CN114006883A - Cross-network data penetration interaction method, device, equipment and storage medium - Google Patents
Cross-network data penetration interaction method, device, equipment and storage medium Download PDFInfo
- Publication number
- CN114006883A CN114006883A CN202111201791.0A CN202111201791A CN114006883A CN 114006883 A CN114006883 A CN 114006883A CN 202111201791 A CN202111201791 A CN 202111201791A CN 114006883 A CN114006883 A CN 114006883A
- Authority
- CN
- China
- Prior art keywords
- data
- center
- cross
- adapter
- interaction
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Granted
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L61/00—Network arrangements, protocols or services for addressing or naming
- H04L61/09—Mapping addresses
- H04L61/10—Mapping addresses of different types
- H04L61/106—Mapping addresses of different types across networks, e.g. mapping telephone numbers to data network addresses
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L41/00—Arrangements for maintenance, administration or management of data switching networks, e.g. of packet switching networks
- H04L41/02—Standardisation; Integration
- H04L41/0226—Mapping or translating multiple network management protocols
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Landscapes
- Engineering & Computer Science (AREA)
- Computer Networks & Wireless Communication (AREA)
- Signal Processing (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a method, a device, equipment and a storage medium for cross-network data penetration interaction. On one hand, the method for cross-network data penetration interaction provides a concept of an adapter to adapt to different data sources with commonalities, and data of each data island can be taken in real time due to adaptation of all market universal data sources; on the other hand, the method and the device have the advantages that the forwarding center is arranged, the penetration strategy is simplified, the problem that the penetration strategy is increased due to the increase of point to point is avoided, the problem of data circulation is mainly solved, fewer IP are solidified, the change safety strategy is reduced, the data circulation is safer, the cross-domain interaction of the latest data can be carried out rapidly in real time, and the problem of data isolated island is solved.
Description
Technical Field
The present application relates to the field of information interaction technologies, and in particular, to a method, an apparatus, a device, and a storage medium for cross-network data penetration interaction.
Background
In the communication field, a domain represents a physically independent local area network, the domain is divided into security levels, and the domain with high security level can access the domain with low security level. Cross-domain penetration refers to the interaction of two physically isolated local area network data, for example: the industry backgrounds of banks, public security and the like can set networks of a plurality of domains, and the fields need cross-domain data interaction requirements to guarantee data safety. However, the stored data cannot be associated due to the fact that the network is obstructed, a mature scheme for supporting cross-domain interaction is not provided in the prior art, the traditional scheme is based on functions of importing and exporting, multiple departments need to examine and approve the data interaction layer by layer each time of the cross-domain interaction, the efficiency is low, resource waste is caused, and the defects include:
1. problem of adaptation of data sources
The data diversity stored in different domains comprises a traditional database, a big data platform, a gallery and the like, and different data sources need to be customized to be adapted by different adapters.
2. Problem of network obstruction
The cross-network jumping machine is used as an agent, and normal data can be handed over to the two-domain data penetration situation of one network only by the fact that multi-layer support is stable.
Therefore, it is necessary to quickly perform cross-domain interaction on the latest data in real time.
Disclosure of Invention
On one hand, the application provides a cross-network data penetration interaction method to solve the technical problems of low efficiency, high cost and information isolated island caused by the existing cross-domain interaction.
The technical scheme adopted by the application is as follows:
a method of cross-network data penetration interaction, comprising the steps of:
after receiving a cross-network data penetration interaction task sent by a client, a scheduling center respectively sends the cross-network data penetration interaction task to an adapter and a forwarding center deployed in a local domain, wherein the adapter is used for adapting to the service of a big data center serving as a data target place or a data source place in the local domain, reading the data of the big data center, and compressing the data into a compressed file for data transmission;
the forwarding center transmits the cross-network data penetrating interaction task to an adapter deployed in a heterogeneous domain to be interacted, wherein the adapter is used for adapting the service of a big data center serving as a data target place or a data source place in the heterogeneous domain, reading the data of the big data center, and compressing the data into a compressed file for data transmission;
the adapter of the data target place acquires corresponding data from the big data center of the data target place according to the cross-network data penetration interaction task and sends the acquired corresponding data and state information to the forwarding center, and meanwhile, the adapter of the data source place acquires corresponding data from the big data center of the data source place according to the cross-network data penetration interaction task and sends corresponding state information to the forwarding center;
the forwarding center transmits corresponding data acquired from a data target place to an adapter of a data source place, the adapter of the data source place performs memory-based MapReduce calculation on the acquired corresponding data and the corresponding data acquired from the data target place, and pushes a calculation result to the forwarding center;
and the forwarding center pushes the calculation result to an adapter of a data target place, and the adapter of the data target place writes the calculation result into a big data center of the data target place, so that the data target place obtains a desired interaction result.
Further, the big data center refers to a data source for storing data in a domain, the data source adopts a big data cluster, and the big data cluster comprises Huacheng big data, CDH and HDP.
Further, the big data center refers to a data source for storing data in a domain, and the data source adopts a traditional database which comprises oracle, mysql and mp.
Further, the scheduling center receives a cross-network data penetration interaction task sent by the client through a rest interface, and the rest interface is used for standardizing how the application performs data interaction with the API provider on the HTTP layer.
Further, the content of the cross-network data penetration interaction task comprises a data source sql and a calculation result sql.
Further, after receiving the cross-network data penetration interaction task sent by the client, the scheduling center sends the cross-network data penetration interaction task to an adapter and a forwarding center deployed in the local domain, and the forwarding center sends the cross-network data penetration interaction task to a message queue and takes the task through the message queue.
Another aspect of the present application further provides an apparatus for cross-network data penetration interaction, including:
the local domain task sending module is used for respectively sending the cross-network data penetrating interactive tasks to an adapter and a forwarding center which are deployed in the local domain after the scheduling center receives the cross-network data penetrating interactive tasks which are sent by a client through a rest interface, wherein the content of the cross-network data penetrating interactive tasks comprises a data source sql and a calculation result sql, the adapter is used for adapting the service of a large data center serving as a data target place or a data source place in the local domain, the data of the large data center is read through the sql, and the data is compressed into a compressed file for data transmission;
the system comprises a different domain task sending module, a forwarding center and a data transmission module, wherein the different domain task sending module is used for the forwarding center to send a cross-network data penetrating interaction task to an adapter deployed in a different domain to be interacted, the adapter is used for adapting the service of a big data center serving as a data target place or a data source place in the different domain, the data of the big data center is read through sql, and compressed into a compressed file for data transmission;
the data acquisition module is used for acquiring corresponding data from the big data center of the data target site by the adapter of the data target site according to the cross-network data penetration interaction task and sending the acquired corresponding data and state information to the forwarding center, and meanwhile, the adapter of the data source site acquires corresponding data from the big data center of the data source site according to the cross-network data penetration interaction task and sends corresponding state information to the forwarding center;
the data calculation module is used for the forwarding center to transmit corresponding data acquired from a data target place to an adapter of a data source place, the adapter of the data source place performs memory-based MapReduce calculation on the acquired corresponding data and the corresponding data acquired from the data target place, and pushes a calculation result to the forwarding center;
and the data writing module is used for pushing the calculation result to an adapter of a data target place by the forwarding center, and writing the calculation result into a big data center of the data target place by the adapter of the data target place so that the data target place obtains a desired interaction result.
In another aspect, the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, wherein the processor implements the steps of the method for cross-network data penetration interaction when executing the program.
In another aspect, the present application further provides a storage medium, where the storage medium includes a stored program, and when the program runs, the apparatus on which the storage medium is located is controlled to perform the steps of the method for cross-network data penetration interaction.
Compared with the prior art, the method has the following beneficial effects:
1. the method starts from the structure of the model, the complex task is interfaced in the process, and the demander transmits the parameters to the interface, so that the desired data result set can be taken quickly.
2. The method and the device have the advantages that the complex scene is streamlined, the practicability of the adapter is improved, the adapter can be adapted to various complex data sources with commonalities, and parallelization development can be effectively realized;
3. this application has set up the forwarding center, the effect of forwarding center can pierce through the tactics simplization with data, avoids increasing the problem of piercing through the tactics complexity because of point-to-point increase, only with effective port fixed open, solidified IP less, reduced the change security policy, improve the interactive security of data, make the data circulation safer.
4. The method can quickly carry out cross-domain interaction on the latest data in real time, and solve the problem of data isolated island.
In addition to the objects, features and advantages described above, other objects, features and advantages will be apparent from the present application. The present application will now be described in further detail with reference to the accompanying drawings.
Drawings
The accompanying drawings, which are incorporated in and constitute a part of this application, illustrate embodiments of the application and, together with the description, serve to explain the application and are not intended to limit the application. In the drawings:
fig. 1 is a flowchart illustrating a method for cross-network data penetration interaction according to a preferred embodiment of the present application.
Fig. 2 is a schematic data interaction diagram of a cross-network data penetration interaction method according to a preferred embodiment of the present application.
Fig. 3 is a flow chart of a method for cross-network data penetration interaction according to another preferred embodiment of the present application.
Fig. 4 is a schematic data interaction diagram of a cross-network data penetration interaction method according to another preferred embodiment of the present application.
Fig. 5 is a schematic diagram of data interaction of a forwarding center in the preferred embodiment of the present application.
Fig. 6 is a schematic diagram of data interaction when a forwarding center is not provided.
Fig. 7 is a block diagram of an apparatus for cross-network data penetration interaction according to the preferred embodiment of the present application.
Fig. 8 is a schematic block diagram of an electronic device entity of the preferred embodiment of the present application.
Fig. 9 is an internal structural view of a computer device of the preferred embodiment of the present application.
Detailed Description
It should be noted that the embodiments and features of the embodiments in the present application may be combined with each other without conflict. The present application will be described in detail below with reference to the embodiments with reference to the attached drawings.
Cross-network data penetration interaction:
the model belongs to two owners, the data related to model reasoning is divided into local data and allopatric data, the local data must penetrate through allopatric interaction according to the requirement of the first part, the result penetrates back to the local, and two scene conditions are shared:
assume a common two domains, domain A and domain B
1. Domain A sends a request, penetrates to domain B, which penetrates the result to domain A
2. Domain B sends a request, penetrates to domain A, which penetrates the result to domain B
Related art terminology interpretation definitions:
a rest interface: to specify how applications interact with API providers at the HTTP layer.
An adapter: and the service for adapting the data source can read the data of the data source through sql and compress the data into a compressed file for data transmission.
The forwarding center: for interacting data with an adapter, comprising:
1. the local domain data is sent to the forwarding center, and the forwarding center sends the local domain data to the different domain adapter.
2. And the different domain data is sent to the forwarding center, and the forwarding center sends the different domain data to the local domain adapter.
The dispatching center: the method is used for sending scheduling instructions to the adapter, including sending tasks and recording task states.
Referring to fig. 1 and 2, a preferred embodiment of the present application provides a method for cross-network data penetration interaction, which is used for realizing that a domain a takes an interaction result from a domain B, and comprises the following steps:
s1, after receiving a cross-network data penetration interaction task sent by a client through a rest interface, a scheduling center respectively sends the cross-network data penetration interaction task to an adapter A and a forwarding center deployed in a domain A, wherein the adapter A is used for adapting to the service of a big data center A of a data target place city A in the domain A, reading the data of the big data center A through sql, and compressing the data into a compressed file for data transmission;
s2, the forwarding center sends the cross-network data penetrating interaction task to an adapter B deployed in a domain B to be interacted, the adapter B is used for adapting the service of a big data center B of a city B serving as a data source in the domain B, the data of the big data center B is read through sql, and the data is compressed into a compressed file for data transmission;
s3, the adapter A of the data target place city A acquires corresponding data A from the big data center A of the data target place city A according to the cross-network data penetration interaction task, and sends the acquired corresponding data A and the state information to the forwarding center, meanwhile, the adapter B of the data source place city B acquires corresponding data B from the big data center B of the data source place city B according to the cross-network data penetration interaction task, and sends the corresponding state information to the forwarding center;
s4, the forwarding center transmits the corresponding data A acquired from the data target place city A to the adapter B of the data source place city B;
s5, the adapter B of the data source place city B performs memory-based MapReduce calculation on the obtained corresponding data B and the corresponding data A obtained from the data target place city A (MapReduce is a high-performance parallel calculation frame based on a cluster), and pushes the calculation result to the forwarding center;
s6, the forwarding center pushes the calculation result to an adapter A of a data target place city A;
and S7, the adapter A of the data target place city A writes the calculation result into the big data center A of the data target place city A, so that the data target place city A obtains the desired interaction result.
Specifically, the big data center a and the big data center B refer to data sources for storing data in domains, the data sources adopt big data clusters, and the big data clusters include Huacheng big data, CDH, and HDP. Or the data source adopts a traditional database, wherein the traditional database comprises oracle, mysql and mpp.
Specifically, the scheduling center receives a cross-network data penetration interaction task sent by a client through a rest interface, and the rest interface is used for standardizing how an application performs data interaction with an API (application programming interface) provider on an HTTP (hyper text transport protocol) layer. Specifically, the content of the cross-network data penetration interaction task comprises a data source sql and a calculation result sql, wherein sql is standard sql99, the data source sql is used for querying and extracting required data, and the calculation result sql is used for calculating queried and extracted data, such as cross-linking, merging and the like.
Referring to fig. 3 and 4, another preferred embodiment of the present application provides a method for cross-network data penetration interaction, which is used for implementing that domain B takes an interaction result from domain a, and comprises the following steps: the method comprises the following steps:
s1, after receiving a cross-network data penetration interaction task sent by a client through a rest interface, a scheduling center respectively sends the cross-network data penetration interaction task to an adapter A and a forwarding center deployed in a domain A, wherein the adapter A is used for adapting the service of a big data center A of a data source city A in the domain A, reading the data of the big data center A through sql, and compressing the data into a compressed file for data transmission;
s2, the forwarding center sends the cross-network data penetrating interaction task to an adapter B deployed in a domain B to be interacted, the adapter B is used for adapting the service of a big data center B serving as a data target city B in the domain B, the data of the big data center B is read through sql, and the data is compressed into a compressed file for data transmission;
s3, the adapter B of the data target place city B acquires corresponding data B from the big data center B of the data target place city B according to the cross-network data penetration interaction task, and sends the data B and the state information to the forwarding center; meanwhile, the adapter A of the city A of the data source place acquires corresponding data A from a big data center A of the city A of the data source place according to the cross-network data penetration interaction task and sends corresponding state information to a forwarding center;
s4, the forwarding center transmits the corresponding data B acquired from the data target place city B to the adapter A of the data source place city A;
s5, the adapter A of the data source place city A carries out MapReduce calculation based on the memory on the obtained corresponding data A and the corresponding data B obtained from the data target place city B, and pushes the calculation result to the forwarding center;
s6, the forwarding center pushes the calculation result to an adapter B of a data target place city B;
and S7, writing the calculation result into the big data center B of the data target place city B by the adapter B of the data target place city B, and enabling the data target place city B to obtain the desired interaction result.
The method for cross-network data penetration interaction of the embodiment comprises the following advantages:
1. from the structure of the model, the complex tasks are interfaced in the process, and the demander transmits the parameters to the interface, so that a desired data result set can be quickly obtained.
2. The complex scene is streamlined, the adapter is adapted to various complex data sources, and the parallel development can be effectively realized.
3. The effect of the forwarding center can simplify the data penetration strategy, only the effective port is fixedly opened, and the safety is improved.
As shown in fig. 5, when the forwarding center is provided, the forwarding center and the city B port include:
20667: data transmission port, IP check;
20668: message communication port, IP check;
the forwarding center can interact data with a plurality of same-domain cities of the domain A only by opening a plurality of fixed ports with the city B in the domain B.
As shown in fig. 6, if there is no forwarding center, a port needs to be added or an IP check needs to be added every time a city of domain a is added, and after there is a forwarding center, the penetration policy only needs to be done once.
The foregoing embodiments of the present application provide an adapter for adapting to different types of data sources having a common property, for example:
data source commonality and adapter development:
oracle: oracle adapter
Mysql: mysql adapter
…
Hua is big data: huacheng big data adapter
CDH: CDh adapter
…
The data sources of each city are diverse, but the data sources are generally stored by a specific database or a large data platform, and the data sources are adapted to all market universal data sources, so that the data of each data island can be taken in real time, and the first step of data interaction calculation is solved.
In addition, the method and the device have the advantages that the forwarding center is arranged, the penetration strategy is simplified, the problem that the penetration strategy is increased due to the increase of point-to-point is solved, the problem of data circulation is mainly solved, fewer IP are solidified, the change of security strategies is reduced, and the data circulation is safer.
As shown in fig. 7, another embodiment of the present application further provides an apparatus for cross-network data penetration interaction, including:
the local domain task sending module is used for respectively sending the cross-network data penetrating interactive tasks to an adapter and a forwarding center which are deployed in the local domain after the scheduling center receives the cross-network data penetrating interactive tasks which are sent by a client through a rest interface, wherein the content of the cross-network data penetrating interactive tasks comprises a data source sql and a calculation result sql, the adapter is used for adapting the service of a large data center serving as a data target place or a data source place in the local domain, the data of the large data center is read through the sql, and the data is compressed into a compressed file for data transmission;
the system comprises a different domain task sending module, a forwarding center and a data transmission module, wherein the different domain task sending module is used for the forwarding center to send a cross-network data penetrating interaction task to an adapter deployed in a different domain to be interacted, the adapter is used for adapting the service of a big data center serving as a data target place or a data source place in the different domain, the data of the big data center is read through sql, and compressed into a compressed file for data transmission;
the data acquisition module is used for acquiring corresponding data from the big data center of the data target site by the adapter of the data target site according to the cross-network data penetration interaction task and sending the acquired corresponding data and state information to the forwarding center, and meanwhile, the adapter of the data source site acquires corresponding data from the big data center of the data source site according to the cross-network data penetration interaction task and sends corresponding state information to the forwarding center;
the data calculation module is used for the forwarding center to transmit corresponding data acquired from a data target place to an adapter of a data source place, the adapter of the data source place performs memory-based MapReduce calculation on the acquired corresponding data and the corresponding data acquired from the data target place, and pushes a calculation result to the forwarding center;
and the data writing module is used for pushing the calculation result to an adapter of a data target place by the forwarding center, and writing the calculation result into a big data center of the data target place by the adapter of the data target place so that the data target place obtains a desired interaction result.
The various modules in the above-described apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The modules can be embedded in a hardware form or independent from a processor in the computer device, and can also be stored in a memory in the computer device in a software form, so that the processor can call and execute operations corresponding to the modules.
As shown in fig. 8, the preferred embodiment of the present application further provides an electronic device, which includes a memory, a processor, and a computer program stored in the memory and executable on the processor, and when the processor executes the computer program, the processor implements the method of cross-network data penetration interaction in the above embodiments.
As shown in fig. 9, the preferred embodiment of the present application also provides a computer device, which may be a terminal or a biopsy server, and the internal structure thereof may be as shown in fig. 9. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device comprises a nonvolatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of an operating system and computer programs in the non-volatile storage medium. The network interface of the computer device is used for communicating with other external computer devices through network connection. The computer program is executed by a processor to implement the above-described method of cross-network data penetration interaction.
Those skilled in the art will appreciate that the architecture shown in fig. 9 is merely a block diagram of some of the structures associated with the disclosed aspects and is not intended to limit the computing devices to which the disclosed aspects apply, as particular computing devices may include more or less components than those shown, or may combine certain components, or have a different arrangement of components.
The preferred embodiment of the present application also provides a storage medium, where the storage medium includes a stored program, and when the program runs, the apparatus where the storage medium is located is controlled to execute the method for cross-network data penetration interaction in the foregoing embodiments.
It should be noted that the steps illustrated in the flowcharts of the figures may be performed in a computer system such as a set of computer-executable instructions and that, although a logical order is illustrated in the flowcharts, in some cases, the steps illustrated or described may be performed in an order different than presented herein.
If the functions of the method of the present embodiment are implemented in the form of software functional units and sold or used as independent products, the functions may be stored in one or more storage media readable by a computing device. Based on such understanding, part of the contribution to the prior art of the embodiments of the present application or part of the technical solution may be embodied in the form of a software product stored in a storage medium and including several instructions for causing a computing device (which may be a personal computer, a server, a mobile computing device or a network device) to execute all or part of the steps of the method described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a Read-Only Memory (ROM), a Random Access Memory (RAM), a magnetic disk or an optical disk, and other various media capable of storing program codes.
The above description is only for the purpose of illustrating the preferred embodiments of the present application and is not to be construed as limiting the present application, and any modifications, equivalents, improvements, etc. made within the spirit and principle of the present application should be included in the scope of the present application.
Claims (9)
1. A method for cross-network data penetration interaction, comprising the steps of:
after receiving a cross-network data penetration interaction task sent by a client, a scheduling center respectively sends the cross-network data penetration interaction task to an adapter and a forwarding center deployed in a local domain, wherein the adapter is used for adapting to the service of a big data center serving as a data target place or a data source place in the local domain, reading the data of the big data center, and compressing the data into a compressed file for data transmission;
the forwarding center transmits the cross-network data penetrating interaction task to an adapter deployed in a heterogeneous domain to be interacted, wherein the adapter is used for adapting the service of a big data center serving as a data target place or a data source place in the heterogeneous domain, reading the data of the big data center, and compressing the data into a compressed file for data transmission;
the adapter of the data target place acquires corresponding data from the big data center of the data target place according to the cross-network data penetration interaction task and sends the acquired corresponding data and state information to the forwarding center, and meanwhile, the adapter of the data source place acquires corresponding data from the big data center of the data source place according to the cross-network data penetration interaction task and sends corresponding state information to the forwarding center;
the forwarding center transmits corresponding data acquired from a data target place to an adapter of a data source place, the adapter of the data source place performs memory-based MapReduce calculation on the acquired corresponding data and the corresponding data acquired from the data target place, and pushes a calculation result to the forwarding center;
and the forwarding center pushes the calculation result to an adapter of a data target place, and the adapter of the data target place writes the calculation result into a big data center of the data target place, so that the data target place obtains a desired interaction result.
2. The method of cross-network data penetration interaction of claim 1,
the big data center refers to a data source for storing data in a domain, the data source adopts a big data cluster, and the big data cluster comprises Huacheng big data, CDH and HDP.
3. The method of cross-network data penetration interaction of claim 1,
the big data center refers to a data source for storing data in a domain, and the data source adopts a traditional database which comprises oracle, mysql and mpp.
4. The method of cross-network data penetration interaction of claim 1,
the scheduling center receives a cross-network data penetration interaction task sent by a client through a rest interface, and the rest interface is used for standardizing how application performs data interaction with an API (application programming interface) provider on an HTTP (hyper text transport protocol) layer.
5. The method of cross-network data penetration interaction of claim 1,
the content of the cross-network data penetration interaction task comprises a data source sql and a calculation result sql.
6. The method for cross-network data penetration interaction according to claim 1, wherein after receiving a cross-network data penetration interaction task sent by a client, the scheduling center sends the cross-network data penetration interaction task to an adapter and a forwarding center deployed in a local domain, and the forwarding center sends the cross-network data penetration interaction task to a message queue and takes the task through the message queue.
7. An apparatus for data-penetrating interaction across a network, comprising:
the local domain task sending module is used for respectively sending the cross-network data penetrating interactive tasks to an adapter and a forwarding center which are deployed in the local domain after the scheduling center receives the cross-network data penetrating interactive tasks which are sent by a client through a rest interface, wherein the content of the cross-network data penetrating interactive tasks comprises a data source sql and a calculation result sql, the adapter is used for adapting the service of a large data center serving as a data target place or a data source place in the local domain, the data of the large data center is read through the sql, and the data is compressed into a compressed file for data transmission;
the system comprises a different domain task sending module, a forwarding center and a data transmission module, wherein the different domain task sending module is used for the forwarding center to send a cross-network data penetrating interaction task to an adapter deployed in a different domain to be interacted, the adapter is used for adapting the service of a big data center serving as a data target place or a data source place in the different domain, the data of the big data center is read through sql, and compressed into a compressed file for data transmission;
the data acquisition module is used for acquiring corresponding data from the big data center of the data target site by the adapter of the data target site according to the cross-network data penetration interaction task and sending the acquired corresponding data and state information to the forwarding center, and meanwhile, the adapter of the data source site acquires corresponding data from the big data center of the data source site according to the cross-network data penetration interaction task and sends corresponding state information to the forwarding center;
the data calculation module is used for the forwarding center to transmit corresponding data acquired from a data target place to an adapter of a data source place, the adapter of the data source place performs memory-based MapReduce calculation on the acquired corresponding data and the corresponding data acquired from the data target place, and pushes a calculation result to the forwarding center;
and the data writing module is used for pushing the calculation result to an adapter of a data target place by the forwarding center, and writing the calculation result into a big data center of the data target place by the adapter of the data target place so that the data target place obtains a desired interaction result.
8. An electronic device comprising a memory, a processor, and a computer program stored on the memory and executable on the processor,
the processor, when executing the program, performs the steps of the method for cross-network data penetration interaction of any one of claims 1 to 5.
9. A storage medium including a stored program, characterized in that,
controlling a device on which the storage medium is located to perform the steps of the method of cross-network data penetration interaction as claimed in any one of claims 1 to 5 when the program is run.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111201791.0A CN114006883B (en) | 2021-10-15 | 2021-10-15 | Cross-network data penetration interaction method, device, equipment and storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202111201791.0A CN114006883B (en) | 2021-10-15 | 2021-10-15 | Cross-network data penetration interaction method, device, equipment and storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114006883A true CN114006883A (en) | 2022-02-01 |
| CN114006883B CN114006883B (en) | 2023-06-27 |
Family
ID=79922928
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202111201791.0A Active CN114006883B (en) | 2021-10-15 | 2021-10-15 | Cross-network data penetration interaction method, device, equipment and storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114006883B (en) |
Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110283017A1 (en) * | 2010-05-14 | 2011-11-17 | Microsoft Corporation | Interconnecting Members of a Virtual Network |
| US20120042033A1 (en) * | 2010-08-13 | 2012-02-16 | International Business Machines Corporation | Migrating virtual machines across network separated data centers |
| US20150163179A1 (en) * | 2013-12-09 | 2015-06-11 | Hewlett-Packard Development Company, L.P. | Execution of a workflow that involves applications or services of data centers |
| US20180191856A1 (en) * | 2016-12-29 | 2018-07-05 | Synology Inc. | Cross-domain communication methods and proxy servers using the same |
| US10326830B1 (en) * | 2016-09-02 | 2019-06-18 | Amazon Technologies, Inc. | Multipath tunneling to a service offered at several datacenters |
| CN110502559A (en) * | 2019-07-25 | 2019-11-26 | 浙江公共安全技术研究院有限公司 | A kind of data/address bus and transmission method of credible and secure cross-domain data exchange |
| WO2021057198A1 (en) * | 2019-09-27 | 2021-04-01 | 北京市天元网络技术股份有限公司 | Big data-based cross-domain service whole-process routing and penetration method and apparatus |
| US11121989B1 (en) * | 2020-05-29 | 2021-09-14 | Bank Of America Corporation | Centralized repository and communication system for cross-network interactions |
| CN113422837A (en) * | 2021-07-02 | 2021-09-21 | 广东电网有限责任公司 | Cross-network data transmission device, method, terminal and readable storage medium |
-
2021
- 2021-10-15 CN CN202111201791.0A patent/CN114006883B/en active Active
Patent Citations (9)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US20110283017A1 (en) * | 2010-05-14 | 2011-11-17 | Microsoft Corporation | Interconnecting Members of a Virtual Network |
| US20120042033A1 (en) * | 2010-08-13 | 2012-02-16 | International Business Machines Corporation | Migrating virtual machines across network separated data centers |
| US20150163179A1 (en) * | 2013-12-09 | 2015-06-11 | Hewlett-Packard Development Company, L.P. | Execution of a workflow that involves applications or services of data centers |
| US10326830B1 (en) * | 2016-09-02 | 2019-06-18 | Amazon Technologies, Inc. | Multipath tunneling to a service offered at several datacenters |
| US20180191856A1 (en) * | 2016-12-29 | 2018-07-05 | Synology Inc. | Cross-domain communication methods and proxy servers using the same |
| CN110502559A (en) * | 2019-07-25 | 2019-11-26 | 浙江公共安全技术研究院有限公司 | A kind of data/address bus and transmission method of credible and secure cross-domain data exchange |
| WO2021057198A1 (en) * | 2019-09-27 | 2021-04-01 | 北京市天元网络技术股份有限公司 | Big data-based cross-domain service whole-process routing and penetration method and apparatus |
| US11121989B1 (en) * | 2020-05-29 | 2021-09-14 | Bank Of America Corporation | Centralized repository and communication system for cross-network interactions |
| CN113422837A (en) * | 2021-07-02 | 2021-09-21 | 广东电网有限责任公司 | Cross-network data transmission device, method, terminal and readable storage medium |
Non-Patent Citations (3)
| Title |
|---|
| YANG LU 等: "Cross-area Node Selection Algorithm for Data Replica Storing in Cloud Data Centers", IEEE * |
| 王建冬 等: "东数西算：我国数据跨域流通的总体框架和实施路径研究", 电子政务, no. 03 * |
| 胡腾 等: "面向服务的数据中心安全框架", 电信科学, no. 1 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114006883B (en) | 2023-06-27 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110958218B (en) | Data transmission method based on multi-network communication and related equipment | |
| CN102902805B (en) | A kind of page access method and apparatus | |
| CN110554927A (en) | Micro-service calling method based on block chain | |
| CN102523207A (en) | VNC (Virtual Network Computer)-based remote resource access method and proxy device | |
| CN109104368B (en) | Connection request method, device, server and computer readable storage medium | |
| WO2017192549A1 (en) | System and method for secure and efficient communication within an organization | |
| CN106027534A (en) | System for implementing financial message processing based on Netty | |
| CN109327511A (en) | A kind of data request method and server based on http protocol | |
| CN114124929A (en) | Cross-network data processing method and device | |
| CN108833500B (en) | Service calling method, service providing method, data transmission method and server | |
| US20190370293A1 (en) | Method and apparatus for processing information | |
| EP3276914A1 (en) | Data sharing method and device for virtual desktop | |
| CN102902593B (en) | Agreement distributing and processing system based on caching mechanism | |
| CN114006883A (en) | Cross-network data penetration interaction method, device, equipment and storage medium | |
| CN104333515B (en) | A kind of SWIFT message processing methods and device | |
| CN108111513B (en) | Data management method, device, medium and electronic equipment applied to front-end device | |
| CN115623057A (en) | RDMA (remote direct memory Access) -based connection establishing method, device, equipment and storage medium | |
| CN115499432A (en) | Family terminal computing resource management system and computing resource scheduling method | |
| CN114780361A (en) | Log generation method, device, computer system and readable storage medium | |
| CN114979144A (en) | Cloud edge communication method and device and electronic equipment | |
| CN106899652A (en) | A kind of method and device of transmission service result | |
| CN107707383B (en) | Put-through processing method and device, first network element and second network element | |
| CN108667863B (en) | WEB service adaptation system and method based on local agent | |
| CN104202374A (en) | Breakpoint resume method for medical image transmission | |
| CN106797403B (en) | For handling the server, client device and method therein of the content resource of cache |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |