CN112241540B - Method and device for improving data interaction safety between enterprises - Google Patents

Abstract

The invention discloses a device and a method for improving the data interaction safety between enterprises, wherein the device is arranged in a DMZ space deployed between an enterprise intranet and an external network; the device comprises: the MQ module is used for organizing external incoming data in a message queue mode; the pretreatment module is used for sequentially carrying out pretreatment on the data organized in the message queue, wherein the pretreatment comprises data cleaning, data integration and data transformation; and the CDC module is used for synchronizing the preprocessed data stream to the enterprise intranet database in real time by adopting a changed data grabbing mode. The device and the method improve the high security and the high stability of data exchange between enterprises.

Description

Method and device for improving data interaction safety between enterprises

Technical Field

The invention relates to the field of communication security, in particular to a method and a device for improving the data interaction security between enterprises.

Background

The informatization degree of the existing enterprises is gradually improved, and the information systems used in the enterprises are more and more. Specialized systems used by various production units in the industry chain each operate independently, each generating and managing a certain amount of data. There is no direct relationship between these data, but rather, as an informative industry chain, data flows around each ring, and data is now an important production resource, so that the whole industry is operating as a whole.

There is in fact an inherent logical link between the data within each production unit. If a complete business process is implemented across units, multiple informationized systems of each unit are often required to cooperate, real-time or non-real-time interaction is required between the systems for the data, and enterprises are also required to integrate and integrate the data with different sources. The security of the data in the transmission process is ensured, and the reliability is the task that the interface between the information systems needs special attention. However, this often means huge development workload and maintenance cost, because if the security construction of the system is guaranteed by data security, a lot of manpower is required to be consumed, and the transmission efficiency of information may be affected, the amount of transmitted data is huge, and effective carding and screening are required, so that the lock list and data redundancy caused by repeated read-write modification of the database are avoided. If a specially converted application is built for each different application database storage, the development complexity will be very high and the development period will be very long. In the prior art, it is difficult to have a system which not only satisfies safety and reliability but also can effectively comb programs, the development cost is high, more maintenance cost is required, the development period is long, factors in various aspects are required to be involved, various problems can exist in practice, the development success rate of a software system is low, the development period is long, and the execution of enterprise projects is not facilitated.

There are many authoritative standards in the development process of information technology safety evaluation criteria, such as trusted computer system evaluation criteria, TCSEC for short, information technology safety evaluation criteria, IT SEC for short, and computer information system safety protection class division criteria.

In TCSECI, computer security is divided from high to low into: A. b, C, D, with the improvement of the security level, the reliability of the system is increased, the risk is gradually reduced, and the TCSEC has four security levels: the protection stage is not needed, the protection stage is independent, the protection stage is forced, and the protection stage is verified. From class C (autonomous protection level), the system has certain protection capability, and the adopted measures are autonomous access control and meter trace, which are generally only applicable to multi-user environments with certain level and have the capability of reporting the responsibility of a main body and the actions of the main body.

For data security of computer systems, authoritative standards have been defined very deeply, and in the "information security technology framework", the computer information system is mainly divided into 4 parts: local computing environments, regional boundaries, networks and infrastructures, supporting infrastructures. The boundary area is the part of the area where information exchange occurs with the external network, and the area boundary ensures that the entering information does not affect the security of the resources in the area and the leaving information is legally authorized, so the security policy of the area boundary is very important and effective.

Disclosure of Invention

The invention aims to provide a method and a device for improving the data interaction safety between enterprises, which are used for improving the high safety and the high stability of the data exchange between enterprises.

In order to achieve the above purpose, the technical scheme provided by the invention is as follows:

a device for improving the data interaction safety between enterprises, wherein the device is arranged in a DMZ space deployed between an enterprise intranet and an external network;

the device comprises:

the MQ module is used for organizing external incoming data in a message queue mode;

the pretreatment module is used for sequentially carrying out pretreatment on the data organized in the message queue, wherein the pretreatment comprises data cleaning, data integration and data transformation;

and the CDC (Change Data Capture) module is used for synchronizing the preprocessed data stream to the enterprise intranet database in real time by adopting a changed data grabbing mode.

The data cleaning comprises the steps of processing missing values, smoothing noise data, identifying or deleting outliers and solving inconsistency. The data integration comprises the steps of combining and uniformly storing data in a plurality of data sources, and establishing a data warehouse. The data transformation includes converting the data into a form for storage by means of smooth aggregation, data generalization.

In the invention, the DMZ space is deployed between the enterprise intranet and the external network, and the device for improving the data interaction safety between enterprises is arranged in the DMZ space and is used for carrying out safety protection and carding on the data exchange between enterprises, thereby improving the high safety and the high stability. The DMZ mode can solve the problem that an access user of an external network cannot access an internal network server after installing a firewall, and a buffer zone between a non-secure system and a secure system is set up. The buffer is located in a small network area between the enterprise internal network and the external network. Within this small network area, server facilities may be placed that must be disclosed, such as enterprise Web servers, FTP servers, forums, etc. On the other hand, with such a DMZ zone, the internal network is more effectively protected because the DMZ zone deployment has one more gateway for an attacker from the external network than a general firewall solution.

In the device for improving the inter-enterprise data interaction safety, the data information transmitted by the MQ module can become orderly and queuing stable, then enter the preprocessing module from the MQ module, and the unique data coding ciphertext is filtered by the computing logic in the MQ module, so that other illegal messages can be directly extracted and bounced back. The preprocessing module comprises a preprocessing algorithm, so that preprocessing of data can be realized, the filtering mode can be changed in a customized mode, the response time and the response efficiency of the system are improved, and other security holes in a DMZ area are effectively made up; the response speed of the preprocessed data stream is greatly improved through a CDC module for capturing the preprocessed data stream in real time.

In one embodiment, when the amount of externally penetrating data is large, the preprocessing module dynamically performs elastic expansion, and synchronously processes through the copy mirror image in a horizontal mode according to the required data processing amount. The total data processing efficiency is greatly improved, the compression resistance of the system is greatly improved, and a large amount of data is prevented from being accumulated in the MQ module.

In another embodiment, when the amount of external incoming data is large, the preprocessing module dynamically performs elastic expansion, and simultaneously processes the data by adding multiple threads.

In another embodiment, the preprocessing module performs priority judgment on the external incoming data, directly preprocesses the external incoming data when the priority of the external incoming data is high, and transmits the external incoming data with low priority to the MQ module.

A method for improving security of data interaction between enterprises, comprising the steps of:

the external data transmitted based on https protocol is successfully checked by account name and password and then is transmitted into the firewall;

the external data is checked by the firewall white list and then is transmitted into the boundary of the area;

the device for improving the data interaction safety between enterprises according to any one of claims 1 to 7 is transmitted into the external data after the digital certificate in the SAP PI is checked successfully;

the external data is stored in the enterprise intranet database after being processed by the device for improving the data interaction safety between enterprises.

Compared with the prior art, the invention has the beneficial effects that at least the following steps are included:

by using the organic combination of various security and data processing technologies, a great deal of work such as preprocessing is carried out on data in a DMZ area, the risk is controlled outside the intranet, and the intranet of an enterprise is effectively protected. The invention has high safety and usability by combining some very mature prior art with the dynamic preprocessing technology developed independently, and can be regulated in safety strategy according to different requirements. Unlike other data safety devices in the past, the invention dynamically controls the risk, improves the safety, simultaneously greatly improves the efficiency of data transmission and the use stability of a database, avoids system paralysis and the lock table of the database caused by overlarge data, meets most important indexes mentioned in the system safety standard, has extremely high safety, and is beneficial to the safety and the integrity of data transmission among enterprises.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings that are required in the embodiments or the description of the prior art will be briefly described, and it is obvious that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.

FIG. 1 is a schematic diagram of inter-enterprise data interactions provided by an embodiment;

fig. 2 is a schematic diagram of decoupling of an apparatus for improving security of data interaction between enterprises according to an embodiment.

Detailed Description

The present invention will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present invention more apparent. It should be understood that the detailed description is presented by way of example only and is not intended to limit the scope of the invention.

Based on the extension of enterprise batch management ideas and management, each cooperation unit in the industry develops the construction of a batch system, and the batch data generated by each enterprise needs to be ensured to be safely and efficiently transmitted to the enterprise. Therefore, the invention provides a method for improving the data interaction safety between enterprises, which organically combines a plurality of safety measures, brings out the best in each other for some mature technologies and autonomous research and development technologies, crossly improves the safety and reliability in transmission, builds a platform of a system by means of the existing mature technologies, and uses the autonomous research and development technologies to make up for loopholes and improve efficiency.

The data interaction safety among enterprises mainly realizes layer-by-layer protection of 5 safety measures, so that the data is very safe and reliable in the circulation process. The 5 components are respectively: https transmission protocol, account and password, firewall whitelists, digital certificates, and devices for improving the security of data interactions between enterprises as shown in figure 2. Based on the above, the method for improving the data interaction security between enterprises comprises the following processes:

firstly, based on the Https transmission protocol, when data enter from the outside, the data needs to pass through the corresponding verification of a unique account name and password, after going forward, the data passes through the firewall, the specific source (xx factory/xx server) of the data is defined through the firewall white list verification, if the verification is correct, the data can be transmitted forward continuously, two security measures are mature, efficient and stable technical application, double identity verification can effectively prevent unauthorized data tampering, wiretapping and stealing, defraud, replay attack and the like, and stronger repudiation and confidentiality are shown.

After passing through the firewall, the data enters the region boundary, firstly undergoes digital certificate verification in SAP WEB PI, and can confirm the authorization of the identity again through the digital certificate, thereby ensuring the credibility.

The digital certificate is a relatively authoritative and fair certificate issued by an electronic commerce authentication center, adopts a digital certificate authentication technology taking a digital encryption technology as a core, can encrypt, decrypt, digitally sign and authenticate various kinds of information transmitted on the Internet through the digital certificate, and can also ensure that illegal molecules cannot invade in the digital transmission process or cannot view contents even if the illegal molecules invade. If the user installs the digital certificate in the process of electronic commerce, even if personal information such as an account or a password is stolen, the information and fund security in the account can be effectively ensured. The user can prove the identity of the user through the digital certificate and recognize the identity of the other party when the electronic commerce is carried out. Currently only about 30 businesses nationwide qualify as practitioners of digital authentication. The data flow is firstly checked by the digital certificate to have certain reliability, and then the transmission structure is optimized by means of the integration characteristic of the SAP PI, so that the problems that some data are not instant in transmission, data management is difficult, data adjustment is difficult in the later period and the like are solved, certain safety is provided, and the data transmission is ensured to be correct. And realizing real-time data transmission through Web service. By adopting the safety measure, not only the safety is improved, but also the information management capability of the system is improved, and various information management and inquiry modes are provided, so that the operation and management are convenient. The method can ensure the accuracy of information going to the next step, and can also have an automatic retransmission mechanism if transmission errors occur, and provide a common interface (Adapter) type (A2A/B2B), so as to reduce the interface consideration and development consideration of information transmission between systems.

The data after passing through the SAP Web PI enters a device for improving the data interaction security between enterprises as shown in figure 2. The system consists of 3 parts, namely an MQ module, a preprocessing module and a CDC module. Each of the parts is described in detail below.

1) MQ module

The data first enters the MQ module, and the message queue has become the core means for the internal communication of the enterprise IT system. The system has a series of functions of low coupling, reliable delivery, broadcasting, flow control, final consistency and the like, and becomes one of main means of asynchronous RPC. The interactive systems have no direct calling relation, and only through message transmission, so the system has low invasiveness and low coupling degree. After passing through the MQ message service assembly, the data information transmitted in becomes orderly and queuing stable, then enters the preprocessing module from the data information, and the unique data coding ciphertext is filtered through the calculation logic in the data coding ciphertext, and other illegal messages can be directly extracted and bounced back.

2) Pretreatment module

The main functions of user permission verification such as a firewall, a digital certificate and the like are access validity check, corresponding permission is given to an access subnet or a user according to a permission configuration file, illegal access requests are refused, but the system cannot prevent the problem of distorted data and data loss in legal access. Distorted data or incomplete data can cause errors in data storage and data analysis. Therefore, the preprocessing module can realize preprocessing of data and change the filtering mode in a customized way, improve the response time and the response efficiency of the system, and effectively compensate other security holes in the DMZ area.

The preprocessing module mainly has the functions of data preprocessing, mirror image copying and synchronous processing, priority setting and dynamic capacity expansion.

Aiming at the problems of incomplete, inconsistent, strong dispersibility and the like of production process data transmitted by a plurality of cooperative factories, data preprocessing work is needed to be carried out for improving the data quality, and data sources are processed by methods of data cleaning, data integration, data transformation and the like. The data cleaning comprises the steps of processing missing values, smoothing noise data, identifying or deleting outliers and solving inconsistency; the data integration comprises the steps of combining and uniformly storing data in a plurality of data sources, and establishing a data warehouse; data transformation includes converting data into a suitable form by way of smooth aggregation, data generalization, and the like.

Aiming at the duplication mirror image and synchronous processing, when the data transmission quantity is huge, the preprocessing module dynamically performs elastic expansion to improve the operation efficiency, and the synchronous processing is performed through the duplication module mirror image in a horizontal mode according to the required processing quantity dynamic, so that the total data processing efficiency is greatly improved, the compression resistance of the system is greatly improved, and the data is prevented from being accumulated in the MQ module in a large quantity.

For setting priority, in order to effectively improve the response time of the system, firstly, the sequence is guaranteed, and in most use cases, the sequence of data processing is important. Most message queues are inherently ordered and can ensure that data will be processed in a particular order. Secondly, according to the system requirement, the specific processing priority sequence, such as an original set of information processing system, is realized by a plurality of logics of the whole system; through the design of the preprocessing device architecture, urgent important (needing immediate response) items can be put into the preprocessing module for preferential treatment, the message queues with low response requirements are put into the MQ queues, a part of illegal messages are filtered through preprocessing screening, and the response speed is greatly improved through the CDC module of real-time grabbing of the simplified data stream.

Aiming at dynamic capacity expansion, the device provided by the embodiment also provides services for a big data processing architecture, and the message queue is integrated with the real-time processing architecture through message integration under the background of big data, so that performance support is provided for data processing. And support asynchronous communications, some services do not want or need to process messages immediately. Message queues provide an asynchronous processing mechanism that allows a message to be placed in the queue, but does not process it immediately. How many messages are placed in the queue and then processed again when needed. In the case of large volumes of data, redundancy may cause the process of processing the data to fail. Unless the data is persisted, it will be lost. The message queue persists the data until they have been fully processed, and then hands over to preprocessing filtering, in such a way that the risk of data loss is circumvented. In the "insert-acquire-delete" paradigm employed by many message queues, a processing system is required to explicitly indicate that a message has been processed before it is deleted from the queue, thereby ensuring that data is securely saved until the preprocessing is complete and continues to commit forward.

In the case of very large data volumes, the preprocessing module expands the elastic expansion to perform simultaneous multi-threaded processing of data, and because the message queue decouples the processing procedure and increases the expansibility, it is easy to increase the frequency of message enqueuing and processing, so long as the processing procedure is additionally increased. No code change and no parameter adjustment are required. The distributed capacity expansion is convenient to be more beneficial to the horizontal copying of the preprocessing module, and the message queue also provides enough buffer space for the preprocessing module, so that elements requiring different processing time can be arranged in any important system. The message queues may help the preprocessing module and its dynamic level flexible expansion to perform most efficiently by a buffer layer that helps control and optimize the speed of the data stream through the system. In the event of a sudden increase in access volume, the system still needs to continue to function, but such bursty traffic cannot be predicted, and the preprocessing module has sufficient overload protection capability to enable critical components to withstand the bursty access pressure without completely collapsing due to bursty overload requests.

3) CDC module

After preprocessing analysis, the data enters a CDC module, the CDC synchronization technology can identify the data changed after the last extraction, the data can be extracted by utilizing CDC while INSERT, UPDATE or DELETE and other operations are carried out on a source table, and the changed data is stored in a change table of a database. This allows the changed data to be captured and then provided to the target system in a controlled manner using the database view. Through the processing, the data finally transmitted into the intranet database is effectively managed and monitored, bad data diffusion can be effectively prevented, meanwhile, the system is prevented from being damaged, the data is timed, and most importantly, the operation of large-scale concurrent modification and the like on the database table or continuous cross access of different data are avoided, so that the database lock table is caused, the CDC technology can grasp the database log in real time and play back the database log to the internal area of the user, and the data is dynamically and stably in real time during access and modification.

The MQ module, the preprocessing method module and the CDC module are combined to form a device for improving the data interaction safety among enterprises, the preprocessed data is subjected to a large amount of screening, safety detection and arrangement before being transmitted out of the DMZ area, the safety of data transmission is greatly improved, and meanwhile, the data is simplified and light. And the overall operation efficiency is improved, and the stability is increased. The preprocessed data are transmitted to the point by point through the firewall and the intranet again, the data enter the intranet through the device, the safety, reliability, integrity and consistency of the intranet received data can be guaranteed, and the data processing efficiency is improved.

The method for improving the data interaction safety between enterprises effectively protects the enterprise intranet by using the organic combination of various safety and data processing technologies to perform a great deal of work such as preprocessing on data in a DMZ area and the like to control risk outside the intranet. The invention has high safety and usability by combining some very mature prior art with the dynamic preprocessing technology developed independently, and can be regulated in safety strategy according to different requirements. Unlike other data safety devices in the past, the invention dynamically controls the risk, improves the safety, simultaneously greatly improves the efficiency of data transmission and the use stability of a database, avoids system paralysis and the lock table of the database caused by overlarge data, meets most important indexes mentioned in the system safety standard, has extremely high safety, and is beneficial to the safety and the integrity of data transmission among enterprises. The invention has certain referential and popularization and application values for the safe and efficient data transmission among enterprises in tobacco industry or related industries.

The foregoing detailed description of the preferred embodiments and advantages of the invention will be appreciated that the foregoing description is merely illustrative of the presently preferred embodiments of the invention, and that no changes, additions, substitutions and equivalents of those embodiments are intended to be included within the scope of the invention.

Claims (5)

1. The device for improving the data interaction safety between enterprises is characterized in that the device is arranged in a DMZ space deployed between an enterprise intranet and an external network;

the device comprises:

the MQ module is used for organizing external incoming data in a message queue mode;

the pretreatment module is used for sequentially carrying out pretreatment on the data organized in the message queue, wherein the pretreatment comprises data cleaning, data integration and data transformation;

when the external data quantity is large, the preprocessing module dynamically performs elastic expansion, synchronously processes the data through copying mirror images in a horizontal mode according to the required data processing quantity, and simultaneously processes the data by adding multiple threads;

the preprocessing module is used for judging the priority of the external incoming data, directly preprocessing the external incoming data when the priority of the external incoming data is high, and transmitting the external incoming data with low priority to the MQ module;

and the CDC module is used for synchronizing the preprocessed data stream to the enterprise intranet database in real time by adopting a changed data grabbing mode.

2. The apparatus for improving security of data interactions between enterprises of claim 1, wherein the data cleansing includes processing missing values, smoothing noise data, identifying or deleting outliers and resolving inconsistencies.

3. The apparatus for improving security of data interactions between enterprises of claim 1, wherein the data integration comprises combining and uniformly storing data in a plurality of data sources to create a data warehouse.

4. The apparatus for improving security of data interactions between enterprises of claim 1, wherein said data transformation comprises converting data into a form for storage by means of smooth aggregation, data generalization.

5. A method for improving the security of data interaction between enterprises, comprising the steps of:

the external data transmitted based on https protocol is successfully checked by account name and password and then is transmitted into the firewall;

the external data is checked by the firewall white list and then is transmitted into the boundary of the area;

the device for improving the data interaction safety between enterprises according to any one of claims 1 to 4 is transmitted into the external data after the digital certificate in the SAP PI is checked successfully;

the external data is stored in the enterprise intranet database after being processed by the device for improving the data interaction safety between enterprises.