CN113641740A

CN113641740A - Multi-system-based data processing method, device, equipment and storage medium

Info

Publication number: CN113641740A
Application number: CN202110874925.9A
Authority: CN
Inventors: 陈战仁; 肖群华
Original assignee: Ping An Life Insurance Company of China Ltd
Current assignee: Ping An Life Insurance Company of China Ltd
Priority date: 2021-07-30
Filing date: 2021-07-30
Publication date: 2021-11-12

Abstract

The application relates to the field of data processing, and provides a multi-system-based data processing method, a device, equipment and a storage medium, wherein the method comprises the following steps: acquiring system information of all associated systems corresponding to the target service; acquiring a system docking mode of each associated system based on all system information; judging whether the current time is in a preset leisure time period or not; if yes, for each correlation system, writing the service data corresponding to the target service in the correlation system into a preset data conversion layer based on a data processing rule corresponding to a system docking mode of the correlation system; isomorphic conversion processing is carried out on all service data contained in the data conversion layer to obtain processed service data; and extracting the processed service data from the data conversion layer and storing the service data. The method and the device can realize real-time acquisition and summarization of the service data of the target service. The method and the device can also be applied to the field of block chains, and the processed service data can be stored on the block chains.

Description

Multi-system-based data processing method, device, equipment and storage medium

Technical Field

The present application relates to the field of data technologies, and in particular, to a method, an apparatus, a device, and a storage medium for data processing based on multiple systems.

Background

Currently, a service usually includes a plurality of different service channels, so that the same service is connected to a plurality of service systems corresponding to the service channels, and the data format of the service data collected by each service system may also be different. The conventional method for counting and summarizing the service data of the related service generally includes the steps of directly performing independent collection and statistics on the service data in each service system corresponding to the related service, and reporting the collected and counted service data according to requirements after each system completes the collection and statistics on the service data so as to summarize all the service data. Such a data processing method is time-consuming and labor-consuming, cannot realize real-time collection and summarization of all service data of related services, and is low in efficiency and lack of intelligence.

Disclosure of Invention

The application mainly aims to provide a data processing method, a data processing device, computer equipment and a storage medium based on multiple systems, and aims to solve the technical problems that the existing statistical summary mode of business data of related businesses is time-consuming and labor-consuming, real-time collection and summary of all business data of related businesses cannot be realized, the efficiency is low, and intelligence is lacked.

The application provides a data processing method based on multiple systems, which comprises the following steps:

acquiring system information of all associated systems corresponding to the target service;

based on all the system information, acquiring system docking modes respectively corresponding to the associated systems;

acquiring current time, and judging whether the current time is in a preset leisure time period or not;

if yes, for each association system, writing the service data corresponding to the target service in each association system into a preset data conversion layer based on a data processing rule corresponding to a system docking mode of each association system;

performing isomorphic conversion processing on all service data contained in the data conversion layer based on a preset data conversion thread and a data conversion rule to obtain processed service data;

and extracting the processed service data from the data conversion layer and storing the processed service data.

Optionally, the step of writing, for each of the association systems, service data corresponding to the target service in each of the association systems into a preset data conversion layer based on a data processing rule corresponding to a system docking manner of each of the association systems includes:

acquiring a system docking mode of a designated association system; the appointed correlation system is any one of all correlation systems, and the system docking mode of the appointed correlation system is marked as an appointed correlation system docking mode;

if the designated association system docking mode is a KAFKA docking mode, pulling the service data corresponding to the target service in the designated association system to the data conversion layer according to a first preset rule;

if the designated association system is in a golden gate docking mode, calling a preset golden gate tool to synchronize service data corresponding to the target service in the designated association system to the data conversion layer;

if the appointed association system docking mode is an ActiveMQ docking mode, pulling the service data corresponding to the target service in the appointed association system to the data conversion layer according to a second preset rule;

if the designated association system docking mode is a preset first interface docking mode, calling a preset Http interface to write the service data corresponding to the target service in the designated association system into the data conversion layer;

if the designated association system docking mode is a preset second interface docking mode, connecting the designated association system through JDBC, and synchronizing service data corresponding to the target service in the designated association system to the data conversion layer.

Optionally, the step of pulling, according to a first preset rule, service data corresponding to the target service in the designated association system to the data conversion layer includes:

determining whether the specified association system has used KAFKA as a message middleware;

if KAFKA is used as message middleware, a first request for applying consumption authority of a first TOPIC corresponding to the target service is sent to the specified association system;

judging whether the specified association system successfully processes the first request or not;

if the first request is successfully processed, pulling the service data corresponding to the target service contained in the first TOPIC to the data conversion layer based on the consumption right.

Optionally, the step of pulling, according to a second preset rule, service data corresponding to the target service in the designated association system to the data conversion layer includes:

sending a second request for applying subscription authority of a second TOPIC corresponding to the target service to the specified association system;

judging whether the specified association system successfully processes the second request;

if yes, after the appointed association system transmits the broadcast message of the service data corresponding to the target service into the MQ message queue corresponding to the second TOPIC, establishing a connection relation with the MQ message queue in the appointed association system;

after the connection relationship is successfully established, subscribing the broadcast information in the MQ message queue based on the subscription authority;

and pulling the service data corresponding to the target service in the specified correlation system contained in the broadcast information to the data conversion layer.

Optionally, the step of performing isomorphic conversion processing on all service data included in the data conversion layer based on a preset data conversion thread and a preset data conversion rule to obtain processed service data includes:

acquiring pre-stored configuration information, and acquiring a first number of central processing units from the configuration information;

calculating the product of the first quantity and a preset proportion to obtain a second quantity;

creating a plurality of data conversion threads with the same number as the second number in a preset thread pool;

distributing all the service data into each data conversion thread according to a preset distribution rule;

respectively sending the data conversion rules to each data conversion thread;

and simultaneously carrying out isomorphic conversion processing on the service data contained in each data conversion thread based on the data conversion rule through each data conversion thread to obtain the processed service data.

Optionally, the step of judging whether the current time is in the preset leisure time period includes:

dividing the value according to a preset length, and dividing the time of a day into a plurality of time periods;

screening all the time periods based on a preset busy time list, and screening out a first time period from all the time periods; wherein the number of the first time periods is multiple;

counting the total data processing amount of each first time period in a preset time period based on a pre-stored historical data processing record;

screening out a first data processing total amount which is larger than a preset data processing amount threshold value from all the data processing total amounts;

removing the first data processing total amount from all the data processing total amounts to obtain a processed second data processing total amount;

screening out a second time period corresponding to the second data processing total amount from all the first time periods;

taking the second time period as the leisure time period.

Optionally, after the step of storing the processed service data is completed, data acquisition completion information corresponding to the target service is generated;

acquiring preset mail login information; and the number of the first and second groups,

acquiring a target mail address of a target user;

logging in to a corresponding mail server based on the mail login information;

and sending the data acquisition completion information to the target mail address through the mail server.

The present application further provides a data processing apparatus based on multiple systems, including:

the first acquisition module is used for acquiring system information of all associated systems corresponding to the target service;

a second obtaining module, configured to obtain, based on all the system information, system docking manners respectively corresponding to the associated systems;

the judging module is used for acquiring the current time and judging whether the current time is in a preset leisure time period or not;

the first processing module is used for writing the service data corresponding to the target service in each association system into a preset data conversion layer on the basis of a data processing rule corresponding to a system docking mode of each association system if the association system is in the first state;

the second processing module is used for carrying out isomorphic conversion processing on all service data contained in the data conversion layer based on a preset data conversion thread and a data conversion rule to obtain processed service data;

and the third processing module is used for extracting the processed service data from the data conversion layer and storing the processed service data.

The present application further provides a computer device, comprising a memory and a processor, wherein the memory stores a computer program, and the processor implements the steps of the above method when executing the computer program.

The present application also provides a computer-readable storage medium having stored thereon a computer program which, when being executed by a processor, carries out the steps of the above-mentioned method.

The multisystem-based data processing method, the multisystem-based data processing device, the computer equipment and the storage medium have the following beneficial effects:

the data processing method, the data processing device, the computer equipment and the storage medium based on the multiple systems, which are provided by the application, acquire system information of all associated systems corresponding to a target service and acquire a system docking mode respectively corresponding to each associated system based on all the system information. When judging that the current time is in a free time period, writing the service data corresponding to the target service in each association system into a preset data conversion layer for each association system based on a data processing rule corresponding to a system docking mode of each association system, then performing isomorphic conversion processing on all the service data contained in the data conversion layer based on a preset data conversion thread to obtain processed service data, and finally extracting the processed service data from the data conversion layer and storing the processed service data. According to the method and the device, the system can be in butt joint with each associated system based on the system butt joint mode corresponding to each associated system, and then the service data contained in each associated system is written into the data conversion layer based on the data processing rule corresponding to the system butt joint mode to perform isomorphic conversion processing on the service data to obtain the service data with the same format, so that real-time collection and summarization of the service data corresponding to the target service contained in all associated systems can be effectively and quickly realized, the processing cost of the service data is reduced, and the processing efficiency and the processing intelligence of the service data are improved.

Drawings

FIG. 1 is a flow chart of a multisystem-based data processing method according to an embodiment of the present application;

FIG. 2 is a block diagram of a multisystem-based data processing apparatus according to an embodiment of the present application;

fig. 3 is a schematic structural diagram of a computer device according to an embodiment of the present application.

The implementation, functional features and advantages of the objectives of the present application will be further explained with reference to the accompanying drawings.

Detailed Description

It should be understood that the specific embodiments described herein are merely illustrative of the present application and are not intended to limit the present application.

It will be understood by those skilled in the art that, unless otherwise defined, all terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art to which this invention belongs. It will be further understood that terms, such as those defined in commonly used dictionaries, should be interpreted as having a meaning that is consistent with their meaning in the context of the prior art and will not be interpreted in an idealized or overly formal sense unless expressly so defined herein.

The embodiment of the application can acquire and process related data based on an artificial intelligence technology. Among them, Artificial Intelligence (AI) is a theory, method, technique and application system that simulates, extends and expands human Intelligence using a digital computer or a machine controlled by a digital computer, senses the environment, acquires knowledge and uses the knowledge to obtain the best result.

The artificial intelligence infrastructure generally includes technologies such as sensors, dedicated artificial intelligence chips, cloud computing, distributed storage, big data processing technologies, operation/interaction systems, mechatronics, and the like. The artificial intelligence software technology mainly comprises a computer vision technology, a robot technology, a biological recognition technology, a voice processing technology, a natural language processing technology, machine learning/deep learning and the like.

Referring to fig. 1, a multi-system-based data processing method according to an embodiment of the present application includes:

s1: acquiring system information of all associated systems corresponding to the target service;

s2: based on all the system information, acquiring system docking modes respectively corresponding to the associated systems;

s3: acquiring current time, and judging whether the current time is in a preset leisure time period or not;

s4: if yes, for each association system, writing the service data corresponding to the target service in each association system into a preset data conversion layer based on a data processing rule corresponding to a system docking mode of each association system;

s5: performing isomorphic conversion processing on all service data contained in the data conversion layer based on a preset data conversion thread and a data conversion rule to obtain processed service data;

s6: and extracting the processed service data from the data conversion layer and storing the processed service data.

As described in the above steps S1 to S6, the main implementation of the embodiment of the method is a multisystem-based data processing apparatus. In practical applications, the data processing apparatus based on multiple systems may be implemented by a virtual apparatus, such as a software code, or by an entity apparatus written in or integrated with a relevant execution code, and may perform human-computer interaction with a user through a keyboard, a mouse, a remote controller, a touch panel, or a voice control device, and may specifically be a system having a data collection and aggregation function. The data processing device based on multiple systems in the embodiment can effectively and quickly realize real-time acquisition and summarization of the business data corresponding to the target business contained in all the associated systems, and improves the processing efficiency and the processing intelligence of the business data. Specifically, system information of all associated systems corresponding to the target service is first acquired. Wherein the target service may be a premium transaction service. The system information may include system name information or system id information. The system docking mode related to the associated system can be queried based on the system information. For example, when the target service is a premium transaction service, since premium transaction data of the premium transaction service can be derived from a plurality of premium transaction channels, such as an offline agent sales channel, a bank sales channel, internet sales, a telephone sales channel, and the like, a plurality of association systems corresponding to each premium transaction channel are correspondingly created for the premium transaction service. And then acquiring system docking modes respectively corresponding to the associated systems based on all the system information. The system docking mode refers to a docking mode supported by the association system with other external systems, and the system docking mode may specifically include a KAFKA docking mode, a golden gate docking mode, an ActiveMQ docking mode, a first interface docking mode, and a second interface docking mode. In addition, the system information has a mapping relation with the system docking mode, and a mapping data table for storing the system information and the system docking mode corresponding to the system information can be created in advance based on the mapping relation, so that the system docking modes of all the associated systems corresponding to the target service can be inquired from the mapping data table quickly and conveniently.

And then acquiring the current time, and judging whether the current time is in a preset leisure time period. The time of day can be divided into a plurality of time periods, the first time periods are screened out from all the time periods based on the preset busy time list, the data processing total amount of each first time period in the preset time period is analyzed, and the free time period of the device is intelligently determined based on the analysis result. And if the current time is in the leisure time period, for each association system, writing the service data corresponding to the target service in each association system into a preset data conversion layer based on the data processing rule corresponding to the system docking mode of each association system. Different data processing rules are correspondingly set for different system docking modes, and for any one associated system, the data processing rule corresponding to the associated system is correspondingly called so as to smoothly write the service data corresponding to the target service in the associated system into a preset data conversion layer. In addition, the data conversion layer can be an area in the device for performing data isomorphism processing on the service data.

And subsequently, based on a preset data conversion thread and a preset data conversion rule, performing isomorphic conversion processing on all service data contained in the data conversion layer to obtain processed service data. The method includes the steps of determining a second number based on a first number and a preset proportion of a central processing unit of the device, creating a plurality of data conversion threads which are the same as a product of the first number and the preset proportion of the central processing unit of the device in advance in the device, distributing all service data to each data conversion thread, sending a data conversion rule to each data conversion thread, and performing isomorphic conversion processing on service data contained in the device simultaneously based on the data conversion rule through each data conversion thread to obtain the processed service data. In addition, the specific value of the preset proportion is not limited and can be set according to actual requirements. And finally, extracting the processed service data from the data conversion layer and storing the processed service data. After the processed service data is extracted, the processed service data can be synchronized or stored in the Elasticsearch, so that a subsequent relevant user can quickly search out relevant service data required to be queried from the Elasticsearch in real time. The Elasticsearch is a Lucene-based search server. It provides a distributed multi-user capable full-text search engine based on RESTful web interface. The Elasticsearch is developed in Java language, is released as an open source under Apache licensing terms, and is a popular enterprise-level search engine. For example, if the business data is premium transaction data, the sales condition of the product in each premium transaction channel or level can be queried from the elastic search, so that the sales dynamics of the product and the sales market reverberation of the product can be viewed in real time.

In this embodiment, system information of all associated systems corresponding to a target service is acquired, and system docking manners respectively corresponding to the associated systems are acquired based on all the system information. When judging that the current time is in a free time period, writing the service data corresponding to the target service in each association system into a preset data conversion layer for each association system based on a data processing rule corresponding to a system docking mode of each association system, then performing isomorphic conversion processing on all the service data contained in the data conversion layer based on a preset data conversion thread to obtain processed service data, and finally extracting the processed service data from the data conversion layer and storing the processed service data. According to the embodiment, the association systems can be docked with the association systems based on the system docking modes respectively corresponding to the association systems, and then the service data contained in the association systems are respectively written into the data conversion layer based on the data processing rules corresponding to the system docking modes to perform isomorphic conversion processing on the service data to obtain the service data with the same format, so that real-time collection and summarization of the service data corresponding to the target service contained in all the association systems can be effectively and quickly realized, the processing cost of the service data is reduced, and the processing efficiency and the processing intelligence of the service data are improved.

Further, in an embodiment of the present application, the step S4 includes:

s400: acquiring a system docking mode of a designated association system; the appointed correlation system is any one of all correlation systems, and the system docking mode of the appointed correlation system is marked as an appointed correlation system docking mode;

s401: if the designated association system docking mode is a KAFKA docking mode, pulling the service data corresponding to the target service in the designated association system to the data conversion layer according to a first preset rule;

s402: if the designated association system is in a golden gate docking mode, calling a preset golden gate tool to synchronize service data corresponding to the target service in the designated association system to the data conversion layer;

s403: if the appointed association system docking mode is an ActiveMQ docking mode, pulling the service data corresponding to the target service in the appointed association system to the data conversion layer according to a second preset rule;

s404: if the designated association system docking mode is a preset first interface docking mode, calling a preset Http interface to write the service data corresponding to the target service in the designated association system into the data conversion layer;

s405: if the designated association system docking mode is a preset second interface docking mode, connecting the designated association system through JDBC, and synchronizing service data corresponding to the target service in the designated association system to the data conversion layer.

As described in the foregoing steps S400 to S405, the step of writing, for each of the association systems, the service data corresponding to the target service in each of the association systems into a preset data conversion layer based on the data processing rule corresponding to the system docking manner of each of the association systems may specifically include: firstly, a system docking mode of a specified associated system is obtained. The appointed correlation system is any one of all the correlation systems, and the system docking mode of the appointed correlation system is marked as the appointed correlation system docking mode. In addition, the system docking mode may specifically include a KAFKA docking mode, a golden gate docking mode, an ActiveMQ docking mode, a first interface docking mode, and a second interface docking mode. And if the appointed association system docking mode is a KAFKA docking mode, pulling the service data corresponding to the target service in the appointed association system to the data conversion layer according to a first preset rule. If the system interface mode of the designated association system is the KAFKA interface mode, the device can synchronize the service data corresponding to the target service in the designated association system to the data conversion layer by using the message content acquisition mode corresponding to KAFKA. Specifically, the first preset rule may include: whether the appointed association system uses KAFKA as a message middleware or not is judged, if yes, a first request for applying for the consumption authority of a first TOPIC corresponding to the target service is sent to the appointed association system, and after the appointed association system successfully processes the first request, the service data corresponding to the target service and contained in the first TOPIC is pulled to the data conversion layer based on the consumption authority. And if the designated association system is in a golden gate docking mode, calling a preset golden gate tool to synchronize the service data corresponding to the target service in the designated association system to the data conversion layer. If the system docking mode of the designated association system is a golden gate docking mode, the designated association system can support a device to synchronize service data corresponding to the target service in the designated association system to the data conversion layer by using a corresponding golden gate tool. In addition, the golden gate tool is a tool for real-time synchronization between databases, and supports synchronization and replication of data between databases of the same type and different types. And if the appointed association system is in an ActiveMQ docking mode, pulling the service data corresponding to the target service in the appointed association system to the data conversion layer according to a second preset rule. If the system docking mode of the designated association system is an ActiveMQ docking mode, the system docking mode indicates that the designated association system can support a device to synchronize the service data corresponding to the target service in the designated association system to the data conversion layer by using a message content acquisition mode corresponding to the ActiveMQ. Specifically, the second preset rule may include: the method comprises the steps of firstly sending a second request for applying for subscription authority of a second TOPIC corresponding to the target service to the appointed association system, successfully processing the second request by the appointed association system, transmitting a broadcast message containing service data corresponding to the target service into an MQ message queue corresponding to the second TOPIC, establishing a connection relation with the MQ message queue in the appointed association system, subscribing the broadcast message in the MQ message queue based on the subscription authority, and pulling the service data corresponding to the target service in the appointed association system contained in the broadcast message to the data conversion layer. And if the specified association system docking mode is a preset first interface docking mode, calling a preset Http interface to write the service data corresponding to the target service in the specified association system into the data conversion layer. And the first interface is connected with the Http interface in a corresponding way. If the system docking mode of the designated association system is the first interface docking mode, the designated association system can support the device to call an Http interface to synchronize the service data corresponding to the target service in the designated association system to the data conversion layer. Specifically, the Http interface is an Http interface in the designated association system, and may be first docked with the Http interface of the designated association system, and after the docking is successful, the Http interface acquires the service data corresponding to the target service in the designated association system, and then writes the acquired service data into the data conversion layer. And if the specified association system docking mode is a preset second interface docking mode, connecting the specified association system through JDBC, and synchronizing the service data corresponding to the target service in the specified association system to the data conversion layer. And the second interface docking mode is a docking mode corresponding to the JDBC interface. If the system docking mode of the designated association system is the second interface docking mode, it indicates that the designated association system can support the device to connect the designated association system through JDBC to synchronize the service data corresponding to the target service in the designated association system to the data conversion layer. In addition, JDBC (Java Database Connectivity) is an application program interface in Java language used to specify how a client program accesses a Database, providing methods such as querying and updating data in a Database. Specifically, after obtaining authorization of a source table of service data corresponding to the target service and included in the association system, the JDBC connection is performed based on the obtained authorization, and data synchronization is performed according to data included in each page, so as to synchronize the service data corresponding to the target service and included in the designated association system to the data conversion layer. For each association system, the embodiment can intelligently write the service data corresponding to the target service in the association system into a preset data conversion layer quickly and conveniently based on the data processing rule corresponding to the system docking mode of the association system, so that the subsequent isomorphic conversion processing can be quickly and conveniently performed on all the service data contained in the data conversion layer, and the processed service data can be further obtained, so that the real-time collection and statistics of all the service data of the target service can be completed.

Further, in an embodiment of the application, the step S401 includes:

s4010: determining whether the specified association system has used KAFKA as a message middleware;

s4011: if KAFKA is used as message middleware, a first request for applying consumption authority of a first TOPIC corresponding to the target service is sent to the specified association system;

s4012: judging whether the specified association system successfully processes the first request or not;

s4013: if the first request is successfully processed, pulling the service data corresponding to the target service contained in the first TOPIC to the data conversion layer based on the consumption right.

As described in the foregoing steps S4010 to S4013, the step of pulling the service data corresponding to the target service in the specified association system to the data conversion layer according to a first preset rule may specifically include: it is first determined whether the specified association system has used KAFKA as a message middleware. The message middleware is responsible for storing messages, that is, when a consumer does not have time to process a complete message, the message content can be buffered in the message middleware. Additionally, if the designated association system already uses KAFKA as the message middleware, the messages in KAFKA, such as traffic data for the designated association system, may be stored in the TOPIC of KAFKA. The TOPICs are message classifications in KAFKA, which may contain multiple TOPICs, and different TOPICs are isolated from each other, and different TOPICs are used to store different traffic data according to actual traffic demands. If KAFKA has been used as a message middleware, a first request is issued to the designated association system for a consuming right of a first TOPIC corresponding to the target service. The device needs to successfully apply for the consumption authority of the subject if the device needs to acquire the message in the TOPIC, and then the message contained in the TOPIC is consumed according to the consumption authority. Specifically, if the device needs to acquire the service data corresponding to the target service in the designated association system, it needs to apply for the consumption right of the first TOPIC corresponding to the target service in the designated association system in advance. It is then determined whether the first request was successfully processed by the designated association system. Wherein specifying that the association system successfully processes the first request refers to: the designated association system allows the device to apply for consumption rights of a first TOPIC corresponding to the target traffic to enable the device to possess consumption rights for the first TOPIC. In addition, after the designated association system successfully processes the first request, corresponding successful processing information is returned so as to indicate the application of the consumption authority of the designated association system to the first TOPIC through the device through the successful processing information. If the first request is successfully processed, pulling the service data corresponding to the target service contained in the first TOPIC to the data conversion layer based on the consumption right. When the device has the consumption right for the first TOPIC, the service data corresponding to the target service in the appointed association system can be extracted from the first TOPIC according to the consumption right. Further, if the designated association system has not used KAFKA as a message middleware, a unified write API and TOPIC is provided for association systems that are willing to write traffic data to KAFKA, followed by a pull of the traffic data in the TOPIC to the data translation layer. In this embodiment, after the designated system docking manner is determined to be the KAFKA docking manner and the designated association system already uses KAFKA as the message middleware, the first request for applying for the consumption permission of the first TOPIC corresponding to the target service is intelligently issued to the designated association system according to the corresponding data processing rule, and after the consumption permission is obtained, the service data corresponding to the target service in the designated association system included in the first TOPIC is quickly and conveniently pulled to the data conversion layer based on the consumption permission, so that the subsequent isomorphic conversion processing can be quickly and conveniently performed on all the service data included in the data conversion layer, and the processed service data can be further obtained, so as to complete the real-time acquisition and statistics on all the service data of the target service.

Further, in an embodiment of the present application, the step S405 includes:

s4050: sending a second request for applying subscription authority of a second TOPIC corresponding to the target service to the specified association system;

s4051: judging whether the specified association system successfully processes the second request;

s4052: if yes, after the appointed association system transmits the broadcast message of the service data corresponding to the target service into the MQ message queue corresponding to the second TOPIC, establishing a connection relation with the MQ message queue in the appointed association system;

s4053: after the connection relationship is successfully established, subscribing the broadcast information in the MQ message queue based on the subscription authority;

s4054: and pulling the service data corresponding to the target service in the specified correlation system contained in the broadcast information to the data conversion layer.

As described in steps S4050 to S4054, the step of pulling the service data corresponding to the target service in the designated association system to the data conversion layer according to a second preset rule may specifically include: first, a second request for applying the subscription authority of a second TOPIC corresponding to the target service is sent to the appointed correlation system. Wherein, ActiveMQ is a message service component. The second TOPIC corresponding to the target traffic is a message classification in ActiveMQ, which may also be referred to as a TOPIC or TOPIC pattern. The theme mode has the following characteristics: the client comprises a publisher and a subscriber; messages in the topic are consumed by all subscribers; the consumer cannot consume the message that was sent to the topic before the subscription. Here, the publisher is the designated association system. Because the message in the theme can only be consumed by the subscriber, if the device needs to acquire the message in the theme, the device needs to successfully apply for the subscription authority of the theme first, and then the message contained in the theme is consumed according to the subscription authority. Specifically, if the device needs to acquire service data corresponding to the target service in the designated association system, it needs to apply for the subscription right of the second TOPIC corresponding to the target service in the designated association system in advance. It is then determined whether the second request was successfully processed by the designated association system. Wherein specifying that the association system successfully processes the second request refers to: specifying an association system allows the device to apply for subscription rights of a second TOPIC corresponding to the target traffic to enable the device to possess subscription rights to the second TOPIC. In addition, after the designated association system successfully processes the second request, corresponding successful processing information is returned so as to indicate that the designated association system has passed the application of the subscription authority of the device to the second TOPIC through the successful processing information. If the second request is successfully processed, after the appointed association system transmits the broadcast message of the service data corresponding to the target service into the MQ message queue corresponding to the second TOPIC, establishing a connection relationship with the MQ message queue in the appointed association system. After the appointed association system determines that the device has the requirement for acquiring the service data corresponding to the target service and requests for the subscription authority of the second TOPIC, the broadcast message containing the service data corresponding to the target service is transmitted into the MQ message queue corresponding to the second TOPIC. And then subscribing the broadcast information in the MQ message queue based on the subscription authority after the connection relationship is successfully established. After the device has the subscription right for the second TOPIC, the subscription of the broadcast information in the MQ message queue can be effectively realized according to the subscription right. And finally, pulling the service data corresponding to the target service in the specified correlation system contained in the broadcast information to the data conversion layer. And subscribing the broadcast information in the MQ message queue to extract the service data corresponding to the target service in the specified correlation system from the broadcast information. After the appointed system docking mode is judged to be the ActiveMQ docking mode, the method and the device can intelligently establish a connection relation with the MQ message queue corresponding to the second TOPIC in the appointed association system according to the data processing rule corresponding to the ActiveMQ docking mode, subscribe the broadcast information in the MQ message queue, so that the business data corresponding to the target business in the appointed association system and contained in the broadcast information can be synchronized to the data conversion layer, the method and the device are favorable for carrying out subsequent isomorphic conversion processing on all the business data contained in the data conversion layer quickly and conveniently, and further the processed business data can be obtained, so that real-time acquisition and statistics on all the business data of the target business can be completed.

Further, in an embodiment of the present application, the step S5 includes:

s500: acquiring pre-stored configuration information, and acquiring a first number of central processing units from the configuration information;

s501: calculating the product of the first quantity and a preset proportion to obtain a second quantity;

s502: creating a plurality of data conversion threads with the same number as the second number in a preset thread pool;

s503: distributing all the service data into each data conversion thread according to a preset distribution rule;

s504: respectively sending the data conversion rules to each data conversion thread;

s505: and simultaneously carrying out isomorphic conversion processing on the service data contained in each data conversion thread based on the data conversion rule through each data conversion thread to obtain the processed service data.

As described in the foregoing steps S500 to S505, the step of performing isomorphic conversion processing on all service data included in the data conversion layer based on a preset data conversion thread and a data conversion rule to obtain processed service data may specifically include: first, pre-stored configuration information is obtained, and a first number of central processing units is obtained from the configuration information. The configuration information of the device can be obtained in advance and stored, the configuration information can include internal memory information, kernel information and the like of the device, and the kernel information includes the number of central processing units. The number of the Central Processing units may also be referred to as a CPU core number, where the CPU core number refers to the number of cores of a CPU (Central Processing Unit), and the number of the Central Processing units may also be obtained through an operation instruction of a CPU get. And then calculating the product of the first quantity and a preset proportion to obtain a second quantity. The preset ratio may be a value determined by a preset parameter range, and the parameter range may be, for example, 1.5 to 3.5. Preferably, when the preset ratio is 2 times of the number of the central processing units, that is, when the preset ratio is 2, the cost of thread switching in the CPU of the device is low, which is beneficial to reducing the device loss in the subsequent data processing process. And then creating a plurality of data conversion threads with the same number as the second number in a preset thread pool. After the data conversion processing is completed, the data conversion thread immediately enters a dormant state when not working, so that occupation of a CPU is reduced, and reduction of device loss is facilitated. And after the data conversion threads are obtained, distributing all the service data into each data conversion thread according to a preset distribution rule. The allocation rule is not specifically limited, and it is only required to ensure that a certain part of service data is allocated in each data conversion thread, and the sum of the certain part of service data allocated in all the data conversion threads is all the service data included in the data conversion layer. Preferably, all the service data included in the data conversion layer may be equally divided into a plurality of designated service data equal to the number according to the number of the data conversion threads, and then one designated service data may be randomly allocated to each data conversion thread. And subsequently, the data conversion rules are respectively sent to the data conversion threads. The data conversion rule can be a rule generated by a developer according to actual data conversion requirements. The data conversion rule may include a validity check of the data and a data format conversion step. The validity check is used for checking whether abnormal data such as SQL injection characters and illegal characters exist in the service data or not and rejecting abnormal data. And the data format conversion step is used for converting specific rule character strings such as XML, JSON and the like in the service data into the required rule data object. And finally, performing isomorphic conversion processing on the service data contained in each data conversion thread simultaneously through each data conversion thread based on the data conversion rule to obtain the processed service data. Each data conversion thread simultaneously performs isomorphic conversion processing on the service data contained in the data conversion thread by using the same data conversion rule, so that all the obtained processed service data can be ensured to be isomorphic data with the same format, and isomorphism of all the service data is realized. In this embodiment, after the second number is determined based on the first number and the preset ratio of the central processing unit of the device, a thread pool including a plurality of data conversion threads with the same number as the second number is created in advance in the device, so that a plurality of processing flows of isomorphic conversion processing are performed in parallel by using the data conversion threads in the following. The number of the data conversion threads in the thread pool is matched with the second number, so that the processing efficiency of data conversion processing can be improved to the greatest extent, the waste of device resources is avoided, the processing efficiency of other works except data processing can be prevented from being influenced by the processing threads with excessive number, and the intelligence of processing the service data is effectively improved.

Further, in an embodiment of the present application, before the step S3, the method includes:

s300: dividing the value according to a preset length, and dividing the time of a day into a plurality of time periods;

s301: screening all the time periods based on a preset busy time list, and screening out a first time period from all the time periods; wherein the number of the first time periods is multiple;

s302: counting the total data processing amount of each first time period in a preset time period based on a pre-stored historical data processing record;

s303: screening out a first data processing total amount which is larger than a preset data processing amount threshold value from all the data processing total amounts;

s304: removing the first data processing total amount from all the data processing total amounts to obtain a processed second data processing total amount;

s305: screening out a second time period corresponding to the second data processing total amount from all the first time periods;

s306: taking the second time period as the leisure time period.

As described in the above steps S300 to S306, before the step of determining whether the current time is in the preset leisure time period, a process of generating the leisure time period may be further included. Specifically, the score value is first divided by a preset length, and the time of day is divided into a plurality of time periods. The time period is not specifically limited, and the time length included in each divided time period may also be set according to actual requirements, for example, 4 hours may be used as the value of the length division, that is, the time length included in one time period, and then one day (24 hours) may be divided into 6 time periods from 0, that is, 0: 00-4: 00,4: 00-8: 00,8: 00-12: 00, 12: 00-16: 00, 16: 00-20: 00, 20: 00-24: 00. and then screening all the time periods based on a preset busy time list, and screening out the first time period from all the time periods. Wherein the number of the first time periods is plural. In addition, the busy time list can be a pre-generated table containing time periods with busy traffic, and the time periods with busy traffic can be removed from all the obtained time periods, so that the corresponding first time period is obtained. For example, if the busy time list contains 12: 00-16: 00 and 16: 00-20: 00, the first time period can be selected to be 0: 00-4: 00,4: 00-8: 00,8: 00-12: 00, 20: 00-24: 00. all time periods are preliminarily screened by utilizing the busy time list, so that the data processing total amount of each first time period in a preset time period is only needed to be counted subsequently, the data processing total amount of all time periods is not needed to be counted, the effective reduced data statistics amount is realized, the statistical loss of the device is reduced, and the processing intelligence of the device is improved. And then counting the total data processing amount of each first time period in a preset time period based on a pre-stored historical data processing record. The preset time period is not particularly limited, and may be set according to actual requirements. For example, the preset time period may be the last month adjacent to the current time. For example, if a certain first processing time period is 8: 00-12: 00, the first time period 8: 00-12: 00 the total amount of data processing in a month is the first time period 8: 00-12: 00 contains the sum of all data throughput. And after the total data processing amount is obtained, screening out a first total data processing amount larger than a preset data processing amount threshold value from all the total data processing amounts. The data processing quantity threshold is not particularly limited, and may be set according to actual requirements. And subsequently, removing the first data processing total amount from all the data processing total amounts to obtain a processed second data processing total amount. And finally screening out a second time period corresponding to the second data processing total amount from all the first time periods, and taking the second time period as the leisure time period. According to the method and the device, after the time of one day is divided into the plurality of time periods, the first time periods are screened out from all the time periods based on the preset busy time list, the total data processing amount of each first time period in the preset time period is analyzed, the leisure time period of the device is intelligently determined based on the analysis result, and the accuracy of the generated leisure time period is effectively improved. And the data processing flow of writing the service data corresponding to the target service into the preset data conversion layer can be carried out subsequently in the free time period, and the data writing processing can not be carried out in the service peak period of the device, so that the normal use of a user can not be influenced, the normal operation of the data writing processing can not be influenced, the reasonable utilization of system resources is ensured, and the processing speed and the processing efficiency of the data writing processing are effectively improved.

Further, in an embodiment of the present application, after the step S6, the method includes:

s600: after the step of storing the processed service data is completed, generating data acquisition completion information corresponding to the target service;

s601: acquiring preset mail login information; and the number of the first and second groups,

s602: acquiring a target mail address of a target user;

s603: logging in to a corresponding mail server based on the mail login information;

s604: and sending the data acquisition completion information to the target mail address through the mail server.

As described in the above steps S600 to S604, after the step of extracting the processed service data from the data conversion layer and storing the processed service data is completed, a process of generating data acquisition completion information and sending the data acquisition completion information to a relevant target user may be further included. Specifically, after the step of storing the processed service data is completed, data acquisition completion information corresponding to the target service is generated, and first, data acquisition completion information corresponding to the target service is generated. The data acquisition completion information at least includes service information of the target service, such as a service name or a service code of the target service. In addition, the service information of the target service can be filled to the corresponding position in a pre-created information template to obtain data acquisition completion information, wherein the information template is pre-created according to actual use requirements, and at least a service information field is included in the information template. And then acquiring preset mail login information. And acquiring a target mail address of the target user. Wherein the target user is a relevant system administrator. And subsequently logging in a corresponding mail server based on the mail login information. And finally, sending the data acquisition completion information to the target mail address through the mail server. After the processed service data corresponding to the target service is generated, the data acquisition completion information corresponding to the target service is also intelligently generated and sent to the target mail address corresponding to the target user, so that the target user can timely and conveniently know that the target service completes the acquisition and the summarization of all service data based on the data acquisition completion information, and accordingly, corresponding subsequent processing is timely performed according to the data acquisition completion information, and the use experience of the target user is effectively improved.

The data processing method based on multiple systems in the embodiment of the present application may also be applied to the field of blockchains, for example, data such as the processed service data are stored on a blockchain. By storing and managing the processed service data by using the block chain, the security and the non-tamper property of the processed service data can be effectively ensured.

The block chain is a novel application mode of computer technologies such as distributed data storage, point-to-point transmission, a consensus mechanism and an encryption algorithm. A block chain (Blockchain), which is essentially a decentralized database, is a series of data blocks associated by using a cryptographic method, and each data block contains information of a batch of network transactions, so as to verify the validity (anti-counterfeiting) of the information and generate a next block. The blockchain may include a blockchain underlying platform, a platform product service layer, an application service layer, and the like.

The block chain underlying platform can comprise processing modules such as user management, basic service, intelligent contract and operation monitoring. The user management module is responsible for identity information management of all blockchain participants, and comprises public and private key generation maintenance (account management), key management, user real identity and blockchain address corresponding relation maintenance (authority management) and the like, and under the authorization condition, the user management module supervises and audits the transaction condition of certain real identities and provides rule configuration (wind control audit) of risk control; the basic service module is deployed on all block chain node equipment and used for verifying the validity of the service request, recording the service request to storage after consensus on the valid request is completed, for a new service request, the basic service firstly performs interface adaptation analysis and authentication processing (interface adaptation), then encrypts service information (consensus management) through a consensus algorithm, transmits the service information to a shared account (network communication) completely and consistently after encryption, and performs recording and storage; the intelligent contract module is responsible for registering and issuing contracts, triggering the contracts and executing the contracts, developers can define contract logics through a certain programming language, issue the contract logics to a block chain (contract registration), call keys or other event triggering and executing according to the logics of contract clauses, complete the contract logics and simultaneously provide the function of upgrading and canceling the contracts; the operation monitoring module is mainly responsible for deployment, configuration modification, contract setting, cloud adaptation in the product release process and visual output of real-time states in product operation, such as: alarm, monitoring network conditions, monitoring node equipment health status, and the like.

Referring to fig. 2, an embodiment of the present application further provides a multi-system based data processing apparatus, including:

the first acquisition module 1 is used for acquiring system information of all associated systems corresponding to the target service;

a second obtaining module 2, configured to obtain, based on all the system information, system docking manners respectively corresponding to the associated systems;

the judging module 3 is used for acquiring the current time and judging whether the current time is in a preset leisure time period or not;

the first processing module 4 is configured to, if yes, write, for each of the associated systems, service data corresponding to the target service in each of the associated systems into a preset data conversion layer based on a data processing rule corresponding to a system docking manner of each of the associated systems;

the second processing module 5 is configured to perform isomorphic conversion processing on all service data included in the data conversion layer based on a preset data conversion thread and a data conversion rule, so as to obtain processed service data;

and the third processing module 6 is configured to extract the processed service data from the data conversion layer, and store the processed service data.

In this embodiment, operations that the modules or units are respectively configured to execute correspond to the steps of the multi-system-based data processing method in the foregoing embodiment one to one, and are not described herein again.

Further, in an embodiment of the present application, the first processing module 4 includes:

the first acquisition unit is used for acquiring a system docking mode of a specified correlation system; the appointed correlation system is any one of all correlation systems, and the system docking mode of the appointed correlation system is marked as an appointed correlation system docking mode;

a first pulling unit, configured to pull, according to a first preset rule, service data corresponding to the target service in the designated association system to the data conversion layer if the designated association system docking manner is a KAFKA docking manner;

a first synchronization unit, configured to invoke a preset golden gate tool to synchronize service data corresponding to the target service in the designated association system to the data conversion layer if the designated association system is in a golden gate docking manner;

a second pulling unit, configured to, if the designated association system docking manner is an ActiveMQ docking manner, pull, according to a second preset rule, service data corresponding to the target service in the designated association system to the data conversion layer;

a writing unit, configured to call a preset Http interface to write, to the data conversion layer, service data corresponding to the target service in the designated association system if the designated association system docking mode is a preset first interface docking mode;

and the second synchronization unit is used for connecting the designated association system through JDBC if the docking mode of the designated association system is a preset second interface docking mode, and synchronizing the service data corresponding to the target service in the designated association system to the data conversion layer.

Further, in an embodiment of the present application, the first pulling unit includes:

a first judgment subunit operable to judge whether the specified association system has used KAFKA as a message middleware;

a first sending subunit, configured to send a first request for applying a consumption right of a first TOPIC corresponding to the target service to the specified association system if KAFKA has been used as message middleware;

the second judging subunit is used for judging whether the specified association system successfully processes the first request;

a second processing subunit, configured to, if the first request has been successfully processed, pull, to the data translation layer, traffic data corresponding to the target traffic and included in the first TOPIC based on the consumption right.

Further, in an embodiment of the present application, the second pulling unit includes:

a second sending subunit, configured to send, to the designated association system, a second request for applying for a subscription right of a second TOPIC corresponding to the target service;

a third determining subunit, configured to determine whether the designated association system successfully processes the second request;

the connection subunit is configured to, if yes, establish a connection relationship with the MQ message queue in the designated association system after the designated association system transmits the broadcast message containing the service data corresponding to the target service into the MQ message queue corresponding to the second TOPIC;

a subscription subunit, configured to subscribe, based on the subscription authority, the broadcast information in the MQ message queue after the connection relationship is successfully established;

a second sending subunit, configured to pull, to the data conversion layer, the service data corresponding to the target service in the specified association system included in the broadcast information.

Further, in an embodiment of the present application, the second processing module 5 includes:

the second acquisition unit is used for acquiring prestored configuration information and acquiring a first number of the central processing units from the configuration information;

the calculating unit is used for calculating the product of the first quantity and a preset proportion to obtain a second quantity;

a creating unit, configured to create a plurality of data conversion threads with the same number as the second number in a preset thread pool;

the distribution unit is used for distributing all the service data into each data conversion thread according to a preset distribution rule;

the sending unit is used for respectively sending the data conversion rules to the data conversion threads;

and the processing unit is used for simultaneously carrying out isomorphic conversion processing on the service data contained in the processing unit through each data conversion thread based on the data conversion rule to obtain the processed service data.

Further, in an embodiment of the present application, the multi-system based data processing apparatus includes:

the dividing module is used for dividing the score values according to a preset length and dividing the time of one day into a plurality of time periods;

the first screening module is used for screening all the time periods based on a preset busy time list and screening out first time periods from all the time periods; wherein the number of the first time periods is multiple;

the statistical module is used for counting the total data processing amount of each first time period in a preset time period based on a pre-stored historical data processing record;

the second screening module is used for screening out a first data processing total amount which is larger than a preset data processing amount threshold value from all the data processing total amounts;

the fourth processing module is used for removing the first data processing total amount from all the data processing total amounts to obtain a processed second data processing total amount;

the third screening module is used for screening out a second time period corresponding to the second data processing total amount from all the first time periods;

a determination module for taking the second time period as the leisure time period.

a generating module, configured to generate data acquisition completion information corresponding to the target service after the step of storing the processed service data is completed;

the third acquisition module is used for acquiring preset mail login information; and the number of the first and second groups,

the fourth acquisition module is used for acquiring a target mail address of a target user;

the login module is used for logging in a corresponding mail server based on the mail login information;

and the sending module is used for sending the data acquisition completion information to the target mail address through the mail server.

Referring to fig. 3, a computer device, which may be a server and whose internal structure may be as shown in fig. 3, is also provided in the embodiment of the present application. The computer device comprises a processor, a memory, a network interface, a display screen, an input device and a database which are connected through a system bus. Wherein the processor of the computer device is designed to provide computing and control capabilities. The memory of the computer device comprises a storage medium and an internal memory. The storage medium stores an operating system, a computer program, and a database. The internal memory provides an environment for the operating system and computer programs in the storage medium to run. The database of the computer equipment is used for storing target service information, system docking modes, leisure time periods, data processing rules, data conversion rules and processed service data. The network interface of the computer device is used for communicating with an external terminal through a network connection. The display screen of the computer equipment is an indispensable image-text output equipment in the computer, and is used for converting digital signals into optical signals so that characters and figures are displayed on the screen of the display screen. The input device of the computer equipment is the main device for information exchange between the computer and the user or other equipment, and is used for transmitting data, instructions, some mark information and the like to the computer. The computer program is executed by a processor to implement a multi-system based data processing method.

The processor executes the steps of the multi-system-based data processing method:

Those skilled in the art will appreciate that the structure shown in fig. 3 is only a block diagram of a part of the structure related to the present application, and does not constitute a limitation to the apparatus and the computer device to which the present application is applied.

An embodiment of the present application further provides a computer-readable storage medium, on which a computer program is stored, and when the computer program is executed by a processor, the method for processing data based on multiple systems is implemented, specifically:

It will be understood by those skilled in the art that all or part of the processes of the methods of the embodiments described above can be implemented by a computer program, which can be stored in a computer-readable storage medium, and can include the processes of the embodiments of the methods described above when the computer program is executed. Any reference to memory, storage, database, or other medium provided herein and used in the examples may include non-volatile and/or volatile memory. Non-volatile memory can include read-only memory (ROM), Programmable ROM (PROM), Electrically Programmable ROM (EPROM), Electrically Erasable Programmable ROM (EEPROM), or flash memory. Volatile memory can include Random Access Memory (RAM) or external cache memory. By way of illustration and not limitation, RAM is available in a variety of forms such as Static RAM (SRAM), Dynamic RAM (DRAM), Synchronous DRAM (SDRAM), double-rate SDRAM (SSRSDRAM), Enhanced SDRAM (ESDRAM), synchronous link (Synchlink) DRAM (SLDRAM), Rambus Direct RAM (RDRAM), direct bus dynamic RAM (DRDRAM), and memory bus dynamic RAM (RDRAM).

It should be noted that, in this document, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, apparatus, article, or method that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, apparatus, article, or method. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, apparatus, article, or method that includes the element.

The above description is only a preferred embodiment of the present application, and not intended to limit the scope of the present application, and all modifications of equivalent structures and equivalent processes, which are made by the contents of the specification and the drawings of the present application, or which are directly or indirectly applied to other related technical fields, are also included in the scope of the present application.

Claims

1. A multisystem-based data processing method is characterized by comprising the following steps:

2. The multisystem-based data processing method according to claim 1, wherein the step of writing the service data corresponding to the target service in each of the associated systems to a preset data conversion layer based on a data processing rule corresponding to a system docking manner of each of the associated systems for each of the associated systems comprises:

3. The multisystem-based data processing method according to claim 2, wherein the step of pulling the service data corresponding to the target service in the specified association system to the data conversion layer according to a first preset rule includes:

4. The multisystem-based data processing method according to claim 2, wherein the step of pulling the service data corresponding to the target service in the specified associated system to the data conversion layer according to a second preset rule includes:

5. The multisystem-based data processing method according to claim 1, wherein the step of performing isomorphic conversion processing on all the service data included in the data conversion layer based on a preset data conversion thread and a data conversion rule to obtain processed service data includes:

respectively sending the data conversion rules to each data conversion thread;

6. The multisystem-based data processing method according to claim 1, wherein the step of determining whether the current time is within a preset leisure time period comprises:

taking the second time period as the leisure time period.

7. The multisystem-based data processing method according to claim 1, wherein said step of extracting the processed service data from the data conversion layer and storing the processed service data is followed by:

after the step of storing the processed service data is completed, generating data acquisition completion information corresponding to the target service;

acquiring a target mail address of a target user;

logging in to a corresponding mail server based on the mail login information;

8. A multisystem-based data processing apparatus, comprising:

9. A computer device comprising a memory and a processor, the memory having stored therein a computer program, characterized in that the processor, when executing the computer program, implements the steps of the method according to any one of claims 1 to 7.

10. A computer-readable storage medium, on which a computer program is stored, which, when being executed by a processor, carries out the steps of the method of any one of claims 1 to 7.