CN115269225A

CN115269225A - Data processing method and device and computer equipment

Info

Publication number: CN115269225A
Application number: CN202210885101.6A
Authority: CN
Inventors: 颜小健; 张兆冰
Original assignee: Lenovo Beijing Ltd
Current assignee: Lenovo Beijing Ltd
Priority date: 2022-07-26
Filing date: 2022-07-26
Publication date: 2022-11-01

Abstract

The application provides a data processing method, a data processing device and computer equipment, wherein the computer equipment is deployed between each data source system and a message middleware, and the computer equipment obtains data to be processed to be sent to the message middleware and a characteristic identifier thereof; responding to the data scheduling request, and determining the transmission priority of a plurality of data to be processed according to the characteristic identification of the data to be processed and a preset priority scheduling rule; and sending the data to be processed to the message middleware according to the transmission priority of the data to be processed.

Description

Data processing method and device and computer equipment

Technical Field

The present application relates to the field of information technology, and in particular, to a data processing method, apparatus, and computer device.

Background

Message Middleware (MQ), also called Message Queue, has gradually become the core means of internal communication of enterprise systems by a series of functions such as low coupling, reliable delivery, broadcasting, flow control, final consistency, etc.

Based on the method, under the cross-system synchronization scene, each data source system calls the message middleware, sends the generated data to be processed to the message middleware for caching, and waits for each data processor to read the cached data sequentially by monitoring the message middleware. But limited by the first-in first-out data sequence processing rule followed by the message middleware, the system architecture cannot meet the requirement of quick response of the emergency important data.

Disclosure of Invention

In view of this, the present application proposes a data processing method, which includes:

acquiring to-be-processed data to be sent to a message middleware and a feature identifier of the to-be-processed data;

responding to a data scheduling request, and determining the transmission priority of a plurality of data to be processed according to the feature identification and a preset priority scheduling rule;

and sending the data to be processed to the message middleware according to the transmission priority of the data to be processed.

Optionally, the determining the transmission priorities of the multiple pieces of data to be processed according to the feature identifier and a preset priority scheduling rule includes:

calling a priority scheduling rule associated with the feature identifier;

and determining the transmission priority of each of the plurality of data to be processed with the associated characteristic identifier according to the called priority scheduling rule.

Optionally, the invoking a priority scheduling rule associated with the feature identifier includes:

calling a priority scheduling rule corresponding to the characteristic identifier according to the corresponding relation between different characteristic identifiers and different priority scheduling rules; or,

determining the target data type of the data to be processed according to the feature identifier;

and calling the target data type according to the corresponding relation between the different data types and the different priority scheduling rules.

Optionally, the sending the to-be-processed data to the message middleware according to the transmission priorities of the to-be-processed data includes:

determining a data scheduling frequency associated with the feature identifier; the characteristic identification can represent the data type of the data to be processed with the characteristic identification;

and sending the data to be processed with the characteristic identifier to a message middleware according to the data scheduling frequency associated with the same characteristic identifier and the transmission priority of the data to be processed.

Optionally, the determining a data scheduling frequency associated with the feature identifier includes:

detecting whether historical scheduling information associated with the feature identifier exists; the historical scheduling information is generated in response to a historical data scheduling request which belongs to the historical to-be-processed data with the characteristic identification;

if the historical scheduling frequency exists, determining the historical scheduling frequency contained in the historical scheduling information as a data scheduling frequency;

and if not, processing a plurality of to-be-processed data associated with the characteristic identifier obtained within a preset time length based on a scheduling frequency calculation rule to obtain a data scheduling frequency associated with the characteristic identifier.

Optionally, the processing, based on the scheduling frequency calculation rule, the multiple pieces of to-be-processed data associated with the feature identifier and obtained within a preset time period to obtain the data scheduling frequency associated with the feature identifier includes:

determining a preset time length associated with the characteristic identifier and a preset number of reference analysis time within the preset time length;

in the process of obtaining the data to be processed associated with the feature identifier within the preset time length, obtaining the unit time increment of the data to be processed respectively obtained within the preset number of reference analysis time;

obtaining the increase time of the unit time increase amount of the to-be-processed data within the preset time length to reach the maximum increase amount by using the obtained unit time increase amount of the preset number and the preset number;

and determining the increase time as the time interval of two adjacent data scheduling, and obtaining the data scheduling frequency associated with the characteristic identifier.

Optionally, in the absence of historical scheduling information associated with the feature identifier, the sending, to a message middleware, the to-be-processed data with the feature identifier according to the data scheduling frequency and the transmission priority of the to-be-processed data associated with the same feature identifier includes:

determining the transmission sequence of a plurality of data to be processed with the characteristic identifier according to the transmission priority of the data to be processed associated with the characteristic identifier;

and after the preset time length, sending the corresponding data to be processed to a message middleware according to the data scheduling frequency and the transmission sequence associated with the same characteristic identifier.

Optionally, the method further includes:

outputting a data processing interface, and presenting a plurality of function triggering areas and function display areas in the data processing interface; the plurality of function triggering areas comprise at least one of a data labeling area, a priority scheduling area and a monitoring area;

responding to the triggering operation of the data labeling area, performing feature extraction on the obtained data to be processed, and presenting the obtained feature identification of the data to be processed in the function display area; and/or the presence of a gas in the atmosphere,

responding to the triggering operation of the priority scheduling area, and presenting a priority scheduling interface in the function display area;

responding to the priority script configuration operation on the priority scheduling interface to obtain priority scheduling rules corresponding to different data types; the priority script configuration operation is executed based on the data processing requirement of the corresponding data type; and/or the presence of a gas in the gas,

and responding to the triggering operation of the monitoring area, and presenting a data scheduling process for sending the obtained data to be processed with different data types to the message middleware in the function display area.

The present application further provides a data processing apparatus, the apparatus comprising:

the data acquisition module is used for acquiring data to be processed to be sent to the message middleware and the feature identifier of the data to be processed;

a transmission priority determining module, configured to respond to a data scheduling request, and determine transmission priorities of the multiple pieces of data to be processed according to the feature identifier and a preset priority scheduling rule;

and the data transmission module is used for sending the data to be processed to the message middleware according to the transmission priorities of the data to be processed.

The present application further provides a computer device, comprising:

the system comprises a plurality of communication interfaces, a message middleware and at least one data source system, wherein the communication interfaces are used for being in communication connection with the message middleware and the at least one data source system respectively so as to realize the transmission of data to be processed between the data source system and the message middleware;

a memory for storing a program for implementing the data processing method as described above, and data to be processed from the data source system;

a processor for loading and executing the program stored in the memory to realize the data processing method.

Therefore, the application provides a data processing method, a device and a computer device, wherein the computer device is deployed between each data source system and a message middleware, data to be processed output by each data source system is firstly sent to the computer device for storage, and the computer device obtains a feature identifier of each data to be processed, so that in order to send some data to be processed required by a service to the message middleware preferentially, a corresponding data scheduling request can be responded, and the transmission priority of a plurality of stored data to be processed is determined according to the feature identifier of the data to be processed and a preset priority scheduling rule, so that the data to be processed are sent to the message middleware accordingly, some data to be processed required by the service can be rapidly and efficiently issued to the message middleware without changing the original configuration of the message middleware and the data source system, and a data processor executing the service can read the required data to be processed in time from the message middleware, thereby improving the data processing efficiency.

Drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the drawings used in the embodiments or the prior art descriptions will be briefly described below, it is obvious that the drawings in the following description are only embodiments of the present invention, and for those skilled in the art, other drawings can be obtained according to the provided drawings without creative efforts.

Fig. 1 is a schematic diagram of an alternative system architecture in an application environment suitable for the data processing method proposed in the present application;

FIG. 2 is a schematic diagram of a hardware configuration of an alternative example of a computer device proposed in the present application;

FIG. 3 is a schematic flow chart diagram of an alternative example of the data processing method proposed in the present application;

FIG. 4 is a schematic flow chart diagram of yet another alternative example of the data processing method presented in the present application;

fig. 5 is a schematic flowchart of another alternative example of the data processing method proposed in the present application;

FIG. 6 is a schematic flow chart diagram of yet another alternative example of the data processing method presented in the present application;

FIG. 7 is a schematic flow chart diagram of yet another alternative example of the data processing method presented in the present application;

FIG. 8 is a schematic diagram of an alternative data processing interface in the data processing method proposed in the present application;

fig. 9 is a schematic structural diagram of an alternative example of the data processing apparatus proposed in the present application.

Detailed Description

And aiming at the description of the background technology part, establishing various types of topic topics with different priorities is provided, so that the priority of the topic to which the important urgent data belongs can be improved according to the quick response requirement of the important urgent data, a data processor can preferentially monitor the topic with the highest priority, preferentially process each data to be processed which belongs to the topic in the message middleware until the data to be processed which belongs to the topic does not exist in the message middleware, and then process each data to be processed which corresponds to the topic with lower priority. However, in the data processing mode, various types of topics need to be constructed based on the service processing requirements of the application scenarios, the difficulty of system management is increased, and in the data processing process, if the topic with a high priority always has to process data, each piece of data to be processed of other topics with a low priority cannot be processed, and the service processing requirements are difficult to meet.

In order to improve the above problem, it is proposed to mount a memory-based re-sequencer to introduce a service between a data publisher and a data processor, where the data to be processed cached in an internal buffer of the re-sequencer meets a caching condition (e.g., the data volume reaches a preset capacity, or the caching time reaches a maximum waiting time), and the set of cached data to be processed is sorted and then published based on a string manner according to the determined priority of a kafka (a high throughput distributed publish-subscribe messaging system) topic, so as to preferentially process the urgent important data required by the service ranked earlier.

However, in the process of using the re-sequencer to process data, after the to-be-processed data cached in the buffer area meets the caching condition, the to-be-processed data is sorted and issued according to the service processing requirement, which easily causes a delay problem when a data processor receives the to-be-processed data.

For the above problems, the present application further proposes that, without changing the existing message middleware settings and the original data sending program of the data producer (data source system), a priority scheduling device is configured between each data source system and the message middleware for caching the to-be-processed data generated by each data source system, and a priority scheduling rule meeting the actual service processing requirements is pre-constructed, so that the priority identifiers of the to-be-processed data meeting the actual service processing requirements are obtained by using the feature identifiers of the to-be-processed data and the priority scheduling rule, adjusting the transmission priorities of the cached multiple to-be-processed data (i.e., the transmission sequence of the to-be-processed data to the message middleware), that is, determining the transmission sequence of the cached multiple to-be-processed data meeting the actual service processing requirements, and sending the multiple to-be-processed data to the message middleware according to the newly determined transmission sequence, thereby sending the to-be-processed data urgently needed by the service to the data processor at the first time, and greatly improving the data processing efficiency.

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

Referring to fig. 1, which is a schematic diagram of an alternative system architecture in an application environment suitable for the data processing method provided in the present application, as shown in fig. 1, the system may include at least one data source system 110, a message middleware 120, and a computer device 130 communicatively connected to the data source system 110 and the message middleware 120, respectively, where:

the data source system 110 may be a business system that generates data to be processed, such as a CRM (Customer Relationship Management) system, an ECC (Enterprises Command Center) system, or other business systems, and the application does not limit the type of the business system and the structure thereof, and may be determined according to the situation.

Message middleware (MQ, also called Message Queue) 120 may be used to buffer the data to be processed generated by each data source system 110, wait for the client (i.e. data processor) to monitor and process, avoid data loss, and relieve the data processing pressure between the data source systems and the data processors. In practical application, the type message intermediaries such as Kafka, rabbitMQ, zeroMQ, activeMQ and the like can be flexibly selected and used for data caching according to application requirements, the number and the types of the message intermediaries are not limited in the application, and the structure and the working principle of each type of message middleware are not detailed in the application.

The computer device 130 may be the priority scheduling device described above, and may exist as an independent device, or may be integrated as a custom component in a certain device in the system, so as to implement the data processing method provided by the present application, and the implementation process may be described with reference to the corresponding part of the method embodiment below.

In practical application of the present application, the computer device 130 may provide a communication interface for each data source system to call, and receive data to be processed (such as service data generated by each service system) generated by each data source system for caching, so that, when a certain data processor needs to read a certain type or a plurality of data to be processed first, the data to be processed needs to be sent to the message middleware first, so that the data processor can obtain the required data to be processed as soon as possible according to the first-in first-out characteristic of the queue. Therefore, the computer device 130 may be a device with certain data processing capability and data storage space, and the present application does not limit the device type of the computer device 130, and may be determined according to the circumstances.

In some embodiments, as shown in fig. 2, a hardware structure diagram of an alternative example of the computer device proposed in the present application may include: a plurality of communication interfaces 131, at least one memory 132, and at least one processor 133, wherein:

the plurality of communication interfaces 131 may be used for being respectively in communication connection with the message middleware 120 and the at least one data source system 110, so as to implement transmission of data to be processed between the data source system 110 and the message middleware 120.

In practical applications, the communication interface 131 may include a data interface of a communication module capable of implementing data interaction by using a wireless communication network, such as a data interface of a communication module like a WIFI module, a 5G/6G (fifth generation mobile communication network/sixth generation mobile communication network) module, or a GPRS module; the data processing method may further include an Application calling Interface such as an API (Application Program Interface), so that the data source system sends the generated to-be-processed data to the computer device by calling the API Interface, and the computer device may execute the data processing method provided by the present Application. In addition, the communication interface 131 may further include a communication interface such as a USB interface, a serial/parallel interface, and the like for implementing data interaction inside the computer device, which may be determined according to application requirements, and this application is not described in detail herein by way of example.

The memory 132 may be used to store programs implementing the data processing methods described in the method embodiments below, as well as data to be processed from the data source systems 110; the processor 133 can load and execute the program stored in the memory to implement the steps of the data processing method described in the corresponding method embodiment below, and the detailed implementation process may refer to the description of the corresponding parts of the embodiment below, which is not described in detail herein.

In the embodiment of the present application, the memory 132 may include a high-speed random access memory, and may further include a nonvolatile memory, such as at least one magnetic disk storage device or other volatile solid state storage devices. To implement persistent storage of pending data from the various data source systems, the memory 132 may also include a database(s) ((b))_Data _Base，DB), namely a warehouse for storing data to be processed, and the type of the database and the storage mode of the data to be processed are not limited by the application and can be determined according to the situation.

Optionally, the database may be an independent device, and is connected to the processor 133 of the computer device 130 in a wired or wireless manner; or may be integrated in the computer device 130. It can be seen that the computer device 130 is not limited to a single device, and may also include a plurality of devices connected in communication, which may be determined according to actual needs, and this application is not described in detail herein by way of example.

The processor 133 may be a Central Processing Unit (CPU), an application-specific integrated circuit (ASIC), a Digital Signal Processor (DSP), an Application Specific Integrated Circuit (ASIC), an off-the-shelf programmable gate array (FPGA) or other programmable logic device, and so on.

The communication interface 131, the memory 132, the processor 133 and other computer devices may be connected to a communication bus of the computer device, so as to implement data interaction between the computer devices and other components of the computer device, as well as with other I/O devices of the computer device, and meet various application requirements of the computer device.

In addition, it should be understood that the computer device configuration shown in fig. 2 does not constitute a limitation of the computer device in the embodiments of the present application, and in practical applications, the computer device may include more components than those shown in fig. 2, or may combine some components, such as at least one output component, e.g., a display, a speaker, etc.; at least one input component such as a keyboard, a mouse, a sound pickup, a touch sensing unit, etc., which are not listed herein; similarly, the system architecture applicable to the data processing method provided by the present application is not limited to the system structure shown in fig. 1, and may further include a monitoring device, which may be flexibly adjusted according to the data processing requirement, and the present application is not described in detail herein.

Referring to fig. 3, a schematic flowchart of an optional example of the data processing method proposed in the present application, which can be executed by the computer device 130, based on the system architecture and the constituent devices thereof described above, as shown in fig. 3, the data processing method proposed in this embodiment may include:

step S31, obtaining data to be processed and a characteristic identifier thereof to be sent to the message middleware;

in order to realize the scheduling of the data to be processed to be sent to the message middleware, the data to be processed required by the service can be sent to the message middleware firstly, so that a data processor (such as a client) can obtain the data to be processed required by the service from the message middleware preferentially. The application provides that priority scheduling equipment (namely the computer equipment) is configured between each data source system and the message middleware, and each data source system can call an application call interface API (application program interface) used for realizing the data processing method provided by the application in the computer equipment to send the data to be processed generated by the application to the computer equipment instead of directly sending the data to the message middleware.

In this way, after receiving data to be processed (different types of data such as contract data, order data, customer data, material data, price data and the like, which can be determined according to the types of the data source systems) sent by each data source system, the computer equipment can persist the data to the database so as to avoid data loss and relieve data processing pressure. And then, the computer equipment can label the obtained data to be processed to obtain the characteristic identifier of each data to be processed, so that each data to be processed can be identified later.

Step S32, responding to the data scheduling request, and determining the transmission priority of a plurality of data to be processed according to the characteristic identification of the data to be processed and a preset priority scheduling rule;

in a scene that data to be processed needs to be scheduled, namely, after an original sequence of sending some kind of data to be processed required by a service to a message middleware is compared, the data to be processed needs to be scheduled to a front or even a first transmission position so as to meet a priority processing requirement of a data processor on the data to be processed.

In practical application, for business data that a data processor needs to preferentially read, the business data is often a certain type of data or the type of data under a certain dimension or multiple dimensions, such as price data of a product B transported by the country M through a channel a, in order to read the needed price data as soon as possible from multiple types of data to be processed from various cached data from various data source systems, the transmission priority of the price data to a message middleware needs to be scheduled, and for this reason, a priority scheduling rule for realizing the transmission priority scheduling can be configured. Thus, when responding to the data scheduling request, the data to be processed, such as the price data exemplified above, which belong to the data to be processed that needs to be scheduled, can be determined according to the feature identifier of the data to be processed, and then the respective transmission priorities of the price data, that is, the subsequent transmission order to the message middleware, can be determined according to the priority scheduling rule configured for the price data.

The priority scheduling rules can support script definition, so that after a data scheduling requirement (namely a data processing requirement) changes, the priority scheduling rules meeting the changed data scheduling requirement can be redefined through the script, for example, the priority scheduling rules meeting the current data scheduling requirement are flexibly configured on an output priority scheduling interface, and the dynamic configuration implementation method of the priority scheduling rules is not limited and can be determined according to situations.

And step S33, sending the data to be processed to the message middleware according to the transmission priorities of the data to be processed.

Following the above description of the technical solution of the present application, the transmission priority of the data to be processed may be characterized as follows: the transmission order of the data to be processed to the message middleware in the multiple data to be processed cached by the computer device may configure a priority identifier of a transmission priority of the data to be processed in a priority field of the data to be processed, so that the computer device obtains the transmission priority of each data to be processed by monitoring the priority field, that is, by monitoring a change in the priority field at a field level, thereby obtaining the transmission order of each data to be processed stored by the computer device, and sending the transmission order to the message middleware according to the transmission order.

The transmission channel through which the computer device transmits the stored to-be-processed data to the message middleware for caching may be determined according to a communication manner between the computer device and the message middleware, which is not described in detail herein.

In summary, in the embodiment of the present application, a computer device is deployed between each data source system and a message middleware, and for to-be-processed data output by each data source system, the to-be-processed data is sent to the computer device for storage, and the computer device obtains a feature identifier of each to-be-processed data, so that in order to send some to-be-processed data required by a service to the message middleware preferentially, a corresponding data scheduling request may be responded, and according to the feature identifier of the to-be-processed data and a preset priority scheduling rule, transmission priorities of a plurality of stored to-be-processed data are determined, so that the to-be-processed data are sent to the message middleware accordingly, and some to-be-processed data required by the service may be quickly and efficiently issued to the message middleware without changing original configurations of the message middleware and the data source system, so that a data processor executing the service can read the required to-be-processed data in time from the message middleware, thereby improving data processing efficiency.

Referring to fig. 4, a schematic flow chart of yet another optional example of the data processing method proposed by the present application, which may be an optional detailed implementation of the data processing method described in the foregoing embodiment, but is not limited to the detailed implementation method described in the present embodiment, and as shown in fig. 4, the detailed implementation method may include:

step S41, obtaining data to be processed and a characteristic identifier thereof to be sent to the message middleware;

step S42, responding to the data scheduling request, and calling a priority scheduling rule associated with the characteristic identifier;

step S43, determining the transmission priority of each of a plurality of data to be processed with associated feature marks according to the priority scheduling rule;

in the embodiment of the present application, priority scheduling rules for to-be-processed data corresponding to different feature identifiers may be preconfigured according to actual scheduling requirements of a data scheduling request, the obtained priority scheduling rules are associated with the feature identifiers and then stored, in a scheduling process, classification and identification may be performed according to the feature identifiers of the to-be-processed data, and a transmission sequence of the to-be-processed data having the associated feature identifiers to a message middleware is scheduled according to the priority scheduling rules associated with the feature identifiers, so as to obtain a transmission priority corresponding to the to-be-processed data, but the method is not limited to the transmission priority determining method described in this embodiment.

In this way, the data monitoring module or the database trigger of the computer device can monitor the change in the priority field at the field level to trigger the sending service of the data to be processed, and the specific implementation process can be described according to the context corresponding part.

Step S44, determining the data scheduling frequency associated with the characteristic identifier;

in practical application, the process of generating the data to be processed by the data source system is often executed at different constant speeds but according to actual service fluctuation, and the scheduling time interval, namely the data scheduling frequency, of the data to be processed generated by each data source system can be determined according to the service fluctuation conditions of different data source systems, and is stored after being associated with the feature identifier of the data to be processed generated by the corresponding data source system. It should be noted that, the method for determining the data scheduling frequency associated with each feature identifier and the value thereof are not limited, and for example, how long the shortest time is required to reach the maximum data increment may be predicted based on a prediction model with preset duration, so as to obtain the data scheduling frequency of the corresponding type of data to be processed.

The feature identifier of the data to be processed can represent the data type (which is often a large class to which the data to be processed belongs) to which the data to be processed with the feature identifier belongs, so that the feature identifier can be determined according to information such as key words and topics of the data to be processed, or the feature identifier of the data to be processed generated by a data source system can be determined according to an application type of the data source system of the data to be processed.

And step S45, sending the data to be processed with the characteristic identifier to a message middleware according to the data scheduling frequency associated with the same characteristic identifier and the transmission priority of the data to be processed.

Since the computer device may obtain the data to be processed generated by each of the multiple data source systems at the same time, so that the computer device stores multiple types of data to be processed, according to the above-described method, after determining the feature identifier, the transmission priority, and the data scheduling frequency of each type of data to be processed, for multiple data to be processed associated with the same feature identifier, the transmission sequence of the multiple data to be processed may be determined according to the transmission priority, then, according to the data scheduling frequency, the sending service of the multiple data to be processed having the feature identifier is triggered, and the multiple data to be processed are sent to the message middleware according to the determined transmission sequence.

In summary, after obtaining each piece of to-be-processed data to be sent to the message middleware, the computer device serving as the data scheduling device between the data source system and the message middleware obtains the feature identifier of each piece of to-be-processed data, and the transmission priority and the data scheduling frequency of the to-be-processed data corresponding to the same feature identifier, thereby implementing scheduling of the transmission sequence of the plurality of pieces of to-be-processed data with the feature identifier to the message middleware, so as to preferentially send the to-be-processed data urgently needed by a service to the message middleware, so that a data processor executing the service can quickly read the needed to-be-processed data from the message middleware without adding data subject data or adding a buffer area by defining a re-sequencer in the message middleware, and improving data processing efficiency.

Referring to fig. 5, a flow chart of a further optional example of the data processing method proposed by the present application is schematically illustrated, and the method may be a description of a further optional detailed implementation of the data processing method described in the foregoing embodiment, and as shown in fig. 5, the detailed implementation method may include:

s51, acquiring data to be processed and a characteristic identifier thereof to be sent to a message middleware;

step S52, responding to the data scheduling request, and determining the target data type of the data to be processed according to the characteristic identifier;

step S53, calling a priority scheduling rule corresponding to the target data type according to the corresponding relation between different data types and different priority scheduling rules;

in combination with the above description of the priority scheduling rule invoking process in the embodiment, for the data to be processed of different data types, the corresponding priority scheduling rule may be configured according to the corresponding data processing requirement (i.e., the scheduling requirement of the corresponding service on the data to be processed), so as to obtain the corresponding relationship between the different data types and the different priority scheduling rules. The generation time or the peak generation time period of the to-be-processed data of different data types may be different, and in the same time period, the transmission sequence of the to-be-processed data of one data type may only need to be scheduled, and of course, the transmission sequence of the to-be-processed data of multiple data types may also need to be scheduled, and the scheduling process of the to-be-processed data of each data type is similar, which is not described in detail in this application.

In this way, when a data scheduling request for data to be processed of any data type is responded, a target data type to which each data to be processed belongs can be determined according to the feature identifier of each data to be processed, and then, a priority scheduling rule corresponding to each target data type can be called according to the corresponding relation between preset different data types and different priority scheduling rules, that is, a transmission priority scheduling method for a plurality of data to be processed under the target data type is determined, so that the transmission priorities of the plurality of data to be processed, that is, the transmission sequence to the message middleware, can be determined accordingly.

In still other embodiments, in order to implement scheduling of transmission priority of data to be processed more accurately, this embodiment may determine the feature identifier of the data to be processed more finely, so as to characterize the subclass of the data to be processed, and then may determine the corresponding priority scheduling rule according to the data processing requirement of the class of data to be processed, and generate the corresponding relationship between different feature identifiers and different priority scheduling rules, so that, when responding to the data scheduling request, the feature identifier of the data to be processed may be directly utilized to query the corresponding relationship, and determine the transmission priority of the data to be processed.

In still other embodiments, if priority scheduling needs to be performed on the to-be-processed data of multiple data types at the same time, the transmission priority between the to-be-processed data of different data types may be determined first, so as to determine the transmission order between the to-be-processed data of each data type subsequently, and then the transmission order of the to-be-processed data may be determined by combining the transmission priorities of the to-be-processed data of the same data type described above. The transmission priority between the to-be-processed data of different data types can be determined according to at least one dimension information such as the importance of the to-be-processed data of each data type, a data processing requirement, a service requirement and the like, and the transmission priority is not limited by the application and can be determined according to the situation.

It can be seen that, in the embodiment, for the transmission ordering (i.e., the transmission priority configuration) of the to-be-processed data of each of the multiple data types obtained by the computer device, the transmission ordering may be determined from two dimensions, on one hand, the transmission ordering between the to-be-processed data of the multiple data types may be determined from the dimension of the data type, and on the other hand, the transmission ordering of the multiple to-be-processed data of the data type may be implemented from the characteristics of the to-be-processed data of the same data type (i.e., the priority scheduling rule corresponding to the data type).

Optionally, in the face of some limit scenarios, for data to be processed of multiple data types, the data to be processed of a certain data type with the highest transmission priority may be determined according to the above method, and then, scheduling and issuing of each data to be processed of the data type may be implemented according to the priority scheduling rule and the data scheduling frequency of the data to be processed of the data type; the publication may be suspended for pending data of other data types.

Step S54, detecting whether historical scheduling information of the target data type exists, and if so, entering step S55; if not, go to step S56;

step S55, determining the historical scheduling frequency contained in the historical scheduling information as the data scheduling frequency;

in order to reduce the calculation amount, the generation rule of the data to be processed of the same data type is basically unchanged through analysis, the data scheduling frequency/data scheduling time interval when the transmission priority of the data is scheduled at different time can be the same, and the scheduling parameters do not need to be repeatedly calculated on line, so that the calculation pressure of computer equipment is reduced. Therefore, when responding to the data scheduling request, it may be detected whether the target data type of the to-be-processed data stored in the computer device is scheduled, that is, whether historical scheduling information of the target data type exists, and if so, it is indicated that the to-be-processed data of the target data type is scheduled before, and the current scheduling of the transmission priority of the to-be-processed data of the target data type may be completed by directly using the historical scheduling frequency of the target data type.

It should be understood that the present application does not limit the content of the above-mentioned historical scheduling information, and besides the historical scheduling frequency of the to-be-processed data corresponding to the target data type (i.e. the data scheduling frequency of the historical to-be-processed data of the target data type stored by the computer device in a past certain time period), the historical scheduling time, the historical transmission priority, etc. of the historical to-be-processed data of the target data type may also be included, as the case may be.

Based on this, in still other embodiments, when responding to a data scheduling request, it may be determined whether to-be-processed data having the feature identifier (e.g., to-be-processed data of a target data type) is scheduled, that is, whether to store historical scheduling information, and if so, the data scheduling frequency and the priority scheduling rule for the data scheduling request are determined by using the historical scheduling frequency and the historical priority scheduling rule included in the historical scheduling information, and then, the transmission priority of the to-be-processed data having the associated feature identifier may be determined according to the priority scheduling rule, and a plurality of to-be-processed data having the feature identifier may be transmitted to the message middleware according to the data scheduling frequency and the transmission priority of the to-be-processed data. Therefore, the execution steps of the data processing method provided by the present application include, but are not limited to, the step sequence described in the context, and can be flexibly adjusted according to the actual situation, which is not exemplified in the present application.

Step S56, processing a plurality of data to be processed of the target data type obtained within a preset time length based on a scheduling frequency calculation rule to obtain a data scheduling frequency of the data to be processed of the target data type;

according to the above-described method, it is known that the computer device schedules the to-be-processed data of the target data type for the first time, and the corresponding data scheduling frequency can be obtained according to the scheduling frequency calculation rule based on the increase condition of the to-be-processed data of the target data type obtained within the preset time length, and the implementation process can refer to the following description of the corresponding embodiment, which is not described in detail herein.

And step S57, sending the data to be processed with the characteristic identifier to a message middleware according to the data scheduling frequency associated with the same characteristic identifier and the transmission priority of the data to be processed.

According to the description of the context on the implementation process of step S57, the transmission order of the multiple pieces of to-be-processed data with the feature identifier may be determined according to the transmission priority of the to-be-processed data associated with the feature identifier, so that after a preset time period (after 5 minutes as in the above example), the corresponding to-be-processed data may be sent to the message middleware according to the data scheduling frequency and the transmission order associated with the same feature identifier, and it is ensured that the to-be-processed data required by the service can be issued to the message middleware as soon as possible, so that the data processor can read the required to-be-processed data in time.

Optionally, for the data to be processed of a certain data type that is scheduled for the first time, the data scheduling frequency of the data to be processed of the target data type may be determined by referring to, but not limited to, the following method, as shown in fig. 6, where the method may include:

step S61, determining a preset time length associated with the characteristic identifier and a preset number of reference analysis time within the preset time length;

step S62, in the process of obtaining the data to be processed associated with the characteristic identifier within a preset time length, obtaining the unit time increment of the data to be processed respectively obtained within a preset number of reference analysis time;

step S63, obtaining the increasing time of the unit time increasing amount of the to-be-processed data within the preset time length to reach the maximum increasing amount by using the obtained unit time increasing amount of the preset number and the preset number;

and step S64, determining the increasing time as the time interval of two adjacent data scheduling, and obtaining the data scheduling frequency associated with the characteristic identifier.

According to the embodiment of the application, a prediction model with preset duration can be constructed to predict the shortest sending time interval of the data to be processed corresponding to the data type, namely, the data scheduling frequency of the data to be processed of the data type is determined. In practical applications, the preset durations of different data types (e.g., the preset durations associated with different feature identifiers) may be different, and the corresponding preset durations may be determined according to information such as the data types and the data processing requirements, and the application does not limit the value of the preset durations.

For example, for the type a data to be processed, data enhancement within 5 minutes can be predicted, 6 reference analysis times are selected, as shown in table 1 below, the data increase amount of the type a data to be processed within 5s, 10s, 30s, 1 minute, 2 minutes, and 5 minutes, that is, the total amount of the type a data to be processed obtained within the time, is counted from the start of scheduling timing, and the unit increase amount of the data to be processed within the time is calculated.

Unit time	Growth (data sum)	Unit growth (per second)
			5(S)	1000(S1)	S1/5
10(S)	2000(S2)	S2/10
			30(S)	3000(S3)	S3/30
1 minute	5000(S4)	S4/60
			2 minutes	8000(S5)	S5/120
5 minutes	12000(S6)	S6/300

TABLE 1

Based on each information calculated in table 1, an average value of unit time increment of a preset number, that is, (S1/5 + S2/10+ S3/30+ S4/60+ S5/120+ S6/300)/6 =115, is obtained in 5 minutes according to the prediction, the shortest increment time capable of reaching the maximum increment amount may be 12000/115=104 seconds, and the average value may be directly used as a time interval for scheduling the a-type data to be processed, that is, a transmission service is triggered once every 104 seconds, and the data to be processed is transmitted to the message middleware according to the stored transmission priority of the a-type data to be processed. It can be seen that the predicted data scheduling frequency of the type a data to be processed in this example may be scheduled once every 104 seconds.

Optionally, after determining the increase time according to the method described above, a scheduling time interval for the data to be processed of type a, that is, a time interval between two adjacent data schedules, may also be determined according to the increase time, so as to obtain a corresponding data scheduling frequency, for example, the data scheduling frequency is obtained by adaptively adjusting 104 seconds, and is not limited to determining the increase time as the data scheduling time interval.

It should be noted that, in the data scheduling control process, the data sending service for the message middleware is not limited to be triggered according to the parameter of the data scheduling frequency, and the data sending service can also be implemented according to the data scheduling time interval, and the implementation process is similar, and the detailed description is not given in this application.

In addition, in the prediction example of 5 minutes as described above, the reference analysis time of the preset number to be counted, including but not limited to the contents shown in table 1, may be adaptively adjusted according to the service requirement, and is not limited to the preset time period of 5 minutes. Similar implementation methods for predicting the data scheduling frequency of the to-be-processed data of other data types are not described in detail in this application.

Optionally, in practical application of the present application, a prediction model for predicting data scheduling frequency may be constructed according to the above-described calculation method, so that for to-be-processed data of a data type to be scheduled for the first time, the to-be-processed data acquired within a preset time period may be input to the prediction model to obtain the data scheduling frequency of the to-be-processed data of the data type, and a detailed implementation process is not described in this embodiment.

Based on the above analysis, in the embodiment of the present application, in the process of responding to the data scheduling request, it is determined that the data type to which the stored data to be processed belongs has been scheduled before, the historical data scheduling frequency of the data type may be directly called as the data scheduling frequency of the data scheduling of this time, and the historical priority scheduling rule is the priority scheduling rule of the data scheduling of this time.

If the data to be processed of the data type is scheduled for the first time, predicting the data scheduling frequency of the data to be processed of the data type based on the increase condition of the data to be processed of the data type obtained in the preset time length, so that after the preset time length, data scheduling can be started according to the data scheduling frequency, and when data is scheduled and sent each time, the transmission sequence of the data to be processed can be obtained according to the transmission priority of the data to be processed of the data type, and the data to be processed of the type obtained in the scheduling time interval is sent to the message middleware according to the transmission sequence, so that flexible scheduling of the data to be processed of different data types is realized, and the emergency use requirement of each data processor on the data to be processed of any data type is met.

According to the method, the computer equipment is directly arranged between the data source system and the message middleware to realize that the data to be processed of each data type to be sent to the message middleware is transmitted to the priority scheduling of the message middleware, the original configuration of the message middleware and the data source system is not required to be changed, the transmission sequence of the data to be processed when the current data processing requirements (such as scheduling requirements for quickly acquiring certain data to be processed) are met is determined by the computer equipment according to the method, and then the data to be processed which is urgently needed by a data processor is transmitted to the message middleware so as to be written into the storage queue of the message middleware first, so that the data processor can timely read the needed data to be processed from the storage queue, the reading processing of the data to be processed with lower priority cannot be influenced, and various data types can be compatible.

Based on the data processing method described in the foregoing embodiments, in a case where data scheduling is not required, the computer device does not need to determine the transmission priority of the obtained to-be-processed data from each data source system, and may directly send the obtained to-be-processed data to the message middleware according to a default transmission order (e.g., a sequence of generation time/obtaining time of each to-be-processed data). It can be seen that, when the data scheduling switch is activated, the transmission priorities of all the obtained data to be processed can be determined according to the above-described method, that is, the data to be processed is transmitted and sequenced and then issued; when the data scheduling switch is not activated, the computer equipment can directly forward the obtained data to be processed to the message middleware without processing the data. The form of the data scheduling switch and the activation implementation mode thereof are not limited by the application and can be determined according to the situation.

Referring to fig. 7, which is a flowchart illustrating yet another optional example of the data processing method proposed by the present application, this embodiment may describe the visualization scheme of the data processing method process described in the foregoing embodiment, but is not limited to the visualization implementation described in this embodiment, and as shown in fig. 7, the data processing method proposed by this embodiment may include:

step S71, outputting a data processing interface, and presenting a plurality of function triggering areas and function display areas in the data processing interface;

in order to implement visual monitoring of a data processing method in the embodiments of the present application, when the data processing method is executed, a data processing interface may be triggered and output, as shown in fig. 8, at least one of a plurality of function trigger areas (e.g., button areas shown in fig. 8), such as a data tagging area (e.g., an identification data button in fig. 8), a priority scheduling area (e.g., a definition rule shown in fig. 8), and a monitoring area (e.g., a monitoring main screen shown in fig. 8) is presented on the data processing interface.

Step S72, responding to the trigger operation of the data labeling area, extracting the characteristics of the obtained data to be processed, and displaying the characteristic identification of the obtained data to be processed in the function display area;

the computer device obtains the data to be processed sent by each data source system, and persists the data to the database DB, where the data information required to persist may include but is not limited to: request header, request json, request parameters, request URL (Uniform Resource Locator), etc. In order to facilitate subsequent identification of each piece of data to be processed, labeling can be performed according to information such as key words of the data to be processed, and the feature identifier of the data to be processed is determined.

For this, the dispatcher may trigger a data tagging area (e.g., a function button of the identification data shown in fig. 8) in the button area, read the Json body information of the request through the Json parsing function, extract the key word of the key field of the data to be processed, store the key word into the request data record that has been persisted, determine the feature identification of the data to be processed by using the key field according to the preset classification standard, and define the data type of the data to be processed. And then, classifying and labeling the obtained data to be processed according to the above, and presenting a persistent classification structure of the data to be processed in the corresponding function display area, but not limited to the classification mode and the display content and mode thereof shown in fig. 8, and can be flexibly adjusted according to actual requirements.

Step S73, responding to the triggering operation of the priority scheduling area, and presenting a priority scheduling interface in the function display area;

step S74, responding to the priority script configuration operation on the priority scheduling interface to obtain the priority scheduling rules corresponding to different data types;

in the embodiment of the present application, priority script configuration operation may be executed based on the data processing requirement of the corresponding data type, that is, a priority scheduling rule, such as "condition number/condition table/condition type/sales organization/material/number", is defined for the identified data to be processed through a script. For example, after the priority scheduling area is triggered, various information for defining the priority scheduling rule, such as selectable items of various data types and various types of conditions, may be displayed in the output priority scheduling interface, so that the scheduler selects or drags the corresponding option to form the priority scheduling rule, but the method is not limited to this implementation method.

Step S75, in response to the triggering operation on the monitoring area, presenting a data scheduling process for sending the obtained to-be-processed data of different data types to the message middleware in the function display area.

According to the method, after the monitoring area is triggered, the layout and the presentation content of a scheduling monitoring interface output by the function display area can be determined according to visual requirements, for example, to-be-processed data and data volume thereof under each data type, the transmission priority of each to-be-processed data determined according to the priority scheduling rule, the transmission progress transmitted to the message middleware according to the transmission priority, and in the process, information such as new to-be-processed data and data volume thereof obtained by the computer equipment are used for visually seeing the whole dynamic scheduling process of the to-be-processed data, and the embodiment of the actual display method is not described in detail herein.

Therefore, the visualized monitoring of the data processing method provided by the application can monitor the data synchronization flow intuitively and in real time, and can adjust the corresponding priority scheduling rules, data scheduling frequency and other contents in time according to the above-described method when the scheduling requirement changes, so as to ensure the high efficiency and convenience of data processing.

Referring to fig. 9, a schematic structural diagram of an alternative example of the data processing apparatus proposed in the present application may include:

a data obtaining module 91, configured to obtain to-be-processed data to be sent to a message middleware, and a feature identifier of the to-be-processed data;

a transmission priority determining module 92, configured to respond to a data scheduling request, and determine transmission priorities of multiple pieces of data to be processed according to the feature identifier and a preset priority scheduling rule;

a data transmission module 93, configured to send the to-be-processed data to the message middleware according to the transmission priorities of the to-be-processed data.

In some embodiments, the transmission priority determination module 92 may include:

a priority scheduling rule calling unit, configured to call a priority scheduling rule associated with the feature identifier;

a transmission priority determining unit, configured to determine, according to the called priority scheduling rule, a transmission priority of each of the multiple pieces of to-be-processed data having the associated feature identifier.

Optionally, the priority scheduling rule invoking unit may include:

the first calling unit is used for calling the priority scheduling rules corresponding to the characteristic identifications according to the corresponding relations between the different characteristic identifications and the different priority scheduling rules; or,

the target data type determining unit is used for determining the target data type of the data to be processed according to the characteristic identifier;

and the second calling unit is used for calling the target data type according to the corresponding relation between the different data types and the different priority scheduling rules.

In still other embodiments, the data transmission module 93 may include:

a data scheduling frequency determining unit, configured to determine a data scheduling frequency associated with the feature identifier; the characteristic identification can represent the data type of the data to be processed with the characteristic identification;

and the first sending unit is used for sending the data to be processed with the characteristic identifier to message middleware according to the data scheduling frequency associated with the same characteristic identifier and the transmission priority of the data to be processed.

Optionally, the data scheduling frequency determining unit may include:

a detecting unit, configured to detect whether there is historical scheduling information associated with the feature identifier; the historical scheduling information is generated in response to a historical data scheduling request which belongs to the historical to-be-processed data with the characteristic identification;

a first determining unit, configured to determine, when the detection result of the detecting unit is present, a historical scheduling frequency included in the historical scheduling information as a data scheduling frequency;

and the first calculating unit is used for processing the plurality of to-be-processed data which are obtained within a preset time length and are associated with the characteristic identifier on the basis of a scheduling frequency calculation rule under the condition that the detection result of the detecting unit does not exist, so that the data scheduling frequency associated with the characteristic identifier is obtained.

Optionally, the first calculating unit may include:

the second determining unit is used for determining a preset time length associated with the characteristic identifier and a preset number of reference analysis time within the preset time length;

a unit time increment obtaining unit, configured to obtain, in the process of obtaining the to-be-processed data associated with the feature identifier within the preset duration, unit time increments of the to-be-processed data obtained within the preset number of reference analysis times, respectively;

a growth time obtaining unit, configured to obtain, by using the obtained unit time growth amount of the preset number and the preset number, a growth time at which the unit time growth amount of the to-be-processed data obtained within the preset time length reaches a maximum growth amount;

and a third determining unit, configured to determine the increase time as a time interval between two adjacent data scheduling, and obtain a data scheduling frequency associated with the feature identifier.

Optionally, the first sending unit may include:

a transmission sequence determining unit, configured to determine a transmission sequence of the plurality of pieces of data to be processed having the feature identifier according to the transmission priority of the piece of data to be processed associated with the feature identifier;

and the second sending unit is used for sending the corresponding data to be processed to the message middleware according to the data scheduling frequency and the transmission sequence associated with the same characteristic identifier after the preset time length.

In still other embodiments, the apparatus may further include:

the data processing interface output module is used for outputting a data processing interface and presenting a plurality of function triggering areas and function display areas in the data processing interface; the plurality of function triggering areas comprise at least one of a data labeling area, a priority scheduling area and a monitoring area;

the data labeling module is used for responding to the triggering operation of the data labeling area, extracting the characteristics of the obtained data to be processed and displaying the characteristic identification of the obtained data to be processed in the function display area; and/or the presence of a gas in the gas,

the priority scheduling interface presenting module is used for responding to the triggering operation of the priority scheduling area and presenting a priority scheduling interface in the function display area;

the priority scheduling rule obtaining module is used for responding to the priority script configuration operation on the priority scheduling interface to obtain the priority scheduling rules corresponding to different data types; the priority script configuration operation is executed based on the data processing requirement of the corresponding data type; and/or the presence of a gas in the gas,

and the monitoring display module is used for responding to the triggering operation of the monitoring area and presenting a data scheduling process of sending the obtained data to be processed with different data types to the message middleware in the function display area.

It should be noted that, various modules, units, and the like in the embodiments of the foregoing apparatuses may be stored in the memory as program modules, and the processor executes the program modules stored in the memory to implement corresponding functions, and for the functions implemented by the program modules and their combinations and the achieved technical effects, reference may be made to the description of corresponding parts in the embodiments of the foregoing methods, which is not described in detail in this embodiment.

The present application also provides a computer-readable storage medium, on which a computer program can be stored, which can be called and loaded by a processor to implement the steps of the data processing method described in the above embodiments.

Finally, it should be noted that, with respect to the above embodiments, unless the context clearly dictates otherwise, the words "a", "an" and/or "the" do not denote a singular number, but may include a plurality. In general, the terms "comprises" and "comprising" merely indicate that steps and elements are included which are explicitly identified, that the steps and elements do not form an exclusive list, and that a method or apparatus may include other steps or elements. An element defined by the phrase "comprising a … …" does not exclude the presence of additional identical elements in a process, method, article, or apparatus that comprises the element.

In the description of the embodiments herein, "/" means "or" unless otherwise specified, for example, a/B may mean a or B; "and/or" herein is merely an association relationship describing an associated object, and means that there may be three relationships, for example, a and/or B, and may mean: a exists alone, A and B exist simultaneously, and B exists alone. In addition, in the description of the embodiments of the present application, "a plurality" means two or more than two.

This application is directed to terms such as "first," "second," and the like, which are used for descriptive purposes only to distinguish one operation, element, or module from another operation, element, or module and do not necessarily require or imply any actual relationship or order between such elements, operations, or modules. And are not to be construed as indicating or implying relative importance or implicitly indicating the number of technical features indicated, whereby a feature defined as "first" or "second" may explicitly or implicitly include one or more of such features.

The embodiments in the present description are described in a progressive or parallel manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other. The device, the computer device and the system disclosed by the embodiment correspond to the method disclosed by the embodiment, so that the description is relatively simple, and the relevant points can be referred to the method part for description.

The previous description of the disclosed embodiments is provided to enable any person skilled in the art to make or use the present invention. Various modifications to these embodiments will be readily apparent to those skilled in the art, and the generic principles defined herein may be applied to other embodiments without departing from the spirit or scope of the invention. Thus, the present invention is not intended to be limited to the embodiments shown herein but is to be accorded the widest scope consistent with the principles and novel features disclosed herein.

Claims

1. A method of data processing, the method comprising:

acquiring data to be processed to be sent to a message middleware and a feature identifier of the data to be processed;

2. The method according to claim 1, wherein the determining the transmission priority of the plurality of data to be processed according to the feature identifier and a preset priority scheduling rule comprises:

calling a priority scheduling rule associated with the feature identifier;

3. The method of claim 2, the invoking of the priority scheduling rule associated with the feature identification comprising:

calling a priority scheduling rule corresponding to the characteristic identifier according to the corresponding relation between the different characteristic identifiers and the different priority scheduling rules; or,

and calling the target data type according to the corresponding relation between different data types and different priority scheduling rules.

4. The method according to any one of claims 1 to 3, wherein the sending the to-be-processed data to the message middleware according to a plurality of transmission priorities of the to-be-processed data comprises:

5. The method of claim 4, the determining a data scheduling frequency associated with the feature identification, comprising:

6. The method according to claim 5, wherein the processing the to-be-processed data associated with the feature identifier obtained within a preset time period based on the scheduling frequency calculation rule to obtain the data scheduling frequency associated with the feature identifier includes:

obtaining the increase time of the unit time increase amount of the data to be processed reaching the maximum increase amount within the preset time length by using the unit time increase amount of the obtained preset number and the preset number;

7. The method according to claim 5, in the absence of historical scheduling information associated with the feature identifier, said sending the pending data with the feature identifier to a message middleware according to the data scheduling frequency and the transmission priority of the pending data associated with the same feature identifier, comprising:

8. The method according to any one of claims 1-3, further comprising:

performing feature extraction on the obtained data to be processed in response to the triggering operation on the data labeling area, and presenting the obtained feature identification of the data to be processed in the function display area; and/or the presence of a gas in the gas,

9. A data processing apparatus, the apparatus comprising:

the data acquisition module is used for acquiring to-be-processed data to be sent to the message middleware and the characteristic identifier of the to-be-processed data;

and the data transmission module is used for sending the data to be processed to the message middleware according to the transmission priority of the data to be processed.

10. A computer device, the computer device comprising:

the system comprises a plurality of communication interfaces, a message middleware and at least one data source system, wherein the communication interfaces are used for being respectively in communication connection with the message middleware and the at least one data source system so as to realize the transmission of data to be processed between the data source system and the message middleware;

a memory for storing a program for implementing the data processing method according to any one of claims 1 to 8, and data to be processed from the data source system;

a processor for loading and executing the program stored in the memory to implement the data processing method of any one of claims 1 to 8.