CN117875893A - Data processing method, system, device, medium and program product - Google Patents

Abstract

The present disclosure provides a data processing method, apparatus, device, storage medium, and program product, which can be applied to the technical field of big data. The data processing method comprises the following steps: in response to acquiring the real-time data, constructing service flow data; wherein the traffic stream data comprises a plurality of consecutive traffic events arranged in time sequence within a specified time window; performing real-time stream calculation operation on the service stream data to obtain real-time operation and maintenance data; acquiring corresponding offline operation and maintenance data based on the real-time data; the offline operation and maintenance data are obtained by calculating offline data in a specified time range; calculating current operation data of the data processing system based on the offline operation data and the real-time operation data; and determining a business handling flow based on the current operation data, and feeding back the business handling flow to the client so that the client executes business handling in the data processing system according to the business handling flow.

Description

Data processing method, system, device, medium and program product

Technical Field

The present disclosure relates to the field of big data technologies, and more particularly, to a business handling method, apparatus, device, medium, and program product.

Background

As market competition increases, banking businesses are becoming more abundant, and when customers transact business, the customers need to be guided based on banking business navigation so that the customers transact business smoothly.

However, since banking businesses are more, a large amount of data can be generated in the running process of each banking business system, which is easy to cause untimely data processing, and the running condition of each node in the business handling system can not be obtained quickly and accurately, so that the operation efficiency of the whole navigation system is lower, and timely and accurate customer guidance can not be performed.

Disclosure of Invention

In view of the foregoing, the present disclosure provides a data processing method, apparatus, device, medium, and program product that improve data processing accuracy.

According to a first aspect of the present disclosure, there is provided a data processing method comprising: in response to acquiring the real-time data, constructing service flow data; wherein the traffic stream data comprises a plurality of consecutive traffic events arranged in time sequence within a specified time window; performing real-time stream calculation operation on the service stream data to obtain real-time operation and maintenance data; the real-time operation and maintenance data are used for reflecting the current occupation condition and/or queuing condition of each node in the data processing system; acquiring corresponding offline operation and maintenance data based on the real-time data; the offline operation and maintenance data are obtained by calculating offline data in a specified time range; the offline operation and maintenance data are used for reflecting the change condition of the states of all nodes of the data processing system in a certain past time period; calculating current operation data of the data processing system based on the offline operation data and the real-time operation data; and determining a business handling flow based on the current operation data, and feeding back the business handling flow to the client so that the client executes business handling in the data processing system according to the business handling flow.

According to an embodiment of the present disclosure, in response to acquiring real-time data, constructing traffic flow data includes: acquiring the data type of real-time data; determining a packet to which real-time data belongs based on the data type; acquiring event time of real-time data; the event time is the time of actual generation of real-time data; and distributing the real-time data to the corresponding time window in the belonging packet based on the event time to obtain the service flow data.

According to an embodiment of the present disclosure, acquiring corresponding offline operation and maintenance data based on real-time data includes: acquiring a corresponding offline operation and maintenance data set according to the data type of the real-time data; and querying the offline operation and maintenance data at a specified time in the offline operation and maintenance data set.

According to an embodiment of the present disclosure, the offline operation data at a specified time includes: offline operation data closest to the time range of the time window in which the real-time data is located; and/or offline operation and maintenance data containing the current time window on different dates.

According to an embodiment of the present disclosure, further comprising calculating offline operation data based on the offline data: initiating an offline processing task based on a preset time threshold; in response to receiving the offline processing task, obtaining offline data within a time threshold from the data storage module; and transmitting the offline data into a pre-training model, and calculating to obtain the offline operation and maintenance data.

According to an embodiment of the present disclosure, calculating current operational data of a data processing system based on offline operational data and real-time operational data includes: predicting the state of each node at the next moment based on the offline operation and maintenance data and the real-time operation and maintenance data to obtain state change data of each node in the data processing system; the operating state of the data processing system is determined based on the real-time operation and maintenance data and the state change data of each node.

According to an embodiment of the present disclosure, further comprising: monitoring the operating state of the data processing system; executing alarm operation on abnormal nodes in the data processing system based on preset rules; the types of the abnormal nodes at least comprise: nodes with processing speed lower than a preset threshold value and nodes with processing data higher than a preset value; alarm operations corresponding to different types of abnormal nodes are different.

According to an embodiment of the present disclosure, further comprising: transmitting the request data to a message queue; and sequentially sending the request data in the message queue to the data processing module for processing according to the sequence, obtaining real-time data and sending the real-time data to the data storage module.

According to an embodiment of the present disclosure, sequentially sending request data in a message queue to a data processing module for processing, including: analyzing the latest uploaded request data in the message queue; preprocessing operation is carried out on the parsed request data, and preprocessed data is obtained; wherein the preprocessing operation comprises: at least one of data cleansing, data deduplication, data association, data merging; performing a secondary processing operation based on the data type of the pre-processed data to obtain real-time data; wherein the secondary treatment operation comprises an aggregation calculation and/or a desensitization operation.

A second aspect of the present disclosure provides a data processing system comprising: the construction module is used for responding to the acquired real-time data and constructing service flow data; wherein the traffic stream data comprises a plurality of consecutive traffic events arranged in time sequence within a specified time window; the real-time flow calculation module is used for executing real-time flow calculation operation on the business flow data to obtain real-time operation and maintenance data; the real-time operation and maintenance data are used for reflecting the current occupation condition and/or queuing condition of each node in the data processing system; the acquisition module is used for acquiring corresponding offline operation and maintenance data based on the real-time data; the offline operation and maintenance data are obtained by calculating offline data in a specified time range; the offline operation and maintenance data are used for reflecting the change condition of the states of all nodes of the data processing system in a certain past time period; the computing module is used for computing the current operation data of the data processing system based on the offline operation and maintenance data and the real-time operation and maintenance data; the determining module is used for determining a business handling flow based on the current operation data and feeding back the business handling flow to the client so that the client can execute business handling in the data processing system according to the business handling flow.

A third aspect of the present disclosure provides an electronic device, comprising: one or more processors; and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the data processing method described above.

A fourth aspect of the present disclosure also provides a computer-readable storage medium having stored thereon executable instructions that, when executed by a processor, cause the processor to perform the above-described data processing method.

A fifth aspect of the present disclosure also provides a computer program product comprising a computer program which, when executed by a processor, implements the above-described data processing method.

Drawings

The foregoing and other objects, features and advantages of the disclosure will be more apparent from the following description of embodiments of the disclosure with reference to the accompanying drawings, in which:

FIG. 1 schematically illustrates an application scenario diagram of a data processing method, apparatus, device, medium and program product according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a data processing method according to an embodiment of the disclosure;

FIG. 3 schematically illustrates a flow chart of constructing traffic flow data according to an embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow chart for acquiring corresponding offline operation and maintenance data based on real-time data according to an embodiment of the present disclosure;

FIG. 5 schematically illustrates a flow chart for calculating offline operation and maintenance data based on offline data, in accordance with an embodiment of the present disclosure;

FIG. 6 schematically illustrates a flow chart of determining current operational data of a data processing system in accordance with an embodiment of the present disclosure;

FIG. 7 schematically illustrates a flow chart of monitoring exception data according to an embodiment of the present disclosure;

FIG. 8 schematically illustrates a flow chart for retrieving and storing user request data according to an embodiment of the disclosure;

FIG. 9 schematically illustrates a flow diagram of processing request data by a data processing module according to an embodiment of the disclosure;

FIG. 10 schematically illustrates a block diagram of a data processing system in accordance with an embodiment of the present disclosure; and

fig. 11 schematically illustrates a block diagram of an electronic device adapted to implement a data processing method according to an embodiment of the disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.

Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

In the technical solution of the present disclosure, the related user information (including, but not limited to, user personal information, user image information, user equipment information, such as location information, etc.) and data (including, but not limited to, data for analysis, stored data, displayed data, etc.) are information and data authorized by the user or sufficiently authorized by each party, and the related data is collected, stored, used, processed, transmitted, provided, disclosed, applied, etc. and processed, all in compliance with the related laws and regulations and standards of the related country and region, necessary security measures are taken, no prejudice to the public order, and corresponding operation entries are provided for the user to select authorization or rejection.

The embodiment of the disclosure provides a data processing method, which comprises the following steps: in response to acquiring the real-time data, constructing service flow data; wherein the traffic stream data comprises a plurality of consecutive traffic events arranged in time sequence within a specified time window; performing real-time stream calculation operation on the service stream data to obtain real-time operation and maintenance data; the real-time operation and maintenance data are used for reflecting the current occupation condition and/or queuing condition of each node in the data processing system; acquiring corresponding offline operation and maintenance data based on the real-time data; the offline operation and maintenance data are obtained by calculating offline data in a specified time range; the offline operation and maintenance data are used for reflecting the change condition of the states of all nodes of the data processing system in a certain past time period; calculating current operation data of the data processing system based on the offline operation data and the real-time operation data; and determining a business handling flow based on the current operation data, and feeding back the business handling flow to the client so that the client executes business handling in the data processing system according to the business handling flow.

Fig. 1 schematically illustrates an application scenario diagram of a data processing method, system, device, medium and program product according to an embodiment of the present disclosure.

As shown in fig. 1, the application scenario 100 according to this embodiment may include a terminal device 101, a terminal device 102, a terminal device 103, a network 104, and a server 105. The network 104 is used as a medium to provide communication links between the terminal devices 101, 102, 103 and the server 105. The network 104 may include various connection types, such as wired, wireless communication links, or fiber optic cables, among others.

The user may interact with the server 105 via the network 104 using the terminal devices 101, 102, 103 to receive or send messages or the like. Various communication client applications, such as shopping class applications, web browser applications, search class applications, instant messaging tools, mailbox clients, social platform software, etc. (by way of example only) may be installed on the terminal devices 101, 102, 103.

The terminal devices 101, 102, 103 may be a variety of electronic devices having a display screen and supporting web browsing, including but not limited to smartphones, tablets, laptop and desktop computers, and the like.

The server 105 may be a server providing various services, such as a background management server (by way of example only) providing support for websites browsed by users using the terminal devices 101, 102, 103. The background management server may analyze and process the received data such as the user request, and feed back the processing result (e.g., the web page, information, or data obtained or generated according to the user request) to the terminal device.

It should be noted that the service transacting method provided by the embodiments of the present disclosure may be generally executed by the server 105. Accordingly, the business transaction system provided by the embodiments of the present disclosure may be generally disposed in the server 105. The service transacting method provided by the embodiments of the present disclosure may also be performed by a server or a server cluster that is different from the server 105 and is capable of communicating with the terminal devices 101, 102, 103 and/or the server 105. Accordingly, the service handling system provided by the embodiments of the present disclosure may also be provided in a server or a server cluster different from the server 105 and capable of communicating with the terminal devices 101, 102, 103 and/or the server 105.

It should be understood that the number of terminal devices, networks and servers in fig. 1 is merely illustrative. There may be any number of terminal devices, networks, and servers, as desired for implementation.

The service transacting method of the disclosed embodiment will be described in detail with reference to fig. 2 to 6 based on the scenario described in fig. 1.

Fig. 2 schematically illustrates a flow chart of a data processing method according to an embodiment of the present disclosure.

As shown in fig. 2, the data processing method of this embodiment includes operations S210 to S250.

In response to acquiring the real-time data, traffic stream data is constructed in operation S210.

In some embodiments, the bank systems operate to generate a large amount of data, the data is processed and stored in the data storage module, and the data calculation module acquires real-time data from the data storage module. And responding to the data calculation module to acquire real-time data from the data storage module, and constructing service flow data.

Stream data is data that is continuously generated by multiple data sources, is a continuous, endless sequence of data, and is typically divided by a specified time window. The traffic stream data in this application refers to a plurality of consecutive traffic events arranged in time sequence within a specified time window.

In operation S220, a real-time stream calculation operation is performed on the traffic stream data to obtain real-time operation and maintenance data.

In some embodiments, after the service flow data is obtained, the service flow data is uploaded to an offline model trained based on the offline data, and real-time flow calculation operation is performed on the service flow by the offline model to obtain real-time operation and maintenance data. The real-time operation and maintenance data are used for reflecting the current occupation condition and/or queuing condition of each node in the data processing system.

According to the embodiment of the disclosure, through carrying out real-time stream calculation operation on the service stream data, real-time analysis and processing of the data can be realized, immediate insight and feedback are provided, so that the current state of each calculation node in the data processing system can be quickly found, the calculation nodes can be timely optimized and adjusted, and the performance of the data processing system is ensured.

In operation S230, corresponding offline operation and maintenance data is acquired based on the real-time data.

In some embodiments, in addition to performing real-time streaming computation operations on real-time data to obtain real-time operational dimension data, it is desirable to obtain corresponding offline operational dimension data based on the real-time data. The offline operation and maintenance data are calculated by a batch calculation submodule in the data calculation module. The offline data is data before a certain designated time, the offline operation data is obtained by calculating the offline data within a designated time range, and the offline operation data is used for reflecting the change condition of the states of all nodes of the data processing system in a certain past time period.

Compared with real-time operation and maintenance data, the offline operation and maintenance data is obtained by batch calculation of batch data in the data storage module, and can effectively reflect the change rule of the states of all nodes in the data processing system. The batch calculation can execute calculation processing on a large amount of data, and accurately and objectively summarize the operation rule of the data processing system.

In operation S240, current operation data of the data processing system is calculated based on the offline operation and maintenance data and the real-time operation and maintenance data.

In some embodiments, the real-time operation and maintenance data has the characteristics of strong timeliness and capability of reflecting the current operation condition of the data processing system in real time, and the offline operation and maintenance data is obtained by jointly calculating a large amount of offline data and can objectively reflect the historical operation condition and the historical operation rule of the data processing system. Therefore, the method combines the offline operation and maintenance data with the real-time operation and maintenance data, and determines the current operation data of the data processing system from the timeliness and regularity dimensions together so as to effectively evaluate the current operation condition of the data processing system. The current operation data comprise the occupation condition, queuing condition and processing speed of each node in the data processing system at the current moment.

In operation S250, a business process is determined based on the current operation data, and the business process is fed back to the client, so that the client performs business in the data processing system according to the business process.

In some embodiments, the current operation data of the data processing system is used for determining the business handling flow of the current business handling request, and the business handling flow is fed back to the client side, so that the client performs business handling in the data processing system according to the business handling flow. The service handling process of the current service handling request is determined, for example, the service handling process with the shortest waiting time for the user can be used for effectively realizing client distribution and improving the service handling experience of the client.

According to the method and the device, the current running condition of the data processing system is determined jointly by combining the real-time streaming data and the offline data, the corresponding business handling strategy is determined based on the current running condition of the data processing system, the customer business handling experience is improved, meanwhile, customer shunting is achieved, and the probability of congestion of the data processing system due to improper customer shunting is reduced.

Fig. 3 schematically illustrates a flow chart of constructing traffic flow data according to an embodiment of the present disclosure.

As shown in fig. 3, the construction traffic stream data of this embodiment includes operations S310 to S340.

In operation S310, a data type of real-time data is acquired.

In some embodiments, since the service requests to be handled by the clients are different and the handling procedures of different services are different, the corresponding processing nodes in the data processing system will also be different from each other in the different service handling procedures. Therefore, real-time data can be classified based on service types, and the data types corresponding to different real-time data are different.

In operation S320, a packet to which real-time data belongs is determined based on the data type.

In some embodiments, the real-time data is grouped according to its data type.

In operation S330, acquiring event time of real-time data; the event time is the time of actual generation of real-time data; and

in operation S340, real-time data is allocated to a corresponding time window in the belonging packet based on the event time, resulting in traffic stream data.

In some embodiments, a time window corresponding to the real-time data is determined according to the event time of the real-time data to obtain the service data stream. Wherein the event time is determined based on a time stamp of real-time data, and refers to the time when the event actually occurs. The event time is taken as the basis of stream data processing, and real-time data can be accurately distributed to a corresponding time window, so that the real occurrence condition of the event is reflected.

Determining stream data based on event time and executing stream processing can solve the problem of data disorder caused by different devices and different data transmission conditions. Even if the data arrives in disorder, the data is still calculated according to the event actually happening to ensure the consistency and accuracy of the result in the data disorder scene, thereby ensuring the accuracy of the real-time operation and maintenance data.

Fig. 4 schematically illustrates a flow chart for acquiring corresponding offline operation and maintenance data based on real-time data according to an embodiment of the present disclosure.

As shown in fig. 4, the method for acquiring corresponding offline operation and maintenance data based on real-time data in this embodiment includes operations S410 to S420.

In operation S410, a corresponding offline operation and maintenance data set is acquired according to the data type of the real-time data.

In some embodiments, the offline operation and maintenance data is obtained by calculating after the offline data is transmitted to the pre-training model, so that the change condition of each node in different time ranges can be effectively reflected.

The batch processing module acquires the offline data based on a preset time interval, acquires all the offline data generated in the time interval from the data storage module, and transmits the offline data to the pre-training model for calculation to obtain the offline operation and maintenance data in the current time range. Wherein the offline operation and maintenance data set comprises a plurality of offline operation and maintenance data of different time ranges.

In operation S420, offline operation and maintenance data for a designated time is queried in the offline operation and maintenance data set.

In some embodiments, the offline operation and maintenance data set is queried for offline operation and maintenance data closest to the time range of the time window in which the real-time data is located, and/or offline operation and maintenance data containing the time range of the current time window on different dates.

In the implementation process, for example, the offline data acquisition and calculation operation is preset to be performed every 2 hours, that is, an offline operation and maintenance data is generated every 2 hours. The time window in which the real-time data is located is 13: 45-13: 50, find generation time and 13:45 off-line operational data sets with shortest intervals. And/or, on the previous day, 13: 45-13: 50 this time frame of offline operation and maintenance data.

Fig. 5 schematically illustrates a flowchart for calculating offline operation and maintenance data based on offline data according to an embodiment of the present disclosure.

As shown in fig. 5, the offline operation and maintenance data is calculated based on the offline data in this embodiment, including operations S510 to S530.

In operation S510, an offline processing task is initiated based on a preset time threshold.

In some embodiments, an offline processing task may be initiated to the data computation module by a preset time threshold to regularly obtain offline operation and maintenance data. The preset time threshold may be, for example, 2 hours, 10 hours, 24 hours, 36 hours, or the like.

In response to receiving the offline processing task, offline data within a time threshold is retrieved from the data storage module in operation S520.

In some embodiments, in response to receiving the offline processing task, the data calculation module obtains offline data generated within a time threshold from the data storage module.

In operation S530, the offline data is transferred to the pre-training model, and the offline operation and maintenance data is calculated.

In some embodiments, after the offline data is acquired, the offline data is transferred to a pre-training model, which computes the offline data to obtain offline operational dimensional data. The pre-training model is a machine learning model which is obtained based on a large amount of historical data and can be adaptively updated. And each time the offline data is transmitted into the pre-training model, the parameters of the pre-training model are updated in real time based on the calculation process besides the offline operation and data obtained through calculation, and the updated pre-training model is utilized to execute the next offline processing task.

FIG. 6 schematically illustrates a flow chart for determining current operational data of a data processing system in accordance with an embodiment of the present disclosure.

As shown in fig. 6, the present operation data of the data processing system according to this embodiment includes operations S610 to S620.

In operation S610, the state of each node at the next moment is predicted based on the offline operation and maintenance data and the real-time operation and maintenance data, so as to obtain state change data of each node in the data processing system.

In some embodiments, the real-time operation and maintenance data may timely reflect the current operation status of the data processing system, and the offline operation and maintenance data may reflect the processing rule and the change condition of each node in the data processing system, so the disclosure proposes to predict the state of each node at the next moment based on the real-time operation and maintenance data and the offline operation and maintenance data, so as to obtain the state change data of each node in the data processing system.

In operation S620, an operation state of the data processing system is commonly determined based on the real-time operation and maintenance data and the state change data of each node.

In some embodiments, the operation state of the data processing system is determined together according to the real-time operation and maintenance data and the state change data of each node, so that the operation state of the data processing system is monitored accurately in real time.

In a specific implementation process, the method and the device adopt a data computing framework with high throughput and a plurality of program interfaces (Application Programming Interface, API) to perform data computation, and different APIs are adopted for stream processing and batch processing in the data framework, so that the coupling degree among programs in a data processing system is effectively reduced. And the higher throughput can effectively improve the processing capacity of large-batch data, improve the resource utilization rate and reduce the system overhead.

The data processing method provided by the embodiment of the disclosure further comprises the step of monitoring abnormal data in the data processing system.

Fig. 7 schematically illustrates a flow chart of monitoring exception data according to an embodiment of the present disclosure.

As shown in fig. 7, the monitoring of abnormal data of this embodiment includes operations S710 to S720.

In operation S710, an operational state of the data processing system is monitored.

In operation S720, an alarm operation is performed on an abnormal node in the data processing system based on a preset rule.

In some embodiments, the abnormal data is discovered in time by monitoring the running state of the data processing system, and the abnormal data is processed in time, so that the safe and stable running of the data processing system is ensured. The types of the abnormal nodes at least comprise: nodes with processing speed lower than a preset threshold value and nodes with processing data higher than a preset value. Alarm operations corresponding to different types of abnormal nodes are different, so that operation and maintenance personnel can find the abnormal nodes in time and realize quick investigation of abnormal conditions.

Further, the data processing method provided by the present disclosure further includes: and visually displaying the analyzed data by utilizing a data visualization tool, and displaying the health degree of the system in a multi-dimensional and full-view angle mode by graphically displaying each index. The operation and maintenance personnel can visually check the overall state of the system, wherein each display graph can drill down layer by layer, fine-grained display of the operation state of the system is realized, and the rapid determination of the fault source is facilitated.

The data processing method provided by the embodiment of the disclosure further comprises the steps of acquiring and storing the user request data.

Fig. 8 schematically illustrates a flow chart for retrieving and storing user request data according to an embodiment of the disclosure.

As shown in fig. 8, the acquisition and storage of user request data in this embodiment includes operations S810 to S820.

In operation S810, request data is transmitted to a message queue.

In operation S820, the request data in the message queue are sequentially sent to the data processing module for processing, so as to obtain real-time data, and the real-time data is sent to the data storage module.

In some embodiments, the message queue is used for forwarding the request data, so that on one hand, the response speed of the message sending end can be improved, and on the other hand, the decoupling of the message sending end and the data processing system can be realized by sending the message through the message queue, and the stability of the data processing system is effectively improved.

Fig. 9 schematically illustrates a flow chart of processing of request data by a data processing module according to an embodiment of the disclosure.

As shown in fig. 9, the data processing module of this embodiment processes the request data, including operations S910 to S930.

In operation S910, the newly uploaded request data in the message queue is parsed.

In operation S920, performing a preprocessing operation on the parsed request data to obtain preprocessed data; wherein the preprocessing operation comprises: at least one of data cleansing, data deduplication, data association, data merging.

In operation S930, a secondary processing operation is performed based on the data type of the pre-processed data, resulting in real-time data; wherein the secondary treatment operation comprises an aggregation calculation and/or a desensitization operation.

In some embodiments, before storing the request data in the data storage module, the request data needs to be sent to the data processing module, the data processing module processes the request data and sends the processed request book data to the data storage module for storage, so as to ensure the normalization and correctness of the data in the data storage module.

In an implementation, the data storage module may be configured, for example, by a data warehouse to store different types of data in a sorted manner, for example, time series data such as index data is stored in a time series database, and relationship class data and data collected in a recursive query based on relationships are stored in a graph database. Because the business data are more various, the data of each heterogeneous data source database can be uniformly managed by utilizing the data warehouse, and the data of different data sources can be integrated on a higher level of abstraction according to the actual demands of users, so that all the data can be organized around a certain theme. And the data stored in the data warehouse is usually a series of historical snapshots, and is not allowed to be modified, so that the safety and the authenticity of the stored data can be effectively ensured.

Based on the data processing method, the disclosure also provides a data processing system. The device will be described in detail below in connection with fig. 10.

FIG. 10 schematically illustrates a block diagram of a data processing system according to an embodiment of the present disclosure.

As shown in fig. 10, the data processing system 1000 of this embodiment includes a construction module 1010, a real-time stream calculation module 1020, an acquisition module 1030, a calculation module 1040, and a determination module 1050.

The construction module 1010 is configured to construct service flow data in response to acquiring real-time data; wherein the traffic stream data comprises a plurality of consecutive traffic events arranged in time sequence within a specified time window. In an embodiment, the construction module 1010 may be configured to perform the operation S210 described above, which is not described herein.

The real-time flow calculation module 1020 is configured to perform a real-time flow calculation operation on the traffic flow data to obtain real-time operation and maintenance data; the real-time operation and maintenance data are used for reflecting the current occupation condition and/or queuing condition of each node in the data processing system. In an embodiment, the real-time stream calculation module 1020 may be used to perform the operation S220 described above, which is not described herein.

The acquiring module 1030 is configured to acquire corresponding offline operation and maintenance data based on the real-time data; the offline operation and maintenance data are obtained by calculating offline data in a specified time range; the offline operation data is used to reflect changes in the state of each node of the data processing system over a period of time. In an embodiment, the obtaining module 1030 may be configured to perform the operation S230 described above, which is not described herein.

The calculation module 1040 is used to calculate current operational data of the data processing system based on the offline operational data and the real-time operational data. In an embodiment, the calculating module 1040 may be used to perform the operation S240 described above, which is not described herein.

The determining module 1050 is configured to determine a business process based on the current operation data, and feed back the business process to the client, so that the client performs business in the data processing system according to the business process. In an embodiment, the determining module 1050 may be configured to perform the operation S250 described above, which is not described herein.

In addition, the data processing system 1000 provided by the present disclosure further includes: a data processing module 1060 and a data storage module 1070.

The data processing module 1060 is configured to process the request data sent by the message queue. The data storage module 1070 is used for storing real-time data processed by the data processing module 1060 and offline data.

Any of the building module 1010, the real-time stream computing module 1020, the obtaining module 1030, the computing module 1040, the determining module 1050 may be combined in one module to be implemented, or any of the modules may be split into a plurality of modules, according to embodiments of the present disclosure. Alternatively, at least some of the functionality of one or more of the modules may be combined with at least some of the functionality of other modules and implemented in one module. At least one of the building module 1010, the real-time stream computing module 1020, the acquisition module 1030, the computing module 1040, the determination module 1050 may be implemented at least in part as hardware circuitry, such as a Field Programmable Gate Array (FPGA), a Programmable Logic Array (PLA), a system on a chip, a system on a substrate, a system on a package, an Application Specific Integrated Circuit (ASIC), or in hardware or firmware in any other reasonable manner of integrating or packaging the circuitry, or in any one of or a suitable combination of three of software, hardware, and firmware, according to embodiments of the present disclosure. Alternatively, at least one of the building module 1010, the real-time stream computing module 1020, the obtaining module 1030, the computing module 1040, the determining module 1050 may be at least partially implemented as a computer program module, which when executed, may perform the corresponding functions.

Fig. 11 schematically illustrates a block diagram of an electronic device adapted to implement a data processing method according to an embodiment of the disclosure.

As shown in fig. 11, an electronic device 1100 according to an embodiment of the present disclosure includes a processor 1101 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 1102 or a program loaded from a storage section 1108 into a Random Access Memory (RAM) 1103. The processor 1101 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. The processor 1101 may also include on-board memory for caching purposes. The processor 1101 may comprise a single processing unit or a plurality of processing units for performing the different actions of the method flow according to embodiments of the present disclosure.

In the RAM 1103, various programs and data necessary for the operation of the electronic device 1100 are stored. The processor 1101, ROM 1102, and RAM 1103 are connected to each other by a bus 1104. The processor 1101 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM 1102 and/or the RAM 1103. Note that the program may be stored in one or more memories other than the ROM 1102 and the RAM 1103. The processor 1101 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in the one or more memories.

According to an embodiment of the disclosure, the electronic device 1100 may also include an input/output (I/O) interface 1105, the input/output (I/O) interface 1105 also being connected to the bus 1104. The electronic device 1100 may also include one or more of the following components connected to the I/O interface 1105: an input section 1106 including a keyboard, a mouse, and the like; an output portion 1107 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, a speaker, and the like; a storage section 1108 including a hard disk or the like; and a communication section 1109 including a network interface card such as a LAN card, a modem, and the like. The communication section 1109 performs communication processing via a network such as the internet. The drive 1110 is also connected to the I/O interface 1105 as needed. Removable media 1111, such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like, is installed as needed in drive 1110, so that a computer program read therefrom is installed as needed in storage section 1108.

The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs which, when executed, implement methods in accordance with embodiments of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, the computer-readable storage medium may include ROM 1102 and/or RAM 1103 described above and/or one or more memories other than ROM 1102 and RAM 1103.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowcharts. The program code, when executed in a computer system, causes the computer system to implement the item recommendation method provided by embodiments of the present disclosure.

The above-described functions defined in the system/apparatus of the embodiments of the present disclosure are performed when the computer program is executed by the processor 1101. The systems, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program can also be transmitted, distributed over a network medium in the form of signals, downloaded and installed via the communication portion 1109, and/or installed from the removable media 1111. The computer program may include program code that may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such an embodiment, the computer program can be downloaded and installed from a network via the communication portion 1109, and/or installed from the removable media 1111. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by the processor 1101. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

According to embodiments of the present disclosure, program code for performing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, such computer programs may be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, such as Java, c++, python, "C" or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be provided in a variety of combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.

Claims (13)

1. A data processing method, comprising:

in response to acquiring the real-time data, constructing service flow data; wherein the traffic stream data comprises a plurality of consecutive traffic events arranged in time sequence within a specified time window;

Performing real-time stream calculation operation on the service stream data to obtain real-time operation and maintenance data; the real-time operation and maintenance data are used for reflecting the current occupation condition and/or queuing condition of each node in the data processing system;

acquiring corresponding offline operation and maintenance data based on the real-time data; the offline operation and maintenance data are obtained by calculating offline data in a specified time range; the offline operation and maintenance data are used for reflecting the change condition of the states of all nodes of the data processing system in a certain past time period;

calculating current operation data of a data processing system based on the offline operation and maintenance data and the real-time operation and maintenance data;

and determining a business handling flow based on the current operation data, and feeding back the business handling flow to a client so that the client executes business handling in the data processing system according to the business handling flow.

2. The data processing method according to claim 1, wherein the constructing the traffic stream data in response to acquiring the real-time data includes:

acquiring the data type of the real-time data;

determining a packet to which the real-time data belongs based on the data type;

acquiring event time of the real-time data; the event time is the time of the real-time data actually generated; and

And distributing the real-time data to a corresponding time window in the belonging packet based on the event time to obtain service flow data.

3. The data processing method according to claim 1, wherein the acquiring corresponding offline operation and maintenance data based on the real-time data includes:

acquiring a corresponding offline operation and maintenance data set according to the data type of the real-time data;

and querying the offline operation and maintenance data of the designated time in the offline operation and maintenance data set.

4. A data processing method according to claim 3, wherein the offline operation data at the specified time comprises:

offline operation and maintenance data closest to the time range of the time window in which the real-time data is located; and/or

Offline operation and maintenance data containing the current time window at different dates.

5. The data processing method of claim 4, further comprising computing the offline operational dimension data based on the offline data:

initiating an offline processing task based on a preset time threshold;

in response to receiving the offline processing task, obtaining offline data within a time threshold from the data storage module;

and transmitting the offline data into a pre-training model, and calculating to obtain offline operation and maintenance data.

6. The data processing method of claim 1, the calculating current operational data of a data processing system based on the offline operational data and the real-time operational data, comprising:

predicting the state of each node at the next moment based on the offline operation and maintenance data and the real-time operation and maintenance data to obtain state change data of each node in the data processing system;

and determining the running state of the data processing system based on the real-time operation and maintenance data and the state change data of each node.

7. The data processing method of claim 6, further comprising:

monitoring an operational state of the data processing system;

executing alarm operation on abnormal nodes in the data processing system based on preset rules;

wherein, the types of the abnormal nodes at least comprise: nodes with processing speed lower than a preset threshold value and nodes with processing data higher than a preset value; alarm operations corresponding to different types of abnormal nodes are different.

8. The data processing method of claim 1, further comprising:

transmitting the request data to a message queue;

and sequentially sending the request data in the message queue to a data processing module for processing according to the sequence, obtaining real-time data and sending the real-time data to a data storage module.

9. The data processing method according to claim 8, wherein the sequentially sending the request data in the message queue to the data processing module for processing includes:

analyzing the latest uploaded request data in the message queue;

preprocessing operation is carried out on the parsed request data, and preprocessed data is obtained; wherein the preprocessing operation includes: at least one of data cleansing, data deduplication, data association, data merging;

performing secondary processing operation based on the data type of the preprocessed data to obtain the real-time data; wherein the secondary treatment operation comprises an aggregation calculation and/or a desensitization operation.

10. A data processing system, comprising:

the construction module is used for responding to the acquired real-time data and constructing service flow data; wherein the traffic stream data comprises a plurality of consecutive traffic events arranged in time sequence within a specified time window;

the real-time flow calculation module is used for executing real-time flow calculation operation on the service flow data to obtain real-time operation and maintenance data; the real-time operation and maintenance data are used for reflecting the current occupation condition and/or queuing condition of each node in the data processing system;

The acquisition module is used for acquiring corresponding offline operation and maintenance data based on the real-time data; the offline operation and maintenance data are obtained by calculating offline data in a specified time range; the offline operation and maintenance data are used for reflecting the change condition of the states of all nodes of the data processing system in a certain past time period;

the calculating module is used for calculating the current operation data of the data processing system based on the offline operation and maintenance data and the real-time operation and maintenance data; and

and the determining module is used for determining a business handling flow based on the current operation data and feeding the business handling flow back to the client so that the client executes business handling in the data processing system according to the business handling flow.

11. An electronic device, comprising:

one or more processors;

storage means for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-9.

12. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1 to 9.

13. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 9.