CN117493426A

CN117493426A - Data visualization processing method, system, electronic equipment and storage medium

Info

Publication number: CN117493426A
Application number: CN202311566838.2A
Authority: CN
Inventors: 赵鑫
Original assignee: Industrial and Commercial Bank of China Ltd ICBC
Current assignee: Industrial and Commercial Bank of China Ltd ICBC
Priority date: 2023-11-22
Filing date: 2023-11-22
Publication date: 2024-02-02

Abstract

The disclosure provides a data visualization processing method, a system, electronic equipment and a storage medium, relates to the technical field of big data, and can be applied to the financial field or other fields. The data visualization processing method comprises the following steps: receiving a data request; data segmentation is carried out on the data request of the data visualization processing system according to a preset rule, N visualized sub-jobs are generated, and N is an integer greater than 1; correspondingly distributing N visualized sub-jobs of the data visualization processing system to N job executors, wherein each job executor of the data visualization processing system performs data processing on the corresponding visualized sub-jobs of the data visualization processing system to generate N processed data; and carrying out visual combination and processing on N pieces of processing data of the data visual processing system, and outputting visual data.

Description

Data visualization processing method, system, electronic equipment and storage medium

Technical Field

The present disclosure relates to the field of big data technologies, and may be applied to the financial field or other fields, and in particular, to a data visualization processing method, a system, an electronic device, and a storage medium.

Background

Data visualization refers to theory, method and technology that data in a large data set is represented in a graphic image form and then interactive processing is carried out.

In the data visualization process, the process of converting input data into visualized data takes up a significant portion of the overall time consumption. When the scale of the data is too large, the visualization processing process is longer, and the user experience is affected.

Disclosure of Invention

The present disclosure proposes a data visualization processing method, system, electronic device, storage medium and computer program product.

According to one aspect of the present disclosure, there is provided a data visualization processing method including: receiving a data request; data segmentation is carried out on the data request according to a preset rule, N visualized sub-jobs are generated, and N is an integer greater than 1; correspondingly distributing the N visualized sub-jobs to N job executors, wherein each job executor carries out data processing on the corresponding visualized sub-job to generate N processing data; and carrying out visual combination and processing on the N processing data, and outputting visual data.

According to an embodiment of the present disclosure, performing data segmentation on a data request according to a preset rule includes: and uniformly dividing the data request into N visualized sub-jobs with the same granularity according to the granularity of the preset batch.

According to an embodiment of the present disclosure, before the step of distributing the N visualized sub-jobs to the N job executors, further includes: and executing parameter initialization processing on each job executor based on the preset parameters so as to make the parameters of the visual coordinate system of each job executor consistent.

According to an embodiment of the present disclosure, further comprising: generating a sub-job list, wherein the sub-job list stores visualized sub-jobs which are processed correspondingly by different job executors; each job executor completes data processing and outputs a completion identification matched with the corresponding processed visualized sub-job; after the N job executors finish data processing, matching each output completion identifier with each visualized sub-job stored in the sub-job list; if the visualized sub-job with the failed matching exists in the sub-job list, the job executor carries out data processing on the visualized sub-job again.

According to an embodiment of the present disclosure, further comprising: monitoring whether the operation executor is abnormal; if the operation executor is abnormal, the residual visualized sub-operation which is not processed by the operation executor is acquired, and the residual visualized sub-operation is distributed to at least one of the residual operation executors for data processing.

According to an embodiment of the present disclosure, a poll distribution manner is employed to distribute N visualization sub-jobs to N job executors.

According to an embodiment of the present disclosure, the visual data is a non-aggregated chart, the method comprising: each job executor performs visualization processing on the corresponding visualized sub-job to generate N pieces of processing data; visual compounding and processing includes: and merging and rendering the N pieces of processing data, and outputting visual data.

According to an embodiment of the present disclosure, each job executor performs visualization processing on a corresponding visualization sub-job to generate N pieces of processing data includes: each job executor performs visualization processing and primary rendering on the corresponding visualized sub-job based on the visualized coordinate system with the same parameters, and generates a relative coordinate position of the visualized sub-job in the visualized coordinate system and a corresponding primary rendering result value; visual compounding and processing includes: the relative coordinate position in each processing data is converted into an absolute coordinate position, and final rendering is performed based on the primary rendering result value to generate visualized data.

According to an embodiment of the present disclosure, the visual data is an aggregated chart, the method comprising: each job executor performs data first aggregation processing on the corresponding visualized sub-job to generate N pieces of processing data; visual compounding and processing includes: performing second aggregation on the N pieces of processing data to generate aggregated data; and performing visualization processing and rendering processing on the aggregated data, and outputting the visualized data.

According to another aspect of the present disclosure, there is provided a data visualization processing system including: the data segmentation module is used for receiving the data request, carrying out data segmentation on the data request according to a preset rule, generating N visualized sub-jobs, wherein N is an integer greater than 1; the job coordination module is used for correspondingly distributing the N visualized sub-jobs to N job executors, and each job executor carries out data processing on the corresponding visualized sub-job to generate N processing data; and the merging module is used for carrying out visual combination and processing on the N processing data and outputting visual data.

According to an embodiment of the present disclosure, the job coordination module is further configured to: and executing parameter initialization processing on each job executor based on the preset parameters so as to make the parameters of the visual coordinate system of each job executor consistent.

According to the embodiment of the disclosure, the job executor distributes N visualized sub-jobs to N job executors based on a sub-job list, wherein the sub-job list stores visualized sub-jobs processed by different job executors; the job executor completes data processing and outputs a completion identification matched with the corresponding processed visualized sub-job; the job coordination module is also for: after the N job executors finish data processing, matching each output completion identifier with each visualized sub-job stored in the sub-job list; if the visualized sub-job with the failed matching exists in the sub-job list, the job executor carries out data processing on the visualized sub-job again.

According to an embodiment of the present disclosure, the job coordination module is further configured to: detecting whether the operation executor is abnormal, if so, acquiring the residual visualized sub-operation which is not processed by the operation executor, and distributing the residual visualized sub-operation to at least one of the residual operation executors for data processing.

According to an embodiment of the present disclosure, the visual data is a non-aggregated chart, and the job executor is configured to: performing visualization processing on the visualized sub-jobs to generate N pieces of processing data; the merge module is configured to: and merging and rendering the N pieces of processing data, and outputting visual data.

According to an embodiment of the present disclosure, the visual data is an aggregated graph, and the job executor is configured to: performing data first aggregation processing on the visualized sub-jobs to generate N pieces of processing data; the merge module is configured to: performing second aggregation on the N pieces of processing data to generate aggregated data; and performing visualization processing and rendering processing on the aggregated data, and outputting the visualized data.

According to another aspect of the present disclosure, there is provided an electronic device including: one or more processors; and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the data visualization processing method as described above.

According to another aspect of the present disclosure, there is provided a computer-readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform a data visualization processing method as described above.

According to another aspect of the present disclosure, there is provided a computer program product comprising a computer program which, when executed by a processor, implements a data visualization processing method as above.

It should be understood that the description in this section is not intended to identify key or critical features of the embodiments of the disclosure, nor is it intended to be used to limit the scope of the disclosure. Other features of the present disclosure will become apparent from the following specification.

Drawings

The above and other objects, features and advantages of the present disclosure will become more apparent from the following description of embodiments thereof with reference to the accompanying drawings in which:

FIG. 1 schematically illustrates a system architecture 100 of a data visualization processing method and system according to an embodiment of the present disclosure;

FIG. 2 schematically illustrates a flow chart of a data visualization processing method according to an embodiment of the present disclosure;

FIG. 3 schematically illustrates a flow chart in a data visualization processing method according to another embodiment of the present disclosure;

FIG. 4 schematically illustrates a flow chart of a data visualization processing method according to yet another embodiment of the present disclosure;

FIG. 5 schematically illustrates a flow chart of a data visualization processing method according to yet another embodiment of the present disclosure;

FIG. 6 schematically illustrates a functional block diagram of a data visualization processing system, in accordance with an embodiment of the present disclosure;

FIG. 7 schematically illustrates a schematic diagram of a data slicing principle of a data visualization processing system, in accordance with an embodiment of the present disclosure;

FIG. 8 schematically illustrates a schematic diagram of a data visualization processing system in accordance with an embodiment of the present disclosure;

FIG. 9 schematically illustrates a schematic diagram of a data visualization processing system in accordance with another embodiment of the present disclosure;

fig. 10 schematically illustrates a block diagram of an electronic device adapted for a data visualization processing method according to an embodiment of the present disclosure.

Detailed Description

Hereinafter, embodiments of the present disclosure will be described with reference to the accompanying drawings. It should be understood that the description is only exemplary and is not intended to limit the scope of the present disclosure. In the following detailed description, for purposes of explanation, numerous specific details are set forth in order to provide a thorough understanding of the embodiments of the present disclosure. It may be evident, however, that one or more embodiments may be practiced without these specific details. In addition, in the following description, descriptions of well-known structures and techniques are omitted so as not to unnecessarily obscure the concepts of the present disclosure.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the disclosure. The terms "comprises," "comprising," and/or the like, as used herein, specify the presence of stated features, steps, operations, and/or components, but do not preclude the presence or addition of one or more other features, steps, operations, or components.

All terms (including technical and scientific terms) used herein have the same meaning as commonly understood by one of ordinary skill in the art unless otherwise defined. It should be noted that the terms used herein should be construed to have meanings consistent with the context of the present specification and should not be construed in an idealized or overly formal manner.

Where expressions like at least one of "A, B and C, etc. are used, the expressions should generally be interpreted in accordance with the meaning as commonly understood by those skilled in the art (e.g.," a system having at least one of A, B and C "shall include, but not be limited to, a system having a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.). Where a formulation similar to at least one of "A, B or C, etc." is used, in general such a formulation should be interpreted in accordance with the ordinary understanding of one skilled in the art (e.g. "a system with at least one of A, B or C" would include but not be limited to systems with a alone, B alone, C alone, a and B together, a and C together, B and C together, and/or A, B, C together, etc.).

Some of the block diagrams and/or flowchart illustrations are shown in the figures. It will be understood that some blocks of the block diagrams and/or flowchart illustrations, or combinations of blocks in the block diagrams and/or flowchart illustrations, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, or other programmable data processing apparatus, such that the instructions, when executed by the processor, create means for implementing the functions/acts specified in the block diagrams and/or flowchart. The techniques of this disclosure may be implemented in hardware and/or software (including firmware, microcode, etc.). Additionally, the techniques of this disclosure may take the form of a computer program product on a computer-readable storage medium having instructions stored thereon, the computer program product being for use by or in connection with an instruction execution system.

In the technical solution of the present disclosure, the related user information (including, but not limited to, user personal information, user image information, user equipment information, such as location information, etc.) and data (including, but not limited to, data for analysis, stored data, displayed data, etc.) are information and data authorized by the user or sufficiently authorized by each party, and the related data is collected, stored, used, processed, transmitted, provided, disclosed, applied, etc. and processed, all in compliance with the related laws and regulations and standards of the related country and region, necessary security measures are taken, no prejudice to the public order, and corresponding operation entries are provided for the user to select authorization or rejection.

In the data visualization process, the scale of the input data is sometimes large, for example, the data visualization corresponding to the performance analysis data, the biological information analysis data, and the like, and the data amount of the input data often reaches the millions or even tens of millions. Thus, the process of data visualization can be quite lengthy, resulting in a poor user experience. In the current data visualization method, the task granularity is limited, the visualization processing of a single data set is a single service node, namely, the granularity of the distributed processing is taken as the whole single table data, and when the input data set is large in scale, the visualization processing speed is limited by single node resources, and the processing delay is generated.

Aiming at the technical problems in the related art, the embodiment of the disclosure provides a data visualization processing method, which generates N visualized sub-jobs after data segmentation is performed on a data request, and distributes the visualized sub-jobs to different job executors for processing. Because the visualized sub-job is obtained by data segmentation of the data request, the task granularity of the visualized sub-job is smaller than that of the data request. Compared with the operation executor for processing the data request, the operation executor has smaller processing load on the visualized sub-operation, and can greatly shorten the processing time. Each job executor performs data processing on the corresponding visualized sub-job, so that the workload of each node can be balanced, and load balancing is realized. Finally, the N processing data are subjected to visual combination and processing, so that the time consumption for performing visual processing on the data request can be greatly shortened, and the user experience is improved.

Fig. 1 schematically illustrates a system architecture 100 of a data visualization processing method and system according to an embodiment of the present disclosure. It should be noted that fig. 1 is only an example of a system architecture to which embodiments of the present disclosure may be applied to assist those skilled in the art in understanding the technical content of the present disclosure, but does not mean that embodiments of the present disclosure may not be used in other devices, systems, environments, or scenarios.

As shown in fig. 1, a system architecture 100 according to this embodiment may include a client 101, a network 102, and a server 103. Network 102 is used to provide a communication link between client 101 and server 103.

The client 101 may be an electronic device such as a desktop computer, a mobile phone, a tablet computer, etc. that includes a display function and an input function, and a user may send a data request through the client.

Network 102 may include various connection types such as wired, wireless communication links, or fiber optic cables, among others. The wired mode can be, for example, connection by adopting any one of the following interfaces: the wireless mode may be, for example, a wireless mode connection, where the wireless mode may be, for example, any one of a plurality of wireless technology standards such as bluetooth, wi-Fi, infrared, zigBee, etc.

The server 103 may be a background server capable of performing data visualization processing, where the server 103 obtains a data request sent by the client 101 through the network 102, and after performing data segmentation on the data request, generates N visualization sub-jobs, and distributes the multiple visualization sub-jobs to different job executors for processing.

It should be noted that the data visualization processing method provided by the embodiment of the present disclosure may be executed by the server 103. Accordingly, the data visualization processing system provided by the embodiments of the present disclosure may be disposed in the server 103. Alternatively, the data visualization processing method provided by the embodiment of the present disclosure may also be performed by a server or a server cluster that is different from the server 103 and is capable of communicating with the client 101 and/or the server 103. Accordingly, the data visualization processing system provided by the embodiments of the present disclosure may also be disposed in a server or server cluster that is different from the server 103 and is capable of communicating with the client 101 and/or the server 103. Alternatively, the data visualization processing method provided by the embodiment of the present disclosure may be partially executed by the server 103 and partially executed by the client 101. Accordingly, the data visualization processing system provided in the embodiments of the present disclosure may be partially disposed in the server 103 and partially disposed in the client 101.

It should be understood that the number of image acquisitions, networks, and servers in fig. 1 are merely illustrative. There may be any number of image acquisitions, networks, and servers, as desired for implementation.

It should be understood that the data visualization processing method provided by the embodiments of the present disclosure is not limited to application in the field of financial technology, but may be applied to any field other than the financial field. The above description is merely exemplary, and the data visualization processing method according to the embodiments of the present disclosure may be applied to any field having data visualization processing, such as electronic commerce, logistics, and other technical fields.

The virtual object rendering method according to the embodiment of the present disclosure will be described in detail below with reference to fig. 2 to 5, describing a scene of data visualization processing based on fig. 1. Fig. 2 schematically illustrates a flow chart of a data visualization processing method according to an embodiment of the present disclosure.

As shown in fig. 2, the data visualization processing method may include operations S1 to S4.

In operation S1, a data request is received.

In some embodiments, the data request is report data uploaded by the user. Report data includes a plurality of lines of data, which may be, for example, several lines, several hundred lines, several thousand lines, several ten thousand lines, several hundred thousand lines, millions of lines, or ten million lines. The embodiment of the disclosure does not limit the specific line number of the report data.

In operation S2, data segmentation is performed on the data request according to a preset rule, N visualized sub-jobs are generated, and N is an integer greater than 1.

In some embodiments, the data request is report data, the report data includes a plurality of rows of data, and the preset rule may be: and carrying out data segmentation on the report data based on the preset line number to generate a visualized sub-job with smaller granularity.

For example, the report data includes m rows of data, where n is less than m, and the preset number of rows is n. The data splitting of the data request according to the preset rule may be: and selecting n rows from the m rows of data as 1 visualization sub-operation, selecting n rows from the rest m-n rows of data as 1 visualization sub-operation, and repeatedly executing the steps until the m rows of data are selected. If the number of data lines remaining in the request data is less than N after the N-1 visualized sub-jobs have been selected, the last remaining number of data lines is still segmented into 1 visualized sub-job.

In some embodiments, performing data segmentation on the data request according to the preset rule includes: and uniformly dividing the data request into N visualized sub-jobs with the same granularity according to the granularity of the preset batch.

The batch size refers to the data size of each visualization sub-job. In the embodiment of the present disclosure, the batch size refers to a preset number of lines set based on a preset rule. The predetermined batch size may be determined based on the configuration.

The data request is uniformly segmented into N visualized sub-jobs with the same granularity, namely, for a data report with m rows of data, the preset batch size is N rows, m=nn, and thus, the number of rows of each visualized sub-job after the data request is segmented based on the preset batch size is N.

The granularity of each visualized sub-job is set to be the same, so that the load of each job executor is the same, and the load balancing is facilitated to be improved. And according to the preset batch size as granularity, the data request is uniformly segmented into N visual sub-jobs with the same granularity, so that the data segmentation operation is easier, and when the N visual sub-jobs are distributed, the size difference of the data volume among different visual sub-jobs is not required to be considered, and further the distribution operation of the visual sub-jobs can be simplified.

In operation S3, the N visualized sub-jobs are correspondingly distributed to N job executors, and each job executor performs data processing on the corresponding visualized sub-job to generate N processing data.

Because the visualized sub-jobs are obtained by data segmentation of the data requests, the task granularity of each visualized sub-job is much smaller than that of the data requests. The smaller load of the job executor to process the visualized sub-job may not significantly reduce the processing time of the data request compared to the job executor to process the data request.

And each job executor carries out data processing on the corresponding visualized sub-job, so that the workload of each node can be balanced, and the load balance is realized.

In some embodiments, a poll distribution scheme is employed to distribute the N visualization sub-jobs to the N job executors.

It will be appreciated that, in the case where the requested data is report data, the complexity of the data visualization process for the requested data is related to the data volume of the report data. After the data request is split, the data volume of the N visualization sub-jobs is the same or nearly the same. Thus, the computational resources consumed to process the N visualization sub-jobs are substantially uniform. That is, the load required to process any of the N visualization sub-jobs is the same or nearly the same for the same job executor. Thus, N visualized sub-job correspondence may be distributed to N job executors in a round robin fashion.

Specifically, the polling mode may be: one of the N visualized sub-jobs is selected, whether the N job executors give the visualized sub-job data processing service or not is sequentially inquired, and if one of the N visualized sub-jobs is answered, the inquiry of the visualized sub-job is ended. And then selecting 1 visual sub-job from the remaining N-1 visual sub-jobs, and sequentially inquiring whether the remaining N-1 job executors give the visual sub-job data processing service or not until the job executors answer. Repeating the steps for the remaining visualized sub-jobs until each visualized sub-job is distributed to the corresponding job executor for processing.

In some embodiments, before the step of distributing the N visualization sub-jobs to the N job executors, further comprising: and executing parameter initialization processing on each job executor based on the preset parameters so as to make the parameters of the visual coordinate system of each job executor consistent.

Since data visualization requires mapping a data line in a data request to a certain pixel point or block of pixels in the visualization coordinate system. Therefore, the visualized coordinate system adopted by each job executor in data processing needs to be subjected to parameter initialization processing, so that each job executor can process the visualized sub-job based on the same visualized coordinate system, each processing data is combined in the same visualized coordinate system in the subsequent combination processing, and the image is legally output.

In some embodiments, performing the parameter initialization process by each job executor to reconcile parameters of the visual coordinate system of each job executor may include:

setting coordinate axis unit parameters of a visual coordinate system to be consistent; setting the coordinate axis of the visual coordinate system to be consistent in upper and lower intervals; setting the consistent color theme of the visual coordinate system; and setting the visualization canvas of the visualization coordinate system to be consistent in size.

In some embodiments, the visualization coordinate system may be a geographic coordinate system, and latitude and longitude coordinate points of the geographic coordinate system may be used for visualization. In some embodiments, the visualization coordinate system may also be a spatial rectangular coordinate system, in which 3D coordinate points may be used for visualization.

Fig. 3 schematically illustrates a flow chart in a data visualization processing method according to another embodiment of the present disclosure.

As shown in fig. 3, in some embodiments, the data visualization processing method further includes operations S10 to S13:

in operation S10, a sub-job list storing visualized sub-jobs corresponding to processes of different job executors is generated.

In operation S11, each job executor completes data processing, and outputs a completion identification matching the visualized sub-job of the corresponding processing.

It should be noted that, when the job executor completes data processing, the job executor successfully performs data processing on the visualized sub-job, and outputs a completion identifier matched with the corresponding visualized sub-job. If the operation executor fails to process the visualized sub-operation data, the completion identification is not output. Different visualization sub-jobs match different completion identifications.

In operation S12, after the N job executors complete the data processing, each of the output completion identifications is matched with each of the visualized sub-jobs stored in the sub-job list.

It will be appreciated that because the completion identification output by the job executor matches the visualized sub-job, and the visualized sub-job executed by each job executor is stored in the sub-job list, the completion identification may be matched with the visualized sub-job stored in the sub-job list.

In one particular example, matching each completion identification of the output with each visualized sub-job stored in the sub-job list may include:

and matching each completion identifier with each visualized sub-job stored in the sub-job list in turn, and marking the visualized sub-job as successful if the visualized sub-job is matched with the sub-job in the sub-job list. Repeating the steps until all the completion identifications are matched with the visualized sub-jobs in the sub-job list.

And checking the visualized sub-job which is not marked as successful in matching in the sub-job list, and considering that the visualized sub-job fails in matching.

In operation S13, if there is a visualized sub-job whose matching fails in the sub-job list, the job executor performs data processing again on the visualized sub-job.

In some embodiments, after the data processing is performed on the failed visualized sub-job again, if the completion identifier is output, the completion identifier is matched with the visualized sub-job in the sub-job list again, and if the matching is failed, the job executor performs the data processing on the visualized sub-job again, so that the device matching is successful.

Fig. 4 schematically illustrates a flow chart of a data visualization processing method according to a further embodiment of the present disclosure.

As shown in fig. 4, in some embodiments, the data visualization processing method further includes operations S20 to S21:

in operation S20, it is monitored whether an abnormality occurs in the job executor. For example, when a job executor exits for various reasons, the job executor is considered to be abnormal.

In operation S21, if an abnormality occurs in the job executor, the remaining visualized sub-jobs that are not processed by the job executor are acquired, and the remaining visualized sub-jobs are distributed to at least one of the remaining job executors for data processing.

In one particular example, the remaining visualization sub-jobs may be distributed to any of the remaining normal job executors for processing.

In another specific example, the data of the remaining visualized sub-jobs may be segmented to generate N-1 sub-visualized sub-jobs with smaller granularity, and the N-1 sub-visualized sub-jobs may be distributed to the remaining N-1 job executors for data processing.

In some embodiments, the number of actuators may also be monitored prior to performing operation S3. If the number of the executors is less than N, creating new job executors until the number of the job executors is N.

In operation S4, the N pieces of processing data are visually combined and processed, and visual data are output.

Because the data request is segmented in advance, the visualized sub-jobs with smaller granularity are generated, the N visualized sub-jobs are simultaneously processed by adopting N job executors, the load of each job executor is reduced, the processing time is shortened, the N processing data are subjected to visual combination and processing in the follow-up process, the time for carrying out the visualized processing on the data request can be greatly shortened, and the user experience is improved.

In some embodiments, the visual data may include non-aggregated charts as well as aggregated charts. Wherein the non-aggregated graph may be, for example, a scatter plot, a thermodynamic diagram, etc., each line of the data request will map to an element in the visual coordinate system. Aggregation charts require aggregation analysis of data requests, such as histogram statistics, pie chart statistics, etc.

Fig. 5 schematically illustrates a flow chart of a data visualization processing method according to a further embodiment of the present disclosure.

As shown in fig. 5, in some embodiments, since the non-aggregated graph does not need to aggregate data, the aggregated graph needs to aggregate data, and thus the data visualization processing method is different for the case where the visualized data is the non-aggregated graph and the aggregated graph. Based on this, operation S2 is performed: the method further comprises the steps of after data segmentation is carried out on the data request and N visualized sub-jobs are generated: operations S30 to S31.

In operation S30, N visualization sub-jobs need to be traversed.

In operation S31, it is determined whether or not the visualized sub-job needs to be aggregated, and if so, the visualized sub-job is an aggregated graph, and if not, the visualized sub-job is a non-aggregated graph.

In some embodiments, where the visual data is a non-aggregated chart, operation S3 may be:

each job executor performs visualization processing on the corresponding visualization sub-job to generate N pieces of processing data.

In a specific example, each job executor performs visualization processing on a corresponding visualization sub-job to generate N pieces of processing data includes: and each job executor performs visualization processing and primary rendering on the corresponding visualized sub-job based on the visualized coordinate system with the same parameters, and generates a relative coordinate position of the visualized sub-job in the visualized coordinate system and a corresponding primary rendering result value.

Specifically, drawing calculation is performed on each visualized sub-job in a unified visualized coordinate system space after parameter initialization, and data corresponding to each visualized sub-job is converted into a data mark set in the visualized coordinate space, wherein the data mark set comprises a relative coordinate position and an initial rendering result value.

For example, if the visual coordinate system is a space rectangular coordinate system, the visual sub-jobs can be visualized according to the rendering subspaces with fixed length, width and height, and each visual sub-job is converted into a relative coordinate position and a corresponding rendering result value in the rendering subspace. The rendering result value may be an RGB value.

Operation S4 may be: and merging and rendering the N pieces of processing data, and outputting visual data.

In one specific example, the visual compounding and processing includes: the relative coordinate position in each processing data is converted into an absolute coordinate position, and final rendering is performed based on the primary rendering result value to generate visualized data.

That is, in operation S4, the data marking set is summarized, the data marking result is generated, and the data marking result is rendered to output the visualized data

Specifically, the relative coordinate position in each processing data may be converted into an absolute coordinate position in the same visual coordinate system, and each processing data may be merged under the same visual coordinate system. The primary rendering result value of each processing data is unchanged, and the final rendering is formed and visualized data is output.

In some embodiments, the visual data is an aggregated chart, and operation S3 may be: each job executor performs data first aggregation processing on the corresponding visualized sub-job to generate N pieces of processing data.

Specifically, the first polymerization process may be: and summarizing and counting the data corresponding to each visualized sub-job to obtain N pieces of processed data after summarizing and counting. The processing data may be aggregate statistics of individual visualization sub-jobs.

Operation S4 may be: performing second aggregation on the N pieces of processing data to generate aggregated data; and performing visualization processing and rendering processing on the aggregated data, and outputting the visualized data.

The second aggregation process may be to aggregate and combine each of the above-described process data in the same visual coordinate space. Namely, all the aggregation statistical results are summarized to obtain the aggregation statistical results. And after merging, carrying out visualization processing and rendering on the aggregation statistical result, and outputting visualized data.

It can be understood that, for the visual data being a non-aggregation chart, since the visualization processing is performed on each visualization sub-job in operation S3, the load of each job executor is reduced, so that the time consumption of the visualization processing can be reduced, and finally, only the processing data after the visualization is combined in operation S4, so that the time consumption is greatly shortened, and the user experience is improved.

For the visualization data being an aggregation chart, in operation S3, aggregation statistics is first performed for each visualization sub-job, and then in operation S4, the processing data after the aggregation statistics is summarized and visualization processing is performed. Because the aggregation statistics is performed on the data, the data size of the visualization in the operation S4 is far smaller than the data size in the original data request, so that the time consumption of the visualization processing can be reduced.

In the data visualization processing method provided by the embodiment, after the data request is subjected to data segmentation, N visualized sub-jobs with smaller task granularity are generated, and the visualized sub-jobs are distributed to different job executors for processing. In this way, the load of the job executor for processing the visualized sub-job can be reduced, and the processing time can be greatly shortened. Each job executor performs data processing on the corresponding visualized sub-job, so that the workload of each node can be balanced, and load balancing is realized. Finally, the N processing data are visually combined and processed, so that the time consumption of the visual processing of the data request can be greatly shortened, and the user experience is improved.

According to another aspect of the present disclosure, there is provided a data visualization processing system that can be applied to the data visualization processing method provided by the above embodiment.

FIG. 6 schematically illustrates a functional block diagram of a data visualization processing system, in accordance with an embodiment of the present disclosure; fig. 7 schematically illustrates a schematic diagram of a data slicing principle of a data visualization processing system according to an embodiment of the present disclosure.

Referring to fig. 6 and 7, the data visualization processing system includes: the data splitting module 101 is configured to receive the data request 1, and perform data splitting on the data request 1 according to a preset rule, so as to generate N visualized sub-jobs 2, where N is an integer greater than 1.

In some embodiments, data request 1 is report data uploaded by a user. Report data includes a plurality of lines of data, which may be, for example, several lines, several hundred lines, several thousand lines, several ten thousand lines, several hundred thousand lines, millions of lines, or ten million lines.

In some embodiments, the data request 1 is report data, where the report data includes a plurality of rows of data, the preset rule may be: and carrying out data segmentation on the report data based on the preset line number to generate a visualized sub-job 2 with smaller granularity.

In some embodiments, the data splitting module 101 splits the data request 1 into N visualized sub-jobs 2 with the same granularity according to the preset batch size as granularity.

The data visualization processing system may further include: the job coordination module 102 is configured to correspondingly distribute the N visualized sub-jobs 2 to N job executors 103, where each job executor 103 performs data processing on the corresponding visualized sub-job 2 to generate N pieces of processing data.

In some embodiments, job coordination module 102 may distribute N visualization sub-jobs 2 to N job executors 103 using a poll distribution manner.

In some embodiments, the job coordination module is further to: a parameter initialization process is performed on each job executor 103 based on a preset parameter so that the parameters of the visual coordinate system of each job executor 103 are uniform.

In some embodiments, performing the parameter initialization process by each job executor 103 to reconcile parameters of the visual coordinate system of each job executor 103 may include:

In some embodiments, the job executor distributes N visualized sub-jobs 2 to N job executors 103 based on a sub-job list storing visualized sub-jobs 2 processed by different job executors 103; the job executor 103 completes data processing and outputs a completion identifier matched with the corresponding processed visualized sub-job 2; the job coordination module 102 is also configured to:

after the N job executors 103 complete the data processing, each completion flag output is matched with each visualized sub job 2 stored in the sub job list. If there is a visualized sub-job 2 whose matching fails in the sub-job list, the job executor 103 performs data processing again on the visualized sub-job 2.

In a specific example, the job coordination module 102 matches each completion identifier with each visualized sub-job 2 stored in the sub-job list in turn, and if there is a match between the visualized sub-job 2 in the sub-job list, the visualized sub-job 2 is marked as successful. Repeating the steps until all the completion identifications are matched with the visualized sub-job 2 in the sub-job list.

In some embodiments, job coordination module 102 is further to: whether the job executor 103 is abnormal or not is detected, if the job executor 103 is abnormal, the residual visualized sub-job 2 which is not processed by the job executor 103 is obtained, and the residual visualized sub-job 2 is distributed to at least one of the residual job executors 103 for data processing.

In one specific example, the remaining visualization sub-job 2 may be distributed to any of the remaining normal job executors 103 for processing.

In another specific example, the remaining visualized sub-jobs 2 may be subjected to data segmentation to generate N-1 visualized sub-jobs 2 with smaller granularity, and the N-1 visualized sub-jobs 2 may be distributed to the remaining N-1 job executors 103 for data processing.

In some embodiments, job coordination module 102 may also monitor the number of actuators. If the number of actuators is less than N, a new job actuator 103 is created until the number of job actuators 103 is N.

The data visualization processing system may further include: and the merging module 104 is used for carrying out visual combination and processing on the N processing data and outputting visual data.

Fig. 8 schematically illustrates a schematic diagram of a data visualization processing system according to an embodiment of the present disclosure.

Referring to FIG. 8, in some embodiments, the visualization data is a non-aggregated graph, and the job executor 103 is configured to: performing visualization processing on the visualization sub-job 2 to generate N pieces of processing data; the merge module 104 is configured to: and merging and rendering the N pieces of processing data, and outputting visual data.

Specifically, the job executor 103 performs visualization processing and primary rendering on the corresponding visualized sub-job 2 based on the visualized coordinate system of the same parameters, and generates a data marker set 21 in the visualized coordinate space, where the data marker set 21 may include the relative coordinate position of the visualized sub-job 2 in the visualized coordinate system and the corresponding primary rendering result value.

The merge module 104 sums the data-markup sets 21, generates the data-markup results 22, and renders the data-markup results 22 to output the visual data 3. Specifically, the merging module 104 may convert the relative coordinate position in each processing data into an absolute coordinate position, and perform final rendering based on the primary rendering result value to generate the visualized data 3.

Fig. 9 schematically illustrates a schematic diagram of a data visualization processing system according to another embodiment of the present disclosure.

Referring to FIG. 9, in some embodiments, the visualization data is an aggregated graph, and the job executor 103 is configured to: performing data first aggregation processing on the visualized sub-job 2 to generate N pieces of processing data; the merge module 104 is configured to: performing second aggregation on the N pieces of processing data to generate aggregated data; the aggregated data is visualized and rendered, and visualized data 3 is output.

The first polymerization process may be: and summarizing and counting the data corresponding to each visualized sub-job 2 to obtain N pieces of processed data after summarizing and counting. The processing data may be segmented aggregate statistics 23 for each visualization sub-job 2.

The second aggregation process may be to aggregate and combine each of the above-described process data in the same visual coordinate space. I.e. the individual segment aggregate statistics 23 are summarized to obtain a summarized aggregate statistics 24. After merging, the summary aggregation statistics 24 are visualized and rendered, and visual data 3 is output.

In the data visualization processing system provided in the foregoing embodiment, after the data segmentation module 101 performs data segmentation on the data request 1, N visualized sub-jobs 2 with smaller task granularity are generated. The job coordination module 102 distributes the plurality of visualization sub-jobs 2 to different job executors 103 for processing. In this way, the load of the job executor 103 for processing the visualized sub-job 2 can be reduced, and the processing time can be greatly shortened. Each job executor 103 performs data processing on the corresponding visualized sub-job 2, so that the workload of each node can be balanced, and load balancing is realized. The merging module 104 performs visual combination and processing on the N processing data, so that the time consumption of performing visual processing on the data request 1 can be greatly shortened, and the user experience is improved.

According to another aspect of the present disclosure, there is also provided an electronic device including: one or more processors; and a memory for storing one or more programs, wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform a data visualization processing method provided by implementing the above embodiments.

As shown in fig. 10, the electronic device according to the embodiment of the present disclosure includes a processor 201 that can perform various appropriate actions and processes according to a program stored in a Read Only Memory (ROM) 202 or a program loaded from a storage section 208 into a Random Access Memory (RAM) 203. The processor 201 may include, for example, a general purpose microprocessor (e.g., a CPU), an instruction set processor and/or an associated chipset and/or a special purpose microprocessor (e.g., an Application Specific Integrated Circuit (ASIC)), or the like. Processor 201 may also include on-board memory for caching purposes. The processor 201 may include a single processing unit or multiple processing units for performing different actions of the method flows according to embodiments of the disclosure.

In the RAM203, various programs and data required for the operation of the electronic device are stored. The processor 201, ROM202, and RAM203 are connected to each other via a bus 204. The processor 201 performs various operations of the method flow according to the embodiments of the present disclosure by executing programs in the ROM202 and/or the RAM 203. Note that the program may be stored in one or more memories other than the ROM202 and the RAM 203. The processor 201 may also perform various operations of the method flow according to embodiments of the present disclosure by executing programs stored in one or more memories.

According to embodiments of the present disclosure, the electronic device may further include an input/output (I/O) interface 205, the input/output (I/O) interface 205 also being connected to the bus 204. The electronic device may also include one or more of the following components connected to the I/O interface 205: an input section 206 including a keyboard, a mouse, and the like; an output portion 207 including a Cathode Ray Tube (CRT), a Liquid Crystal Display (LCD), and the like, and a speaker, and the like; a storage section 208 including a hard disk or the like; and a communication section 209 including a network interface card such as a LAN card, a modem, and the like. The communication section 209 performs communication processing via a network such as the internet. The drive 210 is also connected to the I/O interface 205 as needed. A removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, a semiconductor memory, or the like is installed on the drive 210 as needed, so that a computer program read out therefrom is installed into the storage section 208 as needed.

The present disclosure also provides a computer-readable storage medium that may be embodied in the apparatus/device/system described in the above embodiments; or may exist alone without being assembled into the apparatus/device/system. The computer-readable storage medium carries one or more programs that, when executed, implement a data visualization processing method according to an embodiment of the present disclosure.

According to embodiments of the present disclosure, the computer-readable storage medium may be a non-volatile computer-readable storage medium, which may include, for example, but is not limited to: a portable computer diskette, a hard disk, a Random Access Memory (RAM), a read-only memory (ROM), an erasable programmable read-only memory (EPROM or flash memory), a portable compact disc read-only memory (CD-ROM), an optical storage device, a magnetic storage device, or any suitable combination of the foregoing. In the context of this disclosure, a computer-readable storage medium may be any tangible medium that can contain, or store a program for use by or in connection with an instruction execution system, apparatus, or device. For example, according to embodiments of the present disclosure, a computer-readable storage medium may include ROM and/or RAM and/or one or more memories other than ROM and RAM as described above.

Embodiments of the present disclosure also include a computer program product comprising a computer program containing program code for performing the methods shown in the flowcharts. The program code means for causing a computer system to carry out the data visualization processing methods provided by the embodiments of the present disclosure when the computer program product is run on the computer system.

The above-described functions defined in the system/apparatus of the embodiments of the present disclosure are performed when the computer program is executed by a processor. The systems, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

In one embodiment, the computer program may be based on a tangible storage medium such as an optical storage device, a magnetic storage device, or the like. In another embodiment, the computer program may also be transmitted, distributed over a network medium in the form of signals, downloaded and installed via a communication section, and/or installed from a removable medium. The computer program may include program code that may be transmitted using any appropriate network medium, including but not limited to: wireless, wired, etc., or any suitable combination of the foregoing.

In such embodiments, the computer program may be downloaded and installed from a network via a communication portion, and/or installed from a removable medium. The above-described functions defined in the system of the embodiments of the present disclosure are performed when the computer program is executed by a processor. The systems, devices, apparatus, modules, units, etc. described above may be implemented by computer program modules according to embodiments of the disclosure.

According to embodiments of the present disclosure, program code for performing computer programs provided by embodiments of the present disclosure may be written in any combination of one or more programming languages, and in particular, such computer programs may be implemented in high-level procedural and/or object-oriented programming languages, and/or assembly/machine languages. Programming languages include, but are not limited to, such as Java, c++, python, "C" or similar programming languages. The program code may execute entirely on the user's computing device, partly on the user's device, partly on a remote computing device, or entirely on the remote computing device or server. In the case of remote computing devices, the remote computing device may be connected to the user computing device through any kind of network, including a Local Area Network (LAN) or a Wide Area Network (WAN), or may be connected to an external computing device (e.g., connected via the Internet using an Internet service provider).

The flowcharts and block diagrams in the figures illustrate the architecture, functionality, and operation of possible implementations of systems, methods and computer program products according to various embodiments of the present disclosure. In this regard, each block in the flowchart or block diagrams may represent a module, segment, or portion of code, which comprises one or more executable instructions for implementing the specified logical function(s). It should also be noted that, in some alternative implementations, the functions noted in the block may occur out of the order noted in the figures. For example, two blocks shown in succession may, in fact, be executed substantially concurrently, or the blocks may sometimes be executed in the reverse order, depending upon the functionality involved. It will also be noted that each block of the block diagrams or flowchart illustration, and combinations of blocks in the block diagrams or flowchart illustration, can be implemented by special purpose hardware-based systems which perform the specified functions or acts, or combinations of special purpose hardware and computer instructions.

Those skilled in the art will appreciate that the features recited in the various embodiments of the disclosure and/or in the claims may be combined in various combinations and/or combinations, even if such combinations or combinations are not explicitly recited in the disclosure. In particular, the features recited in the various embodiments of the present disclosure and/or the claims may be variously combined and/or combined without departing from the spirit and teachings of the present disclosure. All such combinations and/or combinations fall within the scope of the present disclosure.

The embodiments of the present disclosure are described above. However, these examples are for illustrative purposes only and are not intended to limit the scope of the present disclosure. Although the embodiments are described above separately, this does not mean that the measures in the embodiments cannot be used advantageously in combination. The scope of the disclosure is defined by the appended claims and equivalents thereof. Various alternatives and modifications can be made by those skilled in the art without departing from the scope of the disclosure, and such alternatives and modifications are intended to fall within the scope of the disclosure.

Claims

1. A data visualization processing method, comprising:

receiving a data request;

performing data segmentation on the data request according to a preset rule to generate N visualized sub-jobs, wherein N is an integer greater than 1;

correspondingly distributing the N visualized sub-jobs to N job executors, wherein each job executor carries out data processing on the corresponding visualized sub-job to generate N processing data;

and carrying out visual combination and processing on the N processing data, and outputting visual data.

2. The data visualization processing method according to claim 1, wherein the performing data segmentation on the data request according to a preset rule includes:

And uniformly dividing the data request into N visualized sub-jobs with the same granularity according to the granularity of the preset batch.

3. The data visualization processing method according to claim 1, wherein before the step of distributing the N visualization sub-jobs to N job executors, further comprising:

and executing parameter initialization processing on each job executor based on preset parameters so as to make the parameters of the visual coordinate system of each job executor consistent.

4. The data visualization processing method according to claim 1, further comprising:

generating a sub-job list, wherein the sub-job list stores visualized sub-jobs which are processed correspondingly by different job executors;

each job executor completes data processing and outputs a completion identification matched with the corresponding processed visualized sub-job;

after finishing data processing by the N job executors, matching each output finishing identifier with each visualized sub-job stored in the sub-job list;

and if the visualized sub-job with the failed matching exists in the sub-job list, the job executor carries out data processing on the visualized sub-job again.

5. The data visualization processing method according to claim 4, further comprising:

monitoring whether the operation executor is abnormal;

if the job executor is abnormal, acquiring residual visualized sub-jobs which are not processed by the job executor, and distributing the residual visualized sub-jobs to at least one of the residual job executors for data processing.

6. The data visualization processing method according to claim 1, wherein the N visualized sub-jobs are distributed to N job executors by a polling distribution method.

7. The data visualization processing method according to any one of claims 1 to 6, wherein the visualized data is a non-aggregated chart, the method comprising:

each job executor performs visualization processing on the corresponding visualization sub-job to generate the N pieces of processing data;

the visual compound and treatment comprises:

and merging and rendering the N pieces of processing data, and outputting the visualized data.

8. The data visualization processing method according to claim 7, wherein the each of the job executors performs visualization processing on the corresponding visualized sub-job to generate the N pieces of processing data includes:

Each job executor performs visualization processing and primary rendering on the corresponding visualized sub-job based on a visualized coordinate system with the same parameters, and generates a relative coordinate position of the visualized sub-job in the visualized coordinate system and a corresponding primary rendering result value;

the visual compound and treatment comprises:

and converting the relative coordinate position in each processing data into an absolute coordinate position, and finally rendering based on the primary rendering result value to generate the visualized data.

9. The data visualization processing method according to any one of claims 1 to 6, wherein the visualized data is an aggregated graph, the method comprising:

each job executor performs data first aggregation processing on the corresponding visualized sub-job to generate the N pieces of processing data;

the visual compound and treatment comprises:

performing second aggregation processing on the N pieces of processing data to generate aggregated data;

and performing visualization processing and rendering processing on the aggregated data, and outputting the visualized data.

10. A data visualization processing system, comprising:

the data segmentation module is used for receiving the data request, carrying out data segmentation on the data request according to a preset rule, generating N visualized sub-jobs, wherein N is an integer greater than 1;

The job coordination module is used for correspondingly distributing the N visualized sub-jobs to N job executors, and each job executor carries out data processing on the corresponding visualized sub-job to generate N processing data;

and the merging module is used for carrying out visual combination and processing on the N processing data and outputting visual data.

11. An electronic device, comprising:

one or more processors;

a memory for storing one or more programs,

wherein the one or more programs, when executed by the one or more processors, cause the one or more processors to perform the method of any of claims 1-9.

12. A computer readable storage medium having stored thereon executable instructions which, when executed by a processor, cause the processor to perform the method according to any of claims 1 to 9.

13. A computer program product comprising a computer program which, when executed by a processor, implements the method according to any one of claims 1 to 9.