CN117971952A

CN117971952A - Data processing method, device, computer equipment and storage medium

Info

Publication number: CN117971952A
Application number: CN202311355456.5A
Authority: CN
Inventors: 程祥一
Original assignee: Mashang Consumer Finance Co Ltd
Current assignee: Mashang Consumer Finance Co Ltd
Priority date: 2023-10-18
Filing date: 2023-10-18
Publication date: 2024-05-03

Abstract

The present application relates to a data processing method, apparatus, computer device, storage medium and computer program product. The method comprises the following steps: acquiring a data loading request, and determining a data loading strategy based on the data loading request; loading target data based on the data loading strategy, and visually displaying the target data; in the process of visual display, responding to grouping operation, determining a data loading strategy, obtaining grouping data after grouping operation based on the determined data loading strategy, and performing visual display on the grouping data; in the visual display process, a data loading strategy is determined in response to the aggregation operation, an aggregate data processing result after the aggregation operation is obtained based on the determined data loading strategy, and the aggregate data processing result is visually displayed. The method can realize the visualization of data exploration processing and improve the efficiency of data exploration.

Description

Data processing method, device, computer equipment and storage medium

Technical Field

The present application relates to the field of computer technology, and in particular, to a data processing method, apparatus, computer device, storage medium, and computer program product.

Background

With the increasing data volume, how to quickly and accurately find the rules in the data is a long-standing hot spot problem in the data analysis field. Data exploration (Data Exploration) is a common data analysis method that can exploring valuable information hidden in data. However, in the current data searching method, when a large data set is processed, the same strategy is adopted for searching and processing, so that the problems of low efficiency caused by slow and unstable processing are easily caused.

Disclosure of Invention

In view of the foregoing, it is desirable to provide a data processing method, apparatus, computer device, computer-readable storage medium, and computer program product that can improve analysis processing efficiency.

In a first aspect, the present application provides a data processing method. The method comprises the following steps:

Acquiring a data loading request, and determining a data loading strategy based on the data loading request;

loading target data based on the data loading strategy, and visually displaying the target data;

In the process of visual display, responding to grouping operation, determining a data loading strategy, obtaining grouping data after grouping operation based on the determined data loading strategy, and performing visual display on the grouping data;

In the visual display process, a data loading strategy is determined in response to the aggregation operation, an aggregate data processing result after the aggregation operation is obtained based on the determined data loading strategy, and the aggregate data processing result is visually displayed.

In a second aspect, the application further provides a data processing method. The method comprises the following steps:

Acquiring a data loading request and determining a data loading strategy corresponding to the data loading request;

acquiring target data corresponding to the data loading request, and transmitting the target data to a sender of the data loading request based on the data loading strategy so as to visually display the target data on the sender;

Acquiring a packet request of the sender, determining a data loading strategy corresponding to the packet request, screening and acquiring corresponding packet data based on the packet request, and sending the packet data to the sender based on the determined data loading strategy so as to visually display the packet data on the sender;

Acquiring an aggregation request of the sender, determining a data loading strategy corresponding to the aggregation request, performing aggregation operation on target aggregation data based on the aggregation request, acquiring an aggregation data processing result after the aggregation operation, and sending the aggregation data processing result to the sender based on the determined data loading strategy so as to visually display the aggregation data processing result on the sender.

In a third aspect, the present application further provides a data processing apparatus. The device comprises:

The first loading strategy determining module is used for acquiring a data loading request and determining a data loading strategy corresponding to the data loading request;

The first data loading module is used for loading target data based on the data loading strategy and carrying out visual display on the target data;

The first packet processing module is used for responding to the packet operation in the process of carrying out the visual chart display on the target data, determining a data loading strategy, obtaining packet data after the packet operation based on the determined data loading strategy, and carrying out the visual display on the packet data;

the first aggregation processing module is used for responding to the aggregation operation in the visual display process, determining a data loading strategy, obtaining an aggregate data processing result after the aggregation operation based on the determined data loading strategy, and visually displaying the aggregate data processing result.

In a fourth aspect, the present application further provides a data processing apparatus. The device comprises:

the second loading strategy determining module is used for acquiring a data loading request and determining a data loading strategy corresponding to the data loading request;

The second data loading module is used for acquiring target data corresponding to the data loading request, and sending the target data to a sender of the data loading request based on the data loading strategy so as to visually display the target data on the sender;

The second packet processing module is used for acquiring a packet request of the sender, determining a data loading strategy corresponding to the packet request, screening and acquiring corresponding packet data based on the packet request, and sending the packet data to the sender based on the determined data loading strategy so as to visually display the packet data on the sender;

The second aggregation processing module is used for acquiring the aggregation request of the sender, determining a data loading strategy corresponding to the aggregation request, carrying out aggregation operation on target aggregation data based on the aggregation request, obtaining an aggregation data processing result after the aggregation operation, and sending the aggregation data processing result to the sender based on the determined data loading strategy so as to carry out visual display on the aggregation data processing result at the sender.

In a fifth aspect, the present application also provides a computer device. The computer device comprises a memory storing a computer program and a processor implementing the steps of the method in any of the embodiments described above when the computer program is executed.

In a sixth aspect, the present application also provides a computer readable storage medium. The computer readable storage medium having stored thereon a computer program which, when executed by a processor, implements the steps of the method in any of the embodiments described above.

In a seventh aspect, the present application also provides a computer program product. The computer program product comprising a computer program which, when executed by a processor, implements the steps of the method in any of the embodiments described above.

The data processing method, the device, the computer equipment, the storage medium and the computer program product firstly determine the data loading strategy corresponding to the data loading request after the data loading request is obtained, then load the target data corresponding to the data loading request based on the data loading strategy, and visually display the loaded target data, so that the determined data loading strategy corresponds to the data loading request, and accordingly, the adaptive data loading strategy can carry out data loading.

Drawings

FIG. 1 is a diagram of an application environment for a data processing method in one embodiment;

FIG. 2 is a flow diagram of a data processing method in one embodiment;

FIG. 3 is a flow diagram of a process for obtaining configured data quality management rules in one embodiment;

FIG. 4 is a schematic diagram of an interface for custom configuration of data quality management rules in one embodiment;

FIG. 5 is a flow diagram of determining a data loading policy corresponding to the data loading request in one embodiment;

FIG. 6 is a flow diagram of packet processing in one embodiment;

FIG. 7 is a flow diagram of an aggregation process in one embodiment;

FIG. 8 is a flow chart of a data processing method in another embodiment;

FIG. 9 is a flow chart of a data processing method in another embodiment;

FIG. 10 is a schematic diagram of a data processing method and schematic architecture in one specific example;

FIG. 11 is a block diagram of a data processing apparatus in one embodiment;

FIG. 12 is a block diagram of a data processing apparatus in one embodiment;

FIG. 13 is an internal block diagram of a computer device in one embodiment;

Fig. 14 is an internal structural view of a computer device in another embodiment.

Detailed Description

The present application will be described in further detail with reference to the drawings and examples, in order to make the objects, technical solutions and advantages of the present application more apparent. It should be understood that the specific embodiments described herein are for purposes of illustration only and are not intended to limit the scope of the application.

With the increasing data volume, how to quickly and accurately find the rules in the data is a long-standing hot spot problem in the data analysis field. Data exploration (Data Exploration) is a common data analysis method that can exploring valuable information hidden in data.

However, in the existing data processing process of data exploration, the data processing capability is limited, when a large data set is processed, the data processing capability may become slow or unstable, and when the data loading amount is large, page locking is easy to occur, so that the data processing efficiency is not high. On the other hand, the data quality cannot be accurately controlled, the accuracy and the reliability of data exploration are easily reduced, in the process of data exploration, only a single user can conduct the data exploration, team cooperation cannot be conducted, and the efficiency of data exploration during team work is seriously affected.

Based on the method, the method and the device can solve the problem of low processing efficiency by combining a dynamic data loading strategy through a visual data exploration process, solve the problem of low accuracy and reliability of data exploration based on a user-defined data quality management rule, and realize collaborative sharing aiming at the visual data exploration process so as to improve the efficiency of data exploration during team work.

Accordingly, the present application is directed to a data processing method that, in one or more of the above-mentioned ways, solves one or more of the above-mentioned problems.

The data processing method provided by the embodiment of the application can be applied to an application environment shown in figure 1. Wherein the terminal 102 communicates with the server 104 via a network. The data storage system may store data that the server 104 needs to process. The data storage system may be integrated on the server 104 or may be located on a cloud or other network server. The terminal 102 may perform data exploration processing by accessing data of the server 104. For example, the terminal 102 obtains a data loading request and determines a data loading policy corresponding to the data loading request; based on the data loading strategy, loading target data corresponding to the data loading request from the server 104, and visually displaying the loaded target data. In the process of performing visual chart display on the target data, responding to the grouping operation, determining a data loading strategy, acquiring grouping data after the grouping operation from the server 104 based on the determined data loading strategy, and performing visual display on the grouping data; in the process of performing visual display, a data loading strategy is determined in response to the aggregation operation, an aggregate data processing result after the aggregation operation is obtained from the server 104 based on the determined data loading strategy, and the aggregate data processing result is subjected to visual display.

Further, as shown in fig. 1, the application scenario may further involve a plurality of terminals 102, for example, one terminal 102 obtains sharing operation data of the data exploration process, and if a cooperative sharing condition is met, the sharing operation data is sent to the server 104, so that the sharing operation data is sent to another terminal 102 performing cooperative sharing through the server 104, and can be visually displayed at the other terminal 102, thereby implementing cooperative sharing in the data exploration process.

The terminal 102 may be, but not limited to, various personal computers, notebook computers, smart phones, tablet computers, internet of things devices, and portable wearable devices, where the internet of things devices may be smart speakers, smart televisions, smart air conditioners, smart vehicle devices, and the like. The portable wearable device may be a smart watch, smart bracelet, headset, or the like. The server 104 may be implemented as a stand-alone server or as a server cluster of multiple servers.

In some embodiments, as shown in fig. 2, a data processing method is provided, which is illustrated by using the method applied to the terminal 102 in fig. 1 as an example, and at least includes the following steps S202 to S208.

Step S202: and acquiring a data loading request and determining a data loading strategy corresponding to the data loading request.

Wherein the data loading request is a request for indicating loading of data, which the terminal can obtain in various possible ways. For example, in some embodiments, taking the example of running a data exploration related tool (e.g., an application or browser capable of accessing a server) at the terminal 102, it may be considered to obtain a data load request when starting to launch the application. In other embodiments, the data loading request may be obtained based on a processing procedure in the process of data exploration after the terminal starts the relevant tool for data exploration, for example, a data loading request generated based on an operation procedure of a user, and for example, a data loading request generated based on a preset time period, which is not limited in the embodiments of the present application.

When determining the data loading strategy corresponding to the data loading request, the data loading strategy can be determined according to the actual technical requirement, and in some embodiments of the present application, the data loading strategy for the data loading request can be determined after the data loading request is received, so as to implement intelligent data loading.

Step S204: and loading target data based on the data loading strategy, and visually displaying the target data.

The target data refers to data to be obtained based on the data loading request, and may be obtained from a data source, where the data source is a data set that needs to be explored and managed, such as a local file, a local database, or a remote API (Application Programming Interface, an application program interface, also called an application programming interface), and so on. In some embodiments, taking a data source as a local database as an example, the terminal may be provided with the local database, and target data to be loaded may be obtained from the local database directly based on a data loading policy. Taking a data source as a remote API as an example, the terminal can obtain target data from a server through the remote API and load the target data based on a data loading strategy.

In some specific examples, when obtaining target data, the target data may be obtained based on Mithril, mithril is a client javascript framework that enables application code to be divided into a data layer, a UI (user interface) layer, and an adhesive layer, which can be used for high-performance rendering, making the readability and maintainability of the application code stronger. For example, data may be loaded via an AJAX request or a fetch API using the Mithril request module and stored in the data model of the front-end application.

When the obtained target data is visually displayed, the visual display can be performed in combination with the actual technical requirement, and the specific visual display mode is not limited, for example, the visual display can be performed in a chart mode. By taking the display mode as an example, the graph can be drawn and displayed in a D3.js mode, and the D3.js is a JavaScript program library for data visualization by using dynamic graphs, and the visual display of the graphs can be realized by inputting simple data, so that the method is simple and easy to use, and is beneficial to improving the processing efficiency. For example, the acquired data can be bound with the front-end page through a data binding function provided by the Mithril framework, and the data is displayed in a chart form by using D3.js according to a display mode selected by a user.

Step S206: in the process of visual display, a data loading strategy is determined in response to the grouping operation, grouping data after the grouping operation is obtained based on the determined data loading strategy, and the grouping data is subjected to visual display.

In related art processing of data exploration, based on the needs of different data exploration, it may be necessary to group visually presented data to explore, or discover, the regularity of such data. The grouping operation may refer to a screening operation of data, and in the case that the grouping operation is the screening operation, the obtained grouping data after the grouping operation may refer to the screened data obtained after the screening operation, which is not specifically limited in the embodiment of the present application.

When the packet data after the packet operation is obtained, firstly, a data loading strategy corresponding to the packet operation is determined in response to the packet operation, and then the packet data is loaded based on the determined data loading strategy corresponding to the packet operation, so that intelligent dynamic loading of the packet data is realized, and the processing efficiency is improved.

In the process of visually displaying the packet data, a specific visual display manner is not limited, and may be, for example, a chart display.

Step S208, in the process of visual display, responding to the aggregation operation, determining a data loading strategy, obtaining an aggregate data processing result after the aggregation operation based on the determined data loading strategy, and performing visual display on the aggregate data processing result.

In the related art processing of data exploration, based on the requirements of different data exploration, data in visual display may need to be aggregated to explore and find the rules of the data. The data used for the aggregation operation may be data after the grouping operation as described above, or may be other data. The manner of the specific aggregation operation may be set differently based on the actual data exploration requirement, for example, in some embodiments, the aggregation operation may be multi-level grouping, nested combination, time sequence aggregation, and the like, and the embodiments of the present application are not limited specifically.

When an aggregate data processing result is obtained, firstly, a data loading strategy corresponding to the aggregate operation is determined in response to the aggregate operation, and then the aggregate data processing result after the aggregate operation is obtained based on the determined data loading strategy corresponding to the aggregate operation, so that intelligent dynamic loading of the aggregate data processing result is realized, and the processing efficiency is improved.

In the process of visually displaying the aggregate data processing result, a specific visual display mode is not limited, and may be, for example, chart display.

According to the data processing method in the embodiment, after the data loading request is obtained, the data loading strategy corresponding to the data loading request is determined, then the target data corresponding to the data loading request is loaded based on the data loading strategy, and the loaded target data is subjected to visual display, so that the determined data loading strategy corresponds to the data loading request, and accordingly, the data loading can be carried out by the adaptive data loading strategy.

Wherein, in some embodiments, before the acquiring the data loading request, the method further comprises:

and acquiring a data quality management rule, and sending the data quality management rule to the server, so that the server performs data cleaning on an original data source based on the data quality management rule to acquire the data source for returning the target data.

The configured data quality management rule may be a data quality management rule generated based on an operation at the terminal. In some embodiments, the data quality management rule may be configured in advance in the terminal and stored in the terminal, and the stored data quality management rule is acquired in the data processing process and sent to the server, where the data quality management rule stored in the terminal may include a plurality of data quality management rules, and one or more data quality management rules may be selected from the plurality of data quality management rules to be combined and sent to the server.

Therefore, the server can clean the data of the original data source based on the data quality management rule configured at the terminal in a self-defining way by acquiring the data quality management rule configured at the terminal, thereby realizing personalized data quality management.

Through the data quality management rule, the quality management of the data can be realized, and the accuracy, consistency, integrity and reliability of the data in the data exploration and processing process are improved, so that the efficiency and reliability of the data exploration and processing can be improved.

In some embodiments, referring to fig. 3, the above-mentioned data quality management rule of the acquisition configuration includes steps S302 to S306.

Step S302: and acquiring a rule adding instruction, and displaying a rule adding form based on the rule adding instruction.

The rule adding instruction is an instruction for indicating that a new data quality management rule needs to be configured, and the terminal may obtain the rule adding instruction through various possible manners, for example, in some embodiments, the rule adding instruction may be received through a rule adding control disposed on the visual display interface.

After the rule adding instruction is obtained, a rule adding form can be displayed based on the rule adding instruction, and an area for receiving each data processing rule input by a user can be displayed in the rule adding form so as to obtain the data processing rule determined and input by the user.

Step S304: and obtaining the data processing rule based on the input operation of the user on the rule newly-added interface.

Based on the displayed rule newly-added form, the user can operate on the displayed rule newly-added form to determine the data processing rule. In some embodiments, the user may obtain the data processing rule configured at this time by selecting a preset data processing rule, that is, by selecting an operation, in the rule newly-added form, and in other embodiments, the user may obtain the data processing rule configured at this time in the rule newly-added form by means of a custom input, that is, by means of a custom input operation, which is not specifically limited in embodiments of the present application.

The data processing rule refers to a rule for processing the data corresponding to the data processing rule, and may include a rule for the condition that the data needs to satisfy, such as a data format that needs to satisfy, a range of data size that needs to satisfy, or a rule for processing the data, such as a processing mechanism when the data does not satisfy the range of data size.

Step S306: and acquiring a data quality management rule based on the data processing rule.

When obtaining the data processing rules obtained by the user based on the processing of the rule added form, these data processing rules may be combined to obtain the configured data quality management rules, where the specific combination manner is not limited. It may be understood that, when the user sets more than two data processing rules based on the rule newly added form, the obtained configured data quality management rule may include more than two data processing rules set by the user, and the data processing rules may be a relationship (i.e. need to be satisfied at the same time) or a relationship (i.e. need to satisfy only one rule) between the data processing rules, and may be set based on a requirement of actual data processing.

In some embodiments, the data processing rule obtained based on the processing of the rule added form by the user may include a data collection rule.

The data acquisition rule can be a standard and a specification for data acquisition of the data source, and the quality of the data source can be ensured to a certain extent by setting the data acquisition rule. The data acquisition rules may be set in connection with actual technical needs, and may include, for example, data acquisition time, data acquisition fields, and so forth.

In some embodiments, the data processing rule obtained based on the processing of the rule newly added form by the user may include a data input rule.

Data input rules refer to input rules that need to be satisfied when collecting data of a data source, and may implement control over input fields of the data, for example, may include necessary-to-fill fields of the data, data format requirements, and so on.

In some embodiments, the data processing rule obtained based on the processing of the rule added form by the user may include a data verification rule.

Data validation rules refer to rules used to validate data, and in some embodiments, data validation rules may include, but are not limited to, at least one of data type, data format, and data range. Where the data type is a requirement of the data type to be satisfied by the data, such as integer, long integer, floating point, etc., but not limited thereto. The data format is a requirement of a format that the data needs to satisfy, such as a numerical value, a character string, a binary, and the like, but is not limited thereto. The data range is a range of values that the data should satisfy, for example, the value of a certain data should be between a first value threshold and a second value threshold, or the value of a certain data should be greater than a third value threshold, etc., but is not limited thereto.

In some embodiments, the data processing rule obtained based on the processing of the rule added form by the user may include a data cleansing rule.

The data cleaning rule refers to a rule formulated for processing data in order to meet quality requirements such as data integrity and consistency. In some embodiments, the data cleansing rules include, but are not limited to: at least one of redundant data cleansing rules, format normalization rules, invalid data identification and handling rules, outlier handling rules.

The redundant data cleansing rule refers to a rule for cleansing redundant data, and the redundant data refers to data repeatedly existing in data, for example, two or more identical data exist. The existence of redundant data occupies more storage space, and can lead to repeated processing in the data processing process, thereby influencing the processing efficiency. Therefore, by formulating the redundant data cleaning rule, the optimization of the storage space and the improvement of the processing efficiency are facilitated.

The format standardization rule refers to a rule for converting various data into the data with the same format, and is beneficial to realizing the consistency of the formats of various data and the processing efficiency of the subsequent data processing by converting the various data into the data with the same format. The standardized format of the data standardized rule can be set in combination with the actual data processing requirement.

The invalid data identifying and processing rule refers to a rule that identifies invalid data and processes the identified invalid data. By identifying invalid data and processing the invalid data, the data processing efficiency is improved.

By identifying and deleting redundant or invalid data, consistency of the data set can be ensured. By converting the data into a unified format and standard, subsequent processing and analysis are facilitated to improve the efficiency of subsequent data processing and analysis.

The outlier processing rule refers to a determination method of an outlier and a rule of how to process an existing outlier, and the outlier may be a missing value or an outlier. In some embodiments, the determination of the outliers, including but not limited to the normal range of the key indicators, the definition of the outliers, and the threshold setting of the outliers, etc., may be set based on domain knowledge, business requirements, and data characteristics. For abnormal values, based on actual technical requirements, different modes can be adopted for processing, for example, for missing data, the mode of filling default values, interpolating or deleting missing values can be adopted for processing, and manual auditing processing can also be adopted. In the event that an outlier is identified, it may also be processed by sending an alarm, generating a report, or taking other action.

The default values and interpolation values can be determined according to service requirements and data characteristics. For example, the distribution, range, and missing values of the data are known by analyzing and exploring the raw data. Depending on the data characteristics, suitable default values may be selected, for example using average, median, mode etc. as default values, or using specific marker values to represent missing data. For another example, interpolation methods (e.g., linear interpolation, polynomial interpolation, KNN interpolation, etc.) may be selected to fill in missing values or fix outliers, and appropriate default values and interpolation methods may be determined based on knowledge and experience in different data exploration fields. In some embodiments, some configurable options may also be provided, and the user may customize the default values and interpolation methods based on the configurable options.

In some embodiments, the data quality management rules may also include data documents and metadata management rules, such as recording metadata information such as data structures, field descriptions, data dictionaries, and the like, and establishing data blood-edge relationships to facilitate tracking of the source, conversion, and use of data. In some embodiments, the data quality management rules may also include user feedback rules to receive user-reported data quality problems and to be able to receive user-entered data error correction information and/or user audit information for the data.

In some embodiments, an interface diagram for customizing configuration data quality management rules at a terminal may be as shown in fig. 4.

Referring to fig. 4, a new rule group may be generated by clicking on the new rule group, each rule group may include a plurality of rule points, and by clicking on the new rule point, a new rule point may be added, which is or between the rule group and the rule group, and which is or between the rule points. Wherein each rule set or rule point represents a set of data processing rules. The drop-down box data in the rule points generates drop-down options by reading the data fields of the database, wherein the filtering condition options may include, but are not limited to (greater than, equal to, less than, including, function, regular, etc.), and the custom rule is to determine the current input box rule (e.g., function, regular, etc., in some embodiments, code blocks may also be input) according to the filtering condition.

Based on the customized data quality management rule, in the process of performing data quality management, the original data can be read and parsed, for example, by acquiring the data from a data source such as a file, a database, an API and the like, and parsing the data into an operable data structure such as an object and an array, and then applying the customized data quality management rule to the data. Traversing the data field by field based on the configured data quality management rule, and performing corresponding cleaning operation according to the rule. In this process, operations such as data conversion, formatting, filtering, merging, etc. may be involved, and the data after the operations may be output, for example, saved to a file, database, or directly presented to the user in a user interface.

In some specific examples, data quality management may be implemented through the setting and coordination of correlation functions on the basis of the Mithril framework. For example, by executing manageDataQuality (rawData) to run a data quality management flow, manageDataQuality () function call VALIDATEDATA () validates data, and if the data validation passes, then CLEANDATA () and deduplicateData () are called in sequence to perform data cleaning and deduplication operations. And finally outputting the processed data or corresponding error information. The custom created data quality management rule may be transferred manageDataQuality in a function form to run and perform quality management flow. VALIDATEDATA () function is used to verify the integrity and accuracy of the data. It traverses each entry in the original data, for example, checks whether there is a missing value, whether the age field is a number, and whether the mailbox format is correct. CLEANDATA () function is used to clean the data, it traverses each entry in the original data, e.g., removes the front and back spaces of the name and mailbox fields, and converts the age field to a digital type. deduplicateData () function is used to de-duplicate the data, e.g., it traverses the cleaned data, by comparing the ID fields to determine if duplicate entries exist.

In some embodiments, referring to fig. 5, the determining the data loading policy corresponding to the data loading request in step S202 includes steps S502 to S508.

Step S502: user behavior data and/or performance detection data are obtained.

User behavior data refers to data related to user behavior in a data processing process, such as user operation type, user interaction time, page residence time and the like. The user's behavioral data during the data exploration process may be collected by user behavioral analysis tools, such as Google analysis, mixpanel, etc.

The performance detection data refers to data related to the loading performance. In some embodiments, the performance detection data may include data obtained by detection during data processing, such as page load time, resource load time, first rendering time, and so forth. The performance detection data may be monitored by a performance monitoring tool, such as WebPageTest, lighthouse, etc.

In some embodiments, the performance test data may also be determined based on performance test data. For example, the test page may be preloaded, and the performance test data such as the page loading time, the resource loading time and the like of the test page may be obtained by inserting performance monitoring codes at key positions of the test page. For another example, setting event tracking in a user behavior analysis tool, obtaining performance test data of an experiment group through processing operation performed by a user on a page before executing the data processing method of the present application, and then obtaining performance test data of a control group through obtaining processing operation performed by the user on the page during executing the data processing method of the present application.

Step S504: and predicting behavior influence parameters corresponding to each data loading mode based on the user behavior data.

By analyzing the user behavior data, the influence of different data loading modes on the user experience can be obtained through analysis, so that behavior influence parameters are obtained, and the influence degree of different data loading modes on the user experience is characterized to a certain extent by the behavior influence parameters. The analysis mode of the user behavior data is not limited, and in practical technical application, any analysis mode capable of analyzing the user behavior data and obtaining behavior influence parameters can be adopted. For example, user behavior data may be analyzed by user behavior analysis tools, such as Google analysis, mixpanel, etc., to obtain the impact of different data loading modes on the user experience, thereby obtaining behavior impact parameters.

The method for estimating the behavior influence parameters corresponding to each data loading mode based on the user behavior data is not limited, and the data loading mode includes paging loading and incremental loading as an example: based on the user behavior data, assuming that the residence time of the user on the page is longer and the user interaction time is shorter, the user may consider that more data needs to be acquired each time: combining the data volume of one page loaded by paging, the data volume of the page browsed by the current user, the user interaction time and the loading time of the data volume of the loaded page, determining the probability that the paging loading mode can meet the user behavior data, namely the probability of quickly looking up the next page can be timely met, and taking the obtained probability as a behavior influence parameter; combining the incremental data quantity loaded by the increment, the user interaction time and the loading time for loading the incremental data quantity, determining the probability that the incremental loading mode can meet the user behavior data, namely, the probability that the next page can be quickly checked in time, and taking the obtained probability as a behavior influence parameter. It should be understood that in other embodiments, the behavior influencing parameters may be determined in other manners, as long as the behavior influencing parameters can characterize the influence degree of different data loading manners on the user experience.

Step S506: and estimating performance influence parameters corresponding to each data loading mode based on the performance detection data.

By analyzing the performance detection data, the influence of different data loading modes on the loading performance can be obtained through analysis, so that performance influence parameters are obtained, and the performance influence parameters represent the influence degree of different data loading modes on the data loading performance of the terminal to a certain extent. The analysis method of the performance detection data is not limited, and in practical technical application, any analysis method capable of analyzing the performance detection data and obtaining performance influence parameters can be adopted. For example, the performance impact parameters may be obtained by analyzing the impact of different data loading patterns on loading performance through performance monitoring tools, such as WebPageTest, lighthouse, etc. Page loading time, resource loading time, first rendering time

The method for estimating the performance influence parameters corresponding to each data loading mode based on the performance detection data is not limited, and the data loading mode includes paging loading and incremental loading as an example: determining the time length required by loading the next page in the paging loading mode based on the page loading time of the performance detection data, and further processing (such as scoring or normalizing) the time length to obtain the score of the loading time length of the paging loading mode as a performance influence parameter; and determining the loading time for loading the incremental data in the incremental loading mode based on the page loading time of the performance detection data, and further processing (such as scoring or normalizing) the loading time to obtain the score of the loading time of the incremental loading mode as a performance influence parameter. It should be understood that in other embodiments, the behavior influencing parameters may be determined in other manners, as long as the behavior influencing parameters can characterize the influence degree of different data loading manners on the user experience.

Step S508: a data loading policy is determined based on the behavior influencing parameters and/or the performance influencing parameters.

The data loading policy of the data loading request may be comprehensively determined by the behavior influencing parameters and/or the performance influencing parameters. The determined data loading policy may be an optimization of the data loading policy, for example, optimizing a resource loading sequence, applying a cache policy, performing code optimization, for example, reducing unnecessary resource requests, compressing a file size, etc., which is not limited in particular.

In some embodiments, the corresponding one or more loading policies may be determined in combination with the behavior-affecting parameter, and the corresponding one or more loading policies may be determined in combination with the performance-affecting parameter, and the data loading policies of the final data loading request may be determined comprehensively based on the one or more loading policies corresponding to the behavior-affecting parameter and the one or more loading policies corresponding to the performance-affecting parameter. The determining the data loading policy of the final data loading request may be determining, by weighting or comprehensively considering the priority, one or more loading policies corresponding to the performance influencing parameters, or using the one or more loading policies corresponding to the performance influencing parameters and the one or more loading policies corresponding to the performance influencing parameters as the data loading policy of the final data loading request. In other embodiments, the comprehensive determination may be performed in other ways. The embodiment of the present application is not particularly limited thereto.

In some embodiments, it may also be possible to combine the behavior impact parameters and the performance impact parameters, determine the composite impact parameters, and determine the data loading policy of the data loading request based on the composite impact parameters. The comprehensive influence parameters may be determined by weighting based on the behavior influence parameters and the performance influence parameters, or may be determined by comprehensively considering other modes, which is not particularly limited in the embodiment of the present application.

In some embodiments, the data loading policy of the data loading request determined above may include one or more of paging loading, delta loading, asynchronous loading, and preloading.

The paging loading refers to dividing data into a plurality of pages or batches, and gradually loading the data of each page or batch according to a user request. For example, for a large dataset, only the first page of data is loaded at initialization, and the next page of data is gradually loaded as the user scrolls or clicks the "load more" button. In a specific implementation process, a request for loading new data of a next page or next batch can be triggered according to interaction actions of a user by monitoring a scrolling event or a button clicking event.

Incremental loading refers to loading only the latest data based on the update frequency of the data and the time stamp of the data, e.g., data whose time stamp is between the last loading time point to the current time point, instead of reloading all the data. In a specific implementation process, a time stamp or a version number of the last loaded data may be recorded, the time stamp or the version number is compared with data of a background server, only updated data is requested based on a comparison result, and the obtained updated data may be combined or added to an existing data set.

Asynchronous loading refers to the process of loading data while other processes may be performed. By using an asynchronous loading mechanism, the situation that the user interface is blocked in the data loading process can be avoided, and the responsiveness of the user interface can be maintained. After the data loading is completed, the user interface may be updated. In the process of loading data, a loading indicator or a placeholder can be displayed on a display interface, so that a user realizes that the data is being loaded, and user experience is improved.

Preloading refers to loading data in advance, so that when the data needs to be used, the data can be directly used and displayed based on the loaded data, and the processing efficiency of the data processing can be improved. The data area which the user may need to view and the data which the data area needs to present can be predicted, and the data can be loaded in advance with the prediction result. In the specific prediction, the user behavior information can be monitored, and the prediction can be performed based on the user behavior information. For example, when a user browses or approaches a certain data region, the data region is considered to be a data region that the user may need to view, and the data of the region is preloaded to reduce the delay of subsequent loading. In some embodiments, the preloaded data may also be stored in a local cache, and by combining the mechanisms of preloading and data caching, the loading speed at the time of revisiting may be increased by storing the loaded data locally.

It should be understood that, in other embodiments, the above-determined data loading policy of the data loading request may also include other data loading policies, and embodiments of the present application are not limited in particular.

In some embodiments, the determined data loading policy may be as follows. Based on the behavior influence parameters, it is predicted that the user will access a smaller data set, and one of the determined data loading strategies can be to load all data at one time, so that the data size is smaller, the loading time is short, and the user experience is not greatly influenced. Based on the behavior influence parameters, it is predicted that the user will access a large data set, and one of the determined data loading strategies may be paging loading, so as to improve data loading efficiency. Based on the behavior influence parameters, static data which is not frequently updated is predicted to be accessed by a user, and one of the data loading strategies is determined to be one-time loading when the page is loaded, and updating is not performed in the process of subsequent user operation. For data updated in real time, the determined data loading strategy can be incremental loading, and updated data can be acquired through polling, webSocket or server pushing and the like. For data which needs to be interacted by a user, the determined data loading strategy can be asynchronous loading, corresponding data is dynamically loaded when the user initiates operation, and instant feedback and response are provided. For data that a user may want to access, the determined data loading policy may be pre-loaded, loaded in advance before the user accesses, to improve loading speed and response performance.

It should be appreciated that during data loading, a data request may be sent based on a data loading policy, which may be an initial load request, a page flip request, an incremental load request, etc. The background processes the data according to the request and returns corresponding data results. The front end receives the data result, and updates the page display content according to the loading strategy, which can be replaced, added or combined. Depending on the user's operation or behavior, the above steps may need to be repeated to meet the user's needs.

Based on the dynamically determined data loading strategy, intelligent data loading is realized, and the performance and user experience of data exploration processing can be improved. According to the size, the updating frequency and the user behavior of the data, a proper loading strategy is selected, and data analysis and optimization are performed on the basis of the loading strategy, so that the aims of efficiently loading the data, reducing delay and providing smooth user experience can be achieved.

In some embodiments, referring to fig. 6, in the process of visual presentation in step S206, in response to the packet operation, determining a data loading policy, and obtaining packet data after the packet operation based on the determined data loading policy, and visually presenting the packet data, steps S602 to S610 may be included.

Step S602: and displaying a grouping screening interface in the visual display process.

The grouping screening form is used for a user to select and determine grouping conditions. A group screening area may be present on the display interface during the process of displaying the visual chart, and the group screening form may be displayed in the group screening area. The grouping screening control can also exist on the display interface in the process of visual chart display, and the grouping screening form is displayed by responding to the operation of the user on the grouping screening control. In other embodiments, the packet screening form may be displayed if other conditions are met.

Step S604: and obtaining packet screening parameters in response to the operation on the packet screening interface.

Based on the displayed packet screening form, the user can operate on the packet screening form and obtain packet screening parameters based on the operation result.

Step S606: and constructing a packet screening object based on the packet screening parameters, wherein the packet screening object comprises the packet screening parameters.

The constructed packet screening object may contain one or more packet screening parameters set by the user on the packet screening form. The method can be that a grouping screening object is built in real time in the process of operating the grouping screening form by a user, and the grouping screening object is updated in real time based on the operation of the user, or the grouping screening object is built based on the monitored grouping screening parameters after the user finishes operating the grouping screening form and clicks buttons such as 'confirm', 'submit' and the like in the grouping screening form.

The data packets may be grouped according to actual needs, for example, the data packets may be grouped according to a specific field, where the packet filtering parameter may include the specific field.

Step S608: and determining a data loading strategy corresponding to the grouping screening object.

After determining the packet screening object, a data loading policy corresponding to the packet screening object may be determined. The manner of determining the data loading policy may be the same as that referred to in the above embodiments, for example, in combination with a behavior influencing parameter and/or a performance influencing parameter, to determine the data loading policy corresponding to the packet screening object.

Step S610: and acquiring packet data corresponding to the packet screening object from a server based on the determined data loading strategy, wherein the packet data comprises data obtained by the server through grouping operation based on the packet screening parameters.

Based on the determined data loading strategy, the packet screening object can be used to send a data request to the server, wherein the data request refers to a request of the server to obtain data matched with the packet screening correspondence, after the data request is sent to the server, the server screens and obtains the corresponding screening data based on each packet screening parameter contained in the packet screening object, and takes the screening data as the packet data after the packet operation, and the data loading strategy is based on the determination, and the data loading strategy is returned to the terminal, so that the terminal obtains the packet data after the packet operation.

Step S612: and visually displaying the grouping data.

In the process of visually displaying the packet data, the terminal can visually display the packet data based on actual technical requirements. For example, in some embodiments, it may be displayed in a data table, drawn in a chart, and so forth.

In some embodiments, referring to fig. 7, in the process of performing visual presentation in step S208, in response to the aggregation operation, determining a data loading policy, obtaining an aggregate data processing result after the aggregation operation based on the determined data loading policy, and performing visual presentation on the aggregate data processing result, and may include steps S702 to S710.

Step S702: and displaying the aggregation option area in the visual display process.

The aggregation option area is an area for a user to select and determine aggregation parameters. An aggregation option area may be present on the display interface during the visual presentation. The aggregation option control can also exist on the display interface in the visual display process, and the aggregation option area is displayed in response to the operation of the user on the aggregation option control. In other embodiments, the aggregation option area may be displayed if other conditions are satisfied.

Step S704: in response to an operation in the aggregation option area, an aggregation parameter is obtained.

Based on the displayed aggregation option area, the user can perform an operation on the aggregation option area and obtain the aggregation option area based on the operation result.

In some embodiments, the aggregation option area may include an aggregation field selection area and an aggregation function selection area, where obtaining the aggregation parameter in response to the operation in the aggregation option area may include:

obtaining an aggregation field in response to an operation in the aggregation field selection area;

obtaining an aggregation function in response to an operation in the aggregation function selection area;

wherein the aggregation parameter includes the aggregation field and the aggregation function.

Thus, by operating in the aggregation option area based on the user, the aggregation parameter including the aggregation field and the aggregation function can be obtained. The embodiment of the application is not particularly limited, and the different aggregation fields can adopt different aggregation functions, or the same aggregation function can be adopted for each aggregation field, or the same aggregation function can be used for carrying out aggregation processing on the data of a plurality of different aggregation fields.

Where the aggregation parameters may be determined according to actual needs, in some embodiments, the functions or libraries provided by the Mithril framework, such as lodash, may be used as the aggregation functions. For example, an aggregation function such as summation, average, maximum, minimum, etc. may be selected, from which the aggregate calculation operation is performed on the packet data. As another example, the aggregate computing operation is implemented using functions provided by the Mithril framework or a related data processing library, such as D3.js, or the like.

Step S706: and constructing an aggregation object based on the aggregation parameters, wherein the aggregation object comprises the aggregation parameters.

The built aggregate object may contain one or more aggregate parameters set by the user in the aggregate options area. The aggregation object may be constructed in real time during the process of operating the aggregation parameter by the user, and updated in real time based on the operation of the user, or may be constructed based on the monitored aggregation parameter after the user finishes operating in the aggregation option area and clicks the buttons such as "confirm", "submit" in the aggregation option area.

Step S708: and determining a data loading strategy corresponding to the aggregation object.

After determining the aggregate object, a data loading policy corresponding to the aggregate object may be determined. The manner of determining the data loading policy may be the same as that referred to in the above embodiments, for example, in combination with a behavior influencing parameter and/or a performance influencing parameter, to determine the data loading policy corresponding to the aggregate object.

Step S710: and acquiring an aggregate data processing result corresponding to the aggregate object from a server based on the determined data loading strategy, wherein the aggregate data processing result is obtained by the server for performing aggregate operation on the corresponding data based on the aggregate parameter.

Based on the determined data loading strategy, the aggregation object can be used for sending a data request to a server so as to obtain an aggregate data processing result after the server performs aggregation operation on the corresponding data based on the aggregation parameters. The data request refers to a request of a server to obtain data corresponding to an aggregation object, and aggregate the data in a mode corresponding to the aggregation object, after the data request is sent to the server, the server obtains data corresponding to each aggregation field based on each aggregation field contained in the aggregation object, performs aggregation processing based on an aggregation mode corresponding to an aggregation function, obtains an aggregation data processing result, and returns the aggregation data processing result to the terminal based on the determined data loading strategy, so that the terminal obtains the aggregation data processing result.

Step S712: and visually displaying the aggregate data processing result.

In the process of visually displaying the aggregated data processing result, the terminal can perform visual display based on actual technical requirements. For example, in some embodiments, the aggregate data handling results may be presented in a data table, in a chart (e.g., bar graph, line graph, pie graph), and so forth.

In some embodiments, the aggregated data processing results may be bound to the visualization component and presented on the user interface using the view component of the Mithril framework. In some embodiments, the user interface may have an interactive function (such as a drop-down menu, a slider, a check box, etc.) to enable receiving an operation of a user, so as to implement screening, sorting, or switching a display manner of the aggregated data processing result, so that the user can flexibly operate on the aggregated data.

Taking an implementation using Mithril framework as an example, in some embodiments, advanced aggregation functions, such as multi-level grouping, nested aggregation, time-series aggregation, etc., may be implemented using the functionality provided by Mithril framework or a related database, such as PivotTable. Js, etc. In other embodiments, performance optimization tools provided by Mithril framework may be used to perform performance optimization, such as data indexing, caching of computing results, distributed computing, and the like, and embodiments of the present application are not limited in detail.

In some embodiments, the group screening form and the aggregation option area may be displayed simultaneously during the visual display process, or the group screening form and the aggregation option area may be contained in the same data exploration page. Taking the example that the grouping screening form and the aggregation option area are simultaneously contained in the same data exploration page, a view component of Mithril frames can be used for constructing the data exploration page in a declarative mode, and the elements such as the data display area, the screening component and the aggregation option are arranged in the page. For example, in the 'view' method of the 'UserList' component, the'm' function provided by Mithril is used to create the HTML elements and components, and to dynamically render in conjunction with the user information in the data model. The rendering result is mounted under the 'body' element of the HTML document, and dynamic updating and responsive design of the page are achieved through a virtual DOM technology provided by a Mithril framework.

In some specific examples, a container element, such as a 'div' element, may be declared in the HTML file as the root element of the page. A page component is then created using Mithril and mounted to the root element, defining an area in the page component for exposing data. May be a table, chart. The data presentation area may be laid out and stylized using HTML tags and CSS styles.

A screening component is created that acts as a group screening form, which may be a form, drop down menu, or other interactive element, for allowing a user to select particular data screening conditions. The components provided by the HTML tags and framework can be used to create a filtering component and add event processing logic thereto. The grouping screening form may include an input field for selecting a screening condition, and a submit button, and the user may input the screening condition in the grouping screening form, click the submit button, and obtain the screening condition input by the user based on the operation of clicking the submit button, so as to construct the grouping screening object.

An aggregation option component is created and presented in an aggregation option area for allowing a user to select the manner in which data is aggregated. The aggregation option component may be a drop down menu, a set of check boxes. Similar to the filtering component, the components provided by the HTML tags and framework can be used to create and add event processing logic to the aggregation options. In some embodiments, the aggregation option area may include an input field for selecting an aggregation function (such as summation, average, maximum, etc.) and specifying an aggregation field, and the user selects the aggregation function and specifies the aggregation field in the aggregation option area, and obtains the aggregation function and the aggregation field selected by the user to construct the aggregation parameter object.

Referring to fig. 8, in other embodiments, on the basis of the above embodiments, the method further includes:

Step S210: and acquiring the sharing operation data of the data exploration processing, and sending the sharing operation data to a server under the condition that the cooperative sharing condition is met, so that the sharing operation data is sent to a cooperative sharing client through the server for visual display.

The shared operation data is data that needs to be shared with other terminals that perform cooperative sharing during the data search process. The collaboration sharing condition refers to a condition that sharing operation data is sent to a server so that the server sends the sharing operation data to a collaboration sharing client, and may be considered to satisfy the collaboration sharing condition every time the latest sharing operation data is received, or may be considered to satisfy the collaboration sharing condition every preset time period, or may be considered to satisfy the collaboration sharing condition after receiving an operation of "confirmation", "save", "submit" or other similar processing confirmation components by a user.

Taking as an example that the cooperative sharing condition is considered to be satisfied after the operation of the processing confirmation component by the user is received, in some embodiments, on the basis of the method of the foregoing embodiments, the method further includes:

creating and displaying a data exploration processing component, wherein the data exploration processing component comprises a text box input component and a processing confirmation component;

updating and storing sharing operation data based on the input information of the text box input component;

And under the condition that a confirmation instruction is received through the processing confirmation component, the sharing operation data is sent to a server, so that the sharing operation data is sent to a cooperative sharing client through the server for display.

In some implementations, a component may be created at the terminal that includes a text box and a save button. When a user inputs content in a text box, the terminal updates the value of vnode.state.shareddata, and sends the data to the server when clicking a save button, thereby realizing the sending of sharing operation data.

Therefore, the sharing operation data is sent to the server in the data exploration processing process so as to be sent to the auxiliary sharing client, and the cooperative sharing in the data exploration process is realized.

In some embodiments, on the basis of the above embodiments, the method further includes:

under the condition that the sharing operation data of the cooperative sharing client sent by the server is received, updating the stored sharing operation data by using the sharing operation data of the cooperative sharing client;

and rendering the visual display page by using the updated sharing operation data.

In some specific examples, taking the example that the terminal updates the value of vnode.state.shareddata to obtain the sharing operation data, the terminal may monitor the message from the server through WebSocket, and when receiving the message, update the value of vnode.state.shareddata, and re-render the page to reflect the latest sharing data.

Therefore, under the condition that the sharing operation data of the current client side is sent to other cooperative sharing client sides for sharing, the sharing operation data of the other cooperative sharing client sides can be obtained and updated and displayed, so that convenience of cooperative sharing is improved, and user experience can be improved.

In some embodiments, the sharing operation data may include: annotating the information. The annotation information may be information generated by a user based on an annotation operation, such as annotation information, tag information, or comment information. Thus, by sending annotation information to the collaboration sharing client, discussion and communication among team members in team collaboration is facilitated. When new annotation information exists, reminding information can be sent, and notification or reminding can be timely sent to relevant team members, so that the possibility of effective communication and cooperation is improved.

And receiving a data sharing instruction, and generating a data link based on the data sharing instruction, wherein the data link corresponds to the corresponding data content. The data link may be sent to other terminals so that the other terminals may also access the corresponding data content based on the data link so that they may view and access the corresponding data content.

and receiving a data export instruction, exporting data based on the data export instruction and storing the data as a target format file, such as an EXCEL format file, so that the data of the target format file can be conveniently shared with other people or further analyzed by exporting the data of the target format file.

In some embodiments, version change information for the data may also be recorded to enable different versions of the data to be retrospectively and compared based on the version change information. In some embodiments, the performed data processing operations may also be revoked based on the undo operation to correct the described. In some embodiments, the previous data processing state may also be restored based on the restore operation to restore the data changes. Specific technical implementation embodiments of the present application are not particularly limited.

Based on the embodiment, collaborative sharing of data exploration is realized, and the collaborative sharing focuses on analysis, exploration and interaction of data, so that efficiency of team in data exploration is improved.

Taking three clients A, B, C for collaborative sharing data exploration, it is assumed that client a is responsible for initial data loading, and client a needs to load data from a server to the front end of client a. The server receives the data loading request of the client A and returns corresponding data results, and the client A performs screening and aggregation operations on the loaded data based on the mode in the embodiment, selects interesting data dimensions and metrics and generates a data display view.

And the server receives the screening and aggregation request of the client A, processes data according to the request and returns a screened data result. The client a interacts and explores the data presentation view, such as marking data points, adding annotations, exporting data, and the like. And the server dynamically updates the data display view according to the interactive operation of the client A and sends the updated result to all the clients participating in the collaboration.

Client B, C joins the collaboration sharing task and communicates with client a through a server. Client B interacts and explores the data presentation view, e.g., shares labeled data points with client a, adds new data labels, etc. Client C may receive the data interaction results of client a and client B via the server and share their own data tags and comments with them.

When any client modifies or interacts with the data, the server is responsible for synchronizing the data to other clients to ensure that all clients participating in the collaboration can see the latest data state and interaction results. The clients can participate in screening, aggregation, interaction and exploration of data simultaneously in the collaborative sharing process of data exploration, share own data marks and comments, and synchronize operation results of other clients in real time. Such collaborative sharing functionality helps users better understand data, discovery patterns, and insight in data exploration, and make deeper analyses and decisions.

In the collaboration sharing process, a user authority control mechanism may be set, for example, the user needs to register, log in and authenticate, and only the authorized user can perform the collaboration sharing function. For example, the users with different roles or authority levels, such as an administrator, an editor, a viewer, etc., have different data access and operation authorities, ensuring that sensitive data is only visible or editable to a particular user.

It should be understood that in the process of data processing, after the data is encrypted in the process of data transmission, a secure transmission protocol is adopted to perform data transmission, so as to prevent the data from being accessed or stolen by unauthorized people and protect the security of the data in the transmission process.

In some embodiments, as shown in fig. 9, a data processing method is provided, which is illustrated by using the method applied to the server 104 in fig. 1 as an example, and at least includes the following steps S902 to S908.

Step S902: and acquiring a data loading request and determining a data loading strategy corresponding to the data loading request.

The data loading request is a request for indicating loading data, and the server can obtain the data loading request sent by the terminal.

When determining the data loading strategy corresponding to the data loading request, the data loading strategy can be determined by combining with the actual technical requirement, and in some embodiments of the application, the data loading strategy can be determined in real time by combining with the actual application, so that intelligent data loading is realized. The data loading strategy determined in real time can be a data loading strategy determined by a terminal and is carried in a data loading request to be provided for a server. The server may determine the data loading policy in real time after receiving the data loading request.

Step S904: and acquiring target data corresponding to the data loading request, and transmitting the target data to a sender of the data loading request based on the data loading strategy so as to visually display the target data on the sender.

The target data refers to data that needs to be obtained as determined based on the data load request. The process of obtaining the target data based on the data loading request may be determined based on actual technical requirements, for example, the target data corresponding to the data loading request may be obtained from the data storage system.

Taking the example of obtaining the data loading request from the terminal, the server may send the target data to the terminal, so as to perform visual display on the target data at the terminal.

Step S906: the method comprises the steps of obtaining a packet request of a sender, determining a data loading strategy corresponding to the packet request, screening and obtaining corresponding packet data based on the packet request, and sending the packet data to the sender based on the determined data loading strategy so as to carry out visual display on the packet data at the sender.

Wherein the packet request is a request for indicating to perform a packet operation or a data screening operation, and the server can obtain the packet request sent by the terminal.

The grouping request may include a grouping screening object, the grouping screening object may include one or more grouping screening parameters, and the server may screen and obtain grouping data corresponding to the grouping screening parameters based on the grouping screening parameters, and send the grouping data to the terminal, so as to perform visual display at the terminal.

Step S908: acquiring an aggregation request of the sender, determining a data loading strategy corresponding to the aggregation request, performing aggregation operation on target aggregation data based on the aggregation request, acquiring an aggregation data processing result after the aggregation operation, and sending the aggregation data processing result to the sender based on the determined data loading strategy so as to perform visual display on the aggregation data processing result at the sender.

The aggregation request is a request for indicating to perform aggregation processing, and the server can obtain the aggregation request sent by the terminal.

The aggregation request may include an aggregation object, the aggregation request may include one or more aggregation fields and one or more aggregation functions, and the server may obtain data corresponding to each aggregation field based on the aggregation fields, and perform aggregation processing on the data based on an aggregation mode corresponding to the aggregation functions, so as to obtain an aggregate data processing result, and send the aggregate data processing result to the terminal, so as to perform visual display at the terminal.

In some of these embodiments, the method further comprises: acquiring a data quality management rule sent by the sender; collecting data from a data source, carrying out data quality management on the collected data based on the data quality management rule, obtaining data after the data quality management, and storing the data after the data quality management;

At this time, the acquiring the target data corresponding to the data loading request includes: and acquiring target data corresponding to the data loading request from the stored data after the data quality management.

Therefore, by acquiring the data quality management rule sent by the sender (such as the terminal), the server can return the target data corresponding to the data loading request based on the data quality management rule custom configured at the terminal, thereby realizing personalized data quality management.

In some of these embodiments, on the basis of the above embodiments, the method may further include:

And acquiring the sharing operation data sent by the sender, updating the stored sharing operation data based on the sharing operation data, and then sending the updated sharing operation data to the collaborative sharing client for visual display.

The shared operation data is data that needs to be shared with other terminals that perform cooperative sharing during the data search process. Therefore, the sharing operation data is sent to the auxiliary sharing client side participating in the collaborative sharing in the data exploration processing process, so that the collaborative sharing in the data exploration process is realized.

In some of these embodiments, the method may further include, on the basis of the foregoing embodiments:

And when the connection of the new client is monitored, the currently stored sharing operation data is sent to the new client under the condition that the new client is the cooperative sharing client of the sender.

Wherein it can be determined in various possible ways whether the new client is the sender's collaborative sharing client. In some embodiments, a team ID may be created and the client ID bound to the team ID. When the connection of the new client is monitored, the sharing operation data corresponding to the team ID is obtained based on the team ID bound with the client ID of the new client, and the sharing operation data of the sender can be timely sent to the new client due to the binding of the team ID and the client ID of the sender.

Therefore, under the condition that a new client is connected, even if the new client does not perform any processing, the currently stored sharing operation data can be obtained, the condition of data exploration can be quickly and conveniently obtained, the convenience of collaborative sharing in the data exploration process is improved, and the user experience is improved.

Based on the scheme of the embodiment of the application, the processing function of data exploration of each client can be realized, and collaborative sharing among a plurality of clients can be realized. Referring to fig. 10, after any one of the clients (e.g., client 1 or client 2) starts the data exploration function, the processing procedure of the data processing method in the above embodiments of the present application may be executed, for example, performing data quality management based on the configured data quality management rule, filtering and filtering the data in response to the grouping operation, loading the filtered and filtered data based on the intelligently determined data loading policy, performing aggregation processing in response to the aggregation operation after loading the data, binding the result of the aggregation processing with the front page, constructing the front display page, and performing data visualization presentation based on the constructed front page. Under the condition that the cooperative sharing is satisfied, if the client 1 and the client 2 are both bound to the same team ID, and the client 1 and the client 2 both have the authority of cooperative sharing, the sharing operation data of the client 1 and the client 2 in the data exploration process can be shared through forwarding (not shown in fig. 10) of a server, so that the cooperative sharing in the data exploration process is realized.

Based on the scheme of the embodiment of the application, the data processing and analyzing capability in the data exploration and processing process is improved, the changing requirements of users in the aspects of data exploration and analyzing are met, and a more powerful, flexible and innovative data exploration and processing mode is provided. Which can identify and address data quality issues. Through technologies such as data cleaning, abnormal value detection and data restoration, the data quality can be effectively improved, the accuracy and reliability of data exploration are improved, the data management rules can be customized, and quality management of different data types is realized. Through a dynamic data loading strategy, data loading can be performed according to the quantity and the user demands by adopting paging, asynchronous functions, cache and other modes, so that the data loading performance and user experience are improved, the problem of page blocking when the quantity of the data is large is solved, the page response speed is improved, and the aims of efficiently loading the data, reducing delay and providing smooth user experience are fulfilled. By creating webSocket a server at the server, monitoring user behavior, creating team IDs, monitoring and sending messages when clients with the same team IDs are connected, updating operation content at the clients in real time, users can conduct real-time collaborative editing, share data exploration results, and export data reports or files so as to better cooperate with the team and share analysis results.

It should be understood that, although the steps in the flowcharts related to the embodiments described above are sequentially shown as indicated by arrows, these steps are not necessarily sequentially performed in the order indicated by the arrows. The steps are not strictly limited to the order of execution unless explicitly recited herein, and the steps may be executed in other orders. Moreover, at least some of the steps in the flowcharts described in the above embodiments may include a plurality of steps or a plurality of stages, which are not necessarily performed at the same time, but may be performed at different times, and the order of the steps or stages is not necessarily performed sequentially, but may be performed alternately or alternately with at least some of the other steps or stages.

Based on the same inventive concept, the embodiment of the application also provides a data processing device for realizing the above related data processing method. The implementation of the solution provided by the device is similar to the implementation described in the above method, so the specific limitation of one or more embodiments of the data processing device provided below may refer to the limitation of the data processing method hereinabove, and will not be repeated herein.

In one embodiment, as shown in fig. 11, there is provided a data processing apparatus including: a first loading policy determination module 110, a first data loading module 120, a first packet processing module 130, and a first aggregation process 140, wherein:

a first loading policy determining module 110, configured to obtain a data loading request, and determine a data loading policy corresponding to the data loading request;

The first data loading module 120 is configured to load target data based on the data loading policy, and perform visual display on the target data;

the first packet processing module 130 is configured to determine a data loading policy in response to a packet operation in a process of performing a visual chart presentation on the target data, obtain packet data after the packet operation based on the determined data loading policy, and perform a visual presentation on the packet data;

The first aggregation processing module 140 is configured to determine a data loading policy in response to an aggregation operation in a process of performing visual display, obtain an aggregate data processing result after the aggregation operation based on the determined data loading policy, and perform visual display on the aggregate data processing result.

In some of these embodiments, further comprising:

the first quality management rule acquisition module acquires a data quality management rule and sends the data quality management rule to the server, so that the server performs data cleaning on an original data source based on the data quality management rule to acquire the data source for returning the target data.

In some embodiments, the first quality management rule obtaining module is configured to obtain a rule adding instruction, and display a rule adding interface based on the rule adding instruction; based on the input operation of the user to the rule newly added interface, obtaining a data processing rule; and acquiring a data quality management rule based on the data processing rule.

In some embodiments, the first loading policy determination module 110 is configured to obtain user behavior data and/or performance detection data; estimating behavior influence parameters corresponding to each data loading mode based on the user behavior data; estimating performance influence parameters corresponding to each data loading mode based on the performance detection data; a data loading policy is determined based on the behavior influencing parameters and/or the performance influencing parameters.

In some of these embodiments, the determined data loading policy for the data loading request includes one or more of a page load, a delta load, an asynchronous load, and a preload.

In some embodiments, the first packet processing module 130 is configured to display a packet screening interface during the visual presentation; responding to the operation on the grouping screening interface to obtain grouping screening parameters; constructing a packet screening object based on the packet screening parameters, wherein the packet screening object comprises the packet screening parameters; determining a data loading strategy corresponding to the grouping screening object; based on the determined data loading strategy, acquiring packet data corresponding to the packet screening object from a server, wherein the packet data comprises data obtained by the server through grouping operation based on the packet screening parameters; and visually displaying the grouping data.

In some embodiments, the first aggregation processing module 140 is configured to display an aggregation option area during the visual presentation; obtaining an aggregation parameter in response to an operation in the aggregation option area; constructing an aggregation object based on the aggregation parameters, wherein the aggregation object comprises the aggregation parameters; determining a data loading strategy corresponding to the aggregation object; acquiring an aggregate data processing result corresponding to the aggregate object from a server based on the determined data loading strategy, wherein the aggregate data processing result is obtained by the server performing aggregate operation on the corresponding data based on the aggregate parameter; and visually displaying the aggregate data processing result.

In some of these embodiments, the aggregation option area includes an aggregation field selection area and an aggregation function selection area; a first aggregation processing module 140, configured to obtain an aggregation field in response to an operation in the aggregation field selection area; acquiring user-defined aggregation function information input by a user in the aggregation function selection area, and acquiring an aggregation function; the aggregation parameter includes the aggregation field and the aggregation function.

In some of these embodiments, further comprising:

The first cooperative sharing processing module is used for acquiring the sharing operation data of the data exploration processing, and sending the sharing operation data to a server under the condition that the cooperative sharing condition is met, so that the sharing operation data is sent to a cooperative sharing client side through the server for visual display.

In some of these embodiments, the first collaborative sharing processing module is further configured to create and display a data exploration processing component that includes a text box input component and a processing validation component; updating and storing sharing operation data based on the input information of the text box input component; and under the condition that a confirmation instruction is received through the processing confirmation component, the sharing operation data is sent to a server, so that the sharing operation data is sent to a cooperative sharing client through the server for display.

In some embodiments, the first cooperative sharing processing module is further configured to update, when receiving the shared operation data of the cooperative sharing client sent by the server, stored shared operation data with the shared operation data of the cooperative sharing client; and rendering the visual display page by using the updated sharing operation data.

In one embodiment, as shown in FIG. 12, there is provided a data processing apparatus comprising: a second loading policy determination module 210, a second data loading module 220, a second packet processing module 230, and a second aggregation process 240, wherein:

a second loading policy determining module 210, configured to obtain a data loading request, and determine a data loading policy corresponding to the data loading request;

the second data loading module 220 is configured to obtain target data corresponding to the data loading request, and send the target data to a sender of the data loading request based on the data loading policy, so as to perform visual display on the target data at the sender;

A second packet processing module 230, configured to obtain a packet request of the sender, determine a data loading policy corresponding to the packet request, screen and obtain corresponding packet data based on the packet request, and send the packet data to the sender based on the determined data loading policy, so as to visually display the packet data on the sender;

The second aggregation processing module 240 is configured to obtain an aggregation request of the sender, determine a data loading policy corresponding to the aggregation request, perform an aggregation operation on the target aggregate data based on the aggregation request, obtain an aggregate data processing result after the aggregation operation, and send the aggregate data processing result to the sender based on the determined data loading policy, so as to visually display the aggregate data processing result on the sender.

In some embodiments, the method further includes a second quality management rule obtaining module, configured to obtain a data quality management rule sent by the sender; collecting data from an original data source, carrying out data quality management on the collected data based on the data quality management rule, obtaining data after the data quality management, and storing the data after the data quality management;

At this time, the second data loading module 220 is configured to obtain, from the stored data after quality management, target data corresponding to the data loading request.

In some of these embodiments, further comprising:

And the second cooperative sharing processing module is used for acquiring the sharing operation data sent by the sender, updating the stored sharing operation data based on the sharing operation data, and then sending the updated sharing operation data to the cooperative sharing client for visual display.

In some embodiments, the second cooperative sharing processing module is further configured to, when a new client connection is monitored, send currently stored sharing operation data to the new client if the new client is a cooperative sharing client of the sender.

Each of the modules in the above-described data processing apparatus may be implemented in whole or in part by software, hardware, and combinations thereof. The above modules may be embedded in hardware or may be independent of a processor in the computer device, or may be stored in software in a memory in the computer device, so that the processor may call and execute operations corresponding to the above modules.

In one embodiment, a computer device is provided, which may be a server, and the internal structure of which may be as shown in fig. 13. The computer device includes a processor, a memory, and a network interface connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system, computer programs, and a database. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The database of the computer device is used to store data, such as data sources or target data, that requires data exploration. The network interface of the computer device is used for communicating with an external terminal through a network connection. The computer program is executed by a processor to implement a data processing method.

In one embodiment, a computer device is provided, which may be a terminal, and the internal structure thereof may be as shown in fig. 14. The computer device includes a processor, a memory, a communication interface, a display screen, and an input device connected by a system bus. Wherein the processor of the computer device is configured to provide computing and control capabilities. The memory of the computer device includes a non-volatile storage medium and an internal memory. The non-volatile storage medium stores an operating system and a computer program. The internal memory provides an environment for the operation of the operating system and computer programs in the non-volatile storage media. The communication interface of the computer device is used for carrying out wired or wireless communication with an external terminal, and the wireless mode can be realized through WIFI, a mobile cellular network, NFC (near field communication) or other technologies. The computer program is executed by a processor to implement a data processing method. The display screen of the computer equipment can be a liquid crystal display screen or an electronic ink display screen, and the input device of the computer equipment can be a touch layer covered on the display screen, can also be keys, a track ball or a touch pad arranged on the shell of the computer equipment, and can also be an external keyboard, a touch pad or a mouse and the like.

It will be appreciated by persons skilled in the art that the structures shown in fig. 13 and 14 are merely block diagrams of portions of structures associated with aspects of the application and are not intended to limit the computer apparatus to which aspects of the application may be applied, and that a particular computer apparatus may include more or less components than those shown, or may combine some of the components, or have a different arrangement of components.

In an embodiment, a computer device is provided comprising a memory and a processor, the memory having stored therein a computer program, the processor implementing the steps of the method of any of the embodiments described above when the computer program is executed.

In one embodiment, a computer readable storage medium is provided, on which a computer program is stored which, when executed by a processor, implements the steps of the method of any of the embodiments described above.

In an embodiment, a computer program product is provided comprising a computer program which, when executed by a processor, implements the steps of the method in any of the embodiments described above.

The user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or sufficiently authorized by each party.

Those skilled in the art will appreciate that implementing all or part of the above described methods may be accomplished by way of a computer program stored on a non-transitory computer readable storage medium, which when executed, may comprise the steps of the embodiments of the methods described above. Any reference to memory, database, or other medium used in embodiments provided herein may include at least one of non-volatile and volatile memory. The nonvolatile Memory may include Read-Only Memory (ROM), magnetic tape, floppy disk, flash Memory, optical Memory, high density embedded nonvolatile Memory, resistive random access Memory (ReRAM), magneto-resistive random access Memory (Magnetoresistive Random Access Memory, MRAM), ferroelectric Memory (Ferroelectric Random Access Memory, FRAM), phase change Memory (PHASE CHANGE Memory, PCM), graphene Memory, and the like. Volatile memory can include random access memory (Random Access Memory, RAM) or external cache memory, and the like. By way of illustration, and not limitation, RAM can be in various forms such as static random access memory (Static Random Access Memory, SRAM) or dynamic random access memory (Dynamic Random Access Memory, DRAM), etc. The databases referred to in the embodiments provided herein may include at least one of a relational database and a non-relational database. The non-relational database may include, but is not limited to, a blockchain-based distributed database, and the like. The processor referred to in the embodiments provided in the present application may be a general-purpose processor, a central processing unit, a graphics processor, a digital signal processor, a programmable logic unit, a data processing logic unit based on quantum computing, or the like, but is not limited thereto.

The technical features of the above embodiments may be arbitrarily combined, and all possible combinations of the technical features in the above embodiments are not described for brevity of description, however, as long as there is no contradiction between the combinations of the technical features, they should be considered as the scope of the description.

The foregoing examples illustrate only a few embodiments of the application and are described in detail herein without thereby limiting the scope of the application. It should be noted that it will be apparent to those skilled in the art that several variations and modifications can be made without departing from the spirit of the application, which are all within the scope of the application. Accordingly, the scope of the application should be assessed as that of the appended claims.

Claims

1. A method of data processing, the method comprising:

2. The method of claim 1, further comprising, prior to the obtaining the data load request:

3. The method of claim 2, wherein the obtaining the configured data quality management rule comprises:

acquiring a rule adding instruction, and displaying a rule adding interface based on the rule adding instruction;

Based on the input operation of the user to the rule newly added interface, obtaining a data processing rule;

And acquiring a data quality management rule based on the data processing rule.

4. A method according to any one of claims 1 to 3, wherein said determining a data loading policy comprises:

Acquiring user behavior data and/or performance detection data;

estimating behavior influence parameters corresponding to each data loading mode based on the user behavior data;

Estimating performance influence parameters corresponding to each data loading mode based on the performance detection data;

a data loading policy is determined based on the behavior influencing parameters and/or the performance influencing parameters.

5. The method of claim 4, wherein the data loading policy includes one or more of paging loading, delta loading, asynchronous loading, and preloading.

6. A method according to any one of claims 1 to 3, wherein in the process of visual presentation, in response to a grouping operation, determining a data loading policy, and obtaining grouping data after the grouping operation based on the determined data loading policy, and visually presenting the grouping data, includes:

Displaying a grouping screening interface in the visual display process;

responding to the operation on the grouping screening interface to obtain grouping screening parameters;

Constructing a packet screening object based on the packet screening parameters, wherein the packet screening object comprises the packet screening parameters;

Determining a data loading strategy corresponding to the grouping screening object;

based on the determined data loading strategy, acquiring packet data corresponding to the packet screening object from a server, wherein the packet data comprises data obtained by the server through grouping operation based on the packet screening parameters;

And visually displaying the grouping data.

7. A method according to any one of claims 1 to 3, wherein in the process of performing visual presentation, in response to an aggregation operation, determining a data loading policy, and obtaining an aggregate data processing result after the aggregation operation based on the determined data loading policy, and performing visual presentation on the aggregate data processing result, and the method comprises:

Displaying an aggregation option area in the visual display process;

obtaining an aggregation parameter in response to an operation in the aggregation option area;

Constructing an aggregation object based on the aggregation parameters, wherein the aggregation object comprises the aggregation parameters;

determining a data loading strategy corresponding to the aggregation object;

Acquiring an aggregate data processing result corresponding to the aggregate object from a server based on the determined data loading strategy, wherein the aggregate data processing result is obtained by the server performing aggregate operation on the corresponding data based on the aggregate parameter;

and visually displaying the aggregate data processing result.

8. The method of claim 7, wherein the aggregation option area comprises an aggregation field selection area and an aggregation function selection area;

the obtaining the aggregation parameter in response to the operation in the aggregation option area comprises the following steps:

Acquiring user-defined aggregation function information input by a user in the aggregation function selection area, and acquiring an aggregation function;

The aggregation parameter includes the aggregation field and the aggregation function.

9. A method of data processing, the method comprising:

10. The method according to claim 9, wherein:

the method further comprises the steps of: acquiring a data quality management rule sent by the sender; collecting data from an original data source, carrying out data quality management on the collected data based on the data quality management rule, obtaining data after the data quality management, and storing the data after the data quality management;

The obtaining the target data corresponding to the data loading request comprises the following steps: and acquiring target data corresponding to the data loading request from the stored data after the data quality management.

11. A data processing apparatus, the apparatus comprising:

12. A data processing apparatus, the apparatus comprising:

13. A computer device comprising a memory and a processor, the memory storing a computer program, characterized in that the processor implements the steps of the method of any one of claims 1 to 10 when the computer program is executed.

14. A computer readable storage medium, on which a computer program is stored, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any of claims 1 to 10.

15. A computer program product comprising a computer program, characterized in that the computer program, when being executed by a processor, implements the steps of the method of any one of claims 1 to 10.