CN113704298A

CN113704298A - Data processing method and device, electronic equipment and storage medium

Info

Publication number: CN113704298A
Application number: CN202010432775.1A
Authority: CN
Inventors: 谢明枫; 燕存
Original assignee: Cainiao Smart Logistics Holding Ltd
Current assignee: Cainiao Smart Logistics Holding Ltd
Priority date: 2020-05-20
Filing date: 2020-05-20
Publication date: 2021-11-26

Abstract

The embodiment of the application provides a data processing method, a data processing device, electronic equipment and a storage medium, wherein the method comprises the following steps: receiving task related information, wherein the task related information comprises task analysis information and task configuration information; determining analysis data according to the task analysis information; determining and configuring a corresponding data analyzer based on the task configuration information; determining a data analysis result based on the configured data parser and the analysis data. The data processing efficiency can be improved.

Description

Data processing method and device, electronic equipment and storage medium

Technical Field

The present application relates to the field of computer technologies, and in particular, to a data processing method, a data processing apparatus, an electronic device, and a storage medium.

Background

The data analysis means that a large amount of collected data is analyzed by using a proper statistical analysis method, and the collected data is summarized, understood and digested so as to maximally develop the function of the data and play the role of the data. Data analysis is the process of studying and summarizing data in detail to extract useful information and to form conclusions.

One existing analytical process is: the user provides the required data to be analyzed to the technical staff offline, and the technical staff develops an algorithm model according to the requirements of the user so as to analyze the related data through the algorithm model to determine the corresponding analysis result, the prediction result and the like.

However, in this way, for some similar requirements, it also takes a lot of time to re-develop the algorithm model, and the process of data analysis takes a long time.

Disclosure of Invention

The embodiment of the application provides a data processing method to improve processing efficiency.

Correspondingly, the embodiment of the application also provides a data processing device, an electronic device and a storage medium, which are used for ensuring the realization and the application of the system.

In order to solve the above problem, an embodiment of the present application discloses a data processing method, including: receiving task related information, wherein the task related information comprises task analysis information and task configuration information; determining analysis data according to the task analysis information; determining and configuring a corresponding data analyzer based on the task configuration information; determining a data analysis result based on the configured data parser and the analysis data.

In order to solve the above problem, an embodiment of the present application discloses a data processing method, including: acquiring a target processing task, wherein the target processing task is determined according to analysis data and a configured data analyzer, the analysis data is determined according to task analysis information, and the configured data analyzer is configured according to task configuration information; analyzing and processing the analysis data through a configured data analyzer to determine a data analysis result; and outputting the data analysis result.

In order to solve the above problem, an embodiment of the present application discloses a data processing method, including: providing a task receiving page; determining task related information according to operation on the task receiving page, wherein the task related information comprises task analysis information and task configuration information, the task analysis information is used for determining analysis data, and the task configuration information is used for configuring a data analyzer so as to analyze and process the analysis data; and sending the task related information.

In order to solve the above problem, an embodiment of the present application discloses a data processing method, including: providing a fifth page, wherein the fifth page comprises an uploading control; determining a data analyzer and corresponding parameter information according to the triggering of the uploading control, wherein the data analyzer is used for configuring according to task configuration information and executing analysis processing of analysis data; and uploading the data parser and the corresponding parameter information.

In order to solve the above problem, an embodiment of the present application discloses a data processing apparatus, including: the task receiving module is used for receiving task related information, and the task related information comprises task analysis information and task configuration information; the data determining module is used for determining analysis data according to the task analysis information; the analyzer configuration module is used for determining and configuring a corresponding data analyzer based on the task configuration information; and the result determining module is used for determining a data analysis result based on the configured data parser and the analysis data.

In order to solve the above problem, an embodiment of the present application discloses a data processing apparatus, including: the receiving module is used for acquiring a target processing task, the target processing task is determined according to analysis data and a configured data analyzer, the analysis data is determined according to task analysis information, and the configured data analyzer is configured according to task configuration information; the analysis module is used for analyzing and processing the analysis data through a configured data analyzer and determining a data analysis result; and the result output module is used for outputting the data analysis result.

In order to solve the above problem, an embodiment of the present application discloses a data processing apparatus, including: the page providing module is used for providing a task receiving page; the task determining module is used for determining task related information according to operation on the task receiving page, wherein the task related information comprises task analysis information and task configuration information, the task analysis information is used for determining analysis data, and the task configuration information is used for configuring a data analyzer so as to analyze and process the analysis data; and the sending module is used for outputting the task related information.

In order to solve the above problem, an embodiment of the present application discloses a data processing apparatus, including: the providing module is used for providing a fifth page, and the fifth page comprises an uploading control; the analyzer uploading module is used for determining a data analyzer and corresponding parameter information according to the triggering of the uploading control; and uploading the data analyzer and the corresponding parameter information, wherein the data analyzer is used for configuring according to the task configuration information and executing analysis processing of analysis data.

In order to solve the above problem, an embodiment of the present application discloses an electronic device, including: a processor; and a memory having executable code stored thereon, which when executed, causes the processor to perform the method as described in one or more of the above embodiments.

To address the above issues, embodiments of the present application disclose one or more machine-readable media having executable code stored thereon that, when executed, cause a processor to perform a method as described in one or more of the above embodiments.

Compared with the prior art, the embodiment of the application has the following advantages:

in the embodiment of the application, after receiving the task related information, generating analysis data according to the task analysis information of the task related information; according to the task configuration information of the task related information, the corresponding data parser in the server side can be called, the called data parser is configured, the data analysis process can be completed more quickly through the configured data parser, and the data processing efficiency is improved.

Drawings

FIG. 1 is a schematic structural diagram of a data analysis system according to an embodiment of the present application;

FIG. 2 is a flow chart of the steps of an embodiment of a data processing method of the present application;

FIG. 3A is a schematic diagram of a first example page of an embodiment of the present application;

FIG. 3B is a diagram illustrating a second example page in accordance with an embodiment of the present application;

FIG. 3C is a diagram illustrating a third example page in accordance with an embodiment of the present application;

FIG. 3D is a diagram illustrating a fourth example page in accordance with an embodiment of the present application;

FIG. 3E is a diagram illustrating an example of a fifth page in an embodiment of the present application;

FIG. 4 is a diagram illustrating an example of a process for permissions of a user group according to an embodiment of the present application;

fig. 5 is a schematic diagram of an example of a module architecture of a server according to an embodiment of the present application;

FIG. 6 is a flow chart of steps in another data processing method embodiment of the present application;

FIG. 7 is a flow chart of steps of yet another data processing method embodiment of the present application;

FIG. 8 is a flow chart of steps of yet another data processing method embodiment of the present application;

FIG. 9 is a flow chart of steps of yet another data processing method embodiment of the present application;

FIG. 10 is a block diagram of an embodiment of a data processing apparatus of the present application;

FIG. 11 is a schematic block diagram of another embodiment of a data processing apparatus according to the present application;

FIG. 12 is a schematic block diagram of another embodiment of a data processing apparatus according to the present application;

FIG. 13 is a schematic block diagram of another embodiment of a data processing apparatus according to the present application;

fig. 14 is a schematic structural diagram of an apparatus according to an embodiment of the present application.

Detailed Description

In order to make the aforementioned objects, features and advantages of the present application more comprehensible, the present application is described in further detail with reference to the accompanying drawings and the detailed description.

The embodiment of the application can be applied to the field of cloud computing, wherein cloud computing is a service related to information technology, software and the Internet, computing resources can be integrated, and users can execute required processing by using the computing resources provided by the cloud computing. With the continuous development of the technology, cloud computing can be understood as a product of the convergence of computer technology and network technology development, such as distributed computing, utility computing, load balancing, parallel computing, network storage, hot backup redundancy, content distribution and virtualization, and the like. The embodiment of the application can be applied to various service scenes based on cloud computing and can also be applied to various distributed processing scenes.

Fig. 1 illustrates a data processing system according to an embodiment of the present application, which can be applied to various scenarios of data analysis and processing, and can obtain a scenario of an analysis result by collecting relevant data and performing analysis and planning on the relevant data by using a corresponding algorithm. The method is applied to data analysis scenes such as traffic route planning, product operation, logistics transportation planning, comprehensive traffic planning and smart cities. The system can provide various algorithms, computing resources and the like for users to use, so that the users can select the needed algorithms, import data to create analysis tasks of corresponding methods, perform computing analysis on related data according to the used algorithms and determine data analysis results. For example, in a traffic route planning scenario, the system may collect existing traffic routes within a city and analyze the relevant route data according to a corresponding route planning algorithm to determine a traffic route planning result. For example, in a product operation scenario, sales data of a plurality of products in a historical period may be collected, and sales of the products may be predicted by using a sales analysis algorithm, so as to provide suggestions to users. For example, in a logistics transportation planning scenario, operation data of a logistics transportation tool may be collected, and the operation data may be calculated and analyzed according to a corresponding planning algorithm to determine a transportation plan of a logistics object.

The data processing system of the embodiment of the application comprises: the system comprises a user side, a server side and a computing side, wherein electronic equipment of the user side can comprise terminals such as a computer and the like, and can also comprise mobile terminals such as a mobile phone and a tablet personal computer. The user can interact with the server through the electronic device at the user side, so as to create an analysis task, view an analysis result and the like, for example, the data to be analyzed can be determined, a data parser corresponding to a required analysis algorithm can be selected, and the data parser can be understood as a component for performing data analysis processing based on the corresponding algorithm.

The service end can provide various services for users, can be connected with the computing end to provide computing processing of various services and the like, can manage various data and services, and can call computing resources to analyze and compute corresponding analysis tasks. The server can provide various pages, so that a user can conveniently operate through the pages, use required services, view analysis results and the like.

The computing end comprises a plurality of computing devices for providing data computing processing, and can call a required algorithm to execute a corresponding analysis processing task on the data, so that a corresponding analysis result is obtained and returned to the server end. The device on the computing side may provide distributed computing processing services, and may include various edge computing devices, for example, so that the computing services can be provided in combination with a distributed architecture.

The user may perform processing steps such as task creation and calculation at the user side, and the corresponding server side may perform steps of task generation, task deployment and result determination, as shown in fig. 2:

task related information is received in step 202, the task related information comprising task analysis information and task configuration information. The task related information may be understood as various data required for analyzing the task, the task analysis information is used for determining analysis data to be analyzed, the task configuration information is used for determining a required algorithm and related parameter configuration information, and the like.

In order to facilitate user operation, the server side can provide a page, and a user can conveniently set required information based on the page. One page is a task receiving page, and task related information can be received through the task receiving page. In an example of the task receiving page, the task receiving page includes various controls for setting task related information, and the controls may be input controls, selection controls, click controls, and the like, and may be specifically set according to requirements. The user can set various task related information including task names, task types, data ranges, algorithms, data resolvers and the like based on the task receiving page, wherein the data ranges can be set by setting data constraint conditions, and other related information related to the tasks can be set, such as area ranges, industries, industry sets and the like, and can be determined according to the analyzed scene. For example, in the planning of the logistics network, the task related information may include basic information such as a name and an area, and may also include data constraints such as a sorting flow direction, a coverage area, a warehouse fixed flow direction, warehouse syntax information, auxiliary configuration information, and a distribution station.

Wherein the creation of the task and the execution of the task are separable, so that the task receiving page may include a first page and a second page, wherein the first page is used to create the task and the second page is used to execute the task. The task analysis information may be received through a first page, which may include various controls related to creating the task, through which a task name, a type, a scope, a product, an industry set, and the like of the task may be set, and data constraints of data corresponding to the task may also be set, as an example shown in fig. 3A, so as to facilitate determining the required data. Therefore, various task analysis information can be set based on the first page, and a corresponding analysis task is created.

The second page may receive task configuration information, and based on the second page, the configuration of various information required by the task may be performed, for example, in the example shown in fig. 3B, the data parser for processing the task is configured, the required content may be configured, the data parser is selected, and the corresponding number is configured, and the like, and a threshold for processing other configuration data may also be performed, so that by setting various required task configuration information on the second page, and after the user clicks the save setting, the server may correspondingly receive task analysis information. Based on the second page, running analysis aiming at the task can be started, so that running can be triggered after configuration of task configuration information is completed, analysis data corresponding to the task analysis information can be called, in order to guarantee accuracy of the data, the data can be checked at first, and the analysis processing is executed after the data passes the checking. Taking the logistics field as an example, the data parser, such as the heavy goods parser, the delivery parser, the order parser, etc., may be constructed based on the logistics characteristics of transportation, delivery, etc. in the logistics field, so that the corresponding parser may be selected according to the requirements of the task.

Therefore, when a user needs to set task analysis information, a request can be sent to the server, the server can send page data of the task receiving page, the user side analyzes the page data and displays the task receiving page on a display assembly of the electronic equipment, and the user can set various task related information on the task receiving page through a control and trigger execution of the task.

In an alternative embodiment, the task receiving page may be sent to the user side according to a request of a user at the user side, and the task related information may be received according to an operation of the user at the user side on the task receiving page. The server side sends page data of the task receiving page according to the task creating request so as to display the task receiving page; and receives related information of the task-based reception page. The server side sends a task receiving page to the user side according to a task creating request sent by a user of the user side, and the task receiving page is displayed to the user through the user side. And the user side receives the operation of the user on the task receiving page, determines the task analysis information and the task configuration information and sends the task analysis information and the task configuration information to the server side.

In an alternative embodiment, the task analysis information and the task configuration information may be received through two pages, respectively. The task receiving page comprises a first page and a second page. The server receiving the relevant information of the task-based receiving page comprises the following steps: receiving task analysis information based on a first page; task configuration information based on the second page is received. The first page and the second page can be displayed to the user at the user side respectively, and the first page and the second page can also be displayed to the user at the same time, which is not limited in the present application. The user may determine the task analysis information in a first page and the task configuration information in a second page.

The server can establish connection with various data sources such as databases, and then can acquire required data from the databases and also can store the data in the databases of the server. And then the server side converts the data in the database to divide the data into data of corresponding versions, and stores the data into the application database. The server side can acquire data from each data source, and the acquired data can be divided according to scenes corresponding to the data, the type of the data, the acquisition time of the data, the source of the data and the like. Therefore, when the user sets the task related information in the task receiving pages such as the first page, the second page and the like, the selection can be provided for the user, and the required data can be determined conveniently and quickly.

In an alternative embodiment, the task configuration information of the task related data may include parser information for invoking the data parser and parameter configuration information for configuring the invoked data parser so that the configured data parser can parse the analysis data selected by the user. Task configuration information such as resolver information, parameter configuration information and the like can be set through the second page, and the corresponding server can receive the task configuration information. In an optional example, the server sends the page data of the second page to the user side, and the user displays the second page on the user side, sets the parser information and the parameter configuration information through the second page, and sends the parser information and the parameter configuration information to the server side. The user can also set memory resources occupied by the operation of the parser aiming at the tasks, so that the processing tasks corresponding to the data parser can be distributed to the preferred computing equipment for computing processing.

After receiving the task related information, the server may generate analysis data according to the task analysis information in step 204; and, in step 206, based on the task configuration information, determining and configuring the corresponding data parser. The configured data analyzer is used for analyzing the analysis data. That is, the user side can set the task analysis information based on the first page, so that the server side can generate analysis data according to the task analysis information, for example, obtain corresponding data according to data constraint conditions, configure parameters such as names, types, industries and the like corresponding to the tasks, complete configuration of the data corresponding to the tasks, and generate an analysis task to be processed. And then, selecting an analysis task to be processed through a second page, setting task configuration information of the analysis task, and after receiving the task configuration information, the server can set a data parser of the analysis task and other configuration parameters required for task processing to complete the configuration of the task, wherein the data parser required to be called by the analysis task and the related configuration information of the parser can be determined based on the task configuration information, so that the data parser can be called and the configuration of the parser can be carried out, and other parameters related to the task parsing can be configured based on the other configuration parameters, such as requirement parameters for computing equipment and the like. And then, the user can trigger the running of the task on the second page, correspondingly call the analysis data of the analysis task, check the analysis data, and call the data analyzer to process the analysis task after the data is checked to be correct.

And extracting data corresponding to the task analysis information from an application database of the server side as analysis data according to the task analysis information. And calling the corresponding data analyzer according to the analyzer information of the task configuration information, configuring the called data analyzer according to the parameter configuration information of the task configuration information, and analyzing the analysis data by the configured data analyzer.

In an optional embodiment, the step of generating the analysis data according to the task analysis information includes: acquiring target data from a target data source according to the task analysis information; and converting the target data into analysis data of a target version according to a preset conversion rule. The target data source can be a database, an Excel table and the like, target data can be obtained through timing obtaining and manual uploading by a user, then a corresponding conversion rule is determined based on a format to be converted, the target data is converted according to the conversion rule, corresponding analysis data is obtained, different analysis data can also correspond to different version information, and therefore after the analysis data is converted, version information of the analysis data can also be set, the analysis data of a target version is obtained, and a subsequent data analysis process is carried out.

In an optional embodiment, after the server establishes the connection with the target data source, the server may periodically collect the analysis data, and therefore, the analysis task of the server may be periodically executed according to the collection time and the collection frequency of the analysis data to dynamically obtain the corresponding data analysis result, for example, the update time of a set of analysis data is zero point every day, and the corresponding analysis task may be set to be executed every zero point, so that after the analysis data is updated, the updated analysis data is used for analysis to obtain the latest data analysis result, and the latest data analysis result is displayed to the user. By dynamically executing the analysis task, the data analysis result can be dynamically updated, and the user can conveniently check the data analysis result.

In the embodiment of the application, the data parser can be coupled to the server side, and can perform data analysis depending on the processing capability of the server side and perform data analysis depending on the processing capability of the computing device.

Specifically, after determining the analysis data and the configured data parser, the server determines a data analysis result based on the configured data parser and the analysis data in step 208. In one example, the server may parse the analysis data using a configured data parser to determine data analysis results. In another example, the server may perform an analysis process on the task using the computing device to obtain a data analysis result. Determining a target processing task based on the analysis data and the configured data parser; sending indication information to target computing equipment to deploy the target processing task to the target computing equipment for analysis processing; and receiving a corresponding data analysis result. The target computing device can be determined based on the task configuration information, then indication information is sent to the target computing device, the execution information is used for indicating the analysis processing of the executed task, the corresponding target computing device can call the configured data analyzer to analyze and process the analysis data, and then a corresponding data analysis result is obtained and returned to the server side. Therefore, the analysis data can be analyzed by using the computing power of the target computing server, decoupling between the data analyzer and the server is realized, the analysis process of the data analyzer does not depend on the server, and distributed computing can be realized.

Determining corresponding analysis data based on the task analysis information; the corresponding data analyzer is called and configured by utilizing the task configuration information, the data analysis process can be completed based on the data analyzer, the data analysis process can be completed more quickly, and the data processing efficiency is improved.

In a data planning scenario, an analysis task is usually combined with data to perform planning, prediction and other processing, so in the embodiment of the present application, the analysis task can be optimized according to a data analysis result corresponding to the analysis task and actual data in an actual application process. By analyzing the data and the actual data back flow, the corresponding data parser can be optimized to provide more accurate analysis results. In one example, in a scenario where a product sales is predicted, the analysis task may predict the predicted sales for the next week after the actual sales for the next week are actually obtained. The server side can optimize the data analyzer by combining the deviation between the predicted sales volume and the actual sales volume so as to predict the sales volume more accurately. In another example, in a scenario of planning a transportation route, a planned route from a departure place to an arrival place (data analysis result) can be analyzed by an analysis task, and then an actual route taken from the departure place to the arrival place is collected. And optimizing the data parser by combining the planned route and the actual route so as to more accurately plan the route.

The server side of the embodiment of the application can manage and schedule a plurality of tasks, so that each task can be distributed to the computing equipment through the task queue, and the parallel processing of the tasks is realized. The server can add the target processing tasks into the task queue, take out the target processing tasks according to the sequence in the task queue, and deploy the target processing tasks to corresponding target computing equipment for processing.

In an alternative embodiment, the server may determine the computing device corresponding to the target processing task according to the usage of the used memory and the available memory of the computing device. Specifically, the server side obtains running resource information of at least one computing device; and determining corresponding target computing equipment based on the running resource information and the occupation information of the target processing task on the running resources. The operating resource information may include processor resources and memory resource information, the memory resource information may include at least two of used memory information, available memory information, and total memory information, and the occupation information of the target processing task on the operating resources may be understood as the amount of memory that needs to be used when the target processing task operates. And determining the corresponding computing equipment as the target computing equipment according to the available memory amount of the computing equipment and the occupied memory amount of the target processing task. The target computing device can have the computing resources needed by the task, and meanwhile, the occupancy rate of the resources for the task to run relative to other computing devices is low.

In an optional embodiment, in order to enable the target processing task to operate normally, the server may further monitor an operating state of the target processing task in the target computing device, specifically, the server monitors the operating state of the target computing in the target computing device, and sends a retry instruction to the target computing device when the operating state of the target computing device on the target processing task is abnormal, so that the target computing device re-operates the target processing task. Determining alternative computing equipment under the condition that the running state of the target computing equipment on the target processing task is failure; deploying the target processing task to the alternative computing device for parsing by the alternative computing device.

After the computing device receives the processing task, the computing device can send the running state information of the processing task to the server at regular time, and the server can monitor the running state of the processing task according to the running state information of the processing task. When the target processing task is stopped unexpectedly (for example, when the computing device is in a failure), the operation state is confirmed to be abnormal, and a retry instruction is sent to the target computing device to enable the target computing device to operate again. And under the condition that the target processing task stops again in the target computing equipment or the task fails to run, confirming that the running state is failure, determining the alternative computing equipment, deploying the target processing task to the alternative computing equipment, and analyzing by utilizing the computing capacity of the alternative computing equipment. By re-running the computing equipment in the target computing equipment and deploying the target processing task to the alternative computing equipment for analysis, the analysis process of the data can be normally completed.

In an optional embodiment, the server may provide a third page, where the third page is used to display the data analysis result. A display request may be received; feeding back page data of a third page according to the display request so as to display a data analysis result of the target processing task in the third page. So that the user can view the data analysis result of the target processing task in the third page. The third page can display the data analysis result of the target processing task and also can provide the data analysis results of a plurality of processing tasks, so that the third page can be used for selecting the required processing task based on the requirement and checking the corresponding data analysis result. As shown in fig. 3C, a third page of an analysis task based on a logistics field can show data analysis results of a plurality of different types of logistics services, and a user can select a task of a corresponding type of logistics service based on a requirement and view the data analysis result of the corresponding task. Various controls based on the task can also be provided in the page, for example, an export control can provide a function of exporting the data analysis result; the overlay original routing control can provide a function of modifying the routing parameters of the task, so that the parameters of the task such as the routing and the like are modified, and analysis is performed based on the new routing; the generated examination order control provides an examination order generating function aiming at the task, and the examination order corresponding to the task can be generated by triggering the control. Thereby conveniently carrying out task examination and approval; the release scheme function can provide a release function aiming at the task, and after the data analysis result is checked and the task meets certain service requirements, the task can be released through the control, so that the task can be conveniently applied to the service of the corresponding scene.

The embodiment of the application further provides a fourth page for checking the established task, a user can request the fourth page through the task display request, and the corresponding server can send page data of the fourth page based on the task display request so as to display the established analysis task based on the fourth page, so that the user can conveniently check the established analysis task, and can conveniently and rapidly perform corresponding processing. An example schematic diagram of a fourth page, as shown in FIG. 3D, in which a user may view various information for a task and provide some controls to facilitate the editing process for the task. For example, the difference of the divided tasks can be compared through a comparison control, for example, the version difference of the task data corresponding to different tasks is compared, the operation difference of the two tasks is compared, and the like; the copying control is used for copying analysis tasks, and a user can conveniently copy relevant information of some tasks, so that the tasks are conveniently created; and the newly added control is used for newly adding an analysis task, and the control is triggered to jump to the first page to set task analysis information.

In the embodiment of the application, the analysis task can be determined according to the configured data analyzer and the analysis data, so that the analysis task can be analyzed, and the analysis result can be determined. In some alternative embodiments, in order to configure the analysis tasks more flexibly, the data resolvers, analysis data, and the like corresponding to each analysis task may be dynamically adjusted, for example, new analysis data is inserted into the analysis tasks, or other data resolvers are inserted into the analysis tasks, and in some examples, part of the analysis data may be deleted from the analysis tasks, or some data resolvers may be deleted from the tasks, so as to adjust the analysis tasks, so that the analysis tasks can be adjusted more flexibly.

The user can also modify the analysis task in the fourth page to more flexibly analyze and process the data. In one example, the user may modify the parameter configuration information in the analysis task to optimize the analysis task and obtain a more accurate analysis result. In another example, multiple versions of the data parser in the server may exist, and the user may further modify the parser information of the analysis task to invoke another version of the data parser to perform analysis processing on the data, so as to obtain a more accurate analysis result. In another example, the user may modify the analysis data in the analysis task to analyze other analysis data. By modifying the analysis data, the data can be analyzed more flexibly; by modifying the analyzer information and the data configuration information, different analyzers and data configuration information can be matched to perform data analysis, and the diversity of analysis tasks is improved by combining the analyzers and the data configuration information.

For example, as shown in fig. 3C, the third page in fig. 3C shows a plurality of different data analysis results, and when a certain data analysis result is no longer needed, the user may delete the analysis task corresponding to the data analysis result in the fourth page, so as to manage the data analysis result more flexibly.

In the embodiment of the application, in order to provide better analysis service, a fifth page is further provided, and the fifth page is used for receiving the data parser, so that the system can expand the data parser, is convenient to access various analysis tasks, and can be convenient to maintain the data parser, and better analysis service is provided for a user.

The server side can receive an uploading request and send page data of a fifth page according to the uploading request, wherein the fifth page comprises an uploading control; and receiving the uploaded data parser. The development user can develop the data analyzer and upload the data analyzer through a fifth page, so that the data analyzer is expanded in the system, wherein various parameters of data analysis can be set in the page, basic parameters such as the name, the type, the classification name, the operating environment and the description of the data analyzer can be set, after the setting of the basic parameters is completed, the data analyzer can be uploaded through the uploading control, the address of the data analyzer can be bound through the binding control, so that the uploading of the analyzer is realized, in addition, the compiling control can be provided for compiling the uploaded data analyzer and then packaging the compiled data analyzer and putting the compiled data analyzer into a server side and a computing device, and the use of the data analyzer is facilitated. And the fifth page also provides the setting of the operation parameters, and the user can set the operation parameters of the data parser in the operation parameter setting area, thereby facilitating the use of the data parser.

In an optional embodiment, a resolver list may be preset in the server, when the user requests the second page, the server may send the resolver list to the user as page data of the second page, and the user may search for a corresponding resolver in the resolver list in the second page to select the resolver. After the development user uploads the data parser through the fifth page, the server may adjust the parser list, and send the adjusted parser list to the user as page data of the second page to replace the old version of the parser list. When a new resolver is added, the server side can adjust the corresponding resolver list according to the new resolver, and sends the adjusted resolver list to the user, so that the user can conveniently select the resolver.

The data parser of the embodiment of the application may be generated by various algorithms, for example, some model algorithms such as machine learning algorithms, so that mathematical models obtained through processes such as model training may be uploaded as the data parser, where the corresponding mathematical models may be obtained through model algorithm training, the mathematical models are scientific or engineering models constructed by using mathematical logic methods and mathematical languages, the mathematical models are mathematical structures that are generally or approximately expressed by using mathematical languages with respect to the characteristic or quantity dependency relationships of some object systems, and the mathematical structures are pure relationship structures of some systems described by means of mathematical symbols. The mathematical model may be one or a set of algebraic, differential, integral or statistical equations, and combinations thereof, by which the interrelationships or causal relationships between the variables of the system are described quantitatively or qualitatively. In addition to mathematical models described by equations, there are also models described by other mathematical tools, such as algebra, geometry, topology, mathematical logic, etc. Mathematical models describe the behavior and characteristics of a system rather than the actual structure of the system.

In the embodiment of the application, each user can perform operations such as task creation, task running and result viewing through the system. Thus, in some optional embodiments, a user group may also be established for the user, such that the users within the user group may share data. Different users in the user group can enjoy different user permissions, and the task information associated with the user group is shared based on the different permissions. As in some examples, the creator (creating user) of each task has various rights to the task, and may set sharing rights for the task, such as rights to share the results of the task to other users, rights to share the task to other users, and so forth. In other examples, a user group leader may be set in the user group, and the user group leader has a higher authority and may configure the authority for other users, so that users in the same work group can perform work processing based on the system, and each user corresponding to the same task can complete work based on the system and the work group together, thereby providing convenience for the users. As shown in the processing diagram of the user group shown in fig. 4, the user group includes a user 1 and a user 2, after the user 1 creates a certain analysis task, corresponding task data can be obtained, and also, the analysis processing of the task can be performed to obtain a corresponding analysis result, the user 1 can open the authority to view the analysis result for the user 2, so that the second user can view the partial result, and in addition, based on the processing task generated by the task data, if the user 1 views the authority for the user 2, the user 2 can also view the processing task, and can use the processing task on the basis of obtaining the authority, for example, copy part of information of the processing task, so as to create a related task by himself. In addition, based on the task, analytics data, which may also be referred to as metadata, may be determined that user 1 has permission, while other users do not have permission, and so on. The method can be determined according to the scene requirements.

For example, in a logistics scenario, a certain part of a logistics provider has a demand of a logistics network planning class, and can register a user and create a user group, and each user of the department can join the user group, thereby sharing task-related information in the group. And then determining resolvers such as algorithms and models which need to be used according to the requirements of the logistics network planning class, wherein if the resolvers exist in the system, the resolvers can be called and configured, and if the resolvers do not exist in the system, the resolvers can be developed and uploaded to the system. After that, the task related information may be created, and then the data parser and the analysis data are configured to obtain a data analysis result, such as a data analysis result shown in fig. 3C, which may be used to view information of distribution, trunk lines, and the like in the logistics network.

On the basis of the above embodiment, the system may be hierarchically divided and may include: the system comprises a basic data layer, a data collection layer, a data management layer, a data processing layer and a data application layer.

The basic data layer can be connected with other data sources, the data warehouse can synchronize data of the data sources in an incremental or full-scale mode, and correspondingly, the data warehouse can synchronize the data to the data collection layer in the incremental or full-scale mode. The data collection layer can configure data based on tasks, including generating various versions of analysis data, executing various processes of the analysis data, such as version comparison, setting user rights and the like, and facilitating sharing of data and analysis results among users. The parser management layer may provide various data parsers, and related configurations for tasks, such as configuration of data, configuration of parsers, and the like. The data processing layer may perform data processing tasks including mining of data, feature extraction, and persistence processing for models, among others. The data application layer can provide various application functions based on data analysis, such as providing online services, facilitating the creation and operation of tasks by users, providing visual services of data, facilitating the creation, operation, result viewing and other management functions of tasks by users based on various interfaces, facilitating the uploading of a parser by development users, facilitating the expansion and maintenance of various functions of the system, providing a timed automatic operation function based on the setting of users, facilitating the viewing of corresponding analysis reports by users, providing some cooperative functions, such as approval and release based on tasks, and providing various data downloading functions, such as exporting data results and the like, thereby providing various convenient services for users.

The data processing layer may select data to form a corresponding analysis task based on the requirements, wherein data characteristics of the analysis data may be extracted, the analysis data may be data mined, and the data may be saved. And the data processing layer performs mining analysis on the acquired data and provides reference for a user to select analysis data. Specifically, the data processing layer is configured to perform feature analysis and data mining on data related to a user, and display a data mining result to the user (for example, data in a data selection area of a first page in fig. 3A is a corresponding data mining result), and the user may select the corresponding data mining result as task analysis information.

And a meta-information management layer can be further included, wherein the meta-information management layer can be understood as a database for storing data, and a meta-information management module stores the meta-data of the analysis data, the source data of the processing task and the meta-data of the analysis result. The meta-information management layer can provide data support for other layers in the server, such as providing data meta-data for the task management layer and the data processing layer, and providing task meta-data and result meta-data for the data application layer.

In the embodiment of the application, the data processing system provides functions of data development, data mining, data analysis and the like, can conveniently acquire data and assemble tasks, schedules the tasks to perform distributed analysis processing, and provides convenient and efficient data services for users.

On the basis of the above embodiment, as shown in fig. 5, the server includes, according to module division: the system comprises a service management module, a data management module, a resource management module and an interface management module.

The resource management module is used for managing the processing tasks and the processing process of the processing tasks by the computing equipment. The resource management module can flexibly manage computing resources, the resource management device can manage running resource information of a plurality of computing devices, such as memory and CPU (central processing unit) resources for storing each computing device, can place tasks into a task queue, determines target computing devices corresponding to processing tasks in the queue according to a scheduling strategy, schedules the processing tasks into the target computing devices for processing, and can keep high availability of the computing devices. After receiving the processing task, the target computing device feeds back the running state information of the processing task to the resource management module at regular time, the resource management module monitors the running of the processing task according to the running state information and can issue a retry instruction to enable the processing task to run again in the target computing device, and can also determine an alternative computing device and deploy the processing task which fails to run into the alternative computing device for computing. The resource management device may also manage information such as processing logs related to processing tasks. And the display of the task related data can be carried out by combining the interface management module, and a user can check the data analysis result.

The data management module is used for managing, processing, displaying and the like of the data. Specifically, the data management module may perform data docking with a service database of a user, acquire target data, convert and clean the target data according to a preset conversion rule, convert the target data into analysis data, and store the analysis data. The data management module can also store data such as data analysis results. The data management module may present the data to the user as page data for a page. The data management module can also provide data uploading service and data downloading service for the user.

The service management module is used for managing information, data, processing tasks and the like corresponding to the user. Specifically, the service management module may add the relevant user to the corresponding user group according to the operation of the user, so that the users in the group may share data. The service management module may also manage the processing task and the parser, and may present it to a corresponding user, for example, may present the processing task to the user as page data of a page.

The interface management module is used for managing the page so as to enable the server side to interact with the user. Pages such as a first page for managing a creation task, a second page for configuring a data parser, a fourth page for presenting a processing task, a third page for presenting analysis data and analysis results, a fifth page for uploading a data parser, and the like.

On the basis of the foregoing embodiments, the present application further provides a data processing method, which is applicable to a computing device, as shown in fig. 6, and includes:

step 602, a target processing task is obtained, the target processing task is determined according to the analysis data and the configured data parser, the analysis data is determined according to the task analysis information, and the configured data parser is configured according to the task configuration information.

And step 604, analyzing and processing the analysis data through a configured data analyzer, and determining a data analysis result.

And step 606, outputting a data analysis result.

In the embodiment of the application, after the computing device receives the target processing task, the computing device determines analysis processing and a configured data parser according to the target processing task, then performs data analysis on the analysis data by using the configured data parser, performs processing through an algorithm corresponding to the data parser to obtain a corresponding data analysis result, and feeds the data analysis result back to the server. The computing equipment can also report the running resource information to the server, so that the server can distribute processing tasks in a balanced manner based on the running resource information of each computing equipment, and the processing efficiency of the tasks is improved.

On the basis of the above embodiment, the application further provides a data processing method, which can conveniently set and check the tasks based on the page. As shown in fig. 7:

step 702, a task receipt page is provided.

The user side can send the task creating request to the server side, so that page data of a task receiving page returned by the server side based on the task creating request is received, the page data is analyzed, and the task receiving page is displayed.

In an optional embodiment, the task receiving page comprises a first page for determining task analysis information and a second page for determining task configuration information. The task configuration information comprises resolver information and parameter configuration information, the second page comprises a running control, and the running control is used for triggering indication information. Therefore, a user can set task analysis information on the first page to generate corresponding analysis tasks, and each analysis task can determine analysis data of a corresponding version. The task configuration information can be set through the second page, so that the configuration of the data analyzer and other running configurations of the task can be carried out, and the analysis processing of the task can be triggered through the running control of the second page.

In still other alternative embodiments, a task view request may be sent; and receiving page data of a fourth page returned by the task display request, displaying the fourth page, and checking the established analysis task. The user can check the established analysis task on the fourth page, edit the established analysis task based on the fourth page, and trigger a task establishment request based on the newly added control to display the first page.

Step 704, determining task related information according to an operation on the task receiving page, where the task related information includes task analysis information and task configuration information, the task analysis information is used to determine analysis data, and the task configuration information is used to configure a data parser so as to analyze and process the analysis data.

Step 706, sending the task related information.

In some optional embodiments of the present application, a presentation request is sent; receiving page data of a third page fed back according to the display request; and analyzing the page data, and displaying a data analysis result of the target processing task in the third page.

In other optional embodiments, a task presentation request is sent; receiving page data of a fourth page, wherein the page data of the fourth page is determined according to the task display request; and displaying the fourth page so as to display the created analysis tasks.

On the basis of the foregoing embodiments, the present application further provides a data processing method, which can conveniently extend and maintain a data parser of a system, as shown in fig. 8:

and step 802, providing a fifth page, wherein the fifth page comprises an uploading control.

And step 804, determining a data analyzer and corresponding parameter information according to the trigger of the uploading control, wherein the data analyzer is used for configuring according to the task configuration information and executing analysis processing of analysis data.

And 806, uploading the data parser and the corresponding parameter information.

In order to provide better analysis service, a fifth page is also provided, and the fifth page is used for receiving the data parser, so that the system can expand the data parser, is convenient to access various analysis tasks, and can be convenient to maintain the data parser, and better analysis service is provided for users.

On the basis of the above embodiments, the embodiments of the present application further provide a data processing method, which can create various tasks and execute the processing of the tasks, and is convenient for users to use.

Referring to fig. 9, a flowchart illustrating steps of a data processing method according to an embodiment of the present application is shown.

Step 902, receiving task related information through a task receiving page, where the task related information includes task analysis information and task configuration information.

In an optional embodiment, based on a task creation request, page data of a task receiving page is sent to display the task receiving page; task related information based on the task receiving page is received. The task receiving page comprises a first page used for determining task analysis information and a second page used for determining task configuration information. The task configuration information comprises resolver information and parameter configuration information, the second page comprises a running control, and the running control is used for triggering indication information.

In another alternative embodiment, a task view request is received; and sending page data of a fourth page based on the task display request so as to display the established analysis task based on the fourth page.

And 904, determining analysis data according to the task analysis information, and generating a corresponding analysis task.

Target data can be obtained from a target data source according to the task analysis information; and converting the target data into analysis data of a target version according to a preset conversion rule.

Step 906, determining and configuring a corresponding data parser based on the task configuration information.

Step 908, determining a target processing task based on the analysis task and the configured data parser.

Step 910, adding the target processing task into a task queue;

step 912, deploying the target processing task according to the sequence of the target processing task in the task queue.

And step 914, sending indication information to the target computing device, so as to deploy the target processing task to the target computing device for analysis processing.

Wherein, the running resource information of at least one computing device can be obtained; and determining corresponding target computing equipment based on the running resource information and the occupation information of the target processing task on the running resources.

The embodiment of the application can also monitor the running state of the target processing task in the target computing equipment; and sending a retry instruction to the target computing device to enable the target computing device to rerun the target processing task when the running state of the target computing device to the target processing task is abnormal.

In another optional embodiment, when the running state of the target processing task by the target computing device is a failure, determining an alternative computing device; deploying the target processing task to the alternative computing device for parsing by the alternative computing device.

At step 916, a corresponding data analysis result is received.

In an optional embodiment, a third page is provided, and the third page is used for displaying the data analysis result.

At step 918, a display request is received.

Step 920, feeding back page data of a third page according to the display request, so as to display a data analysis result of the target processing task in the third page.

The steps in the embodiments of the present application are similar to those in the embodiments described above, and specific reference may be made to the embodiments described above.

Therefore, the page can be provided for the user, the user can conveniently set information and check the task, and the analysis task and the data analyzer can be set based on the data fed back by the page, so that the corresponding processing task is generated and deployed into the computing equipment and is processed by the distributed computing equipment, the computing resource can be effectively used, and the processing efficiency is improved.

After the target processing task and the target computing device are determined, the target processing task is deployed to the target computing device, and the target processing task is processed by utilizing the computing capability of the computing device to determine a data analysis result. In the embodiment of the application, the corresponding analysis data is determined by using the task analysis information; the corresponding data analyzer is called and configured by using the task configuration information, the data analysis process can be completed by using the data analyzer of the server side, and the data analysis process can be completed more quickly.

The embodiment of the application realizes the communication between the processing system of each scene and the data processing system. And synchronizing data in other systems into the database through the database timing. Then, the data are synchronized from the database to the server side at regular time, and a data processing flow can be triggered through asynchronous messages, so that the data are converted into a uniform data model format, and the use and display of subsequent data are facilitated.

The user can receive operation through the browser display surface through the page, so that the user can use the service of the system on any computer at any time and any place as long as the user has network access authority.

The embodiment of the application manages the data and the computing resources, and the computing node (computing equipment) reports the state of the available computing resources (such as a memory and a CPU) to the platform. The server-side can schedule the operation tasks to the appointed computing nodes for operation according to a certain scheduling strategy. And meanwhile, the running state of the system is monitored, and when the task fails, the task is retried. Meanwhile, under the condition of multiple users and multiple tasks, the computing nodes can be expanded horizontally, and the server side can manage the newly added computing nodes and distribute the tasks to all the machines in a balanced manner.

A development user can develop an algorithm model for solving the service scenario by using a corresponding programming language, and a corresponding data parser is generated and uploaded for the user to use. Therefore, aiming at some new analysis requirements, the data analyzer can be uploaded correspondingly without other improvements, the realization speed of new functions is increased, and the efficiency is improved.

It should be noted that, for simplicity of description, the method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the embodiments are not limited by the order of acts described, as some steps may occur in other orders or concurrently depending on the embodiments. Further, those skilled in the art will also appreciate that the embodiments described in the specification are presently preferred and that no particular act is required of the embodiments of the application.

On the basis of the foregoing embodiment, this embodiment further provides a data processing apparatus, as shown in fig. 10, which may specifically include the following modules:

the task receiving module 1002 is configured to receive task related information, where the task related information includes task analysis information and task configuration information.

And a data determining module 1004, configured to determine analysis data according to the task analysis information.

And the parser configuration module 1006 is configured to determine and configure a corresponding data parser based on the task configuration information.

A result determination module 1008 for determining a data analysis result based on the configured data parser and the analysis data.

In conclusion, after receiving the task related information, generating analysis data according to the task analysis information of the task related information; according to the task configuration information of the task related information, the corresponding data parser in the server side can be called, the called data parser is configured, the data analysis process can be completed more quickly through the configured data parser, and the data processing efficiency is improved.

Wherein the result determination module 1008 is configured to determine a target processing task based on the analysis data and the configured data parser; sending indication information to target computing equipment to deploy the target processing task to the target computing equipment for analysis processing; and receiving a corresponding data analysis result.

The device also includes: the queue processing module is used for adding the target processing task into a task queue; and deploying the target processing tasks according to the sequence of the target processing tasks in the task queue.

The resource processing module is used for acquiring running resource information of at least one computing device; and determining corresponding target computing equipment based on the running resource information and the occupation information of the target processing task on the running resources.

The monitoring module is used for monitoring the running state of the target processing task in the target computing equipment; and sending a retry instruction to the target computing device to enable the target computing device to rerun the target processing task when the running state of the target computing device to the target processing task is abnormal.

The result determination module 1008 is further configured to determine an alternative computing device if the running state of the target processing task by the target computing device is a failure; deploying the target processing task to the alternative computing device for parsing by the alternative computing device.

And the user group establishing module is used for establishing a user group and setting the user permission of the user in the user group so as to share the processing task related to the user group according to the user permission.

The data determining module 1004 is configured to obtain target data from a target data source according to the task analysis information; and converting the target data into analysis data of a target version according to a preset conversion rule.

The device also includes: the page providing module is used for sending page data of a task receiving page based on a task creating request so as to display the task receiving page; task related information based on the task receiving page is received. The task receiving page comprises a first page used for determining task analysis information and a second page used for determining task configuration information. The task configuration information comprises resolver information and parameter configuration information, the second page comprises a running control, and the running control is used for triggering indication information.

The page providing module is further used for providing a third page, and the third page is used for displaying a data analysis result; receiving a display request; feeding back page data of a third page according to the display request so as to display a data analysis result of the target processing task in the third page.

The page providing module is also used for receiving a task viewing request; and sending page data of a fourth page based on the task display request so as to display the established analysis task based on the fourth page.

The page providing module is further configured to provide a fifth page, where the fifth page is used to receive a data parser; receiving an uploading request, and sending page data of a fifth page according to the uploading request, wherein the fifth page comprises an uploading control; and receiving the uploaded data parser.

In the embodiment of the application, a user can send a task creating request through a user side, a server side returns a first page and a second page to the user side according to the task creating request, the user determines task analysis information in the first page, determines task configuration information in the second page, and then the user side sends task related information to the server side. After receiving the task related information, the server generates analysis data according to the task analysis information of the task related information, calls a corresponding data parser in the server according to the task configuration information of the task related information, and configures the called data parser, so that the server can complete analysis of the analysis data. And then packaging the data into a target processing task by using the configured data parser and the analysis data, and determining the preferred computing equipment as the target computing equipment according to the running resource information of the plurality of computing equipment.

On the basis of the foregoing embodiment, this embodiment further provides a data processing apparatus, as shown in fig. 11, which may specifically include the following modules:

a receiving module 1102, configured to obtain a target processing task, where the target processing task is determined according to analysis data and a configured data parser, the analysis data is determined according to task analysis information, and the configured data parser is configured according to task configuration information;

the analysis module 1104 is used for analyzing and processing the analysis data through a configured data analyzer to determine a data analysis result;

a result output module 1106, configured to output the data analysis result.

The computing equipment calls the configured data analyzer to analyze and process the analysis data to obtain a data analysis result, so that distributed computing resources can be fully utilized, and the processing efficiency is improved.

On the basis of the foregoing embodiment, this embodiment further provides a data processing apparatus, as shown in fig. 12, which may specifically include the following modules:

a page providing module 1202 for providing a task receiving page.

A task determining module 1204, configured to determine task-related information according to an operation on the task receiving page, where the task-related information includes task analysis information and task configuration information, the task analysis information is used to determine analysis data, and the task configuration information is used to configure a data parser, so as to analyze and process the analysis data.

A sending module 1206, configured to output the task related information.

The page providing module 1202 is further configured to send a task creating request to obtain page data of the task receiving page.

The task receiving page comprises a first page used for determining task analysis information and a second page used for determining task configuration information. The task configuration information comprises resolver information and parameter configuration information, the second page comprises a running control, and the running control is used for triggering indication information.

The page providing module 1202 is further configured to send a display request; receiving page data of a third page fed back according to the display request; and analyzing the page data, and displaying a data analysis result of the target processing task in the third page.

The page providing module 1202 is further configured to send a task display request; receiving page data of a fourth page, wherein the page data of the fourth page is determined according to the task display request; and displaying the fourth page so as to display the created analysis tasks.

Various pages can be provided, user operation is facilitated, and therefore a user can conveniently check and set related information of tasks and check data analysis results based on the pages, operation efficiency is improved, and user experience is improved.

On the basis of the foregoing embodiment, this embodiment further provides a data processing apparatus, as shown in fig. 13, which may specifically include the following modules:

a providing module 1302, configured to provide a fifth page, where the fifth page includes an upload control.

The analyzer uploading module 1024 is used for determining a data analyzer and corresponding parameter information according to the triggering of the uploading control; and uploading the data analyzer and the corresponding parameter information, wherein the data analyzer is used for configuring according to the task configuration information and executing analysis processing of analysis data.

The present application further provides a non-transitory, readable storage medium, where one or more modules (programs) are stored, and when the one or more modules are applied to a device, the device may execute instructions (instructions) of method steps in this application.

Embodiments of the present application provide one or more machine-readable media having instructions stored thereon, which when executed by one or more processors, cause an electronic device to perform the methods as described in one or more of the above embodiments. In the embodiment of the application, the electronic device includes a server, a terminal device and other devices.

Embodiments of the present disclosure may be implemented as an apparatus, which may comprise a server (cluster), a terminal, etc., electronic device, using any suitable hardware, firmware, software, or any combination thereof, in a desired configuration. Fig. 14 schematically illustrates an example apparatus 1400 that can be used to implement various embodiments described herein.

For one embodiment, fig. 14 illustrates an exemplary apparatus 1400 having one or more processors 1402, a control module (chipset) 1404 coupled to at least one of the processor(s) 1402, a memory 1406 coupled to the control module 1404, a non-volatile memory (NVM)/storage 1408 coupled to the control module 1404, one or more input/output devices 1410 coupled to the control module 1404, and a network interface 1412 coupled to the control module 1404.

Processor 1402 may include one or more single-core or multi-core processors, and processor 1402 may include any combination of general-purpose or special-purpose processors (e.g., graphics processors, application processors, baseband processors, etc.). In some embodiments, the apparatus 1400 can be used as a server, a terminal, or other devices described in this embodiment.

In some embodiments, apparatus 1400 may include one or more computer-readable media (e.g., memory 1406 or NVM/storage 1408) having instructions 1414 and one or more processors 1402 in combination with the one or more computer-readable media and configured to execute instructions 1414 to implement modules to perform the actions described in this disclosure.

For one embodiment, the control module 1404 may include any suitable interface controller to provide any suitable interface to at least one of the processor(s) 1402 and/or any suitable device or component in communication with the control module 1404.

The control module 1404 may include a memory controller module to provide an interface to the memory 1406. The memory controller module may be a hardware module, a software module, and/or a firmware module.

The memory 1406 may be used, for example, to load and store data and/or instructions 1414 for the apparatus 1400. For one embodiment, memory 1406 may comprise any suitable volatile memory, such as suitable DRAM. In some embodiments, the memory 1406 may comprise double data rate type four synchronous dynamic random access memory (DDR4 SDRAM).

For one embodiment, control module 1404 may include one or more input/output controllers to provide an interface to NVM/storage 1408 and input/output device(s) 1410.

For example, NVM/storage 1408 may be used to store data and/or instructions 1414. NVM/storage 1408 may include any suitable non-volatile memory (e.g., flash memory) and/or may include any suitable non-volatile storage device(s) (e.g., one or more Hard Disk Drives (HDDs), one or more Compact Disk (CD) drives, and/or one or more Digital Versatile Disk (DVD) drives).

The NVM/storage 1408 may be a storage resource that is part of the device on which the apparatus 1400 is installed, or it may be accessible by the device and need not be part of the device. For example, NVM/storage 1408 may be accessible over a network via input/output device(s) 1410.

Input/output device(s) 1410 may provide an interface for apparatus 1400 to communicate with any other suitable device, input/output devices 1410 may include communication components, audio components, sensor components, and so forth. Network interface 1412 may provide an interface for device 1400 to communicate over one or more networks, and device 1400 may wirelessly communicate with one or more components of a wireless network according to any of one or more wireless network standards and/or protocols, such as access to a communication standard-based wireless network, e.g., WiFi, 2G, 3G, 4G, 5G, etc., or a combination thereof.

For one embodiment, at least one of the processor(s) 1402 may be packaged together with logic for one or more controller(s) (e.g., memory controller module) of control module 1404. For one embodiment, at least one of the processor(s) 1402 may be packaged together with logic for one or more controller(s) of control module 1404 to form a System In Package (SiP). For one embodiment, at least one of the processor(s) 1402 may be integrated on the same die with logic for one or more controller(s) of the control module 1404. For one embodiment, at least one of the processor(s) 1402 may be integrated on the same die with logic for one or more controller(s) of control module 1404 to form a system on chip (SoC).

In various embodiments, the apparatus 1400 may be, but is not limited to being: a server, a desktop computing device, or a mobile computing device (e.g., a laptop computing device, a handheld computing device, a tablet, a netbook, etc.), among other terminal devices. In various embodiments, the apparatus 1400 may have more or fewer components and/or different architectures. For example, in some embodiments, device 1400 includes one or more cameras, a keyboard, a Liquid Crystal Display (LCD) screen (including a touch screen display), a non-volatile memory port, multiple antennas, a graphics chip, an Application Specific Integrated Circuit (ASIC), and speakers.

The detection device can adopt a main control chip as a processor or a control module, sensor data, position information and the like are stored in a memory or an NVM/storage device, a sensor group can be used as an input/output device, and a communication interface can comprise a network interface.

An embodiment of the present application further provides an electronic device, including: a processor; and a memory having executable code stored thereon that, when executed, causes the processor to perform a method as described in one or more of the embodiments of the application.

Embodiments of the present application also provide one or more machine-readable media having executable code stored thereon that, when executed, cause a processor to perform a method as described in one or more of the embodiments of the present application.

For the device embodiment, since it is basically similar to the method embodiment, the description is simple, and for the relevant points, refer to the partial description of the method embodiment.

The embodiments in the present specification are described in a progressive manner, each embodiment focuses on differences from other embodiments, and the same and similar parts among the embodiments are referred to each other.

Embodiments of the present application are described with reference to flowchart illustrations and/or block diagrams of methods, terminal devices (systems), and computer program products according to embodiments of the application. It will be understood that each flow and/or block of the flow diagrams and/or block diagrams, and combinations of flows and/or blocks in the flow diagrams and/or block diagrams, can be implemented by computer program instructions. These computer program instructions may be provided to a processor of a general purpose computer, special purpose computer, embedded processor, or other programmable data processing terminal to produce a machine, such that the instructions, which execute via the processor of the computer or other programmable data processing terminal, create means for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing terminal to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.

These computer program instructions may also be loaded onto a computer or other programmable data processing terminal to cause a series of operational steps to be performed on the computer or other programmable terminal to produce a computer implemented process such that the instructions which execute on the computer or other programmable terminal provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.

While preferred embodiments of the present application have been described, additional variations and modifications of these embodiments may occur to those skilled in the art once they learn of the basic inventive concepts. Therefore, it is intended that the appended claims be interpreted as including the preferred embodiment and all such alterations and modifications as fall within the true scope of the embodiments of the application.

Finally, it should also be noted that, herein, relational terms such as first and second, and the like may be used solely to distinguish one entity or action from another entity or action without necessarily requiring or implying any actual such relationship or order between such entities or actions. Also, the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or terminal that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or terminal. Without further limitation, an element defined by the phrase "comprising an … …" does not exclude the presence of other like elements in a process, method, article, or terminal that comprises the element.

The foregoing detailed description has provided a data processing method, a data processing apparatus, an electronic device, and a storage medium, and the principles and embodiments of the present application are described herein using specific examples, which are only used to help understand the method and the core ideas of the present application; meanwhile, for a person skilled in the art, according to the idea of the present application, there may be variations in the specific embodiments and the application scope, and in summary, the content of the present specification should not be construed as a limitation to the present application.

Claims

1. A method of data processing, the method comprising:

receiving task related information, wherein the task related information comprises task analysis information and task configuration information;

determining analysis data according to the task analysis information;

determining and configuring a corresponding data analyzer based on the task configuration information;

determining a data analysis result based on the configured data parser and the analysis data.

2. The method of claim 1, wherein determining a data analysis result based on the configured data parser and the analysis data comprises:

determining a target processing task based on the analysis data and the configured data parser;

sending indication information to target computing equipment to deploy the target processing task to the target computing equipment for analysis processing;

and receiving a corresponding data analysis result.

3. The method of claim 2, further comprising:

adding the target processing task into a task queue;

and deploying the target processing tasks according to the sequence of the target processing tasks in the task queue.

4. The method of claim 3, further comprising:

acquiring running resource information of at least one computing device;

and determining corresponding target computing equipment based on the running resource information and the occupation information of the target processing task on the running resources.

5. The method of claims 2-4, further comprising:

monitoring the running state of the target processing task in the target computing equipment;

and sending a retry instruction to the target computing device to enable the target computing device to rerun the target processing task when the running state of the target computing device to the target processing task is abnormal.

6. The method of claim 5, further comprising:

determining alternative computing equipment under the condition that the running state of the target computing equipment on the target processing task is failure;

deploying the target processing task to the alternative computing device for parsing by the alternative computing device.

7. The method of claim 1, further comprising:

and establishing a user group and setting the user permission of the user in the user group so as to share the processing task related to the user group according to the user permission.

8. The method of claim 1, wherein said determining analysis data from said task analysis information comprises:

acquiring target data from a target data source according to the task analysis information;

and converting the target data into analysis data of a target version according to a preset conversion rule.

9. The method of claim 1, further comprising:

based on a task creation request, sending page data of a task receiving page to display the task receiving page;

task related information based on the task receiving page is received.

10. The method of claim 1, wherein the task receiving page comprises a first page for determining task analysis information and a second page for determining task configuration information.

11. The method of claim 1, wherein the task configuration information comprises parser information and parameter configuration information, and wherein the second page comprises a launch control, and wherein the launch control is used for triggering the indication information.

12. The method of claim 1, further comprising:

providing a third page, wherein the third page is used for displaying a data analysis result;

receiving a display request;

feeding back page data of a third page according to the display request so as to display a data analysis result of the target processing task in the third page.

13. The method of claim 2, further comprising:

receiving a task viewing request;

and sending page data of a fourth page based on the task display request so as to display the established analysis task based on the fourth page.

14. The method of claim 1, further comprising:

providing a fifth page, wherein the fifth page is used for receiving a data parser;

receiving an uploading request, and sending page data of a fifth page according to the uploading request, wherein the fifth page comprises an uploading control;

and receiving the uploaded data parser.

15. A data processing method, comprising:

acquiring a target processing task, wherein the target processing task is determined according to analysis data and a configured data analyzer, the analysis data is determined according to task analysis information, and the configured data analyzer is configured according to task configuration information;

analyzing and processing the analysis data through a configured data analyzer to determine a data analysis result;

and outputting the data analysis result.

16. A data processing method, comprising:

providing a task receiving page;

determining task related information according to operation on the task receiving page, wherein the task related information comprises task analysis information and task configuration information, the task analysis information is used for determining analysis data, and the task configuration information is used for configuring a data analyzer so as to analyze and process the analysis data;

and sending the task related information.

17. The method of claim 16, further comprising:

and sending a task creating request to acquire page data of the task receiving page.

18. The method of claim 17, wherein the task receiving page comprises a first page for determining task analysis information and a second page for determining task configuration information.

19. The method of claim 18, wherein the task configuration information comprises parser information and parameter configuration information, and wherein the second page comprises a launch control for triggering the indication information.

20. The method of claim 16, further comprising:

sending a display request;

receiving page data of a third page fed back according to the display request;

and analyzing the page data, and displaying a data analysis result of the target processing task in the third page.

21. The method of claim 16, further comprising:

sending a task display request;

receiving page data of a fourth page, wherein the page data of the fourth page is determined according to the task display request;

and displaying the fourth page so as to display the created analysis tasks.

22. A data processing method, comprising:

providing a fifth page, wherein the fifth page comprises an uploading control;

determining a data analyzer and corresponding parameter information according to the triggering of the uploading control, wherein the data analyzer is used for configuring according to task configuration information and executing analysis processing of analysis data;

and uploading the data parser and the corresponding parameter information.

23. A data processing apparatus, comprising:

the task receiving module is used for receiving task related information, and the task related information comprises task analysis information and task configuration information;

the data determining module is used for determining analysis data according to the task analysis information;

the analyzer configuration module is used for determining and configuring a corresponding data analyzer based on the task configuration information;

and the result determining module is used for determining a data analysis result based on the configured data parser and the analysis data.

24. A data processing apparatus, comprising:

the receiving module is used for acquiring a target processing task, the target processing task is determined according to analysis data and a configured data analyzer, the analysis data is determined according to task analysis information, and the configured data analyzer is configured according to task configuration information;

the analysis module is used for analyzing and processing the analysis data through a configured data analyzer and determining a data analysis result;

and the result output module is used for outputting the data analysis result.

25. A data processing apparatus, comprising:

the page providing module is used for providing a task receiving page;

the task determining module is used for determining task related information according to operation on the task receiving page, wherein the task related information comprises task analysis information and task configuration information, the task analysis information is used for determining analysis data, and the task configuration information is used for configuring a data analyzer so as to analyze and process the analysis data;

and the sending module is used for outputting the task related information.

26. A data processing apparatus, comprising:

the providing module is used for providing a fifth page, and the fifth page comprises an uploading control;

the analyzer uploading module is used for determining a data analyzer and corresponding parameter information according to the triggering of the uploading control; and uploading the data analyzer and the corresponding parameter information, wherein the data analyzer is used for configuring according to the task configuration information and executing analysis processing of analysis data.

27. An electronic device, comprising: a processor; and

memory having stored thereon executable code which, when executed, causes the processor to perform the method of one or more of claims 1-14.

28. One or more machine-readable media having executable code stored thereon that, when executed, causes a processor to perform the method of one or more of claims 1-14.

29. An electronic device, comprising: a processor; and

a memory having executable code stored thereon that, when executed, causes the processor to perform the method of claim 15.

30. One or more machine-readable media having executable code stored thereon that, when executed, causes a processor to perform the method of claim 15.

31. An electronic device, comprising: a processor; and

memory having stored thereon executable code which, when executed, causes the processor to perform the method of one or more of claims 16-21.

32. One or more machine-readable media having executable code stored thereon that, when executed, causes a processor to perform the method of one or more of claims 16-21.

33. An electronic device, comprising: a processor; and

a memory having executable code stored thereon that, when executed, causes the processor to perform the method of claim 22.

34. One or more machine-readable media having executable code stored thereon that, when executed, causes a processor to perform the method of claim 22.