CN117215700A

CN117215700A - Parallel data processing method, device, electronic equipment and storage medium

Info

Publication number: CN117215700A
Application number: CN202311205152.0A
Authority: CN
Inventors: 张威
Original assignee: Bank of China Ltd
Current assignee: Bank of China Ltd
Priority date: 2023-09-18
Filing date: 2023-09-18
Publication date: 2023-12-12

Abstract

The application provides a parallel data processing method, a device, electronic equipment and a storage medium, which can be used in the field of distributed frameworks. The method comprises the following steps: the parallel processing process responds to data processing operation triggered by a target user aiming at a data processing interface, and a data set generated by the target user is obtained, wherein the data set comprises a plurality of pieces of data to be processed; performing data slicing processing on the data set to obtain a plurality of data slicing sets; distributing the plurality of data fragment sets to corresponding data processing processes, so that each data processing process performs data processing on the received data fragment sets to generate corresponding data processing results; and receiving data processing results fed back by each data processing process. The method improves the data processing efficiency.

Description

Parallel data processing method, device, electronic equipment and storage medium

Technical Field

The present application relates to the field of distributed frameworks, and in particular, to a parallel data processing method, apparatus, electronic device, and storage medium.

Background

The data processing process generally refers to distributing data to be processed to an application program, and performing data processing on the data by the application program to generate a data processing result. In practical application, the quantity of data to be processed is huge and the data types are different. The application needs to process each piece of data separately, resulting in less efficient data processing.

Disclosure of Invention

The application provides a parallel data processing method, a device, electronic equipment and a storage medium, which are used for solving the technical problem that when more data needs to be processed, an application program needs to process each piece of data respectively, so that the data processing efficiency is low.

In a first aspect, the present application provides a parallel data processing method, including:

the parallel processing process responds to data processing operation triggered by a target user aiming at a data processing interface, and a data set generated by the target user is obtained, wherein the data set comprises a plurality of pieces of data to be processed;

performing data slicing processing on the data set to obtain a plurality of data slicing sets;

distributing the plurality of data fragment sets to corresponding data processing processes, so that each data processing process performs data processing on the received data fragment sets to generate corresponding data processing results;

and receiving data processing results fed back by each data processing process.

In a second aspect, the present application provides a parallel data processing apparatus comprising:

the data processing system comprises a set acquisition unit, a processing unit and a processing unit, wherein the set acquisition unit is used for responding to data processing operation triggered by a target user for a data processing interface to acquire a data set generated by the target user, and the data set comprises a plurality of pieces of data to be processed;

The data slicing unit is used for performing data slicing processing on the data set to obtain a plurality of data slicing sets;

the process processing unit is used for distributing the plurality of data fragment sets to corresponding data processing processes so that each data processing process can process the received data fragment sets to generate corresponding data processing results;

and the result acquisition unit is used for receiving the data processing results fed back by the data processing processes respectively.

In a third aspect, the present application provides an electronic device comprising: a processor, and a memory communicatively coupled to the processor;

the memory stores computer-executable instructions;

the processor executes computer-executable instructions stored in the memory to implement the method as described in the first aspect and the various possibilities of the first aspect.

In a fourth aspect, the present application provides a computer-readable storage medium having stored therein computer-executable instructions which, when executed by a processor, are adapted to carry out the method of the first aspect and the various possibilities of the first aspect.

In a fifth aspect, the application provides a computer program product comprising a computer program which, when executed by a processor, implements the steps of the first aspect and the various possible methods involved in the first aspect.

The parallel processing program provided by the application can be configured in the electronic equipment. The parallel processing program may obtain a target user-generated data set, which may include a plurality of pieces of data to be processed, in response to a data processing operation performed by the target user for the data processing interface. And performing data slicing processing on the data set to obtain a plurality of data slicing sets. The data volume in the data slicing and centralizing obtained after the data set is sliced is smaller, and the parallel processing of the aging data can be realized without excessively high authority. And distributing the plurality of data fragment sets to corresponding data processing processes, so that the data fragment sets received by each data processing process are subjected to data processing, and a data processing result is generated. The data slicing sets are processed by the corresponding data processing processes, so that parallel processing of the data slicing sets is realized, and the processing efficiency of the data slicing can be improved through parallel processing. And further, data processing results fed back by the data processing processes respectively can be received. By means of data slicing and parallel execution of data processing processes, rapid processing of multiple pieces of data to be processed can be achieved, system safety is guaranteed, and meanwhile data processing efficiency is improved.

Drawings

The accompanying drawings, which are incorporated in and constitute a part of this specification, illustrate embodiments consistent with the application and together with the description, serve to explain the principles of the application.

FIG. 1 is a flow chart of a parallel data processing method according to an embodiment of the present application;

FIG. 2 is an exemplary diagram of a data processing interface provided by an embodiment of the present application;

FIG. 3 is an exemplary diagram of an editing interface for slicing logic provided by an embodiment of the present application;

FIG. 4 is a flow chart of yet another embodiment of a parallel data processing method according to an embodiment of the present application;

FIG. 5 is an application example diagram of a parallel data processing method according to an embodiment of the present application;

FIG. 6 is a schematic diagram of a parallel data processing apparatus according to an embodiment of the present application;

fig. 7 is a block diagram of an electronic device with a parallel data processing method according to an embodiment of the present application.

Specific embodiments of the present application have been shown by way of the above drawings and will be described in more detail below. The drawings and the written description are not intended to limit the scope of the inventive concepts in any way, but rather to illustrate the inventive concepts to those skilled in the art by reference to the specific embodiments.

Detailed Description

Reference will now be made in detail to exemplary embodiments, examples of which are illustrated in the accompanying drawings. When the following description refers to the accompanying drawings, the same numbers in different drawings refer to the same or similar elements, unless otherwise indicated. The implementations described in the following exemplary examples do not represent all implementations consistent with the application. Rather, they are merely examples of apparatus and methods consistent with aspects of the application as detailed in the accompanying claims.

It should be noted that, the user information (including but not limited to user equipment information, user personal information, etc.) and the data (including but not limited to data for analysis, stored data, presented data, etc.) related to the present application are information and data authorized by the user or fully authorized by each party, and the collection, use and processing of the related data need to comply with related laws and regulations and standards, and provide corresponding operation entries for the user to select authorization or rejection.

It should be noted that the parallel data processing method, apparatus, electronic device and storage medium provided by the present application may be used in the field of distributed frameworks, and may also be used in any field other than the field of distributed frameworks.

In the related art, data processing generally develops an application program for data of a type according to the type of the data. And sending the data to be processed to an application program corresponding to the data type of the data to be processed, and running the application program to obtain a data processing result. However, in practical applications, the amount of data to be processed is large. When the data is processed by the application program for data development, the application program needs to wait for processing each piece of data one by one, and the processing efficiency of the data is low.

In order to solve the above-described problems, embodiments of the present application contemplate using a parallel processing framework by which a large amount of data is processed in parallel. However, when a large number of data line processing frameworks are used at present, a higher authority needs to be set for users. However, for the data processing scenario with higher security requirements, for example, the scenario of banks, financial institutions, etc., setting higher authority for the user may have a greater risk of data leakage, and reduce the security of data processing. In order to realize parallel processing of data, the safety of the system is ensured. In the embodiment, the data to be processed is sliced by considering the use of data slicing, so that the data quantity required to be processed by each slicing is reduced, the parallel processing of the data can be completed without permission, the safety of a system is ensured, and the data processing efficiency is improved.

In the embodiment of the application, the parallel processing program can be configured in the electronic equipment. The parallel processing program may obtain a target user-generated data set, which may include a plurality of pieces of data to be processed, in response to a data processing operation performed by the target user for the data processing interface. And performing data slicing processing on the data set to obtain a plurality of data slicing sets. The data volume in the data slicing and centralizing obtained after the data set is sliced is smaller, and the parallel processing of the aging data can be realized without excessively high authority. And distributing the plurality of data fragment sets to corresponding data processing processes, so that the data fragment sets received by each data processing process are subjected to data processing, and a data processing result is generated. The data slicing sets are processed by the corresponding data processing processes, so that parallel processing of the data slicing sets is realized, and the processing efficiency of the data slicing can be improved through parallel processing. And further, data processing results fed back by the data processing processes respectively can be received. By means of data slicing and parallel execution of data processing processes, rapid processing of multiple pieces of data to be processed can be achieved, system safety is guaranteed, and meanwhile data processing efficiency is improved.

The application provides a parallel data processing method, which aims to solve the technical problems in the prior art.

The following describes the technical scheme of the present application and how the technical scheme of the present application solves the above technical problems in detail with specific embodiments. The following embodiments may be combined with each other, and the same or similar concepts or processes may not be described in detail in some embodiments. Embodiments of the present application will be described below with reference to the accompanying drawings.

For easy understanding, as shown in fig. 1, a flowchart of a parallel data processing method according to an embodiment of the present application may include:

step 101, a parallel processing process responds to a data processing operation triggered by a target user aiming at a data processing interface to obtain a data set generated by the target user, wherein the data set comprises a plurality of pieces of data to be processed.

The parallel processing process can be used for realizing the parallel data processing method provided by the embodiment of the application, and can be used for executing the creation and the cancellation of the data processing process. As an alternative design, the parallel processing process may be a Master process, and the data processing process may be a Slave process.

A data set may refer to a set of large amounts of data. The data set may be any one of a data table, a data packet or a database table, and of course, the data set may be other data structures for storing a large amount of data, and the specific data structure of the data set is not limited in this embodiment. The data set may include a plurality of pieces of data to be processed, each piece of data to be processed may include at least one data element, and the meaning of the data referred to by each data element may be set according to the usage scenario of the data. For example, in a fund scenario, a piece of data may include data elements such as user information, fund identification, fund hold, net daily value, and the like. The data element may be the smallest data unit in a piece of data, and may be used to store a piece of information, for example, when the values corresponding to different keys are stored in a key value mode, the data element may be the value corresponding to any key, and the key may refer to the actual meaning of the data. The data formats of the plurality of pieces of data to be processed in the data set can be the same or different.

For ease of understanding, fig. 2 shows an example diagram of a data manipulation interface, which data manipulation interface 200 may include a data selection area 201 and a data input area 202.

The data selection area 201 may include, for example, a prompt text 2011 corresponding to "please select a data set" and a selection control 2012 of the data set, and a confirm button 2013. The user may be detected to make a data set, e.g., a selection of a data table, based on the selection control 2012 of the data set, which may be pre-stored at the user terminal.

In addition, the data input area 202 may include a prompt text 2021 to input prompt information, such as to "please input a data set", and an input control 2022 to the data set, and a confirm button 2023. It is also possible to detect that the user performs real-time data input through the data input control 2022, such as a data set directly filled in by way of paste, input, or the like.

Referring to the above description, in step 101, obtaining the data set generated by the target user may refer to obtaining the data set obtained by the target user through the data selection operation performed by the data selection area. It may also mean that a data set input by the target user through the data input area is obtained.

Of course, in practical applications, the storage address of the data set may be located in the data processing operation by providing the storage address of the data set, such as an ftp (File Transfer Protocol, text transfer protocol) address. For example, the memory address of the data set may be read from the data processing operation, and the data set may be read from the terminal device storing the data set based on the memory address of the data set.

In practical application, the parallel data processing method provided by the application can be executed by the user terminal or the electronic equipment which is in communication connection with the user terminal.

As an alternative embodiment, the user terminal may detect a data processing operation performed by the target user with respect to the data processing interface. The user terminal may send the detected operation to the electronic device, so that the electronic device performs the step 101.

As yet another alternative embodiment, the electronic device may directly detect the data processing operation performed by the target user with respect to the data processing interface, and perform step 101 described above.

And 102, performing data slicing processing on the data set to obtain a plurality of data slicing sets.

Alternatively, step 102 may still be performed by a parallel processing process, i.e., the parallel processing process may perform a slicing process on the data set to obtain multiple data slicing sets. Step 102 may specifically include: and determining a plurality of pieces of data to be processed contained in the data set, determining the number of data fragments as N, and dividing the plurality of pieces of data to be processed in the data set into N data fragment sets. N is a positive integer greater than 1.

Illustratively, subjecting the data set to a data slicing process, obtaining a plurality of data slicing sets may include: and carrying out data slicing processing on the data set according to the preset slicing quantity and/or slicing size to obtain a plurality of data slicing sets.

The data sharded set may be a subset of the data set. The data set can be obtained by combining a plurality of data fragment sets.

And 103, distributing the plurality of data fragment sets to corresponding data processing processes, so that each data processing process performs data processing on the received data fragment sets to generate corresponding data processing results.

Alternatively, step 103 may still be performed by a parallel processing process, i.e., the parallel processing process may distribute multiple data shard sets to corresponding data processing processes.

In this embodiment, the electronic device or the user terminal may execute the methods of steps 101-104 through parallel processing procedures, as well as the methods shown in other embodiments. Distributing the plurality of data shard sets to corresponding data manipulation processes in step 103 may include: the parallel processing process distributes each data fragment set to the corresponding data processing process based on the communication interface with each data processing process.

The data processing process can process the data to be processed in the data slicing set by utilizing the data processing program to obtain the processing result of each piece of data to be processed in the data slicing set. The processing results may include a result type and result content. The type of result may be either processing success or processing failure. The result content is a data processing result of the data to be processed, for example, if the data processing program is an update program of the fund unit, the result content may be the updated fund unit.

Step 104, receiving data processing results fed back by each data processing process.

Alternatively, step 104 may still be performed by a parallel processing process, that is, the parallel processing process may receive the data processing results fed back by each data processing process separately.

In step 104, the data processing results fed back by the data processing interfaces can be received through the communication interfaces of the parallel processing process and the data processing processes. The data processing result fed back by the data processing process can be the data processing result of the data to be processed in the data slicing set processed by the data processing process.

In order to achieve the efficient relationship of the slicing, step 102 performs data slicing processing on the data set to obtain a plurality of data slicing sets, which may include:

and carrying out data slicing processing on the data set based on preset data slicing logic to obtain a plurality of data slicing sets.

Alternatively, the data slicing logic may include information on the number of slices, the size of the slices, the slicing algorithm, etc. The slicing algorithm may comprise, for example, a hash slicing algorithm. Of course, in practical applications, the algorithm used for slicing may be set according to the requirement of use, and the data slicing algorithm in this embodiment is not limited too.

The slicing step may specifically be: and dividing the data set into a plurality of data fragment sets according to the logic information such as the fragment data, the fragment size, the fragment algorithm and the like in the data fragment logic.

In this embodiment, the data set is subjected to data slicing processing according to the data slicing logic, so as to realize effective slicing of the data set and realize rapid and accurate slicing of multiple data slicing sets.

The data slicing logic can be preset default slicing logic, and also can be personalized slicing logic set by a user.

Further, based on preset data slicing logic, performing data slicing processing on the data set, and before obtaining a plurality of data slicing sets, further including:

Displaying an editing interface of the slicing logic;

and responding to the editing operation executed by the target user for the editing interface, and obtaining the data slicing logic generated by the target user.

For ease of understanding, FIG. 3 shows an exemplary diagram of an editing interface for the slicing logic. The editing interface 300 of the slicing logic may include: a tile number setting area 301 or a tile size setting area 302. The target user can set the number N of the divided plurality of data fragment sets through the fragment number control 3011 in the fragment number setting area 301. The target user may also set the sizes of the divided plurality of data shard sets through the shard size control 3012 in the shard size setting area 302, and the number N of data shard sets may be determined according to the quotient of the total number of data sets and the data shard size.

In addition, the editing interface may also include the setting or selection logic of the slicing algorithm. For example, several candidate slicing algorithms may be provided, and a selection operation performed by the user from the displayed at least one candidate slicing algorithm may be detected to obtain a target slicing algorithm. Data slicing logic may be determined based on at least one of a slice size, a number of slices, and/or a target slicing algorithm.

In this embodiment, an editing interface of the slicing logic may be displayed, and the data slicing logic generated by the target user may be obtained in response to an editing operation performed by the target user with respect to the editing interface. And the personalized setting of the target user on the data slicing logic is realized, so that the data slicing logic is more matched with the slicing requirement of the target user.

As shown in fig. 4, a flowchart of another embodiment of a parallel data processing method according to an embodiment of the present application is different from any of the foregoing embodiments in that, in step 103, a plurality of data slicing sets are distributed to corresponding data processing processes, so that each data processing process performs data processing on a received data slicing set to generate a corresponding data processing result, which may include:

step 401, creating N data processing processes according to the number N of the plurality of data fragment sets.

Optionally, in step 401, creating N data processing processes may include: and creating N data processing processes according to the process creation strategy. The N data processing processes may be sub-processes of the parallel processing process. The parallel processing process may create N data processing processes and determine the N data processing processes as sub-processes thereof. Illustratively, the parallel processing process may be a Master process, and the data processing process may be a Slave process. The process creation process using the process creation policy may specifically include setting up a process block and establishing control and management information of the created process block by the parallel processing process to obtain a data processing process.

Step 402, acquiring a data processing program corresponding to a plurality of pieces of data to be processed in the data set.

The data processing program may be computer software set for data processing requirements of the data to be processed, and may be packaged as a data processing program. The data processing program may be set for the target user, that is, the data processing program set by the target user may be acquired.

For example, the data manipulation program may be located in a data manipulation operation, i.e., the data manipulation operation may include settings for the data manipulation program to make the data manipulation program more compatible with the user-provided data set.

And step 403, transmitting the data slicing set and the data processing program to a data processing process, so that the data processing process utilizes the data processing program to process the data to be processed in the data slicing set, and obtaining a data processing result of the data slicing set.

In this embodiment, by creating N data processing processes, the number of data processing processes may be consistent with the number of multiple data fragment sets, that is, even if the data processing processes correspond to the data fragment sets one by one, after the data fragment sets and the data processing programs are transmitted to the data processing processes, the N data processing processes may be executed in parallel, so as to implement parallel processing of the multiple data fragment sets.

Further, on the basis of any one of the above embodiments, acquiring a data processing program corresponding to a plurality of pieces of data to be processed in a data set includes:

acquiring program setting operation executed by a target user aiming at a program setting interface, and acquiring a data processing program set by the target user;

or, acquiring a software compression package matched with the data set; the software compression package is parsed to obtain a data processing program.

Alternatively, the program setting interface may be a sub-interface in the data processing interface, that is, the data processing interface may further include the program setting interface. In addition, the program setting interface can also be an interface displayed after the data processing interface. That is, after the user performs the data processing operation on the data processing interface, the program setting interface can be switched to be displayed, and the user can view and perform the program setting operation on the program setting interface to obtain the data processing program set by the target user.

Wherein obtaining the data manipulation program of the target user setting may include receiving the data manipulation program provided by the target user. For example, the data processing program may be an installation package stored in the user terminal, and the user may be detected to upload the installation package of the data processing program to obtain the program setting operation. The program setting operation may include an installation package of the data processing program.

Alternatively, obtaining a software compression package that matches the data set may include: and receiving a program storage address set by the target user, and reading the software compression package from the program storage address. The program storage address may be a storage address set by the user, may be selected by the user through an interface (a plurality of data processing programs), or may be an address directly provided by the user.

In addition, a plurality of compression packets can be preset, and according to the data processing type set by each compression packet, the target data processing type which is the same as the data type of the data to be processed in the data set is determined according to the data processing type corresponding to each compression packet, so that the compression packet corresponding to the target data processing type is a software compression packet matched with the data set.

In this embodiment, after the installation package or the software compression package obtained by the parallel processing program, the installation package or the software compression package may be run to start the data processing program, and the started data processing program is distributed to the data processing process.

In this embodiment, the program setting operation executed by the target user for the program setting interface may be obtained, the data processing program set by the target user may be obtained, the personalized setting of the data processing program by the user may be implemented, and the setting efficiency and accuracy of the data processing program may be improved. In addition, the embodiment also provides a mode of directly matching the data set with the software compression package, so that the corresponding data processing program is automatically determined for the data set, and the acquisition efficiency and accuracy of the data processing program are improved.

In order to timely acquire the running condition of each data processing process, the data state of each data processing process can be detected.

Further, after the data fragment set and the data processing program are transferred to the data processing process according to any one of the above embodiments, the method further includes:

detecting the execution states corresponding to the N data processing processes respectively;

if the abnormal execution state exists in the execution states corresponding to the N data processing processes respectively, determining the abnormal data processing process in the abnormal execution state;

generating abnormal prompt information corresponding to the abnormal data processing process, and outputting the abnormal prompt information.

Optionally, a status flag bit may be set during the data processing process. In the running process of the data processing process, the value of the state zone bit can be updated in real time, if the abnormal execution state is determined to exist, the state zone bit can be updated to 0, and if the data processing process is in the normal execution state, the state zone bit can be set to 1.

The parallel processing process can read the value of the status flag bit of the data processing process according to a certain frequency. And determining the execution state of the data processing process according to the value of the read state flag bit. The execution state may include an abnormal execution state or a normal execution state.

In this embodiment, the detection of the execution state may be performed on the N data processing processes, so that the parallel processing process may perform state detection on the N data processing processes in time, obtain the execution state of each data processing process, quickly determine the data processing process in the abnormal execution state, and output the abnormal prompt information for the data processing process in the abnormal execution state. The target user can timely acquire the execution state of each data processing process and timely respond when the data processing process in the abnormal execution state exists, so that the stability of the data processing process is effectively improved, and the data processing efficiency is further effectively improved.

As one embodiment, the data processing results include: a first result corresponding to the data which is successfully processed and a second result corresponding to the data which is failed to be processed; the method further comprises the steps of:

determining a second result corresponding to the data failing to be processed in each of the plurality of data slicing sets according to the data processing results corresponding to each of the plurality of data slicing sets;

according to the second results corresponding to the data which are failed to be processed in the plurality of data slicing sets, carrying out fusion processing on the data results which are failed to be processed, and obtaining a failure result set which is failed to be processed in the data processing;

And outputting the failure result set for the target user to check.

Alternatively, the second result may be a result corresponding to data of which the data slicing centralized processing failed. The first result may be a result corresponding to data that is successfully processed in the data shard set.

The fusing of the data results with processing failure may refer to establishing a failure result set for data in the data results with processing failure. The output failure result set may refer to outputting data of processing failure in a data table or the like.

In the embodiment of the application, the second result corresponding to the data identified by the respective processing in the plurality of data slicing sets can be determined according to the data processing results respectively corresponding to the plurality of data slicing sets, so that the independent acquisition of the data failing to be processed in the respective data slicing sets is realized. And meanwhile, according to a second result corresponding to each processing failure data in the plurality of data slicing sets, the processing failure data results are fused to obtain a failure result set of the data processing failure, namely, the integration of the processing identification data and the processing result is realized, and the failure result set can be output for a target user to check. Through the output of the failure result, the target user can acquire the data which is failed to be processed in time, interaction with the target user from the angle of data processing failure is realized, and user experience is improved.

In order to improve the success rate of data processing, after obtaining the failure result set of data processing identification on the basis of any embodiment, the method further includes:

reading data of processing failure from a failure result set;

generating a new data fragment set according to the data of which the data processing fails;

and (3) carrying out data processing treatment again on the new data fragment set according to the data processing program to obtain a reprocessing result.

The data processing is performed again on the new data fragment set according to the data processing program, so as to obtain a reprocessing result, which may include: and establishing a data processing process, transmitting the data processing program and the new data fragment set to the data processing process, and processing the data in the new data fragment set again through the data processing process by the data processing program to obtain a reprocessing result. And obtaining a reprocessing result fed back by the data processing process.

In this embodiment, for the data failing to be centrally processed in the identification result, a new data slicing set may be separately and again combined, and the data processing program is utilized to perform data processing again on the data in the new data slicing set, so as to obtain a reprocessing result. By arranging the data with processing failure, the processing can be performed again on the data with processing failure, so that the probability of successful data processing can be improved, the data with processing failure can be reduced, and the user experience can be improved.

In order to summarize the processing result of each piece of data to be processed, further, on the basis of any one of the embodiments, after receiving the data processing result fed back by each data processing process, the method further includes:

carrying out result fusion processing according to data processing results respectively corresponding to the plurality of data fragment sets to obtain processing results of each piece of data to be processed in the data set;

and outputting processing results respectively corresponding to the plurality of pieces of data to be processed in the data set.

Optionally, performing the result fusion processing according to the data processing results respectively corresponding to the plurality of data fragment sets may include: and sequencing the data processing results corresponding to the data to be processed respectively according to the sequence of the data to be processed in the data set, so as to obtain the processing result of each piece of data to be processed in the data set.

The processing result of each piece of data to be processed in the data set can be updated as a data element or added into the piece of data to be processed, so that the result of the data to be processed in the data set is updated, and the updated data set is obtained. That is, the processing result of each piece of data to be processed is utilized to update the piece of data to be processed, and the updated target data of each piece of data to be processed is obtained. The updated data set may include multiple-entry tag data updated as a result.

In this embodiment, the result fusion processing may be performed on the data processing results corresponding to the multiple sliced data sets respectively, so as to obtain the processing result of each piece of data to be processed in the data set, and achieve the result acquisition of each piece of data to be processed in the data set. And then, the result output of each piece of data to be processed in the data set can be realized by outputting the processing results respectively corresponding to the pieces of data to be processed in the data set, thereby providing convenience for a user to check the data processing results of each piece of data to be processed in the data set and improving the user experience.

In order to describe the application mode of the application in detail, the technical scheme of the application is described in detail by taking a data set as a data table as an example. As shown in fig. 5, the electronic device 500 may include a display 501. The display device 501 may be located on the user side. The display device 501 may display a data processing interface. The target user may perform a data manipulation operation with respect to the data manipulation interface, which may then be provided to the processing device 502 of the electronic apparatus 500 after detection.

The processing device 502 may perform any of the parallel data processing methods described above. In the parallel data processing method, after the data processing operation is acquired, the data set generated by the target user can be acquired in response to the data processing operation. The data set may include a plurality of pieces of data to be processed.

Then, the processing device 502 may further perform data slicing processing on the data set to obtain a plurality of data slicing sets, which are respectively data slicing set 1 to data slicing set N. And creating a corresponding data processing process for each data fragment set to obtain N data processing processes. And distributing the data fragment set and the data processing program to the corresponding data processing process. And processing the data to be processed in the data slicing set one by one through a data processing program in the data processing process so as to obtain processing results of the data to be processed, wherein the processing results of the data to be processed are data processing results of the data processing process on the data slicing set. The processing device 502 may obtain data processing results fed back by each data processing process.

Of course, the structure of the electronic device 500 described above is merely exemplary. In practical applications, the above parallel data processing method may also be performed by a parallel data processing system. The parallel data processing system may include a user terminal and a server. The user terminal may display a data manipulation interface and detect a data manipulation operation performed by the target user. The user terminal may also send data processing operations to the server. The server may execute the parallel data processing method as shown in the processing device 502, so as to implement parallel execution of data slicing and data, and obtain data processing results respectively generated by each data processing process.

As shown in fig. 6, a schematic structural diagram of a parallel data processing apparatus according to an embodiment of the present application may be configured in a parallel processing process, where the parallel processing apparatus may include:

the set acquisition unit 601: the data processing method comprises the steps that a parallel processing process responds to data processing operation triggered by a target user for a data processing interface, a data set generated by the target user is obtained, and the data set comprises a plurality of pieces of data to be processed;

data slicing unit 602: the method comprises the steps of performing data slicing processing on a data set to obtain a plurality of data slicing sets;

process processing unit 603: the data processing method comprises the steps of distributing a plurality of data fragment sets to corresponding data processing processes, so that each data processing process performs data processing on the received data fragment sets to generate corresponding data processing results;

the result acquisition unit 604: and the data processing device is used for receiving the data processing results fed back by the data processing processes respectively.

As an embodiment, the data slicing unit 602 may include:

the logic slicing module is used for performing data slicing processing on the data set based on preset data slicing logic to obtain a plurality of data slicing sets.

As yet another embodiment, further comprising:

The interface display unit is used for displaying an editing interface of the slicing logic;

and the logic editing unit is used for responding to the editing operation executed by the target user for the editing interface and obtaining the data slicing logic generated by the target user.

As yet another embodiment, a process processing unit includes:

the process creation module is used for creating N data processing processes according to the number N of the data fragment sets;

the program acquisition module is used for acquiring a data processing program corresponding to a plurality of pieces of data to be processed in the data set;

the process processing module is used for transmitting the data fragment set and the data processing program to a data processing process, so that the data processing process utilizes the data processing program to process the data to be processed in the data fragment set, and a data processing result of the data fragment set is obtained.

As still another embodiment, a program acquisition module includes:

the program setting sub-module is used for acquiring program setting operation executed by a target user aiming at a program setting interface and acquiring a data processing program set by the target user;

or the software analysis sub-module is used for acquiring a software compression package matched with the data set; the software compression package is parsed to obtain a data processing program.

As yet another embodiment, further comprising:

the state detection unit is used for detecting the execution states corresponding to the N data processing processes respectively;

the abnormality detection unit is used for determining an abnormal data processing process in the abnormal execution state if the abnormal execution state exists in the execution states corresponding to the N data processing processes respectively;

the abnormal prompting unit is used for generating abnormal prompting information corresponding to the abnormal data processing process and outputting the abnormal prompting information.

As yet another embodiment, the data processing result includes: a first result corresponding to the data which is successfully processed and a second result corresponding to the data which is failed to be processed; further comprises:

the failure determining unit is used for determining a second result corresponding to the data which are failed to be processed in each of the plurality of data slicing sets according to the data processing results corresponding to each of the plurality of data slicing sets;

the failure fusion unit is used for carrying out fusion processing on the data results which are failed to be processed according to the second results corresponding to the data which are failed to be processed in the plurality of data slicing sets, so as to obtain a failure result set which is failed to be processed by the data;

and the result storage unit is used for storing the failure result set for the target user to check.

As yet another embodiment, further comprising:

a data reading unit for reading data of failure processing from the failure result set;

the slicing splicing unit is used for generating a new data slicing set according to the data which fail in data processing;

and the secondary processing unit is used for carrying out data processing treatment on the new data slicing set again according to the data processing program to obtain a reprocessing result.

As yet another embodiment, further comprising:

the result fusion unit is used for carrying out result fusion processing according to the data processing results respectively corresponding to the plurality of data fragment sets to obtain the processing result of each piece of data to be processed in the data set;

and the result output unit is used for outputting processing results respectively corresponding to the plurality of pieces of data to be processed in the data set.

Fig. 7 is a block diagram of an electronic device, which may be a mobile phone, computer, digital broadcast terminal, messaging device, game console, tablet device, medical device, exercise device, personal digital assistant, or the like, in accordance with an exemplary embodiment.

The electronic device 700 may include one or more of the following components: a processing component 702, a memory 704, a power component 706, a multimedia component 708, an audio component 710, an input/output (I/O) interface 712, a sensor component 714, and a communication component 716.

The processing component 702 generally controls overall operation of the apparatus 700, such as operations associated with display, telephone calls, data communications, camera operations, and recording operations. The processing component 702 may include one or more processors 720 to execute instructions to perform all or part of the steps of the methods described above. Further, the processing component 702 can include one or more modules that facilitate interaction between the processing component 702 and other components. For example, the processing component 702 may include a multimedia module to facilitate interaction between the multimedia component 708 and the processing component 702.

The memory 704 is configured to store various types of data to support operations at the apparatus 700. Examples of such data include instructions for any application or method operating on the apparatus 700, contact data, phonebook data, messages, pictures, videos, and the like. The memory 704 may be implemented by any type or combination of volatile or nonvolatile memory devices such as Static Random Access Memory (SRAM), electrically erasable programmable read-only memory (EEPROM), erasable programmable read-only memory (EPROM), programmable read-only memory (PROM), read-only memory (ROM), magnetic memory, flash memory, magnetic or optical disk.

The power supply component 706 provides power to the various components of the device 700. The power components 706 may include a power management system, one or more power sources, and other components associated with generating, managing, and distributing power for the device 700.

The multimedia component 708 includes a screen between the device 700 and the user that provides an output interface. In some embodiments, the screen may include a Liquid Crystal Display (LCD) and a Touch Panel (TP). If the screen includes a touch panel, the screen may be implemented as a touch screen to receive input signals from a user. The touch panel includes one or more touch sensors to sense touches, swipes, and gestures on the touch panel. The touch sensor may sense not only the boundary of a touch or sliding action, but also the duration and pressure associated with the touch or sliding operation. In some embodiments, the multimedia component 708 includes a front-facing camera and/or a rear-facing camera. The front-facing camera and/or the rear-facing camera may receive external multimedia data when the apparatus 700 is in an operational mode, such as a photographing mode or a video mode. Each front camera and rear camera may be a fixed optical lens system or have focal length and optical zoom capabilities.

The audio component 710 is configured to output and/or input audio signals. For example, the audio component 710 includes a Microphone (MIC) configured to receive external audio signals when the device 700 is in an operational mode, such as a call mode, a recording mode, and a voice recognition mode. The received audio signals may be further stored in the memory 704 or transmitted via the communication component 716. In some embodiments, the audio component 710 further includes a speaker for outputting audio signals.

The I/O interface 712 provides an interface between the processing component 702 and peripheral interface modules, which may be a keyboard, click wheel, buttons, etc. These buttons may include, but are not limited to: homepage button, volume button, start button, and lock button.

The sensor assembly 714 includes one or more sensors for providing status assessment of various aspects of the apparatus 700. For example, the sensor assembly 714 may detect an on/off state of the device 700, a relative positioning of the assemblies, such as a display and keypad of the device 700, a change in position of the device 700 or one of the assemblies of the device 700, the presence or absence of user contact with the device 700, an orientation or acceleration/deceleration of the device 700, and a change in temperature of the device 700. The sensor assembly 714 may include a proximity sensor configured to detect the presence of nearby objects without any physical contact. The sensor assembly 714 may also include a light sensor, such as a CMOS or CCD image sensor, for use in imaging applications. In some embodiments, the sensor assembly 714 may also include an acceleration sensor, a gyroscopic sensor, a magnetic sensor, a pressure sensor, or a temperature sensor.

The communication component 716 is configured to facilitate communication between the apparatus 700 and other devices in a wired or wireless manner. The apparatus 700 may access a wireless network based on a communication standard, such as WiFi,2G or 3G, or a combination thereof. In one exemplary embodiment, the communication component 716 receives broadcast signals or broadcast related information from an external broadcast management system via a broadcast channel. In one exemplary embodiment, the communication component 716 further includes a Near Field Communication (NFC) module to facilitate short range communications. For example, the NFC module may be implemented based on Radio Frequency Identification (RFID) technology, infrared data association (IrDA) technology, ultra Wideband (UWB) technology, bluetooth (BT) technology, and other technologies.

In an exemplary embodiment, the apparatus 700 may be implemented by one or more Application Specific Integrated Circuits (ASICs), digital Signal Processors (DSPs), digital Signal Processing Devices (DSPDs), programmable Logic Devices (PLDs), field Programmable Gate Arrays (FPGAs), controllers, microcontrollers, microprocessors, or other electronic elements for executing the methods described above.

In an exemplary embodiment, a non-transitory computer readable storage medium is also provided, such as memory 704, including instructions executable by processor 720 of apparatus 700 to perform the above-described method. For example, the non-transitory computer readable storage medium may be ROM, random Access Memory (RAM), CD-ROM, magnetic tape, floppy disk, optical data storage device, etc.

The present application provides an electronic device including: a processor, a memory communicatively coupled to the processor;

the memory stores computer-executable instructions;

the processor executes the computer-executable instructions stored in the memory to implement a method as possible in any of the embodiments described above.

The present application provides a computer readable storage medium having stored therein computer executable instructions which when executed by a processor are adapted to carry out a method as possible in any of the embodiments.

The present application provides a computer program product comprising a computer program which, when executed by a processor, implements a method as possible in any of the embodiments.

Other embodiments of the application will be apparent to those skilled in the art from consideration of the specification and practice of the application disclosed herein. This application is intended to cover any variations, uses, or adaptations of the application following, in general, the principles of the application and including such departures from the present disclosure as come within known or customary practice within the art to which the application pertains. It is intended that the specification and examples be considered as exemplary only, with a true scope and spirit of the application being indicated by the following claims.

It is to be understood that the application is not limited to the precise arrangements and instrumentalities shown in the drawings, which have been described above, and that various modifications and changes may be effected without departing from the scope thereof. The scope of the application is limited only by the appended claims.

Claims

1. A method of parallel data processing, comprising:

and receiving data processing results fed back by each data processing process.

2. The method of claim 1, wherein the performing data slicing on the data set to obtain a plurality of data slicing sets comprises:

displaying an editing interface of the slicing logic;

responding to the editing operation executed by the target user for the editing interface, and obtaining data slicing logic generated by the target user;

And carrying out data slicing processing on the data set based on preset data slicing logic to obtain the plurality of data slicing sets.

3. The method of claim 1, wherein distributing the plurality of data shard sets to corresponding data processing processes such that each data processing process performs data processing on the received data shard sets to generate corresponding data processing results, comprises:

creating N data processing processes according to the number N of the plurality of data fragment sets;

acquiring a data processing program corresponding to the plurality of pieces of data to be processed in the data set;

and transmitting the data slicing set and the data processing program to the data processing process, so that the data processing process utilizes the data processing program to process the data to be processed in the data slicing set, and a data processing result of the data slicing set is obtained.

4. A method according to claim 3, wherein the acquiring a data processing program corresponding to the plurality of pieces of data to be processed in the data set includes:

acquiring program setting operation executed by the target user aiming at a program setting interface, and acquiring a data processing program set by the target user;

Or, acquiring a software compression package matched with the data set; parsing the software compression package to obtain the data processing program;

after the data fragment set and the data processing program are transmitted to the data processing process, the method further comprises:

if the abnormal execution state exists in the execution states corresponding to the N data processing processes, determining the abnormal data processing process in the abnormal execution state;

5. The method of any of claims 1-4, wherein the data processing results comprise: a first result corresponding to the data which is successfully processed and a second result corresponding to the data which is failed to be processed; the method further comprises the steps of:

determining a second result corresponding to the data which are failed to be processed in each of the plurality of data slicing sets according to the data processing results corresponding to each of the plurality of data slicing sets;

according to the second results corresponding to the data which are failed to be processed in the plurality of data slicing sets, carrying out fusion processing on the data results which are failed to be processed, and obtaining a failure result set of the data processing failure;

And outputting the failure result set for the target user to check.

6. The method of claim 5, wherein after obtaining the set of failure results identified by the data processing, further comprising:

reading data of processing failure from the failure result set;

generating a new data fragment set according to the data of the data processing failure;

and carrying out data processing treatment again on the new data fragment set according to the data processing program to obtain a reprocessing result.

7. The method according to any one of claims 1 to 4, wherein after receiving the data processing results fed back by each data processing process, the method further comprises:

performing result fusion processing according to data processing results respectively corresponding to the plurality of data fragment sets to obtain processing results of each piece of data to be processed in the data set;

8. A parallel data processing apparatus, comprising:

9. An electronic device, comprising: a processor, and a memory communicatively coupled to the processor;

the memory stores computer-executable instructions;

the processor executes computer-executable instructions stored in the memory to implement the method of any one of claims 1 to 7.

10. A computer readable storage medium having stored therein computer executable instructions which when executed by a processor are adapted to carry out the method of any one of claims 1 to 7.