CN111176797B - Data concurrency processing method and device, electronic equipment and readable storage medium - Google Patents
Data concurrency processing method and device, electronic equipment and readable storage medium Download PDFInfo
- Publication number
- CN111176797B CN111176797B CN201911310868.0A CN201911310868A CN111176797B CN 111176797 B CN111176797 B CN 111176797B CN 201911310868 A CN201911310868 A CN 201911310868A CN 111176797 B CN111176797 B CN 111176797B
- Authority
- CN
- China
- Prior art keywords
- concurrent processing
- flow stage
- result data
- stage
- processing result
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
- 238000003672 processing method Methods 0.000 title abstract description 17
- 238000012545 processing Methods 0.000 claims abstract description 331
- 238000000034 method Methods 0.000 claims abstract description 93
- 230000008569 process Effects 0.000 claims abstract description 66
- 230000015654 memory Effects 0.000 claims description 19
- 230000001419 dependent effect Effects 0.000 claims description 4
- 238000005516 engineering process Methods 0.000 abstract description 3
- 238000006243 chemical reaction Methods 0.000 description 22
- 238000012821 model calculation Methods 0.000 description 11
- 238000010586 diagram Methods 0.000 description 5
- 238000004891 communication Methods 0.000 description 4
- 238000004590 computer program Methods 0.000 description 4
- 230000006870 function Effects 0.000 description 3
- 238000012986 modification Methods 0.000 description 3
- 230000004048 modification Effects 0.000 description 3
- 230000003993 interaction Effects 0.000 description 2
- 239000004973 liquid crystal related substance Substances 0.000 description 2
- 238000004364 calculation method Methods 0.000 description 1
- 230000001413 cellular effect Effects 0.000 description 1
- 238000010276 construction Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 230000001151 other effect Effects 0.000 description 1
- 230000009467 reduction Effects 0.000 description 1
- 230000001953 sensory effect Effects 0.000 description 1
- 239000004984 smart glass Substances 0.000 description 1
- 238000006467 substitution reaction Methods 0.000 description 1
- 230000000007 visual effect Effects 0.000 description 1
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/465—Distributed object oriented systems
Landscapes
- Engineering & Computer Science (AREA)
- Software Systems (AREA)
- Theoretical Computer Science (AREA)
- Physics & Mathematics (AREA)
- General Engineering & Computer Science (AREA)
- General Physics & Mathematics (AREA)
- Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
Abstract
The application discloses a data concurrency processing method, a device, electronic equipment and a readable storage medium, and relates to the data processing technology. The specific implementation scheme is as follows: acquiring first concurrent processing result data of a first process stage output by the first process stage; carrying out concurrent processing of a second flow stage on the first concurrent processing result data according to a concurrent processing threshold of the second flow stage so as to obtain second concurrent processing result data of the second flow stage; and outputting the second concurrent processing result data.
Description
Technical Field
The present application relates to computer technology, and in particular, to a data processing technology, and more particularly, to a data concurrency processing method, apparatus, electronic device, and readable storage medium.
Background
In general, in a traditional data concurrency processing manner, a unified concurrency architecture is adopted for splitting data in each different flow stage, and each split set of data adopts an independent thread to complete each flow stage.
However, the conventional data concurrency processing manner may cause serious unbalance between the resource consumption and the time consumption of the partial flow stage, thereby causing the reduction of the system performance.
Disclosure of Invention
Aspects of the present application provide a data concurrency processing method, apparatus, electronic device, and readable storage medium, so as to improve system performance.
In one aspect of the present application, a method for processing data concurrency is provided, including:
acquiring first concurrent processing result data of a first process stage output by the first process stage;
carrying out concurrent processing of a second flow stage on the first concurrent processing result data according to a concurrent processing threshold of the second flow stage so as to obtain second concurrent processing result data of the second flow stage;
and outputting the second concurrent processing result data.
In the aspect and any possible implementation manner described above, there is further provided an implementation manner, where the performing, according to a concurrent processing threshold of a second flow stage, concurrent processing of the first concurrent processing result data in the second flow stage to obtain second concurrent processing result data in the second flow stage includes:
According to the concurrency processing threshold value of the second process stage, carrying out grouping processing on all the obtained first concurrency processing result data so as to obtain at least one data grouping;
and carrying out concurrent processing of the second flow stage on the first concurrent processing result data in each data packet in the at least one data packet to obtain processing result data of each data packet, wherein the processing result data is used as second concurrent processing result data of the second flow stage.
In the aspect and any possible implementation manner described above, there is further provided an implementation manner, where the performing, according to a concurrent processing threshold of a second flow stage, concurrent processing of the first concurrent processing result data in the second flow stage to obtain second concurrent processing result data in the second flow stage includes:
acquiring first concurrent processing result data which is acquired in real time and does not undergo concurrent processing in the second flow stage;
and if the number of the first concurrent processing result data which is not subjected to the concurrent processing of the second flow stage is greater than or equal to a concurrent processing threshold of the second flow stage, performing the concurrent processing of the second flow stage on the first concurrent processing result data which is not subjected to the concurrent processing threshold of the first concurrent processing result data of the second flow stage so as to obtain second concurrent processing result data of the second flow stage.
In accordance with aspects and any of the possible implementations described above, there is further provided an implementation in which the concurrent processing threshold of the second process stage is dependent on at least one of a resource consumption and a time consumption of the second process stage.
In accordance with the above aspect and any possible implementation manner, there is further provided an implementation manner, where the concurrent processing threshold of the first procedure stage is set independently from the concurrent processing threshold of the second procedure stage.
In another aspect of the present application, there is provided a data concurrency processing apparatus, including:
the data acquisition unit is used for acquiring first concurrent processing result data of a first process stage output by the first process stage;
the concurrency control unit is used for carrying out concurrency processing of the second flow stage on the first concurrency processing result data according to a concurrency processing threshold value of the second flow stage so as to obtain second concurrency processing result data of the second flow stage;
and the data output unit is used for outputting the second concurrent processing result data.
Aspects and any one of the possible implementations as described above, further provide an implementation, the concurrency control unit is specifically configured to
According to the concurrency processing threshold value of the second process stage, carrying out grouping processing on all the obtained first concurrency processing result data so as to obtain at least one data grouping; and
and carrying out concurrent processing of the second flow stage on the first concurrent processing result data in each data packet in the at least one data packet to obtain processing result data of each data packet, wherein the processing result data is used as second concurrent processing result data of the second flow stage.
Aspects and any one of the possible implementations as described above, further provide an implementation, the concurrency control unit is specifically configured to
Acquiring first concurrent processing result data which is acquired in real time and does not undergo concurrent processing in the second flow stage; and
and if the number of the first concurrent processing result data which is not subjected to the concurrent processing of the second flow stage is greater than or equal to a concurrent processing threshold of the second flow stage, performing the concurrent processing of the second flow stage on the first concurrent processing result data which is not subjected to the concurrent processing threshold of the first concurrent processing result data of the second flow stage so as to obtain second concurrent processing result data of the second flow stage.
In accordance with aspects and any of the possible implementations described above, there is further provided an implementation in which the concurrent processing threshold of the second process stage is dependent on at least one of a resource consumption and a time consumption of the second process stage.
In accordance with the above aspect and any possible implementation manner, there is further provided an implementation manner, where the concurrent processing threshold of the first procedure stage is set independently from the concurrent processing threshold of the second procedure stage.
In another aspect of the present invention, there is provided an electronic apparatus including:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the aspects and methods of any one of the possible implementations described above.
In another aspect of the invention, there is provided a non-transitory computer-readable storage medium storing computer instructions for causing a computer to perform the method of the aspects and any possible implementation described above.
As can be seen from the above technical solution, in the embodiment of the present application, by acquiring the first concurrent processing result data of the first flow stage output by the first flow stage, further, the first concurrent processing result data may be subjected to concurrent processing of the second flow stage according to the concurrent processing threshold of the second flow stage, so as to obtain second concurrent processing result data of the second flow stage, so that the second concurrent processing result data may be output.
In addition, by adopting the technical scheme provided by the application, no dependency relationship exists between the concurrent processing of each flow stage, and the concurrent processing of the flow stage can be independently carried out, so that the automatic parallel of the concurrent processing of part of the flow stages can be realized.
In addition, by adopting the technical scheme provided by the application, the experience of the user can be effectively improved.
Other effects of the above aspects or possible implementations will be described below in connection with specific embodiments.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present application, the following description will briefly explain the drawings used in the embodiments or the description of the prior art, and it is obvious that the drawings in the following description are some embodiments of the present application, and other drawings can be obtained according to these drawings without inventive effort for a person skilled in the art. The drawings are only for a better understanding of the present solution and are not to be construed as limiting the application. Wherein:
FIG. 1A is a flow chart illustrating a data concurrency processing method according to an embodiment of the present application;
FIG. 1B is a schematic diagram of concurrent processing of four flow stages in the embodiment corresponding to FIG. 1A;
FIG. 2 is a schematic diagram of a data concurrency processing apparatus according to another embodiment of the present application;
fig. 3 is a schematic diagram of an electronic device for implementing a data concurrency processing method provided by an embodiment of the present application.
Detailed Description
Exemplary embodiments of the present application will now be described with reference to the accompanying drawings, in which various details of the embodiments of the present application are included to facilitate understanding, and are to be considered merely exemplary. Accordingly, those of ordinary skill in the art will recognize that various changes and modifications of the embodiments described herein can be made without departing from the scope and spirit of the application. Also, descriptions of well-known functions and constructions are omitted in the following description for clarity and conciseness.
It will be apparent that the described embodiments are some, but not all, embodiments of the application. All other embodiments, which can be made by those skilled in the art based on the embodiments of the application without making any inventive effort, are intended to be within the scope of the application.
It should be noted that, the terminal according to the embodiments of the present application may include, but is not limited to, a mobile phone, a personal digital assistant (Personal Digital Assistant, PDA), a wireless handheld device, a Tablet Computer (Tablet Computer), a personal Computer (Personal Computer, PC), an MP3 player, an MP4 player, a wearable device (e.g., smart glasses, smart watches, smart bracelets, etc.), and so on.
In addition, the term "and/or" herein is merely an association relationship describing an association object, and means that three relationships may exist, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. In addition, the character "/" herein generally indicates that the front and rear associated objects are an "or" relationship.
Fig. 1A is a flow chart of a data concurrency processing method according to an embodiment of the application, as shown in fig. 1A.
101. And acquiring first concurrent processing result data of the first process stage output by the first process stage.
102. And carrying out concurrent processing of the second flow stage on the first concurrent processing result data according to a concurrent processing threshold of the second flow stage so as to obtain second concurrent processing result data of the second flow stage.
103. And outputting the second concurrent processing result data.
The execution bodies 101 to 103 may be part or all of applications located in the local terminal, or may be functional units such as plug-ins or software development kits (Software Development Kit, SDKs) provided in the applications located in the local terminal, or may be processing engines located in a server on the network side, or may be distributed systems located on the network side, for example, processing engines or distributed systems in a data processing platform on the network side, which is not particularly limited in this embodiment.
It will be appreciated that the application may be a native program (native app) installed on the terminal, or may also be a web page program (webApp) of a browser on the terminal, which is not limited in this embodiment.
In this way, by acquiring the first concurrent processing result data of the first flow stage output by the first flow stage, further, the first concurrent processing result data may be subjected to concurrent processing of the second flow stage according to the concurrent processing threshold of the second flow stage, so as to obtain second concurrent processing result data of the second flow stage, so that the second concurrent processing result data may be output.
The technical scheme provided by the application can be applied to various data processing schemes of multiple process stages, such as a feature acquisition process stage, a feature conversion process stage, a model acquisition process stage and a model conversion model stage, which are included in the prediction processing of the recommended article.
In the present application, the consumption of resources and the consumption of time by the various flow phases included in the various data processing schemes are not safe and even differ significantly from time to time. If the concurrent architecture related to the concurrent processing of each flow stage is configured uniformly, then the resource consumption and the time consumption of part of the flow stages are seriously unbalanced, and the system performance is reduced. Meanwhile, the concurrency architecture has poor flexibility and cannot be flexibly configured for concurrency processing of each flow stage.
The present application addresses the above-mentioned shortcomings by providing a flexible configurable concurrency architecture for improving system performance.
In the present application, the concurrency processing threshold refers to the maximum number of data objects allowed to be processed by each thread executing concurrently in each flow stage.
Specifically, the concurrent processing threshold value of each flow stage may be configured in advance according to the resource consumption and the time consumption of each flow stage in each data processing scheme. The concurrent processing thresholds of the flow stages are independent of each other without any dependency relationship.
In the application, the concurrent processing threshold values suitable for the execution characteristics of each flow stage are adopted respectively, so that the concurrent processing of each flow stage can effectively ensure the balance of the resource consumption and the time consumption of the flow stage, thereby improving the system performance.
The first and second process stages referred to in the present application are not specific to which process stage, and may be any two consecutive process stages of the process stages included in the data processing scheme.
Accordingly, the concurrency processing threshold of the second process stage depends on at least one of the resource consumption and the time consumption of the second process stage, which is not particularly limited in this embodiment.
It may be understood that, in 101, the obtained first concurrent processing result data of the first flow stage output by the first flow stage is obtained by performing the concurrent processing of the first flow stage on the previous concurrent processing result data of the previous flow stage output by the previous flow stage of the first flow stage according to the concurrent processing threshold of the first flow stage. Accordingly, the concurrency processing threshold of the first process stage depends on at least one of the resource consumption and the time consumption of the first process stage, which is not particularly limited in the present embodiment.
In the application, the concurrent processing threshold of the first procedure stage and the concurrent processing threshold of the second procedure stage are independently set.
Optionally, in one possible implementation manner of this embodiment, in 102, all acquired first concurrent processing result data may be specifically subjected to packet processing according to a concurrent processing threshold of the second procedure stage, so as to obtain at least one data packet. The number of the first concurrent processing result data included in each data packet is the concurrent processing threshold of the second flow stage, and since the number of the first concurrent processing result data may not be an integer multiple of the concurrent processing threshold of the second flow stage, the number of the first concurrent processing result data included in the last data packet may be less than the concurrent processing threshold of the second flow stage.
After obtaining at least one data packet, the concurrent processing of the second flow stage may be performed on first concurrent processing result data in each of the at least one data packet to obtain processing result data for each data packet as second concurrent processing result data for the second flow stage.
Specifically, a thread may be started for each data packet, and the concurrent processing of the second flow stage may be performed on the first concurrent processing result data in the data packet. After the thread completes concurrent processing of the first concurrent processing result data in the data packet, the thread may then be released.
After all the obtained first concurrent processing result data complete the concurrent processing of the second flow stage, the related processing of entering the next flow stage is started uniformly, that is, the steps 101 to 103 are repeatedly executed.
In the implementation mode, all the process stages are sequentially and uniformly entered according to the sequence, and all threads are uniformly started and run in the same process stage.
Optionally, in one possible implementation manner of this embodiment, in 102, first concurrent processing result data that is not subjected to concurrent processing in the second flow stage in the first concurrent processing result data acquired in real time may be specifically acquired. And if the number of the first concurrent processing result data which is not subjected to the concurrent processing of the second flow stage is greater than or equal to the concurrent processing threshold of the second flow stage, performing the concurrent processing of the second flow stage on the first concurrent processing result data which is not subjected to the concurrent processing threshold of the first concurrent processing result data of the second flow stage so as to obtain second concurrent processing result data of the second flow stage.
Specifically, the concurrency processing threshold of the second flow stage may be used to determine, from the first concurrency processing result data obtained in real time, first concurrency processing result data that is not subjected to concurrency processing of the second flow stage, and determine whether the number of the first concurrency processing result data that is not subjected to concurrency processing of the second flow stage meets the concurrency processing threshold of the second flow stage, so that the first concurrency processing result data that meets the concurrency processing threshold of the second flow stage can quickly enter the second flow stage, perform concurrency processing of the second flow stage, and then enter the second flow stage without waiting for obtaining all the first concurrency processing result data.
Because no dependency relationship exists between the concurrent processing of each flow stage, the concurrent processing of the flow stages can be independently performed, and thus, the automatic parallelization of the concurrent processing of part of the flow stages can be realized.
The technical scheme provided by the application will be described in detail below by taking four process stages, namely a feature acquisition process stage, a feature conversion process stage, a model acquisition process stage and a model conversion model stage, included in the prediction processing of the recommended article as an example. As shown in fig. 1B, the input estimated content of the estimated task includes 24 estimated objects, as shown by 21 small squares in fig. 1B. The concurrent processing threshold value of the feature acquisition flow stage is 6; the concurrent processing threshold value of the feature conversion flow stage is 8; the concurrency processing threshold value of the model acquisition flow stage is 12; the concurrency processing threshold of the model calculation flow stage is 8.
In the feature acquisition flow stage, the concurrent processing of the feature acquisition flow stage can be performed according to the concurrent processing threshold 6 of the feature acquisition flow stage, wherein every 6 pieces of estimated object data are integrated.
Specifically, in the feature acquisition stage, if the number of the acquired predicted object data to be processed reaches the concurrent processing threshold 6, one feature acquisition task is sequentially started, feature acquisition processing is performed on the 6 predicted object data, and a total of 4 feature acquisition tasks, namely, a feature acquisition task 1, a feature acquisition task 2, a feature acquisition task 3 and a feature acquisition task 4, can be started.
After each feature acquisition task is completed, the processing result data sequentially enter the next flow stage, namely a feature conversion flow stage, and serve as estimated object data of the next flow stage.
In the feature conversion flow stage, the concurrent processing of the feature conversion flow stage can be performed according to the concurrent processing threshold 8 of the feature conversion flow stage, wherein each 8 pieces of estimated object data are integrated.
Specifically, in the feature conversion stage, if the number of the obtained estimated object data to be processed reaches the concurrent processing threshold 8, one feature conversion task is started in sequence, feature conversion processing is performed on the 8 estimated object data, and a total of 3 feature acquisition tasks, namely, feature conversion task 1, feature conversion task 2 and feature conversion task 3, can be started.
For example, after the feature acquisition task 1 and the feature acquisition task 2 are completed, 12 pieces of predicted object data to be processed are acquired, at this time, the feature conversion task 1 may be started to perform feature conversion processing on 8 pieces of predicted object data in the 12 pieces of predicted object data, and the remaining 4 pieces of predicted object data to be processed. After the feature acquisition task 4 is completed, 6 pieces of predicted object data to be processed are further acquired, and the remaining 4 pieces of predicted object data are added, wherein the total is 10 pieces of predicted object data to be processed. At this time, the feature conversion task 2 may be started to perform feature conversion processing on 8 pieces of estimated object data out of the 10 pieces of estimated object data, and the remaining 2 pieces of estimated object data to be processed. After the feature acquisition task 3 is completed, 6 pieces of predicted object data to be processed are further acquired, and the total of the remaining 2 pieces of predicted object data is 8 pieces of predicted object data to be processed. At this time, the feature conversion task 3 may be started to perform feature conversion processing on 8 pieces of predicted object data, and there is no predicted object data remaining to be processed.
After each feature conversion task is completed, the processing result data sequentially enter the next flow stage, namely the model acquisition flow stage, and serve as estimated object data of the next flow stage.
Similarly, in the model acquisition flow stage, the concurrency processing of the model acquisition flow stage may be performed according to the concurrency processing threshold 12 of the model acquisition flow stage, with every 12 pieces of estimated object data as a whole.
Specifically, in the model acquisition stage, if the number of the acquired estimated object data to be processed reaches the concurrent processing threshold 12, one model acquisition task is started in sequence, model acquisition processing is performed on the 12 estimated object data, and a total of 2 model acquisition tasks, namely a model acquisition task 1 and a model acquisition task 2, can be started.
After the completion of each model acquisition task, the processing result data sequentially enter the next flow stage, namely a model calculation flow stage, and serve as estimated object data of the next flow stage.
In the model calculation flow stage, according to the concurrent processing threshold 8 of the model calculation flow stage, the concurrent processing of the model calculation flow stage can be performed for each 8 pieces of estimated object data as a whole.
Specifically, in the model calculation stage, if the number of the obtained estimated object data to be processed reaches the concurrent processing threshold 8, one model obtaining task is sequentially started, model calculation processing is performed on the 8 estimated object data, and a total of 3 model calculation tasks, namely, a model calculation task 1, a model calculation task 2 and a model calculation task 3, can be started.
After the calculation tasks of the models are completed, the processing result data can be output to serve as task results of the estimated tasks.
The technical scheme of the application can support the adoption of different concurrent processing thresholds in different process stages for concurrent processing, and can effectively balance the resource consumption and time consumption of each process stage, thereby improving the system performance.
In this embodiment, by acquiring the first concurrent processing result data of the first flow stage output by the first flow stage, further, the first concurrent processing result data may be subjected to concurrent processing of the second flow stage according to a concurrent processing threshold of the second flow stage, so as to obtain second concurrent processing result data of the second flow stage, so that the second concurrent processing result data may be output.
In addition, by adopting the technical scheme provided by the application, no dependency relationship exists between the concurrent processing of each flow stage, and the concurrent processing of the flow stage can be independently carried out, so that the automatic parallel of the concurrent processing of part of the flow stages can be realized.
In addition, by adopting the technical scheme provided by the application, the experience of the user can be effectively improved.
It should be noted that, for simplicity of description, the foregoing method embodiments are all described as a series of acts, but it should be understood by those skilled in the art that the present application is not limited by the order of acts described, as some steps may be performed in other orders or concurrently in accordance with the present application. Further, those skilled in the art will also appreciate that the embodiments described in the specification are all preferred embodiments, and that the acts and modules referred to are not necessarily required for the present application.
In the foregoing embodiments, the descriptions of the embodiments are emphasized, and for parts of one embodiment that are not described in detail, reference may be made to related descriptions of other embodiments.
Fig. 2 is a schematic structural diagram of a data concurrency processing device according to another embodiment of the present application, as shown in fig. 2. The processing apparatus 200 for data concurrency of the present embodiment may include a data acquisition unit 201, a concurrency control unit 202, and a data output unit 203. The data acquisition unit 201 is configured to acquire first concurrent processing result data of a first process stage output by the first process stage; a concurrency control unit 202, configured to perform concurrency processing of the second flow stage on the first concurrency processing result data according to a concurrency processing threshold of the second flow stage, so as to obtain second concurrency processing result data of the second flow stage; and a data output unit 203, configured to output the second concurrent processing result data.
It should be noted that, part or all of the execution body of the processing apparatus for data concurrency provided in this embodiment may be an application located at a local terminal, or may also be a functional unit such as a plug-in unit or a software development kit (Software Development Kit, SDK) disposed in an application located at a local terminal, or may also be a processing engine located in a server on a network side, or may also be a distributed system located on a network side, for example, a processing engine or a distributed system in a video processing platform on a network side, which is not limited in this embodiment.
It will be appreciated that the application may be a native program (native app) installed on the terminal, or may also be a web page program (webApp) of a browser on the terminal, which is not limited in this embodiment.
Optionally, in one possible implementation manner of this embodiment, the concurrency control unit 202 may specifically be configured to perform packet processing on all acquired first concurrency processing result data according to a concurrency processing threshold of the second procedure stage, so as to obtain at least one data packet; and performing concurrent processing of the second flow stage on the first concurrent processing result data in each data packet in the at least one data packet to obtain processing result data of each data packet as second concurrent processing result data of the second flow stage.
Optionally, in one possible implementation manner of this embodiment, the concurrency control unit 202 may be specifically configured to acquire first concurrency processing result data that is not subjected to concurrency processing in the second flow stage in the first concurrency processing result data acquired in real time; and if the number of the first concurrent processing result data which is not subjected to the concurrent processing of the second flow stage is greater than or equal to a concurrent processing threshold of the second flow stage, performing the concurrent processing of the second flow stage on the first concurrent processing result data which is not subjected to the concurrent processing threshold of the first concurrent processing result data of the second flow stage, so as to obtain second concurrent processing result data of the second flow stage.
The first and second process stages referred to in the present application are not specific to which process stage, and may be any two consecutive process stages of the process stages included in the data processing scheme.
The concurrent processing threshold of the second process stage depends on at least one of the resource consumption and the time consumption of the second process stage, which is not particularly limited in this embodiment.
It may be understood that the data obtaining unit 201 obtains the first concurrent processing result data of the first process stage output by the first process stage, which is obtained by performing the concurrent processing of the first process stage on the previous concurrent processing result data of the previous process stage output by the previous process stage of the first process stage according to the concurrent processing threshold of the first process stage. Accordingly, the concurrency processing threshold of the first process stage depends on at least one of the resource consumption and the time consumption of the first process stage, which is not particularly limited in the present embodiment.
In the application, the concurrent processing threshold of the first procedure stage and the concurrent processing threshold of the second procedure stage are independently set.
It should be noted that, the method in the embodiment corresponding to fig. 1A may be implemented by the processing device for data concurrency provided in this embodiment. For detailed description, reference may be made to the relevant content in the corresponding embodiment of fig. 1A, which is not repeated here.
In this embodiment, by acquiring the first concurrent processing result data of the first flow stage output by the first flow stage, further, the first concurrent processing result data may be subjected to concurrent processing of the second flow stage according to a concurrent processing threshold of the second flow stage, so as to obtain second concurrent processing result data of the second flow stage, so that the second concurrent processing result data may be output.
In addition, by adopting the technical scheme provided by the application, no dependency relationship exists between the concurrent processing of each flow stage, and the concurrent processing of the flow stage can be independently carried out, so that the automatic parallel of the concurrent processing of part of the flow stages can be realized.
In addition, by adopting the technical scheme provided by the application, the experience of the user can be effectively improved.
According to an embodiment of the present application, there is also provided an electronic device and a non-transitory computer-readable storage medium storing computer instructions.
Fig. 3 is a schematic diagram of an electronic device for implementing a data concurrency processing method provided by an embodiment of the present application. Electronic devices are intended to represent various forms of digital computers, such as laptops, desktops, workstations, personal digital assistants, servers, blade servers, mainframes, and other appropriate computers. The electronic device may also represent various forms of mobile devices, such as personal digital processing, cellular telephones, smartphones, wearable devices, and other similar computing devices. The components shown herein, their connections and relationships, and their functions, are meant to be exemplary only, and are not meant to limit implementations of the applications described and/or claimed herein.
As shown in fig. 3, the electronic device includes: one or more processors 301, memory 302, and interfaces for connecting the various components, including high-speed interfaces and low-speed interfaces. The various components are interconnected using different buses and may be mounted on a common motherboard or in other manners as desired. The processor may process instructions executing within the electronic device, including instructions stored in or on memory to display graphical information of a Graphical User Interface (GUI) on an external input/output device, such as a display device coupled to the interface. In other embodiments, multiple processors and/or multiple buses may be used, if desired, along with multiple memories and multiple memories. Also, multiple electronic devices may be connected, each providing a portion of the necessary operations (e.g., as a server array, a set of blade servers, or a multiprocessor system). One processor 301 is illustrated in fig. 3.
Memory 302 is a non-transitory computer readable storage medium provided by the present application. The memory stores instructions executable by the at least one processor to cause the at least one processor to perform the data concurrency processing method provided by the present application. The non-transitory computer readable storage medium of the present application stores computer instructions for causing a computer to execute the data concurrency processing method provided by the present application.
The memory 302 is used as a non-transitory computer readable storage medium, and may be used to store a non-transitory software program, a non-transitory computer executable program, and units, such as program instructions/units (e.g., the acquisition unit 201, the query unit 202, and the output unit 203 shown in fig. 2) corresponding to a data concurrency processing method in an embodiment of the present application. The processor 301 executes various functional applications of the server and data processing, i.e., implements the data concurrency processing method in the above-described method embodiments, by running non-transitory software programs, instructions, and units stored in the memory 302.
Memory 302 may include a storage program area that may store an operating system, at least one application program required for functionality, and a storage data area; the storage data area may store data created according to the use of the electronic device implementing the data concurrency processing method provided by the embodiment of the present application, and the like. In addition, memory 302 may include high-speed random access memory, and may also include non-transitory memory, such as at least one magnetic disk storage device, flash memory device, or other non-transitory solid-state storage device. In some embodiments, memory 302 may optionally include memory located remotely from processor 301, which may be connected via a network to an electronic device implementing the data-concurrent processing methods provided by embodiments of the present application. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof.
The electronic device of the data concurrency processing method may further include: an input device 303 and an output device 304. The processor 301, memory 302, input device 303, and output device 304 may be connected by a bus or other means, for example in fig. 3.
The input device 303 may receive input numeric or character information and generate key signal inputs related to user settings and function control of an electronic device implementing the data-concurrent processing method provided by embodiments of the present application, such as a touch screen, a keypad, a mouse, a track pad, a touch pad, a pointer stick, one or more mouse buttons, a track ball, a joystick, etc. The output device 304 may include a display apparatus, auxiliary lighting devices (e.g., LEDs), haptic feedback devices (e.g., vibration motors), and the like. The display device may include, but is not limited to, a Liquid Crystal Display (LCD), a Light Emitting Diode (LED) display, and a plasma display. In some implementations, the display device may be a touch screen.
Various implementations of the systems and techniques described here can be realized in digital electronic circuitry, integrated circuitry, application Specific Integrated Circuits (ASICs), computer hardware, firmware, software, and/or combinations thereof. These various embodiments may include: implemented in one or more computer programs, the one or more computer programs may be executed and/or interpreted on a programmable system including at least one programmable processor, which may be a special purpose or general-purpose programmable processor, that may receive data and instructions from, and transmit data and instructions to, a storage system, at least one input device, and at least one output device.
These computing programs (also referred to as programs, software applications, or code) include machine instructions for a programmable processor, and may be implemented in a high-level procedural and/or object-oriented programming language, and/or in assembly/machine language. As used herein, the terms "machine-readable medium" and "computer-readable medium" refer to any computer program product, apparatus, and/or device (e.g., magnetic discs, optical disks, memory, programmable Logic Devices (PLDs)) used to provide machine instructions and/or data to a programmable processor, including a machine-readable medium that receives machine instructions as a machine-readable signal. The term "machine-readable signal" refers to any signal used to provide machine instructions and/or data to a programmable processor.
To provide for interaction with a user, the systems and techniques described here can be implemented on a computer having: a display device (e.g., a Cathode Ray Tube (CRT) or Liquid Crystal Display (LCD) monitor) for displaying information to a user; and a keyboard and pointing device (e.g., a mouse or trackball) by which a user can provide input to the computer. Other kinds of devices may also be used to provide for interaction with a user; for example, feedback provided to the user may be any form of sensory feedback (e.g., visual feedback, auditory feedback, or tactile feedback); and input from the user may be received in any form, including acoustic input, speech input, or tactile input.
The systems and techniques described here can be implemented in a computing system that includes a background component (e.g., as a data server), or that includes a middleware component (e.g., an application server), or that includes a front-end component (e.g., a user computer having a graphical user interface or a web browser through which a user can interact with an implementation of the systems and techniques described here), or any combination of such background, middleware, or front-end components. The components of the system can be interconnected by any form or medium of digital data communication (e.g., a communication network). Examples of communication networks include: local Area Networks (LANs), wide Area Networks (WANs), and the internet.
The computer system may include a client and a server. The client and server are typically remote from each other and typically interact through a communication network. The relationship of client and server arises by virtue of computer programs running on the respective computers and having a client-server relationship to each other.
It should be appreciated that various forms of the flows shown above may be used to reorder, add, or delete steps. For example, the steps described in the present application may be performed in parallel, sequentially, or in a different order, so long as the desired results of the technical solution disclosed in the present application can be achieved, and are not limited herein.
The above embodiments do not limit the scope of the present application. It will be apparent to those skilled in the art that various modifications, combinations, sub-combinations and alternatives are possible, depending on design requirements and other factors. Any modifications, equivalent substitutions and improvements made within the spirit and principles of the present application should be included in the scope of the present application.
Claims (12)
1. A method for processing data concurrency, comprising:
acquiring first concurrent processing result data of a first flow stage, which is output by the first flow stage, wherein the first concurrent processing result data is obtained by performing concurrent processing of the first flow stage on data to be processed in the first flow stage according to a concurrent processing threshold of the first flow stage, and the concurrent processing threshold of the first flow stage refers to the maximum number of data objects allowed to be processed by each thread concurrently executed in the first flow stage;
carrying out concurrent processing of a second flow stage on the first concurrent processing result data according to a concurrent processing threshold of the second flow stage to obtain second concurrent processing result data of the second flow stage, wherein the concurrent processing threshold of the second flow stage refers to the maximum number of data objects allowed to be processed by each thread concurrently executed in the second flow stage;
And outputting the second concurrent processing result data.
2. The method of claim 1, wherein the performing the concurrent processing of the second process stage on the first concurrent processing result data according to a concurrent processing threshold of the second process stage to obtain second concurrent processing result data of the second process stage comprises:
according to the concurrency processing threshold value of the second process stage, carrying out grouping processing on all the obtained first concurrency processing result data so as to obtain at least one data grouping;
and carrying out concurrent processing of the second flow stage on the first concurrent processing result data in each data packet in the at least one data packet to obtain processing result data of each data packet, wherein the processing result data is used as second concurrent processing result data of the second flow stage.
3. The method of claim 1, wherein the performing the concurrent processing of the second process stage on the first concurrent processing result data according to a concurrent processing threshold of the second process stage to obtain second concurrent processing result data of the second process stage comprises:
Acquiring first concurrent processing result data which is acquired in real time and does not undergo concurrent processing in the second flow stage;
and if the number of the first concurrent processing result data which is not subjected to the concurrent processing of the second flow stage is greater than or equal to the concurrent processing threshold of the second flow stage, performing the concurrent processing of the second flow stage on the first concurrent processing result data of the concurrent processing threshold of the second flow stage in the first concurrent processing result data which is not subjected to the concurrent processing of the second flow stage, so as to obtain second concurrent processing result data of the second flow stage.
4. The method of claim 1, wherein the concurrent processing threshold of the second process stage is dependent on at least one of a resource consumption and a time consumption of the second process stage.
5. The method of any of claims 1-4, wherein the concurrent processing threshold for the first process stage is set independently of the concurrent processing threshold for the second process stage.
6. A data concurrency processing apparatus, comprising:
The data acquisition unit is used for acquiring first concurrent processing result data of a first flow stage, which is output by the first flow stage, wherein the first concurrent processing result data is obtained by performing concurrent processing of the first flow stage on data to be processed in the first flow stage according to a concurrent processing threshold of the first flow stage, and the concurrent processing threshold of the first flow stage refers to the maximum number of data objects allowed to be processed by each thread concurrently executed in the first flow stage;
the concurrency control unit is used for carrying out concurrency processing of the second flow stage on the first concurrency processing result data according to a concurrency processing threshold value of the second flow stage so as to obtain second concurrency processing result data of the second flow stage, wherein the concurrency processing threshold value of the second flow stage refers to the maximum number of data objects allowed to be processed by each thread executed in concurrency in the second flow stage;
and the data output unit is used for outputting the second concurrent processing result data.
7. The apparatus according to claim 6, wherein the concurrency control unit is specifically configured to
According to the concurrency processing threshold value of the second process stage, carrying out grouping processing on all the obtained first concurrency processing result data so as to obtain at least one data grouping; and
and carrying out concurrent processing of the second flow stage on the first concurrent processing result data in each data packet in the at least one data packet to obtain processing result data of each data packet, wherein the processing result data is used as second concurrent processing result data of the second flow stage.
8. The apparatus according to claim 6, wherein the concurrency control unit is specifically configured to
Acquiring first concurrent processing result data which is acquired in real time and does not undergo concurrent processing in the second flow stage; and
and if the number of the first concurrent processing result data which is not subjected to the concurrent processing of the second flow stage is greater than or equal to the concurrent processing threshold of the second flow stage, performing the concurrent processing of the second flow stage on the first concurrent processing result data of the concurrent processing threshold of the second flow stage in the first concurrent processing result data which is not subjected to the concurrent processing of the second flow stage, so as to obtain second concurrent processing result data of the second flow stage.
9. The apparatus of claim 6, wherein the concurrent processing threshold for the second process stage is dependent on at least one of a resource consumption and a time consumption of the second process stage.
10. The apparatus of any of claims 6-9, wherein the concurrent processing threshold for the first flow stage is set independently of the concurrent processing threshold for the second flow stage.
11. An electronic device, comprising:
at least one processor; and
a memory communicatively coupled to the at least one processor; wherein,,
the memory stores instructions executable by the at least one processor to enable the at least one processor to perform the method of any one of claims 1-5.
12. A non-transitory computer readable storage medium storing computer instructions for causing the computer to perform the method of any one of claims 1-5.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911310868.0A CN111176797B (en) | 2019-12-18 | 2019-12-18 | Data concurrency processing method and device, electronic equipment and readable storage medium |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN201911310868.0A CN111176797B (en) | 2019-12-18 | 2019-12-18 | Data concurrency processing method and device, electronic equipment and readable storage medium |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN111176797A CN111176797A (en) | 2020-05-19 |
| CN111176797B true CN111176797B (en) | 2023-10-27 |
Family
ID=70655545
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN201911310868.0A Active CN111176797B (en) | 2019-12-18 | 2019-12-18 | Data concurrency processing method and device, electronic equipment and readable storage medium |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN111176797B (en) |
Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1868224A (en) * | 2003-08-13 | 2006-11-22 | 北方电讯网络有限公司 | Method, system and program product for indicating concurrent service capability with enhanced precision |
| CN102110079A (en) * | 2011-03-07 | 2011-06-29 | 杭州电子科技大学 | Tuning calculation method of distributed conjugate gradient method based on MPI |
| CN102779062A (en) * | 2011-05-09 | 2012-11-14 | 阿里巴巴集团控股有限公司 | Method and device for controlling number of threads accessing system resources |
| CN104394353A (en) * | 2014-10-14 | 2015-03-04 | 浙江宇视科技有限公司 | Video compression method and device |
| CN105450445A (en) * | 2015-11-17 | 2016-03-30 | 武汉日电光通信工业有限公司 | High-performance alarm processing system under large capacity packet transmission system and method thereof |
| CN108206854A (en) * | 2016-12-20 | 2018-06-26 | 广州市动景计算机科技有限公司 | Browser downloading task management method, device and user terminal |
| CN110515710A (en) * | 2019-08-06 | 2019-11-29 | 深圳市随手科技有限公司 | Asynchronous task scheduling method, apparatus, computer equipment and storage medium |
Family Cites Families (3)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| JP5900061B2 (en) * | 2012-03-19 | 2016-04-06 | 富士通株式会社 | Test method, test apparatus and program |
| CN105159654B (en) * | 2015-08-21 | 2018-06-12 | 中国人民解放军信息工程大学 | Integrity measurement hashing algorithm optimization method based on multi-threaded parallel |
| CN109542600B (en) * | 2018-11-15 | 2020-12-25 | 口碑(上海)信息技术有限公司 | Distributed task scheduling system and method |
-
2019
- 2019-12-18 CN CN201911310868.0A patent/CN111176797B/en active Active
Patent Citations (7)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN1868224A (en) * | 2003-08-13 | 2006-11-22 | 北方电讯网络有限公司 | Method, system and program product for indicating concurrent service capability with enhanced precision |
| CN102110079A (en) * | 2011-03-07 | 2011-06-29 | 杭州电子科技大学 | Tuning calculation method of distributed conjugate gradient method based on MPI |
| CN102779062A (en) * | 2011-05-09 | 2012-11-14 | 阿里巴巴集团控股有限公司 | Method and device for controlling number of threads accessing system resources |
| CN104394353A (en) * | 2014-10-14 | 2015-03-04 | 浙江宇视科技有限公司 | Video compression method and device |
| CN105450445A (en) * | 2015-11-17 | 2016-03-30 | 武汉日电光通信工业有限公司 | High-performance alarm processing system under large capacity packet transmission system and method thereof |
| CN108206854A (en) * | 2016-12-20 | 2018-06-26 | 广州市动景计算机科技有限公司 | Browser downloading task management method, device and user terminal |
| CN110515710A (en) * | 2019-08-06 | 2019-11-29 | 深圳市随手科技有限公司 | Asynchronous task scheduling method, apparatus, computer equipment and storage medium |
Non-Patent Citations (2)
| Title |
|---|
| Reachability in Binary Multithreaded Programs Is Polynomial;Malkis等;《37TH INTERNATIONAL CONFERENCE ON DISTRIBUTED COMPUTING SYSTEMS (ICDCS 2017)》;2083-2088 * |
| 多线程编程模型的研究与应用;吴丹等;《广东工业大学学报》(第1期);47-49 * |
Also Published As
| Publication number | Publication date |
|---|---|
| CN111176797A (en) | 2020-05-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN110806923B (en) | Parallel processing method and device for block chain tasks, electronic equipment and medium | |
| CN111967568B (en) | Adaptation method and device for deep learning model and electronic equipment | |
| CN111046257B (en) | Session recommendation method and device and electronic equipment | |
| CN111597028B (en) | Method and device for task scheduling | |
| CN111967569B (en) | Neural network structure generation method and device, storage medium and electronic equipment | |
| CN111177476B (en) | Data query method, device, electronic equipment and readable storage medium | |
| CN110706147B (en) | Image processing environment determination method, device, electronic equipment and storage medium | |
| CN111611449B (en) | Information encoding method, apparatus, electronic device, and computer-readable storage medium | |
| CN111241234B (en) | Text classification method and device | |
| CN111949272B (en) | Compiling optimization method and device for hosted application, electronic equipment and readable storage medium | |
| CN112527474B (en) | Task processing method and device, equipment, readable medium and computer program product | |
| CN111178937B (en) | User rewarding method and device for application, electronic equipment and readable storage medium | |
| CN111782181A (en) | Code generation method and device, electronic equipment and storage medium | |
| CN112446574B (en) | Product evaluation method, device, electronic equipment and storage medium | |
| CN112329919B (en) | Model training method and device | |
| CN111738325B (en) | Image recognition method, device, equipment and storage medium | |
| CN111176797B (en) | Data concurrency processing method and device, electronic equipment and readable storage medium | |
| CN111177479B (en) | Method and device for acquiring feature vector of node in relational network graph | |
| CN111176838B (en) | Method and device for distributing embedded vector to node in bipartite graph | |
| CN111125597B (en) | Webpage loading method, browser, electronic equipment and storage medium | |
| CN111767059A (en) | Deployment method and device of deep learning model, electronic equipment and storage medium | |
| CN111694648A (en) | Task scheduling method and device and electronic equipment | |
| CN111857825A (en) | Instruction execution method and device, electronic equipment and computer-readable storage medium | |
| CN111292223B (en) | Graph calculation processing method and device, electronic equipment and storage medium | |
| CN111783872B (en) | Method, device, electronic equipment and computer readable storage medium for training model |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |