CN114924877B - Dynamic allocation calculation method, device and equipment based on data stream - Google Patents
Dynamic allocation calculation method, device and equipment based on data stream Download PDFInfo
- Publication number
- CN114924877B CN114924877B CN202210538816.4A CN202210538816A CN114924877B CN 114924877 B CN114924877 B CN 114924877B CN 202210538816 A CN202210538816 A CN 202210538816A CN 114924877 B CN114924877 B CN 114924877B
- Authority
- CN
- China
- Prior art keywords
- processed
- task
- data
- result
- services
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Active
Links
Classifications
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F9/00—Arrangements for program control, e.g. control units
- G06F9/06—Arrangements for program control, e.g. control units using stored programs, i.e. using an internal store of processing equipment to receive or retain programs
- G06F9/46—Multiprogramming arrangements
- G06F9/50—Allocation of resources, e.g. of the central processing unit [CPU]
- G06F9/5005—Allocation of resources, e.g. of the central processing unit [CPU] to service a request
- G06F9/5027—Allocation of resources, e.g. of the central processing unit [CPU] to service a request the resource being a machine, e.g. CPUs, Servers, Terminals
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06F—ELECTRIC DIGITAL DATA PROCESSING
- G06F2209/00—Indexing scheme relating to G06F9/00
- G06F2209/50—Indexing scheme relating to G06F9/50
- G06F2209/5011—Pool
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D10/00—Energy efficient computing, e.g. low power processors, power management or thermal management
Abstract
The invention relates to a dynamic allocation calculation method, a device and equipment based on data flow, which comprise the steps of obtaining flow data and a data flow diagram of a job to be processed, wherein flow nodes in the data flow diagram correspond to a plurality of services one by one, extracting tasks to be processed from the flow data to be processed, and storing the tasks to be processed into a task pool to be processed corresponding to initial services of the data flow diagram; deploying all services corresponding to the dataflow graph to each of a plurality of processing units; invoking one of a plurality of processing units to process a task to be processed, and generating a new task to be processed; and judging that the new task to be processed is a result task to be processed, calling a processing unit to process the result task to be processed, generating result data and outputting the result data. Each processing unit in the invention can support all services related to a complete stream data processing flow, can schedule the processing units in real time, fully utilizes system resources and realizes high-efficiency processing of stream big data.
Description
Technical Field
The present invention relates to the field of computer technologies, and in particular, to a method, an apparatus, and a device for dynamic allocation calculation based on data flow.
Background
The data flow technology is different from the existing mainstream system architecture and programming mode based on control flow, data is used as a drive to generate tasks, and a processing unit carries out corresponding processing according to task logic corresponding to the data, generates new data and generates new tasks. The task corresponding to the data is used as a scheduling unit, and the data flow technology can realize flexible resource scheduling. However, on the one hand, the existing data stream scheduling method needs to preset the processing logic of each processing module, and then the data stream is sent to the corresponding processing module according to the required processing logic. The task scheduling and resource allocation mode limits the resource utilization elasticity and efficiency of the processing module, and cannot exert the resource capacity of the whole processing module system to the maximum extent. On the other hand, whether it is a single machine or a cluster, the processing resources generally have isomerism, such as a single machine or a cluster generally comprises a CPU and a GPU, and may further comprise other professional processing modules, such as artificial intelligent chips, chilies, TPU, and the like. Different processing modules are good at different processing tasks, such as CPU is good at complex logic processing or high-precision floating point operations, GPU is good at parallel processing of massive data, processing of pictures, etc., TPU is good at processing of machine learning tasks, etc. The existing task allocation algorithm does not consider the resource heterogeneity, so the processing efficiency for stream data is not high enough.
Disclosure of Invention
In view of the foregoing, it is desirable to provide a method for dynamic allocation computation based on data streams, and to provide a computing device and/or apparatus based on dynamic allocation accordingly.
A data stream based dynamic allocation computing method, adapted to be executed in a computing device, comprising the steps of:
obtaining stream data to be processed and a data flow graph, wherein flow nodes in the data flow graph correspond to a plurality of services one by one, extracting tasks to be processed from the stream data to be processed, and storing the tasks to be processed into a task pool to be processed corresponding to initial services in the data flow graph;
deploying all services corresponding to the dataflow graph to each of a plurality of processing units; invoking one of a plurality of processing units to process the task to be processed, and generating a new task to be processed;
and judging that the new task to be processed is a task to be processed result, calling a processing unit to process the task to be processed result, generating result data and outputting the result data.
The invention also provides a computing device based on dynamic allocation, which comprises:
the preprocessing module is used for acquiring stream data to be processed and a data flow graph, wherein flow nodes in the data flow graph correspond to a plurality of services one by one, extracting tasks to be processed from the stream data to be processed, and storing the tasks to be processed into a task pool to be processed, which corresponds to initial services of the data flow graph;
the distribution module is used for deploying all the services corresponding to the data flow graph to each of the processing units; invoking one of a plurality of processing units to process the task to be processed, and generating a new task to be processed;
and the output module is used for judging that the new task to be processed is a task to be processed result, calling the processing unit to process the task to be processed result, generating result data and outputting the result data.
The invention also provides a computing device based on dynamic allocation, comprising a processor and a memory for storing processor executable instructions; wherein the processor is configured to:
acquiring stream data to be processed and a data flow graph, automatically deploying all services involved in the data flow graph, wherein flow nodes in the data flow graph correspond to a plurality of services one by one, extracting tasks to be processed from the stream data to be processed, and storing the tasks to be processed into a task pool to be processed corresponding to initial services of the data flow graph;
deploying all services corresponding to the dataflow graph to each of a plurality of processing units; invoking one of a plurality of processing units to process the task to be processed, and generating a new task to be processed;
and judging that the new task to be processed is a task to be processed result, calling a processing unit to process the task to be processed result, generating result data and outputting the result data.
The dynamic allocation calculation method and the dynamic allocation calculation equipment based on the data flow are characterized in that the flow data are divided into tasks to be processed one by one according to the logic shown by the data flow diagram, the tasks to be processed are allocated to a service task pool corresponding to the preset data flow diagram, the tasks to be processed are then transferred into proper processing units for processing according to a preset scheduling algorithm, each processing unit can support all services related to a complete flow data processing flow, therefore, the processing units are not bound with single service, but have the characteristic of full service, and a task scheduling and resource management module can perform real-time scheduling according to the resource characteristics and the resource states of the processing units required by the tasks to be processed all the time so as to exert the processing capability characteristics of different processing units, fully utilize the system resources and realize the efficient processing of the flow large data.
Drawings
FIG. 1 is a flow chart of a method for dynamic allocation calculation based on data flow in an embodiment of the invention;
FIG. 2 is a flowchart illustrating a method according to an embodiment of the invention in step S100;
FIG. 3 is a flowchart illustrating a method in step S200 according to an embodiment of the present invention;
FIG. 4 is a flowchart illustrating a method in step S210 according to an embodiment of the present invention;
fig. 5 is a schematic structural diagram of a dynamic allocation computing device based on data flow according to an embodiment of the present invention.
Detailed Description
In order that the invention may be readily understood, a more complete description of the invention will be rendered by reference to the appended drawings. Preferred embodiments of the present invention are shown in the drawings. This invention may, however, be embodied in many different forms and should not be construed as limited to the embodiments set forth herein. Rather, these embodiments are provided so that this disclosure will be thorough and complete.
In one embodiment of the present invention, a dynamic allocation computing method based on data flow is provided, which is suitable for being executed in a computing device, as shown in fig. 1, and includes the following steps S100 to S300:
step S100: obtaining stream data to be processed and a data flow graph, wherein flow nodes in the data flow graph are in one-to-one correspondence with a plurality of services, extracting tasks to be processed from the stream data to be processed, and storing the tasks to be processed into a task pool to be processed corresponding to initial services in the data flow graph. Before a stream data processing job is performed, a data flow graph, i.e., a predetermined processing flow for stream data to be processed, is first designed according to data processing requirements. The setting of the data flow diagram or the predetermined flow is known to those skilled in the art related to the data flow, and will not be described herein. The data flow graph comprises a plurality of flow nodes with a certain sequence, and each flow node corresponds to a data processing service and can be designed through different programming languages. After the above work is ready, the stream data to be processed can be directly accessed and sorted according to a preset input format, and the tasks to be processed are divided one by one, as shown in fig. 2, the specific steps include steps S110 to S120:
step S110: and arranging stream data to be processed according to a preset input format, and sequentially generating a plurality of key value pair data.
Step S120: and adding initial service labels to the preprocessed key values one by one to generate tasks to be processed, wherein the initial service labels are used for designating initial services. A service tag is an information carrier carrying context information. And when the initial service label is added in the data to be processed, the task to be processed is classified into a task pool to be processed corresponding to the initial service in the data flow diagram. The pool of tasks to be processed can exist in a memory in a queue.
Therefore, once the preset flow for processing certain stream data is determined, a series of tasks to be processed can be automatically generated one by one after the stream data is processed by a specific input module, and the tasks to be processed enter a task pool to be processed corresponding to the initial service in sequence. For example, a video stream is decomposed into a plurality of picture frames according to a preset input format through a data input module, each picture frame is processed according to an initial service indicated by a first flow node in the flow chart, after the initial service processes the picture frames according to a preset requirement, an output result of the initial service is transferred to other subsequent services as input for continuous processing, namely a new task to be processed is formed and a task pool to be processed of the corresponding service is entered. And the method is repeated until the data reaches the end service according to the logic shown in the data flow diagram and the final result is output.
Step S200: deploying all services corresponding to the dataflow graph to each of a plurality of processing units; and calling one of the processing units to process the task to be processed, and generating a new task to be processed. After submitting the job from the user, all the services related in the data flow graph are automatically deployed on each processing unit, namely the deployment characteristic of the full service, specifically, each service comprises a program set for processing the task to be processed, the program set is packed into containers according to a standard packing format, and the containers corresponding to all the services are allocated on each processing unit according to a standard container deployment method to form the micro-service. Each micro-service is assigned an IP address and Port (< IP: port >), and the system uses the < IP: port > of each micro-service as an identifier to allocate a task to be processed or a new task to be processed, i.e., a specific way of performing full service deployment on the processing unit. Preferably, the containers are associated with processing unit types, and the data processor is divided into a first processing container and a second processing container according to the program set and the system resource characteristics, for example, when the system has a CPU and a GPU at the same time, the CPU container and the GPU container are respectively generated by packaging according to the characteristics. In other words, the program of each flow node and the environment required for running the program are made into a container according to the generation mode of a standard container, and are delivered to a container manager (such as Kubernetes) for management. To achieve full service deployment, the number of boot container image copies is specified to be equal to the number of system physical resources, i.e., for each container, by specifying a number of container image copies equal to the number of corresponding processing units, e.g., if a machine has 8 CPUs, then all of the service containers involved in the dataflow graph boot 8 copies.
After the processing, the processing may enter another processing stage according to the set flow, and other data results are generated, so that a new task to be processed is formed after the processing, as shown in fig. 3, the specific process includes steps S210 to S220:
step S210: and calling the task to be processed in the task pool to be processed, and starting the processing unit to process the task to be processed and generate output data. The task scheduling and resource management module is used for scheduling the task to be processed in the task pool to be processed, distributing the task to be processed to a proper processing unit according to the state of the real-time processing unit and the resource demand characteristic of the task to be processed, and processing the data in the task to be processed according to the service required by the task to be processed by the processing unit and generating output data.
As a preferred solution, as shown in fig. 4, the detailed steps of the task scheduling and resource management module calling processing unit to process the task to be processed include steps S211 to S213:
step S211: and acquiring the real-time utilization rate of a plurality of processing units and the resource demand characteristics of the task to be processed or the task to be processed as a result. The task scheduling and resource management module contains predefined physical resource information including the physical resource type and/or physical resource quantity, such as the number of CPUs and/or the number of GPUs. The predefined physical resources generally need to be less than or equal to the physical resource quantity that can be provided by the system, and the system forms a physical resource pool and initializes the resource state of each processing unit according to the specification of related personnel. The system checks the busy state and processing power of the processing unit before processing, ready for allocation.
Step S212: and determining an actual processing unit for processing the task to be processed or the task to be processed as a result according to the resource scheduling algorithm.
Step S213: and calling the task to be processed or the task to be processed result into the actual processing unit for processing. And updating the resource state of the actual processing unit, and reporting and updating the resource state to the task scheduling and resource management module after the processing unit finishes task processing.
The overall idea of step S211 to step S213 is to determine the real-time utilization rate of each of the plurality of processing units and the resource status of each processing unit, select the processing unit with the lowest real-time utilization rate and capable of supporting the task to be processed, add the standby processing unit, and determine the resource demand characteristic of the task to be processed corresponding to a certain service in the data flow diagram to be processed, for example, if the processing of the picture is involved, the GPU is more suitable, if the calculation of the artificial intelligence is involved, the chip (such as the junior and TPU) supporting the artificial intelligence is more suitable, and if the complex logic operation or the high-precision operation is involved, the CPU is more suitable. And combining the two characteristics, selecting an optimal processing module from a plurality of processing units according to a resource scheduling algorithm, and taking the optimal processing module as an actual processing unit for processing stream data to be processed.
In principle, the task scheduling and resource management module automatically analyzes (Profiling) the load of each processing unit for different services, if the CPU1 can support at most 2 tasks corresponding to S1 services, 3 tasks corresponding to S2 services, 4 tasks corresponding to S3, and 1 task corresponding to S4 services at the same time, the initial resource state of the CPU1 is described as a vector<2，3，4，1>Other processing units (CPU or GPU, etc.) may be pushed in this manner. Initializing the resource state description of all processing units according to the Profiling result, and initializing the resource utilization rate to be 0. After the actual task is allocated, the resource utilization and resource status description of the allocated processing unit are updated. Assuming that the initial resource state is described as<r 1 ,r 2 ,r 3 ,…,r S >When a certain service S is allocated to the processing unit i After the corresponding task to be processed, the resource utilization rate is updated to be p=p+1/r i Accordingly, the resource status description thereof is updated as: wherein->Representation pair r S (1-p) rounding down. For example when the initial resource state is<2,3,4,1>After the processing unit of the processing unit allocates a task to be processed corresponding to the service S1, the resource utilization rate of the processing unit is updated to 0.5, and the corresponding resource state is updated to<1,1,2,0>The processing unit will no longer be suitable for assigning to it the task to be processed corresponding to the service S4. The system initializes the task pool to be processed of each service related to the data flow diagram, allocates memory space for each service and builds a first-in first-out queue as the task pool to be processed.
Step S220: and adding a new service tag to the output data to form a new task to be processed, wherein the new service tag is used for designating intermediate service or terminal service in a plurality of services.
According to the pre-allocated initial service labels, the tasks to be processed are stored in a task pool to be processed corresponding to the initial service in the data flow diagram, along with the initial allocation processing of the streaming data, the tasks to be processed carrying the initial service labels are continuously stored in the task pool to be processed corresponding to the initial service, the tasks to be processed are transferred into a processing unit according to a first-in first-out sequence for processing, new service labels are added to the processed data to form new tasks to be processed, the new service labels are used for corresponding to other services possibly existing in the flow diagram, and the new service labels are stored in task pools to be processed corresponding to the services, and of course, the storing is performed according to the pre-set sequence of the data flow diagram, so that the tasks to be processed corresponding to the next service determined according to the preset flow diagram in the follow-up processing are conveniently carried out according to the need.
When a certain processing unit processes the task allocated to the processing unit, a new task to be processed is generated according to a specified processing flow, and the task scheduling and resource management module is required to report the task, the utilization rate of the processing unit and the corresponding resource state description are updated, and the specific process principle is as described above.
Step S300: and judging that the new task to be processed is a task to be processed result, calling the processing unit to process the task to be processed result, generating result data and outputting the result data.
After a new task to be processed is generated, the new task to be processed needs to be identified and judged to be classified, if the processed task to be processed is a task to be processed result, the processed result is directly output as a final result, and the specific process is as follows: and if the new task pool to be processed is judged to be the task pool to be processed corresponding to the terminal service in the plurality of services, confirming that the new task to be processed is a task with a result to be processed. It can be further subdivided into steps S310 to S330:
step S310: and acquiring a new task to be processed. The new task to be processed is obtained from a task pool to be processed corresponding to the service required by storing the task to be processed.
Step S320: judging that the service label in the new task to be processed is matched with the attribute information of the end node of the flow in the data flow graph. The data flow graph comprises a plurality of flow nodes with specific sequences, such as an initial flow node, an intermediate flow node and an end flow node, wherein each node has own attribute information which is predetermined based on a data flow technology and is stored in a corresponding node position.
Step S330: and determining the new task to be processed as a result to be processed. When it is determined in step S320 that the new task to be processed comes from the end node of the flow, it is indicated that the new task to be processed is already at the end of the dataflow graph, and no new service processing is needed, so that the new task to be processed is used as the task to be processed as the result.
As a preferable scheme, when judging that the new task to be processed is a task with a non-processing result, namely, the data flow diagram does not correspond to only two services (an initial service and an end service), storing the new task to be processed into a task pool to be processed corresponding to an intermediate service in a plurality of services. Judging that the new task to be processed is not a task with a result to be processed, that is, the generated new task to be processed is not generated by the last node in the preset flow, putting the new task to be processed into a task pool again by the system, and enabling the task pool to correspond to the intermediate service to repeatedly perform the step S200, namely, calling the processing unit again to process the new task to be processed to generate an updated task to be processed until the system judges that the new task with the result to be processed is generated. Of course, the processing unit for processing the new task to be processed is not necessarily the processing unit for processing the original task to be processed, and since each processing unit can process all services corresponding to the data flow graph, other processing units can be called for processing according to the related selection rule.
As another preferable scheme, the task pools to be processed are in one-to-one correspondence with preset priority tokens, the preset priority tokens are used for identifying the processing sequence priority of the task pools to be processed, and the tasks to be processed in the task pools to be processed are called according to the priority tokens to enter the processing unit for processing. With the deep task processing, as the stream data enters the processing flow of the data flow diagram, the task to be processed is continuously generated to enter the task pool to be processed corresponding to the initial service, a task queue is formed in the task pool to be processed corresponding to the initial service, the processing unit processes the task to be processed to form a new task to be processed, the new task enters the service corresponding to the second flow node in the data flow diagram, the service is stored in the task pool to be processed corresponding to the service, and so on, the task queues are formed in the task pools to be processed of the services corresponding to the subsequent flow nodes such as the third flow node, the fourth flow node and the like. The processing efficiency affecting the whole system is the maximum, namely the length of a task queue in a to-be-processed task pool of the service corresponding to the endmost flow node, so that the to-be-processed task pool of the service corresponding to each flow node is given priority, and the to-be-processed tasks stored in the corresponding to-be-processed task pools are timely processed according to the priority, so that the stream data processing efficiency can be better improved. Preferably, the terminal service is given the corresponding task pool to be processed with the highest priority, and the tasks to be processed in the task pool to be processed are processed preferentially.
Through the processing, the resources in the system can be fully invoked, and as each processing unit can process all the services in the data flow diagram, the defect that each processing unit is required to preset a single service in the traditional data flow technology and data is sent to the corresponding processing unit according to the required service is overcome, all the services corresponding to all the processes in the data flow diagram can be processed by each processing unit based on the full-service data flow processing technology, resources can be allocated and fully utilized according to the requirements, and then real-time processing of tasks is realized, so that stream data with heterogeneous characteristics can be flexibly and efficiently processed.
Based on the idea of the computer system, the invention also provides a computing device based on dynamic allocation, as shown in fig. 5, which comprises:
the preprocessing module 10 is configured to obtain stream data to be processed and a data flow graph, wherein flow nodes in the data flow graph correspond to a plurality of services one by one, extract tasks to be processed from the stream data to be processed, and store the tasks to be processed into a task pool to be processed corresponding to an initial service of the data flow graph. A user submits a job, designates a streaming data source (such as an RTSP address of a video stream), acquires streaming data (such as a video stream, sensor data, a user access log and the like which are continuously color-generated along with time) to be processed by the job, extracts a task to be processed from the streaming data to be processed by the preprocessing module 10, and stores the task to be processed into a task pool to be processed corresponding to an initial service in the data flow diagram.
The allocation module 20 is used for calling one of the processing units to process a task to be processed and generating a new task to be processed; all services corresponding to the dataflow graph are deployed on each processing unit.
And the output module 30 is used for judging that the new task to be processed is a task to be processed result, calling the processing unit to process the task to be processed result, generating result data and outputting the result data.
It should be emphasized that the allocation module includes an allocation unit, where the allocation unit is configured to invoke the processing unit to process the task to be processed or the task to be processed as a result, and specifically includes steps S211 to S213.
The implementation process of the execution method between the modules and/or units is the same as the dynamic allocation calculation method based on the data stream, and the description is not repeated.
Meanwhile, the invention also provides a computing device based on dynamic allocation, which comprises a processor and a memory for storing executable instructions of the processor, wherein the processor is configured to execute steps S100-S300 in the computing method based on dynamic allocation of data streams:
step S100: obtaining stream data to be processed and a data flow graph, wherein flow nodes in the data flow graph are in one-to-one correspondence with a plurality of services, extracting tasks to be processed from the stream data to be processed, and storing the tasks to be processed into a task pool to be processed corresponding to initial services of the data flow graph.
Step S200: deploying all services corresponding to the dataflow graph to each of a plurality of processing units; and calling one of the processing units to process the task to be processed, and generating a new task to be processed.
Step S300: and judging that the new task to be processed is a task to be processed result, calling the processing unit to process the task to be processed result, generating result data and outputting the result data.
For a better understanding of the present invention, one example of an application is set forth below:
a section of video data is accessed into a system through a video streaming RTSP address, the system comprises a data flow diagram preset by a programmer and a preset input format, the data flow diagram comprises a flow node 1, a flow node 2, a flow node 3 and a flow node 4, services corresponding to the flow nodes are S1, S2, S3 and S4, each service comprises a specific processing program, for example, S1 is used for extracting a specific object in a picture, S2 is used for distinguishing one type of object in the specific object, S3 is used for distinguishing one type of object, and S4 is used for sending distinguishing result information. Each service corresponds to a task pool to be processed, which is defined as V1, V2, V3 and V4, and is used for storing the task to be processed of each node. The system includes several processing units including a CPU and/or a GPU. Based on the deployment strategy of the full service, the processing programs of the S1, the S2, the S3 and the S4 are deployed in each processing unit in a micro-service mode, so that each processing unit can provide the services S1 to S4 and can be started according to the requirements of the tasks to be processed. The video data is processed by a preprocessing module to form frame picture data, an initial service tag is added in the data, a task M1 to be processed is continuously formed, M1 is stored in a task pool V1 corresponding to the task S1 to form an M1 queue, according to a first-in first-out rule, M1 is called out and enters a proper processing unit, a new service tag is added after processing to form M2, it is judged that the V2 is not the terminal service, the task pool V2 corresponding to the task S2 is stored, the processing is continued, the new service tag is added, the task pool V2 is formed into M3 and M4 respectively, finally M4 is stored in the V4, the system judges that the M4 carries a service tag pointing to the V4, the task pool V4 is determined to be a task pool corresponding to the terminal service, then the M4 is confirmed to be a task as a result to be processed, finally the proper processing unit is called to process the M4 and output, and a result of a specific object is found from a section of video stream. Each time the processing unit is used for processing the task to be processed, the most suitable processing unit before processing is selected, and each processing unit can process S1-S4 and can be selected, but the utilization rate, the resource quantity and the processing characteristics of each processing unit are different and are required to be actually determined according to the related calling algorithm. When a large amount of video data is processed, a waiting task queue with a certain length is stored in each waiting task pool, and the waiting tasks in V4 are preferentially processed, so that the waiting tasks in V3 are stored in V4 after being processed, and the result can be output faster.
The apparatus or module set forth in the above embodiments may be implemented in particular by a computer chip or entity, or by a product having a certain function. A typical implementation device is a computer, which may be in the form of a personal computer, laptop computer, cellular telephone, camera phone, smart phone, personal digital assistant, media player, navigation device, email device, game console, tablet computer, wearable device, or a combination of any of these devices.
One skilled in the art will appreciate that one or more embodiments of the present description may be provided as a method, system, or computer program product. Accordingly, one or more embodiments of the present description may take the form of an entirely hardware embodiment, an entirely software embodiment or an embodiment combining software and hardware aspects. Moreover, one or more embodiments of the present description can take the form of a computer program product on one or more computer-usable storage media (including, but not limited to, disk storage, CD-ROM, optical storage, etc.) having computer-usable program code embodied therein.
These computer program instructions may also be stored in a computer-readable memory that can direct a computer or other programmable data processing apparatus to function in a particular manner, such that the instructions stored in the computer-readable memory produce an article of manufacture including instruction means which implement the function specified in the flowchart flow or flows and/or block diagram block or blocks.
These computer program instructions may also be loaded onto a computer or other programmable data processing apparatus to cause a series of operational steps to be performed on the computer or other programmable apparatus to produce a computer implemented process such that the instructions which execute on the computer or other programmable apparatus provide steps for implementing the functions specified in the flowchart flow or flows and/or block diagram block or blocks.
It should also be noted that the terms "comprises," "comprising," or any other variation thereof, are intended to cover a non-exclusive inclusion, such that a process, method, article, or apparatus that comprises a list of elements does not include only those elements but may include other elements not expressly listed or inherent to such process, method, article, or apparatus. Without further limitation, an element defined by the phrase "comprising one … …" does not exclude the presence of other like elements in a process, method, article or apparatus that comprises the element.
One or more embodiments of the present specification may be described in the general context of computer-executable instructions, such as program modules, being executed by a computer. Generally, program modules include routines, programs, objects, components, data structures, etc. that perform particular tasks or implement particular abstract data types. One or more embodiments of the present description may also be practiced in distributed computing environments where tasks are performed by remote processing devices that are linked through a communications network. In a distributed computing environment, program modules may be located in both local and remote computer storage media including memory storage devices.
In this specification, each embodiment is described in a progressive manner, and identical and similar parts of each embodiment are all referred to each other, and each embodiment mainly describes differences from other embodiments. In particular, for the server device embodiment, since it is substantially similar to the method embodiment, the description is relatively simple, and the relevant points are referred to in the description of the method embodiment.
The foregoing describes specific embodiments of the present disclosure. Other embodiments are within the scope of the following claims. In some cases, the actions or steps recited in the claims can be performed in a different order than in the embodiments and still achieve desirable results. In addition, the processes depicted in the accompanying figures do not necessarily require the particular order shown, or sequential order, to achieve desirable results. In some embodiments, multitasking and parallel processing are also possible or may be advantageous.
The foregoing description of the preferred embodiment(s) is merely illustrative of one or more embodiments of the present disclosure and is not intended to limit the disclosure to the details shown, but on the contrary, the intention is to cover all modifications, equivalents, alternatives, and alternatives falling within the spirit and principles of the disclosure.
Claims (9)
1. A method of data stream based dynamic allocation computation, adapted to be executed in a computing device, comprising the steps of:
obtaining stream data to be processed and a data flow graph, wherein flow nodes in the data flow graph correspond to a plurality of services one by one, extracting tasks to be processed from the stream data to be processed, and storing the tasks to be processed into a task pool to be processed corresponding to initial services in the data flow graph;
deploying all services corresponding to the dataflow graph to each of a plurality of processing units;
invoking one of a plurality of processing units to process the task to be processed, and generating a new task to be processed;
judging that the new task to be processed is a task to be processed result, calling a processing unit to process the task to be processed result, generating result data and outputting the result data;
the step of calling the processing unit to process the task to be processed or the task to be processed result specifically comprises the following steps:
acquiring the real-time utilization rate of a plurality of processing units and the resource demand characteristics of the task to be processed or the result task to be processed;
determining an actual processing unit for processing the task to be processed or the task to be processed result according to a resource scheduling algorithm;
and calling the task to be processed or the task to be processed result into the actual processing unit for processing.
2. The method for computing data flow-based dynamic allocation according to claim 1, wherein the step of generating new tasks to be processed further comprises:
judging that the new task to be processed is a non-result task to be processed, and storing the new task to be processed into a task pool to be processed corresponding to an intermediate service in the plurality of services;
and calling a processing unit to process the new task to be processed, and generating an updated task to be processed.
3. The method for dynamically allocating and calculating based on data stream according to claim 1, wherein the process of judging that the new task to be processed is a task to be processed result is specifically as follows:
judging that the new task pool to be processed is the task pool to be processed corresponding to the terminal service in the plurality of services, and confirming that the new task to be processed is a task to be processed as a result.
4. The method for dynamically allocating data flows according to claim 1, wherein the step of deploying all services corresponding to the dataflow graph to each of the plurality of processing units comprises:
each service comprises a program set for processing the task to be processed or the new task to be processed, the system resource demand characteristic corresponding to each program set is obtained, and the program sets and the system resource demand characteristics are packed into a container according to a standard packing format;
allocating containers corresponding to all services to each processing unit to form micro services by designating the number of container mirror image copies equal to the number of corresponding processing units for each container;
and designating the IP address and the port of the micro service, and distributing the task to be processed or the new task to be processed to the micro service according to the IP address and the port.
5. The method for computing dynamic allocation based on data stream according to claim 1, wherein the step of extracting the task to be processed from the stream to be processed data specifically comprises:
arranging the stream data to be processed according to a preset input format, and sequentially generating a plurality of key value pair data;
and adding initial service labels to the data one by one according to the key values, and generating tasks to be processed one by one, wherein the initial service labels are used for designating the initial services.
6. The method for dynamically allocating and calculating a data stream according to claim 1, wherein the step of calling the processing unit to process the task to be processed and generating a new task to be processed specifically comprises:
invoking a task to be processed in the task pool to be processed, and enabling a processing unit to process the task to be processed and generate output data;
and adding a new service tag to the output data to form a new task to be processed, wherein the new service tag is used for designating intermediate service or terminal service in a plurality of services.
7. The method for dynamically allocating and calculating data flow according to claim 1, wherein the task pools to be processed are in one-to-one correspondence with preset priority tokens, the preset priority tokens are used for identifying the processing sequence priorities of the task pools to be processed, and the tasks to be processed in the task pools to be processed are called according to the priority tokens to enter the processing unit for processing.
8. A data flow based dynamic allocation computing device for implementing the data flow based dynamic allocation computing method according to any one of claims 1 to 7, comprising:
the preprocessing module is used for acquiring stream data to be processed and a data flow graph, wherein flow nodes in the data flow graph correspond to a plurality of services one by one, extracting tasks to be processed from the stream data to be processed, and storing the tasks to be processed into a task pool to be processed, which corresponds to initial services of the data flow graph;
the distribution module is used for deploying all the services corresponding to the data flow graph to each of the processing units; invoking one of a plurality of processing units to process the task to be processed, and generating a new task to be processed;
and the output module is used for judging that the new task to be processed is a task to be processed result, calling the processing unit to process the task to be processed result, generating result data and outputting the result data.
9. A data flow based dynamic allocation computing device for implementing the data flow based dynamic allocation computing method of any one of claims 1 to 7, comprising: a processor; a memory for storing processor-executable instructions; wherein the processor is configured to:
obtaining stream data to be processed and a data flow graph, wherein flow nodes in the data flow graph correspond to a plurality of services one by one, extracting tasks to be processed from the stream data to be processed, and storing the tasks to be processed into a task pool to be processed corresponding to initial services of the data flow graph;
deploying all services corresponding to the dataflow graph to each of a plurality of processing units; invoking one of a plurality of processing units to process the task to be processed, and generating a new task to be processed;
and judging that the new task to be processed is a task to be processed result, calling a processing unit to process the task to be processed result, generating result data and outputting the result data.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210538816.4A CN114924877B (en) | 2022-05-17 | 2022-05-17 | Dynamic allocation calculation method, device and equipment based on data stream |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210538816.4A CN114924877B (en) | 2022-05-17 | 2022-05-17 | Dynamic allocation calculation method, device and equipment based on data stream |
Publications (2)
| Publication Number | Publication Date |
|---|---|
| CN114924877A CN114924877A (en) | 2022-08-19 |
| CN114924877B true CN114924877B (en) | 2023-10-17 |
Family
ID=82809034
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210538816.4A Active CN114924877B (en) | 2022-05-17 | 2022-05-17 | Dynamic allocation calculation method, device and equipment based on data stream |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN114924877B (en) |
Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN110287016A (en) * | 2019-07-01 | 2019-09-27 | 武汉兆格信息技术有限公司 | A kind of distribution flow chart Heterogeneous Computing dispatching method |
| CN111352726A (en) * | 2018-12-24 | 2020-06-30 | 北京奇虎科技有限公司 | Streaming data processing method and device based on containerized micro-service |
| WO2020233037A1 (en) * | 2019-05-20 | 2020-11-26 | 深圳壹账通智能科技有限公司 | Data processing method and apparatus, and storage medium |
| WO2021197392A1 (en) * | 2020-04-02 | 2021-10-07 | 支付宝(杭州)信息技术有限公司 | Task queue generation |
| CN114416340A (en) * | 2021-12-15 | 2022-04-29 | 深圳先进技术研究院 | Intelligent space concurrent service flow execution method and system based on micro-service |
Family Cites Families (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| US10432532B2 (en) * | 2016-07-12 | 2019-10-01 | Cisco Technology, Inc. | Dynamically pinning micro-service to uplink port |
| CN107885594B (en) * | 2016-09-30 | 2020-06-12 | 腾讯科技（深圳）有限公司 | Distributed resource scheduling method, scheduling node and access node |
| US10320643B2 (en) * | 2017-11-10 | 2019-06-11 | Netscout Systems, Inc. | Cloud computing environment system for automatically determining over-the-top applications and services |
| CN107885590A (en) * | 2017-11-30 | 2018-04-06 | 百度在线网络技术（北京）有限公司 | Task processing method and device for smart machine |
| WO2020106973A1 (en) * | 2018-11-21 | 2020-05-28 | Araali Networks, Inc. | Systems and methods for securing a workload |
-
2022
- 2022-05-17 CN CN202210538816.4A patent/CN114924877B/en active Active
Patent Citations (5)
| Publication number | Priority date | Publication date | Assignee | Title |
|---|---|---|---|---|
| CN111352726A (en) * | 2018-12-24 | 2020-06-30 | 北京奇虎科技有限公司 | Streaming data processing method and device based on containerized micro-service |
| WO2020233037A1 (en) * | 2019-05-20 | 2020-11-26 | 深圳壹账通智能科技有限公司 | Data processing method and apparatus, and storage medium |
| CN110287016A (en) * | 2019-07-01 | 2019-09-27 | 武汉兆格信息技术有限公司 | A kind of distribution flow chart Heterogeneous Computing dispatching method |
| WO2021197392A1 (en) * | 2020-04-02 | 2021-10-07 | 支付宝(杭州)信息技术有限公司 | Task queue generation |
| CN114416340A (en) * | 2021-12-15 | 2022-04-29 | 深圳先进技术研究院 | Intelligent space concurrent service flow execution method and system based on micro-service |
Also Published As
| Publication number | Publication date |
|---|---|
| CN114924877A (en) | 2022-08-19 |
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111768006A (en) | Artificial intelligence model training method, device, equipment and storage medium | |
| US8650570B2 (en) | Method of assigning instructions in a process to a plurality of scheduler instances based on the instruction, in which each scheduler instance is allocated a set of negoitaited processor resources | |
| US20050081208A1 (en) | Framework for pluggable schedulers | |
| US11210131B2 (en) | Method and apparatus for assigning computing task | |
| CN114741207B (en) | GPU resource scheduling method and system based on multi-dimensional combination parallelism | |
| CN113535367A (en) | Task scheduling method and related device | |
| CN111861412B (en) | Completion time optimization-oriented scientific workflow scheduling method and system | |
| CN107085605B (en) | Method and device for data packaging and electronic equipment | |
| CN113301151B (en) | Low-delay containerized task deployment method and device based on cloud edge cooperation | |
| US7243354B1 (en) | System and method for efficiently processing information in a multithread environment | |
| CN110109748A (en) | A kind of hybrid language task executing method, device and cluster | |
| CN111124644B (en) | Method, device and system for determining task scheduling resources | |
| CN111258655A (en) | Fusion calculation method and readable storage medium | |
| CN114924877B (en) | Dynamic allocation calculation method, device and equipment based on data stream | |
| CN116069500A (en) | Model training task processing method and device, electronic equipment and readable medium | |
| CN111813541B (en) | Task scheduling method, device, medium and equipment | |
| CN113254200B (en) | Resource arrangement method and intelligent agent | |
| CN114327856A (en) | Data processing method and device, electronic equipment and storage medium | |
| CN112835626A (en) | Pipeline configuration method and device, readable storage medium and electronic equipment | |
| EP1630671A1 (en) | Framework for pluggable schedulers | |
| CN113254180B (en) | Data matching method and device, electronic equipment and storage medium | |
| CN117149440B (en) | Task scheduling method and device, electronic equipment and storage medium | |
| CN113778458B (en) | Data processor function development system, method and computing device | |
| CN111290850B (en) | Data storage method, device and equipment | |
| CN116541018B (en) | Distributed model compiling system, method, device, medium and equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| GR01 | Patent grant | ||
| GR01 | Patent grant |