CN114756371A - A method and system for optimal configuration of terminal edge joint resources - Google Patents

Abstract

The invention discloses a method and a system for optimizing the configuration of terminal edge joint resources. The method includes: when an edge controller records videos with different frame numbers and is input to a neural network, calculating the multiplication and summation required by the neural network and identifying the neural network complete the control task according to the video recognition request of the mobile device, and the control task includes: sending sampling frame number control information to the video sampling management module corresponding to the mobile device based on the frame number of each mobile device video sampling; The offloading decision determines whether the user's inference task is completed in the local DNN inference module or in the edge DNN inference module; based on the allocation strategy of the communication resources of each mobile video device, the time proportion of each mobile device's uplink video transmission is determined. Based on this optimization method, the average inference delay of the system and the average energy consumption of mobile devices are reduced, and the accuracy of model inference is improved.

Description

A method and system for optimal configuration of terminal edge joint resources

technical field

The invention mainly relates to the field of computer technology, and in particular relates to a method and system for optimal configuration of terminal edge joint resources.

Background technique

The development of technologies such as network, cloud computing, edge computing, and artificial intelligence has triggered people's infinite imagination of the metaverse. Augmented reality (AR) technology plays a vital role in enabling users to interact between the real world and the virtual world. At the same time, artificial intelligence plays an important role in automatic speech recognition, natural language processing, computer vision and other fields due to its learning and reasoning capabilities. With the assistance of AI technology, AR can carry out deeper scene understanding and more immersive interaction.

However, the computational complexity of artificial intelligence algorithms, especially deep neural networks (DNNs), is usually very high. On mobile devices with limited computing and energy capacity, it is difficult to complete neural network inference reliably in a timely manner. Experiments show that even with the acceleration of mobile GPUs, a typical single-frame image processing AI inference task takes about 600 milliseconds. Furthermore, continuous execution of the above inference tasks can only last up to 2.5 hours on commodity devices. The above problems result in that only a few AR applications currently use deep learning. To reduce the inference time of DNNs, one approach is to perform network pruning on the neural network. However, if you prune too many channels, you may break the model, and fine-tuning may not restore satisfactory accuracy.

Mobile edge computing assisted AI is another way to address these issues. The integration of mobile edge computing and AI technologies has recently emerged as a promising paradigm to support computationally intensive tasks. Edge computing moves the inference and training process of AI models to the edge of the network close to the data source. Therefore, it will alleviate network traffic load, latency and privacy concerns. However, for the task of edge computing-assisted AI applications, there are still a lot of challenges, which are as follows:

(1) Although the computing resources of the edge are far stronger than those of the end user, they are also limited, and blindly relying on the computing power of the edge device cannot solve the problem of insufficient computing power of the terminal;

(2) Users offload AI reasoning tasks to edge computing, which will reduce the impact of insufficient computing power to a certain extent, but will also introduce communication delays;

(3) Delay, energy consumption, and accuracy restrict each other, and blindly improving the performance of one of them will inevitably lead to a decline in the performance of the other two.

SUMMARY OF THE INVENTION

The purpose of the present invention is to overcome the deficiencies of the prior art. The present invention provides a method and system for optimal configuration of terminal edge joint resources. Based on the optimization method, the average inference delay of the system and the average energy consumption of the mobile device are simultaneously reduced, and the Improve the accuracy of model inference.

The present invention provides a method for optimal configuration of terminal edge joint resources, the method comprising:

When the video with different frame numbers recorded by the edge controller is input to the neural network, the multiplication and summation required by the neural network and the accuracy of the neural network recognition are calculated; when receiving the video recognition request of each mobile device, the video Identifying requests to complete control tasks, the control tasks include:

Determine the frame number of each mobile device video sampling, and send sampling frame number control information to the video sampling management module corresponding to the mobile device based on the frame number of each mobile device video sampling;

Determine the user's unloading decision, and determine whether the user's reasoning task is completed in the local DNN reasoning module or in the edge DNN reasoning module based on the user's unloading decision;

Based on the allocation strategy of the communication resources of each mobile video device, the time ratio of each mobile device to transmit video upstream is determined;

Determine the resource allocation strategy of the edge DNN inference module to determine the CPU computing frequency of each offload user.

The method also includes:

The edge DNN inference module obtains the uploaded video from the edge device to be uninstalled, then completes the inference according to the computing resources allocated by the edge controller, and sends the inference result to each mobile device.

The method also includes:

The video sampling management module obtains the sampling frame number control information of edge computing, controls the sampling frame number of the mobile device based on the sampling frame control information, and determines the input video frame number for neural network inference.

The method also includes:

The local controller obtains the video from the video sampling management module, and decides whether to transmit the video to the edge server according to the user uninstallation decision obtained from the edge controller.

The local controller obtains the video from the video sampling management module, and decides whether to transmit the video to the edge server according to the user uninstallation decision obtained from the edge controller, including:

If the user's unloading decision is 1, the video is transmitted to the edge server for inference, and the transmitted communication resources are configured by the base station;

If the user's unloading decision is 0, the video is allowed to complete inference locally, and local CPU computing resources are allocated according to local device information.

When the mobile device needs to complete the DNN inference locally, the local DNN inference module obtains the video and uses the allocated local computing resources to complete the DNN inference.

Correspondingly, the present invention also provides a side-end collaborative video AI inference system, the system includes:

The edge controller is used to record the multiplication and summation required by the neural network and the accuracy of the neural network recognition when videos of different frame numbers are input to the neural network; when receiving video recognition requests from various mobile devices, according to The video recognition request of the mobile device completes the control task;

The edge DNN inference module is used to obtain the uploaded video from the edge device that needs to be uninstalled, and then complete the inference according to the computing resources allocated by the edge controller, and send the inference result to each mobile device;

The video sampling management module is used to obtain the sampling frame number control information of edge computing, control the sampling frame number of the mobile device based on the sampling frame control information, and determine the input video frame number for neural network inference;

The local controller is used to obtain the video from the video sampling management module, and decide whether to transmit the video to the edge server according to the user uninstallation decision obtained from the edge controller;

The local DNN inference module is used to obtain video when the mobile device needs to complete DNN inference locally, and use the allocated local computing resources to complete DNN inference.

The described completion of the control task according to the video recognition request of the mobile device includes:

Determine the frame number of each mobile device video sampling, and send sampling frame number control information to the video sampling management module corresponding to the mobile device based on the frame number of each mobile device video sampling;

Determine the user's unloading decision, and determine whether the user's reasoning task is completed in the local DNN reasoning module or in the edge DNN reasoning module based on the user's unloading decision;

Based on the allocation strategy of the communication resources of each mobile video device, the time ratio of each mobile device to transmit video upstream is determined;

Determine the resource allocation strategy of the edge DNN inference module to determine the CPU computing frequency of each offload user.

The local controller transmits the video to the edge server for inference if the user's unloading decision is 1, and the transmitted communication resources are configured by the base station.

The local controller allows the video to complete inference locally if the user's uninstallation decision is 0, and allocates local CPU computing resources according to local device information.

The beneficial effects that the embodiment of the present invention has are as follows:

(1) The present invention provides the architecture of a video AI inference system based on side-end collaboration based on the AI inference algorithm offloading architecture. The edge server can determine the number of video frames used by the user for detection according to the number of requesting users, and provide the The user's offloading strategy and the user's communication computing resource allocation scheme are determined.

(2) Multi-dimensional performance optimization. Under the given system architecture, considering inference delay, terminal energy consumption and recognition accuracy jointly, an effective algorithm is proposed, which can improve the recognition accuracy of neural network while reducing delay and energy consumption.

(3) Performance trade-off analysis. The invention provides a trade-off relationship between inference delay, terminal energy consumption and recognition accuracy. By using this relationship, targeted system optimization can be performed, and the system performance of AI inference under different applications can be performed.

Description of drawings

In order to more clearly illustrate the embodiments of the present invention or the technical solutions in the prior art, the following briefly introduces the accompanying drawings that need to be used in the description of the embodiments or the prior art. Obviously, the accompanying drawings in the following description are only These are some embodiments of the present invention, and for those of ordinary skill in the art, other drawings can also be obtained from these drawings without creative effort.

1 is a schematic structural diagram of a video AI inference system with side-end collaboration in an embodiment of the present invention;

Fig. 2 is the performance comparison schematic diagram of different unloading strategies in the embodiment of the present invention;

FIG. 3 is a schematic diagram of a trade-off relationship among delay, energy consumption, and recognition accuracy in an embodiment of the present invention.

Detailed ways

The technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the accompanying drawings in the embodiments of the present invention. Obviously, the described embodiments are only a part of the embodiments of the present invention, but not all of the embodiments. Based on the embodiments of the present invention, all other embodiments obtained by those of ordinary skill in the art without creative efforts shall fall within the protection scope of the present invention.

The main technical problem to be solved by the present invention is to propose an architecture of an edge-to-end collaborative video AI inference system for video recognition tasks based on deep learning, and based on the architecture, propose a simultaneous optimization of the delay for executing deep learning inference tasks , energy consumption, and method of accuracy. At the same time, the present invention provides a trade-off relationship between delay, energy consumption and accuracy, and the relationship can be used as a reference for system design.

The present invention proposes an architecture of an edge-end collaborative video AI inference system, the function of which is that in the scenario where an edge server serves multiple mobile devices through wireless access, when these devices all need to perform AI-based video recognition tasks , by reasonably adjusting the number of video frames for identification and jointly configuring wireless and computing resources to reduce the inference delay and energy consumption of the neural network, and at the same time improve the accuracy of the neural network inference.

Specifically, FIG. 1 shows a schematic structural diagram of a video AI inference system with side-end collaboration in an embodiment of the present invention, where the system includes:

The edge controller is used to record the multiplication and summation required by the neural network and the accuracy of the neural network recognition when videos of different frame numbers are input to the neural network; when receiving video recognition requests from various mobile devices, according to The video recognition request of the mobile device completes the control task;

The edge DNN inference module is used to obtain the uploaded video from the edge device that needs to be uninstalled, and then complete the inference according to the computing resources allocated by the edge controller, and send the inference result to each mobile device;

The video sampling management module is used to obtain the sampling frame number control information of edge computing, control the sampling frame number of the mobile device based on the sampling frame control information, and determine the input video frame number for neural network inference;

The local controller is used to obtain the video from the video sampling management module, and decide whether to transmit the video to the edge server according to the user uninstallation decision obtained from the edge controller;

The local DNN inference module is used to obtain video when the mobile device needs to complete DNN inference locally, and use the allocated local computing resources to complete DNN inference.

The described completion of the control task according to the video recognition request of the mobile device includes: determining the frame number of each mobile device video sampling, and sending the sampling frame number to the video sampling management module corresponding to the mobile device based on the frame number of each mobile device video sampling Control information; determine user offloading decision, and determine whether the user's reasoning task is completed in the local DNN inference module or in the edge DNN inference module based on the user's offloading decision; determine the uplink transmission of each mobile device based on the allocation strategy of the communication resources of each mobile video device The time scale of the video; determine the resource allocation strategy of the edge DNN inference module to determine the CPU computing frequency of each offloading user.

When the user's unloading decision is 1, the local controller transmits the video to the edge server for reasoning, and the transmitted communication resources are configured by the base station; when the user's unloading decision is 0, the video is allowed to complete the reasoning locally. , and allocate local CPU computing resources according to local device information.

Based on the system structure shown in FIG. 1 , the method for optimizing the configuration of terminal edge joint resources in an embodiment of the present invention includes: when the edge controller records videos with different frame numbers and inputs them to the neural network, calculating the required amount of the neural network. When receiving the video recognition request of each mobile device, complete the control task according to the video recognition request of the mobile device, and the control task includes: determining the frame of each mobile device video sampling and send sampling frame number control information to the video sampling management module corresponding to the mobile device based on the frame number of the video sampling of each mobile device; determine the user uninstallation decision, and determine the user's inference task based on the user uninstallation decision is in the local DNN inference module Whether it is completed or completed in the edge DNN inference module; based on the allocation strategy of the communication resources of each mobile video device to determine the time proportion of each mobile device uplink video transmission; determine the resource allocation strategy of the edge DNN inference module to determine the CPU computing of each offloading user frequency.

Further, the edge DNN inference module obtains the uploaded video from the edge device to be uninstalled, then completes the inference according to the computing resources allocated by the edge controller, and sends the inference result to each mobile device.

Further, the video sampling management module obtains the sampling frame number control information of edge computing, controls the sampling frame number of the mobile device based on the sampling frame control information, and determines the input video frame number for neural network inference.

Further, the local controller obtains the video from the video sampling management module, and decides whether to transmit the video to the edge server according to the user uninstallation decision obtained from the edge controller.

Further, the local controller obtains the video from the video sampling management module, and decides whether to transmit the video to the edge server according to the user unloading decision obtained from the edge controller. The edge server performs inference, and the communication resources transmitted at the same time are configured by the base station; if the user's unloading decision is 0, the video is allowed to complete inference locally, and local CPU computing resources are allocated according to local device information.

Further, when the mobile device needs to complete the DNN inference locally, the local DNN inference module obtains the video, and uses the allocated local computing resources to complete the DNN inference.

It should be noted that, according to the above system architecture, the present invention proposes a corresponding optimization scheme, which simultaneously reduces the average inference delay of the system and the average energy consumption of the mobile device, and improves the accuracy of model inference. The specific algorithm is as follows:

First, the multiplier-adder C(M _n ) required to complete the inference of the neural network model is obtained by using the multiplier-adder model of the neural network. Considering that for each layer of the neural network, the multiplier-adder required to complete the inference is proportional to the input size, so After derivation, the total calculated multiplication and addend can be roughly represented by a linear function of the number of input video frames, denoted as:

C(M _n )=m _{c, 0} M _n +m _{c, 1}

Among them, m _{c, 0} and m _{c, 1} are parameters obtained by fitting, which are determined by the architecture of the model. M _n is the sampling frame number control information, and then the calculation delay D _n and energy of the neural network are given according to the multiplication and addend. The expression that consumes _En :

表示分配给本地的CPU计算资源，

表示根据边缘控制器分配的计算资源，x_n代表是否卸载到边缘计算(如果x_n＝1则代表卸载，否则就在本地计算)。where ρ represents the number of CPU revolutions required for each multiplication and addition operation, d represents the size of each frame of video, R _n represents the communication rate of the nth mobile device, and _tn represents the communication time ratio of the nth mobile device (that is, the communication resource allocation ratio), κ represents the energy consumption coefficient, p _n represents the transmit power of the mobile device, x _n represents the user’s unloading decision,

Indicates the CPU computing resources allocated to the local,

Indicates the computing resources allocated according to the edge controller, and x _n represents whether to offload to edge computing (if x _n =1, it represents offloading, otherwise, it is calculated locally).

For accuracy, the more frames of the input video, the higher the accuracy of the model prediction. As the number of video input frames increases, the gain of the model prediction accuracy will gradually decrease. Therefore, the accuracy rate Φ(M _n ) can be expressed as the following function:

where m _{a, 0} , m _{a, 1} and m _{a, 2} are parameters obtained by fitting, and the values are determined by the model and task of the neural network.

In summary, the optimized objective function is:

的值，⑥x_n为卸载或者不卸载。Among them: β ₁ , β ₂ , β ₃ are the weight coefficients respectively, and the optimization goal is to reduce the total delay D _n and the total energy consumption E _n , and improve the user identification accuracy Φ(M _n ), the constraint condition is ① the number of user frames In line with the given range, ② the sum of the proportion of the communication time of the devices participating in offloading is less than 1, ③ the sum of the allocated computing frequencies of the edge devices is less than the upper limit, ④ the allocated communication time and the edge computing frequency need to be greater than 0, ⑤ the mobile device’s The calculation frequency is assigned to be greater than 0 and less than

, ⑥x _n means uninstall or not.

In order to solve the optimization problem, the offloading strategy x _n is given, and then the problem is decomposed into two sub-problems, namely, the resource optimization problem of mobile devices that complete inference locally and the resource optimization problem of mobile devices that complete inference at the edge. question. Assume that the set of users who complete inference locally is N ₀ , and the set of users who complete inference at the edge is N ₁ , so the optimization problem is transformed into two sub-optimization problems.

For the resource optimization problem of N ₀ , the problem is expressed as:

勾在本地计算时的代价函数。需要满足的约束条件为①用户帧数符合给定的范围，②移动设备的计算频率分配为大于0小于

的值in

Check the cost function when calculating locally. The constraints that need to be met are ① the number of user frames conforms to the given range, ② the calculation frequency of the mobile device is allocated to be greater than 0 and less than

the value of

By derivation, the closed-form expression for solving this subproblem can be obtained:

For the resource optimization problem of N ₁ , the problem is formulated as:

勾卸载到边缘计算时的代价函数。需要满足的约束条件为①用户帧数符合给定的范围，②参与卸载的设备其通信时间比例之和小于1，③边缘设备的分配计算频率之和小于其上限，④分配的通信时间以及边缘计算频率需要大于0。in

Check the cost function when offloading to edge computing. The constraints that need to be met are: ① the number of user frames conforms to the given range, ② the sum of the communication time ratios of the devices participating in the offloading is less than 1, ③ the sum of the allocated calculation frequencies of the edge devices is less than the upper limit, ④ the allocated communication time and the edge The calculation frequency needs to be greater than 0.

与M_n的关系表达式为：By derivation, t _n can be obtained,

The relational expression with _Mn is:

Then the problem can be solved by the method of convex optimization.

和优化变量M_n，

只依赖于设备自身的参数，不受其他设备参数的影响。但是，对于边集

代价函数与集合

中的设备数量和参数有关。下面介绍基于算法的原理。首先，计算每个设备的任务在本地执行时集合

的成本函数

其次，舍得所有设备都卸载到边缘服务器进行推理并且

在每次迭代中，得到

的每个设备对应的代价函数

比较成本函数

和代价函数

在

集合中的设备，可以得到

和

之间的差值，并选择差值最大的设备记为设备y。尝试将集合

中的设备y放入集合

并计算新集合的成本。如果新集合的总成本降低，则继续下一次迭代。否则，将设备y放回集合

算法结束。For the problem of solving the unloading strategy x _n , the embodiments of the present invention are implemented based on a greedy iterative algorithm. It can be observed that when performing inference locally, the cost function

and the optimization variable M _n ,

It only depends on the parameters of the device itself and is not affected by the parameters of other devices. However, for edge sets

Cost Functions and Sets

The number of devices in is related to the parameters. The algorithm-based principle is described below. First, compute the set of tasks for each device when executed locally

cost function of

Second, it is worthwhile to offload all devices to edge servers for inference and

At each iteration, we get

The cost function corresponding to each device of

Compare cost functions

and cost function

exist

The devices in the collection can be obtained

and

The difference between and select the device with the largest difference as device y. try to set

put the device y in the collection

and calculate the cost of the new set. If the total cost of the new set decreases, proceed to the next iteration. Otherwise, put device y back in the collection

The algorithm ends.

系数κ＝10²⁸，由对应的设备确定。输入视频的大小为112*112*M_n。此外，计算复杂度系数设置为ρ＝12这是在通过多次实验获得的。权重β₁，β₂，β₃分别设置为0.2，0.2，0.6。Setting the downlink bandwidth to 5Mhz, the path loss is modeled as PL=128.1+37.6log ₁₀ (D), where D is the distance between the device and the AP in kilometers. Devices are randomly distributed in the [500m 500m] range. The computing resources of the MEC server and device are set to 1.8GHz and 22GHz, respectively. The recognition accuracy requirements and the maximum number of input video frames are set to

The coefficient κ=10 ²⁸ is determined by the corresponding equipment. The size of the input video is 112*112* _Mn . Furthermore, the computational complexity factor is set to ρ=12 which is obtained through multiple experiments. The weights β ₁ , β ₂ , and β ₃ are set to 0.2, 0.2, and 0.6, respectively.

The proposed offloading scheme is compared with the local reasoning scheme (Local), the edge reasoning scheme (Edge), and the random offloading scheme (Random). The experimental results are shown in Figure 2. When the number of devices is less than 10, the cost of only executing tasks at the edge is almost equal to the cost of proposing an offloading solution. This is because when the number of devices is small, all devices can benefit from performing inference on edge servers. If the inference task is only performed locally, the average cost of the device does not change because the local resources between devices do not affect each other. The curve of the Edge scheme is linear because the AI model is the same for all users in the experiment.

This experiment uses different weights β ₁ , β ₂ , β ₃ to analyze the trade-off relationship between average delay, energy consumption and accuracy. The performance of the trade-off surface is obtained by the proposed unloading and allocation scheme with the constraint β ₁ +β ₂ +β ₃ =1. As shown in Figure 3, latency, energy consumption and accuracy are mutually limited, and the three are trade-offs. When the delay is constant, higher recognition accuracy requires higher energy consumption. From another point of view, in order to improve the accuracy, it is necessary to sacrifice the performance of latency and energy consumption. Additionally, with the same accuracy, higher power consumption makes the device more inclined to perform inference tasks locally, resulting in lower latency.

The beneficial effects that the embodiment of the present invention has are as follows:

(1) The present invention provides the architecture of a video AI inference system based on side-end collaboration based on the AI inference algorithm offloading architecture. The edge server can determine the number of video frames used by the user for detection according to the number of requesting users, and provide the The user's offloading strategy and the user's communication computing resource allocation scheme are determined.

(2) Multi-dimensional performance optimization. Under the given system architecture, considering inference delay, terminal energy consumption and recognition accuracy jointly, an effective algorithm is proposed, which can improve the recognition accuracy of neural network while reducing delay and energy consumption.

(3) Performance trade-off analysis. The invention provides a trade-off relationship between inference delay, terminal energy consumption and recognition accuracy. By using this relationship, targeted system optimization can be performed to improve the system performance of AI inference in different applications.

Those of ordinary skill in the art can understand that all or part of the steps in the various methods of the above embodiments can be completed by instructing relevant hardware through a program, and the program can be stored in a computer-readable storage medium, and the storage medium can include: Read only memory (ROM, ReadOnly Memory), random access memory (RAM, Random Access Memory), magnetic disk or optical disk, etc.

In addition, the above embodiments of the present invention have been introduced in detail, and specific examples should be used in this paper to illustrate the principles and implementations of the present invention, and the descriptions of the above embodiments are only used to help understand the method of the present invention and its core idea; Meanwhile, for those of ordinary skill in the art, according to the idea of the present invention, there will be changes in the specific embodiments and application scope. In summary, the contents of this specification should not be construed as limiting the present invention.

Claims (10)

1. A method for optimal configuration of joint resources at an edge of a terminal, the method comprising:

When videos with different frame numbers are recorded by the edge controller and input to the neural network, calculating the multiply-add number required by the neural network and the accuracy rate of neural network identification; when receiving a video identification request of each mobile device, completing a control task according to the video identification request of the mobile device, wherein the control task comprises the following steps:

determining the frame number of video samples of each mobile device, and sending the control information of the frame number of the samples to a video sample management module corresponding to the mobile device based on the frame number of the video samples of each mobile device;

determining a user offload decision, and determining whether the inference task of the user is completed in the local DNN inference module or the edge DNN inference module based on the user offload decision;

determining the time proportion of uplink video transmission of each mobile device based on the allocation strategy of the communication resources of each mobile video device;

and determining a resource allocation strategy of the edge DNN inference module to decide the CPU calculation frequency of each uninstalled user.

2. The method for optimal configuration of terminal edge joint resources according to claim 1, wherein the method further comprises:

the edge DNN reasoning module obtains the uploaded video from the edge equipment needing unloading, then completes reasoning according to the computing resources distributed by the edge controller, and sends a reasoning result to each piece of mobile equipment.

3. The method for optimal configuration of joint resources at an edge of a terminal as claimed in claim 1, wherein the method further comprises:

the video sampling management module acquires the control information of the sampling frame number calculated at the edge, controls the sampling frame number of the mobile equipment based on the control information of the sampling frame, and determines the frame number of the input video for neural network inference.

4. The method for optimal configuration of joint resources at an edge of a terminal as claimed in claim 1, wherein the method further comprises:

and the local controller acquires the video from the video sampling management module and determines whether to transmit the video to the edge server according to the user unloading decision acquired from the edge controller.

5. The method of claim 4, wherein the local controller obtaining video from a video sampling management module and deciding whether to transmit the video to the edge server based on the user offload decision obtained from the edge controller comprises:

if the user unloading decision is 1, transmitting the video to an edge server for reasoning, and meanwhile, configuring a transmitted communication resource by a base station;

and if the user unloading decision is 0, allowing the video to locally finish reasoning, and distributing local CPU computing resources according to the local equipment information.

6. The method for optimized configuration of terminal-edge federated resources of claim 1, wherein when the mobile device needs to complete DNN inference locally, the local DNN inference module obtains video and completes DNN inference using the allocated local computing resources.

7. An edge-side collaborative video AI inference system, the system comprising:

the edge controller is used for recording the multiplication and addition number required by the neural network and the accuracy rate of neural network identification when videos with different frame numbers are input to the neural network; when receiving the video identification request of each mobile device, completing a control task according to the video identification request of the mobile device;

the edge DNN reasoning module is used for obtaining the uploaded video from the edge equipment needing unloading, finishing reasoning according to the computing resources distributed by the edge controller and sending a reasoning result to each piece of mobile equipment;

the video sampling management module is used for acquiring the control information of the sampling frame number of the edge calculation, controlling the sampling frame number of the mobile equipment based on the control information of the sampling frame, and determining the frame number of the input video for neural network inference;

the local controller is used for acquiring the video from the video sampling management module and determining whether to transmit the video to the edge server according to the user unloading decision acquired from the edge controller;

And the local DNN reasoning module is used for acquiring the video when the mobile equipment needs to locally finish DNN reasoning and finishing DNN reasoning by using the distributed local computing resources.

8. The frontend coordinated video AI inference system of claim 7 wherein said completing a control task based on a video identification request of a mobile device comprises:

determining the frame number of video samples of each mobile device, and sending the control information of the frame number of the samples to a video sample management module corresponding to the mobile device based on the frame number of the video samples of each mobile device;

determining a user offload decision, and determining whether the inference task of the user is completed in the local DNN inference module or the edge DNN inference module based on the user offload decision;

determining the time proportion of uplink video transmission of each mobile device based on the allocation strategy of the communication resource of each mobile video device;

and determining a resource allocation strategy of the edge DNN reasoning module to decide the CPU calculation frequency of each uninstalling user.

9. The frontend collaborative video AI inference system of claim 8, wherein the local controller transmits video to an edge server for inference if a user offload decision is 1, with the transmitted communication resources configured by a base station.

10. The frontend collaborative video AI inference system according to claim 8, wherein the local controller allows videos to complete inference locally if a user offload decision is 0, and allocates local CPU computational resources based on local device information.

Images