TW202326526A - Method for reducing network bandwidth required for image streaming by using artificial intelligence processing module - Google Patents

Abstract

A method for reducing network bandwidth required for image streaming by using an Artificial Intelligence (AI) processing module. The resolution of the images to be transmitted are firstly reduced on the server side, and then the resolution-reduced low-resolution images are transmitted to the client device through the network, thereby reducing the network bandwidth required to transmit the image streams. A pre-trained AI processing module is used on the client device to restore the received low-resolution images to high-resolution images, allowing users to enjoy high-quality image streaming and low-bandwidth consumption of network at the same time.

Description

Method of Using Artificial Intelligence Processing Module to Reduce Network Bandwidth Required for Video Streaming

The present invention relates to a method for reducing the network bandwidth required for image streaming by using an artificial intelligence processing module, especially a method of reducing the resolution of the image to be transmitted on the server side before transmitting it through the network to the client, and then use a pre-trained artificial intelligence (AI) processing module on the client to restore the received image to high resolution, thereby reducing the network bandwidth required for image streaming Methods.

In recent years, online games have become more and more popular all over the world. With the development of Cloud-Based computing related systems and technologies, a cloud-based technology has also been developed in which the server transmits the game content to the players through the network as an image stream to provide online game services.

Traditionally, the way to provide such cloud-based online game (On-Line Game) service is to let the server perform almost all calculations. In other words, when online game services are provided, a specific application program will be executed in the server to generate a virtual 3D environment including many 3D (Three Dimensional) objects, which include 3D objects that can be controlled or moved by players. Then, according to the player's control result, the server renders the 3D objects and the virtual 3D environment into a 2D (Two Dimensional) game screen for display on the player's device. Then, the server compresses the rendered image code (Encode) into a 2D image stream and sends it to the player's device through the network. The player device only needs to decode the received 2D image stream and then "play" it without performing 3D rendering calculations. However, there are still several problems to be paid attention to in this kind of cloud online game service, such as the high load of the server when providing 3D rendering programs for a large number of players at the same time, the degradation of image quality caused by encoding compression and streaming programs, and Send 2D images via network Streaming consumes a lot of communication bandwidth.

A known way to solve the picture quality reduction is to increase the resolution of the original image generated by the game application on the server side, and increase the bit rate (Bitrate) when transmitting the image, that is, reduce the server's original image resolution. The compression ratio (Compression Ratio) when the video is encoded into a 2D video stream. However, it is obvious that doing so will result in a significant increase in server load and bandwidth consumption due to the high resolution and high transmission bit rate of the video. For example, assuming that the frame rate (Frame Rate) and the encoding compression rate are both constant, when the resolution of the original image image generated by the game application on the server side is increased from 720p to 1080p, the computing load of the server And the required network transmission bit rate will be increased by 2.25 times. Conversely, when trying to reduce server load or network bandwidth consumption, the picture quality of game graphics will be sacrificed. Therefore, it becomes a dilemma that both perfect image quality and economical bandwidth consumption can be obtained at the same time.

Another way to solve this problem is to reduce the resolution of the original image generated by the server-side game application, or encode the original image into a 2D video stream with a higher compression rate, or both . In this way, the bandwidth consumption of transmitting 2D video streams via the network can be reduced, although the picture quality of the game video will also be sacrificed. At the same time, an image enhancement technology is used on the client device. Once receiving the 2D video stream, the client device will decode the video stream and use the video enhancement technology to improve the visual effect of the video. Histogram equalization (HE for short) is one of the most commonly used methods for improving image contrast because of its simplicity and efficiency. However, HE can lead to excessive contrast enhancement and feature loss problems, resulting in unnatural appearance and loss of processed image details. Furthermore, not only HE but all other image enhancement techniques known in the art suffer from the same dilemma, namely, they all try to use the same set of algorithms to process various images with completely different picture content, which is not possible. OK. Taking the cloud online game service as an example, the screen content of the original image image generated by the server will change significantly due to the change of the game scene. For example, a city game scene may cause the original image of the game to contain many colors with simple and clear outlines and different but roughly the same color family. A dark cave game scene would fill the game's original image with flat, low-key and low-chroma-value colors, but with irregular but inconspicuous landscape outlines. A lush garden scene would make the game's artwork include many vibrant and brightly colored objects with detailed and complex silhouettes. Undoubtedly, none of the conventional enhancement techniques can equally provide good image enhancement effects for various scenes with completely different picture contents.

In addition, another disadvantage of these conventional image enhancement techniques is that although the mathematical calculation formulas of these conventional image enhancement techniques can improve image effects such as contrast, sharpness, saturation, etc., these calculation formulas and their parameters are not It has nothing to do with the original image generated by the server. Therefore, the enhancement process of these conventional image enhancement technologies will never make the enhanced image visually closer to its corresponding original image, and therefore gamers on the client end cannot fully enjoy the game played by the server end. The screen effect of the original image generated by the application.

Therefore, the main purpose of the present invention is to provide a method for reducing the network bandwidth required for image streaming by using artificial intelligence processing modules. By first reducing the resolution of the image to be transmitted on the server side and then transmitting the low-resolution image to the client through the network, the network bandwidth required to transmit the image stream is reduced. Then, on the client side, a pre-trained artificial intelligence (AI) processing module restores the received low-resolution images to high-resolution images, while enjoying high image quality and low network frequency. The double advantage of wide consumption.

To achieve the above-mentioned purpose, an embodiment of the method for reducing the network bandwidth required for image streaming by using an artificial intelligence processing module in the present invention includes: Step (A): Executing a first application program in a server to generate A plurality of source images corresponding to a plurality of original image images; the plurality of source images have a first resolution; the plurality of source images are encoded and compressed by an encoder in the server to generate corresponding plurality an encoded image; step (B): execute a second application program in a client device away from the server; the second application program is associated with and cooperates with the first application program; step (C): The client device is connected to the server via a network, and then receives the encoded images generated by the server via the network in an image stream; step (D): the client device will The encoded images are decoded into a corresponding plurality of decoded images, and an artificial intelligence (AI) processing module is used to improve the resolution of the decoded images to generate a corresponding plurality of high-resolution images. The plurality of high-resolution images have a second resolution; wherein, the second resolution is higher than the first resolution, and the resolution of the plurality of original image images is equal to the second resolution ; Step (E): The client device sequentially outputs a plurality of the high-resolution images to a screen as a plurality of output images to be played; wherein, the AI processing module analyzes the decoded images and Corresponding to at least one mathematical operation formula and a plurality of weighting parameters obtained in advance corresponding to the differences between the original images to process the decoded images; thereby, the obtained high-resolution The resolution of the image will be equal to the corresponding original images and higher than the resolution of the plurality of source images; the at least one mathematical operation formula and the plurality of weighting parameters of the AI processing module are pre-passed by a It is defined by a training program executed by an artificial neural network module in the training server.

Preferably, the plurality of encoded images described in step (A) are generated by the following steps: executing the first application program in the server to generate a plurality of the original image images, and the plurality of the original images The image image has the second resolution; using a resolution reduction program to reduce the resolution of the plurality of the original image images to the first resolution to obtain the corresponding plurality of the source images; and using the encoder to encode the plurality of source images to obtain the corresponding plurality of encoded images.

Preferably, the server includes an AI encoding module; the plurality of encoded images described in step (A) are generated by the following steps: executing the first application program in the server to generate a plurality of a plurality of the original image images, the plurality of the original image images have the second resolution; use the AI coding module to reduce the resolution of the plurality of the original image images to obtain the corresponding plurality of the source images, and A plurality of the source images are encoded to obtain a corresponding plurality of the encoded images; wherein, the AI encoding module includes at least one preset AI encoding formula; the at least one AI encoding formula includes a plurality of preset Encode weighting parameters.

Preferably, the at least one mathematical calculation formula of the AI processing module includes a first preset AI calculation formula and a second preset AI calculation formula; the first preset AI calculation formula includes a plurality of first A weighting parameter; the second preset AI calculation formula includes a plurality of second weighting parameters; wherein, the first preset AI calculation formula and a plurality of the first weighting parameters can be used to improve the resolution of the image, thereby , the resolution of the image processed by the first preset AI calculation formula combined with a plurality of the first weighting parameters can be increased from the first resolution to the second resolution; wherein, the second preset The AI calculation formula combined with a plurality of the second weighting parameters can be used to enhance the quality of the image, whereby the quality of the image processed by the second preset AI calculation formula with the plurality of the second weighting parameters is better than that of the decoded The quality of the image is higher and closer to that of the original image.

Preferably, after the client device decodes the received plurality of encoded images into corresponding plurality of decoded images, the client device will use one of the following methods to process the plurality of decoded images image:

Method 1: The client device first uses the first preset AI calculation formula and multiple The first weighting parameter is used to process a plurality of the decoded images to generate a plurality of corresponding resolution-upgraded images with a second resolution; then, the client device uses the second preset AI calculation formula and the plurality of The second weighting parameter is used to process the plurality of the resolution-enhanced images to generate the plurality of the high-resolution images with high image quality and the second resolution;

Method 2: the client device first uses the second preset AI calculation formula and the plurality of second weighting parameters to process the plurality of decoded images to generate corresponding plurality of quality-enhanced images with high image quality; Then, the client device uses the first preset AI calculation formula and the first weighting parameters to process the plurality of the quality-enhanced images to generate the plurality of the high-quality images with the second resolution and high image quality. resolution image.

1, 501, 701: server

2, 21, 22, 23, 502, 702: client device

3: base station

30:Router

4: Network

100, 200: Application (App)

101, 201: memory

102: Coding

103: Streaming

104:Network equipment

105:Artificial Neural Network Module

106: Neural Networks

107:Decoding module

108:Comparing and training modules

202: Network module

203: decoding module

204: AI Enhanced Module

205: Output module

301-308, 400-466, 711-723, 7161-7229: Procedure

The preferred embodiment of the present invention will be described in conjunction with the following drawings, wherein:

Figure 1 schematically introduces the system of the present invention that uses artificial intelligence processing modules to reduce the network bandwidth required for video streaming;

FIG. 2 is a schematic diagram of an embodiment of the system architecture of the present invention using artificial intelligence processing modules to reduce the network bandwidth required for video streaming;

FIG. 3 is a schematic diagram of a first embodiment of a method for processing video streams using an artificial intelligence processing module of the present invention;

FIG. 4 is a schematic diagram of a first embodiment of the training program of the artificial neural network module 105 according to the present invention;

FIG. 5 is a schematic diagram of a second embodiment of the training program of the artificial neural network module 105 according to the present invention;

FIG. 6 is a schematic diagram of a third embodiment of the training program of the artificial neural network module 105 according to the present invention;

Fig. 7 is a schematic diagram of an embodiment of the training program of the discriminator as shown in Fig. 6;

Figure 8, which discloses an embodiment of the training process of the neural network of the present invention, wherein the original image is YUV420, and the output image is RGB or YUV420;

FIG. 9 is a schematic diagram of an embodiment of a program for processing a decoded image in YUV420 format according to the present invention;

Fig. 10 is another program of the present invention to process the decoded image with YUV420 format Example schematic diagram;

FIG. 11A is a schematic diagram of a second embodiment of the method for reducing the network bandwidth required for image streaming by using the artificial intelligence processing module of the present invention;

FIG. 11B is a schematic diagram of the third embodiment of the method for reducing the network bandwidth required for image streaming by using the artificial intelligence processing module of the present invention;

FIG. 12A is a schematic diagram of a fourth embodiment of a method for reducing network bandwidth required for image streaming by using an artificial intelligence processing module in the present invention;

FIG. 12B is a schematic diagram of a fifth embodiment of the method for reducing the network bandwidth required for image streaming by using the artificial intelligence processing module of the present invention;

Fig. 13 is a schematic diagram of an embodiment of the first preset AI calculation formula and the training method of the first weighting parameter of the AI processing module of the present invention;

FIG. 14A is a schematic diagram of the sixth embodiment of the method for reducing the network bandwidth required for image streaming by using the artificial intelligence processing module of the present invention;

FIG. 14B is a schematic diagram of the seventh embodiment of the method for reducing the network bandwidth required for image streaming by using the artificial intelligence processing module of the present invention;

Fig. 15 is a schematic diagram of an embodiment of the training method of the first preset AI calculation formula, the second preset AI calculation formula, the first weighting parameter and the second weighting parameter of the AI processing module of the present invention;

FIG. 16 is a schematic diagram of the eighth embodiment of the method for reducing the network bandwidth required for image streaming by using the artificial intelligence processing module of the present invention;

Fig. 17 shows the AI coding formula, the AI decoding formula, the first preset AI formula, the second preset AI formula, the first weighting parameter and the AI coding formula of the artificial neural network module of the present invention. A schematic diagram of an embodiment of the training method of the second weighting parameter.

The invention relates to a method for reducing the network bandwidth required for image streaming by using an artificial intelligence processing module. By first reducing the resolution of the image to be transmitted on the server side and then transmitting the low-resolution image to the client through the network, the network bandwidth required to transmit the image stream is reduced. Then, on the client side, a pre-trained artificial intelligence (AI) processing module restores the received low-resolution images to high-resolution images, allowing you to enjoy high image quality and low network connectivity at the same time Double advantage of bandwidth consumption.

One application of the present invention is cloud-based online games, wherein players use client devices to connect to a server through a network to play games provided by the server. The server can respond to the commands input by the players and generate corresponding images and videos. Therefore, for example, the player can execute a movement command on the client device. The movement command is sent to the server through the network, and then the server calculates an image according to the movement command, returns the image and plays it on the client device. In many games, the server generates a 2D image that contains several 3D rendered objects in view.

Please refer to FIG. 1 , which schematically introduces the system of the present invention that utilizes artificial intelligence processing modules to reduce network bandwidth required for image streaming. A server 1 is used to provide a service executed by an application program on the server 1; the service may be, but not limited to, a cloud online game service. A plurality of client devices 21 , 22 , 23 can connect (log in) to the server 1 via a network 4 to use the services provided by the application program running on the server 1 . In this embodiment, the network 4 is the Internet, and the client devices 21, 22, 23 can be any type of electronic devices that can be connected to the Internet, such as (but not limited to) smart phones 21 , digital tablet, laptop 22, desktop 23, video game console, or even a smart TV. Some client devices 21, 22 are wirelessly connected to the network 4 through a wireless communication base station 3 or a wireless router 30, while others are connected to the network through a network router or a network sharer in a wired manner 4. The application running on the server 1 generates a virtual 3D environment which includes a plurality of 3D objects; some of the 3D objects can be moved or destroyed according to the user's operation, while others cannot. In a preferred embodiment, for each client device, the application program will have an independent running instance; that is, each application program only provides the service to one client device, but the server 1, multiple applications can be executed simultaneously to provide services to multiple client devices. The client devices 21 , 22 , 23 are connected to the server 1 via the network 4 to receive images generated by the application program and including at least a part of 3D objects. The structure and functions of the system of the present invention will be described in detail through Figure 2 and its related descriptions.

FIG. 2 is a schematic diagram of an embodiment of the system architecture of the present invention. An application program (App) 100 is an application program (usually a 3D game program) stored in the memory 101 and executed on the server 1, which can generate a 3D image rendering result composed of a series of original image images. The encoding 102 and the streaming 103 are an encoding module and a streaming module respectively, which accept the original images generated by the application 100, and encode and stream them into a 2D image stream. The 2D video stream is then passed through the server The network device 104 of the server is transmitted to the remote client device 2 through the network 4 . Each client device 2 is pre-installed with an application program 200 . The application program 200 is stored in the memory 201 of the client device 2 and can be associated and cooperated with the application program 100 on the server 1 . The application program 200 of the client device 2 can establish a connection with the application program 100 of the server 1 and receive the encoded 2D image stream from the server 1 through the network module 202 . The encoded 2D image stream is then decoded by the decoding module 203 to generate decoded images. Due to these encoding, streaming and decoding procedures, the quality of the decoded image will obviously be much worse than the original image. The AI processing module 204 built in the client device 2 can enhance the quality of those decoded images to generate corresponding enhanced images. Wherein, the AI processing module 204 processes the decoded images by analyzing at least one mathematical formula obtained by analyzing the difference between the decoded images and the corresponding original image images; thereby, The resulting enhanced image will be visually closer to the original image than the decoded image. Afterwards, the enhanced image is output (played) on the screen (display panel) of the client device 2 via the output module 205 . In the present invention, the mathematical calculation formula used by the AI processing module 204 of the client device 2 is a training performed by the artificial neural network (Artificial Neural Network) module 105 located on the server 1 program to define. The artificial neural network module 105 is set in the server 1 and includes: an artificial neural network 106 , a decoding module 107 and a comparison and training module 108 . The embodiment of the training program of the artificial neural network module 105 described in the present invention will be described in detail later.

FIG. 3 is a schematic diagram of a first embodiment of a method for processing video streams using an artificial intelligence processing module according to the present invention. By utilizing the system and architecture of the present invention as shown in Figures 2 and 3, the method generally includes the following steps:

Step 301: Execute a first application in a server. The first application program generates a plurality of original image images according to at least one instruction (step 302). Next, the original image is subjected to resolution reduction processing by a resolution reduction module in the server (step 3021), and then encoded and compressed by an encoder in the server (step 303) to generate a complex code of the image. The encoded images are then sent to the client device via the network in the form of a 2D image stream (step 304 ). Since the video is downscaled before being sent to the client device, the network bandwidth required to transmit the video stream is also reduced.

A second application is executed in a client device remote from the server (step 305). The second application is related to and cooperates with the first application, whereby the customer The client device can be operated by a user to generate and send instructions to the server to enjoy the service provided by the first application program of the server. The client device sends the command to the server via the network, and then receives the encoded image generated by the server and corresponding to the command via the network. Then, the client device decodes (step 306) the encoded images into complex decoded images, and uses an AI processing module (step 307) to enhance the quality of the decoded images to generate complex enhancements after the image. Wherein, the AI processing module processes the decoded images by analyzing the difference between the decoded images and the corresponding original images by at least one mathematical operation formula obtained in advance; thereby, The resulting enhanced image will be visually closer to the original image than the decoded image. Afterwards, the client device outputs (step 308 ) the enhanced image to a screen (display panel) as an output image to be played.

In the present invention, the at least one mathematical operation formula used by the AI processing module in the client device includes a plurality of weighted parameters (Weighted Parameters). The weighting parameters are related to the difference between the decoded image and the corresponding original image, and are defined by a training program executed by an artificial neural network module in the server. In an embodiment of the present invention, the weighting parameters are pre-stored in the client device. In another embodiment, the weighting parameter is downloaded from the server to the client device when the client device executes the second application program.

In one embodiment of the present invention, the screen content contained in the original image generated by the server will change drastically due to different game scenes. For example, a city game scene may cause the original image of the game to contain many colors with simple and clear outlines and different but roughly the same color family. Another game scene of a dark cave would fill the game's original image with flat, low-key and low-chroma colors, but with irregular but inconspicuous landscape outlines. Yet another lush garden scene would make the game's artwork image contain many vibrant and brightly colored objects with detailed and complex silhouettes. The method of the present invention uses different sets of weighting parameters to adapt to these different game scenes, whereby the quality of the output image enhanced by the same AU enhancement module can be maintained at a high-quality and stable level. Even if the screen content of the original image changes drastically.

Preferably, the original image images generated by the first application program can be divided into a plurality of scene-modes, and each scene includes a plurality of the original image images. These weighting parameters are also divided into complex groups, each group contains a plurality of weighting parameters and is corresponding to its One of the scenarios. The decoded images corresponding to the original images of different scenes will be enhanced by the same AI processing module using the set of weighting parameters corresponding to the scene among the different sets of weighting parameters. In an embodiment of the present invention, the different sets of weighting parameters are all pre-stored in the client device, and whenever the scene changes, the set of weighting parameters corresponding to the changed new scene will be used It is used in the AI processing module to generate the enhanced image. In another embodiment, these different sets of weighting parameters are all stored on the server side, and whenever the scene changes, the set of weighting parameters corresponding to the changed new scene will be sent to the client by the server The end device is then used in the AI processing module to generate the enhanced image.

FIG. 4 is a schematic diagram of the first embodiment of the training program of the artificial neural network module 105 of the present invention. In the present invention, the mathematical calculation formula used by the AI processing module 204 of the client device 2 is trained and defined by a training program executed by the artificial neural network module 105 in the server 1 . The training procedure includes the following steps:

Step 400: Execute the first application program in a training mode to generate a plurality of training original image images (step 401), and perform resolution reduction processing on these original image images (step 4011);

Step 402: encoding the original training image images with reduced resolution into a plurality of training encoded images by the encoder;

Step 403: Decoding the training coded images into a plurality of training decoded images by a training decoder in the server;

Step 404: The artificial neural network module receives the training decoded images and uses at least one training mathematical operation to process the training decoded images one by one to generate a plurality of training output images (step 405); the at least one training mathematical operation The formula contains a plurality of training weighting parameters; and

Step 406: using the comparison and training module to compare the differences between the training output image and the corresponding training original image one by one, and accordingly adjust the training weighting parameters of the at least one training mathematical operation formula; These training weighting parameters will be adjusted to minimize the difference between the training output image and the corresponding training original image; each time these training weighting parameters are adjusted, the adjusted training weighting parameters It will be fed back to the at least one training mathematical operation formula for processing the next training decoded image in step 404 . After a predetermined number of After the comparison between the training output image and the corresponding training original image, and the adjustment procedure of the training weighting parameters for a predetermined number of times, the training weighting parameters obtained after the training (step 407) will be taken out and applied to the client The AI processing module of the terminal device is used as the weighting parameter of the mathematical operation formula.

In the first embodiment of the present invention, the training decoding image is input to the artificial neural network module to generate the corresponding training output image. Then, the training output image and the corresponding training original image are compared to calculate a difference value. Then, use mathematical optimization algorithms such as Adam algorithm, stochastic gradient descent (SGD for short), or root mean square gradient descent algorithm (Root Mean Square Propagation; RMSProp for short), etc. to learn all The weighting parameters of the artificial neural network (usually referred to as weight w, bias b), the smaller the difference, the better, so that the training output image can be closer to its corresponding original training image. Different methods can be used to calculate the difference value (or approximate value) to suit different needs; for example: mean square error (mean square error; MSE for short), L1 regularization (L1 regularization) (using absolute value error) , peak signal-to-noise ratio (PSNR for short), structure similarity (SSIM for short), generative adversarial networks loss (GAN loss for short) and/or other methods, etc. wait. In the first embodiment, the following methods are used to calculate the difference value: (1) weighted average of MSE, L1, and GAN loss; (2) MSE; (3) GAN loss and simultaneously train the discriminator (Discriminator); ( 4) The weighted average of MSE and the edge of MSE (Edge of MSE). More details of this training procedure will be described later.

FIG. 5 is a schematic diagram of a second embodiment of the training program of the artificial neural network module 105 of the present invention. In the present invention, the training program of the second embodiment includes the following steps:

Step 410: Execute the first application program in a training mode to generate a plurality of training original image images (step 411), wherein the color format of these training original image images is the three primary colors of color and light (RGB); and, for these Training the original image to perform resolution reduction processing (step 4111);

Step 412: Encode the original training image images with reduced resolution into a plurality of training encoded images by the encoder;

Step 413: Decoding the training coded images into a plurality of training decoded images by a training decoder in the server;

Step 414: In the second embodiment, when the training decoded image and the training input When the color formats of the output images are the same (both are RGB in the second embodiment), the residual network module (residual network module) can also be called a convolutional neural network (Convolutional Neural Network; CNN for short) Can be used in the artificial neural network module; the output of the residual network module for processing the corresponding training decoding image will be summed up with the corresponding training decoding image (step 415); Then, the sum of the output of the residual network module and the corresponding training decoded image will be output as the training output image (step 416); and

Step 417: Using the comparison and training module to compare the difference between the training output image and the corresponding training original image one by one (calculate the difference value), and adjust the at least one training mathematical operation formula accordingly Training weighting parameters; these training weighting parameters will be adjusted to minimize the difference between the training output image and the corresponding training original image; each time these training weighting parameters are adjusted, these adjustments The final training weighting parameters will be fed back to the artificial neural network for processing the next training decoded image in step 414 . After performing a predetermined number of comparisons between the training output image and the corresponding training original image, and a predetermined number of adjustments to the training weighting parameters, the training weighting parameters obtained after the training is finally completed (step 418) will be Taken out and applied in the AI processing module of the client device as the weighting parameter of the mathematical operation formula.

FIG. 6 is a schematic diagram of a third embodiment of the training program of the artificial neural network module 105 of the present invention. In the third embodiment, the comparison and training module uses a discriminator (Discriminator) to compare the difference between the training output image and the corresponding training original image and adjust the training weighting parameters accordingly. The training procedure of this third embodiment comprises the following steps:

Step 420: Execute the first application program in a training mode to generate a plurality of training original image images (step 421), wherein the training original image images include n channels, where n is a positive integer greater than 2; and , performing resolution reduction processing on these training original image images (step 4211);

Step 422: Encode the original training image images with reduced resolution into a plurality of training encoded images by the encoder;

Step 423: Decode the training coded images into a plurality of training decoded images by a training decoder in the server; wherein the training decoded images include m channels, where m is a positive integer greater than 2; and

Step 424: The artificial neural network module receives the training decoded images (m channels) and uses at least one training mathematical operation formula to process the training decoded images one by one to generate a plurality of training output images (n channels) ( Step 425); the at least one training mathematical operation formula includes a plurality of training weighting parameters; the training output image (n channel) and the corresponding training decoded image (m channel) are combined (step 426) to generate a plurality of training combinations images (with m+n channels); then, the training combined images are fed back to a discriminator (step 427) for analyzing the quality of the training output images, thereby training the artificial neural network.

FIG. 7 is a schematic diagram of an embodiment of the training procedure of the discriminator shown in FIG. 6 . The training procedure for this discriminator consists of the following steps:

Step 430: Execute the first application program in a training mode to generate a plurality of training original image images (step 431), wherein the training original image images include n channels, where n is a positive integer greater than 2; and , and perform resolution reduction processing on these training original image images (step 4311);

Step 432: Encode the original training image images with reduced resolution into a plurality of training encoded images by the encoder;

Step 433: Decode the training coded images into a plurality of training decoded images by a training decoder in the server; wherein the training decoded images include m channels, where m is a positive integer greater than 2; and

Step 434: The artificial neural network module accepts the training decoding images and uses at least one training mathematical operation formula to process the training decoding images (m channels) one by one to generate a plurality of training output images (step 435); the at least A training mathematical operation formula includes a plurality of training weighting parameters; the training output image includes n channels;

Step 436: The n-channel training output image is combined with the corresponding m-channel training decoding image to generate a plurality of false samples (false samples) with m+n channels; and, the n-channel training original Combining the image image and its corresponding m-channel training decoding image to generate a plurality of true samples (true samples) with m+n channels (step 437); and

Step 438: The simulated fake samples of the m+n channels and the simulated real samples of the m+n channels are fed back to the discriminator of the comparison and training module for training the discriminator to detect and distinguish the simulated fake samples and The ability to simulate real samples.

The artificial neural network 105 (as shown in Figure 2) is properly trained on the server 1 side Afterwards, the obtained weighting parameters (weight w, bias b) will be applied to the AI processing module 204 in the client device. The AI processing module 204 and its associated weighting parameters (weight w, bias b) are stored in the client device 2 . Afterwards, whenever the client device receives and decodes the encoded image contained in the 2D image stream from the server, each encoded image will be processed by the AI processing module to generate an enhanced of the image. The client device then plays the enhanced images on its screen as an output image. The neural network can learn and enhance the color, brightness and detail of the image. Since some details of the original image will be damaged or lost in the process of encoding and streaming, a properly trained neural network can repair these damaged or lost details. In the embodiment of the present invention, the neural network of the AI processing module requires the following information to operate:

Related functions and parameters:

X: input image;

Conv2d(X,a,b,c,d,w,b): Execute on X; the number of output channels is a(amount of output channel=a); the core size is b(kernel_size=b); the step value is c (stride=c); the padding size is 2d convolution and its bias is d (padding size=2d convolution with bias of d); the weighting parameters of this training are core w (kernel w) and bias b (bias b);

Conv2dTranspose(X,a,b,c, w , b )): Execute on X; the number of output channels is a(amount of output channel=a); the core size is b(kernel_size=b); the step value is c( stride=c); the cropping size is 2d transpose convolution and its deviation is d(cropping size=2d transpose convolution with bias of d); the weighting parameters of this training are core w(kernel w) and deviation b(bias b);

σ(X): a non-linear activation function working on X;

uint8(x): Used to control and limit the value of floating-point x between 0 and 255 (including 255), u uses the unconditional rounding method and converts to unsigned int8;

R(X,w): Residual blocks working on X, including many conv2d and batchnorm, each of which contains its own weighted parameters for training (more information can be used as a reference through the following website: https:// stats.stackexchange.com/questions/246928/what-exactly-is-a-residual-learning-block-in-the-context-of-deep-residual-networ ).

Since the input and output images may have different color formats, such as RGB, YUV420, YUV444, etc., so, more about input and output images with different color formats will be discussed below.

The first case: the original image is RGB, and the output image is also RGB.

This case is the simplest since both input and output images are RGB images. In order to improve the processing speed, a relatively large kernel size (such as 8x8, stride=4 in the convolution and transposed convolution structure) is used to accelerate the calculation as soon as possible to cope with the high resolution of Full HD (Full HD) images Spend. Residual networks are used in this case to make convergence easier and more stable.

Related functions and parameters:

X: input image, which is in RGB format, and each color is in unsigned int8 format;

；

;

Y=uint8((Conv2dTranspose(σ(Conv2d(X2,a,b,c,d, w _1, b _1)), w _2, b _2)+X2)*128+128);

w _1 is a matrix whose size is b*b*3*a; b _1 is a vector whose size is a;

w _2 is a matrix whose size is b*b*3*a; b _2 is a vector whose size is 3;

The parameters used include:

The resolution of X is 1280x720;

a=128,b=10,c=5,d=0,σ=leaky relu with alpha=0.2;

a=128,b=9,c=5,d=4,σ=leaky relu with alpha=0.2;

a=128,b=8,c=4,d=0,σ=leaky relu with alpha=0.2;

Provided that the client device has a faster processing speed, the following formula can be used:

Y=uint8((Conv2dTranspose(R(σ(Conv2d(X2,a,b,c,d, w _1, b _1)), w _R), w _2, b _2)+X2)*128+128);

w _1 is a matrix whose size is b*b*3*a; b _1 is a vector whose size is a;

w _2 is a matrix whose size is b*b*3*a; b _2 is a vector whose size is 3;

where R is a residual block (residual blocks) which has n layers;

Among them, a lot of neural network layers are included, and each layer has its trained weighted parameters, which are collectively referred to as w _ R ;

The parameters used include:

a=128,b=8,c=4,d=0, σ =leaky relu with alpha=0.2; n=2;

a=128, b=8, c=4, d=0, σ=leaky relu with alpha=0.2; n=6.

The second case: the original image is YUV420, the output image is RGB or YUV444;

If the input original image is YUV420 and the output image is RGB or YUV444, the residual network cannot be directly applied to this situation because the resolution and format of the input and output images are different. The method of the present invention will first decode the input image of YUV420, and then use another neural network (called A network, wherein N=3) to process the decoded image and obtain an image in RGB or YUV444 format (called A network) X2). Then, this X2 image is fed into the neural network (residual network) described in the first case for training. Moreover, the same training method is also applied to the A network to compare the difference between X2 and the original image, thereby training the A network.

X_y is the Y of the input image with YUV420 format, and its format is unsigned int8;

X_uv is the uv of the input image with YUV420 format, and its format is unsigned int8;

；

;

；

;

X2=Conv2d( X2_y ,3,e,1, w_y , b_y )+Conv2dTranspose(X2_uv , 3,f,2, w_uv , b_uv );

w_y is a matrix whose size is e*e*1 * 3; b_y is a vector whose size is 3;

w_uv is a matrix whose size is f*f*3*2; b _uv is a vector whose size is 3;

The above is the first embodiment of the A network (neural network number A);

Finally, the mathematical formula used to output the output image is the same as the mathematical formula used in the first case above when both the input and output images are in RGB format:

Y=uint8((Conv2dTranspose(σ(Conv2d(X2,a,b,c,d, w _1, b _1)), w _2, b _2)+X2)*128+128);

w _1 is a matrix whose size is b*b*3*a; b _1 is a vector whose size is a;

w _2 is a matrix whose size is b*b*3*a; b _2 is a vector whose size is 3;

The parameters used are also the same as those used above when the input and output images are in RGB format:

The resolution of X is 1280x720;

a=128,b=8,c=4,d=0,e=1,f=2,σ=leaky relu with alpha=0.2;

a=128, b=8, c=4, d=0, e=1, f=2, σ=leaky relu with alpha=0.2.

Please refer to FIG. 8, which discloses an embodiment of the training process of the neural network of the present invention, wherein the original image is YUV420, and the output image is RGB or YUV420. The training process of the neural network includes the following steps:

Step 440: Execute the first application program in a training mode to generate a plurality of training original image images, wherein the training original image images are in RGB or YUV444 format; and perform resolution reduction on the training original image images Processing (step 4401);

Step 441: Encode the original training image images with reduced resolution into a plurality of training encoded images by the encoder;

Step 442: Decoding the training coded images into a plurality of training decoded images by a training decoder in the server; wherein the training decoded images are in YUV420 format;

Step 443: The artificial neural network module includes a first neural network and a second neural network; the first neural network (also referred to as A network) accepts these trainings to decode images and uses at least one training A mathematical operation formula is used to process the training decoded image (YUV420) one by one to generate a plurality of first output images X2 (also referred to as X2; such as step 444) which have the same encoding format as the training original image; the at least one training The mathematical expression contains a plurality of training weighting parameters;

Step 445: The second neural network is a convolutional neural network (CNN for short); the second neural network (also called CNN network) receives the first output image X2 and uses at least one training a mathematical operation formula to process the first output image X2 one by one to generate a plurality of second output images; the at least one training mathematical operation formula includes a plurality of training weighting parameters; then, the first output image X2 and the second output The two images are added (step 446) to generate the training output image (step 447);

The comparison and training module includes a first comparator and a second comparator; in step 448, the first comparator compares the difference between the first output image X2 and its corresponding original image for training training the first neural network; in step 449, the second comparator compares the training The difference between the training output image and its corresponding training original image is used for training the second neural network.

FIG. 9 is a schematic diagram of an embodiment of a procedure for processing a decoded image in YUV420 format according to the present invention. The present invention processes the program of the decoded image with YUV420 format including:

Step 451: The step of the first neural network accepting and processing the training decoded image with YUV420 color format includes:

Step 452: Extract the Y-part data in the training decoding image, and process the Y-part data of the training decoding image by the neural network with a standard size (original size) to generate Y-part output data with N channels (For example: step value Stride=1 in convolution; as in step 454);

Step 453: extract the UV-part data in the training decoded image, and process the UV-part data of the trained decoded image by a double-amplified neural network to produce UV-part output data with N channels (for example: steps Value Stride=2 in the transposed convolution; as in step 455);

Step 456: Add the Y-part output data and the UV-part output data to generate the training output image (Step 457).

The third case: the original image is YUV420, and the output image is YUV444, processed in another faster way.

If the input image is YUV420 and the output image is YUV444, besides the aforementioned method, there is another way to implement the first neural network (A network), which is a special case with faster speed. The decoded image in YUV420 format is first converted into an image in YUV444 format (also known as X2) using the first neural network (A network); after that, X2 is sent to the aforementioned neural network (residual network road) for training. Moreover, the same training method is also implemented on the A network to compare the difference between X2 and the original image, so as to train the A network.

X_y is the Y of the input image with YUV420 format, and its format is unsigned int8;

X_uv is the uv of the input image with YUV420 format, and its format is unsigned int8;

；

;

；

;

X3_uv =Conv2dTranspose( X2_uv ,2,2,2, w_uv , b_uv );

w _ uv is a matrix whose size is 2*2*2*2; b _ uv is a vector whose size is 2;

X2=concat(X2_y,X3_uv);

The above is another embodiment of the A network (neural network number A), in which the "concat" function connects the input according to the direction of the channel;

Finally, the mathematical formula used to output the output image is the same as the mathematical formula used in the first case above when both the input and output images are in RGB format:

Y=uint8((Conv2dTranspose(σ(Conv2d(X2,a,b,c,d, w _1, b _1)), w _2, b _2)+X2)*128+128);

w _1 is a matrix whose size is b*b*3*a; b _1 is a vector whose size is a;

w _2 is a matrix whose size is b*b*3*a; b _2 is a vector whose size is 3;

The parameters used are also the same as those used above when the input and output images are in RGB format:

The resolution of X is 1280x720;

a=128,b=10,c=5,d=0,σ=leaky relu with alpha=0.2;

a=128,b=9,c=5,d=4,σ=leaky relu with alpha=0.2;

a=128, b=8, c=4, d=0, σ=leaky relu with alpha=0.2.

FIG. 10 is a schematic diagram of another embodiment of the process of processing a decoded image in YUV420 format according to the present invention. As shown in Figure 10, the program of the present invention for processing the decoded image in YUV420 format includes:

Step 461: The first neural network accepts and processes the training decoded image in YUV420 color format through the following steps, wherein the training decoded image includes N channels, and N is a positive integer greater than 2;

Step 462: extracting Y-part data from the training decoded image to generate Y-part output data;

Step 463: extracting the UV-part data in the training decoded image, and using the first neural network with double magnification to process the UV-part data of the training decoded image to generate UV-part output with N-1 channels Data (for example: stride=2 in transposed convolution; as in step 464);

Step 465 : Process the Y-part data and the UV-part data with a merging function Concat (concatenates) to generate the training output image (Step 466 ).

The fourth case: the original image is YUV420, and the output image is also YUV420.

If the input image is YUV420 and the output image is also YUV420, the processing method will be similar to the aforementioned RGB-to-RGB method. However, since the input format and output format are different, different convolution methods will be applied to different channels. For example, when the kernel size of the neural network is 8x8 and the stride value is 4 to process the Y-part of the image, the neural network can be changed to a kernel size of 4x4 and the stride value is 2 to process the UV-part of the image. part.

X_y is the Y of the input image with YUV420 format, and its format is unsigned int8;

X_uv is the uv of the input image with YUV420 format, and its format is unsigned int8;

；

;

；

;

X3=σ(Conv2d ( X2_y,a,b,c, w_y , b_y ) +Conv2d(X2_uv,a,b/ 2 ,c / 2, w_uv , b_uv ));

w_y is a matrix whose size is b*b*1 * a; b_y is a vector whose size is a ;

w _ uv is a matrix whose size is (b/2)*(b/2)*2*a; b _ uv is a vector whose size is a;

X4_y=Conv2dTranspose(X3,1 , b,c, w_1 , b_1 )+ X2_y ;

X4_uv=Conv2dTranspose(X3,2,b/2,c/2, w_2 ,b_2) + X2_uv;

w _1 is a matrix whose size is b*b*1*a; b _1 is a vector whose size is 1;

w _2 is a matrix whose size is (b/2)*(b/2)*2*a; b _2 is a vector whose size is 2;

The above is another embodiment of the A network (neural network number A), in which the "concat" function connects the input according to the direction of the channel;

Final output:

Y_y=uint8(X4_y*128+128);

Y_uv=uint8(X4_uv*128+128);

Parameters used:

a=128, b=8, c=4, d=0, e=2, f=2, σ=leaky relu with alpha=0.2.

The detailed description of the parameters used in the present invention is as follows:

Training parameters:

The initial value of the weighting parameter is based on the Gaussian distribution (Gaussian distribution), mean=0, stddev=0.02;

The Adam algorithm is used in the training process, the learning rate learning rate=1e-4, beta1=0.9;

Micro batch size mini batch size=1;

The primary error function is:

100*(L2+L2e)+λ*L1+γ*D+α*Lg;

The parameters used have standard values:

λ=0, γ=0, α=0;

λ=0, γ=0, α=100;

λ=0, γ=1, α=0;

λ=10, γ=0, α=0;

λ=10, γ=0, α=100;

λ=10, γ=1, α=0;

in:

L2= mean (( T - Y ) ² ); where mean refers to the average value and T is the training target;

L1= mean (| T - Y |); where mean refers to the average value and T is the training target;

D is to generate a confrontational network loss (GAN loss), using the general GAN training method to train the discriminator (Discriminator) to distinguish between (X,Y) and (X,T);

The mathematical formula of Lg is:

For WxH images,

Y_dx(i,j)=Y(i+1,j)-Y(i,j)0<=i<W-1,0<=j<H

T_dx(i,j)=T(i+1,j)-T(i,j)0<=i<W-1,0<=j<H

Y_dy(i,j)=Y(i,j+1)-Y(i,j)0<=i<W,0<=j<H-1

T_dy(i,j)=T(i,j+1)-T(i,j)0<=i<W,0<=j<H-1

L _g = mean (( T _dx - Y _dx ) ² )+ mean (( T _dy - Y _dy ) ² )

In the RGB mode, the training target T is the original image of the RGB game image;

In YUV444 mode, the training target T is the original image of the RGB game image;

In RGB->RGB, and YUV420->YUV420 mode, L2e=0;

In YUV420->RGB and YUV420->YUV444 mode,

L _{2 e} = mean (( T - X ₂ ) ² ).

As can be seen from the foregoing description, the method of the present invention has the following advantages:

The training of the neural network can be kept at any time according to various images with different contents, so as to perform different enhancement effects on different image contents; for example, for images with cartoon style, realistic style or different scenes, etc., different weighting parameters w, b can be pre-stored in the client device, or dynamically downloaded to the client device;

Regarding the method of judging which mode the original image should belong to, the neural network on the server side can automatically determine the mode of the original image and transmit such information to the client device; because the content of the original image is consistent, this This determination process can be performed periodically by the server, such as once per second; however, in another embodiment, the process of determining the image mode can also be performed periodically by the client device, such as once every few seconds, depending on the client. Depends on the computing power of the terminal device;

Training is carried out based on real video images, and the degree of enhancement can be actually measured; for example, when using the method of the present invention to enhance a video image with a resolution of 1280x720 and a bit rate (bitrate) of 3000, the PSNR value of similar scenes can be increased 1.5~2.2db or so, which proves that the method of the present invention can really improve the quality of the output image, and make the output image visually closer to the quality of the original image; moreover, the present invention is different from those well-known image enhancement techniques , they can only increase the contrast, smoothing and color filtering of the output image, but cannot make the output image visually closer to the original image as in the present invention;

With the simplified model of the neural network algorithm, and the use of large cores and large step values, the resolution of the neural network can be reduced rapidly, and the processing speed of the model can be greatly improved; even the client device with limited computing power can reach 60fps and HD resolution output image targets; and

By bringing the conversion of color formats (YUV420 and RGB) into neural networks, And taking advantage of the lower resolution of the UV channel than the Y channel, setting the step value of the UV channel to half of the Y channel can improve the calculation speed of the neural network.

FIG. 11A is a schematic diagram of a second embodiment of a method for reducing network bandwidth required for image streaming by using an artificial intelligence processing module according to the present invention, which includes the following steps:

Step 711 : Execute a first application in a server 701 . The first application program generates a plurality of original image images with high resolution according to at least one instruction. The resolution of these original image images may be 4K resolution or higher (hereinafter also referred to as the second resolution). The at least one command is generated by the client device 702 and sent to the server 701 through the network.

Step 712: Use a known sampling method in the server 701 to reduce the resolution of the original image to obtain a source image with a low resolution (such as 1080i, 720p or lower, hereinafter also referred to as the first resolution) , the first resolution is lower than the second resolution.

Step 713: Use an encoder in the server 701 to encode and compress the source images to generate a plurality of corresponding encoded images.

Step 714 : The server 701 transmits the encoded images to the client device 702 via the network in the form of a 2D video stream (step 304 ) according to the instruction from the client device 702 . Since the video is downscaled before being sent to the client device, the network bandwidth required to transmit the video stream is also reduced.

Step 715: The client device 702 receives the encoded images and decodes them into a plurality of corresponding decoded images.

In the present invention, the client device 702 includes an AI processing module, which includes at least one preset mathematical operation formula. The at least one mathematical operation formula includes a plurality of weighting parameters. The weighting parameters are predefined by a training mode of an artificial neural network module of a training server. A second application program is executed on the client device 702, which is associated with and cooperates with the first application program for the user to operate the client device 702 to generate the command. The client device 702 sends the command to the server 701 through the network, and receives the encoded image generated according to the command from the server.

In this embodiment, the at least one mathematical calculation formula includes a first preset AI calculation formula and a second preset AI calculation formula. The first preset AI calculation formula includes a plurality of first weighting parameters. The second preset AI calculation formula includes a plurality of second weighting parameters. The first preset AI calculation formula combined with a plurality of the first weighting parameters can be used to improve the resolution of the image, thereby, by The resolution of the image processed by the first preset AI calculation formula combined with the plurality of first weighting parameters can be increased from the first resolution to the second resolution. The second preset AI calculation formula combined with a plurality of the second weighting parameters can be used to enhance the quality of the image, thereby, the image processed by the second preset AI calculation formula with the plurality of the second weighting parameters The quality of the decoded image is higher than that of the decoded image and closer to the quality of the original image.

Step 716: After the client device 702 decodes the received plurality of encoded images into corresponding plurality of decoded images, the client device first uses the first preset AI formula and the plurality of The first weighting parameter is used to process the plurality of decoded images to generate corresponding plurality of resolution-enhanced images with the second resolution. Then, in step 717, the client device 702 uses the second default AI calculation formula and the second weighting parameters to process the plurality of the resolution-enhanced images to generate high image quality with the second A plurality of the high-resolution images of resolutions. Afterwards, in step 718 , the client device 702 takes the high-resolution images as output images and outputs them to the screen (display).

Wherein, the first weighting parameter of the first preset AI calculation formula is pre-defined by analyzing the difference between the source image with low resolution and the corresponding original image, so that the resolution can be improved The image is visually closer to the original image than to the source image. In addition, the second weighting parameter of the second preset AI calculation formula is pre-defined by analyzing the difference between the decoded image and the corresponding original image, so that the high-resolution image is visually It is closer to the original image than the decoded image.

FIG. 11B is a schematic diagram of a third embodiment of a method for reducing network bandwidth required for image streaming by using an artificial intelligence processing module according to the present invention. Since most of the steps shown in FIG. 11B are the same as those shown in FIG. 11A, the same or similar steps will be given the same numbers and the details will not be repeated. In the embodiment shown in FIG. 11B, the first application running on the server 701 generates a plurality of source images with a first resolution (step 719), in other words, the server 701 will directly generate low-resolution source images , so no additional downscaling is required. Afterwards, these source images are processed according to the same steps 713-718 described in FIG. 11A. Since the server 701 directly generates the low-resolution source image, the computing resource required for it is lower than that for generating the high-resolution original image; In addition to the benefits of widening, the embodiment shown in FIG. 11B also has the advantage of saving computing resources of the server.

Figure 12A shows the use of artificial intelligence processing modules in the present invention to reduce the need for image streaming A schematic diagram of a fourth embodiment of the network bandwidth method. Since most of the steps shown in FIG. 12A are the same as those shown in FIG. 11A and FIG. 11B, the same or similar steps will be given the same numbers and their details will not be repeated. In the embodiment shown in FIG. 12A, the first application program executed in the server 701 generates a plurality of original image images with the second resolution (step 711). These original images are then processed by reducing the resolution to become a plurality of source images corresponding to the first resolution (step 712 ). Afterwards, these source images are encoded (step 713 ) into encoded images and sent to the client device 702 (step 714 ). The client device 702 decodes the received encoded video into a decoded video (step 715 ). Then, in step 717 of the embodiment shown in FIG. 12A, the client device 702 first uses the second preset AI calculation formula and the plurality of second weighting parameters to process a plurality of the decoded images to generate A plurality of upscaled images with high image quality but still at the first resolution. Then, the client device 702 uses the first preset AI calculation formula and the first weighting parameters to process the quality-enhanced images to generate the second resolution and high image quality images. High resolution imagery. Afterwards, in step 718 , the client device 702 takes the high-resolution images as output images and outputs them on the screen.

FIG. 12B is a schematic diagram of a fifth embodiment of a method for reducing network bandwidth required for image streaming by using an artificial intelligence processing module according to the present invention. Since most of the steps shown in FIG. 12B are the same as those shown in FIG. 12A and FIG. 11B, the same or similar steps will be given the same numbers and the details will not be repeated. In the embodiment shown in FIG. 12B, the first application running on the server 701 generates multiple source images with a first resolution (step 719), in other words, the server 701 directly generates source images with low resolution. , so no additional downscaling is required. Afterwards, these source images are processed according to the same steps 713-718 described in FIG. 12A.

FIG. 13 is a schematic diagram of an embodiment of the training method of the first preset AI calculation formula and the first weighting parameter of the AI processing module of the present invention. In the present invention, the first preset AI calculation formula and the complex first weighting parameters in the AI processing module in the client device 702 are obtained by executing an artificial neural network training program on the training server. predefined. After the training is completed, the first preset AI calculation formula and the complex first weighting parameters will be applied to the AI processing module of the client device 702 to execute Step 716 shown in FIG. 2B shows the step of improving the resolution of AI. The steps of training the first preset AI calculation formula and the complex first weighting parameters in the training server include:

Step 7161: Enable a training mode in the training server to generate a plurality of training sessions Training the original image (step 7162); a plurality of the original training images have the second resolution (high resolution).

Step 7163: Execute a resolution reduction procedure to reduce the resolution of the plurality of training original image images from the second resolution to the first resolution, so as to generate a plurality of training low-resolution images with the first resolution (step 7164).

Step 7165: Accept and use a first training algorithm to process the plurality of training low-resolution images one by one by the artificial neural network module to generate a plurality of training output images corresponding to the second resolution (step 7166); the first training expression has a plurality of first training weighting parameters.

Step 7167: Use a comparison module to compare the differences between the plurality of training output images and the corresponding plurality of training original image images one by one, and adjust the first training weights of the first training formula accordingly parameter. The first training weighting parameters are adjusted to minimize the difference between the training output image and the corresponding training original image. Every time the first training weighting parameters are adjusted, the adjusted first training weighting parameters are fed back to the first training calculation formula for processing the next training low-resolution image. After a predetermined number of comparisons between the training output image and the corresponding training original image, and a predetermined number of adjustment procedures for the first training weighting parameters, the finally obtained first training weighting parameters Will be applied in the AI processing module of the client device 702 as a plurality of the weighting parameters of at least one mathematical operation formula, so as to perform as shown in Figures 11A, 11B, 12A, and 12B. The AI upscaling step described in step 716 is shown.

In this embodiment, the training method for the second preset AI calculation formula and the second weighting parameter of the AI processing module of the client device 702 is the same as that shown in Figure 4, Figure 5, or Figure 6. The network module 105 is trained in the same way. After the training is completed, the obtained second training weighting parameters will be applied in the AI processing module of the client device 702 as a plurality of the weighting parameters of at least one mathematical operation formula, so as to execute as shown in FIG. 10 1A, 11B, 12A, and 12B show the step of AI enhancing image quality described in step 717.

FIG. 14A is a schematic diagram of a sixth embodiment of a method for reducing network bandwidth required for image streaming by using an artificial intelligence processing module according to the present invention. Since most of the steps shown in FIG. 14A are the same as those shown in FIG. 11A, the same or similar steps will be given the same numbers and the details will not be repeated. In the embodiment shown in FIG. 14A, the first application program executed in the server 701 generates a A plurality of original image images of the second resolution (step 711). These original images are then processed by reducing the resolution to become a plurality of source images corresponding to the first resolution (step 712 ). Afterwards, these source images are encoded (step 713 ) into encoded images and sent to the client device 702 (step 714 ). The client device 702 decodes the received encoded video into a decoded video (step 715 ). In this embodiment, the first preset AI calculation formula, the second preset AI calculation formula, the plurality of the first weighting parameters, and the plurality of the second weighting parameters are all included in the client device In 702, the same AI processing module is used for directly processing the plurality of decoded images into a plurality of the high-resolution images with high image quality and the second resolution. Therefore, in step 720, the AI processing module of the client device 702 accepts and uses the first preset AI calculation formula, the second preset AI calculation formula, the plurality of the first weighting parameters, and the plurality of The second weighting parameter is used to process the decoded images to generate a plurality of the high-resolution images corresponding to the second resolution. Afterwards, in step 718 , the client device 702 takes the high-resolution images as output images and outputs them on the screen.

FIG. 14B is a schematic diagram of a seventh embodiment of a method for reducing network bandwidth required for image streaming by using an artificial intelligence processing module according to the present invention. Since most of the steps shown in FIG. 14B are the same as those shown in FIG. 14A and FIG. 11B, the same or similar steps will be given the same numbers and the details will not be repeated. In the embodiment shown in FIG. 14B, the first application program running on the server 701 generates a plurality of source images with a first resolution (step 719), in other words, the server 701 will directly generate low-resolution source images , so no additional downscaling is required. Afterwards, these source images are processed according to the same steps 713, 714, 715, 720 and 718 as described in FIG. 14A.

FIG. 15 is a schematic diagram of an embodiment of the training method of the first preset AI calculation formula, the second preset AI calculation formula, the first weighting parameter and the second weighting parameter of the AI processing module of the present invention. In the present invention, the first preset AI calculation formula, the second preset AI calculation formula, the first weighting parameter and the second weighting parameter in the AI processing module in the client device 702 are obtained by training A pre-defined training program of an artificial neural network is executed on the server. After the training is completed, the first preset AI calculation formula, the second preset AI calculation formula, the first weighting parameter and the second weighting parameter are applied to the AI processing module of the client device 702 for execution as shown in FIG. 4A and step 720 shown in FIG. 14B, the steps of AI enhancement resolution+enhancement. The steps of training the first preset AI calculation formula, the second preset AI calculation formula, the first weighting parameter and the second weighting parameter in the training server include:

Step 7201: Enable a training mode in the training server to generate a plurality of training original images (step 7202); the plurality of training original images have the second resolution (high resolution).

Step 7203: Execute a resolution reduction procedure to reduce the resolution of the plurality of training original image images from the second resolution to the first resolution, so as to generate a plurality of training low resolutions with the first resolution Image (step 7204).

Step 7205: Execute an encoding process to encode the plurality of training low-resolution images into corresponding plurality of training encoded images by means of an encoder in the training server.

Step 7206: Execute a decoding program to decode the plurality of training coded images into corresponding plurality of training decoded images by a decoder in the training server; the plurality of training decoded images have the first resolution.

Step 7207: The artificial neural network module receives and uses a first training formula and a second training formula to process a plurality of the training decoded images one by one to generate corresponding complex numbers with the second resolution training output images (step 7208). The first training formula has a plurality of first training weighting parameters. The second training formula has a plurality of second training weighting parameters.

Step 7209: Use a comparison module to compare the differences between the plurality of training output images and the corresponding plurality of training original image images one by one, and adjust the first training weights of the first training formula accordingly parameters and the second training weighting parameters of the second training formula. The first training weighting parameters and the second training weighting parameters are adjusted to minimize the difference between the training output image and the corresponding training original image. Every time when the first training weighting parameters and the second training weighting parameters are adjusted, the adjusted first training weighting parameters and the second training weighting parameters will be fed back to the first training operation The formula and the second training calculation formula are used for processing the next training low-resolution image. After a predetermined number of comparisons between the training output image and the corresponding training original image, and a predetermined number of adjustment procedures for the first training weighting parameters and the second training weighting parameters, the final result obtained The first training weight parameters and the second training weight parameters will be applied in the AI processing module of the client device as the first training formula contained in at least one mathematical formula and The weighting parameters of the second training calculation formula are used to perform the steps of AI improvement resolution + image quality enhancement as described in step 720 in FIG. 14A and FIG. 14B step.

In a preferred embodiment of the present invention, the AI processing module of the client device 702 only includes a single set of AI calculation formulas and a plurality of weighting parameters, which is achieved through steps 7201 to 7209 as described in FIG. 15 For training, it is also possible to provide the merging function of "AI enhancement of resolution + enhancement of image quality" as described in step 720 in FIG. 14A and FIG. 14B.

FIG. 16 is a schematic diagram of an eighth embodiment of a method for reducing network bandwidth required for image streaming by using an artificial intelligence processing module according to the present invention. Since most of the steps shown in FIG. 16 are the same as those shown in FIG. 14A and FIG. 14B, the same or similar steps will be given the same numbers and their details will not be repeated. In the embodiment shown in FIG. 16, the server 701 further includes an AI encoding module. The first application program executed on the server 701 generates a plurality of original image images with a second resolution according to the instruction (step 721 ). Next, in step 722, the server 701 uses the AI encoding module to reduce the resolution of the plurality of original images to obtain the corresponding plurality of source images, and encode the plurality of source images to obtain Corresponding plural encoded images. The AI coding module includes at least one preset AI coding formula; the at least one AI coding formula includes a plurality of preset coding weighting parameters. Then, the encoded image is transmitted to the client device 750 in the form of an image stream (step 714 ). In this embodiment, the AI processing module of the client device 702 further includes an AI decoding algorithm for decoding the received encoded image into a corresponding decoded image. In other words, the AI decoding formula, the first preset AI formula, the second preset AI formula, the plurality of the first weighting parameters, and the plurality of the second weighting parameters are all included in the client In the same AI processing module of the terminal device 702, the plurality of received encoded images are directly processed into a plurality of decoded high-resolution images with high image quality and the second resolution . Therefore, in step 723, the AI processing module of the client device 702 receives and uses the AI decoding formula, the first preset AI formula, the second preset AI formula, and a plurality of the first The weighting parameters and the plurality of second weighting parameters are used to process the encoded images to directly generate the plurality of high-resolution images corresponding to the second resolution. Afterwards, in step 718 , the client device 702 takes the high-resolution images as output images and outputs them on the screen.

Fig. 17 shows the AI coding formula, the AI decoding formula, the first preset AI formula, the second preset AI formula, the first weighting parameter and the AI coding formula of the artificial neural network module of the present invention. A schematic diagram of an embodiment of the training method of the second weighting parameter. In the present invention, the AI coding formula and its weighting parameters in the server 701, and the AI processing module in the client device 702 The AI decoding formula and its weighting parameters, the first preset AI formula, the second preset AI formula, the first weighting parameter and the second weighting parameter are all obtained by executing the artificial neural network on the training server. The training program of the road is pre-defined. After the training is completed, the AI coding formula and its weighted parameters will be applied to the AI coding module of the server 701 to execute step 722 (AI coding step) shown in FIG. 16; meanwhile, the AI decoding formula and Its weighting parameters, the first preset AI calculation formula, the second preset AI calculation formula, the first weighting parameter and the second weighting parameter are applied in the AI processing module of the client device 702 to execute as shown in FIG. 16 Step 723 shown (the step of combining AI decoding + improving resolution + enhancing image quality). Train the AI encoding formula and its weighting parameters, the AI decoding formula and its weighting parameters, the first preset AI formula, the second preset AI formula, the first weighting parameter and the second formula in the training server The steps for the second weighting parameter include:

Step 7221: Enable a training mode in the training server and execute the first application program in the training mode to generate a plurality of training original image images (step 7222); a plurality of the training original image images have the second resolution ( High resolution).

Step 7223: Execute a resolution reduction procedure to reduce the resolution of the plurality of training original image images from the second resolution to the first resolution, so as to generate a plurality of training low resolutions with the first resolution Image (step 7224).

Step 7225: Use a first artificial neural network module to receive and use a training encoding algorithm to process the plurality of training low-resolution images one by one to generate corresponding training encoding images with the first resolution (Step 7226); the training encoding formula has a plurality of training encoding weighting parameters.

Step 7227: Use a second artificial neural network module to receive and use a training decoding algorithm to process the plurality of training encoding images one by one to generate corresponding training output images with the second resolution (step 7228); the training decoding operation formula has a plurality of training decoding weighting parameters.

Step 7229: Use a comparison module to compare the differences between the plurality of training output images and the corresponding plurality of training original images one by one, and adjust the training encoding weighting parameters and the training encoding formula accordingly. The training decoding weighting parameters of the training decoding operation formula. The training encoding weighting parameters and the training decoding weighting parameters are adjusted to minimize the difference between the training output image and the corresponding training original image. Each time when the training encoding weight parameters and the training decoding weight parameters are adjusted, the adjusted The training encoding weighting parameters and the training decoding weighting parameters are respectively fed back to the training encoding formula and the training decoding formula for processing the next training low-resolution image. In step 7220, after performing a predetermined number of comparisons between the training output image and the corresponding training original image, and a predetermined number of adjustment procedures for the training encoding weighting parameters and the training decoding weighting parameters, The finally obtained training encoding weighting parameters will be applied to the AI encoding calculation formula of the AI encoding module of the server; and the obtained training decoding weighting parameters will be applied to the client device. In at least one mathematical operation formula of the AI processing module. Thereby, the AI encoding module of the server can process the process of reducing the resolution and encoding of the original image in a combined manner as in step 722 of FIG. 16; and, the AI processing module of the client device can be as follows Step 723 in FIG. 16 generally executes the procedures of decoding, resolution enhancement, and image quality enhancement on the received encoded image in combination.

In a preferred embodiment, the AI processing module of the client device 702 only includes a single set of AI calculation formulas and a plurality of weighting parameters, which are trained by means of steps 7221 to 7229 as described in FIG. 17 , so the merging function of "AI decoding + AI enhancement of resolution + AI enhancement of image quality" as described in step 723 in FIG. 16 can also be provided.

In one embodiment of the present invention, any of the following artificial neural network technologies in the prior art can be used as the first artificial neural network module to perform the AI encoding step in the server: Autoencoder (Autoencoder; AE for short), Denoising Autoencoder (DAE for short), Variational autoencoder (VAE for short) and Vector-Quantized Variational Autoencoder (Vector-Quantized Variational Autoencoder; VQ-VAE for short). The second artificial neural network module for performing AI decoding, AI upscaling and AI enhanced image quality in the client device may be selected from the following existing artificial neural network technologies: SRCNN, EDSR, RCAN, EnhanceNet, SRGAN and ESRGAN.

In a preferred embodiment, the plurality of original image images may be three-dimensional (3D for short) images; each 3D image includes a left-eye view and a right-eye view combined in an image frame in a side-by-side manner view. Therefore, the corresponding output image generated on the client device will also be a 3D image.

In a preferred embodiment, the system for reducing network bandwidth required for image streaming by using artificial intelligence processing modules according to the present invention can also be applied to a robot remote control system. The server of the present invention can be a robot, which includes a motion module, a camera module, a communication module and a controller Modeling set. The client device of the present invention may be a robot control device including a controller module and a display. The robot is remotely connected with the control equipment through the Internet or other wireless communication technologies. The controller module can be operated by the user to send control instructions to the robot, so as to remotely control and operate the movement and action of the robot. The camera module of the robot includes a binocular image capture module to obtain 3D images (the left eye view and the right eye view are combined side by side in one image frame). According to the control instructions received from the control device, the robot can move and perform other actions, and can also take 3D images of the robot's surrounding environment, and then send these 3D images back to the control device and display them on the monitor. By using the method of the present invention, the client device (control device) can be equipped with a pre-trained AI processing module; thus, the binocular image capture module of the robot only needs to capture low-resolution images with a small amount of data and consume less The low network bandwidth is quickly transmitted to the client device, and then the client device can use the AI processing module to restore the high resolution and high image quality of the 3D image. In addition, since the robot shoots and processes low-resolution images, it consumes relatively less computing resources and saves power, which can extend the remote working time of the robot.

The above descriptions are only preferred embodiments of the present invention, and are not intended to limit the applicable scope of the present invention. The scope of protection of the present invention should be based on the contents of the scope of the patent application. Any person familiar with the art may make various modifications and changes to the present invention without departing from the spirit and scope of the present invention, and are still covered by the content of the patent application of the present invention.

701: server

702: client device

711-718: Steps

Claims (11)

A method for reducing network bandwidth required for image streaming by using an artificial intelligence processing module, comprising: Step (A): executing a first application program in a server to generate a plurality of source images corresponding to the plurality of original image images; the plurality of source images have a first resolution; the plurality of source images are An encoder in the server performs encoding and compression to generate a plurality of corresponding encoded images; Step (B): executing a second application program in a client device remote from the server; the second application program is associated with and cooperates with the first application program; Step (C): The client device is connected to the server via a network, and then receives the encoded images generated by the server via the network in an image stream; Step (D): The client device decodes the encoded images into a plurality of corresponding decoded images, and uses an artificial intelligence (AI) processing module to improve the analysis of the decoded images degree, to generate a corresponding plurality of high-resolution images; the plurality of the high-resolution images have a second resolution; wherein, the second resolution is higher than the first resolution, and the plurality of the original the resolution of the graphic image is equal to the second resolution; Step (E): The client device sequentially outputs the plurality of high-resolution images to a screen as the plurality of output images to be played; Wherein, the AI processing module processes the decoded images by analyzing the difference between the decoded images and the corresponding original images at least one mathematical operation formula and a plurality of weighting parameters obtained in advance; Thereby, the resolution of the obtained high-resolution images will be equal to the corresponding original image images, and higher than the resolution of the plurality of source images; the at least one mathematical operation formula of the AI processing module And the plurality of weighting parameters are pre-defined by a training program executed by an artificial neural network module in a training server. The method as described in claim 1, wherein the plurality of encoded images described in step (A) are generated by the following steps: executing the first application program in the server to generate a plurality of the original image images, the plurality of the original image images having the second resolution; using a resolution reduction procedure to reduce the resolution of the plurality of original image images to the first resolution, so as to obtain the corresponding plurality of the source images; and The encoder is used to encode the plurality of source images to obtain the corresponding plurality of encoded images. The method as described in claim 1, wherein the server includes an AI encoding module; the plurality of encoded images described in step (A) are generated by the following steps: executing the first application program in the server to generate a plurality of the original image images, the plurality of the original image images having the second resolution; Using the AI encoding module to reduce the resolution of the plurality of original images to obtain corresponding plurality of source images, and encode the plurality of source images to obtain corresponding plurality of encoded images; Wherein, the AI coding module includes at least one preset AI coding formula; the at least one AI coding formula includes a plurality of preset coding weighting parameters. The method as described in claim 1, wherein the at least one mathematical calculation formula of the AI processing module includes a first preset AI calculation formula and a second preset AI calculation formula; the first preset The AI formula includes a plurality of first weighting parameters; the second preset AI formula includes a plurality of second weighting parameters; Wherein, the first preset AI calculation formula with a plurality of the first weighting parameters can be used to improve the resolution of the image, thereby, the first preset AI calculation formula with the plurality of the first weighting parameters is processed The resolution of the processed image can be increased from the first resolution to the second resolution; Wherein, the second preset AI calculation formula with a plurality of the second weighting parameters can be used to enhance the quality of the image, thereby, the second preset AI calculation formula with the plurality of the second weighting parameters processed The quality of the image is higher than that of the decoded image and closer to the quality of the original image. The method as described in claim 4, wherein, after the client device decodes the received plurality of encoded images into corresponding plurality of decoded images, the client device will use one of the following way to process multiple decoded images: Way 1: The client device first uses the first preset AI calculation formula and the first weighting parameters to process the decoded images to generate corresponding resolution enhancements with the second resolution. image; then, the client device uses the second default AI calculation processing a plurality of the resolution-enhanced images to generate a plurality of the high-resolution images with high image quality and the second resolution; Method 2: the client device first uses the second preset AI calculation formula and the plurality of second weighting parameters to process the plurality of decoded images to generate corresponding plurality of quality-enhanced images with high image quality; Then, the client device uses the first preset AI calculation formula and the first weighting parameters to process the plurality of the quality-enhanced images to generate the plurality of the high-quality images with the second resolution and high image quality. resolution image. The method according to claim 4, wherein the first preset AI formula, the second preset AI formula, the plurality of first weighting parameters, and the plurality of second weighting parameters are all Included in the same AI processing module of the client device, for processing the plurality of decoded images into a plurality of the high-resolution images with high image quality and the second resolution. The method as described in claim 4, wherein the AI processing module further includes an AI decoding operation formula for decoding the received plurality of encoded images into a plurality of decoded images; wherein, the AI The decoding calculation formula, the first preset AI calculation formula, the second preset AI calculation formula, the plurality of the first weighting parameters, and the plurality of the second weighting parameters are all contained in the same client device. One of the AI processing modules is used to process the received plurality of encoded images into a plurality of the high-resolution images with high image quality and the second resolution at one time. The method as described in claim 4, wherein the training program of the artificial neural network module executed in the training server includes the following steps: enabling a training mode in the training server to generate a plurality of training original image images; the plurality of training original image images have the second resolution; performing a resolution reduction procedure to reduce the resolution of the plurality of training original image images from the second resolution to the first resolution, so as to generate a plurality of training low resolution images with the first resolution; receiving and using a first training algorithm by the artificial neural network module to process a plurality of the training low-resolution images one by one to generate a corresponding plurality of training output images with the second resolution; the first training The expression has a plurality of first training weighting parameters; and Use a comparison module to compare one by one the plurality of the training output images and the corresponding complex The differences between the training original image images, and adjust the first training weighting parameters of the first training calculation formula accordingly; the first training weighting parameters will be adjusted so that the training output image corresponds to the corresponding training output image. The difference between the training original images is minimized; each time when the first training weighting parameters are adjusted, the adjusted first training weighting parameters will be fed back to the first training calculation formula for Process the next training low-resolution image; Wherein, after a predetermined number of comparisons between the training output image and the corresponding training original image, and a predetermined number of adjustment procedures of the first training weighting parameters, the finally obtained first training The weighting parameters will be applied in the AI processing module of the client device as the plurality of weighting parameters of at least one of the mathematical calculation formulas. The method as described in claim 4, wherein the training program of the artificial neural network module executed in the training server includes the following steps: enabling a training mode in the training server to generate a plurality of training original image images; the plurality of training original image images have the second resolution; performing a resolution reduction procedure to reduce the resolution of the plurality of training original image images from the second resolution to the first resolution, so as to generate a plurality of training low resolution images with the first resolution; Executing an encoding process to encode the plurality of training low-resolution images into corresponding plurality of training encoded images by means of an encoder in the training server; Executing a decoding program to decode the plurality of training encoded images into corresponding plurality of training decoded images by a decoder in the training server; the plurality of training decoded images have the first resolution; The artificial neural network module receives and uses a first training formula and a second training formula to process a plurality of the training decoded images one by one to generate corresponding training outputs with the second resolution image; the first training expression has a plurality of first training weighting parameters; the second training expression has a plurality of second training weighting parameters; and Using a comparison module to compare the differences between the plurality of training output images and the corresponding plurality of training original images one by one, and adjust the first training weighting parameters and the first training weighting parameters of the first training formula accordingly. The second training weighting parameters of the second training formula; the first training weighting parameters and the second training weighting parameters will be adjusted so that the training output image and the corresponding training original image The difference is minimized; each time when these first training After the weighting parameters and the second training weighting parameters are adjusted, the adjusted first training weighting parameters and the second training weighting parameters will be fed back to the first training formula and the second training formula for processing the next training low-resolution image; Wherein, after performing a predetermined number of comparisons between the training output image and the corresponding training original image, and a predetermined number of adjustment procedures for the first training weighting parameters and the second training weighting parameters, finally The obtained first training weighting parameters and the second training weighting parameters will be applied in the AI processing module of the client device as the first training operation included in at least one mathematical operation formula. formula and the weighting parameters of the second training formula. The method as described in claim 3, wherein the training program of the artificial neural network module executed in the training server includes the following steps: enabling a training mode in the training server to generate a plurality of training original image images; the plurality of training original image images have the second resolution; performing a resolution reduction procedure to reduce the resolution of the plurality of training original image images from the second resolution to the first resolution, so as to generate a plurality of training low resolution images with the first resolution; using a first artificial neural network module to receive and use a training encoding algorithm to process the plurality of training low-resolution images one by one to generate corresponding training encoding images with the first resolution; the training The encoding operation formula has a plurality of training encoding weighting parameters; using a second artificial neural network module to receive and use a training decoding algorithm to process the plurality of training encoding images one by one to generate corresponding training output images with the second resolution; the training decoding algorithm The formula has a plurality of training and decoding weighting parameters; Using a comparison module to compare the differences between the plurality of training output images and the corresponding plurality of training original image images one by one, and adjust the training encoding weighting parameters of the training encoding formula and the training decoding accordingly. The training decoding weighting parameters of the calculation formula; the training coding weighting parameters and the training decoding weighting parameters will be adjusted to minimize the difference between the training output image and the corresponding training original image; each Once the training encoding weighting parameters and the training decoding weighting parameters are adjusted, the adjusted training encoding weighting parameters and the training decoding weighting parameters will be fed back to the training encoding formula and the training decoding operation formula for processing the next training low-resolution image; Wherein, after performing a predetermined number of comparisons between the training output image and the corresponding training original image, and a predetermined number of adjustment procedures for the training encoding weighting parameters and the training decoding weighting parameters, the finally obtained The training encoding weighting parameters will be applied to the AI encoding calculation formula of the AI encoding module of the server, and the obtained training decoding weighting parameters will be applied to the AI processing of the client device In at least one mathematical operation formula of the module; thereby, the AI processing module of the client device can execute the procedures of decoding, resolution improvement and image quality enhancement on the received encoded image at one time. The method as described in claim 2, wherein the plurality of original image images are three-dimensional (three-dimensional; 3D for short) images, and each of the 3D images is combined in an image frame in a side-by-side manner left eye view and right eye view of .

Images