CN111681167B

CN111681167B - Image quality adjusting method and device, storage medium and electronic equipment

Info

Publication number: CN111681167B
Application number: CN202010496819.7A
Authority: CN
Inventors: 吴家平
Original assignee: Tencent Technology Shenzhen Co Ltd
Current assignee: Tencent Technology Shenzhen Co Ltd
Priority date: 2020-06-03
Filing date: 2020-06-03
Publication date: 2021-03-12
Anticipated expiration: 2040-06-03
Also published as: CN111681167A

Abstract

The invention discloses an image quality adjusting method and device, a storage medium and electronic equipment. Wherein, the method comprises the following steps: acquiring picture coding data sent by a server of a target cloud application; decoding the picture coding data to obtain picture resources to be displayed; adjusting the image quality of the image resource from a first resolution to a second resolution, wherein the first resolution is smaller than the second resolution; and displaying the picture with the picture quality of the second resolution. The invention solves the technical problem that the processing load of the cloud application server is aggravated when the image quality adjusting method provided by the related technology is adopted.

Description

Image quality adjusting method and device, storage medium and electronic equipment

Technical Field

The present invention relates to the field of computers, and in particular, to a method and an apparatus for adjusting image quality, a storage medium, and an electronic device.

Background

With the increasing demand of users on the quality (i.e., image quality) of pictures presented by an application client, in order to ensure that a cloud application (e.g., a cloud game application) client can present pictures with high image quality, some cloud application developers usually directly render and encode pictures to be presented with high resolution in a cloud server, and send encoded picture resources with high resolution to the cloud application client, so that the cloud application client can directly obtain the picture resources with high resolution after decoding, and perform picture presentation.

However, in the above method for optimizing and adjusting image quality, the cloud application server often needs to render and encode the image with high resolution, so that the processing load of the cloud application server is greatly increased.

In view of the above problems, no effective solution has been proposed.

Disclosure of Invention

The embodiment of the invention provides an image quality adjusting method and device, a storage medium and electronic equipment, and aims to at least solve the technical problem that the processing load of a cloud application server is increased when the image quality adjusting method provided by the related art is adopted.

According to an aspect of the embodiments of the present invention, there is provided a method for adjusting image quality, including: acquiring picture coding data sent by a server of a target cloud application; decoding the picture coding data to obtain picture resources to be displayed; adjusting the image quality of the image resource from a first resolution to a second resolution, wherein the first resolution is smaller than the second resolution; and displaying the picture with the picture quality of the second resolution.

According to another aspect of the embodiments of the present invention, there is provided a method for adjusting image quality, including: acquiring an original resolution of an original picture, wherein the original resolution is the unprocessed resolution of the original picture; adjusting the resolution of the original picture to a first resolution under the condition that the original resolution is greater than the first resolution, wherein the original resolution is greater than the first resolution; rendering and coding the original picture according to the first resolution to obtain picture coded data; and sending the picture coded data to a client of the target cloud application, so that the client adjusts the image quality of the picture resources decoded from the picture coded data to a second resolution and displays the picture with the image quality of the second resolution, wherein the first resolution is smaller than the second resolution.

According to another aspect of the embodiments of the present invention, there is provided an image quality adjusting apparatus, including: the acquisition unit is used for acquiring picture coding data sent by a server of the target cloud application; a decoding unit, configured to decode the picture coding data to obtain a picture resource to be displayed; an adjusting unit, configured to adjust the image quality of the image resource from a first resolution to a second resolution, where the first resolution is smaller than the second resolution; and the display unit is used for displaying the picture with the image quality of the second resolution.

According to another aspect of the embodiments of the present invention, there is also provided a computer-readable storage medium, in which a computer program is stored, wherein the computer program is configured to execute the image quality adjusting method when running.

According to another aspect of the embodiments of the present invention, there is also provided an electronic device, including a memory and a processor, where the memory stores a computer program, and the processor is configured to execute the image quality adjustment method through the computer program.

In the embodiment of the invention, after the picture coding data sent by the server of the target cloud application is acquired, the picture coding data is decoded to obtain the picture resource to be displayed. Then, the image quality of the image resources is adjusted from a first resolution to a second resolution to display the image with the image quality of the second resolution, wherein the first resolution is smaller than the second resolution. That is to say, after the picture coding data sent by the server of the target cloud application is acquired, the picture coding data can be decoded, and then the resolution of the image quality of the picture resource obtained by decoding is improved, and the server does not directly perform rendering coding processing on the picture with high resolution, so as to reduce the number of pixels needing to be processed by the server, thereby achieving the purpose of reducing the processing load of the server, and further overcoming the problem that the processing load of the server is aggravated in the related art.

Drawings

The accompanying drawings, which are included to provide a further understanding of the invention and are incorporated in and constitute a part of this application, illustrate embodiment(s) of the invention and together with the description serve to explain the invention without limiting the invention. In the drawings:

fig. 1 is a diagram illustrating a hardware environment of an alternative method for adjusting image quality according to an embodiment of the present invention;

fig. 2 is a flowchart illustrating an alternative method for adjusting image quality according to an embodiment of the invention;

fig. 3 is a flowchart illustrating another alternative image quality adjustment method according to an embodiment of the invention;

fig. 4 is a flowchart illustrating another alternative method for adjusting image quality according to an embodiment of the invention;

fig. 5 is a flowchart illustrating another alternative method for adjusting image quality according to an embodiment of the invention;

fig. 6 is a schematic structural diagram of an alternative image quality adjustment apparatus according to an embodiment of the present invention;

fig. 7 is a schematic structural diagram of another alternative image quality adjustment apparatus according to an embodiment of the invention;

fig. 8 is a schematic structural diagram of an alternative electronic device according to an embodiment of the invention.

Detailed Description

In order to make the technical solutions of the present invention better understood, the technical solutions in the embodiments of the present invention will be clearly and completely described below with reference to the drawings in the embodiments of the present invention, and it is obvious that the described embodiments are only a part of the embodiments of the present invention, and not all of the embodiments. All other embodiments, which can be derived by a person skilled in the art from the embodiments given herein without making any creative effort, shall fall within the protection scope of the present invention.

It should be noted that the terms "first," "second," and the like in the description and claims of the present invention and in the drawings described above are used for distinguishing between similar elements and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used is interchangeable under appropriate circumstances such that the embodiments of the invention described herein are capable of operation in sequences other than those illustrated or described herein. Furthermore, the terms "comprises," "comprising," and "having," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or apparatus that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed, but may include other steps or elements not expressly listed or inherent to such process, method, article, or apparatus.

According to an aspect of the embodiments of the present invention, there is provided an image quality adjustment method, and optionally, as an optional implementation manner, the image quality adjustment method may be applied to, but not limited to, an image quality adjustment system in a hardware environment as shown in fig. 1. The image quality adjustment system may include, but is not limited to, the terminal device 102, the network 104, and the server 106. Here, the terminal device 102 includes a human-machine interaction screen 1022, a processor 1024, and a memory 1026. The human-computer interaction screen 1022 is used for displaying the picture with the second resolution. The processor 1024 is configured to obtain the picture coding data sent by the server, decode the picture coding data to obtain a picture resource to be displayed, and adjust the image quality of the picture resource from a first resolution to a second resolution. The memory 1026 is used for storing the above-mentioned picture coding data and decoded picture resources.

The server 106 includes a database 1062 and a processing engine 1064, and the database 1062 is used to store the screen coded data. The processing engine 1064 is configured to perform rendering and encoding processing on the original picture to obtain the above picture encoded data.

The specific process comprises the following steps: in steps S102 to S104, picture coding data obtained by rendering and coding an original picture is acquired in the server 106 of the target cloud application, and is sent to the terminal device 102 through the network 104. The terminal device 102 will then perform steps S106-S110: and decoding the picture coding data to obtain the picture resource to be displayed. The terminal device 102 adjusts the image quality of the image resource from the first resolution to the second resolution, and displays the image with the image quality of the second resolution.

In addition, in this embodiment, after the picture coding data sent by the server of the target cloud application is acquired, the picture coding data is decoded to obtain the picture resource to be displayed. Then, the image quality of the image resources is adjusted from a first resolution to a second resolution to display the image with the image quality of the second resolution, wherein the first resolution is smaller than the second resolution. That is to say, after the picture coding data sent by the server of the target cloud application is acquired, the picture coding data can be decoded, and then the resolution of the image quality of the picture resource obtained by decoding is improved, and the server does not directly perform rendering coding processing on the picture with high resolution, so as to reduce the number of pixels needing to be processed by the server, thereby achieving the purpose of reducing the processing load of the server, and further overcoming the problem that the processing load of the server is aggravated in the related art.

Optionally, in this embodiment, the terminal device may be a terminal device configured with a client of the target cloud application, and may include but is not limited to at least one of the following: mobile phones (such as Android phones, iOS phones, etc.), notebook computers, tablet computers, palm computers, MID (Mobile Internet Devices), PAD, desktop computers, smart televisions, etc. Such networks may include, but are not limited to: a wired network, a wireless network, wherein the wired network comprises: a local area network, a metropolitan area network, and a wide area network, the wireless network comprising: bluetooth, WIFI, and other networks that enable wireless communication. The server may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or a cloud server providing cloud computing services. The above is merely an example, and this is not limited in this embodiment.

The target cloud application can be but not limited to an application in which a terminal and a service (cloud) terminal interact, the terminal operates a synchronous cloud terminal, and the terminal data are also reserved through cloud terminal backup when the local space is occupied. In this embodiment, the target cloud application may be a cloud education application, a cloud game application, or the like. Taking Cloud gaming (Cloud gaming) application as an example, Cloud gaming may also be called game on demand (gaming), which is an online gaming technology based on Cloud computing technology. Cloud game technology enables light-end devices (thin clients) with relatively limited graphics processing and data computing capabilities to run high-quality games. In a cloud game scene, a game is not operated in a player game terminal but in a cloud server, and the cloud server renders the game scene into a video and audio stream which is transmitted to the player game terminal through a network. The player game terminal does not need to have strong graphic operation and data processing capacity, and only needs to have basic streaming media playing capacity and capacity of acquiring player input instructions and sending the instructions to the cloud server.

Cloud computing (cloud computing) involved in the cloud game application is a computing mode, and distributes computing tasks on a resource pool formed by a large number of computers, so that various application systems can acquire computing power, storage space and information services according to needs. The network that provides the resources is referred to as the "cloud". Resources in the "cloud" appear to the user as being infinitely expandable and available at any time, available on demand, expandable at any time, and paid for on-demand.

As a basic capability provider of cloud computing, a cloud computing resource pool (called as an ifas (Infrastructure as a Service) platform for short is established, and multiple types of virtual resources are deployed in the resource pool and are selectively used by external clients.

According to the logic function division, a PaaS (Platform as a Service) layer can be deployed on an IaaS (Infrastructure as a Service) layer, a SaaS (Software as a Service) layer is deployed on the PaaS layer, and the SaaS can be directly deployed on the IaaS. PaaS is a platform on which software runs, such as a database, a web container, etc. SaaS is a variety of business software, such as web portal, sms, and mass texting. Generally speaking, SaaS and PaaS are upper layers relative to IaaS.

Optionally, as an optional implementation manner, the image quality adjustment method is applied to a client running on a terminal device, and as shown in fig. 2, the method includes: the method comprises the following steps:

s202, acquiring picture coding data sent by a server of the target cloud application;

s204, decoding the picture coding data to obtain picture resources to be displayed;

s206, adjusting the image quality of the picture resource from a first resolution to a second resolution, wherein the first resolution is smaller than the second resolution;

s208, displaying the picture with the second resolution.

Optionally, in this embodiment, the image quality adjustment method may be applied, but not limited to, in a process of performing optimization adjustment on the image quality of a virtual scene picture in a game scene in a cloud game application. In the cloud game application, the method provided in the embodiment of the present application may be adopted, after the terminal device acquires the picture encoded data from the server, the image quality of the picture resource acquired by decoding is adjusted, so as to achieve the technical effect of completing the image quality improvement operation on the terminal device side, thereby reducing the processing load of the server. The above application scenarios are examples, and this is not limited in this embodiment.

Optionally, in this embodiment, before obtaining, by a client of a target cloud application running in a terminal device, screen encoding data sent by a server of the target cloud application, the method further includes: and receiving a code stream sent by the server through a software development tool in the terminal equipment, wherein the code stream comprises picture coding data, and the picture coding data is data obtained after the server renders and codes an original picture with a first resolution.

In this embodiment, the code stream refers to a data flow used by the video file in a unit time. At the same resolution, the larger the code stream of the video file, the smaller the compression ratio, and the better the picture quality (i.e., picture quality). In addition, in this embodiment, the client may receive, but is not limited to, a code stream encoded by rendering of the server through a built-in software development tool (SDK), and decode the code stream to obtain the picture resource. Further, the image quality of the image resources is adjusted, for example, the image with the first resolution is adjusted to obtain the image with the second resolution.

Optionally, in this embodiment, the coded picture data may be decoded by, but not limited to, an Advanced Video Coding (AVC) technology, so as to obtain a picture resource in a YUV format after decoding. Wherein "Y" is used to identify brightness (luminea), i.e. gray scale value, "U" and "V" are used to identify chroma (chroma), i.e. to describe image color and saturation.

Optionally, in this embodiment, the adjusting the image quality from the first resolution to the second resolution is implemented by a target processor configured in the terminal device, where the target processor may be, but is not limited to, store a resolution adjustment network model for adjusting the image quality. Here, the resolution adjustment network model may, but is not limited to, apply a super-resolution algorithm to perform deconvolution processing on the extracted picture features in the picture, so as to further obtain the picture with the adjusted resolution based on the deconvolution result.

It should be noted that, in this embodiment, the super-resolution algorithm may be, but is not limited to, recovering a high-resolution image from a low-resolution image to recover the high-resolution image in texture details, and the edge details should be as clear as possible. The super resolution algorithm here can be, but is not limited to, adding a deconvolution process to the neural network model for generating a high resolution image. Here, the deconvolution layer is an inverse process of the convolution layer for magnifying an image by K times by a plurality of deconvolution kernels. The algorithm used here is an example, and this is not limited in this embodiment.

The description is made with reference to the example shown in fig. 3: assuming that the target cloud application is a cloud game application, the image quality adjustment method is applied to an interaction process between the cloud game server 302 and the terminal device 304. Reference is made in detail to the following steps:

assume that a container 3022 is created in the cloud game server 302 for a cloud game, the cloud game runs in the container 3022, and the OpenGLES interface is invoked to render a frame (e.g., with a resolution of 1280x720) in the cloud game. The rendering instruction is further transmitted to the GPU driver in the cloud game server 302, and is finally rendered by the GPU on the cloud game server 302 in step S302. The rendered 1280x720 pictures are sent back to the container so as to be captured by the encoder, and then step S304 is executed to encode the pictures according to the 1280x720 resolution by means of AVC encoding, and the generated AVC bitstream is sent to the cloud game client 3042 in the terminal device 304 through the network.

After the cloud game client 3042 obtains the encoded code stream, in step S306, the picture encoded data in the code stream is decoded to obtain the picture resource. Then, as in step S308, it is determined whether the NPU is configured in the terminal apparatus 304. If the NPU is determined to be configured, executing step S310-1, and adjusting the picture in the decoded picture resource by using a super-resolution algorithm in the NPU to obtain a picture with a resolution of 1920x 1080; if the NPU is not configured, step S310-2 is executed to determine whether the GPU is configured in the terminal device 304. If the configuration of the GPU is determined, executing step S312, adjusting the pictures in the decoded picture resources by using a super-resolution algorithm in the GPU to obtain pictures with a resolution of 1920x1080, and then executing step S314 to display the pictures with the resolution of 1920x 1080; if no GPU is configured, step S314 is executed, but the frame with the resolution of 1280 × 720 is directly displayed.

Optionally, in this embodiment, displaying the picture corresponding to the picture resource according to the second resolution includes: and under the condition that the target cloud application is the cloud game application, displaying the virtual scene picture in the cloud game application in the client according to the second resolution.

Optionally, in this embodiment, before acquiring the screen encoding data sent by the server of the target cloud application, the method further includes: the server determines the original resolution of an original picture, wherein the original resolution is the unprocessed resolution of the original picture; in the case that the original resolution is greater than the first resolution, the server will adjust the resolution of the original picture to the first resolution, wherein the original resolution is greater than the first resolution; and the server carries out rendering coding processing on the original picture according to the first resolution to obtain picture coding data.

It should be noted that, in this embodiment, the original resolution is a resolution of an original picture that has not been subjected to any encoding compression processing before being encoded. In order to save transmission bandwidth and resources when the original resolution is greater than the first resolution, the resolution of the original picture may be adjusted to the first resolution in this embodiment, so as to perform the rendering and encoding processing with a lower resolution. Therefore, the processing difficulty and cost of rendering and coding processing are reduced, and the transmission efficiency of the server for sending the picture coding resources to the terminal is improved. In the case where the original resolution is smaller than the first resolution, the rendering-encoding process may be performed directly using the original resolution.

In addition, the original resolution may be greater than the second resolution, or may be less than the second resolution. That is, after the image quality is adjusted from the first resolution to the second resolution in the terminal device, the second resolution may be higher than the original resolution to achieve the purpose of performing high-definition restoration processing on the image, or may be lower than the original resolution to reduce the processing difficulty in the image quality adjustment. In this embodiment, the processing may be performed in different manners according to different application scenarios.

According to the embodiment provided by the application, after the picture coding data sent by the server of the target cloud application is obtained, the picture coding data is decoded, and the picture resource to be displayed is obtained. Then, the image quality of the image resources is adjusted from a first resolution to a second resolution to display the image with the image quality of the second resolution, wherein the first resolution is smaller than the second resolution. That is to say, after the picture coding data sent by the server of the target cloud application is acquired, the picture coding data can be decoded, and then the resolution of the image quality of the picture resource obtained by decoding is improved, and the server does not directly perform rendering coding processing on the picture with high resolution, so as to reduce the number of pixels needing to be processed by the server, thereby achieving the purpose of reducing the processing load of the server, and further overcoming the problem that the processing load of the server is aggravated in the related art.

As an alternative, the adjusting the image quality of the picture resource from the first resolution to the second resolution includes:

s1, inputting the picture with the picture quality of the first resolution in the picture resources into a resolution adjustment network model under the condition that the equipment condition for adjusting the picture quality is achieved, wherein the resolution adjustment network model is a neural network model which is obtained by training a plurality of sample data and is used for adjusting the resolution of the picture quality;

and S2, acquiring a picture with the second resolution of the image quality output by the resolution adjustment network model.

Optionally, in this embodiment, after inputting the picture with the first resolution in the picture resource into the resolution adjustment network model, the method further includes: extracting picture characteristics from a picture with the picture quality of a first resolution through a resolution adjustment network model; carrying out deconvolution processing on the picture characteristics by adopting a super-resolution algorithm to obtain a deconvolution result; and calculating based on the deconvolution result to obtain a picture with the image quality of the second resolution.

For example, the super-resolution algorithm may include, but is not limited to, at least one of the following algorithms provided in the related art: 1) the neural network model-based approach includes: the method comprises the steps of image super-resolution reconstruction based on a depth recursive convolutional network, real-time single-image and video super-resolution reconstruction based on a high-efficiency sub-pixel convolutional neural network, and quick and accurate super-resolution reconstruction based on a deep laplacian pyramid network. 2) The mode of improvement based on the loss function comprises the following steps: real-time style transfer and super-resolution reconstruction based on perceptual loss, single image super-resolution reconstruction based on realistic pictures that generate a confrontational network. The above is an example, and the specific algorithm used in the present embodiment is not limited at all.

Optionally, in this embodiment, the device condition may be, but is not limited to, that a target processor is configured in a terminal device where a client of the cloud application is located, and the target processor stores a model parameter of the resolution adjustment network model. For example, embedded Neural Network Processing Units (NPUs), which may also be referred to as artificial intelligence processors or image processors (GPUs).

In this embodiment, before adjusting the image quality of the screen resource from the first resolution to the second resolution, the method further includes training a resolution adjustment model in the following manner: acquiring a plurality of sample data; inputting a plurality of sample data into the initialized resolution adjustment network model for training until the resolution adjustment network model meeting the convergence condition is obtained, wherein the convergence condition comprises one of the following conditions: and when the training iteration times reach a first threshold value, the continuous N training output results of the resolution adjustment network model are smaller than a second threshold value, wherein N is an integer larger than 2.

In this embodiment, the more the number of iterations is, the more stable the training result is, but the higher the training cost is. Thus, the first threshold herein may be, but is not limited to, set to different values according to different actual scenarios. In addition, the N continuous training output results are smaller than the second threshold, which indicates that the output of the network model has converged stably, and can be applied to an actual scene to solve corresponding problems. Thus, the second threshold herein may be, but is not limited to, set to different values according to different training requirements.

According to the embodiment provided by the application, under the condition that the equipment condition for adjusting the picture quality is achieved, the picture with the picture quality of the first resolution in the picture resources is input into the resolution adjustment network model, and the picture quality is optimized and adjusted through the resolution adjustment model and the super-resolution algorithm, so that the picture quality adjustment operation is automatically executed under the condition that the terminal equipment meets the equipment condition, the server is not relied on any more, and the aim of reducing the processing load of the server is fulfilled.

As an optional scheme, before adjusting the image quality of the picture resource from the first resolution to the second resolution, the method further includes:

and S1, under the condition that the device identification of the target processor is detected in the terminal equipment where the client of the target cloud application is located, determining the equipment condition for adjusting the picture quality, wherein the target processor stores the model parameters of the resolution adjustment network model.

Optionally, in this embodiment, the target processor may include, but is not limited to, a processor whose image Processing speed is greater than the target threshold, for example, the target processor may be an embedded Neural Network Processing Unit (NPU), which may also be referred to as an artificial intelligence processor. The method mainly adopts a data-driven parallel computing architecture to process massive multimedia data such as videos and images. For another example, the target processor may also be a Graphics Processing Unit (GPU), which is also called a display core, a visual processor, etc. The microprocessor is specially used for running drawing operation on personal computers, workstations, game machines and some mobile devices. The display control circuit is mainly used for converting and driving display information required by a computer system, providing a line scanning signal for a display and controlling the display to be correct, and is an important element for connecting the display and a personal computer mainboard. That is, when the device identifier of the processor is detected in the terminal device, it is determined that the terminal device currently has a device condition for adjusting the image quality of the decoded picture resource.

It should be noted that, after the target processor is configured to the terminal device, the model parameters of the resolution adjustment network model need to be loaded from the outside during the first operation, and are cached in the memory, so that after the picture resources to be displayed are obtained, the target processor can be applied to the picture quality adjustment process as soon as possible without waiting for extra loading time, and the processing efficiency of the target processor is improved.

According to the embodiment provided by the application, under the condition that the device identification of the target processor is detected in the terminal equipment where the client of the target cloud application is located, the equipment condition for adjusting the image quality is determined, so that whether the image quality adjustment process is automatically triggered or not is determined according to the detection result of the terminal equipment, the image quality adjustment process is performed after the equipment condition is met, and the processing efficiency of the target processor is improved.

As an optional scheme, before determining that the device condition for adjusting the picture quality is reached, the method further includes:

s1, sending an inquiry request carrying the equipment identifier of the terminal equipment to the server, wherein the inquiry request is used for inquiring the attribute information of the processor configured in the terminal equipment, and the attribute information comprises the device identifier of the processor;

s2, acquiring a processor information list inquired by the server according to the device identifier, wherein the processor information list comprises attribute information of one or more processors configured in the terminal device indicated by the device identifier;

s3, in a case where the device identification of the target processor is included in the processor information list, it is determined that the device identification of the target processor is detected.

Still taking a cloud game application as an example, the following description is specifically made in conjunction with the example shown in fig. 4, and it is assumed that the application is applied to an interaction process between the cloud game server 302 and the terminal device 304. Reference is made in detail to the following steps:

when the cloud game client 3042 in the terminal device 304 is started, in step S402, a query request carrying a device identifier of the terminal device is first sent to the database of the cloud game server 302 to query the device identifier of the processor configured in the current terminal device. Further in steps S404-S406, the cloud game server 302 determines whether the terminal device is configured with an NPU or a GPU according to the device identifier, and returns a query result to the cloud game client 3042.

The terminal device 304 executes step S408 to determine whether to use the super resolution algorithm according to the query result. Specifically, the method comprises the following steps: it is determined whether there is an NPU on the terminal device 304. If so, the resolution adjustment model running in the NPU is adopted; if not, continuously judging whether the terminal equipment has the GPU, if so, adopting a resolution adjustment model running in the GPU; if the terminal device has neither NPU nor GPU, the super-resolution reconstruction process is skipped over, and the decoded picture is directly displayed according to the resolution ratio during coding.

According to the embodiment provided by the application, under the condition of the NPU or the GPU configured in the terminal equipment, the image quality of the picture with the first resolution is improved through the resolution adjusting model stored in the processor, and the picture with the second resolution is obtained, so that the optimal adjustment operation of the image quality is executed on the terminal equipment, and the processing load on the server side is reduced.

According to an aspect of the embodiments of the present invention, there is provided a method for adjusting image quality, optionally, as an optional implementation manner, the method for adjusting image quality may be applied to a server, as shown in fig. 5, and the method includes:

s502, acquiring the original resolution of an original picture, wherein the original resolution is the unprocessed resolution of the original picture;

s504, under the condition that the original resolution is greater than the first resolution, adjusting the resolution of the original picture to the first resolution, wherein the original resolution is greater than the first resolution;

s506, rendering and coding the original picture according to the first resolution to obtain picture coded data;

and S508, sending the picture coded data to a client of the target cloud application, so that the client adjusts the image quality of the picture resources decoded from the picture coded data to a second resolution, and displays the picture with the image quality of the second resolution, wherein the first resolution is smaller than the second resolution.

Optionally, in this embodiment, the image quality adjustment method may be but is not limited to be applied to a server, where the server performs rendering and encoding processing on an original picture according to a first resolution to obtain picture encoding data includes: under the condition that an independent operation container is created for the target cloud application in the server, the server calls a target graphic rendering interface to render an original picture to obtain a rendered original picture; the server stores the rendered original picture to the operation container; and the server calls an encoder to encode the rendered original picture captured from the running container to obtain picture encoded data.

For example, taking a cloud game application as an example, the following examples are specifically combined for explanation:

assuming that the original resolution of the original picture is 1280x720, the cloud game server side will render and encode with 1280x720 resolution. Specifically, an operation container (also referred to as a container for short) is created in the cloud game server for each type of cloud game, and the cloud game application is to be operated in the container of the cloud game server and call the OpenGLES interface for rendering. The rendering instructions are further transmitted to a GPU in the cloud game server to drive the GPU to perform rendering. The rendered frames with original resolution 1280x720 are sent back to the container. And further adopting an encoder to capture the rendered pictures, then adopting an AVC coding mode to code the captured pictures according to the resolution 1280x720 so as to generate an AVC bit stream, and sending the AVC bit stream to the cloud game client through the network.

Here, without considering the bottleneck of the memory, the operation load of the GPU and the Central Processing Unit (CPU) on the cloud game server is proportional to the number of the Processing pixels. From the above situation, it can be known that, when rendering and encoding are performed at the original resolution 1280x720, compared with the rendering and encoding at the resolution 1920x1080, the relationship between the processing loads is as follows:

(1280*720)/(1920*1080)＝0.44

further, assuming that the original resolution of the original picture is 1920 × 1080, where the original resolution is greater than 1280 × 720 used for rendering coding, the original picture is further adjusted so as to meet the requirements of rendering coding, and is encoded at 1280 × 720, which is a lower resolution, to generate an AVC bitstream, and the AVC bitstream is transmitted to the cloud game client via the network.

According to the embodiment provided by the application, the lower first resolution is adopted to perform rendering coding on the server side, so that the aims of reducing the data processing amount of the cloud game server and improving the processing efficiency are fulfilled. In addition, the effect of improving the transmission efficiency between the cloud game server and the cloud game client can be further realized.

For specific embodiments, reference may be made to the above embodiments, which are not described herein again.

The beneficial effects of the above embodiments on the picture quality improvement are specifically described with reference to the following contents:

assuming that the super-resolution reconstruction adopts a mature FSRCNN-s algorithm, it can be known from the performance data of the FSRCNN-s algorithm that the processing time of 2 times super-resolution processing on a common CPU (such as a processor with Intel i 74 GHz main frequency) is 24ms to 61ms, as shown in Table 1.

TABLE 1

The general CPU with Intel i 74 GHz main frequency is assumed to be 8 cores, and the single core has 4GFLOPS capability, and has 32 GFLOPS. In contrast, the FSCNN-s algorithm is run on the NPU, with time roughly:

(1280*720)/(500*500)*61ms/60＝3.7ms

that is, if the terminal device is equipped with the artificial intelligence chip NPU, after decoding a 1280x720 picture, the cloud game client can reconstruct a 1920x1080 picture by using the super-resolution algorithm only by 3.7ms, and then send the picture to the terminal device for display.

Assuming that the client decodes a picture having a resolution of 1280x720, the required decoding time is about 16 ms. The total processing time is here 19.7ms in total, and most of the 30fps game frames can be processed.

If the NPU is not equipped in the terminal equipment, the fact that the GPU is not equipped in the terminal equipment is continuously judged. At present, GPUs in the related art have a processing capacity of more than 500GFLOPS, and assuming that Google pixel3 is taken as an example, the performance of an Adreno 630 GPU equipped in the terminal device (i.e., a mobile phone) is 727 GFLOPS. The GPU runs the FSCNN-s super-resolution algorithm, and the time consumption is as follows:

3.7ms*1.92 TFLOPS/0.5 TFLOPS＝14.8ms

the total processing time here would be 4.8 ms.

And if the fact that the terminal equipment has neither NPU nor GPU is detected, skipping a super-resolution process, and sending the directly decoded picture to a display screen of the terminal equipment for displaying.

It should be noted that, for simplicity of description, the above-mentioned method embodiments are described as a series of acts or combination of acts, but those skilled in the art will recognize that the present invention is not limited by the order of acts, as some steps may occur in other orders or concurrently in accordance with the invention. Further, those skilled in the art should also appreciate that the embodiments described in the specification are preferred embodiments and that the acts and modules referred to are not necessarily required by the invention.

According to another aspect of the embodiments of the present invention, there is also provided an image quality adjusting apparatus for implementing the image quality adjusting method, the apparatus being applied in a terminal device. As shown in fig. 6, the apparatus includes:

1) an obtaining unit 602, configured to obtain picture coding data sent by a server of a target cloud application;

2) a decoding unit 604, configured to decode the picture coding data to obtain a picture resource to be displayed;

3) an adjusting unit 606, configured to adjust the image quality of the image resource from a first resolution to a second resolution, where the first resolution is smaller than the second resolution;

4) the display unit 608 is configured to display a picture with the second resolution.

Optionally, in this embodiment, as shown in fig. 7, the adjusting unit 606 includes: an input module 702, configured to input, to a resolution adjustment network model, a picture with a first resolution in picture resources when a device condition for adjusting picture quality is met, where the resolution adjustment network model is a neural network model for adjusting the resolution of the picture quality, and is obtained after training with a plurality of sample data; the obtaining module 704 is configured to obtain a picture with the second resolution output by the resolution adjustment network model.

Optionally, in this embodiment, the apparatus is further configured to: after inputting a picture with the first resolution in picture resources into a resolution adjustment network model, extracting picture features from the picture with the first resolution through the resolution adjustment network model; carrying out deconvolution processing on the picture characteristics by adopting a super-resolution algorithm to obtain a deconvolution result; and calculating based on the deconvolution result to obtain a picture with the image quality of the second resolution.

Optionally, in this embodiment, the apparatus is further configured to: before the image quality of the image resource is adjusted from the first resolution to the second resolution, determining an equipment condition for adjusting the image quality under the condition that a device identification of a target processor is detected in terminal equipment where a client of a target cloud application is located, wherein the target processor stores model parameters of a resolution adjustment network model.

Optionally, in this embodiment, the apparatus is further configured to: acquiring a plurality of sample data before adjusting the image quality of the image resource from a first resolution to a second resolution; inputting a plurality of sample data into the initialized resolution adjustment network model for training until the resolution adjustment network model meeting the convergence condition is obtained, wherein the convergence condition comprises one of the following conditions: and when the training iteration times reach a first threshold value, the continuous N training output results of the resolution adjustment network model are smaller than a second threshold value, wherein N is an integer larger than 2.

Optionally, in this embodiment, the apparatus is further configured to: before determining that the equipment condition for adjusting the picture quality is achieved, sending a query request carrying an equipment identifier of the terminal equipment to a server, wherein the query request is used for querying attribute information of a processor configured in the terminal equipment, and the attribute information comprises a device identifier of the processor; acquiring a processor information list inquired by a server according to an equipment identifier, wherein the processor information list comprises attribute information of one or more processors configured in terminal equipment indicated by the equipment identifier; in a case where the device identification of the target processor is included in the processor information list, it is determined that the device identification of the target processor is detected.

Optionally, in this embodiment, the specific embodiment may refer to the above-mentioned embodiment, and this embodiment is not described herein again.

According to another aspect of the embodiment of the present invention, there is also provided an electronic device for implementing the image quality adjustment method, where the electronic device may be the terminal device shown in fig. 1. As shown in fig. 8, the electronic device comprises a memory 802 and a processor 804, the memory 802 having a computer program stored therein, the processor 804 being arranged to perform the steps of any of the above-described method embodiments by means of the computer program.

Optionally, in this embodiment, the electronic device may be located in at least one network device of a plurality of network devices of a computer network.

Optionally, in this embodiment, the processor may be configured to execute the following steps by a computer program:

s1, acquiring picture coding data sent by a server of the target cloud application;

s2, decoding the picture coding data to obtain the picture resource to be displayed;

s3, adjusting the image quality of the picture resource from a first resolution to a second resolution, wherein the first resolution is smaller than the second resolution;

s4, displaying the picture with the second resolution.

Alternatively, it can be understood by those skilled in the art that the structure shown in fig. 8 is only an illustration, and the electronic device may also be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palmtop computer, a Mobile Internet Device (MID), a PAD, and the like. Fig. 8 is a diagram illustrating a structure of the electronic device. For example, the electronics may also include more or fewer components (e.g., network interfaces, etc.) than shown in FIG. 8, or have a different configuration than shown in FIG. 8.

The memory 802 may be used to store software programs and modules, such as program instructions/modules corresponding to the image quality adjusting method and apparatus according to the embodiments of the present invention, and the processor 804 executes various functional applications and data processing by operating the software programs and modules stored in the memory 802, so as to implement the image quality adjusting method. The memory 802 may include high-speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory 802 can further include memory located remotely from the processor 804, which can be connected to the terminal over a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The memory 802 may be, but not limited to, specifically used for storing information such as decoded picture resources. As an example, as shown in fig. 8, the memory 802 may include, but is not limited to, an obtaining unit 602, a decoding unit 604, an adjusting unit 606, and a displaying unit 608 of the image quality adjusting apparatus. In addition, the image quality adjusting apparatus may further include, but is not limited to, other module units in the image quality adjusting apparatus, which is not described in detail in this example.

Optionally, the transmitting device 806 is configured to receive or transmit data via a network. Examples of the network may include a wired network and a wireless network. In one example, the transmission device 806 includes a Network adapter (NIC) that can be connected to a router via a Network cable and other Network devices to communicate with the internet or a local area Network. In one example, the transmission device 806 is a Radio Frequency (RF) module, which is used for communicating with the internet in a wireless manner.

In addition, the electronic device further includes: a display 808 for displaying the picture with the adjusted resolution; and a connection bus 810 for connecting the respective module parts in the above-described electronic apparatus.

In other embodiments, the terminal device may be a node in a distributed system, where the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting a plurality of nodes through a network communication. Nodes can form a Peer-To-Peer (P2P, Peer To Peer) network, and any type of computing device, such as a server, a terminal, and other electronic devices, can become a node in the blockchain system by joining the Peer-To-Peer network.

According to another aspect of the embodiment of the present invention, there is also provided an electronic device for implementing the image quality adjustment method, where the electronic device may be a server shown in fig. 1. The electronic device comprises a memory having a computer program stored therein and a processor arranged to perform the steps of any of the above method embodiments by means of the computer program.

s1, acquiring the original resolution of the original picture, wherein the original resolution is the unprocessed resolution of the original picture;

s2, under the condition that the original resolution is larger than the first resolution, adjusting the resolution of the original picture to the first resolution, wherein the original resolution is larger than the first resolution;

s3, performing rendering coding processing on the original picture according to the first resolution to obtain picture coded data;

and S4, sending the picture coded data to the client of the target cloud application, so that the client adjusts the image quality of the picture resources decoded from the picture coded data to a second resolution and displays the picture with the image quality of the second resolution, wherein the first resolution is smaller than the second resolution.

Alternatively, as will be understood by those skilled in the art, the electronic device may also be a terminal device such as a smart phone (e.g., an Android phone, an iOS phone, etc.), a tablet computer, a palm computer, and a Mobile Internet Device (MID), a PAD, etc.

The memory may be used to store software programs and modules, such as program instructions/modules corresponding to the image quality adjustment method and apparatus in the embodiments of the present invention, and the processor executes various functional applications and data processing by running the software programs and modules stored in the memory, so as to implement the image quality adjustment method. The memory may include high speed random access memory, and may also include non-volatile memory, such as one or more magnetic storage devices, flash memory, or other non-volatile solid-state memory. In some examples, the memory may further include memory located remotely from the processor, and these remote memories may be connected to the terminal through a network. Examples of such networks include, but are not limited to, the internet, intranets, local area networks, mobile communication networks, and combinations thereof. The memory may be, but not limited to, specifically configured to store information such as encoded picture data.

Optionally, the transmission device is used for receiving or sending data via a network. Examples of the network may include a wired network and a wireless network. In one example, the transmission device includes a Network adapter (NIC) that can be connected to the router via a Network cable and other Network devices to communicate with the internet or a local area Network. In one example, the transmission device is a Radio Frequency (RF) module, which is used to communicate with the internet in a wireless manner.

In addition, the electronic device further includes: a display for displaying the rendered picture; and a connection bus for connecting the respective module parts in the electronic apparatus.

In other embodiments, the server may be a node in a distributed system, wherein the distributed system may be a blockchain system, and the blockchain system may be a distributed system formed by connecting a plurality of nodes through a network communication. Nodes can form a Peer-To-Peer (P2P, Peer To Peer) network, and any type of computing device, such as a server, a terminal, and other electronic devices, can become a node in the blockchain system by joining the Peer-To-Peer network.

According to a further aspect of an embodiment of the present invention, there is also provided a computer-readable storage medium having a computer program stored thereon, wherein the computer program is arranged to perform the steps of any of the above method embodiments when executed.

Alternatively, in the present embodiment, the above-mentioned computer-readable storage medium may be configured to store a computer program for executing the steps of:

s4, displaying the picture with the second resolution.

Optionally, in this embodiment, the computer-readable storage medium may be further configured to store a computer program for executing the following steps:

Alternatively, in this embodiment, a person skilled in the art may understand that all or part of the steps in the methods of the foregoing embodiments may be implemented by a program instructing hardware associated with the terminal device, where the program may be stored in a computer-readable storage medium, and the storage medium may include: flash disks, Read-Only memories (ROMs), Random Access Memories (RAMs), magnetic or optical disks, and the like.

The above-mentioned serial numbers of the embodiments of the present invention are merely for description and do not represent the merits of the embodiments.

The integrated unit in the above embodiments, if implemented in the form of a software functional unit and sold or used as a separate product, may be stored in the above computer-readable storage medium. Based on such understanding, the technical solution of the present invention may be embodied in the form of a software product, which is stored in a storage medium and includes several instructions for causing one or more computer devices (which may be personal computers, servers, network devices, etc.) to execute all or part of the steps of the method according to the embodiments of the present invention.

In the above embodiments of the present invention, the descriptions of the respective embodiments have respective emphasis, and for parts that are not described in detail in a certain embodiment, reference may be made to related descriptions of other embodiments.

In the several embodiments provided in the present application, it should be understood that the disclosed client may be implemented in other manners. The above-described embodiments of the apparatus are merely illustrative, and for example, the division of the units is only one type of division of logical functions, and there may be other divisions when actually implemented, for example, a plurality of units or components may be combined or may be integrated into another system, or some features may be omitted, or not executed. In addition, the shown or discussed mutual coupling or direct coupling or communication connection may be an indirect coupling or communication connection through some interfaces, units or modules, and may be in an electrical or other form.

The units described as separate parts may or may not be physically separate, and parts displayed as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units can be selected according to actual needs to achieve the purpose of the solution of the embodiment.

In addition, functional units in the embodiments of the present invention may be integrated into one processing unit, or each unit may exist alone physically, or two or more units are integrated into one unit. The integrated unit can be realized in a form of hardware, and can also be realized in a form of a software functional unit.

The foregoing is only a preferred embodiment of the present invention, and it should be noted that, for those skilled in the art, various modifications and decorations can be made without departing from the principle of the present invention, and these modifications and decorations should also be regarded as the protection scope of the present invention.

Claims

1. An image quality adjustment method, comprising:

sending a query request carrying the equipment identifier of the terminal equipment where the client of the target cloud application is located to a server of the target cloud application; the server determines whether an embedded neural Network Processor (NPU) or an image processor (GPU) is configured in the terminal equipment or not according to the equipment identifier;

receiving a query result returned by the server;

acquiring picture coding data sent by a server of a target cloud application, wherein the picture coding data is data obtained after the server renders and codes an original picture with a first resolution;

decoding the picture coding data to obtain picture resources to be displayed;

determining whether to adopt a resolution adjustment network model according to the query result, including: judging whether the terminal equipment is configured with an NPU (network provider Unit), if so, adjusting the image quality of the image resources from a first resolution to a second resolution by adopting a resolution adjustment network model operated in the NPU, wherein the first resolution is smaller than the second resolution; if the NPU is not configured, judging whether the terminal equipment is configured with the GPU, and if the terminal equipment is configured with the GPU, adjusting the image quality of the image resources from a first resolution to a second resolution by adopting a resolution adjustment network model operated in the GPU; displaying the picture with the picture quality of the second resolution;

and if the NPU and the GPU are not configured, directly displaying the picture with the resolution being the first resolution.

2. The method of claim 1, wherein adjusting the quality of the picture resource from a first resolution to a second resolution comprises:

inputting a picture with the picture quality of the first resolution in the picture resources into a resolution adjustment network model, wherein the resolution adjustment network model is a neural network model for adjusting the resolution of the picture quality, which is obtained after training by using a plurality of sample data;

and acquiring the picture quality output by the resolution adjustment network model as the picture of the second resolution.

3. The method of claim 2, wherein after inputting the picture of the picture resource at the first resolution into the resolution adjustment network model, further comprising:

extracting picture characteristics from the picture with the picture quality of the first resolution through the resolution adjustment network model;

performing deconvolution processing on the picture characteristics by adopting a super-resolution algorithm to obtain a deconvolution result;

and calculating based on the deconvolution result to obtain a picture with the image quality of the second resolution.

4. The method of claim 2, wherein the NPU or GPU has model parameters for the resolution-adjusted network model stored therein.

5. The method of claim 2, further comprising, prior to the adjusting the quality of the picture resource from the first resolution to the second resolution:

acquiring the plurality of sample data;

inputting the sample data into an initialized resolution adjustment network model for training until the resolution adjustment network model meeting a convergence condition is obtained, wherein the convergence condition comprises one of the following conditions: the number of training iterations reaches a first threshold, and N continuous training output results of the resolution adjustment network model are smaller than a second threshold, wherein N is an integer larger than 2.

6. The method according to claim 1, wherein before acquiring the picture encoding data transmitted by the server of the target cloud application, the method further comprises:

the server determines the original resolution of an original picture, wherein the original resolution is the unprocessed resolution of the original picture;

in the case that the original resolution is greater than the first resolution, the server will adjust the resolution of the original picture to the first resolution, wherein the original resolution is greater than the first resolution;

and the server carries out rendering coding processing on the original picture according to the first resolution to obtain the picture coding data.

7. The method according to any one of claims 1 to 6, wherein the displaying the picture with the picture quality at the second resolution comprises:

and displaying a game picture in the cloud game application in the client according to the second resolution when the target cloud application is the cloud game application.

8. An image quality adjustment method, comprising:

receiving an inquiry request which is sent by a client of a target cloud application and carries an equipment identifier of a terminal equipment where the client is located;

determining whether an embedded neural Network Processor (NPU) or an image processor (GPU) is configured in the terminal equipment according to the equipment identifier;

returning a query result to a client of the target cloud application;

acquiring an original resolution of an original picture, wherein the original resolution is the unprocessed resolution of the original picture;

if the original resolution is greater than a first resolution, adjusting the resolution of the original picture to the first resolution, wherein the original resolution is greater than the first resolution;

rendering and coding the original picture according to the first resolution to obtain picture coded data;

sending the picture coding data to a client of a target cloud application so that the client can determine whether to adopt a resolution adjustment network model according to the query result, wherein the method comprises the following steps: judging whether the terminal equipment is configured with an NPU or not, if so, adjusting the image quality of the picture resources decoded from the picture coding data to a second resolution by adopting a resolution adjusting network model operated in the NPU, if not, judging whether a GPU is configured in the terminal equipment, and if so, adjusting the image quality of the picture resources from a first resolution to a second resolution by adopting a resolution adjusting network model operated in the GPU; displaying a picture with the picture quality of the second resolution, wherein the first resolution is smaller than the second resolution; and if the NPU and the GPU are not configured, directly displaying the picture with the resolution being the first resolution.

9. The image quality adjusting device is characterized by being used for sending a query request carrying an equipment identifier of a terminal device where a client of a target cloud application is located to a server of the target cloud application; the server determines whether an embedded neural Network Processor (NPU) or an image processor (GPU) is configured in the terminal equipment or not according to the equipment identifier; receiving a query result returned by the server; the device comprises:

the system comprises an acquisition unit, a processing unit and a processing unit, wherein the acquisition unit is used for acquiring picture coding data sent by a server of target cloud application, and the picture coding data is data obtained by rendering and coding an original picture by the server with a first resolution;

the decoding unit is used for decoding the picture coding data to obtain picture resources to be displayed;

the adjusting unit is used for determining whether to adopt a resolution ratio adjusting network model according to the query result, and comprises: judging whether the terminal equipment is configured with an NPU (network provider Unit), if so, adjusting the image quality of the image resources from a first resolution to a second resolution by adopting a resolution adjustment network model operated in the NPU, wherein the first resolution is smaller than the second resolution; if the NPU is not configured, judging whether the terminal equipment is configured with the GPU, and if the terminal equipment is configured with the GPU, adjusting the image quality of the image resources from a first resolution to a second resolution by adopting a resolution adjustment network model operated in the GPU; if the NPU and the GPU are not configured, skipping the super-resolution reconstruction process;

and the display unit is used for displaying the picture with the image quality of the second resolution or the picture with the resolution of the first resolution.

10. A computer-readable storage medium comprising a stored program, wherein the program when executed performs the method of any of claims 1 to 7 or claim 8.

11. An electronic device comprising a memory and a processor, wherein the memory has stored therein a computer program and the processor is arranged to execute the method of any of claims 1 to 7 or 8 by means of the computer program.