CN116567287A - Image quality enhancement method and device and electronic equipment - Google Patents
Image quality enhancement method and device and electronic equipment Download PDFInfo
- Publication number
- CN116567287A CN116567287A CN202210103029.7A CN202210103029A CN116567287A CN 116567287 A CN116567287 A CN 116567287A CN 202210103029 A CN202210103029 A CN 202210103029A CN 116567287 A CN116567287 A CN 116567287A
- Authority
- CN
- China
- Prior art keywords
- image quality
- quality enhancement
- region
- terminal equipment
- interest
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 114
- 230000002708 enhancing effect Effects 0.000 claims abstract description 22
- 238000001514 detection method Methods 0.000 claims description 294
- 230000006870 function Effects 0.000 claims description 183
- 238000012360 testing method Methods 0.000 claims description 131
- 238000004590 computer program Methods 0.000 claims description 20
- 238000003860 storage Methods 0.000 abstract description 20
- 238000005516 engineering process Methods 0.000 abstract description 4
- 238000013473 artificial intelligence Methods 0.000 abstract description 3
- 238000009877 rendering Methods 0.000 description 52
- 238000012545 processing Methods 0.000 description 30
- 238000013139 quantization Methods 0.000 description 22
- 208000037170 Delayed Emergence from Anesthesia Diseases 0.000 description 17
- 238000010586 diagram Methods 0.000 description 16
- 238000001914 filtration Methods 0.000 description 14
- 230000004044 response Effects 0.000 description 12
- 230000006835 compression Effects 0.000 description 10
- 238000007906 compression Methods 0.000 description 10
- 238000004422 calculation algorithm Methods 0.000 description 9
- 238000004891 communication Methods 0.000 description 9
- 230000033001 locomotion Effects 0.000 description 9
- 238000004458 analytical method Methods 0.000 description 8
- 238000004364 calculation method Methods 0.000 description 7
- 230000009466 transformation Effects 0.000 description 7
- 230000003993 interaction Effects 0.000 description 6
- 230000008569 process Effects 0.000 description 6
- 230000000694 effects Effects 0.000 description 5
- 230000005540 biological transmission Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 101150114515 CTBS gene Proteins 0.000 description 2
- 241000023320 Luma <angiosperm> Species 0.000 description 2
- 230000003044 adaptive effect Effects 0.000 description 2
- 230000000903 blocking effect Effects 0.000 description 2
- 238000013500 data storage Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 239000011159 matrix material Substances 0.000 description 2
- OSWPMRLSEDHDFF-UHFFFAOYSA-N methyl salicylate Chemical compound COC(=O)C1=CC=CC=C1O OSWPMRLSEDHDFF-UHFFFAOYSA-N 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000000638 solvent extraction Methods 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 230000001360 synchronised effect Effects 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 230000001131 transforming effect Effects 0.000 description 2
- 230000000007 visual effect Effects 0.000 description 2
- 238000003491 array Methods 0.000 description 1
- 238000013461 design Methods 0.000 description 1
- 238000009826 distribution Methods 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 238000010191 image analysis Methods 0.000 description 1
- 239000004973 liquid crystal related substance Substances 0.000 description 1
- 230000003287 optical effect Effects 0.000 description 1
- 238000005457 optimization Methods 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 238000007781 pre-processing Methods 0.000 description 1
- 238000012958 reprocessing Methods 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000007787 solid Substances 0.000 description 1
- 238000001228 spectrum Methods 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
- 230000002123 temporal effect Effects 0.000 description 1
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/40—Client devices specifically adapted for the reception of or interaction with content, e.g. set-top-box [STB]; Operations thereof
- H04N21/41—Structure of client; Structure of client peripherals
- H04N21/426—Internal components of the client ; Characteristics thereof
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04L—TRANSMISSION OF DIGITAL INFORMATION, e.g. TELEGRAPHIC COMMUNICATION
- H04L67/00—Network arrangements or protocols for supporting network services or applications
- H04L67/01—Protocols
- H04L67/12—Protocols specially adapted for proprietary or special-purpose networking environments, e.g. medical networks, sensor networks, networks in vehicles or remote metering networks
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N19/00—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals
- H04N19/10—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding
- H04N19/134—Methods or arrangements for coding, decoding, compressing or decompressing digital video signals using adaptive coding characterised by the element, parameter or criterion affecting or controlling the adaptive coding
- H04N19/146—Data rate or code amount at the encoder output
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N21/00—Selective content distribution, e.g. interactive television or video on demand [VOD]
- H04N21/20—Servers specifically adapted for the distribution of content, e.g. VOD servers; Operations thereof
- H04N21/21—Server components or server architectures
- H04N21/226—Characteristics of the server or Internal components of the server
-
- Y—GENERAL TAGGING OF NEW TECHNOLOGICAL DEVELOPMENTS; GENERAL TAGGING OF CROSS-SECTIONAL TECHNOLOGIES SPANNING OVER SEVERAL SECTIONS OF THE IPC; TECHNICAL SUBJECTS COVERED BY FORMER USPC CROSS-REFERENCE ART COLLECTIONS [XRACs] AND DIGESTS
- Y02—TECHNOLOGIES OR APPLICATIONS FOR MITIGATION OR ADAPTATION AGAINST CLIMATE CHANGE
- Y02D—CLIMATE CHANGE MITIGATION TECHNOLOGIES IN INFORMATION AND COMMUNICATION TECHNOLOGIES [ICT], I.E. INFORMATION AND COMMUNICATION TECHNOLOGIES AIMING AT THE REDUCTION OF THEIR OWN ENERGY USE
- Y02D30/00—Reducing energy consumption in communication networks
- Y02D30/70—Reducing energy consumption in communication networks in wireless communication networks
Landscapes
- Engineering & Computer Science (AREA)
- Signal Processing (AREA)
- Multimedia (AREA)
- Health & Medical Sciences (AREA)
- Computing Systems (AREA)
- General Health & Medical Sciences (AREA)
- Medical Informatics (AREA)
- Computer Networks & Wireless Communication (AREA)
- Compression Or Coding Systems Of Tv Signals (AREA)
Abstract
The application provides a method, a device, equipment and a storage medium for enhancing image quality, which can be applied to various scenes such as cloud technology, artificial intelligence, intelligent traffic, auxiliary driving, video and the like, and the method comprises the following steps: the cloud server determines image quality enhancement information of the corresponding terminal equipment, and skips image quality enhancement operation and encodes a target video which is not subjected to image quality enhancement if the image quality enhancement information of the terminal equipment indicates that preset image quality enhancement conditions are met, so as to obtain a first code stream; if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is not met, the cloud server performs complete image quality enhancement operation on the target video and encodes the target video with enhanced image quality to obtain a second code stream. That is, according to the method and the device, the cloud server judges whether the image quality enhancement operation is executed by the terminal device or the cloud server according to the image quality enhancement information of the terminal device, so that the computing capability of a decoding end can be fully utilized, the computing resource of an encoding end can be saved, and the encoding efficiency is improved.
Description
Technical Field
The embodiment of the application relates to the technical field of image processing, in particular to an image quality enhancement method, an image quality enhancement device, image quality enhancement equipment and a storage medium.
Background
In the process of video acquisition, the acquired video image quality cannot meet the requirements due to shaking of photographic equipment, rapid movement of objects and the like. In order to improve the display effect of the video or improve the accuracy of task analysis based on the video, the video needs to be subjected to image quality enhancement processing.
Because the computing power of the cloud server is strong, the image quality enhancement operation of the video is usually realized at the encoding end. For example, before encoding a video to be encoded, the cloud server first performs image quality enhancement on the video, and then encodes the video with the image quality enhanced.
However, the cloud server needs to process more tasks, and when the cloud server is used for enhancing the image quality of the video, the workload of the cloud server is additionally increased, the computing resources of the cloud server are excessively consumed, and the coding efficiency is further reduced.
Disclosure of Invention
The application provides an image quality enhancement method, an image quality enhancement device and a storage medium, so that the workload of a cloud server is reduced, and the coding efficiency is improved.
In a first aspect, the present application provides an image quality enhancement method, applied to a cloud server, including:
determining image quality enhancement information of corresponding terminal equipment;
If the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is met, skipping image quality enhancement operation, and encoding a target video which is not subjected to image quality enhancement to obtain a first code stream;
and if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is not met, carrying out image quality enhancement on the target video, and encoding the target video with the enhanced image quality to obtain a second code stream.
In a second aspect, the present application provides an image quality enhancement method, applied to a terminal device, including:
decoding the obtained code stream to obtain a target video;
if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is met, performing image quality enhancement on the decoded target video to obtain the target video with enhanced image quality;
and if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is not met, taking the decoded target video as the target video with enhanced image quality.
In a third aspect, the present application provides an image quality enhancement device, applied to a cloud server, including:
A determining unit for determining image quality enhancement information of the corresponding terminal device;
the coding unit is used for skipping image quality enhancement operation and coding the target video which is not subjected to image quality enhancement if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is met, so as to obtain a first code stream; and if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is not met, carrying out image quality enhancement on the target video, and encoding the target video with the enhanced image quality to obtain a second code stream.
In a fourth aspect, the present application provides an image quality enhancement apparatus, applied to a terminal device, including:
the decoding unit is used for decoding the acquired code stream to obtain a target video;
the processing unit is used for carrying out image quality enhancement on the decoded target video to obtain the target video with enhanced image quality if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is met; and if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is not met, taking the decoded target video as the target video with enhanced image quality.
In a fifth aspect, there is provided an electronic device comprising: a processor and a memory for storing a computer program, the processor being for invoking and running the computer program stored in the memory to perform the method of the first or second aspect.
In a sixth aspect, a computer readable storage medium is provided for storing a computer program for causing a computer to perform the method of the first or second aspect.
A seventh aspect provides a chip for implementing the method of any one of the first to second aspects or each implementation manner thereof. Specifically, the chip includes: a processor for calling and running a computer program from a memory, so that a device on which the chip is mounted performs the method as in any one of the first to second aspects or implementation manners thereof.
In an eighth aspect, there is provided a computer program product comprising computer program instructions for causing a computer to perform the method of any one of the first to second aspects or implementations thereof.
A ninth aspect provides a computer program which, when run on a computer, causes the computer to perform the method of any one of the first to second aspects or implementations thereof.
In summary, in the present application, the cloud server determines image quality enhancement information of a corresponding terminal device; if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is met, skipping image quality enhancement operation, and encoding a target video which is not subjected to image quality enhancement to obtain a first code stream; and if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is not met, enhancing the image quality of the target video, and encoding the target video with enhanced image quality to obtain a second code stream. In other words, in the embodiment of the present application, the cloud server determines, according to the image quality enhancement information of the terminal device, whether the image quality enhancement operation is performed by the terminal device or the cloud server, so that the computing capability of the decoding end can be fully utilized, and computing resources of the encoding end can be saved, thereby improving the encoding efficiency.
Drawings
In order to more clearly illustrate the technical solutions of the embodiments of the present invention, the drawings required for the description of the embodiments will be briefly described below, and it is apparent that the drawings in the following description are only some embodiments of the present invention, and other drawings may be obtained according to these drawings without inventive effort for a person skilled in the art.
Fig. 1 is a schematic view of an application scenario according to an embodiment of the present application;
FIG. 2 is a schematic block diagram of a video encoder provided by an embodiment of the present application;
FIG. 3 is a schematic block diagram of a video decoder provided by an embodiment of the present application;
FIG. 4A is a schematic diagram of task allocation;
FIG. 4B is a schematic diagram of another task allocation;
FIG. 4C is a schematic diagram of another task allocation;
FIG. 4D is a schematic diagram of another task allocation;
FIG. 5 is a flowchart of an image quality enhancement method according to an embodiment of the present disclosure;
FIG. 6 is a flowchart of an image enhancement method according to an embodiment of the present disclosure;
FIG. 7 is a flowchart of an image enhancement method according to an embodiment of the present disclosure;
FIG. 8 is a schematic diagram illustrating an interaction flow of an image quality enhancement method according to an embodiment of the present application
FIG. 9 is a schematic diagram of an image enhancement device according to an embodiment of the present disclosure;
FIG. 10 is a schematic structural diagram of an image enhancement device according to an embodiment of the present disclosure;
fig. 11 is a schematic block diagram of an electronic device provided in an embodiment of the present application.
Detailed Description
The following description of the embodiments of the present invention will be made clearly and completely with reference to the accompanying drawings, in which it is apparent that the embodiments described are only some embodiments of the present invention, but not all embodiments. All other embodiments, which can be made by those skilled in the art based on the embodiments of the invention without making any inventive effort, are intended to be within the scope of the invention.
It should be noted that the terms "first," "second," and the like in the description and the claims of the present invention and the above figures are used for distinguishing between similar objects and not necessarily for describing a particular sequential or chronological order. It is to be understood that the data so used may be interchanged where appropriate such that the embodiments of the invention described herein may be implemented in sequences other than those illustrated or otherwise described herein. Furthermore, the terms "comprises," "comprising," and any variations thereof, are intended to cover a non-exclusive inclusion, such that a process, method, system, article, or server that comprises a list of steps or elements is not necessarily limited to those steps or elements expressly listed or inherent to such process, method, article, or apparatus.
Fig. 1 is a schematic view of an application scenario according to an embodiment of the present application, including a cloud server 101 and a terminal device 102. The cloud server 101 may be understood as an encoding device, and the terminal device 102 may be understood as a decoding device.
The cloud server 101 is configured to encode (may be understood as compressing) video data to generate a code stream, and transmit the code stream to the terminal device 102.
The cloud server 101 of the present embodiment may be understood as a device with a video encoding function, and the terminal device 102 may be understood as a device with a video decoding function, that is, the embodiments of the present application include a wider device for the cloud server 101 and the terminal device 102, such as a smart phone, a desktop computer, a mobile computing device, a notebook (e.g., laptop) computer, a tablet computer, a set-top box, a television, a camera, a display device, a digital media player, a video game console, a vehicle-mounted computer, and the like.
In some embodiments, cloud server 101 may transmit encoded video data (e.g., a code stream) to terminal device 102 via channel 103. Channel 103 may include one or more media and/or devices capable of transmitting encoded video data from cloud server 101 to terminal device 102.
In one example, channel 103 includes one or more communication media that enable cloud server 101 to transmit encoded video data directly to terminal device 102 in real-time. In this example, cloud server 101 may modulate the encoded video data according to a communication standard and transmit the modulated video data to terminal device 102. Where the communication medium comprises a wireless communication medium, such as a radio frequency spectrum, the communication medium may optionally also comprise a wired communication medium, such as one or more physical transmission lines.
In another example, channel 103 comprises a storage medium that may store video data encoded by cloud server 101. Storage media include a variety of locally accessed data storage media such as compact discs, DVDs, flash memory, and the like. In this example, the terminal device 102 may obtain encoded video data from the storage medium.
In another example, channel 103 may comprise a storage server that may store video data encoded by cloud server 101. In this example, the terminal device 102 may download stored encoded video data from the storage server. Alternatively, the storage server may store the encoded video data and may transmit the encoded video data to the terminal device 102, such as a web server (e.g., for a website), a File Transfer Protocol (FTP) server, or the like.
In some embodiments, cloud server 101 includes a video encoder and an output interface. The output interface may comprise, among other things, a modulator/demodulator (modem) and/or a transmitter.
In some embodiments, cloud server 101 may include a video source in addition to a video encoder and an input interface.
The video source may include at least one of a video capture device (e.g., a video camera), a video archive, a video input interface for receiving video data from a video content provider, a computer graphics system for generating video data.
A video encoder encodes video data from a video source to produce a bitstream. The video data may include one or more pictures (pictures) or sequences of pictures (sequence of pictures). The code stream contains encoded information of the image or image sequence in the form of a bit stream. The encoded information may include encoded image data and associated data. The associated data may include a sequence parameter set (sequence parameter set, SPS for short), a picture parameter set (picture parameter set, PPS for short), and other syntax structures. An SPS may contain parameters that apply to one or more sequences. PPS may contain parameters that apply to one or more pictures. A syntax structure refers to a set of zero or more syntax elements arranged in a specified order in a bitstream.
The video encoder transmits the encoded video data directly to the terminal device 102 via an output interface. The encoded video data may also be stored on a storage medium or a storage server for subsequent reading by the terminal device 102.
In some embodiments, the terminal device 102 includes an input interface and a video decoder.
In some embodiments, the terminal device 102 may include a display device in addition to the input interface and the video decoder.
Wherein the input interface comprises a receiver and/or a modem. The input interface may receive the encoded video data through a channel.
The video decoder is used for decoding the encoded video data to obtain decoded video data, and transmitting the decoded video data to the display device.
The display device displays the decoded video data. The display means may be integrated with the terminal device or external to the terminal device. The display device may include a variety of display devices, such as a Liquid Crystal Display (LCD), a plasma display, an Organic Light Emitting Diode (OLED) display, or other types of display devices.
Alternatively, the cloud server 101 may be one or more. When the cloud server 101 is plural, there are at least two servers for providing different services, and/or there are at least two servers for providing the same service, for example, providing the same service in a load balancing manner, which is not limited in the embodiment of the present application.
Alternatively, the cloud server 101 may be an independent physical server, a server cluster or a distributed system formed by a plurality of physical servers, or may be a cloud server that provides cloud services, cloud databases, cloud computing, cloud functions, cloud storage, network services, cloud communication, middleware services, domain name services, security services, CDN (Content Delivery Network ), and basic cloud computing services such as big data and an artificial intelligence platform. Cloud server 101 may also become a node of the blockchain.
In some embodiments, the cloud server 101 is a cloud server with powerful computing resources, and is characterized by high virtualization and high distribution.
In some embodiments, the present application may be applied to the field of image encoding and decoding, the field of video encoding and decoding, the field of hardware video encoding and decoding, the field of dedicated circuit video encoding and decoding, the field of real-time video encoding and decoding, and the like. For example, the schemes of the present application may be incorporated into audio video coding standards (audio video coding standard, AVS for short), such as the h.264/audio video coding (audio video coding, AVC for short) standard, the h.265/high efficiency video coding (high efficiency video coding, HEVC for short) standard, and the h.266/multi-function video coding (versatile video coding, VVC for short) standard. Alternatively, the schemes of the present application may operate in conjunction with other proprietary or industry standards including ITU-T H.261, ISO/IECMPEG-1Visual, ITU-T H.262 or ISO/IECMPEG-2Visual, ITU-T H.263, ISO/IECMPEG-4Visual, ITU-T H.264 (also known as ISO/IECMPEG-4 AVC), including Scalable Video Coding (SVC) and Multiview Video Coding (MVC) extensions. It should be understood that the techniques of this application are not limited to any particular codec standard or technique.
The following describes a video coding framework according to an embodiment of the present application.
Fig. 2 is a schematic block diagram of a video encoder provided by an embodiment of the present application. It should be appreciated that the video encoder 200 may be used for lossy compression of images (lossy compression) and may also be used for lossless compression of images (lossless compression). The lossless compression may be visual lossless compression (visually lossless compression) or mathematical lossless compression (mathematically lossless compression).
The video encoder 200 may be applied to image data in luminance and chrominance (YCbCr, YUV) format.
For example, the video encoder 200 reads video data, divides a frame of image into a number of Coding Tree Units (CTUs) for each frame of image in the video data, and in some examples, CTBs may be referred to as "treeblocks", "maximum coding units" (Largest Coding unit, LCUs) or "coding treeblocks" (coding tree block, CTBs). Each CTU may be associated with a block of pixels of equal size within the image. Each pixel may correspond to one luminance (or luma) sample and two chrominance (or chroma) samples. Thus, each CTU may be associated with one block of luma samples and two blocks of chroma samples. One CTU size is, for example, 128×128, 64×64, 32×32, etc. One CTU may be further divided into several Coding Units (CUs), where a CU may be a rectangular block or a square block. The CU may be further divided into a Prediction Unit (PU) and a Transform Unit (TU), so that the encoding, the prediction, and the transform are separated, and the processing is more flexible. In one example, CTUs are divided into CUs in a quadtree manner, and CUs are divided into TUs, PUs in a quadtree manner.
Video encoders and video decoders may support various PU sizes. Assuming that the size of a particular CU is 2n×2n, video encoders and video decoders may support 2n×2n or n×n PU sizes for intra prediction, and support 2n×2n, 2n× N, N ×2N, N ×n or similar sized symmetric PUs for inter prediction. The video encoder and video decoder may also support 2 nxnu, 2 nxnd, nL x 2N, and nR x 2N asymmetric PUs for inter prediction.
In some embodiments, as shown in fig. 2, the video encoder 200 may include: a prediction unit 210, a residual unit 220, a transform/quantization unit 230, an inverse transform/quantization unit 240, a reconstruction unit 250, a loop filtering unit 260, a decoded image buffer 270, and an entropy encoding unit 280. It should be noted that video encoder 200 may include more, fewer, or different functional components.
Alternatively, in this application, a current block (current block) may be referred to as a current Coding Unit (CU) or a current Prediction Unit (PU), or the like. The prediction block may also be referred to as a prediction image block or an image prediction block, and the reconstructed image block may also be referred to as a reconstruction block or an image reconstruction image block.
In some embodiments, prediction unit 210 includes an inter prediction unit 211 and an intra estimation unit 212. Because of the strong correlation between adjacent pixels in a frame of video, intra-prediction methods are used in video coding techniques to eliminate spatial redundancy between adjacent pixels. Because of the strong similarity between adjacent frames in video, the inter-frame prediction method is used in the video coding and decoding technology to eliminate the time redundancy between adjacent frames, thereby improving the coding efficiency.
The inter prediction unit 211 may be used for inter prediction, which may refer to image information of different frames, using motion information to find a reference block from the reference frame, generating a prediction block from the reference block, for eliminating temporal redundancy; the frames used for inter-prediction may be P frames, which refer to forward predicted frames, and/or B frames, which refer to bi-directional predicted frames. The motion information includes a reference frame list in which the reference frame is located, a reference frame index, and a motion vector. The motion vector may be integer or sub-pixel, and if the motion vector is sub-pixel, interpolation filtering is required to make the required sub-pixel block in the re-reference frame, where the integer or sub-pixel block in the reference frame found from the motion vector is referred to as the reference block. Some techniques may use the reference block directly as a prediction block, and some techniques may reprocess the reference block to generate a prediction block. Reprocessing a prediction block on the basis of a reference block is also understood to mean that the reference block is taken as a prediction block and then a new prediction block is processed on the basis of the prediction block.
The most commonly used inter prediction methods at present include: geometric partitioning modes (geometric partitioning mode, GPM) in VVC video codec standard, and angle weighted prediction (angular weighted prediction, AWP) in AVS3 video codec standard. These two intra prediction modes are in principle common.
The intra estimation unit 212 predicts pixel information within the current code image block for eliminating spatial redundancy by referring to only information of the same frame image. The frame used for intra prediction may be an I-frame.
The intra prediction modes used by HEVC are Planar mode (Planar), DC, and 33 angular modes, for a total of 35 prediction modes. The intra modes used by VVC are Planar, DC and 65 angular modes, for a total of 67 prediction modes. The intra modes used by AVS3 are DC, plane, bilinear and 63 angular modes, for a total of 66 prediction modes.
In some embodiments, intra-estimation unit 212 may be implemented using intra-block copy techniques and intra-string copy techniques.
Residual unit 220 may generate a residual block of the CU based on the pixel block of the CU and the prediction block of the PU of the CU. For example, residual unit 220 may generate a residual block of the CU such that each sample in the residual block has a value equal to the difference between: samples in pixel blocks of a CU, and corresponding samples in prediction blocks of PUs of the CU.
The transform/quantization unit 230 may quantize the transform coefficients. Transform/quantization unit 230 may quantize transform coefficients associated with TUs of a CU based on Quantization Parameter (QP) values associated with the CU. The video encoder 200 may adjust the degree of quantization applied to the transform coefficients associated with the CU by adjusting the QP value associated with the CU.
The inverse transform/quantization unit 240 may apply inverse quantization and inverse transform, respectively, to the quantized transform coefficients to reconstruct a residual block from the quantized transform coefficients.
The reconstruction unit 250 may add samples of the reconstructed residual block to corresponding samples of one or more prediction blocks generated by the prediction unit 210 to generate a reconstructed image block associated with the TU. In this way, reconstructing sample blocks for each TU of the CU, video encoder 200 may reconstruct pixel blocks of the CU.
Loop filtering unit 260 may perform a deblocking filtering operation to reduce blocking artifacts of pixel blocks associated with the CU.
In some embodiments, the loop filtering unit 260 includes a deblocking filtering unit for deblocking artifacts and a sample adaptive compensation/adaptive loop filtering (SAO/ALF) unit for removing ringing effects.
The decoded image buffer 270 may store reconstructed pixel blocks. Inter prediction unit 211 may use the reference image containing the reconstructed pixel block to perform inter prediction on PUs of other images. In addition, intra estimation unit 212 may use the reconstructed pixel blocks in decoded image buffer 270 to perform intra prediction on other PUs in the same image as the CU.
The entropy encoding unit 280 may receive the quantized transform coefficients from the transform/quantization unit 230. Entropy encoding unit 280 may perform one or more entropy encoding operations on the quantized transform coefficients to generate entropy encoded data.
Fig. 3 is a schematic block diagram of a video decoder provided by an embodiment of the present application.
As shown in fig. 3, the video decoder 300 includes: an entropy decoding unit 310, a prediction unit 320, an inverse quantization/transformation unit 330, a reconstruction unit 340, a loop filtering unit 350, and a decoded image buffer 360. It should be noted that the video decoder 300 may include more, fewer, or different functional components.
The video decoder 300 may receive the bitstream. The entropy decoding unit 310 may parse the bitstream to extract syntax elements from the bitstream. As part of parsing the bitstream, the entropy decoding unit 310 may parse entropy-encoded syntax elements in the bitstream. The prediction unit 320, the inverse quantization/transformation unit 330, the reconstruction unit 340, and the loop filtering unit 350 may decode video data according to syntax elements extracted from a bitstream, i.e., generate decoded video data.
In some embodiments, prediction unit 320 includes an intra prediction unit 322 and an inter prediction unit 321.
Intra prediction unit 322 may perform intra prediction to generate a prediction block for the PU. Intra-prediction unit 322 may use an intra-prediction mode to generate a prediction block for the PU based on pixel blocks of spatially-neighboring PUs. Intra-prediction unit 322 may also determine an intra-prediction mode for the PU based on one or more syntax elements parsed from the bitstream.
The inter prediction unit 321 may construct a first reference picture list (list 0) and a second reference picture list (list 1) according to syntax elements parsed from the bitstream. Furthermore, if the PU uses inter prediction encoding, entropy decoding unit 310 may parse the motion information of the PU. Inter prediction unit 321 may determine one or more reference blocks of the PU from the motion information of the PU. Inter prediction unit 321 may generate a prediction block of a PU from one or more reference blocks of the PU.
The inverse quantization/transform unit 330 may inverse quantize (i.e., dequantize) transform coefficients associated with the TUs. Inverse quantization/transform unit 330 may determine the degree of quantization using QP values associated with the CU of the TU.
After inverse quantizing the transform coefficients, inverse quantization/transform unit 330 may apply one or more inverse transforms to the inverse quantized transform coefficients in order to generate a residual block associated with the TU.
Reconstruction unit 340 uses the residual blocks associated with the TUs of the CU and the prediction blocks of the PUs of the CU to reconstruct the pixel blocks of the CU. For example, the reconstruction unit 340 may add samples of the residual block to corresponding samples of the prediction block to reconstruct a pixel block of the CU, resulting in a reconstructed image block.
Loop filtering unit 350 may perform a deblocking filtering operation to reduce blocking artifacts of pixel blocks associated with the CU.
The video decoder 300 may store the reconstructed image of the CU in a decoded image buffer 360. The video decoder 300 may use the reconstructed image in the decoded image buffer 360 as a reference image for subsequent prediction or may transmit the reconstructed image to a display device for presentation.
The basic flow of video encoding and decoding is as follows: at the encoding end, a frame of image is divided into blocks, and for a current block, the prediction unit 210 generates a prediction block of the current block using intra prediction or inter prediction. The residual unit 220 may calculate a residual block, which may also be referred to as residual information, based on the difference between the prediction block and the original block of the current block, i.e., the prediction block and the original block of the current block. The residual block may remove information insensitive to human eyes through processes such as transformation and quantization by the transformation/quantization unit 230 to remove visual redundancy. Alternatively, the residual block before being transformed and quantized by the transforming/quantizing unit 230 may be referred to as a time domain residual block, and the time domain residual block after being transformed and quantized by the transforming/quantizing unit 230 may be referred to as a frequency residual block or a frequency domain residual block. The entropy encoding unit 280 receives the quantized change coefficient output from the change quantization unit 230, and may entropy encode the quantized change coefficient to output a bitstream. For example, the entropy encoding unit 280 may eliminate character redundancy according to the target context model and probability information of the binary code stream.
At the decoding end, the entropy decoding unit 310 may parse the code stream to obtain prediction information of the current block, a quantization coefficient matrix, etc., and the prediction unit 320 generates a prediction block of the current block using intra prediction or inter prediction on the current block based on the prediction information. The inverse quantization/transformation unit 330 performs inverse quantization and inverse transformation on the quantized coefficient matrix obtained from the code stream to obtain a residual block. The reconstruction unit 340 adds the prediction block and the residual block to obtain a reconstructed block. The reconstructed blocks constitute a reconstructed image, and the loop filtering unit 350 performs loop filtering on the reconstructed image based on the image or based on the blocks, resulting in a decoded image. The encoding side also needs to obtain a decoded image in a similar operation to the decoding side. The decoded image may also be referred to as a reconstructed image, which may be a subsequent frame as a reference frame for inter prediction.
The block division information determined by the encoding end, and mode information or parameter information such as prediction, transformation, quantization, entropy coding, loop filtering, etc. are carried in the code stream when necessary. The decoding end analyzes the code stream and analyzes and determines the same block division information as the encoding end according to the existing information, and predicts, transforms, quantizes, entropy codes, loop filters and other mode information or parameter information, so that the decoded image obtained by the encoding end is ensured to be the same as the decoded image obtained by the decoding end.
The foregoing is a basic flow of a video codec under a block-based hybrid coding framework, and as technology advances, some modules or steps of the framework or flow may be optimized.
In some embodiments, the methods of embodiments of the present application may apply Yu Duan cloud co-coding. The end cloud collaborative coding refers to a scheme of compressing video by collaborative coding of a cloud end and a terminal. Because the computing power of the video content producer (cloud) and the computing power of the video content consumer (terminal) are different, a relatively complex video compression task can be completed cooperatively by two ends, so that the cloud resources and the powerful computing power (such as encoding power) can be utilized, the data volume of network transmission is reduced, and the computing power (such as decoding power) of the terminal is also effectively utilized. The method can be used for scenes such as cloud games.
The cloud server and the terminal equipment perform data interaction through an end cloud cooperative protocol, wherein the end cloud cooperative protocol refers to a unified protocol for performing data interaction between the cloud server and the terminal equipment.
The terminal equipment cooperative interface is an intelligent terminal software and hardware module interface, and can effectively interact with the intelligent terminal through the interface, configure video coding and rendering parameters and acquire real-time operation performance of hardware.
Decoding performance refers to the highest decoding frame rate and single frame decoding delay supported for a given video size under a particular decoding protocol. The video size is defined as follows: 360p,576p,720p,1080p,2k,4k. The video frame rate is defined as follows: 30fps,40fps,50fps,60fps,90fps,120fps.
The definition of the video resolution and video frame rate of the terminal device is shown in tables 1 and 2.
Table 1 definition of video resolution of terminal device
Table 2 definition of video resolution and video frame rate for terminal equipment
| Video frame rate | Enumeration definition |
| 30fps | 0x1 |
| 40fps | 0x2 |
| 50fps | 0x4 |
| 60fps | 0x8 |
| 90fps | 0x10 |
| 120fps | 0x20 |
Optionally, the decoding performance supported by the terminal device is given in the form of triples, the first element is an enumeration definition of video resolution, the second element is an enumeration definition of video frame rate, the third element is a single frame decoding delay at video resolution and video frame rate, for example, H264 decoding of device a, and the single frame decoding delay at 720p@60fps is 10ms, which is denoted as (4, 8, 10).
In some embodiments, the terminal device decoding capability data structure requirements are shown in table 3.
Table 3 decoding capability data structure requirements for terminal devices
And the cloud server determines the optimal encoding and decoding configuration such as the decoding protocol, the decoding resolution, the video frame rate and the like of the current equipment, and encoding and decoding strategies such as the number of video encoding reference frames, SVC starting and the like according to the decoding capability of the intelligent terminal and by combining the game type and the user network condition.
In some embodiments, the methods of embodiments of the present application may be applied to video rendering collaboration. And reasonably dividing video rendering tasks according to the graphic processing capacity of the terminal equipment, so that the cloud server and the terminal equipment can effectively cooperate to improve video image quality. In particular, video rendering collaboration includes rendering region collaboration, rendering task collaboration, video analysis and processing collaboration, and the like.
The terminal device rendering capabilities include three classes: (1) not having video image processing capability, (2) having partial video image processing capability, (3) having full video image processing capability. The specific enumeration definitions are shown in table 4.
Table 4 terminal rendering capability definition
The type of video processing algorithm of the terminal device when the terminal device has a partial video image processing capability or a full video image processing capability is shown in table 5.
Table 5 video processing algorithm type definition
| Video processing algorithm type | Enumeration definition |
| Undefined | 0 |
| Video sharpening enhancement | 1 |
| Video HDR enhancement | 2 |
| Video image mode | 3 |
| Video complexity analysis | 1001 |
| Video foreground and background analysis | 1002 |
In some embodiments, the application may be applied to any image quality enhancement technical field, for example, in the field of intelligent task analysis, when task analysis is performed based on video with enhanced image quality, accuracy of task analysis may be improved.
In some embodiments, embodiments of the present invention may be applied to a variety of scenarios, including, but not limited to, cloud technology (e.g., cloud gaming), artificial intelligence, intelligent transportation, assisted driving, and the like.
If the embodiment of the application is applied to the cloud game, the cloud server can determine the rendering function set to be started according to the game type, and then determine the optimal rendering cooperative mode of the current device through the device type and the rendering capability reported by the terminal device. The specific rendering collaborative strategy comprises the following steps: rendering region coordination, rendering task coordination, video analysis coordination. Embodiments of the present application relate generally to rendering task collaboration.
Rendering task collaboration is specific video enhancement task oriented, such tasks may be divided into different independent subtasks, each subtask corresponding to a different video image enhancement algorithm. The video enhancement task a is formed by 3 independent subtasks in cascade as shown in fig. 4A.
And the coordination of rendering tasks can enable one part of video image enhancement tasks to be completed in the cloud server and the other part of video image enhancement tasks to be completed in the intelligent terminal according to the computing capacity of the terminal. The video enhancement task completed by the cloud server is completed before video encoding (video preprocessing), and the video enhancement task completed by the intelligent terminal is completed after video decoding (video post-processing).
(1) For intelligent terminal equipment without video image processing capability, video image enhancement is completed in a cloud server, and the flow is shown in fig. 4B.
(2) For the intelligent terminal equipment with partial video image processing capability, partial tasks can be completed in the cloud server by video image enhancement, and partial tasks are completed in the intelligent terminal, and the flow is shown in fig. 4C.
(3) For intelligent terminal equipment with complete video image processing capability, video image enhancement can be completely finished in the intelligent terminal, and the flow is shown in fig. 4D.
The video rendering co-optimization data structure requirements are shown in table 6.
Table 6 terminal device rendering capability data structure requirements
/>
/>
And the cloud server determines the optimal rendering collaborative mode of the current equipment according to the rendering capability of the intelligent terminal and in combination with the game type, and returns to the completed rendering algorithm, rendering area and image recognition task required by the intelligent terminal.
The following describes the technical solutions of the embodiments of the present application in detail through some embodiments. The following embodiments may be combined with each other, and some embodiments may not be repeated for the same or similar concepts or processes.
First, an encoding end will be described as an example.
Fig. 5 is a flowchart of an image quality enhancement method according to an embodiment of the present application, where the method may be applied to the cloud server 101 shown in fig. 1, or to the encoder shown in fig. 2, or to the cloud servers shown in fig. 4B to 4D.
The cloud server of the embodiment of the application can be understood as an encoder, and the terminal device can be understood as a decoder.
As shown in fig. 5, an embodiment of the present application includes the following steps:
s401, determining image quality enhancement information of the corresponding terminal equipment.
In order to reduce the workload of the cloud server, when the terminal equipment is detected to have enough image quality enhancement capability, the terminal equipment is used for executing corresponding image quality enhancement operation. Specifically, before processing a target video to be encoded, the cloud server first determines image quality enhancement information of a corresponding terminal device, and determines image quality enhancement capability of the terminal device according to the image quality enhancement information of the terminal device. If the terminal equipment has enough image quality enhancement capability, the image quality enhancement operation is determined to be realized by the terminal equipment, and at the moment, the cloud server does not perform the image quality enhancement operation on the target video, but directly encodes the target video to obtain a code stream. In this case, after the terminal device obtains the code stream, the terminal device decodes the code stream to obtain the target video, and performs the image quality enhancement operation on the target video to obtain the target video with enhanced image quality.
If the image quality enhancement capability of the terminal equipment is insufficient, the image quality operation is determined to be realized by the cloud server, and at the moment, the cloud server firstly enhances the image quality of the target video and encodes the target video with enhanced image quality before encoding the target video, so as to obtain a second code stream. In this case, after the terminal device obtains the second code stream, the terminal device decodes the second code stream to obtain the target video, where the decoded target video is a video with enhanced image quality of the cloud server, and at this time, the terminal device does not perform an image quality enhancement operation on the target video, but determines the decoded target video as the target video with enhanced image quality.
The image quality enhancement in the embodiment of the application includes operations such as sharpening, and if the cloud server implements image quality enhancement, code rate reassignment may also be included.
The image quality enhancement information of the terminal device in the embodiment of the present application includes whether the terminal device has an image quality enhancement function, and an image quality enhancement computing capability corresponding to the terminal device when the terminal device has the image quality enhancement function.
In some embodiments, the image quality enhancement computing capability of the embodiments of the present application includes at least one of a time consumed when the terminal device performs the image quality enhancement operation on the video, an enhancement effect obtained after the terminal device performs the image quality enhancement operation on the video, and the like.
The method for the cloud server to acquire the image quality enhancement information of the terminal equipment is not limited.
In one possible implementation, the cloud server transmits first request information to the terminal device, before encoding the target video, the first request information being used to request image quality enhancement information of the terminal device. And the terminal equipment sends the image quality enhancement information to the cloud server when receiving the first request information sent by the cloud server.
In another possible implementation manner, before encoding the target video, the cloud server first sends a section of test video to the terminal device to obtain image quality enhancement information of the terminal device. That is, the step S401 includes the steps of:
S401-A1, acquiring a first test code stream, and sending a first request to the terminal equipment, wherein the first request is used for requesting the image quality enhancement capability of the terminal equipment, and the first request comprises the first test code stream;
S401-A2, receiving first information sent by the terminal equipment, wherein the first information is used for indicating whether the terminal equipment has an image quality enhancement function or not and indicating corresponding image quality enhancement computing capacity when the terminal equipment has the image quality enhancement function;
S401-A3, obtaining the image quality enhancement information of the terminal equipment according to the first information.
In the embodiment of the application, before encoding the target video, the cloud server first acquires a first test code stream and sends the first test code stream to the terminal device. And after receiving the first test code stream, the terminal equipment decodes the first test code stream to obtain a first test video. If the terminal device has the image quality enhancement function, the terminal device performs image quality enhancement on the first test video, and at the same time, the terminal device determines that the terminal device performs image quality enhancement on the first test video, and the terminal device corresponds to the image quality enhancement computing capability, for example, statistics of time consumed by the terminal device to perform image quality enhancement on the first test video and/or determination of enhancement effect of an image obtained after the terminal device performs image quality enhancement on the first test video, that is, the enhanced image is compared with an image before enhancement, so as to obtain enhancement effect.
Then, the terminal device sends first information to the cloud server, wherein the first information is used for indicating whether the terminal device has the image quality enhancement function or not and indicating the corresponding image quality enhancement computing capability when the terminal device has the image quality enhancement function.
In some embodiments, the first information includes a first flag for indicating whether the terminal device is provided with an image quality enhancement function. That is, the terminal device may indicate whether the terminal device has the image quality enhancing function by setting a different value to the first flag. For example, when the terminal device does not have the image quality enhancement function, the value of the first flag is set to the first value, and when the terminal device has the image quality enhancement function, the value of the first flag is set to the second value.
In this embodiment of the present application, if the terminal device has an image quality enhancement function, the first information includes, in addition to the first flag, an image quality enhancement computing capability corresponding to the terminal device.
The specific values of the first value and the second value are not limited in the embodiment of the application. For example, the first value is 0 and the second value is 1.
After obtaining the first information sent by the terminal device, the cloud server analyzes the first information to obtain image quality enhancement information of the terminal device, for example, analyzes the first information to obtain a first mark, and if the value of the first mark is a first value, indicates that the terminal device does not have an image quality enhancement function. And if the value of the first mark is the second value, indicating that the terminal equipment has the image quality enhancement function, and at the moment, continuously analyzing the first information by the cloud server to obtain the image quality enhancement computing capability of the terminal equipment.
In some embodiments, the image quality enhancement computing capability of the terminal device includes a first time consumed by the terminal device in image quality enhancement of the first test video.
In some embodiments, the first time includes: and the terminal equipment is used for carrying out image quality enhancement on the whole image of each image in the first test video.
In some embodiments, the first time includes: the time consumed by the terminal device when performing image quality enhancement on a part of the image area of each image in the first test video, for example, the time consumed by the terminal device when performing image quality enhancement on the region of interest in the first test video.
In some embodiments, the first time comprises: and Shan Zhenping average duration, single frame maximum duration and variance of the first test video when the terminal equipment performs image quality enhancement. The variance refers to the variance of the time period spent by the terminal equipment for respectively enhancing the image quality of all the images in the first test video.
In some embodiments, the step S401 includes the steps of S401-B:
S401-B, if the target video belongs to the preset type video, obtaining the image quality enhancement information of the terminal equipment.
That is, the method of the embodiment of the present application is not performed for all scenes, but is performed for a preset specific application scene. The cloud server judges the type of the target video after obtaining the target video, and if the target video belongs to the preset type of video, the cloud server obtains the image quality enhancement information of the terminal equipment and executes the method of the application embodiment.
The embodiment of the application does not limit the preset type video, and can be cloud game type video, internet of vehicles type video or the like.
The cloud server obtains the image quality enhancement information of the terminal device according to the above manner, and then executes the following steps S402 and S403.
S402, if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is met, skipping image quality enhancement operation, and encoding a target video which is not subjected to image quality enhancement to obtain a first code stream;
and S403, if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is not met, enhancing the image quality of the target video, and encoding the target video with enhanced image quality to obtain a second code stream.
And the cloud server obtains the image quality enhancement information of the terminal equipment according to the steps, judges whether the image quality enhancement information of the terminal equipment meets the preset image quality enhancement condition, and if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is met, the image quality enhancement capability of the terminal equipment is enough. At this time, the cloud server does not perform image quality enhancement on the target video, that is, the cloud server skips the image quality enhancement operation and encodes the target video not subjected to image quality enhancement, thereby obtaining a first code stream.
If the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is not met, the image quality enhancement capability of the terminal equipment is insufficient. At this time, the cloud server performs image quality enhancement on the target video, and encodes the target video with the enhanced image quality to obtain a second code stream.
As can be seen from the above, the image quality enhancement information of the terminal device according to the embodiment of the present application includes whether the terminal device has the image quality enhancement function, and the image quality enhancement computing capability corresponding to the terminal device when the terminal device has the image quality enhancement function. Correspondingly, the preset image quality enhancement condition includes a preset image quality enhancement computing capability, and at this time, the cloud server judges whether the image quality enhancement information of the terminal device meets the preset image quality enhancement condition or not includes the following steps:
step 11, if the terminal equipment has an image quality enhancement function and the image quality enhancement computing capability corresponding to the terminal equipment meets a preset image quality enhancement computing capability, determining that the image quality enhancement information of the terminal equipment meets an image quality enhancement condition;
and step 12, if the terminal equipment does not have the image quality enhancement function, or the terminal equipment has the image quality enhancement function, and the image quality enhancement computing capability of the terminal equipment does not meet the preset image quality enhancement computing capability, determining that the image quality enhancement information of the terminal equipment does not meet the image quality enhancement condition.
Specifically, the cloud server compares the image quality enhancement information of the terminal device with a preset image quality enhancement condition, and first determines whether the terminal device has the image quality enhancement capability. If the terminal device does not have the image quality enhancement capability, it is determined that the image quality enhancement information of the terminal device does not meet the image quality enhancement condition, and at this time, the cloud server performs the step S403, performs image quality enhancement on the target video, and encodes the target video after image quality enhancement to obtain the second code stream.
If the terminal equipment has the image quality enhancement capability, judging whether the image quality enhancement computing capability corresponding to the terminal equipment meets the preset image quality enhancement computing capability. If the image quality enhancement computing capability corresponding to the terminal device meets the preset image quality enhancement computing capability, determining that the image quality enhancement information of the terminal device meets the image quality enhancement condition, at this time, the cloud server executes the step S402, skips the image quality enhancement operation, and encodes the target video which is not subjected to image quality enhancement, thereby obtaining a first code stream. If the image quality enhancement computing capability corresponding to the terminal device does not meet the preset image quality enhancement computing capability, determining that the image quality enhancement information of the terminal device does not meet the image quality enhancement condition, at this time, executing the step S403 by the cloud server, enhancing the image quality of the target video, and encoding the target video after image quality enhancement to obtain a second code stream.
In some embodiments, if the image quality enhancement computing capability of the terminal device includes a first time consumed by the terminal device to enhance the image quality of the first test video, the predetermined image quality enhancement computing capability includes a predetermined image quality enhancement consumed time. At this time, the step 11 includes the following step 11-a, and if the terminal device has the image quality enhancement function and the image quality enhancement computing capability of the terminal device does not satisfy the preset image quality enhancement computing capability in the step 12, the step 12-a of determining that the image quality enhancement information of the terminal device does not satisfy the image quality enhancement bar includes:
step 11-a, if the terminal device has an image quality enhancement function and the first time is less than or equal to a preset image quality enhancement time, determining that the image quality enhancement information of the terminal device meets a preset image quality enhancement condition.
And step 12-A, if the terminal equipment has the image quality enhancement function and the first time is longer than the preset image quality enhancement time, determining that the image quality enhancement information of the terminal equipment does not meet the preset image quality enhancement condition.
The limitation corresponding to the preset image quality enhancement time is different for different application scenes, for example, scenes with strict requirements for low delay such as cloud games, and the requirement for computing power is higher (for example, the upper limit is 1 ms).
The application scenarios of the image quality enhancement method in the embodiment of the present application include, but are not limited to, the following two scenarios:
scene 1, in the embodiment of the present application, the image quality enhancement operation is performed on the entire image of the target video. That is, the image quality enhancement processing is performed on the entire image of each frame image in the target video.
Scene 2, in the embodiment of the present application, image quality enhancement operation is performed on a region of interest of a target video. That is, the image quality enhancement processing is performed on a partial image area of each frame image in the target video.
Region of interest (ROI): is an image region selected from the images, which is the focus of interest in image analysis. In video coding, a region of interest may also be considered as a region of great interest to the human eye.
In scenario 2, if the image quality enhancement operation needs to be performed on the region of interest of the target video, the cloud server needs to obtain not only the image quality enhancement information of the terminal device but also the region of interest detection information of the terminal device, where the step S402 includes the following steps:
S402-A1, if the image quality enhancement information of the terminal equipment meets the preset image quality enhancement condition, acquiring the detection information of the region of interest of the terminal equipment.
In this embodiment of the present invention, if the image quality enhancement information of the terminal device meets a preset image quality enhancement condition, whether the terminal device has sufficient region of interest detection capability is determined, specifically, the region of interest detection information of the terminal device is obtained, and whether the terminal device has sufficient region of interest detection capability is determined according to the region of interest detection information of the terminal device. If the terminal equipment has enough region-of-interest detection capability, determining that the terminal equipment executes the region-of-interest detection operation, namely, the terminal equipment detects the region-of-interest of the decoded target video to obtain the region-of-interest of the target video, and enhancing the image quality of the region-of-interest to obtain the target video with enhanced image quality.
In some embodiments, if the terminal device does not have sufficient region of interest detection capability, it is determined that the region of interest detection operation is performed by the cloud server.
In this embodiment of the present application, the information for detecting the region of interest of the terminal device includes whether the terminal device has the function for detecting the region of interest, and the region of interest detection capability corresponding to the terminal device when the terminal device has the function for detecting the region of interest.
The method for acquiring the detection information of the region of interest of the terminal equipment by the cloud server is not limited.
In one possible implementation manner, before encoding the target video, or when the cloud server determines that the image quality enhancement information of the terminal device meets a preset image quality enhancement condition, the cloud server sends second request information to the terminal device, where the second request information is used to request the region of interest detection information of the terminal device. And the terminal equipment sends the detection information of the interested region to the cloud server when receiving the second request information sent by the cloud server.
In another possible implementation manner, the acquiring the region of interest detection information of the terminal device in S402-A1 includes the following steps:
S402-A11, acquiring a second test code stream, and sending a second request to terminal equipment, wherein the second request is used for requesting the detection capability of the region of interest of the terminal equipment, and the second request comprises the second test code stream;
S402-A12, receiving second information sent by the terminal equipment, wherein the second information is used for indicating whether the terminal equipment has an interesting area detection function or not and indicating the corresponding interesting area detection computing capacity of the terminal equipment when the terminal equipment has the interesting area detection function;
S402-A13, obtaining the detection information of the region of interest of the terminal equipment according to the second information.
In the embodiment of the application, before encoding the target video, the cloud server first acquires a second test code stream and sends the second test code stream to the terminal device. And after receiving the second test code stream, the terminal equipment decodes the second test code stream to obtain a second test video. If the terminal equipment has the function of detecting the region of interest, the terminal equipment detects the region of interest of the second test video, and meanwhile, the terminal equipment determines the region of interest detection computing capability corresponding to the terminal equipment when the terminal equipment detects the region of interest of the second test video, for example, the time consumed by the terminal equipment to detect the region of interest of the second test video is counted, and/or the accuracy of the terminal equipment when the terminal equipment detects the region of interest of the second test video is determined, namely, the detected region of interest is compared with the true value of the region of interest, so that the detection accuracy of the region of interest is obtained.
The method for detecting the region of interest according to the embodiments of the present application is not limited, and for example, the detection of the region of interest includes a method for calculating energy (such as variance) of each region of the image to distinguish a high-frequency region (insensitive region) from a low-frequency region (region of interest).
And then, the terminal equipment sends second information to the cloud server, wherein the second information is used for indicating whether the terminal equipment has the region of interest detection function or not and indicating the corresponding region of interest detection computing capability of the terminal equipment when the terminal equipment has the region of interest detection function.
In some embodiments, the second information comprises a second flag for indicating whether the terminal device is provided with a region of interest detection function. That is, the terminal device may indicate whether the terminal device has the region of interest detection function by setting a different value to the second flag. For example, if the terminal device does not have the region of interest detection function, the value of the second flag is set to the first value, and if the terminal device does have the region of interest detection function, the value of the second flag is set to the second value.
In this embodiment of the present application, if the terminal device has the function of detecting the region of interest, the second information includes, in addition to the second flag, a region of interest detection computing capability corresponding to the terminal device.
The specific values of the first value and the second value are not limited in the embodiment of the application. For example, the first value is 0 and the second value is 1.
After obtaining the second information sent by the terminal equipment, the cloud server analyzes the second information to obtain the detection information of the region of interest of the terminal equipment, for example, analyzes the second information to obtain a second mark, and if the value of the second mark is the first value, indicates that the terminal equipment does not have the detection function of the region of interest. If the value of the second mark is the second value, the terminal equipment is indicated to have the function of detecting the region of interest, and at the moment, the cloud server continues to analyze the second information to obtain the detection computing capacity of the region of interest of the terminal equipment.
In some embodiments, the region of interest detection computing capability of the terminal device includes a second time consumed by the terminal device in performing the region of interest detection on the second test video.
In some embodiments, the second time comprises: and the terminal equipment performs at least one of single-frame average duration, single-frame maximum duration and variance when the second test video is subjected to region-of-interest detection. The variance refers to the variance of the time period spent by the terminal equipment in detecting the region of interest of all the images in the second test video respectively.
According to the above steps, after obtaining the region of interest detection information of the terminal device, the following step S402-A2 is performed.
S402-A2, if the region of interest detection information of the terminal equipment meets the preset region of interest detection condition, skipping image quality enhancement operation and region of interest detection operation, and encoding a target video which is not subjected to image quality enhancement and region of interest detection to obtain a first code stream.
And the cloud server obtains the region-of-interest detection information of the terminal equipment according to the steps, judges whether the region-of-interest detection information of the terminal equipment meets the preset region-of-interest detection condition, and if the region-of-interest detection information of the terminal equipment meets the preset region-of-interest detection condition, the region-of-interest detection capability of the terminal equipment is enough. At this time, the cloud server does not perform region-of-interest detection on the target video, i.e., the cloud server skips the region-of-interest detection operation and encodes the target video not subjected to region-of-interest detection and image quality enhancement, thereby obtaining a first code stream.
If the detection information of the region of interest of the terminal equipment does not meet the preset detection condition of the region of interest, the detection capability of the region of interest of the terminal equipment is insufficient. At this time, the cloud server detects the region of interest of the target video.
In some embodiments, if the region of interest detection condition includes a preset region of interest detection computing capability, the cloud server determining whether the region of interest detection information of the terminal device meets the preset region of interest detection condition includes the following steps:
step 21, if the terminal equipment has the function of detecting the region of interest and the region of interest detection computing capacity corresponding to the terminal equipment meets the preset region of interest detection computing capacity, determining that the region of interest detection information of the terminal equipment meets the region of interest detection condition;
step 22, if the terminal device does not have the function of detecting the region of interest, or if the terminal device has the function of detecting the region of interest and the calculation capability of detecting the region of interest of the terminal device does not meet the preset calculation capability of detecting the region of interest, determining that the detection information of the region of interest of the terminal device does not meet the detection condition of the region of interest.
Specifically, the cloud server compares the region-of-interest detection information of the terminal device with preset region-of-interest detection conditions, and first determines whether the terminal device has the region-of-interest detection capability. If the terminal equipment does not have the region-of-interest detection capability, determining that the region-of-interest detection information of the terminal equipment does not meet the region-of-interest detection condition, and at the moment, realizing the region-of-interest detection operation by the cloud server.
If the terminal equipment has the region-of-interest detection capability, judging whether the region-of-interest detection computing capability corresponding to the terminal equipment meets the preset region-of-interest detection computing capability. If the region of interest detection computing capability corresponding to the terminal equipment meets the preset region of interest detection computing capability, determining that the region of interest detection information of the terminal equipment meets the preset region of interest detection condition, at this time, the cloud server skips the region of interest detection operation and the image quality enhancement operation, and encodes a target video which is not subjected to the region of interest detection and the image quality enhancement, so as to obtain a first code stream. If the detection computing capacity of the region of interest corresponding to the terminal equipment does not meet the preset detection computing capacity of the region of interest, determining that the detection information of the region of interest of the terminal equipment does not meet the detection condition of the region of interest, and at the moment, executing the detection operation of the region of interest by the cloud server.
In some embodiments, if the region of interest detection computing capability of the terminal device includes a second time consumed by the terminal device in detecting the region of interest of the second test video, the preset region of interest detection computing capability includes a preset region of interest detection consumed time. At this time, the above step 21 includes the following step 21-A, and the step 22 includes the following step 22-A:
Step 21-A, if the terminal equipment has the function of detecting the region of interest and the second time is less than or equal to the preset time for detecting the region of interest, determining that the information for detecting the region of interest of the terminal equipment meets the condition for detecting the region of interest.
Step 22-a, if the terminal device has the function of detecting the region of interest and the second time is longer than the preset time for detecting the region of interest, determining that the information for detecting the region of interest of the terminal device does not meet the condition for detecting the region of interest.
For different application scenarios, the preset limitation of the detection time of the region of interest is different, for example, a cloud game or other scenario with strict requirements on low delay, and the requirement on the computing power is higher (for example, the upper limit is 1 ms).
In some embodiments, the method of embodiments of the present application further comprises:
S402-A3, if the region of interest detection information of the terminal equipment does not meet the region of interest detection condition, detecting whether the terminal equipment has a function of receiving externally input image region information;
S402-A4, if the terminal equipment has the function of receiving externally input image area information, detecting an area of interest of the target video to obtain the area of interest information of the target video, sending the area of interest information to the terminal equipment, skipping image quality enhancement operation, and encoding the target video which is not subjected to image quality enhancement to obtain a first code stream;
And S402-A5, if the terminal equipment does not have the function of receiving the externally input image region information, detecting the region of interest of the target video to obtain the region of interest of the target video, enhancing the image quality of the region of interest, and encoding the target video with the enhanced image quality to obtain a second code stream.
The embodiment of the application does not limit the mode of detecting whether the terminal equipment has the function of receiving the externally input image area information or not by the cloud server. For example, the cloud server transmits a test area information to the terminal device, and if the transmission is successful, determines that the terminal device has a function of receiving externally input image area information, and if the transmission is failed, determines that the terminal device does not have a function of receiving externally input image area information.
If the terminal device has a function of receiving the image area information inputted from the outside, it is indicated that the terminal device can receive the area information of interest, and further, the image quality of the area of interest can be enhanced. Specifically, if the terminal device has a function of receiving externally input image region information, the cloud server performs region of interest detection on the target video to obtain region of interest information of the target video, and sends the region of interest information to the terminal device. Meanwhile, before encoding, the cloud server skips image quality enhancement operation and encodes the target video which is not subjected to image quality enhancement to obtain a first code stream. And after receiving the first code stream, the terminal equipment decodes the first code stream to obtain a target video. Optionally, the cloud server may program the region of interest information into the first code stream, so that the terminal device decodes the first code stream and may obtain the region of interest information. And then, the terminal equipment determines an interested region of the target video according to the interested region information, and performs image quality enhancement on the interested region to obtain the target video with enhanced image quality.
In some embodiments, if the terminal device does not have a function of receiving the image area information input from the outside, it indicates that the terminal device cannot receive the area information of interest, and cannot enhance the image quality of the area of interest. At this time, the method of the embodiment of the present application at least includes the following two possible implementations:
in one possible implementation manner, the cloud server detects an interesting region of the target video to obtain the interesting region of the target video, enhances the image quality of the interesting region, and encodes the target video with enhanced image quality to obtain the second code stream.
In another possible implementation manner, the cloud server skips the image quality enhancement operation and the region of interest detection operation, and encodes the target video without image quality enhancement and region of interest detection to obtain the first code stream. And the terminal equipment decodes the first code stream to obtain a target video, and enhances the image quality of the whole frame of the target video to obtain the target video with enhanced image quality.
In some embodiments, the first test code stream and the second test code stream may be the same test code stream, that is, the image quality enhancement computing capability and the region of interest detection computing capability of the detection terminal device may be combined into one detection module. For example, a small video code stream is transmitted to the terminal device in real time before the formal encoding is performed. The terminal equipment detects and outputs the region of interest of the video frame obtained after decoding, then enhances the image quality of the region of interest, and respectively counts the time consumption of the region of interest detection and the image quality enhancement. The terminal equipment sends the counted time consumption for detecting the region of interest and enhancing the image quality to the cloud server, and the cloud server judges whether the time consumption for detecting the region of interest and enhancing the image quality of the terminal equipment respectively accords with corresponding preset conditions. If the detection and calculation capabilities of the region of interest do not meet the corresponding preset conditions or only meet the preset conditions, the same strategy as that when the terminal equipment does not support the image quality enhancement capability is adopted, namely the detection and image quality enhancement of the region of interest are completed by the cloud server before video coding. If the terminal equipment only meets the preset condition, the cloud server completes the detection of the region of interest and then transmits the region information to the terminal equipment. And the terminal equipment enhances the image quality of the region of interest. If both the terminal equipment and the terminal equipment accord with the corresponding preset conditions, the terminal equipment completes the detection of the region of interest and the enhancement of the image quality.
According to the image quality enhancement method provided by the embodiment of the application, the cloud server determines image quality enhancement information of the corresponding terminal equipment; if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is met, skipping image quality enhancement operation, and encoding a target video which is not subjected to image quality enhancement to obtain a first code stream; and if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is not met, enhancing the image quality of the target video, and encoding the target video with the enhanced image quality to obtain a second code stream. In other words, in the embodiment of the present application, the cloud server determines, according to the image quality enhancement information of the terminal device, whether the image quality enhancement operation is performed by the terminal device or the cloud server, or the cloud server and the terminal device are cooperatively performed, so that the computing capability of the decoding end can be fully utilized, the computing resource of the encoding end can be saved, and further the encoding efficiency is improved.
The image quality enhancement method at the encoding end is described above, and the image quality enhancement method at the decoding end is described below.
Fig. 6 is a flowchart of an image quality enhancement method according to an embodiment of the present application, where the execution body of the embodiment of the present application may be the terminal device 102 shown in fig. 1, the decoder shown in fig. 3, or the terminal devices shown in fig. 4B to 4D.
As shown in fig. 6, the method provided in the embodiment of the present application includes:
s501, decoding the obtained code stream to obtain a target video.
S502, if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is met, enhancing the image quality of the decoded target video to obtain the target video with enhanced image quality;
s503, if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is not met, the decoded target video is used as the target video with enhanced image quality.
In this embodiment of the present application, the method for determining, by the terminal device, whether the image quality enhancement information of the terminal device meets the preset image quality enhancement condition at least includes the following modes:
in one mode, the cloud server transmits instruction information to the terminal device, and instructs the terminal device whether or not the image quality enhancement information satisfies a preset image quality enhancement condition. That is, after the cloud server determines the image quality enhancement capability of the terminal device according to the above embodiment, the image quality enhancement capability of the terminal device is sent to the terminal device.
In the second mode, the terminal device compares the image quality enhancement information with a preset image quality enhancement condition to determine whether the image quality enhancement information of the terminal device meets the preset image quality enhancement condition.
In some embodiments, the image quality enhancement information of the terminal device includes whether the terminal device has an image quality enhancement function, and an image quality enhancement computing capability corresponding to the terminal device when the terminal device has the image quality enhancement function.
In some embodiments, the method of embodiments of the present application further comprises:
step 11, receiving a first request sent by a cloud server, where the first request is used to request the image quality enhancement capability of the terminal device, and the first request includes a first test code stream;
step 12, decoding the first test code stream to obtain a first test video;
step 13, counting the time consumed when the image quality of the first test video is enhanced when the terminal equipment is determined to have the image quality enhancement function;
step 14, sending first information to the cloud server, wherein the first information is used for indicating whether the terminal equipment has the image quality enhancement function or not and indicating the corresponding image quality enhancement computing capability when the terminal equipment has the image quality enhancement function.
In some embodiments, the image quality enhancement condition includes a preset image quality enhancement computing capability, and the method of the embodiment of the application further includes:
step A, if the terminal equipment has an image quality enhancement function and the image quality enhancement computing capacity corresponding to the terminal equipment meets the preset image quality enhancement computing capacity, determining that the image quality enhancement information of the terminal equipment meets the image quality enhancement condition;
And B, if the terminal equipment does not have the image quality enhancement function or the terminal equipment has the image quality enhancement function and the image quality enhancement computing capability of the terminal equipment does not meet the preset image quality enhancement computing capability, determining that the image quality enhancement information of the terminal equipment does not meet the image quality enhancement condition.
In some embodiments, if the image quality enhancement computing capability of the terminal device includes a first time consumed by the terminal device when performing image quality enhancement on the first test video, the preset image quality enhancement computing capability includes a preset image quality enhancement elapsed time;
at this time, the step a includes a step A1: if the terminal equipment has the image quality enhancement function and the first time is less than or equal to the preset image quality enhancement time, determining that the image quality enhancement information of the terminal equipment meets the image quality enhancement condition.
The step B includes a step B1: if the terminal device has the image quality enhancement function and the first time is longer than the preset image quality enhancement time, determining that the image quality enhancement information of the terminal device does not meet the image quality enhancement condition.
In some embodiments, the first time comprises: the terminal device can increase the image quality of the whole image of each image in the first test video, or increase the image quality of a part of the image area of each image in the first test video.
In some embodiments, the first time comprises: and Shan Zhenping average duration, single frame maximum duration and variance of the first test video when the terminal equipment performs image quality enhancement.
In some embodiments, if the embodiment of the present application is applied to an application scene for enhancing the image quality of a region of interest of a target video, S502 includes the following cases:
in case 1, if the image quality enhancement information of the terminal device meets a preset image quality enhancement condition, and the region of interest detection information of the terminal device meets a preset region of interest detection condition, detecting the region of interest of the decoded target video to obtain a region of interest of the target video, and enhancing the image quality of the region of interest to obtain the target video with enhanced image quality.
In case 2, if the image quality enhancement information of the terminal device meets the preset image quality enhancement condition, the region of interest detection information of the terminal device does not meet the region of interest detection condition, and the terminal device has the function of receiving the externally input image region information, the region of interest information of the target video sent by the cloud server is received, and the image quality of the region of interest of the target video is enhanced according to the region of interest information, so as to obtain the enhanced target video.
In some embodiments, S502 above includes the following third scenario:
in case 3, if the image quality enhancement information of the terminal device meets the preset image quality enhancement condition, the region detection information of interest of the terminal device does not meet the region detection condition, and the terminal device has the function of not receiving the externally input image region information, the image quality of the whole image of the target video is enhanced, and the enhanced target video is obtained.
In some embodiments, the region of interest detection information of the terminal device includes whether the terminal device has a region of interest detection function, and a region of interest detection capability corresponding to the terminal device when the terminal device has the region of interest detection function.
In some embodiments, the method of embodiments of the present application further comprises the steps of:
step 21, receiving a second request sent by the cloud server, where the second request is used to request the region of interest detection capability of the terminal device, and the second request includes a second test code;
step 22, decoding the second test code stream to obtain a second test video;
step 23, counting a second time consumed when the terminal equipment is determined to have the region-of-interest detection function and the region-of-interest detection is performed on the second test video;
And step 24, sending second information to the cloud server, wherein the second information is used for indicating whether the terminal equipment has the region of interest detection function or not and indicating the corresponding region of interest detection computing capability of the terminal equipment when the terminal equipment has the region of interest detection function.
In some embodiments, if the region of interest detection condition includes a preset region of interest detection computing capability, the determining whether the region of interest detection information of the terminal device meets the preset region of interest detection condition includes the following steps:
step 31, if the terminal device has the function of detecting the region of interest and the computing capability of detecting the region of interest corresponding to the terminal device meets the preset computing capability of detecting the region of interest, determining that the information of detecting the region of interest of the terminal device meets the condition of detecting the region of interest;
step 32, if the terminal device does not have the function of detecting the region of interest, or if the terminal device has the function of detecting the region of interest and the calculation capability of detecting the region of interest of the terminal device does not meet the preset calculation capability of detecting the region of interest, determining that the detection information of the region of interest of the terminal device does not meet the detection condition of the region of interest.
In some embodiments, if the region of interest detection computing capability of the terminal device includes a second time consumed by the terminal device when performing the region of interest detection on the second test video, the corresponding preset region of interest detection computing capability includes a preset region of interest detection consumed time; the step 31 includes the following step 31-a, and the step 32 includes the following step 32-a:
step 31-a, if the terminal device has the function of detecting the region of interest and the second time is less than or equal to the preset time for detecting the region of interest, determining that the information for detecting the region of interest of the terminal device meets the condition for detecting the region of interest.
Step 32-a, if the terminal device has the function of detecting the region of interest and the second time is longer than the preset time for detecting the region of interest, determining that the information for detecting the region of interest of the terminal device does not meet the condition for detecting the region of interest.
In some embodiments, the second time comprises: and the terminal equipment performs at least one of single-frame average duration, single-frame maximum duration and variance when the second test video is subjected to region-of-interest detection.
According to the image quality enhancement method provided by the embodiment of the application, the terminal equipment obtains a target video by decoding the obtained code stream; if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is met, enhancing the image quality of the decoded target video to obtain the target video with enhanced image quality; and if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is not met, taking the decoded target video as the target video with enhanced image quality. In other words, in the embodiment of the present application, whether the image quality enhancement operation is performed by the terminal device or the cloud server is determined according to the image quality enhancement information of the terminal device, or the cloud server is performed in conjunction with the terminal device, so that the computing capability of the decoding end can be fully utilized, the computing resource of the encoding end can be saved, and the encoding efficiency can be further improved.
Fig. 7 is a flowchart of an image quality enhancement method according to an embodiment of the present application, and fig. 7 is a flowchart of an embodiment of the present application, including:
s601, the cloud server determines image quality enhancement information of the corresponding terminal equipment.
The image quality enhancement information of the terminal equipment comprises whether the terminal equipment has an image quality enhancement function or not, and the corresponding image quality enhancement computing capacity of the terminal equipment when the terminal equipment has the image quality enhancement function;
s602, the cloud server judges whether the terminal equipment has an image quality enhancement function.
If the terminal device has the image quality enhancement function, S602 is executed, and if the terminal device does not have the image quality enhancement function, S609 is executed.
S603, the cloud server judges whether the image quality enhancement computing capacity corresponding to the terminal equipment meets the preset image quality enhancement computing capacity.
If the image quality enhancement computing capability corresponding to the terminal device satisfies the preset image quality enhancement computing capability, S604 is executed, and if the image quality enhancement computing capability corresponding to the terminal device satisfies the preset image quality enhancement computing capability, S609 is executed.
And S604, the cloud server acquires the detection information of the region of interest of the terminal equipment.
The information of detecting the region of interest of the terminal equipment comprises whether the terminal equipment has the function of detecting the region of interest or not, and the corresponding capability of detecting the region of interest when the terminal equipment has the function of detecting the region of interest.
S605, the cloud server judges whether the terminal equipment has a region of interest detection function.
If the terminal device has the region of interest detection function, S606 is executed, and if the terminal device does not have the region of interest detection function, S608 is executed.
S606, the cloud server judges whether the detection and calculation capacity of the region of interest corresponding to the terminal equipment meets the preset detection and calculation capacity of the region of interest.
If the region of interest detection computing capability corresponding to the terminal device satisfies the preset region of interest detection computing capability, S607 is executed, and if the region of interest detection computing capability corresponding to the terminal device satisfies the preset region of interest detection computing capability, S608 is executed.
S607, the terminal device performs a region of interest detection operation and an image quality enhancement operation.
And S608, the cloud server performs region-of-interest detection operation, and the terminal equipment performs image quality enhancement operation.
S609, the cloud server performs a region of interest detection operation and an image quality enhancement operation.
According to the image quality enhancement method provided by the embodiment of the application, the cloud server acquires the image quality enhancement information and the region-of-interest detection information of the terminal equipment, and judges whether the image quality enhancement operation and the region-of-interest detection operation are executed by the terminal equipment or the cloud server and the terminal equipment according to the image quality enhancement information and the region-of-interest detection information of the terminal equipment, so that the computing capacity of a decoding end can be fully utilized, the computing resources of an encoding end can be saved, and the encoding efficiency is improved.
The image quality enhancement methods of the decoding end and the encoding end are described above, and the decoder is taken as a terminal device, the encoder is taken as a cloud server as an example, and the image quality enhancement interaction method between the terminal device and the cloud server is described below.
Fig. 8 is a schematic diagram of an interaction flow of an image quality enhancement method according to an embodiment of the present application, where, as shown in fig. 8, the interaction flow includes:
s801, the cloud server sends a rendering capability request to the terminal equipment through the client.
The rendering capability request is used for requesting the rendering capability of the terminal device, and the rendering capability of the embodiment of the application comprises the image quality enhancement capability and/or the region of interest detection capability of the terminal device.
Specifically, the cloud server initiates a rendering capability request to the terminal device through the START client.
Optionally, the rendering capability request protocol field includes at least one of: protocol version number, video frame rate, rendering algorithm type query.
Exemplary, the code for the video rendering capability request is shown in Table 7:
table 7 video rendering capability request
The rendering capability request in the embodiment of the present application may be understood as the first request and/or the second request, where the rendering capability request further includes a first test code stream and/or a second test code stream. The first test code stream is used for requesting the image quality enhancement capability of the terminal equipment, and the second test code stream is used for requesting the region-of-interest detection capability of the terminal equipment.
Optionally, the first test code stream and the second test code stream are the same code stream. That is, after receiving the first test code stream, the terminal device decodes the first test code stream to obtain a first test video. If the terminal equipment has the image quality enhancement function and does not have the region of interest detection function, the terminal equipment carries out image quality enhancement on the whole image region of the first test video. If the terminal equipment has the image quality enhancement function and the region of interest detection function, the terminal equipment detects the region of interest of the first test video and enhances the image quality of the region of interest.
In one possible implementation manner, the cloud server may send the first test code stream to the terminal device first, and send the second test code stream to the terminal device when determining that the specific image quality enhancement function of the terminal device meets the condition for the image quality enhancement capability, so as to obtain the region of interest detection capability of the terminal device.
S802, the terminal equipment returns response information to the cloud server through the client.
In case 1, the first test code stream and the second test code stream are the same code stream.
In case 1, the terminal device receives a request sent by the cloud server through the client, and if the terminal device has an image quality enhancement function, decodes a first test code stream in the request to obtain a first test video. If the terminal equipment has the function of detecting the interest, the terminal equipment detects the interest region of the first test video, enhances the image quality of the detected interest region, and sends information such as the image quality enhancement, the time spent for detecting the interest region and the like to the cloud server.
In case 1, the terminal device receives a request sent by the cloud server through the client, and if the terminal device has an image quality enhancement function, decodes a first test code stream in the request to obtain a first test video. If the terminal equipment does not have the interesting detection function, the terminal equipment performs image quality enhancement on the whole image area of the first test video and sends information such as time spent for image quality enhancement to the cloud server.
In case 1, the terminal device receives, through the client, a request sent by the cloud server, and if the terminal device does not have the image quality enhancement function, the response information sent by the terminal device to the cloud server includes information indicating that the terminal device does not have the image quality enhancement function.
In case 2, the first test code stream and the second test code stream are different code streams.
In case 2, the terminal device receives, through the client, the request sent by the cloud server, and if the terminal device has the image quality enhancement function, decodes the first test code stream in the request to obtain the first test video. And the terminal equipment performs image quality enhancement on the whole image area of the first test video, and sends the information of time information consumed by the image quality enhancement, whether the terminal equipment has an interested area detection function or not and the like to the cloud server.
After the cloud server receives the time information consumed by the image quality enhancement sent by the terminal equipment and the information such as whether the terminal equipment has the region of interest detection function or not, if the image quality enhancement capability of the terminal equipment is judged to meet the requirement, and the terminal equipment has the region of interest detection function, the cloud server sends a second test code stream to the terminal equipment through the client so as to determine the region of interest detection capability of the terminal equipment. Specifically, if the terminal device has the function of detecting the region of interest, the terminal device decodes the second test code stream to obtain a second test video, performs region of interest detection on the second test video, and sends information such as time spent in the region of interest detection to the cloud server through the client.
From the above, when the terminal device receives the capability information acquisition request sent by the cloud server, it returns a status flag (success 0, failure is a specific error code), a supported protocol version number, and capability information of the terminal device (the numerical definition is shown in table 5).
In the embodiment of the present application, the rendering tasks include two rendering tasks, which are respectively an image quality rendering task and an interest detection task, and may be alternatively represented by a render1 and a render 2.
The response information returned by the terminal device includes the following examples:
example 1, when the terminal device supports all requested rendering algorithm capabilities, the code of the video rendering capability response information is as shown in table 8:
table 8 video rendering capability response
/>
In this embodiment of the present application, if the terminal device supports both the image quality enhancement task and the interested detection task, the response information format of the terminal device may be shown in table 8.
Example 2, the terminal device only supports part of the requested rendering algorithm capabilities, and the code of the video rendering capability response information is shown in table 9:
TABLE 9
In this embodiment of the present application, if the terminal device only has the image quality enhancement capability, does not have the region of interest detection capability, or only has the region of interest detection capability, and does not have the image quality enhancement capability, the response information format of the terminal device may be as shown in table 9.
Example 3, if the terminal device does not support rendering capabilities, the code of the video rendering capability response information is as shown in table 10:
table 10
/>
Example 4, terminal device capability information request failed, the code of video rendering capability response information is as shown in table 11:
TABLE 11
It should be noted that, the above tables 7 to 11 are examples of response information of the terminal device, and in the embodiment of the present application, the format of the response information returned by the terminal device includes, but is not limited to, those shown in tables 7 to 11.
S803, the cloud server determines the optimal rendering configuration.
S804, the cloud server sends the optimal rendering configuration to the terminal equipment through the client.
In the embodiment of the application, after receiving the terminal rendering capability information, the cloud server determines the optimal rendering cooperative configuration of the current terminal device, and sends the cooperative task to the terminal device.
In some embodiments, in the embodiments of the present application, the code of the collaborative task is as shown in table 12:
table 12, video rendering task delivery
/>
It should be noted that, the above table 12 is described by taking a rendering task as an example of video sharpening, and in this embodiment of the present application, the rendering task issued by the cloud server to the terminal device is known according to the above embodiment, and includes the following several steps: the method comprises the steps that firstly, a cloud server achieves image quality enhancement and region-of-interest detection tasks, secondly, the cloud server achieves the region-of-interest detection tasks, a terminal device achieves the image quality enhancement tasks, thirdly, the cloud server achieves video coding, and the terminal device achieves the region-of-interest detection and image quality enhancement tasks.
In addition, the parameters shown in tables 7 to 12 are only examples, and the embodiments of the present application are not limited thereto.
It should be understood that fig. 5-8 are only examples of the present application and should not be construed as limiting the present application.
The preferred embodiments of the present application have been described in detail above with reference to the accompanying drawings, but the present application is not limited to the specific details of the above embodiments, and various simple modifications may be made to the technical solutions of the present application within the scope of the technical concept of the present application, and all the simple modifications belong to the protection scope of the present application. For example, the specific features described in the above embodiments may be combined in any suitable manner, and in order to avoid unnecessary repetition, various possible combinations are not described in detail. As another example, any combination of the various embodiments of the present application may be made, as long as it does not depart from the spirit of the present application, and it should also be construed as being disclosed herein.
Method embodiments of the present application are described in detail above in connection with fig. 5-8, and apparatus embodiments of the present application are described in detail below.
Fig. 9 is a schematic structural diagram of an image quality enhancement device according to an embodiment of the present application, where the device is applied to a cloud server, and the device 10 includes:
a determining unit 11, configured to obtain image quality enhancement information of the terminal device;
An encoding unit 12, configured to skip an image quality enhancement operation and encode a target video that is not subjected to image quality enhancement if the image quality enhancement information of the terminal device meets a preset image quality enhancement condition, so as to obtain a first code stream; and if the image quality enhancement information of the terminal equipment does not meet the preset image quality enhancement condition, enhancing the image quality of the target video, and encoding the target video with the enhanced image quality to obtain a second code stream.
In some embodiments, the image quality enhancement information of the terminal device includes whether the terminal device has an image quality enhancement function, and an image quality enhancement computing capability corresponding to the terminal device when the terminal device has the image quality enhancement function.
In some embodiments, the determining unit 11 is specifically configured to obtain a first test code stream, and send a first request to the terminal device, where the first request is used to request an image quality enhancement capability of the terminal device, and the first request includes the first test code stream; receiving first information sent by the terminal equipment, wherein the first information is used for indicating whether the terminal equipment has the image quality enhancement function or not and indicating image quality enhancement computing capacity corresponding to the terminal equipment when the terminal equipment has the image quality enhancement function; and obtaining the image quality enhancement information of the terminal equipment according to the first information.
In some embodiments, the image quality enhancement condition includes a preset image quality enhancement computing capability, and the encoding unit 12 is further configured to determine that the image quality enhancement information of the terminal device meets the image quality enhancement condition if the terminal device has the image quality enhancement function and the image quality enhancement computing capability corresponding to the terminal device meets the preset image quality enhancement computing capability; and if the terminal equipment does not have the image quality enhancement function or the terminal equipment has the image quality enhancement function and the image quality enhancement computing capability of the terminal equipment does not meet the preset image quality enhancement computing capability, determining that the image quality enhancement information of the terminal equipment does not meet the image quality enhancement condition.
In some embodiments, if the image quality enhancement computing capability of the terminal device includes a first time consumed by the terminal device in performing image quality enhancement on the first test video, the preset image quality enhancement computing capability includes a preset image quality enhancement consumption time; the encoding unit 12 is specifically configured to determine that the image quality enhancement information of the terminal device meets the image quality enhancement condition if the terminal device has the image quality enhancement function and the first time is less than or equal to the preset image quality enhancement time; and if the terminal equipment has the image quality enhancement function and the first time is longer than the preset image quality enhancement time, determining that the image quality enhancement information of the terminal equipment does not meet the image quality enhancement condition.
In some embodiments, the first time comprises: and the terminal equipment is used for carrying out image quality enhancement on the whole image of each image in the first test video, or carrying out image quality enhancement on partial image areas of each image in the first test video.
In some embodiments, the first time comprises: and the terminal equipment performs at least one of single-frame average duration, single-frame maximum duration and variance on the first test video when the image quality of the first test video is enhanced.
In some embodiments, if the image quality enhancement information of the terminal device meets a preset image quality enhancement condition, the determining unit 11 is further configured to obtain the region of interest detection information of the terminal device if the image quality enhancement information of the terminal device meets the preset image quality enhancement condition; the encoding unit 12 is specifically configured to skip the image quality enhancement operation and the region of interest detection operation if the region of interest detection information of the terminal device meets a preset region of interest detection condition, and encode the target video that is not subjected to image quality enhancement and region of interest detection, so as to obtain the first code stream.
In some embodiments, the encoding unit 12 is further configured to detect whether the terminal device has a function of receiving externally input image area information if the area of interest detection information of the terminal device does not meet the area of interest detection condition; if the terminal equipment has the function of receiving the image area information input from the outside, detecting the region of interest of the target video to obtain the region of interest information of the target video, sending the region of interest information to the terminal equipment, skipping the image quality enhancement operation, and encoding the target video which is not subjected to image quality enhancement to obtain the first code stream; and if the terminal equipment does not have the function of receiving the externally input image area information, detecting the region of interest of the target video to obtain the region of interest of the target video, enhancing the image quality of the region of interest, and encoding the target video with the enhanced image quality to obtain the second code stream.
In some embodiments, the region of interest detection information of the terminal device includes whether the terminal device has a region of interest detection function, and a region of interest detection capability corresponding to the terminal device when the terminal device has the region of interest detection function.
In some embodiments, the determining unit 11 is specifically configured to obtain a second test code stream, and send a second request to the terminal device, where the second request is used to request the region of interest detection capability of the terminal device, and the second request includes the second test code stream; receiving second information sent by the terminal equipment, wherein the second information is used for indicating whether the terminal equipment has an interesting region detection function or not and indicating the interesting region detection computing capacity corresponding to the terminal equipment when the terminal equipment has the interesting region detection function; and obtaining the detection information of the region of interest of the terminal equipment according to the second information.
In some embodiments, the region of interest detection condition includes a preset region of interest detection computing capability, and the encoding unit 12 is further configured to determine that the region of interest detection information of the terminal device meets the region of interest detection condition if the terminal device has the region of interest detection function and the region of interest detection computing capability corresponding to the terminal device meets the preset region of interest detection computing capability; and if the terminal equipment does not have the region-of-interest detection function, or the terminal equipment has the region-of-interest detection function and the region-of-interest detection computing capacity of the terminal equipment does not meet the preset region-of-interest detection computing capacity, determining that the region-of-interest detection information of the terminal equipment does not meet the region-of-interest detection condition.
In some embodiments, if the region of interest detection computing capability of the terminal device includes a second time consumed by the terminal device in performing the region of interest detection on the second test video, the preset region of interest detection computing capability includes a preset region of interest detection consumed time;
the encoding unit 12 is specifically configured to determine that the region of interest detection information of the terminal device meets the region of interest detection condition if the terminal device has the region of interest detection function and the second time is less than or equal to the preset region of interest detection time; if the terminal equipment has the region-of-interest detection function and the second time is longer than the preset region-of-interest detection time, determining that the region-of-interest detection information of the terminal equipment does not meet the region-of-interest detection condition.
In some embodiments, the second time comprises: and the terminal equipment performs at least one of single-frame average duration, single-frame maximum duration and variance on the second test video when the region of interest is detected.
It should be understood that apparatus embodiments and method embodiments may correspond with each other and that similar descriptions may refer to the method embodiments. To avoid repetition, no further description is provided here. Specifically, the apparatus 10 shown in fig. 9 may perform the above-described method embodiment, and the foregoing and other operations and/or functions of each module in the apparatus 9 are respectively for implementing the above-described method embodiment shown in fig. 4, and are not further described herein for brevity.
Fig. 10 is a schematic structural diagram of an image quality enhancement device according to an embodiment of the present application, where the device is applied to a terminal device, and the device 20 includes:
a decoding unit 21, configured to decode the obtained code stream to obtain a target video;
a processing unit 22, configured to, if the image quality enhancement information of the terminal device meets a preset image quality enhancement condition, perform image quality enhancement on the decoded target video, so as to obtain the target video with enhanced image quality; and if the image quality enhancement information of the terminal equipment does not meet the preset image quality enhancement condition, taking the decoded target video as the target video with enhanced image quality.
In some embodiments, the image quality enhancement information of the terminal device includes whether the terminal device has an image quality enhancement function, and an image quality enhancement computing capability corresponding to the terminal device when the terminal device has the image quality enhancement function.
In some embodiments, the processing unit 22 is further configured to receive a first request sent by the cloud server, where the first request is used to request the image quality enhancement capability of the terminal device, and the first request includes a first test code stream; decoding the first test code stream to obtain a first test video; when the terminal equipment is determined to have the image quality enhancement function, counting the time consumed when the first test video is subjected to image quality enhancement; and sending first information to the cloud server, wherein the first information is used for indicating whether the terminal equipment has the image quality enhancement function or not and indicating the corresponding image quality enhancement computing capability of the terminal equipment when the terminal equipment has the image quality enhancement function.
In some embodiments, the image quality enhancement condition includes a preset image quality enhancement computing capability, and the processing unit 22 is further configured to determine that the image quality enhancement information of the terminal device meets the image quality enhancement condition if the terminal device has the image quality enhancement function and the image quality enhancement computing capability corresponding to the terminal device meets the preset image quality enhancement computing capability; and if the terminal equipment does not have the image quality enhancement function or the terminal equipment has the image quality enhancement function and the image quality enhancement computing capability of the terminal equipment does not meet the preset image quality enhancement computing capability, determining that the image quality enhancement information of the terminal equipment does not meet the image quality enhancement condition.
In some embodiments, if the image quality enhancement computing capability of the terminal device includes a first time consumed by the terminal device in performing image quality enhancement on the first test video, the preset image quality enhancement computing capability includes a preset image quality enhancement consumption time;
a processing unit 22, specifically configured to determine that the image quality enhancement information of the terminal device satisfies the image quality enhancement condition if the terminal device has the image quality enhancement function and the first time is less than or equal to the preset image quality enhancement time; and if the terminal equipment has the image quality enhancement function and the first time is longer than the preset image quality enhancement time, determining that the image quality enhancement information of the terminal equipment does not meet the image quality enhancement condition.
In some embodiments, the first time comprises: and the terminal equipment is used for carrying out image quality enhancement on the whole image of each image in the first test video, or carrying out image quality enhancement on partial image areas of each image in the first test video.
In some embodiments, the first time comprises: and the terminal equipment performs at least one of single-frame average duration, single-frame maximum duration and variance on the first test video when the image quality of the first test video is enhanced.
In some embodiments, the processing unit 22 is specifically configured to perform region of interest detection on the decoded target video to obtain a region of interest of the target video and perform image quality enhancement on the region of interest if the image quality enhancement information of the terminal device meets the preset image quality enhancement condition and the region of interest detection information of the terminal device meets the preset region of interest detection condition, if the image quality enhancement information of the terminal device meets the preset image quality enhancement condition; and if the image quality enhancement information of the terminal equipment meets the preset image quality enhancement condition, and the region of interest detection information of the terminal equipment does not meet the region of interest detection condition, and the terminal equipment has the function of receiving the externally input image region information, receiving the region of interest information of the target video sent by the cloud server, and carrying out image quality enhancement on the region of interest of the target video according to the region of interest information to obtain the enhanced target video.
In some embodiments, if the image quality enhancement information of the terminal device meets the preset image quality enhancement condition, the region of interest detection information of the terminal device does not meet the region of interest detection condition, and the terminal device has a function of not receiving the externally input image region information, the processing unit 22 is further configured to enhance the image quality of the entire image of the target video, and obtain the enhanced target video.
In some embodiments, the region of interest detection information of the terminal device includes whether the terminal device has a region of interest detection function, and a region of interest detection capability corresponding to the terminal device when the terminal device has the region of interest detection function.
In some embodiments, the processing unit 22 is further configured to receive a second request sent by the cloud server, where the second request is used to request the region of interest detection capability of the terminal device, and the second request includes a second test code stream; decoding the second test code stream to obtain a second test video; when the terminal equipment is determined to have the function of detecting the region of interest, counting second time consumed when the second test video is subjected to the detection of the region of interest; and the second information is used for indicating whether the terminal equipment has the region-of-interest detection function or not and indicating the region-of-interest detection computing capacity corresponding to the terminal equipment when the terminal equipment has the region-of-interest detection function.
In some embodiments, the region of interest detection condition includes a preset region of interest detection computing capability, and the processing unit 22 is further configured to determine that the region of interest detection information of the terminal device meets the region of interest detection condition if the terminal device has the region of interest detection function and the region of interest detection computing capability corresponding to the terminal device meets the preset region of interest detection computing capability; and if the terminal equipment does not have the region-of-interest detection function, or the terminal equipment has the region-of-interest detection function and the region-of-interest detection computing capacity of the terminal equipment does not meet the preset region-of-interest detection computing capacity, determining that the region-of-interest detection information of the terminal equipment does not meet the region-of-interest detection condition.
In some embodiments, if the region of interest detection computing capability of the terminal device includes a second time consumed by the terminal device in performing the region of interest detection on the second test video, the preset region of interest detection computing capability includes a preset region of interest detection consumed time;
The processing unit 22 is specifically configured to determine that the region of interest detection information of the terminal device meets the region of interest detection condition if the terminal device has the region of interest detection function and the second time is less than or equal to the preset region of interest detection time; if the terminal equipment has the region-of-interest detection function and the second time is longer than the preset region-of-interest detection time, determining that the region-of-interest detection information of the terminal equipment does not meet the region-of-interest detection condition.
In some embodiments, the second time comprises: and the terminal equipment performs at least one of single-frame average duration, single-frame maximum duration and variance on the second test video when the region of interest is detected.
It should be understood that apparatus embodiments and method embodiments may correspond with each other and that similar descriptions may refer to the method embodiments. To avoid repetition, no further description is provided here. Specifically, the apparatus 20 shown in fig. 10 may perform the above-described method embodiment, and the foregoing and other operations and/or functions of each module in the apparatus 10 are respectively for implementing the above-described method embodiment shown in fig. 5, and are not further described herein for brevity.
The apparatus of the embodiments of the present application are described above in terms of functional modules in conjunction with the accompanying drawings. It should be understood that the functional module may be implemented in hardware, or may be implemented by instructions in software, or may be implemented by a combination of hardware and software modules. Specifically, each step of the method embodiments in the embodiments of the present application may be implemented by an integrated logic circuit of hardware in a processor and/or an instruction in software form, and the steps of the method disclosed in connection with the embodiments of the present application may be directly implemented as a hardware decoding processor or implemented by a combination of hardware and software modules in the decoding processor. Alternatively, the software modules may be located in a well-established storage medium in the art such as random access memory, flash memory, read-only memory, programmable read-only memory, electrically erasable programmable memory, registers, and the like. The storage medium is located in a memory, and the processor reads information in the memory, and in combination with hardware, performs the steps in the above method embodiments.
Fig. 11 is a schematic block diagram of an electronic device provided in an embodiment of the present application, where the electronic device may be the cloud server and/or the terminal device described above.
As shown in fig. 11, the electronic device 40 may include:
a memory 41 and a memory 42, the memory 41 being adapted to store a computer program and to transfer the program code to the memory 42. In other words, the memory 42 may call and run a computer program from the memory 41 to implement the methods in the embodiments of the present application.
For example, the memory 42 may be used to perform the method embodiments described above in accordance with instructions in the computer program.
In some embodiments of the present application, the memory 42 may include, but is not limited to:
general purpose processors, digital signal processors (Digital Signal Processor, DSP), application specific integrated circuits (Application Specific Integrated Circuit, ASIC), field programmable gate arrays (Field Programmable Gate Array, FPGA) or other programmable logic devices, discrete gate or transistor logic devices, discrete hardware components, and the like.
In some embodiments of the present application, the memory 41 includes, but is not limited to:
volatile memory and/or nonvolatile memory. The nonvolatile Memory may be a Read-Only Memory (ROM), a Programmable ROM (PROM), an Erasable PROM (EPROM), an Electrically Erasable EPROM (EEPROM), or a flash Memory. The volatile memory may be random access memory (Random Access Memory, RAM) which acts as an external cache. By way of example, and not limitation, many forms of RAM are available, such as Static RAM (SRAM), dynamic RAM (DRAM), synchronous DRAM (SDRAM), double Data Rate SDRAM (Double Data Rate SDRAM), enhanced SDRAM (ESDRAM), synchronous Link DRAM (SLDRAM), and Direct memory bus RAM (DR RAM).
In some embodiments of the present application, the computer program may be partitioned into one or more modules that are stored in the memory 41 and executed by the memory 42 to perform the methods provided herein. The one or more modules may be a series of computer program instruction segments capable of performing particular functions for describing the execution of the computer program in the video production device.
As shown in fig. 11, the electronic device 40 may further include:
a transceiver 40, the transceiver 43 may be connected to the memory 42 or the memory 41.
The memory 42 may control the transceiver 43 to communicate with other devices, and in particular, may transmit information or data to other devices or receive information or data transmitted by other devices. The transceiver 43 may include a transmitter and a receiver. The transceiver 43 may further include antennas, the number of which may be one or more.
It will be appreciated that the various components in the video production device are connected by a bus system that includes, in addition to a data bus, a power bus, a control bus and a status signal bus.
The present application also provides a computer storage medium having stored thereon a computer program which, when executed by a computer, enables the computer to perform the method of the above-described method embodiments. Alternatively, embodiments of the present application also provide a computer program product comprising instructions which, when executed by a computer, cause the computer to perform the method of the method embodiments described above.
When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When the computer program instructions are loaded and executed on a computer, all or portions of the procedures or functions described in accordance with embodiments of the present application are provided. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line (digital subscriber line, DSL)) or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device including one or more servers, data centers, etc. that can be integrated with the available medium. The usable medium may be a magnetic medium (e.g., a floppy disk, a hard disk, a magnetic tape), an optical medium (e.g., a digital video disc (digital video disc, DVD)), or a semiconductor medium (e.g., a Solid State Disk (SSD)), or the like.
Those of ordinary skill in the art will appreciate that the various illustrative modules and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the present application.
In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, and for example, the division of the modules is merely a logical function division, and there may be additional divisions when actually implemented, for example, multiple modules or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or modules, which may be in electrical, mechanical, or other forms.
The modules illustrated as separate components may or may not be physically separate, and components shown as modules may or may not be physical modules, i.e., may be located in one place, or may be distributed over a plurality of network elements. Some or all of the modules may be selected according to actual needs to achieve the purpose of the solution of this embodiment. For example, functional modules in the embodiments of the present application may be integrated into one processing module, or each module may exist alone physically, or two or more modules may be integrated into one module.
The foregoing is merely specific embodiments of the present application, but the scope of protection of the present application is not limited thereto, and any person skilled in the art who is familiar with the technical scope of the present application can easily think about changes or substitutions, and all the changes and substitutions are included in the scope of protection of the present application. Therefore, the protection scope of the present application shall be subject to the protection scope of the claims.
Claims (20)
1. An image quality enhancement method, which is applied to a cloud server, comprises the following steps:
determining image quality enhancement information of corresponding terminal equipment;
if the image quality enhancement information of the terminal equipment indicates that a preset image quality enhancement condition is met, the cloud server skips image quality enhancement operation and encodes a target video which is not subjected to image quality enhancement to obtain a first code stream;
And if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is not met, enhancing the image quality of the target video, and encoding the target video with enhanced image quality to obtain a second code stream.
2. The method according to claim 1, wherein the image quality enhancement information of the terminal device includes whether the terminal device has an image quality enhancement function, and an image quality enhancement computing capability to which the terminal device corresponds when the terminal device has the image quality enhancement function.
3. The method according to claim 2, wherein the determining the image quality enhancement information of the corresponding terminal device includes:
acquiring a first test code stream, and sending a first request to the terminal equipment, wherein the first request is used for requesting the image quality enhancement capability of the terminal equipment, and the first request comprises the first test code stream;
receiving first information sent by the terminal equipment, wherein the first information is used for indicating whether the terminal equipment has the image quality enhancement function or not and indicating the corresponding image quality enhancement computing capability of the terminal equipment when the terminal equipment has the image quality enhancement function;
And obtaining the image quality enhancement information of the terminal equipment according to the first information.
4. The method according to claim 3, wherein the image quality enhancement condition includes a preset image quality enhancement computing capability, the method further comprising:
if the terminal equipment has the image quality enhancement function and the image quality enhancement computing capability corresponding to the terminal equipment meets the preset image quality enhancement computing capability, determining that the image quality enhancement information of the terminal equipment meets the image quality enhancement condition;
and if the terminal equipment does not have the image quality enhancement function or the terminal equipment has the image quality enhancement function and the image quality enhancement computing capability of the terminal equipment does not meet the preset image quality enhancement computing capability, determining that the image quality enhancement information of the terminal equipment does not meet the image quality enhancement condition.
5. The method according to claim 4, wherein if the image quality enhancement computing capability of the terminal device includes a first time consumed by the terminal device in performing image quality enhancement on a first test video, the preset image quality enhancement computing capability includes a preset image quality enhancement consumed time, and the first test video is a video obtained by the terminal device after decoding the first test bitstream;
And if the terminal device has the image quality enhancement function and the image quality enhancement computing capability corresponding to the terminal device satisfies the preset image quality enhancement computing capability, determining that the image quality enhancement information of the terminal device satisfies the image quality enhancement condition includes:
if the terminal equipment has the image quality enhancement function and the first time is smaller than or equal to the preset image quality enhancement time, determining that the image quality enhancement information of the terminal equipment meets the image quality enhancement condition;
if the terminal device has the image quality enhancement function and the image quality enhancement computing capability of the terminal device does not satisfy the preset image quality enhancement computing capability, determining that the image quality enhancement information of the terminal device does not satisfy the image quality enhancement condition includes:
and if the terminal equipment has the image quality enhancement function and the first time is longer than the preset image quality enhancement time, determining that the image quality enhancement information of the terminal equipment does not meet the image quality enhancement condition.
6. The method of claim 5, wherein the first time comprises: and the terminal equipment is used for carrying out image quality enhancement on the whole image of each image in the first test video, or carrying out image quality enhancement on partial image areas of each image in the first test video.
7. The method of claim 6, wherein the first time comprises: and the terminal equipment performs at least one of single-frame average duration, single-frame maximum duration and variance on the first test video when the image quality of the first test video is enhanced.
8. The method according to any one of claims 1 to 7, wherein if the image quality enhancement information of the terminal device meets a preset image quality enhancement condition, the cloud server skips the image quality enhancement operation and encodes the target video not subjected to the image quality enhancement to obtain a code stream, including:
if the image quality enhancement information of the terminal equipment meets the preset image quality enhancement condition, acquiring the detection information of the region of interest of the terminal equipment;
and if the region of interest detection information of the terminal equipment meets the preset region of interest detection condition, skipping the image quality enhancement operation and the region of interest detection operation, and encoding the target video which is not subjected to image quality enhancement and region of interest detection to obtain the first code stream.
9. The method of claim 8, wherein the method further comprises:
if the region of interest detection information of the terminal equipment does not meet the region of interest detection condition, detecting whether the terminal equipment has a function of receiving externally input image region information;
If the terminal equipment has the function of receiving the image area information input from the outside, detecting the region of interest of the target video to obtain the region of interest information of the target video, sending the region of interest information to the terminal equipment, skipping the image quality enhancement operation, and encoding the target video which is not subjected to image quality enhancement to obtain the first code stream;
and if the terminal equipment does not have the function of receiving the externally input image area information, detecting the region of interest of the target video to obtain the region of interest of the target video, enhancing the image quality of the region of interest, and encoding the target video with the enhanced image quality to obtain the second code stream.
10. The method of claim 8, wherein the region of interest detection information of the terminal device includes whether the terminal device has a region of interest detection function, and a region of interest detection capability corresponding to the terminal device when the terminal device has the region of interest detection function.
11. The method according to claim 10, wherein the acquiring the region of interest detection information of the terminal device includes:
Acquiring a second test code stream, and sending a second request to the terminal equipment, wherein the second request is used for requesting the detection capability of the region of interest of the terminal equipment, and the second request comprises the second test code stream;
receiving second information sent by the terminal equipment, wherein the second information is used for indicating whether the terminal equipment has an interesting region detection function or not and indicating the interesting region detection computing capacity corresponding to the terminal equipment when the terminal equipment has the interesting region detection function;
and obtaining the detection information of the region of interest of the terminal equipment according to the second information.
12. The method of claim 11, wherein the region of interest detection condition includes a preset region of interest detection computing power, the method further comprising:
if the terminal equipment has the function of detecting the region of interest and the region of interest detection computing capacity corresponding to the terminal equipment meets the preset region of interest detection computing capacity, determining that the region of interest detection information of the terminal equipment meets the region of interest detection condition;
and if the terminal equipment does not have the region-of-interest detection function, or the terminal equipment has the region-of-interest detection function and the region-of-interest detection computing capacity of the terminal equipment does not meet the preset region-of-interest detection computing capacity, determining that the region-of-interest detection information of the terminal equipment does not meet the region-of-interest detection condition.
13. The method according to claim 12, wherein if the region of interest detection computing capability of the terminal device includes a second time consumed by the terminal device in performing the region of interest detection on a second test video, the preset region of interest detection computing capability includes a preset region of interest detection consumed time, and the second test video is a video obtained by decoding the second test bitstream by the terminal device;
if the terminal device has the function of detecting the region of interest, and the computing capability of detecting the region of interest corresponding to the terminal device satisfies the preset computing capability of detecting the region of interest, determining that the information of detecting the region of interest of the terminal device satisfies the preset condition of detecting the region of interest includes:
if the terminal equipment has the function of detecting the region of interest and the second time is less than or equal to the preset time for detecting the region of interest, determining that the information for detecting the region of interest of the terminal equipment meets the condition for detecting the region of interest;
if the terminal device has the function of detecting the region of interest and the computing capability of detecting the region of interest of the terminal device does not meet the preset computing capability of detecting the region of interest, determining that the information of detecting the region of interest of the terminal device does not meet the condition of detecting the region of interest includes:
If the terminal equipment has the region-of-interest detection function and the second time is longer than the preset region-of-interest detection time, determining that the region-of-interest detection information of the terminal equipment does not meet the region-of-interest detection condition.
14. The method of claim 13, wherein the second time comprises: and the terminal equipment performs at least one of single-frame average duration, single-frame maximum duration and variance on the second test video when the region of interest is detected.
15. An image quality enhancement method, applied to a terminal device, comprising:
decoding the obtained code stream to obtain a target video;
if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is met, performing image quality enhancement on the decoded target video to obtain the target video with enhanced image quality;
and if the image quality enhancement information of the terminal equipment indicates that the preset image quality enhancement condition is not met, taking the decoded target video as the target video with enhanced image quality.
16. The method of claim 15, wherein the method further comprises:
Receiving a first request sent by a cloud server, wherein the first request is used for requesting the image quality enhancement capability of the terminal equipment, and the first request comprises a first test code stream;
decoding the first test code stream to obtain a first test video;
when the terminal equipment is determined to have the image quality enhancement function, counting the time consumed when the first test video is subjected to image quality enhancement;
and sending first information to the cloud server, wherein the first information is used for indicating whether the terminal equipment has the image quality enhancement function or not and indicating the corresponding image quality enhancement computing capability of the terminal equipment when the terminal equipment has the image quality enhancement function.
17. The method according to claim 15, wherein if the image quality enhancement information of the terminal device satisfies a preset image quality enhancement condition, performing image quality enhancement on the decoded target video to obtain the target video with enhanced image quality, including:
if the image quality enhancement information of the terminal equipment meets the preset image quality enhancement condition, and the region of interest detection information of the terminal equipment meets the preset region of interest detection condition, detecting the region of interest of the decoded target video to obtain a region of interest of the target video, and enhancing the image quality of the region of interest to obtain the target video with enhanced image quality;
And if the image quality enhancement information of the terminal equipment meets the preset image quality enhancement condition, and the region of interest detection information of the terminal equipment does not meet the region of interest detection condition, and the terminal equipment has the function of receiving the externally input image region information, receiving the region of interest information of the target video sent by a cloud server, and enhancing the image quality of the region of interest of the target video according to the region of interest information to obtain the enhanced target video.
18. The method according to claim 17, wherein if the image quality enhancement information of the terminal device satisfies the preset image quality enhancement condition, the region-of-interest detection information of the terminal device does not satisfy the region-of-interest detection condition, and the terminal device has a function of not receiving externally input image region information, the method further comprises:
and carrying out image quality enhancement on the whole image of the target video to obtain the enhanced target video.
19. The method of claim 17, wherein the method further comprises:
receiving a second request sent by the cloud server, wherein the second request is used for requesting the region-of-interest detection capability of the terminal equipment, and the second request comprises a second test code stream;
Decoding the second test code stream to obtain a second test video;
when the terminal equipment is determined to have the function of detecting the region of interest, counting second time consumed when the second test video is subjected to the detection of the region of interest;
and the second information is used for indicating whether the terminal equipment has the region-of-interest detection function or not and indicating the region-of-interest detection computing capacity corresponding to the terminal equipment when the terminal equipment has the region-of-interest detection function.
20. An electronic device, comprising:
a processor and a memory for storing a computer program, the processor being for invoking and running the computer program stored in the memory to perform the method of any of claims 1 to 14 or 15 to 19.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210103029.7A CN116567287A (en) | 2022-01-27 | 2022-01-27 | Image quality enhancement method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202210103029.7A CN116567287A (en) | 2022-01-27 | 2022-01-27 | Image quality enhancement method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN116567287A true CN116567287A (en) | 2023-08-08 |
Family
ID=87488485
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202210103029.7A Pending CN116567287A (en) | 2022-01-27 | 2022-01-27 | Image quality enhancement method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN116567287A (en) |
-
2022
- 2022-01-27 CN CN202210103029.7A patent/CN116567287A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| KR102431537B1 (en) | Encoders, decoders and corresponding methods using IBC dedicated buffers and default value refreshing for luma and chroma components | |
| CN113748677A (en) | Encoder, decoder and corresponding intra prediction method | |
| CN113170143B (en) | Encoder, decoder and corresponding deduction method of boundary strength of deblocking filter | |
| WO2020211765A1 (en) | An encoder, a decoder and corresponding methods harmonzting matrix-based intra prediction and secoundary transform core selection | |
| CN112954367B (en) | Encoder, decoder and corresponding methods using palette coding | |
| CN113545063A (en) | Method and apparatus for intra prediction using linear model | |
| CN112385227A (en) | Method and apparatus for intra prediction | |
| US11496754B2 (en) | Video encoder, video decoder, and corresponding method of predicting random access pictures | |
| CN116016932A (en) | Apparatus and method for deblocking filter in video coding | |
| US11516470B2 (en) | Video coder and corresponding method | |
| CN114205582B (en) | Loop filtering method, device and equipment for video coding and decoding | |
| CN116567256A (en) | Hierarchical coding method, hierarchical coding device, hierarchical coding equipment and storage medium | |
| KR20230129068A (en) | Scalable encoding and decoding method and apparatus | |
| CN116567287A (en) | Image quality enhancement method and device and electronic equipment | |
| CN115052163A (en) | Encoder, decoder and corresponding methods for transform processing | |
| WO2022155922A1 (en) | Video coding method and system, video decoding method and system, video coder and video decoder | |
| RU2789030C2 (en) | Device and method for deblocking filter in video encoding | |
| RU2803063C2 (en) | Encoder, decoder and corresponding methods that are used for the conversion process | |
| CN114900702B (en) | Codec and corresponding method for reducing complexity of intra prediction for planar mode | |
| WO2023221599A1 (en) | Image filtering method and apparatus and device | |
| CN113766227B (en) | Quantization and inverse quantization method and apparatus for image encoding and decoding | |
| RU2801326C2 (en) | Coder, decoder and corresponding methods using allocated ibc buffer and default value updated brightness and colour component | |
| RU2783348C1 (en) | Encoder, decoder and corresponding methods for obtaining the boundary power of the debloking filter | |
| WO2022174475A1 (en) | Video encoding method and system, video decoding method and system, video encoder, and video decoder | |
| CN116567244A (en) | Determination method, device, equipment and storage medium of decoding configuration parameters |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination | ||
| REG | Reference to a national code |
Ref country code: HK Ref legal event code: DE Ref document number: 40093105 Country of ref document: HK |