CN117764834A - Image restoration method and device and electronic equipment - Google Patents
Image restoration method and device and electronic equipment Download PDFInfo
- Publication number
- CN117764834A CN117764834A CN202311600320.6A CN202311600320A CN117764834A CN 117764834 A CN117764834 A CN 117764834A CN 202311600320 A CN202311600320 A CN 202311600320A CN 117764834 A CN117764834 A CN 117764834A
- Authority
- CN
- China
- Prior art keywords
- image
- target image
- residual value
- value
- target
- Prior art date
- Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
- Pending
Links
- 238000000034 method Methods 0.000 title claims abstract description 57
- 238000012545 processing Methods 0.000 claims description 48
- 238000004590 computer program Methods 0.000 claims description 12
- 238000010586 diagram Methods 0.000 claims description 9
- 238000011084 recovery Methods 0.000 claims description 7
- 230000006835 compression Effects 0.000 abstract description 15
- 238000007906 compression Methods 0.000 abstract description 15
- 230000000694 effects Effects 0.000 abstract description 7
- 230000000903 blocking effect Effects 0.000 abstract description 6
- 230000000241 respiratory effect Effects 0.000 abstract description 6
- 230000005540 biological transmission Effects 0.000 description 20
- 230000008569 process Effects 0.000 description 13
- 238000000605 extraction Methods 0.000 description 10
- 230000006870 function Effects 0.000 description 8
- 238000004891 communication Methods 0.000 description 7
- 230000006837 decompression Effects 0.000 description 6
- 230000004927 fusion Effects 0.000 description 6
- 230000000007 visual effect Effects 0.000 description 5
- 230000003287 optical effect Effects 0.000 description 4
- 230000004044 response Effects 0.000 description 4
- 230000008859 change Effects 0.000 description 3
- 230000008878 coupling Effects 0.000 description 3
- 238000010168 coupling process Methods 0.000 description 3
- 238000005859 coupling reaction Methods 0.000 description 3
- 238000013461 design Methods 0.000 description 3
- 238000005516 engineering process Methods 0.000 description 3
- 238000005286 illumination Methods 0.000 description 3
- 238000012937 correction Methods 0.000 description 2
- 238000004519 manufacturing process Methods 0.000 description 2
- 238000012986 modification Methods 0.000 description 2
- 230000004048 modification Effects 0.000 description 2
- 238000011176 pooling Methods 0.000 description 2
- 230000009467 reduction Effects 0.000 description 2
- 238000006467 substitution reaction Methods 0.000 description 2
- 238000012546 transfer Methods 0.000 description 2
- 208000009119 Giant Axonal Neuropathy Diseases 0.000 description 1
- 238000004458 analytical method Methods 0.000 description 1
- 238000013473 artificial intelligence Methods 0.000 description 1
- 238000013528 artificial neural network Methods 0.000 description 1
- 238000013500 data storage Methods 0.000 description 1
- 238000013135 deep learning Methods 0.000 description 1
- 238000001514 detection method Methods 0.000 description 1
- 230000007613 environmental effect Effects 0.000 description 1
- 239000000284 extract Substances 0.000 description 1
- 239000000835 fiber Substances 0.000 description 1
- 201000003382 giant axonal neuropathy 1 Diseases 0.000 description 1
- 238000009499 grossing Methods 0.000 description 1
- 230000010354 integration Effects 0.000 description 1
- 238000010295 mobile communication Methods 0.000 description 1
- 230000001537 neural effect Effects 0.000 description 1
- 238000012805 post-processing Methods 0.000 description 1
- 230000011218 segmentation Effects 0.000 description 1
- 239000004065 semiconductor Substances 0.000 description 1
- 239000004984 smart glass Substances 0.000 description 1
- 230000003068 static effect Effects 0.000 description 1
Abstract
An image restoration method, an image restoration device and electronic equipment. After the electronic device obtains the target image, the target image and the corresponding block-level mean value graph can be subjected to difference to obtain a residual value of the target image, and the residual value of the target image and the residual value of the reference image are input into the image restoration network model. The image restoration network model utilizes the residual value of the target image and the residual value of the reference image to migrate the characteristics of the reference image to the target image, and can accurately restore the details and textures of the original resolution of the target image and restore the loss information of the compressed target image, thereby solving the problems of respiratory effect, tailing effect, blocking effect, ring, pattern and the like generated during the compression of the target image.
Description
Technical Field
The present invention relates to the field of image processing technologies, and in particular, to an image restoration method, an image restoration device, and an electronic device.
Background
The image transmission process is required to go through three steps of compression, transmission and decompression. The visual quality of the display after decompression of the image is related to the bandwidth of the transmission. If the communication bandwidth of the image transmission is relatively poor, the decompression scheme of the related art can only improve the originally poor visual quality in a limited way. The related art decompression scheme can provide better visual quality if the communication bandwidth of the image transmission is better. In the decompression scheme of the related art, in the restoration process, the lost information during compression cannot be restored, and the problems of respiratory effect, tailing effect, blocking effect, ring, pattern and the like generated during compression cannot be solved.
Disclosure of Invention
In order to solve the above-mentioned problems, an embodiment of the present application provides an image restoration method, which introduces residual values of an image, so that an image restoration network model accurately restores an encoded image. In addition, the application also provides an image restoration device and electronic equipment corresponding to the image restoration method.
For this reason, the following technical solutions are adopted in the embodiments of the present application:
in a first aspect, an embodiment of the present application provides an image restoration method, including: receiving image information of a target image; the image information comprises decoded image data of the target image and a block-level mean value diagram of the target image; calculating a residual error value of the target image according to the decoded image data of the target image and the block-level mean value graph of the target image; inputting the residual value of the target image and the residual value of the reference image into an image restoration network model to restore the target image; the reference image refers to an image selected as a reference from a series of images or video streams according to a set rule.
In this embodiment, the image restoration method is generally performed by the electronic device. After the electronic device obtains the target image, the target image and the corresponding block-level mean value graph can be subjected to difference to obtain a residual value of the target image, and the residual value of the target image and the residual value of the reference image are input into the image restoration network model. The image restoration network model utilizes the residual value of the target image and the residual value of the reference image to migrate the characteristics of the reference image to the target image, and can accurately restore the details and textures of the original resolution of the target image and restore the loss information of the compressed target image, thereby solving the problems of respiratory effect, tailing effect, blocking effect, ring, pattern and the like generated during the compression of the target image.
In one embodiment, before the receiving the image information of the target image, the method further includes: sending a generation instruction to an encoder; the generation instruction is used for indicating the encoder to generate a block-level mean value graph of the image after receiving the image.
In this embodiment, the electronic device may send a generation instruction to the encoder, so that, in the process of encoding the image by the encoder, a block-level mean value map is generated according to the image at the same time, so that the electronic device may generate a residual value of the image according to the decoded image and the block-level mean value map of the image, to implement restoration of the target image by the image restoration network model according to the residual value of the image.
In one embodiment, before the receiving the image information of the target image, the method further includes: sending a marking instruction to an encoder; the marking instruction is used for instructing the encoder to select the reference image from a series of images or video streams according to a set rule.
In this embodiment, the electronic device may need to select the reference image during the restoration of the image to provide the restored target image with the reference feature. The electronic device may send a marking instruction to the encoder, instructing the encoder to select a reference image from the received images according to a set rule, so as to obtain the reference image from the received images after the electronic device receives a series of images or video streams.
In one embodiment, before the receiving the image information of the target image, the method further includes: and selecting the reference image from the received series of images or video streams according to a set rule.
In this embodiment, after receiving a series of images or video streams, the electronic device may select a reference image from the received images according to a set rule, which may reduce encoder workload.
In one embodiment, before the calculating the residual value of the target image according to the block-level mean value map of the target image and the block-level mean value map of the reference image, the method further includes: detecting whether the target image is a reference image; and updating the residual value of the self-cached reference image under the condition that the target image is the reference image.
In this embodiment, when the electronic device detects that the target image is the reference image, the residual value of the target image may be buffered in its own buffer. If the buffer has buffered the residual values of the images, the buffered residual values of the reference images may be updated, taking the residual value of the target image as the latest residual value of the reference image. The electronic equipment updates the residual error value of the reference image in real time, so that the error generated in the recovery of the target image due to longer time of the reference image and the target image and larger scene change can be avoided.
In one embodiment, the calculating the residual value of the target image according to the decoded image data of the target image and the block-level mean value graph of the target image specifically includes: and carrying out difference between the decoded image data of the target image and the pixel value at the same position in the block-level mean value graph of the target image to obtain a difference value of the pixel value of each pixel block.
In this embodiment, the electronic device may perform difference between the decoded image data of the target salient and the pixel value at the same position in the block-level mean map of the target image to obtain the residual value of the target image, so as to implement restoration of the target image by the image restoration network model according to the residual value of the image.
In one embodiment, the inputting the residual value of the target image and the residual value of the reference image into an image restoration network model, and restoring the target image specifically includes: extracting the characteristics of the residual value of the target image and the residual value of the reference image to obtain the characteristic representation of the residual value of the target image and the characteristic representation of the residual value of the reference image; migrating a characteristic representation of the residual value of the reference image to the residual value of the target image; fusing the characteristic representation of the residual value of the reference image with the characteristic representation of the residual value of the target image to obtain the characteristic representation of the fused target image; and performing reverse recovery on the characteristic representation of the fused target image to restore the target image.
In a second aspect, an embodiment of the present application provides an image restoration apparatus, including: a first processing unit for receiving image information of a target image; the image information comprises decoded image data of the target image and a block-level mean value diagram of the target image; the second processing unit is used for calculating the residual value of the target image according to the decoded image data of the target image and the block-level mean value graph of the target image; the third processing unit is used for inputting the residual value of the target image and the residual value of the reference image into an image restoration network model to restore the target image; the reference image refers to an image selected as a reference from a series of images or video streams according to a set rule.
In one embodiment, the first processing unit is further configured to send a generation instruction to an encoder; the generation instruction is used for indicating the encoder to generate a block-level mean value graph of the image after receiving the image.
In one embodiment, the first processing unit is further configured to send a marking instruction to the encoder; the marking instruction is used for instructing the encoder to select the reference image from a series of images or video streams according to a set rule.
In one embodiment, the first processing unit is further configured to select the reference image from the received series of images or video streams according to a set rule.
In one embodiment, the second processing unit is further configured to detect whether the target image is a reference image; and updating the residual value of the self-cached reference image under the condition that the target image is the reference image.
In one embodiment, the second processing unit is specifically configured to perform difference between the decoded image data of the target image and a pixel value at the same position in the block-level mean map of the target image, so as to obtain a difference value of the pixel value of each pixel block.
In one embodiment, the third processing unit is specifically configured to perform feature extraction on the residual value of the target image and the residual value of the reference image, so as to obtain a feature representation of the residual value of the target image and a feature representation of the residual value of the reference image; migrating a characteristic representation of the residual value of the reference image to the residual value of the target image; fusing the characteristic representation of the residual value of the reference image with the characteristic representation of the residual value of the target image to obtain the characteristic representation of the fused target image; and performing reverse recovery on the characteristic representation of the fused target image to restore the target image.
In a third aspect, an embodiment of the present application provides an electronic device, including: at least one memory; at least one processor configured to execute instructions stored in the memory to cause the electronic device to perform embodiments as possible in the first aspect.
In a fourth aspect, embodiments of the present application provide a computer-readable storage medium comprising computer program instructions which, when executed by an electronic device, perform embodiments as possible in the first aspect.
In a fifth aspect, embodiments of the present application provide a computer program product comprising instructions, wherein the computer program product stores instructions that, when executed by an electronic device, cause the electronic device to implement the various possible embodiments of the first aspect.
Drawings
The drawings that accompany the detailed description can be briefly described as follows.
Fig. 1 is a schematic architecture diagram of an image transmission system according to an embodiment of the present application;
FIG. 2 is a schematic structural diagram of each unit of image restoration network model division provided in the embodiments of the present application;
Fig. 3 is a flowchart of an image restoration method provided in an embodiment of the present application;
fig. 4 is a schematic structural diagram of an image restoration device according to an embodiment of the present application.
Detailed Description
The technical solutions in the embodiments of the present application will be described below with reference to the drawings in the embodiments of the present application.
The term "and/or" herein is an association relationship describing an associated object, and means that there may be three relationships, for example, a and/or B may mean: a exists alone, A and B exist together, and B exists alone. The symbol "/" herein indicates that the associated object is or is a relationship, e.g., A/B indicates A or B.
The terms "first" and "second" and the like in the description and in the claims are used for distinguishing between different objects and not for describing a particular sequential order of objects. For example, the first response message and the second response message, etc. are used to distinguish between different response messages, and are not used to describe a particular order of response messages.
In the embodiments of the present application, words such as "exemplary" or "such as" are used to mean serving as examples, illustrations, or descriptions. Any embodiment or design described herein as "exemplary" or "for example" should not be construed as preferred or advantageous over other embodiments or designs. Rather, the use of words such as "exemplary" or "such as" is intended to present related concepts in a concrete fashion.
In the description of the embodiments of the present application, unless otherwise specified, the meaning of "a plurality of" means two or more, for example, a plurality of processing units means two or more processing units and the like; the plurality of elements means two or more elements and the like.
In order to solve the problems of the decompression scheme in the related art, the embodiment of the application provides an image restoration method, an image transmission system and electronic equipment. After the electronic device obtains the current image, the current image and the block-level mean value image corresponding to the current image can be subjected to difference to obtain a residual value of the current image, and the residual value of the current image and the residual value of the reference image are input into the image restoration network model. The image restoration network model utilizes the residual value of the current image and the residual value of the reference image to migrate the characteristics of the reference image to the current image, can accurately restore the details and textures of the original resolution of the current image and restore the loss information of the compressed current image, thereby solving the problems of respiratory effect, tailing effect, blocking effect, ring, pattern and the like generated during the compression of the current image.
Fig. 1 is a schematic architecture diagram of an image transmission system according to an embodiment of the present application. As shown in fig. 1, the image transmission system 100 includes a transmitting end 110, a transmission line 120, and a receiving end 130. A communication connection is established between the transmitting end 110 and the receiving end 130 through the transmission line 120 to realize transmission of images or videos.
The transmitting end 110 refers to a terminal that generates an image or video, such as a camera, a surveillance camera, a smart phone, a tablet computer, a notebook computer, an unmanned aerial vehicle, a smart watch, smart glasses, etc. After capturing an image or video by the transmitting end 110 through its own sensor, the image or video stream of each frame is converted into a code stream, and then transmitted to the receiving end 130 through the transmission line 120.
As shown in fig. 1, the transmitting end 110 includes a sensor 111, an image signal processor (image signal processor, ISP) 112, and an encoder 113. The sensor 111 is a device for capturing light and converting the light into an electrical signal. The sensor 111 is typically composed of a number of photosensors, each photosensor corresponding to a pixel in the image. ISP 112 is used to process and optimize the raw image captured by sensor 111 to produce a final image quality. The functions of ISP 112 are color correction, white balancing, noise reduction, sharpening, contrast enhancement, etc. ISP 112 may be able to optimize the image in real time based on environmental conditions and camera parameters to improve the sharpness, color recovery, and detail performance of the image. The encoder 113 is a means for compression-encoding the processed image data. The encoder 113 may compress and package the image data into a format that can be transmitted or stored in order to transmit the image without taking up excessive bandwidth and memory. The coding standard of the image can be JPEG, H.264, H.265, etc.
The transmission line 120 may be a wired transmission, such as an electric wire, an optical cable, or the like. The transmission line 120 may be a wireless transmission mode, such as Bluetooth (BT), wireless fidelity (wireless fidelity, WIFI), mobile communication technology, etc. If the transmission line 120 adopts a wireless transmission manner, the transmitting end 110 and the receiving end 130 each include a corresponding communication module, such as a BT module, a WIFI module, a wireless communication module, and the like.
The receiving end 130 refers to a terminal that performs operations such as receiving, storing, processing, displaying, etc. on an image, such as a smart phone, a tablet computer, a notebook computer, a server, a storage system, etc. After receiving the code stream, the receiving end 130 decodes the code stream to restore the original image, and performs operations such as storage, processing, display, and the like on the image.
As shown in fig. 2, the receiving end 130 includes a decoder 131 and a processor 132. The decoder 131 is a component that decodes a code stream into visual contents. The decoder 131 may decode the compressed image or video code stream into original image data or video data. The decoder 131 generally includes a hardware portion responsible for accelerating the decoding process and a software portion that loads the operation of the decoder 131 and the setting of parameters. The processor 132 is a hardware accelerator specifically designed to handle artificial intelligence (artificial intellectual, AI) and deep learning tasks. The processor 132, upon receiving the image data or video data, may perform processing tasks on the image or video, such as image recognition, object detection, semantic segmentation, and the like.
In this embodiment, the encoder 113 may also generate a block-level mean value map (map) of the current image or the image of the current frame while encoding each frame of image or video. A block-level mean graph is a way of representing image information, describing the mean of pixel values of different blocks in an image. Block-level mean graphs are commonly used for image processing and analysis tasks in computer vision, such as image compression, noise reduction, super resolution, image restoration, and the like.
In one embodiment, the process by which encoder 113 generates the block-level mean map is: first, the encoder 113 defines the block size to decide how the image is divided. Second, the encoder 113 divides the image into equal-sized image blocks and then extracts the image blocks using a sliding window or other method. The encoder 113 may then calculate the average of the pixel values for each image block by calculating the average of the pixels. Finally, the encoder 113 maps the mean value of each image block into a new image to represent the mean value information of each image block in the original image.
After obtaining the block-level mean value map of each frame of image, the encoder 113 may transmit the block-level mean value map of the image together with the code stream to the receiving end 130 through the transmission line 120. If the sender 110 transmits a series of images or video streams, the encoder 113 may select a reference image in the video stream. The reference image may be compared with the current image to obtain a residual of the current image. The reference image may be selected in various ways, including but not limited to:
In one embodiment, encoder 113 may select a frame of image as the reference image after 50 frames or other number of frames apart.
In one embodiment, encoder 113 may select a frame of image as the reference image after an interval of 0.1S or other time intervals.
In one embodiment, the processor 132 in the receiving end 130 may send a marking instruction to the encoder 113, instructing the encoder 113 to detect each frame image of the video stream, taking the high quality image as a reference image. If the encoder 113 does not detect a high quality image for a long time, it may select a frame image as a reference image at a set number of frames or at a set time interval after detecting a last high quality image, and it may be avoided that the time for the last high quality image as the reference image is too long, resulting in a large difference between the reference image and the current image.
After determining the reference image, the encoder 113 may add identification information in the process of encoding or generating the block-level mean value map of the reference image, so that after the receiving end 130 receives the code stream or the block-level mean value map of the reference image, the current image may be identified as the reference image according to the identification information. The identification information may be added to the code stream after the reference image is encoded, the block-level average map generated by the reference frame, or a data packet formed by the code stream and the block-level average map, which is not limited herein.
In the embodiment of the present application, the processor 132 may be a chip with a computing function, such as a central processing unit (central processing unit, CPU), an image processor (graphics processing unit, GPU), a neural network processor (neural processing unit, NPU), and the like. When the processor 132 determines that the decoded image needs to be restored, a generation instruction may be transmitted to the encoder 113. The generation instructions may instruct the encoder 113 to generate a block-level mean map from an image after receiving an image or a frame of an image of a video stream, such that the processor 132 may receive the block-level mean map of the image at the same time as the encoded image.
Processor 132 may also send marking instructions to encoder 113. The marking instructions may instruct the encoder 113 to acquire a reference image in a series of images or video streams according to a set rule when the series of images or video streams are received. After receiving the decoded image and the block-level mean map of the image, the processor 132 may detect whether the current image is a reference image. In one case, the processor 132 may buffer the image data of the target image and the block-level mean map of the target image in its own buffer when it is determined that the current image is a reference image. The processor 132 may replace the reference picture in the buffer with a new reference picture if the reference picture is again available during the continued reception of the video stream. In another case, the processor may buffer the current image and the block-level mean map of the current image when it is determined that the current image is not a reference image, so that residual values are subsequently calculated for the current image.
If the encoder 113 does not select a reference image, the processor 132 may select a reference image for the received series of images or video streams. In the embodiment of the present application, the method of selecting the reference image by the processor 132 is substantially the same as the method of selecting the reference image by the encoder 113. For example, the processor 132 may select one frame image as the reference image after each interval setting number of frames or interval setting time in the time sequence of receiving images. As another example, the processor 132 may detect the pixel quality of each frame of image and take the high quality image as a reference image when receiving the image. As well as other ways.
After receiving the decoded image and the block-level mean map of the image of each frame transmitted from the decoder 131, the processor 132 may calculate a residual value of the image of each frame according to the decoded image and the block-level mean map of the image. In one embodiment, the process of calculating the residual value by the processor 132 may be: the processor 132 may difference the pixel values at the same position in the decoded image and the block-level mean map of the image to obtain a difference value of the pixel values of each pixel block, that is, a residual value of the image. Alternatively, after the processor 132 determines the reference image, the residual values of the reference image may be cached in its own memory.
After obtaining the residual value of the current image, the processor 132 may input the residual value of the current image and the residual value of the reference image currently cached by itself into the image restoration network model to output the restored current image. The image restoration network model is a model running a multi-frame super-resolution algorithm, a multi-frame same-resolution restoration algorithm or other algorithms, can utilize the residual value of the current image and the residual value of the reference image to perform feature extraction, and transfer the features of the reference image to the current image, namely, fusing high-quality image information in the reference image to the decoded current image to restore the current image.
Taking an example of the image restoration network model running multi-frame same-resolution restoration/super-division algorithm. The multi-frame same-resolution restoration/super-resolution algorithm is an image processing technology, and utilizes information of a plurality of same-resolution/low-resolution images to improve the resolution and quality of the images. As shown in fig. 2, the image restoration network model 200 may be divided into a feature extraction unit 210, an information dissemination unit 220, an information fusion unit 230, and an information restoration unit 240 according to the execution function. The process of the image restoration network model 200 restoring the current image is as follows:
The feature extraction unit 210 performs feature extraction on the residual value of the current image and the residual value of the reference image after receiving the residual value of the current image and the residual value of the reference image, so as to obtain a feature representation of the residual value of the current image and a feature representation of the residual value of the reference image. Each feature representation may be seen as an abstract representation of some visual feature in the original image, such as an edge, texture, shape, or higher level of semantic refinement. Specifically, the residual value of an image is generally composed of the residual of each pixel value of the image, and feature extraction can be performed using a residual learning technique. The feature extraction unit 210 may perform feature extraction on the residual values of the image through the convolution and pooling layers of the plurality of layers to capture local features and global features of the image.
After obtaining the image representation of the residual value of the current image and the image representation of the residual value of the reference image, the information propagation unit 220 may spatially align the residual map corresponding to the residual value of the current image with the residual map corresponding to the residual value of the reference image, then migrate the feature representation of the residual value of the current image to the residual map corresponding to the residual value of the reference image, and migrate the feature representation of the residual value of the reference image to the residual map corresponding to the residual value of the current image, so that the residual map corresponding to the residual value of each image contains the feature representations of the residual values of all images, so that the image contains information of other images when being restored, to restore the information lost after self compression. In this embodiment of the present application, since the image restoration network model 200 only needs to output the restored image of the current image, the information propagation unit 220 may migrate the feature representation of the residual value of the reference image to the residual map corresponding to the residual value of the current image, so that the residual map corresponding to the residual value of the current image includes the feature representation of the residual value of the reference image.
The information fusion unit 230 may fuse the feature representation of the residual value of the other transferred image, and then fuse the feature representation of the residual value after fusion with the feature representation of the residual value of the image of the user, so as to obtain the feature of the image after fusion, thereby fusing the information of the other image into the image of the user, so that the information of the other image is fused when the image is restored, and the lost information of the user is restored. In this embodiment of the present application, since only the residual value of the current image and the residual value of the reference image are input, the information fusion unit 230 may directly fuse the feature representation of the residual value of the migrated reference image with the feature representation of the residual value of the current image, so as to fuse the feature representation of the reference image into the current image, so that the current image includes the feature representation of the reference image, so as to recover the information lost by the compression of the current image.
The information restoring unit 240 is configured to reconstruct an image of the fused image-specific representation to restore an original image. Since the feature representation of the fused image includes information of a plurality of images, the information restoring unit 240 may directionally propagate the feature representation of the fused image to the image space through operations such as deconvolution or inverse pooling, or using a generation countermeasure network (generative adversarial networks, GANs) or other inverse restoring methods, and restore the spatial information in the feature map to the original image through inverse operations to restore the current image.
After the current image is restored by the processor 132, the restored current image may be post-processed and additional processing and modifications may be performed on the restored current image to improve image quality, enhance details, change image style, or meet specific requirements. Post-processing operations may include sharpening and deblurring, adjusting brightness, adjusting contrast, denoising and smoothing, color correction, color enhancement, image restoration and stitching, and the like.
The final output current image of the image restoration network model 200 can accurately restore the details and textures of the original resolution and restore the loss information of the compressed current image, so as to solve the problems of respiratory effect, tailing effect, blocking effect, ring, pattern and the like generated during compression. Moreover, the image restoration network model 200 can restore the still region information of video compression loss, and effectively solve the problem that the reference image part information cannot be fully migrated to the current image due to color, illumination and other conditions.
Fig. 3 is a flowchart of an image restoration method provided in an embodiment of the present application. As shown in fig. 3, the image restoration method may be executed by the processor 132 of the receiving end 130, and specifically implemented as follows:
S301, the processor 132 receives image information of a target image. The image information includes decoded image data of the target image and a block-level mean map of the target image.
The target image may be one of a series of images received by the processor 132 or may be one frame of an image in a received video stream. The image data of the target image is decoded image data. If the image data received by the processor 132 is undecoded image data, a decoder may be invoked to decode the target image to obtain decoded image data. The block-level mean value map of the target image refers to a mean value of pixel values of each image block calculated after the target image is divided into a plurality of image blocks.
The block-level mean value map of the target image is generally generated at the target image encoding end. In an embodiment of the present application, the processor 132 may send a generation instruction to the encoder 113. The generation instructions may instruct the encoder 113 to generate a block-level mean value map from an image after receiving an image or a frame of an image of a video stream, and send the block-level mean value map of the image and the code stream to the processor 132 together, so that the processor 132 may receive the block-level mean value map of the image while receiving the encoded image.
S302, the processor 132 calculates a residual value of the target image according to the decoded image data of the target image and the block-level mean map of the target image.
The reference image refers to an image selected as a reference among a series of images or video streams according to a set rule. In an embodiment of the present application, the processor 132 may send a marking instruction to the encoder 113. The marking instructions may instruct the encoder 113 to acquire a reference image in a series of images or video streams according to a set rule when the series of images or video streams are received. There are various ways of selecting the reference image, for example, selecting one image as the reference image according to the time sequence of generating the images, the number of the intervals or the interval setting time, or selecting a high-quality image as the reference image.
Alternatively, when the processor 132 determines that the encoder 113 does not select a reference image, the processor 132 may select a reference image for the received series of images or video streams. The processor 132 selects the reference picture in substantially the same manner as the encoder 113 selects the reference picture. For example, the processor 132 may select one frame image as the reference image after each interval setting number of frames or interval setting time in the time sequence of receiving images. As another example, the processor 132 may detect the pixel quality of each frame of image and take the high quality image as a reference image when receiving the image.
After receiving the decoded image and the block-level mean map of the image of each frame transmitted from the decoder 131, the processor 132 may calculate a residual value of the image of each frame according to the decoded image and the block-level mean map of the image. The processor 132 adds the residual value of the target image in the process of restoring the target image, so that the restoring algorithm is more robust to the changes of external conditions such as color, illumination, brightness and the like, and the original image can be accurately restored. When the processor 132 detects that the target image is a reference image, the residual value of the target image may be buffered in its own buffer. If the buffer has buffered the residual values of the images, the buffered residual values of the reference images may be updated, taking the residual value of the target image as the latest residual value of the reference image.
S303, the processor 132 inputs the residual value of the target image and the residual value of the reference image into the image restoration network model, and restores the target image.
After obtaining the residual value of the target image, the processor 132 may input the residual value of the target image and the residual value of the reference image cached by the target itself into the image restoration network model to output the restored target image.
Taking an example of the image restoration network model running multi-frame same-resolution restoration/super-division algorithm. After receiving the residual value of the target image and the residual value of the reference image, the image restoration network model performs feature extraction on the residual value of the target image and the residual value of the reference image to obtain a feature representation of the residual value of the target image and a feature representation of the residual value of the reference image. After obtaining the image representation of the residual value of the target image and the image representation of the residual value of the reference image, the image restoration network model may spatially align the residual map corresponding to the residual value of the target image with the residual map corresponding to the residual value of the reference image. Because the image restoration network model only needs to output the restored image of the target image, the image restoration network model can transfer the characteristic representation of the residual value of the reference image to the residual image corresponding to the residual value of the target image, so that the residual image corresponding to the residual value of the target image contains the characteristic representation of the residual value of the reference image, and the information lost by compression of the target image is restored. The image restoration network model can directly fuse the characteristic representation of the residual value of the migrated reference image with the characteristic representation of the residual value of the target image, so that the characteristic representation of the reference image is fused into the target image, and the target image contains the characteristic representation of the reference image, thereby restoring the information of compression loss of the target image. Because the characteristic representation of the fused image comprises information of a plurality of images, the image restoration network model propagates the characteristic representation direction of the fused image to an image space, and the spatial information in the characteristic image is restored to the original image through the inverse operation so as to restore the target image.
After the processor 132 restores the target image, the restored target image may be post-processed and additional processing and modifications may be performed on the restored target image to improve image quality, enhance details, change image style, or meet specific requirements.
In this embodiment of the present application, the processor 132 may input the residual value of the target image and the difference value of the reference image to the image restoration network model, where the target image finally output by the image restoration network model may accurately restore details and textures of the original resolution, and restore loss information after the target image is compressed, so as to solve the problems of respiratory effect, tailing effect, blocking effect, ring, pattern, and the like generated during compression. In addition, the image restoration network model can restore the static region information of video compression loss, and effectively solves the problem that the partial information of the reference image cannot be fully migrated to the target image due to the conditions of color, illumination and the like.
Fig. 4 is a schematic structural diagram of an image restoration device according to an embodiment of the present application. As shown in fig. 4, the image restoration apparatus 400 includes a first processing unit 410, a second processing unit 420, and a third processing unit 430. The image restoration apparatus 400 specifically performs the following functions:
The first processing unit 410 is configured to receive image information of a target image. The image information includes decoded image data of the target image and a block-level mean map of the target image. The second processing unit 420 is configured to calculate a residual value of the target image according to the decoded image data of the target image and the block-level mean map of the target image. The third processing unit 430 is configured to input the residual value of the target image and the residual value of the reference image into the image restoration network model, and restore the target image. The reference image refers to an image selected as a reference among a series of images or video streams according to a set rule.
In one embodiment, the first processing unit 410 is further configured to send a generation instruction to the encoder. The generation instructions are used for instructing the encoder to generate a block-level mean value graph of the image after the image is received.
In one embodiment, the first processing unit 410 is further configured to send a marking instruction to the encoder. The marking instructions are used to instruct the encoder to select the reference image in a series of images or video streams according to a set rule.
In one embodiment, the first processing unit 410 is further configured to select a reference image from the received series of images or video streams according to a set rule.
In one embodiment, the second processing unit 420 is further configured to detect whether the target image is a reference image. The second processing unit 420 is further configured to update a residual value of the reference image that has been cached by itself, in the case that the target image is the reference image.
In one embodiment, the second processing unit 420 is specifically configured to perform a difference between the decoded image data of the target image and the pixel value at the same position in the block-level mean map of the target image, to obtain a difference between the pixel values of each pixel block.
In one embodiment, the third processing unit 430 is specifically configured to perform feature extraction on the residual value of the target image and the residual value of the reference image, so as to obtain a feature representation of the residual value of the target image and a feature representation of the residual value of the reference image. The third processing unit 430 is specifically configured to migrate the characteristic representation of the residual value of the reference image to the residual value of the target image. The third processing unit 430 is specifically configured to fuse the feature representation of the residual value of the reference image with the feature representation of the residual value of the target image, so as to obtain a feature representation of the target image after fusion. The third processing unit 430 is specifically configured to perform reverse recovery on the feature representation of the fused target image, and restore the target image.
The embodiment of the application also provides electronic equipment, which comprises a processor, wherein the processor can execute the technical scheme of corresponding protection as shown in fig. 1-3, so that the electronic equipment has the technical effect of the technical scheme of protection.
There is also provided in an embodiment of the present application a computer readable storage medium comprising computer program instructions which, when executed by an electronic device, perform any of the methods recited in fig. 1-3 and corresponding descriptions above.
There is further provided in an embodiment of the present application a computer program product comprising instructions, characterized in that the computer program product stores instructions that, when executed by an electronic device, cause the electronic device to implement any of the methods described in the above figures 1-3 and corresponding descriptions.
Those of ordinary skill in the art will appreciate that the various illustrative elements and algorithm steps described in connection with the embodiments disclosed herein may be implemented as electronic hardware, or combinations of computer software and electronic hardware. Whether such functionality is implemented as hardware or software depends upon the particular application and design constraints imposed on the solution. Skilled artisans may implement the described functionality in varying ways for each particular application, but such implementation decisions should not be interpreted as causing a departure from the scope of the embodiments of the present application.
Furthermore, various aspects or features of embodiments of the present application may be implemented as a method, apparatus, or article of manufacture using standard programming and/or engineering techniques. The term "article of manufacture" as used herein encompasses a computer program accessible from any computer-readable device, carrier, or media. For example, computer-readable media may include, but are not limited to: magnetic storage devices (e.g., hard disk, floppy disk, or magnetic tape, etc.), optical disks (e.g., compact Disk (CD), digital versatile disk (digital versatiledisc, DVD), etc.), smart cards, and flash memory devices (e.g., erasable programmable read-only memory (EPROM), cards, sticks, key drives, etc.). Additionally, various storage media described herein can represent one or more devices and/or other machine-readable media for storing information. The term "machine-readable medium" can include, without being limited to, wireless channels and various other media capable of storing, containing, and/or carrying instruction(s) and/or data.
In the above-described embodiments, the image restoration apparatus 400 in fig. 4 may be implemented in whole or in part by software, hardware, firmware, or any combination thereof. When implemented in software, may be implemented in whole or in part in the form of a computer program product. The computer program product includes one or more computer instructions. When loaded and executed on a computer, produces a flow or function in accordance with embodiments of the present application, in whole or in part. The computer may be a general purpose computer, a special purpose computer, a computer network, or other programmable apparatus. The computer instructions may be stored in a computer-readable storage medium or transmitted from one computer-readable storage medium to another computer-readable storage medium, for example, the computer instructions may be transmitted from one website, computer, server, or data center to another website, computer, server, or data center by a wired (e.g., coaxial cable, fiber optic, digital subscriber line), or wireless (e.g., infrared, wireless, microwave, etc.). The computer readable storage medium may be any available medium that can be accessed by a computer or a data storage device such as a server, data center, etc. that contains an integration of one or more available media. The usable medium may be a magnetic medium (e.g., floppy disk, hard disk, magnetic tape), an optical medium (e.g., DVD), or a semiconductor medium (e.g., SSD), etc.
It should be understood that, in various embodiments of the present application, the sequence numbers of the foregoing processes do not mean the order of execution, and the order of execution of the processes should be determined by the functions and internal logic of the processes, and should not constitute any limitation on the implementation process of the embodiments of the present application.
It will be clear to those skilled in the art that, for convenience and brevity of description, specific working procedures of the above-described systems, apparatuses and units may refer to corresponding procedures in the foregoing method embodiments, and are not repeated herein.
In the several embodiments provided in this application, it should be understood that the disclosed systems, devices, and methods may be implemented in other manners. For example, the apparatus embodiments described above are merely illustrative, e.g., the division of the units is merely a logical function division, and there may be additional divisions when actually implemented, e.g., multiple units or components may be combined or integrated into another system, or some features may be omitted or not performed. Alternatively, the coupling or direct coupling or communication connection shown or discussed with each other may be an indirect coupling or communication connection via some interfaces, devices or units, which may be in electrical, mechanical or other form.
The units described as separate units may or may not be physically separate, and units shown as units may or may not be physical units, may be located in one place, or may be distributed on a plurality of network units. Some or all of the units may be selected according to actual needs to achieve the purpose of the solution of this embodiment.
The functions, if implemented in the form of software functional units and sold or used as a stand-alone product, may be stored in a computer-readable storage medium. Based on such understanding, the technical solution of the embodiments of the present application may be essentially or, what contributes to the prior art, or part of the technical solution may be embodied in the form of a software product stored in a storage medium, comprising several instructions for causing a computer device (which may be a personal computer, a server, or an access network device, etc.) to perform all or part of the steps of the methods described in the embodiments of the present application. And the aforementioned storage medium includes: a U-disk, a removable hard disk, a read-only memory (ROM), a random access memory (random access memory, RAM), a magnetic disk, or an optical disk, or other various media capable of storing program codes.
The foregoing is merely a specific implementation of the embodiments of the present application, but the protection scope of the embodiments of the present application is not limited thereto, and any person skilled in the art may easily think about changes or substitutions within the technical scope of the embodiments of the present application, and all changes and substitutions are included in the protection scope of the embodiments of the present application.
Claims (17)
1. An image restoration method, comprising:
receiving image information of a target image; the image information comprises decoded image data of the target image and a block-level mean value diagram of the target image;
calculating a residual error value of the target image according to the decoded image data of the target image and the block-level mean value graph of the target image;
inputting the residual value of the target image and the residual value of the reference image into an image restoration network model to restore the target image; the reference image refers to an image selected as a reference from a series of images or video streams according to a set rule.
2. The method of claim 1, wherein prior to said receiving image information of the target image, the method further comprises:
sending a generation instruction to an encoder; the generation instruction is used for indicating the encoder to generate a block-level mean value graph of the image after receiving the image.
3. The method according to claim 1 or 2, characterized in that before the receiving of the image information of the target image, the method further comprises:
sending a marking instruction to an encoder; the marking instruction is used for instructing the encoder to select the reference image from a series of images or video streams according to a set rule.
4. The method according to claim 1 or 2, characterized in that before the receiving of the image information of the target image, the method further comprises:
and selecting the reference image from the received series of images or video streams according to a set rule.
5. The method according to any one of claims 1-4, wherein before said calculating a residual value of said target image from a block-level mean map of said target image and a block-level mean map of a reference image, said method further comprises:
detecting whether the target image is a reference image;
and updating the residual value of the self-cached reference image under the condition that the target image is the reference image.
6. The method according to any one of claims 1-5, wherein the calculating the residual value of the target image based on the decoded image data of the target image and the block-level mean map of the target image specifically comprises:
And carrying out difference between the decoded image data of the target image and the pixel value at the same position in the block-level mean value graph of the target image to obtain a difference value of the pixel value of each pixel block.
7. The method according to any one of claims 1-6, wherein the inputting the residual value of the target image and the residual value of the reference image into the image restoration network model, restoring the target image, specifically comprises:
extracting the characteristics of the residual value of the target image and the residual value of the reference image to obtain the characteristic representation of the residual value of the target image and the characteristic representation of the residual value of the reference image;
migrating a characteristic representation of the residual value of the reference image to the residual value of the target image;
fusing the characteristic representation of the residual value of the reference image with the characteristic representation of the residual value of the target image to obtain the characteristic representation of the fused target image;
and performing reverse recovery on the characteristic representation of the fused target image to restore the target image.
8. An image restoration apparatus, comprising:
a first processing unit for receiving image information of a target image; the image information comprises decoded image data of the target image and a block-level mean value diagram of the target image;
The second processing unit is used for calculating the residual value of the target image according to the decoded image data of the target image and the block-level mean value graph of the target image;
the third processing unit is used for inputting the residual value of the target image and the residual value of the reference image into an image restoration network model to restore the target image; the reference image refers to an image selected as a reference from a series of images or video streams according to a set rule.
9. The apparatus of claim 8, wherein the first processing unit is further configured to
Sending a generation instruction to an encoder; the generation instruction is used for indicating the encoder to generate a block-level mean value graph of the image after receiving the image.
10. The apparatus according to claim 8 or 9, wherein the first processing unit is further configured to
Sending a marking instruction to an encoder; the marking instruction is used for instructing the encoder to select the reference image from a series of images or video streams according to a set rule.
11. The apparatus according to claim 8 or 9, wherein the first processing unit is further configured to
And selecting the reference image from the received series of images or video streams according to a set rule.
12. The apparatus according to any of the claims 7-11, wherein the second processing unit is further adapted to
Detecting whether the target image is a reference image;
and updating the residual value of the self-cached reference image under the condition that the target image is the reference image.
13. The device according to any of the claims 7-12, wherein the second processing unit is in particular adapted to
And carrying out difference between the decoded image data of the target image and the pixel value at the same position in the block-level mean value graph of the target image to obtain a difference value of the pixel value of each pixel block.
14. The device according to any of the claims 7-13, characterized in that the third processing unit is in particular adapted to
Extracting the characteristics of the residual value of the target image and the residual value of the reference image to obtain the characteristic representation of the residual value of the target image and the characteristic representation of the residual value of the reference image;
migrating a characteristic representation of the residual value of the reference image to the residual value of the target image;
fusing the characteristic representation of the residual value of the reference image with the characteristic representation of the residual value of the target image to obtain the characteristic representation of the fused target image;
And performing reverse recovery on the characteristic representation of the fused target image to restore the target image.
15. An electronic device, comprising:
at least one memory;
at least one processor configured to execute instructions stored in a memory to cause the electronic device to perform the method of any one of claims 1-7.
16. A computer readable storage medium comprising computer program instructions which, when executed by an electronic device, perform the method of any of claims 1-7.
17. A computer program product comprising instructions, characterized in that the computer program product stores instructions that, when executed by an electronic device, cause the electronic device to implement the method of any of claims 1-7.
Priority Applications (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311600320.6A CN117764834A (en) | 2023-11-24 | 2023-11-24 | Image restoration method and device and electronic equipment |
Applications Claiming Priority (1)
| Application Number | Priority Date | Filing Date | Title |
|---|---|---|---|
| CN202311600320.6A CN117764834A (en) | 2023-11-24 | 2023-11-24 | Image restoration method and device and electronic equipment |
Publications (1)
| Publication Number | Publication Date |
|---|---|
| CN117764834A true CN117764834A (en) | 2024-03-26 |
Family
ID=90311377
Family Applications (1)
| Application Number | Title | Priority Date | Filing Date |
|---|---|---|---|
| CN202311600320.6A Pending CN117764834A (en) | 2023-11-24 | 2023-11-24 | Image restoration method and device and electronic equipment |
Country Status (1)
| Country | Link |
|---|---|
| CN (1) | CN117764834A (en) |
-
2023
- 2023-11-24 CN CN202311600320.6A patent/CN117764834A/en active Pending
Similar Documents
| Publication | Publication Date | Title |
|---|---|---|
| CN111681167B (en) | Image quality adjusting method and device, storage medium and electronic equipment | |
| CN112543347B (en) | Video super-resolution method, device, system and medium based on machine vision coding and decoding | |
| CN111598776B (en) | Image processing method, image processing device, storage medium and electronic apparatus | |
| WO2018150083A1 (en) | A method and technical equipment for video processing | |
| WO2016101883A1 (en) | Method for face beautification in real-time video and electronic equipment | |
| CN114339260A (en) | Image processing method and device | |
| WO2023005740A1 (en) | Image encoding, decoding, reconstruction, and analysis methods, system, and electronic device | |
| CN111738951A (en) | Image processing method and device | |
| CN113628116B (en) | Training method and device for image processing network, computer equipment and storage medium | |
| US20220335560A1 (en) | Watermark-Based Image Reconstruction | |
| CN113409203A (en) | Image blurring degree determining method, data set constructing method and deblurring method | |
| CN112714263A (en) | Video generation method, device, equipment and storage medium | |
| CN116847087A (en) | Video processing method and device, storage medium and electronic equipment | |
| CN114938408B (en) | Data transmission method, system, equipment and medium of cloud mobile phone | |
| WO2023124461A1 (en) | Video coding/decoding method and apparatus for machine vision task, device, and medium | |
| CN117764834A (en) | Image restoration method and device and electronic equipment | |
| CN113658073A (en) | Image denoising processing method and device, storage medium and electronic equipment | |
| CN113613024A (en) | Video preprocessing method and device | |
| US20230100728A1 (en) | A system, an arrangement, a computer software module arrangement, a circuitry arrangement and a method for improved image processing utilzing two entities | |
| CN113781336B (en) | Image processing method, device, electronic equipment and storage medium | |
| US20240095966A1 (en) | Coding of displacements by use of contexts for vertex mesh (v-mesh) | |
| WO2020181540A1 (en) | Video processing method and device, encoding apparatus, and decoding apparatus | |
| CN117333404A (en) | Rendering method and rendering engine | |
| CN115965616A (en) | Iris image processing method and device and electronic equipment | |
| CN113592009A (en) | Image semantic segmentation method and device, storage medium and electronic equipment |
Legal Events
| Date | Code | Title | Description |
|---|---|---|---|
| PB01 | Publication | ||
| PB01 | Publication | ||
| SE01 | Entry into force of request for substantive examination | ||
| SE01 | Entry into force of request for substantive examination |