WO2020187220A1 - 一种图像超分辨重建方法、装置和终端设备 - Google Patents

一种图像超分辨重建方法、装置和终端设备 Download PDF

Info

Publication number
WO2020187220A1
WO2020187220A1 PCT/CN2020/079880 CN2020079880W WO2020187220A1 WO 2020187220 A1 WO2020187220 A1 WO 2020187220A1 CN 2020079880 W CN2020079880 W CN 2020079880W WO 2020187220 A1 WO2020187220 A1 WO 2020187220A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
images
raw images
format
neural network
Prior art date
Application number
PCT/CN2020/079880
Other languages
English (en)
French (fr)
Inventor
戴俊
张一帆
王银廷
Original Assignee
华为技术有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 华为技术有限公司 filed Critical 华为技术有限公司
Publication of WO2020187220A1 publication Critical patent/WO2020187220A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Definitions

  • This application relates to the field of image processing technology, and in particular to an image super-resolution reconstruction method, device and terminal equipment.
  • Smart phones, tablet PCs and other portable terminal devices that have the characteristics of being thin and light and are equipped with fixed-focus lenses. Users can adjust the composition of the photo by zooming.
  • the fixed-focus lens achieves zooming
  • digital zoom technology is used to zoom in and display the subject, but the image is clear
  • the captured image is often unclear, which cannot meet the requirements of users.
  • the image super-resolution reconstruction method can be used.
  • the image super-resolution reconstruction method is based on signal processing technology to reconstruct a high-resolution image from a single or multiple low-resolution images.
  • the image is restored during the acquisition process.
  • the high-frequency detail information lost in the video can achieve the purpose of improving image resolution.
  • the final image has a greater improvement in resolution than the original image, which can provide more detailed information, and the final image is closer to the ideal image.
  • Google has proposed a method of image super-resolution reconstruction, which uses multi-frame registration to demosaicing and uses displacement between frames to fill in missing pixels. Specifically, by sequentially moving the image to the right, down, and left by one pixel, the color image can be completely reconstructed. In this method, the pixel values are recovered from the real image, rather than obtained through interpolation, thereby improving the resolution of the image. It should be noted that this method provided by Google will theoretically increase the resolution of the image, but it is not ideal in actual use, because this method requires a one-pixel offset of the image. In actual use, the image is difficult to accurately The offset of a single pixel is realized, so the image obtained by this method for image super-resolution reconstruction is not ideal.
  • the embodiments of the present application provide an image super-resolution reconstruction method, device, and terminal equipment, which can make the reconstructed image contain more detailed information and be clearer.
  • an embodiment of the present application provides an image super-resolution reconstruction method, including: acquiring N original raw images through a camera, where N is 1 or an integer greater than 1, and performing preliminary operations based on the N raw images , Obtain a first image, the first image is a color three-channel image; input the N raw images into the neural network to obtain the first detailed information output by the neural network; superimpose the first detailed information on For the first image, a super-resolution image corresponding to the N raw images is obtained.
  • the first detail information may be a data set representing image details, or an image representing image details.
  • the user can trigger a photographing instruction by clicking the photographing button on the display screen. After acquiring the photographing instruction, the camera can acquire N raw images. It should be noted that when previewing the shot scene through the display interface before taking a photo, the user can perform a zoom operation. For example, the distance between the two fingers touching the display screen can be increased to increase the zoom magnification; The distance between the two fingers touching the screen becomes smaller, and the zoom ratio can be reduced.
  • acquiring N raw images through a camera may include: cropping the N images acquired by the camera according to a zoom magnification corresponding to a photographing instruction to obtain N raw images corresponding to the zoom magnification.
  • the preliminary operation may include: performing an interpolation operation on any one of the N images to be processed; or, performing an interpolation operation on the first denoised image, where the first denoised image is the to be processed N images are obtained by multi-frame noise reduction.
  • the neural network may take N simulated raw images as input, and use the neural network to be trained to obtain the super-resolution images and high-definition images corresponding to the N simulated raw images.
  • the difference is the loss function, which is obtained by training the neural network to be trained;
  • the N simulated raw images are images generated by degrading the high-definition images and in accordance with the raw image data format;
  • the N images The super-resolution image corresponding to the simulated raw image is the superposition of the second detail information corresponding to the N simulated raw images output by the neural network to be trained and the second image obtained from the N simulated raw images
  • the second image is an image obtained through the preliminary operation according to the N simulated raw images.
  • the format of the N raw images is Bayer Bayer format or Kudro quadra format; if the format of the N raw images is quadra format, the N raw images are passed Preliminary operations to obtain the first image include: converting the N raw images into N images in bayer format through binning processing, and performing the preliminary operation on the N images converted into bayer format to obtain the first image . If the format of the N raw images is a quadra format, the N simulated raw images are images in a quadra format; the second image is the binning process to convert the N simulated raw images into a bayer format, An image obtained by performing the preliminary operation on the N simulated raw images in the bayer format after conversion.
  • an embodiment of the present application provides an image super-resolution reconstruction device, including: an acquisition unit configured to acquire N original raw images through a camera, where N is 1 or an integer greater than 1, and a first processing unit, It is used to obtain a first image through preliminary operations according to the N raw images, where the first image is a color three-channel image.
  • the second processing unit is used to input the N raw images into the neural network to obtain the first detailed information output by the neural network; the superimposing unit is used to superimpose the first detailed information on the first image, Obtain super-resolution images corresponding to the N raw images.
  • the neural network uses N simulated raw images as input, and the difference between the super-resolution images and the high-definition images corresponding to the N simulated raw images obtained by the neural network to be trained Is the loss function, obtained by training the neural network to be trained; the N simulated raw images are images generated by degrading the high-definition images and in accordance with the raw image data format; the N simulated raw images The super-resolution image corresponding to the raw image is the superposition of the second detail information corresponding to the N simulated raw images output by the neural network to be trained and the second image obtained according to the N simulated raw images, The second image is an image obtained through the preliminary operation according to the N simulated raw images.
  • the preliminary operation includes: performing an interpolation operation on any one of the N images to be processed; or, performing an interpolation operation on the first denoised image, and the first denoised image is the The processed N images are obtained by multi-frame noise reduction.
  • the format of the N raw images is Bayer Bayer format or Kudro quadra format; if the format of the N raw images is quadra format, the first processing unit is in accordance with the N raw images are obtained through preliminary operations in terms of the first image, which is used to: convert the N raw images into N images in the bayer format through binning processing, and perform the conversion on the N images converted into the bayer format The first image is obtained by the preliminary operation; if the format of the N raw images is quadra format, the N simulated raw images are quadra format images; the second image is the raw simulated raw images The image is converted into a bayer format through binning processing, and the image obtained by performing the preliminary operation on the N simulated raw images in the bayer format after the conversion.
  • the acquiring unit is specifically configured to crop the N images acquired by the camera according to the zoom magnification corresponding to the photographing instruction to obtain N raw images corresponding to the zoom magnification.
  • an embodiment of the present application provides a terminal device, including a camera, a processor, and a memory.
  • the camera is configured to obtain N original raw images after the processor obtains a photographing instruction, where N is 1 or an integer greater than 1;
  • the memory is used to store a computer program that can be run on the processor;
  • the processor is used to execute the first aspect or any possible embodiment of the first aspect Part or all of the steps of the described method.
  • embodiments of the present application provide a computer-readable storage medium that stores instructions and a computer program corresponding to a neural network.
  • the instructions When the instructions are run on a terminal device, the The terminal device executes the method according to any one of claims 1 to 5.
  • an embodiment of the present application provides a terminal device, including a camera, a processor, and a neural network unit; the camera is used to obtain N original raw images after the processor obtains a photographing instruction. Is 1 or an integer greater than 1; the neural network unit is configured to use the N raw images as input to obtain first detailed information; the processor is configured to execute any one of the first aspect or the first aspect Some or all of the steps of the method described in the possible embodiments.
  • an embodiment of the present application provides a computer-readable storage medium that stores a computer program, and the computer program includes program instructions that, when executed by a processor, cause The processor executes part or all of the steps of the method described in the first aspect or any possible embodiment of the first aspect.
  • the embodiments of the present application provide a computer program product
  • the computer program product includes a computer-readable storage medium storing a computer program
  • the computer program enables a computer to execute the first aspect or any one of the first aspect Part or all of the steps of the method described in the embodiment.
  • FIG. 1 is a schematic flowchart of an image super-resolution reconstruction method provided by an embodiment of the application.
  • FIG. 2 is a schematic structural diagram of an image super-resolution reconstruction apparatus provided by another embodiment of the application.
  • FIG. 3A is a schematic structural diagram of a terminal device provided by another embodiment of this application.
  • FIG. 3B is a schematic structural diagram of a terminal device provided by another embodiment of this application.
  • FIG. 4 is a schematic structural diagram of a terminal device provided by another embodiment of this application.
  • FIG. 5A is a low-resolution image obtained by a super-resolution processing unit in an embodiment of the application.
  • 5B is a detailed schematic diagram corresponding to the red channel in the detail image corresponding to FIG. 5A.
  • FIG. 5C is a detailed schematic diagram corresponding to the green channel in the detail image corresponding to FIG. 5A.
  • FIG. 5D is a detailed schematic diagram corresponding to the blue channel in the detail image corresponding to FIG. 5A.
  • Figure 5E is a high-resolution image obtained after image super-resolution reconstruction in this embodiment.
  • Fig. 6A is a partial schematic diagram of an original image in bayer format in an embodiment of the present application.
  • FIG. 6B is a partial schematic diagram of the original image in the quadra format in an embodiment of the present application.
  • FIG. 6C is a partial schematic diagram of converting the partial schematic diagram of the quadra format shown in FIG. 6B into the bayer format.
  • the image super-resolution reconstruction method includes: acquiring N original raw images through a camera, where N is 1 or an integer greater than 1, and obtaining the first image through preliminary operations based on the N raw images,
  • the first image is a color three-channel image;
  • the N raw images are input to a neural network to obtain first detailed information output by the neural network;
  • the first detailed information is superimposed on the first image, Obtain super-resolution images corresponding to the N raw images.
  • it before acquiring N raw images, it may further include acquiring a photographing instruction, cropping the image acquired by the camera according to the zoom magnification corresponding to the photographing instruction, and the cropped image corresponding to the zoom magnification is used as the raw image.
  • the terminal device may be a portable device with light and thin characteristics and a fixed-focus lens installed, such as a mobile phone with a camera function (or called a "cellular" phone), a smart phone, a portable wearable device (such as Smart watches, etc.), tablet computers, personal computers (PC, Personal Computer), PDAs (Personal Digital Assistant, personal digital assistants), etc.
  • a mobile phone with a camera function or called a "cellular" phone
  • smart phone such as Smart watches, etc.
  • a portable wearable device such as Smart watches, etc.
  • tablet computers personal computers (PC, Personal Computer), PDAs (Personal Digital Assistant, personal digital assistants), etc.
  • PC Personal Computer
  • PDAs Personal Digital Assistant, personal digital assistants
  • FIG. 1 is a schematic flowchart of an image super-resolution reconstruction method provided by an embodiment of the application.
  • the method executed by the processor may include the following steps:
  • Step 101 After acquiring the photographing instruction, crop N images acquired by the camera according to the zoom magnification corresponding to the photographing instruction to obtain N raw images corresponding to the zoom magnification, where N is 1 or an integer greater than 1.
  • the camera instruction can be triggered by the user clicking the camera button on the display screen.
  • the user Before taking a photo, the user can perform a zoom operation when previewing the shot scene through the display interface, for example, the two contacts that are in contact with the display screen.
  • the zoom magnification the N raw images acquired by the camera can be cropped in real time. For example, if the zoom magnification is twice, when any one of the N images acquired by the camera is cropped, the center of the currently displayed image is the center. Both the horizontal and vertical sides are cut in half.
  • the camera specifically collects several images.
  • it can be preset before the terminal device leaves the factory.
  • the specific number of frames can be determined according to the performance of the terminal device. For example, poor performance can be set to capture one frame, better performance can be set to capture 4 frames, and better performance can be set to capture 6 frames, etc. It can be set according to the terminal performance or other standards in advance. it is good.
  • Step 102 Obtain a first image through preliminary operations according to the N raw images, where the first image is a color three-channel image.
  • the preliminary operation includes: performing an interpolation operation on any one of the N images to be processed; or, performing an interpolation operation on the first denoised image, which is the one to be processed N images are obtained by multi-frame noise reduction.
  • the preliminary operation may include one or more of the following processing methods: black level correction, lens vignetting correction, and demosaicing interpolation processing.
  • this image can be any of the N raw images, or an image selected according to preset rules, for example, It can be the clearest image selected from N images. It is not limited here.
  • the method of determining an image and the neural network to be trained determine an image from N simulated raw images. The determination method is the same when performing preliminary operations.
  • the preliminary operation may include one or more of the following operations: multi-frame noise reduction, ghost image noise reduction, black level correction, lens vignetting correction, and demosaicing interpolation processing.
  • the neural network may take N simulated raw images as input, and the difference between the super-resolution images and the high-definition images corresponding to the N simulated raw images obtained by the neural network to be trained is the loss function. It is obtained by training the neural network to be trained; N simulated raw images may be images generated by degrading high-definition images and according to the data format of raw images; N simulated raw images corresponding to super-resolution images It is the superposition of the second detail information corresponding to the N simulated raw images output by the neural network to be trained and the second image obtained according to the N simulated raw images, and the second image is based on the N simulated raw images passed through Describe the image obtained by the preliminary operation.
  • a clear image P is prepared in advance during training, and then 4 degraded images are obtained through a pre-built degradation process on the image P.
  • the pre-built degradation process can be The image obtained by random movement, such as panning random pixels or rotating random angles in a small range, simulates image degradation caused by hand shaking or random displacement when taking pictures, and then obtains the first image through preliminary operations based on 4 raw images.
  • Step 103 Input the N raw images into a neural network to obtain first detailed information output by the neural network.
  • the neural network takes N simulated raw images as input, and uses the difference between the super-resolution image and the high-definition image corresponding to the N simulated raw images obtained through the neural network to be trained as the loss function, and the neural network to be trained
  • the network is trained;
  • the N simulated raw images are images generated by degrading the high-definition images according to the raw image data format;
  • the super-resolution images corresponding to the N simulated raw images are
  • the second detail information corresponding to the N simulated raw images output by the neural network to be trained is superimposed on the second image obtained according to the N simulated raw images, and the second image is based on the N Zhang simulated raw image The image obtained through the preliminary operation.
  • the loss function can use the color brightness of the two images to be directly different or to square the difference.
  • the value of the loss function is also very large. In this case, it can be reversed. How to change the parameters of the neural network to be trained can make the loss function small enough, which is the training process of the neural network to be trained. After the trained neural network inputs at least one image, it can obtain a data set or detailed image representing the details of the image.
  • Step 104 Superimpose the first detail information on the first image to obtain super-resolution images corresponding to the N raw images.
  • the size of the image corresponding to the first detail information may not be consistent with the size of the first image.
  • the first image can be scaled to make the size consistent with the image corresponding to the first detail information, and then the second The image corresponding to one detail information is added to the zoomed first image to obtain a super-resolution image.
  • the image obtained after superposition may be further enhanced.
  • Image enhancement operations may include: gamma correction, contrast enhancement and sharpening.
  • the embodiment of the present application by superimposing the first detailed information output by the neural network on the first image obtained by the preliminary operation, a super-resolution image including more detailed information can be obtained. Therefore, the embodiment of the present application provides The technical solution can get clear images.
  • the embodiments of the present application are mainly used when the terminal device obtains the triggering photographing instruction, and processes the image obtained by the camera to obtain a clear image scene.
  • the terminal device obtains the triggering photographing instruction, and processes the image obtained by the camera to obtain a clear image scene.
  • the following five specific scenarios are described. Understandably, when this application is implemented, it is not limited to the following 5 application scenarios:
  • the terminal equipment corresponding to the above five scenarios may be shown in FIG. 3A.
  • the terminal equipment 300 may include a camera 301, a memory 302, a processor 303, and a display unit 304.
  • the neural network may be software, specifically, application software or dynamic link. The library and other forms are stored in the memory 302. When the processor 303 receives a photographing instruction, the processor 303 can call the neural network stored in the memory 302. It should be noted that in some embodiments, the neural network may also exist in the form of hardware, that is, a hardware structure that integrates software functions.
  • the terminal device 300 may include a camera 301, a memory 302, and a processor 303. , Display unit 304 and neural network unit 305.
  • the camera 301 acquires 4 frames of original images in real time. If the zoom magnification corresponding to the photographing instruction is 2, the processor 4 crops any original image, specifically, the current The center of the displayed image is the center, and the horizontal and vertical sides are cut in half. The four images obtained are T1, T2, T3, and T4.
  • the processor 303 calls the neural network stored in the memory 302, and the neural network outputs detailed information.
  • the processor 303 performs preliminary operations on the four images T1, T2, T3, and T4. For example, performing operations such as multi-frame noise reduction, ghost image noise reduction, black level correction, lens vignetting correction, demosaicing interpolation operation, image scaling, etc., to obtain the first image T5.
  • the neural network also performs the learning process on the four images T1, T2, T3, and T4 input by the processor 303 to obtain the detail image.
  • the detail image corresponds to the three colors of RGB red, green and blue, including the detail image corresponding to the red channel and the detail image corresponding to the green channel.
  • the first image T5 obtained by this embodiment is shown in FIG. 5A.
  • the first detailed information output by the neural network includes three images as shown in FIG. 5B, FIG. 5C, and FIG. 5D.
  • FIG. 5B corresponds to FIG. 5A.
  • FIG. 5C is a detailed schematic diagram corresponding to the green channel in the detail image corresponding to FIG. 5A.
  • FIG. 5D is a detailed schematic diagram corresponding to the blue channel in the detail image corresponding to FIG. 5A.
  • the darker the corresponding color value is, the smaller the corresponding color value is, and the brighter the logo the larger the corresponding color value.
  • the processor can superimpose the color information corresponding to Figures 5B, 5C, and 5D to the color information shown in Figure 5A.
  • the first image is then subjected to image enhancement operations such as gamma correction, contrast enhancement, and sharpening to obtain a clear super-resolution image as shown in FIG. 5E.
  • the preliminary operations are not limited to the above-mentioned multi-frame noise reduction, ghost image noise reduction, black level correction, lens vignetting correction, demosaicing interpolation operation, image scaling and other operations, and can include these operations Part of it can also include the various operations mentioned here, and can also include other image sharpening processing operations, which are not limited here.
  • multi-frame noise reduction is a mature existing technology, which mainly selects an image from the collected multiple images as a reference frame, and performs registration through registration technology, so that the content of the multiple images is aligned with the reference frame. Then the multiple images are averaged to remove noise. If there is a moving object in the image, the position of the object on each image is different. In the process of averaging multiple images, the area of the moving object will not participate in the averaging, but only the content of the reference frame. This process is called For ghost detection. The area detected as a ghost image will have relatively large noise because it is not averaged. Therefore, the ghost image area can be denoised.
  • the existing single-frame noise reduction technology can be used to achieve this function.
  • Image processing such as black level correction and lens vignetting correction is a general processing for obtaining images with correct brightness and color.
  • the processed bayer image undergoes a demosaic interpolation operation to obtain a color three-channel image (each pixel has three components of red, green and blue), where the demosaicing operation can be any interpolation method, usually the simpler the demosaic interpolation and The easier the coordination of the neural network will be, and thus a better effect will be obtained. Therefore, in this embodiment, the bilinear interpolation method can be used to demosaicing.
  • the camera 301 acquires 4 frames of original images in real time. If the zoom magnification corresponding to the photographing instruction is 2, the processor 4 crops any original image, specifically, the current The center of the displayed image is the center, and the horizontal and vertical sides are cut in half. The four images obtained are T1, T2, T3, and T4.
  • the processor 303 calls the neural network stored in the memory 302, and the neural network outputs detailed information.
  • the processor 303 performs preliminary operations on T1, such as performing operations such as black level correction, lens vignetting correction, demosaicing interpolation, image scaling, etc., to obtain the first image T5.
  • the neural network performs the learning process on the four images T1, T2, T3 and T4 input by the processor 303, and outputs the detailed image.
  • the detailed image corresponds to the three colors of RGB red, green and blue, including the detail image corresponding to the red channel and the detail corresponding to the green channel.
  • the image and the detail image corresponding to the blue channel Through testing and adopting the technical solution provided by this embodiment, a clear image can be obtained.
  • the image to be performed preliminary operation can be an image randomly determined from T1, T2, T3, and T4, or an image determined according to other standards, such as T1, T2 The clearest image in T3 and T4.
  • the preliminary operation may not include multi-frame denoising and ghosting denoising.
  • the preliminary operation may not be limited to the above mentioned
  • the black level correction, lens vignetting correction, demosaicing interpolation operation, and image zooming operations can include a part of these operations, can also include the various operations mentioned here, and can also include other single-frame
  • the image sharpening processing operation performed by the image is not limited here.
  • the method for determining the image for which the preliminary operation is performed is the same as the method for determining the image for which the preliminary operation is performed during the training of the neural network to be trained.
  • the processor when constructing super-resolution image resolution, the processor performs preliminary operations on the clearest image among the four images of T1, T2, T3, and T4, and the neural network to be trained will also be the clearest of the four input images during training. Perform preliminary operations on the image.
  • the camera 301 acquires 4 frames of original images in real time. If the zoom magnification corresponding to the photographing instruction is 2, the processor 4 crops any original image, specifically, the current The center of the displayed image is the center, and the horizontal and vertical sides are cut in half. The four images obtained are T1, T2, T3, and T4.
  • the processor 303 performs preliminary operations on the two images T1 and T3, such as multi-frame noise reduction, ghost image noise reduction, black level correction, lens vignetting correction, demosaicing interpolation operation, image scaling, etc. to obtain low-definition images T5.
  • the neural network learns the four images T1, T2, T3 and T4 input by the processor 303 to obtain the detailed image.
  • the detailed image corresponds to the three colors of RGB red, green and blue, including the detail image corresponding to the red channel and the detail image corresponding to the green channel.
  • the detailed image corresponding to the blue channel Through testing and adopting the technical solution provided by this embodiment, a clear image can be obtained.
  • the images that the processor performs preliminary operations can be two images randomly determined from T1, T2, T3, and T4, or two images determined according to other standards, such as These are the two images with the highest definition among T1, T2, T3, and T4. How to determine the details can be set according to needs, and there is no limitation here. It should be noted that the method for determining T1 and T3 is consistent with the method for determining the image for performing preliminary operations when the neural network to be trained is trained.
  • the preliminary operations are not limited to the above-mentioned multi-frame noise reduction, ghost image noise reduction, black level correction, lens vignetting correction, demosaicing interpolation operation, image scaling and other operations, and may include these operations
  • a part of may also include various operations mentioned here, and may also include other image sharpening processing operations, which are not limited here.
  • the camera 301 obtains 1 frame of the original image in real time. If the zoom magnification corresponding to the photographing instruction is 2, the processor 4 crops the original image, specifically, the current display image The center is the center, and the horizontal and vertical sides are cut in half. Get an image as T1.
  • the processor 303 performs preliminary operations on T1, such as performing operations such as black level correction, lens vignetting correction, demosaicing interpolation, and image scaling to obtain image T5.
  • the processor inputs T1 into the neural network, and the neural network performs the learning process to obtain the detail image.
  • the detail image corresponds to the three colors of RGB red, green and blue, including the detail image corresponding to the red channel, the detail image corresponding to the green channel, and the detail corresponding to the blue channel. image.
  • the preliminary operations may not include operations such as multi-frame denoising and ghosting denoising to process multi-frame images.
  • the preliminary operations are not limited to the aforementioned black level correction and lens gradation. Operations such as halo correction, demosaicing interpolation, and image zooming can include some of these operations, and can also include the various operations mentioned here, and can also include other image sharpening operations for single-frame images , There is no limitation here.
  • the format of the 4 original images acquired by the camera is not a conventional bayer format, but a special quadra format.
  • the feature of quadra format images is that four adjacent pixels have the same color.
  • the camera 301 obtains 4 original images in quadra format in real time. If the zoom magnification corresponding to the photographing instruction is 2, the processor 303 crops any original image, specifically, the current display image The center is the center, and the horizontal and vertical sides are cut in half. The four images obtained are T1, T2, T3, and T4.
  • the processor 303 performs binning processing on the four images T1, T2, T3, and T4 to obtain four images T1', T2', T3', and T4'.
  • Binning processing refers to averaging four pixels of the same color to obtain one pixel.
  • Fig. 6A is a partial schematic diagram of the original image in the bayer format. From Fig. 6A, it can be seen that the colors of two adjacent pixels of the image in the bayer format are different.
  • FIGS. 6B and 6C FIG. 6B is a partial schematic diagram of the original image in the quadra format
  • FIG. 6C is a partial schematic diagram of converting the partial schematic diagram of the quadra format shown in FIG. 6B into the bayer format.
  • the processor 303 performs preliminary operations on the four images T1', T2', T3' and T4' after binning processing, such as multi-frame noise reduction, ghost noise reduction, black level correction, lens vignetting correction, Image T5' is obtained by operations such as demosaicing interpolation operation and image scaling.
  • the neural network performs the learning process on the four images T1, T2, T3, and T4 input by the processor 303 to obtain the detail image.
  • the detail image corresponds to the three colors of RGB red, green and blue, including the detail image corresponding to the red channel and the detail corresponding to the green channel.
  • the image and the detail image corresponding to the blue channel are examples of the detail image.
  • the quadra format image obtained by image degradation is also subjected to binning processing, so that the format of the image on which the neural network to be trained performs preliminary operations during training is also in bayer format.
  • the technical solutions provided by this application are not limited to the bayer format and quadra format.
  • the technical solutions provided by this application can also be used, and any desired format can be used.
  • the mosaic interpolation method and neural network can realize the super-resolution reconstruction of the image and obtain a clear image.
  • the reason for binning images in quadra format is that common demosaicing interpolation methods can be used to convert to bayer format, so quadra format is not limited to binning processing, and other demosaicing interpolation methods can be used.
  • an embodiment of the present application also provides an image super-resolution reconstruction apparatus.
  • the image super-resolution reconstruction apparatus 200 provided in the embodiment of the present application includes: an acquisition unit 201, a first processing unit 202, The second processing unit 203 and the superimposing unit 204.
  • the acquiring unit 201 is configured to acquire N original raw images through a camera, where N is 1 or an integer greater than 1.
  • the first processing unit 202 is configured to obtain a first image through preliminary operations according to the N raw images.
  • the second processing unit 203 is configured to input the N raw images into a neural network to obtain first detailed information output by the neural network.
  • the superimposing unit 204 is configured to superimpose the first detail information on the first image to obtain super-resolution images corresponding to the N raw images.
  • the neural network takes N simulated raw images as input, and uses the difference between the super-resolution image and the high-definition image corresponding to the N simulated raw images obtained through the neural network to be trained as the loss function, and the neural network to be trained
  • the network is trained;
  • the N simulated raw images are images generated by degrading the high-definition images according to the raw image data format;
  • the super-resolution images corresponding to the N simulated raw images are
  • the second detail information corresponding to the N simulated raw images output by the neural network to be trained is superimposed on the second image obtained according to the N simulated raw images, and the second image is based on the N Zhang simulated raw image The image obtained through the preliminary operation.
  • Preliminary operations include: performing an interpolation operation on any of the N images to be processed; or, performing an interpolation operation on a first denoised image, where the first denoised image is a multi-frame denoising for the N images to be processed The resulting image.
  • the format of the N raw images can be Bayer Bayer format or Kudro quadra format; if the format of the N raw images is quadra format, the first processing unit obtains the first data through preliminary operations according to the N raw images.
  • the method includes: converting the N raw images into N images in the bayer format through binning processing, and performing the preliminary operation on the N images converted into the bayer format to obtain a first image; If the format of the N raw images is a quadra format, the N simulated raw images are images in a quadra format; the second image is the binning process to convert the N simulated raw images into a bayer format, An image obtained by performing the preliminary operation on the N simulated raw images in the bayer format after conversion.
  • the acquiring unit 201 is specifically configured to crop the N images acquired by the camera according to the zoom magnification corresponding to the photographing instruction to obtain N raw images corresponding to the zoom magnification.
  • FIG. 4 is a schematic structural diagram of a terminal device provided by an embodiment of the application.
  • the terminal device 400 includes: a radio frequency unit 410, a memory 420, an input unit 430, a camera 440, an audio circuit 450, and a processor 460 , External interface 470 and power supply 480.
  • the input unit 430 includes a touch screen 431 and other input devices 432, and the audio circuit 450 includes a speaker 451, a microphone 452, and an earphone jack 453.
  • the touch screen 431 may be a display screen with a touch function.
  • the user can trigger a photographing instruction by clicking the photographing button displayed on the touch screen 431, and change the zoom magnification by changing the contact distance with the touch screen.
  • the processor 460 crops the N frames of original images obtained by the camera 440 according to the zoom magnification corresponding to the photographing instruction to obtain N cropped images, where N is 1 or an integer greater than 1.
  • the processor 460 performs preliminary operations on M images among the N images to obtain the first image, where M is a positive integer less than or equal to N.
  • the processor 460 calls the neural network and inputs the cropped N images to the neural network.
  • the neural network learns the input N images and outputs the first detailed information.
  • the processor 460 superimposes the first detailed information on the first image to obtain super Resolution image. Further, the processor 460 saves the super-resolution image to the memory 420.
  • the processor 460 obtains the super-resolution image from the memory 420 and uses the touch screen as a display interface. 431 is displayed.
  • the neural network is a trained convolutional neural network, taking N simulated raw images as input, and the difference between the super-resolution image and the high-definition image corresponding to the N simulated raw images obtained by the neural network to be trained is Loss function, obtained by training the neural network to be trained; the N simulated raw images are images generated by degrading the high-definition images and in accordance with the raw image data format; the N simulated raw images The super-resolution image corresponding to the raw image is the superposition of the second detail information corresponding to the N simulated raw images output by the neural network to be trained and the second image obtained according to the N simulated raw images, so The second image is an image obtained through the preliminary operation according to the N simulated raw images.
  • the embodiment of the present application also provides a computer-readable storage medium.
  • the computer-readable storage medium stores instructions and a computer program corresponding to a neural network.
  • the terminal device executes any of the preceding instructions. Part or all of the steps of the image super-resolution reconstruction method according to an embodiment.
  • the embodiments of the present application also provide a computer program product, which when the computer program product runs on a computer, causes the computer to execute part or all of the steps of the image super-resolution reconstruction method.
  • each module in the above device is only a division of logical functions, and may be fully or partially integrated into one physical entity during actual implementation, or may be physically separated.
  • each of the above modules can be separately set up processing elements, or they can be integrated in a certain chip of the terminal for implementation.
  • they can also be stored in the storage element of the controller in the form of program codes and processed by a certain processor.
  • the component calls and executes the functions of the above modules.
  • various modules can be integrated together or implemented independently.
  • the processing element described here may be an integrated circuit chip with signal processing capability.
  • each step of the above method or each of the above modules can be completed by hardware integrated logic circuits in the processor element or instructions in the form of software.
  • the processing element may be a general-purpose processor, such as a central processing unit (CPU), or one or more integrated circuits configured to implement the above methods, such as one or more application-specific integrated circuits (application-specific integrated circuits). integrated circuit, ASIC), or, one or more microprocessors (digital signal processor, DSP), or, one or more field-programmable gate arrays (FPGA), etc.
  • CPU central processing unit
  • ASIC application-specific integrated circuits
  • microprocessors digital signal processor, DSP
  • FPGA field-programmable gate arrays

Landscapes

  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Image Processing (AREA)

Abstract

本申请实施例公开了一种图像超分辨重建方法、装置和终端设备，所述方法包括：通过摄像头获取N张原始raw图像，所述N为1或者大于1的整数；根据所述N张raw图像通过初步操作得到第一图像，将所述N张raw图像输入神经网络，得到所述神经网络输出的第一细节信息；将所述第一细节信息叠加到所述第一图像，得到所述N张raw图像对应的超分辨率图像。采用本申请实施例提供的技术方案，通过在初步操作得到的第一图像上叠加神经网络输出的第一细节信息，可以得到包括更多细节信息的超分辨率图像，因此，采用本申请实施例提供的技术方案可以得到清晰的图像。

Description

一种图像超分辨重建方法、装置和终端设备
本申请要求于2019年03月18日提交中国专利局、申请号为201910205538.9、申请名称为“一种图像超分辨重建方法、装置和终端设备”的中国专利申请的优先权，其全部内容通过引用结合在本申请中。
技术领域
本申请涉及图像处理技术领域，尤其涉及一种图像超分辨重建方法、装置和终端设备。
背景技术
智能手机、平板电脑等具有轻薄特性并安装了定焦镜头的便携式终端设备，用户可以通过变焦调整拍照的构图，定焦镜头实现变焦时利用数码变焦技术实现被拍摄景物放大显示，但是图像的清晰度通常有损失，尤其在放大倍数较大时，拍出图像经常不清晰，不能满足用户的要求。
为了得到清晰的图像，可以采用图像超分辨重建方法，图像超分辨重建方法是基于信号处理技术实现从单幅或多幅低分辨率图像中重建出高分辨率图像，通过恢复出图像在采集过程中所丢失的高频细节信息，达到提高图像分辨率的目的。最终图像在分辨率上与原图像相比有较大的提升，可以提供更多的细节信息，最终图像也更接近于理想图像。
图像超分辨重建目前已有多种方法，比如，谷歌提出过一种图像超分辨重建的方法，该方法通过多帧配准的方式进行去马赛克，利用帧间的位移填补缺失的像素。具体地，通过将图像依次向右、向下、向左移动一个像素，就能完整重建出彩色图像。这种方法像素值都是通过真实图像恢复，而不是通过插值得到，从而提升了图像的分辨率。需要说明的是，谷歌提供的这种方法理论上会提高图像的分辨率，但是实际使用时并不理想，因为该方法需要对图像进行一个像素的偏移，实际使用时，图像很难精确地实现单个像素的偏移，因此采用该方法进行图像超分辨重建得到的图像也不理想。
所以，如何使安装定焦镜头的终端设备拍出清晰的图像是目前亟待解决的问题。
发明内容
本申请实施例提供了一种图像超分辨重建方法、装置和终端设备，能够使重建后的图像包含更多的细节信息，更清晰。
第一方面，本申请实施例提供了一种图像超分辨重建方法，包括：通过摄像头获取N张原始raw图像，所述N为1或者大于1的整数；根据所述N张raw图像通过初步操作，得到第一图像，所述第一图像为彩色的三通道图像；将所述N张raw图像输入神经网络，得到所述神经网络输出的第一细节信息；将所述第一细节信息叠加到所述第一图像，得到所述N张raw图像对应的超分辨率图像。
其中，第一细节信息可以是表示图像细节的数据集合，也可以是表示图像细节的图像。
在具体实施时，用户可以通过点击显示屏上的拍照按键触发拍照指令，在获取拍照指令之后，摄像头可以获取N张raw图像。需要说明的是，在拍照之前通过显示界面预览被 拍摄景物时，用户可以执行变焦操作，比如，将与显示屏接触的两个手指之间的距离变大，可增大变焦倍率；将与显示屏接触的两个手指之间的距离变小，可缩小变焦倍率。
在一些可能的实施例中，通过摄像头获取N张raw图像可以包括：根据拍照指令对应的变焦倍率将所述摄像头获取的N张图像进行裁剪得到与所述变焦倍率对应的N张raw图像。
在一些可能的实施例中，初步操作可以包括：对待处理的N张图像中的任一图像进行插值操作；或者，对第一降噪图像进行插值操作，所述第一降噪图像为对待处理的N张图像进行多帧降噪得到的图像。
在一些可能的实施例中，所述神经网络可以是以N张模拟的raw图像为输入，以通过待训练神经网络得到的所述N张模拟的raw图像对应的超分辨率图像与高清图像的差异为损失函数，对所述待训练神经网络进行训练得到；所述N张模拟的raw图像是通过对所述高清图像进行退化处理，并按照raw图像的数据格式生成的图像；所述N张模拟的raw图像对应的超分辨率图像为所述待训练神经网络输出的所述N张模拟的raw图像对应的第二细节信息与根据所述N张模拟的raw图像得到的第二图像的叠加，所述第二图像为根据所述N张模拟的raw图像通过所述初步操作得到的图像。
在一些可能的实施例中，所述N张raw图像的格式为拜耳bayer格式或者库卓quadra格式；若所述N张raw图像的格式为quadra格式，则所述根据所述N张raw图像通过初步操作得到第一图像，包括：将所述N张raw图像经分箱binning处理转换为bayer格式的N张图像，对转换为bayer格式的所述N张图像执行所述初步操作得到第一图像。若所述N张raw图像的格式为quadra格式，所述N张模拟的raw图像是quadra格式的图像；所述第二图像为将所述N张模拟的raw图像经binning处理转换为bayer格式，对转换后bayer格式的所述N张模拟的raw图像执行所述初步操作得到的图像。
采用本申请实施例提供的技术方案，通过在初步操作得到的第一图像上叠加神经网络输出的第一细节信息，可以得到包括更多细节信息的超分辨率图像，因此，采用本申请实施例提供的技术方案可以得到清晰的图像。
第二方面，本申请实施例提供了一种图像超分辨重建装置，包括：获取单元，用于通过摄像头获取N张原始raw图像，所述N为1或者大于1的整数；第一处理单元，用于根据所述N张raw图像通过初步操作，得到第一图像，所述第一图像为彩色的三通道图像。第二处理单元，用于将所述N张raw图像输入神经网络，得到所述神经网络输出的第一细节信息；叠加单元，用于将所述第一细节信息叠加到所述第一图像，得到所述N张raw图像对应的超分辨率图像。
在一些可能的实施例中，所述神经网络是以N张模拟的raw图像为输入，以通过待训练神经网络得到的所述N张模拟的raw图像对应的超分辨率图像与高清图像的差异为损失函数，对所述待训练神经网络进行训练得到；所述N张模拟的raw图像是通过对所述高清图像进行退化处理，并按照raw图像的数据格式生成的图像；所述N张模拟的raw图像对应的超分辨率图像为所述待训练神经网络输出的所述N张模拟的raw图像对应的第二细节信息与根据所述N张模拟的raw图像得到的第二图像的叠加，所述第二图像为根据所述N张模拟的raw图像通过所述初步操作得到的图像。
在一些可能的实施例中，所述初步操作包括：对待处理的N张图像中的任一图像进行插值操作；或者，对第一降噪图像进行插值操作，所述第一降噪图像为对待处理的N张图像进行多帧降噪得到的图像。
在一些可能的实施例中，所述N张raw图像的格式为拜耳bayer格式或者库卓quadra格式；若所述N张raw图像的格式为quadra格式，则所述第一处理单元在根据所述N张raw图像通过初步操作得到第一图像方面，具有用于：将所述N张raw图像经分箱binning处理转换为bayer格式的N张图像，对转换为bayer格式的所述N张图像执行所述初步操作得到第一图像；若所述N张raw图像的格式为quadra格式，所述N张模拟的raw图像是quadra格式的图像；所述第二图像为将所述N张模拟的raw图像经binning处理转换为bayer格式，对转换后bayer格式的所述N张模拟的raw图像执行所述初步操作得到的图像。
在一些可能的实施例中，所述获取单元具体用于，根据拍照指令对应的变焦倍率将所述摄像头获取的N张图像进行裁剪得到与所述变焦倍率对应的N张raw图像。
第三方面，本申请实施例提供了一种终端设备，包括摄像头、处理器和存储器，所述摄像头，用于在所述处理器获取拍照指令后，获取N张原始raw图像，所述N为1或者大于1的整数；所述存储器，用于存储可在所述处理器上运行的计算机程序；所述处理器，用于执行如第一方面或者第一方面任一可能的实施例中所述的方法的部分或全部步骤。
第四方面，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质存储有指令和神经网络对应的计算机程序，当所述指令在终端设备上运行时，使得所述终端设备执行如权利要求1至5中任一项所述的方法。
第五方面，本申请实施例提供了一种终端设备，包括摄像头、处理器和神经网络单元；述摄像头，用于在所述处理器获取拍照指令后，获取N张原始raw图像，所述N为1或者大于1的整数；所述神经网络单元，用于以所述N张raw图像为输入，得到第一细节信息；所述处理器，用于执行如第一方面或者第一方面任一可能的实施例中所述的方法的部分或全部步骤。
第六方面，本申请实施例提供了一种计算机可读存储介质，所述计算机可读存储介质存储有计算机程序，所述计算机程序包括程序指令，所述程序指令当被处理器执行时，使所述处理器执行如第一方面或者第一方面任一可能的实施例中所述的方法的部分或全部步骤。
第七方面，本申请实施例提供了一种计算机程序产品，所述计算机程序产品包括存储了计算机程序的计算机可读存储介质，该计算机程序使得计算机执行如第一方面或者第一方面任一可能的实施例中所述的方法的部分或全部步骤。
采用本申请实施例提供的技术方案，通过在初步操作得到的第一图像上叠加神经网络输出的第一细节信息，可以得到包括更多细节信息的超分辨率图像，因此，采用本申请实施例提供的技术方案可以得到清晰的图像。
附图说明
图1为本申请的一个实施例提供的一种图像超分辨重建方法的流程示意图。
图2为本申请的另一个实施例提供的一种图像超分辨重建装置的结构示意图。
图3A为本申请的另一个实施例提供的一种终端设备的结构示意图。
图3B为本申请的另一个实施例提供的一种终端设备的结构示意图。
图4为本申请的另一个实施例提供的一种终端设备的结构示意图。
图5A为本申请的一个实施例中超分辨处理单元得到的低清图像。
图5B为与图5A对应的细节图像中红色通道对应的细节示意图。
图5C为与图5A对应的细节图像中绿色通道对应的细节示意图。
图5D为与图5A对应的细节图像中蓝色通道对应的细节示意图。
图5E为该实施例经图像超分辨重建后得到的高分辨率图像。
图6A是本申请一实施例中bayer格式的原始图像的局部示意图。
图6B是本申请一实施例中quadra格式的原始图像的局部示意图。
图6C是将图6B所示quadra格式的局部示意图转换为bayer格式的局部示意图。
具体实施方式
下面将结合本申请实施例中的附图，对本申请实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例仅仅是本申请的一部分实施例，并不是全部的实施例。基于本申请中的实施例，本领域普通技术人员在没有做出创造性劳动前提下所获得的所有其他实施例，都属于本申请保护的范围。
本申请实施例提供的图像超分辨重建方法，包括：通过摄像头获取N张原始raw图像，所述N为1或者大于1的整数；根据所述N张raw图像通过初步操作，得到第一图像，所述第一图像为彩色的三通道图像；将所述N张raw图像输入神经网络，得到所述神经网络输出的第一细节信息；将所述第一细节信息叠加到所述第一图像，得到所述N张raw图像对应的超分辨率图像。
在一些可能的实施例中，在获取N张raw图像之前，还可以包括获取拍照指令，根据拍照指令对应的变焦倍率对摄像头获取的图像进行裁剪，裁剪后与变焦倍率对应的图像作为raw图像。
本申请实施例中，终端设备可以是具有轻薄特性并安装了定焦镜头的便携式设备，比如：具有拍照功能的移动电话(或称为“蜂窝”电话)、智能手机、便携式可穿戴设备(如智能手表等)、平板电脑、个人电脑(PC，Personal Computer)、PDA(Personal Digital Assistant，个人数字助理)等。
请参阅图1，图1为本申请的一个实施例提供的一种图像超分辨重建方法的流程示意图，在该实施例中，所述方法由处理器执行可以包括以下步骤：
步骤101、在获取拍照指令后，根据拍照指令对应的变焦倍率对摄像头获取的N张图像进行裁剪，得到与所述变焦倍率对应的N张raw图像，N为1或者大于1的整数。
在一些可能的实施例中，拍照指令可以由用户点击显示屏上的拍照按键触发，在拍照之前，通过显示界面预览被拍摄景物时，用户可以执行变焦操作，比如，将与显示屏接触的两个手指之间的距离变大，可增大变焦倍率；将与显示屏接触的两个手指之间的距离变小，可缩小变焦倍率。根据变焦倍率可以对摄像头实时获取的N张raw图像进行裁剪，举例来说，若变焦倍率为两倍，对摄像头获取的N图像中任一图像进行裁剪时，以当前显示 图像的中心为中心，横边和竖边都裁减掉一半。需要说明的是，在一些可能的实施例中，摄像头具体采集几张图像，对于某种类型的终端设备可以在终端设备出厂之前预先设定好的，具体帧数可以根据终端设备的性能确定，比如，性能较差的可以设为采集一帧，性能较好的可以设为采集4帧，性能更好的可以设为采集6帧等，具体可以根据终端的性能也可以根据其他标准预先设定好。
步骤102、根据所述N张raw图像通过初步操作得到第一图像，所述第一图像为彩色的三通道图像。
在一些可能的实施例中，初步操作包括：对待处理的N张图像中的任一图像进行插值操作；或者，对第一降噪图像进行插值操作，所述第一降噪图像为对待处理的N张图像进行多帧降噪得到的图像。
在一些可能的实施例中，初步操作可以包括如下处理方法中的一个或者多个：黑电平矫正、镜头渐晕矫正、和去马赛克插值处理等操作。需要说明的是，当初步操作是对1张图像进行处理时，这一张图像可以是N张raw图像中的任一张图像，也可以是按照预设的规则选取的一张图像，比如，可以是从N张图像中选取的最清晰的一张图像，这里不做限定，需要说明的是，这里一张图像的确定方法与待训练神经网络从N张模拟的raw图像中确定一张图像来执行初步操作时的确定方法一致。
在一些可能的实施例中，N可以为大于2或者等于2整数，比如，N＝4。初步操作可以包括如下操作中的一个或者多个：多帧降噪、鬼影降噪、黑电平矫正、镜头渐晕矫正、和去马赛克插值处理等操作。
需要说明的是，神经网络可以是以N张模拟的raw图像为输入，以通过待训练神经网络得到的所述N张模拟的raw图像对应的超分辨率图像与高清图像的差异为损失函数，对所述待训练神经网络进行训练得到；N张模拟的raw图像可以是通过对高清图像进行退化处理，并按照raw图像的数据格式生成的图像；N张模拟的raw图像对应的超分辨率图像为待训练神经网络输出的N张模拟的raw图像对应的第二细节信息与根据N张模拟的raw图像得到的第二图像的叠加，第二图像为根据所述N张模拟的raw图像通过所述初步操作得到的图像。
举例来说，若N＝4，则在训练时预先准备1张清晰的图像P，然后对图像P经预先构建的退化过程得到4张退化后的图像，预先构建的退化过程可以是对图像P进行随机移动得到的图像，比如在小范围内平移随机的像素或者转动随机的角度等模拟拍照时手抖或者随机的位移产生的图像退化，然后根据4张raw图像通过初步操作得到第一图像。
步骤103、将所述N张raw图像输入神经网络，得到所述神经网络输出的第一细节信息。
神经网络是以N张模拟的raw图像为输入，以通过待训练神经网络得到的所述N张模拟的raw图像对应的超分辨率图像与高清图像的差异为损失函数，对所述待训练神经网络进行训练得到；所述N张模拟的raw图像是通过对所述高清图像进行退化处理，并按照raw图像的数据格式生成的图像；所述N张模拟的raw图像对应的超分辨率图像为所述待训练神经网络输出的所述N张模拟的raw图像对应的第二细节信息与根据所述N张模拟的raw图像得到的第二图像的叠加，所述第二图像为根据所述N张模拟的raw图像通过所述初步 操作得到的图像。
在一些可能的实施例中，损失函数可以采用两个图像中颜色亮度直接做差或者做差的平方等操作，当两张图像差异很大时，损失函数的值也很大，这时可以反推待训练神经网络中的参数如何改变可以使损失函数足够小，这是待训练神经网络的训练过程。训练好的神经网络输入至少一张图像后，可以得到表示图像细节的数据集合或者细节图像。
步骤104、将所述第一细节信息叠加到所述第一图像，得到所述N张raw图像对应的超分辨率图像。
在一些可能的实施例中，第一细节信息对应的图像与第一图像的尺寸可能不一致，这时可以将第一图像进行缩放使其尺寸与第一细节信息对应的图像一致，然后再将第一细节信息对应的图像与缩放后的第一图像相加，得到超分辨率图像。
在一些可能的实施例中，还可以进一步对叠加后得到的图像增强处理。
图像增强操作可以包括：伽马矫正、对比增强和锐化等。
本申请实施例提供的技术方案，通过在初步操作得到的第一图像上叠加神经网络输出的第一细节信息，可以得到包括更多细节信息的超分辨率图像，因此，采用本申请实施例提供的技术方案可以得到清晰的图像。
本申请实施例主要应用在终端设备获取触发拍照指令后，对摄像头获取的图像进行处理，得到清晰图像的场景。为了方便理解，按照摄像头拍摄的原始图像的格式，以及在获取拍照指令后，摄像头获取的原始图像的个数N，以及执行初步操作的图像的个数M，分如下5个具体场景进行描述。可以理解的，本申请实施时，不限于下面5个应用场景：
应用场景1，N＝4，M＝4，摄像头拍摄的原始图像为bayer格式。
应用场景2，N＝4，M＝1，摄像头拍摄的原始图像为bayer格式。
应用场景3，N＝4，M＝2，摄像头拍摄的原始图像为bayer格式。
应用场景4，N＝1，M＝1，摄像头拍摄的原始图像为bayer格式。
应用场景5，N＝4，M＝4，摄像头拍摄的原始图像为quadra格式。
上述5个场景对应的终端设备可以如图3A所示，终端设备300可以包括：摄像头301、存储器302、处理器303和显示单元304，神经网络可以为软件，具体地可以以应用软件或者动态链接库等形态存储在存储器302中，当处理器303接收到拍照指令后，处理器303可以调用保存在存储器302中的神经网络。需要说明的是，在一些实施例中，神经网络也可以以硬件形式存在，即集成了软件功能的硬件结构，如图3B所示，终端设备300可以包括：摄像头301、存储器302、处理器303、显示单元304和神经网络单元305。图3A和图3B对应的终端设备，在执行图像超分辨重建时除了神经网络存在形态不同，其他部分没有实质区别，为了便于描述，下面在对各应用场景的实施过程进行描述时，统一采用图3A对应的结构进行描述。
实施例一
对应场景1，其中，N＝4，M＝4，摄像头301拍摄的原始图像为bayer格式。
在该实施例中，处理器303获取拍照指令后，摄像头301实时获取4帧原始图像，若 拍照指令对应的变焦倍率为2，则处理器4对任一原始图像进行裁剪，具体地，以当前显示图像的中心为中心，横边和竖边都裁减掉一半。得到四张图像分别为T1、T2、T3和T4。
处理器303调用存储在存储器302中的神经网络，神经网络输出细节信息。
处理器303对T1、T2、T3和T4这四张图像进行初步操作。比如进行多帧降噪、鬼影降噪、黑电平校正、镜头渐晕校正、去马赛克插值操作、图像缩放等操作得到第一图像T5。神经网络还对处理器303输入的T1、T2、T3和T4四张图像执行学习过程，得到细节图像，细节图像对应RGB红绿蓝三种颜色，包括红色通道对应的细节图像、绿色通道对应的细节图像和蓝色通道对应的细节图像。采用该实施例得到的第一图像T5如图5A所示，神经网络输出的第一细节信息包括如图5B、图5C和图5D所示的3张图像，其中，图5B为与图5A对应的细节图像中红色通道对应的细节示意图。图5C为与图5A对应的细节图像中绿色通道对应的细节示意图。图5D为与图5A对应的细节图像中蓝色通道对应的细节示意图。图5B、图5C和图5D中越黑表示对应颜色数值越小，越亮标识对应颜色数值越大，然后处理器可以将图5B、图5C和图5D对应的颜色信息叠加到图5A所示的第一图像上，然后经过伽马矫正、对比增强和锐化等图像增强操作得到图5E所示的清晰的超分辨率图像。
需要说明的是，初步操作不限于上述提到的多帧降噪、鬼影降噪、黑电平校正、镜头渐晕校正、去马赛克插值操作、图像缩放等操作，可以包括这几种操作的一部分，也可以包括这里提到的各种操作，还可以包括其他的图像清晰化处理操作，这里不做限定。
其中，多帧降噪是成熟的现有技术，主要是从采集的多张图像中选取一张图像作为参考帧，通过配准技术进行配准，使得多张图像的内容和参考帧对齐。然后对多张图像进行平均，以去除噪声。图像中如果存在运动物体，那么物体在每一张图像上的位置是不同的，在多张图像平均过程中，运动物体的区域不会参与平均，而是只用参考帧的内容，这个过程称为鬼影检测。检测为鬼影的区域，因为没有经过平均，噪声会比较大，因此可以重点对鬼影区域进行去噪。现有的单帧降噪技术都能用来实现这个功能。
对图像进行黑电平矫正，镜头渐晕矫正等图像处理，是为了得到亮度和颜色正确的图像而进行的一般性处理。处理之后的bayer图像经过去马赛克插值操作，得到彩色的三通道图像(每个像素都有红绿蓝三个分量)，在这里去马赛克操作可以是任意插值方式，通常越简单的去马赛克插值和神经网络的配合会越容易，从而得到更好的效果，因此该实施例中可以采用双线性插值的方法去马赛克。
实施例二
对应场景2，其中，N＝4，M＝1，摄像头301拍摄的原始图像为bayer格式。
在该实施例中，处理器303获取拍照指令后，摄像头301实时获取4帧原始图像，若拍照指令对应的变焦倍率为2，则处理器4对任一原始图像进行裁剪，具体地，以当前显示图像的中心为中心，横边和竖边都裁减掉一半。得到四张图像分别为T1、T2、T3和T4。
处理器303调用存储在存储器302中的神经网络，神经网络输出细节信息。
处理器303对T1进行初步操作，比如进行黑电平校正、镜头渐晕校正、去马赛克插值操作、图像缩放等操作得到第一图像T5。神经网络对处理器303输入的T1、T2、T3和T4四张图像执行学习过程，输出细节图像，细节图像对应RGB红绿蓝三种颜色，包括红色通 道对应的细节图像、绿色通道对应的细节图像和蓝色通道对应的细节图像。通过测试采用该实施例提供的技术方案可以得到清晰的图像。
需要说明的是，M＝1时，被执行初步操作的图像可以是从T1、T2、T3和T4中随机确定的一幅图像，也可以是按照其他标准确定的图像，比如可以是T1、T2、T3和T4中最清晰的一幅图像，在该实施例中，初步操作可以不包括多帧去噪和鬼影去噪等对多帧图像进行处理的操作，初步操作可以不限于上述提到的黑电平校正、镜头渐晕校正、去马赛克插值操作、和图像缩放等操作，可以包括这几种操作的一部分，也可以包括这里提到的各种操作，还可以包括其他的对单帧图像进行的图像清晰化处理操作，这里不做限定。
需要说明的是，当M＝1时，在进行图像超分辨构建时，确定被执行初步操作的图像的方法，与待训练神经网络训练过程中确定执行初步操作的图像的方法相同。比如，若进行超分辨图像分辨构建时，处理器对T1、T2、T3和T4四幅图像中最清晰的图像执行初步操作，则待训练神经网络在训练时也对输入的4张图像中最清晰的图像执行初步操作。
实施例三
对应场景3，其中，N＝4，M＝2，摄像头拍摄的原始图像为bayer格式。
在该实施例中，处理器303获取拍照指令后，摄像头301实时获取4帧原始图像，若拍照指令对应的变焦倍率为2，则处理器4对任一原始图像进行裁剪，具体地，以当前显示图像的中心为中心，横边和竖边都裁减掉一半。得到四张图像分别为T1、T2、T3和T4。
处理器303对T1和T3这两张图像进行初步操作，比如进行多帧降噪、鬼影降噪、黑电平校正、镜头渐晕校正、去马赛克插值操作、图像缩放等操作得到低清图像T5。神经网络对处理器303输入的T1、T2、T3和T4四张图像进行学习，得到细节图像，细节图像对应RGB红绿蓝三种颜色，包括红色通道对应的细节图像、绿色通道对应的细节图像和蓝色通道对应的细节图像。通过测试采用该实施例提供的技术方案可以得到清晰的图像。
需要说明的是，M＝2时，被处理器执行初步操作的图像可以是从T1、T2、T3和T4中随机确定的两张图像，也可以是按照其他标准确定的两张图像，比如可以是T1、T2、T3和T4中清晰度最高的两张图像，具体如何确定可以根据需要进行设定，这里不做限定。需要说明的是，T1和T3的确定方法与待训练神经网络进行训练时确定执行初步操作的图像的确定方法一致。
在该实施例中，初步操作不限于上述提到的多帧降噪、鬼影降噪、黑电平校正、镜头渐晕校正、去马赛克插值操作、图像缩放等操作，可以包括这几种操作的一部分，也可以包括这里提到的各种操作，还可以包括其他的图像清晰化处理操作，这里不做限定。
实施例四
对应场景4，其中，N＝1，M＝1，摄像头301拍摄的原始图像为bayer格式。
在该实施例中，处理器303获取拍照指令后，摄像头301实时获取1帧原始图像，若拍照指令对应的变焦倍率为2，则处理器4对原始图像进行裁剪，具体地，以当前显示图像的中心为中心，横边和竖边都裁减掉一半。得到一张图像为T1。
处理器303对T1执行初步操作，比如进行黑电平校正、镜头渐晕校正、去马赛克插值 操作、图像缩放等操作得到图像T5。处理器将T1输入神经网络，神经网络执行学习过程，得到细节图像，细节图像对应RGB红绿蓝三种颜色，包括红色通道对应的细节图像、绿色通道对应的细节图像和蓝色通道对应的细节图像。通过测试采用该实施例提供的技术方案可以得到清晰的超分辨率图像。
需要说明的是，在该实施例中，初步操作可以不包括多帧去噪和鬼影去噪等对多帧图像进行处理的操作，初步操作不限于上述提到的黑电平校正、镜头渐晕校正、去马赛克插值操作、和图像缩放等操作，可以包括这几种操作的一部分，也可以包括这里提到的各种操作，还可以包括其他的对单帧图像进行的图像清晰化处理操作，这里不做限定。
实施例五
对应场景5，其中，N＝4，M＝4，摄像头301拍摄的原始图像为quadra格式。
在该实施例中，摄像头获取的4张原始图像的格式不是常规的bayer格式，而是一种特殊的quadra格式。quadra格式的图像的特点是相邻的四个像素有相同的颜色。
处理器303获取拍照指令后，摄像头301实时获取4张quadra格式的原始图像，若拍照指令对应的变焦倍率为2，则处理器303对任一原始图像进行裁剪，具体地，以当前显示图像的中心为中心，横边和竖边都裁减掉一半。得到四张图像分别为T1、T2、T3和T4。
处理器303对T1、T2、T3和T4这四张图像进行binning处理得到四张图像T1'、T2'、T3'和T4'。Binning处理指的是将四个颜色相同的像素进行平均，得到一个像素。图6A是bayer格式的原始图像的局部示意图，由图6A可知bayer格式的图像相邻两个像素的颜色是不同的。如图6B和图6C所示，图6B为quadra格式的原始图像的局部示意图，图6C是将图6B所示quadra格式的局部示意图转换为bayer格式的局部示意图。
处理器303对经binning处理后的T1'、T2'、T3'和T4'这四张图像进行初步操作，比如进行多帧降噪、鬼影降噪、黑电平校正、镜头渐晕校正、去马赛克插值操作、图像缩放等操作得到图像T5'。神经网络对处理器303输入的T1、T2、T3和T4四张图像执行学习过程，得到细节图像，细节图像对应RGB红绿蓝三种颜色，包括红色通道对应的细节图像、绿色通道对应的细节图像和蓝色通道对应的细节图像。
需要说明的是，待训练神经网络进行训练时，图像退化得到的quadra格式的图像也进行binning处理，以使待训练神经网络在训练时执行初步操作的图像的格式也是bayer格式。
需要说明的是，本申请提供的技术方案不限于bayer格式和quadra格式，摄像头中的图像传感器为其他排列方式对应的格式为其他格式时，也可以使用本申请提供的技术方案，使用任意的去马赛克插值方式配合神经网络能够实现图像超分辨重建，得到清晰的图像。需要理解的是，quadra格式的图像进行binning处理的原因是转为bayer格式可以采用常用的去马赛克插值方式，所以quadra格式并不限定于binning处理，采用其他去马赛克插值方式就可以。
请参见图2，本申请实施例还提供了一种图像超分辨重建装置，如图2所示，本申请实施例提供的图像超分辨重建装置200包括：获取单元201、第一处理单元202、第二处理单元203和叠加单元204。
获取单元201，用于通过摄像头获取N张原始raw图像，所述N为1或者大于1的整 数。第一处理单元202，用于根据所述N张raw图像通过初步操作得到第一图像。第二处理单元203，用于将所述N张raw图像输入神经网络，得到所述神经网络输出的第一细节信息。叠加单元204，用于将所述第一细节信息叠加到所述第一图像，得到所述N张raw图像对应的超分辨率图像。
神经网络是以N张模拟的raw图像为输入，以通过待训练神经网络得到的所述N张模拟的raw图像对应的超分辨率图像与高清图像的差异为损失函数，对所述待训练神经网络进行训练得到；所述N张模拟的raw图像是通过对所述高清图像进行退化处理，并按照raw图像的数据格式生成的图像；所述N张模拟的raw图像对应的超分辨率图像为所述待训练神经网络输出的所述N张模拟的raw图像对应的第二细节信息与根据所述N张模拟的raw图像得到的第二图像的叠加，所述第二图像为根据所述N张模拟的raw图像通过所述初步操作得到的图像。
初步操作包括：对待处理的N张图像中的任一图像进行插值操作；或者，对第一降噪图像进行插值操作，所述第一降噪图像为对待处理的N张图像进行多帧降噪得到的图像。
N张raw图像的格式可以为拜耳bayer格式或者库卓quadra格式；若所述N张raw图像的格式为quadra格式，则所述第一处理单元在根据所述N张raw图像通过初步操作得到第一图像方面，具有用于：将所述N张raw图像经分箱binning处理转换为bayer格式的N张图像，对转换为bayer格式的所述N张图像执行所述初步操作得到第一图像；若所述N张raw图像的格式为quadra格式，所述N张模拟的raw图像是quadra格式的图像；所述第二图像为将所述N张模拟的raw图像经binning处理转换为bayer格式，对转换后bayer格式的所述N张模拟的raw图像执行所述初步操作得到的图像。
获取单元201具体用于，根据拍照指令对应的变焦倍率将所述摄像头获取的N张图像进行裁剪得到与所述变焦倍率对应的N张raw图像。
请参阅图4，图4为本申请的一个实施例提供的一种终端设备的结构示意图，终端设备400包括：射频单元410、存储器420、输入单元430、摄像头440、音频电路450、处理器460、外部接口470和电源480。其中，输入单元430包括触摸屏431和其他输入设备432，音频电路450包括扬声器451、麦克风452和耳机插孔453。触摸屏431可以是具有触摸功能的显示屏。本实施例中，用户可以通过点击触摸屏431显示的拍照按键触发拍照指令，通过改变与触摸屏接触的距离改变变焦倍率。当处理器460获取拍照指令时，处理器460根据拍照指令对应的变焦倍率将摄像头440获取的N帧原始图像进行裁剪，以得到裁剪后的N张图像，N为1或者大于1的整数。处理器460对N张图像中的M张图像执行初步操作，得到第一图像，M为小于N或者等于N的正整数。处理器460调用神经网络并向神经网络输入裁剪后的N张图像，神经网络对输入的N张图像进行学习，输出第一细节信息处理器460将第一细节信息叠加到第一图像，得到超分辨率图像。进一步地，处理器460将超分辨率图像保存到存储器420，当用户通过触摸屏431触发查看超分辨率图像的指令时，处理器460从存储器420中获取超分辨率图像并通过作为显示界面的触摸屏431显示出来。
神经网络是经过训练后的卷积神经网络，以N张模拟的raw图像为输入，以通过待训 练神经网络得到的所述N张模拟的raw图像对应的超分辨率图像与高清图像的差异为损失函数，对所述待训练神经网络进行训练得到；所述N张模拟的raw图像是通过对所述高清图像进行退化处理，并按照raw图像的数据格式生成的图像；所述N张模拟的raw图像对应的超分辨率图像为所述待训练神经网络输出的所述N张模拟的raw图像对应的第二细节信息与根据所述N张模拟的raw图像得到的第二图像的叠加，所述第二图像为根据所述N张模拟的raw图像通过所述初步操作得到的图像。
本申请实施例还提供了一种计算机可读存储介质，计算机可读存储介质存储有指令和神经网络对应的计算机程序，当所述指令在终端设备上运行时，使得所述终端设备执行前面任一实施例所述图像超分辨重建方法的部分步骤或全部步骤。
本申请实施例还提供了一种计算机程序产品，当所述计算机程序产品在计算机上运行时，使得所述计算机执行图像超分辨重建方法的部分步骤或全部步骤。
上述具体的方法实施例以及实施例中技术特征的解释、表述、以及多种实现形式的扩展也适用于装置中的方法执行，装置实施例中不予以赘述。
应理解以上装置中的各个模块的划分仅仅是一种逻辑功能的划分，实际实现时可以全部或部分集成到一个物理实体上，也可以物理上分开。例如，以上各个模块可以为单独设立的处理元件，也可以集成在终端的某一个芯片中实现，此外，也可以以程序代码的形式存储于控制器的存储元件中，由处理器的某一个处理元件调用并执行以上各个模块的功能。此外各个模块可以集成在一起，也可以独立实现。这里所述的处理元件可以是一种集成电路芯片，具有信号的处理能力。在实现过程中，上述方法的各步骤或以上各个模块可以通过处理器元件中的硬件的集成逻辑电路或者软件形式的指令完成。该处理元件可以是通用处理器，例如处理器(central processing unit，CPU)，还可以是被配置成实施以上方法的一个或多个集成电路，例如：一个或多个特定集成电路(application-specific integrated circuit，ASIC)，或，一个或多个微处理器(digital signal processor，DSP)，或，一个或者多个现场可编程门阵列(field-programmable gate array，FPGA)等。
应理解本发明的说明书和权利要求书及上述附图中的术语“第一”、“第二”等是用于区别类似的对象，而不必用于描述特定的顺序或先后次序。应该理解这样使用的数据在适当情况下可以互换，以便这里描述的实施例能够以除了在这里图示或描述的内容以外的顺序实施。此外，术语“包括”和“具有”以及他们的任何变形，意图在于覆盖不排他的包含，例如，包含了一系列步骤或模块的过程、方法、系统、产品或设备不必限于清楚地列出的那些步骤或模块，而是可包括没有清楚地列出的或对于这些过程、方法、产品或设备固有的其它步骤或模块。
以上所揭露的仅为本发明一种较佳实施例而已，当然不能以此来限定本发明之权利范围，本领域普通技术人员可以理解实现上述实施例的全部或部分流程，并依本发明权利要求所作的等同变化，仍属于发明所涵盖的范围。

Claims (15)

  1. 一种图像超分辨重建方法，其特征在于，包括：
    通过摄像头获取N张原始raw图像，所述N为1或者大于1的整数；
    根据所述N张raw图像通过初步操作，得到第一图像，所述第一图像为彩色的三通道图像；
    将所述N张raw图像输入神经网络，得到所述神经网络输出的第一细节信息；
    将所述第一细节信息叠加到所述第一图像，得到所述N张raw图像对应的超分辨率图像。
  2. 根据权利要求1所述的方法，其特征在于，
    所述神经网络是以N张模拟的raw图像为输入，以通过待训练神经网络得到的所述N张模拟的raw图像对应的超分辨率图像与高清图像的差异为损失函数，对所述待训练神经网络进行训练得到；所述N张模拟的raw图像是通过对所述高清图像进行退化处理，并按照raw图像的数据格式生成的图像；所述N张模拟的raw图像对应的超分辨率图像为所述待训练神经网络输出的所述N张模拟的raw图像对应的第二细节信息与根据所述N张模拟的raw图像得到的第二图像的叠加，所述第二图像为根据所述N张模拟的raw图像通过所述初步操作得到的图像。
  3. 根据权利要求1或2所述的方法，其特征在于，所述初步操作包括：
    对待处理的N张图像中的任一图像进行插值操作；
    或者，对第一降噪图像进行插值操作，所述第一降噪图像为对待处理的N张图像进行多帧降噪得到的图像。
  4. 根据权利要求2所述的方法，其特征在于，所述N张raw图像的格式为拜耳bayer格式或者库卓quadra格式；
    若所述N张raw图像的格式为quadra格式，则所述根据所述N张raw图像通过初步操作得到第一图像，包括：将所述N张raw图像经分箱binning处理转换为bayer格式的N张图像，对转换为bayer格式的所述N张图像执行所述初步操作得到第一图像；
    若所述N张raw图像的格式为quadra格式，所述N张模拟的raw图像是quadra格式的图像；所述第二图像为将所述N张模拟的raw图像经binning处理转换为bayer格式，对转换后bayer格式的所述N张模拟的raw图像执行所述初步操作得到的图像。
  5. 根据权利要求1至4任一项所述的方法，其特征在于，所述通过摄像头获取N张raw图像包括：
    根据拍照指令对应的变焦倍率对摄像头获取的N张图像进行裁剪得到与所述变焦倍率对应的N张raw图像。
  6. 一种图像超分辨重建装置，其特征在于，包括：
    获取单元，用于通过摄像头获取N张原始raw图像，所述N为1或者大于1的整数；
    第一处理单元，用于根据所述N张raw图像通过初步，操作得到第一图像，所述第一图像为彩色的三通道图像；
    第二处理单元，用于将所述N张raw图像输入神经网络，得到所述神经网络输出的第一细节信息；
    叠加单元，用于将所述第一细节信息叠加到所述第一图像，得到所述N张raw图像对应的超分辨率图像。
  7. 根据权利要求6所述的装置，其特征在于，
    所述神经网络是以N张模拟的raw图像为输入，以通过待训练神经网络得到的所述N张模拟的raw图像对应的超分辨率图像与高清图像的差异为损失函数，对所述待训练神经网络进行训练得到；所述N张模拟的raw图像是通过对所述高清图像进行退化处理，并按照raw图像的数据格式生成的图像；所述N张模拟的raw图像对应的超分辨率图像为所述待训练神经网络输出的所述N张模拟的raw图像对应的第二细节信息与根据所述N张模拟的raw图像得到的第二图像的叠加，所述第二图像为根据所述N张模拟的raw图像通过所述初步操作得到的图像。
  8. 根据权利要求6或7所述的装置，其特征在于，所述初步操作包括：
    对待处理的N张图像中的任一图像进行插值操作；
    或者，对第一降噪图像进行插值操作，所述第一降噪图像为对待处理的N张图像进行多帧降噪得到的图像。
  9. 根据权利要求7所述的方法，其特征在于，所述N张raw图像的格式为拜耳bayer格式或者库卓quadra格式；
    若所述N张raw图像的格式为quadra格式，则所述第一处理单元在根据所述N张raw图像通过初步操作得到第一图像方面，具有用于：将所述N张raw图像经分箱binning处理转换为bayer格式的N张图像，对转换为bayer格式的所述N张图像执行所述初步操作得到第一图像；
    若所述N张raw图像的格式为quadra格式，所述N张模拟的raw图像是quadra格式的图像；所述第二图像为将所述N张模拟的raw图像经binning处理转换为bayer格式，对转换后bayer格式的所述N张模拟的raw图像执行所述初步操作得到的图像。
  10. 根据权利要求6至9任一项所述的装置，其特征在于，
    所述获取单元具体用于，根据拍照指令对应的变焦倍率将所述摄像头获取的N张图像进行裁剪得到与所述变焦倍率对应的N张raw图像。
  11. 一种终端设备，其特征在于，包括摄像头、处理器和存储器，其特征在于，
    所述摄像头，用于在所述处理器获取拍照指令后，获取N张原始raw图像，所述N为1或者大于1的整数；
    所述存储器，用于存储可在所述处理器上运行的计算机程序；
    所述处理器，用于执行如权利要求1至5中任一项所述的方法。
  12. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储有指令和神经网络对应的计算机程序，当所述指令在终端设备上运行时，使得所述终端设备执行如权利要求1至5中任一项所述的方法。
  13. 一种终端设备，其特征在于，包括摄像头、处理器和神经网络单元；
    所述摄像头，用于在所述处理器获取拍照指令后，获取N张原始raw图像，所述N为1或者大于1的整数；
    所述神经网络单元，用于以所述N张raw图像为输入，得到第一细节信息；
    所述处理器，用于执行如权利要求1至5中任一项所述的方法。
  14. 一种计算机可读存储介质，其特征在于，所述计算机可读存储介质存储有计算机程序，所述计算机程序包括程序指令，所述程序指令当被处理器执行时，使所述处理器执行如权利要求1至5中任一项所述的方法。
  15. 一种计算机程序产品，其特征在于，所述计算机程序产品包括存储了计算机程序的计算机可读存储介质，该计算机程序使得计算机执行如权利要求1至5中任一项所述的方法的部分或全部步骤。
PCT/CN2020/079880 2019-03-18 2020-03-18 一种图像超分辨重建方法、装置和终端设备 WO2020187220A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910205538.9A CN111724448A (zh) 2019-03-18 2019-03-18 一种图像超分辨重建方法、装置和终端设备
CN201910205538.9 2019-03-18

Publications (1)

Publication Number Publication Date
WO2020187220A1 true WO2020187220A1 (zh) 2020-09-24

Family

ID=72519585

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2020/079880 WO2020187220A1 (zh) 2019-03-18 2020-03-18 一种图像超分辨重建方法、装置和终端设备

Country Status (2)

Country Link
CN (1) CN111724448A (zh)
WO (1) WO2020187220A1 (zh)

Families Citing this family (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN114338958B (zh) * 2020-09-30 2023-07-11 华为技术有限公司 一种图像处理的方法及相关设备
WO2022141265A1 (zh) * 2020-12-30 2022-07-07 华为技术有限公司 图像处理方法和设备
CN113689335B (zh) * 2021-08-24 2024-05-07 Oppo广东移动通信有限公司 图像处理方法与装置、电子设备及计算机可读存储介质

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8705828B2 (en) * 2011-08-31 2014-04-22 Carestream Health, Inc. Methods and apparatus for super resolution scanning for CBCT system and cone-beam image reconstruction
US20150363634A1 (en) * 2014-06-17 2015-12-17 Beijing Kuangshi Technology Co.,Ltd. Face Hallucination Using Convolutional Neural Networks
US20170019615A1 (en) * 2015-07-13 2017-01-19 Asustek Computer Inc. Image processing method, non-transitory computer-readable storage medium and electrical device thereof
CN107492070A (zh) * 2017-07-10 2017-12-19 华北电力大学 一种双通道卷积神经网络的单图像超分辨率计算方法
CN108391060A (zh) * 2018-03-26 2018-08-10 华为技术有限公司 一种图像处理方法、图像处理装置和终端

Family Cites Families (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US5907640A (en) * 1993-03-25 1999-05-25 Live Picture, Inc. Functional interpolating transformation system for image processing
CN107767343B (zh) * 2017-11-09 2021-08-31 京东方科技集团股份有限公司 图像处理方法、处理装置和处理设备
CN109360151B (zh) * 2018-09-30 2021-03-05 京东方科技集团股份有限公司 图像处理方法及系统、分辨率提升方法、可读存储介质

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US8705828B2 (en) * 2011-08-31 2014-04-22 Carestream Health, Inc. Methods and apparatus for super resolution scanning for CBCT system and cone-beam image reconstruction
US20150363634A1 (en) * 2014-06-17 2015-12-17 Beijing Kuangshi Technology Co.,Ltd. Face Hallucination Using Convolutional Neural Networks
US20170019615A1 (en) * 2015-07-13 2017-01-19 Asustek Computer Inc. Image processing method, non-transitory computer-readable storage medium and electrical device thereof
CN107492070A (zh) * 2017-07-10 2017-12-19 华北电力大学 一种双通道卷积神经网络的单图像超分辨率计算方法
CN108391060A (zh) * 2018-03-26 2018-08-10 华为技术有限公司 一种图像处理方法、图像处理装置和终端

Also Published As

Publication number Publication date
CN111724448A (zh) 2020-09-29

Similar Documents

Publication Publication Date Title
CN111641778B (zh) 一种拍摄方法、装置与设备
WO2020187220A1 (zh) 一种图像超分辨重建方法、装置和终端设备
JP5784587B2 (ja) 画像選択および結合の方法およびデバイス
US9363493B2 (en) Image processing apparatus, method, recording medium and image pickup apparatus
JP5464656B2 (ja) 画質評価装置、端末装置、画質評価システム、画質評価方法及びプログラム
JP5867629B2 (ja) 画像処理のためのシステム及び方法
JP5933105B2 (ja) 画像処理装置、撮像装置、フィルタ生成装置、画像復元方法及びプログラム
JP5775977B2 (ja) 画像処理装置、撮像装置、画像処理方法、及び画像処理プログラム
TW201003566A (en) Interpolation system and method
US20150116464A1 (en) Image processing apparatus and image capturing apparatus
WO2013005727A1 (ja) 撮像装置及び記録媒体
US11290635B2 (en) Imaging apparatus and image processing method
US20200412967A1 (en) Imaging element and imaging apparatus
EP2847998B1 (en) Systems, methods, and computer program products for compound image demosaicing and warping
US10559068B2 (en) Image processing device, image processing method, and program processing image which is developed as a panorama
US9374544B2 (en) Image capture device, image processing method, and program
CN113287147A (zh) 一种图像处理方法及装置
CN117440241A (zh) 一种视频处理方法及装置
CN113689335A (zh) 图像处理方法与装置、电子设备及计算机可读存储介质
CN117412185A (zh) 图像生成方法、装置、电子设备和计算机可读存储介质
CN117333366A (zh) 图像处理方法、装置和电子设备
WO2017086021A1 (ja) 画像処理装置、画像処理方法及びコンピュータプログラム
CN116012222A (zh) 图像处理方法、装置、电子设备及存储介质
CN118155009A (zh) 训练数据的生成方法、装置、电子设备和存储介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20774070

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20774070

Country of ref document: EP

Kind code of ref document: A1