WO2020233129A1 - 一种图像超分辨和着色方法、系统及电子设备 - Google Patents

一种图像超分辨和着色方法、系统及电子设备 Download PDF

Info

Publication number
WO2020233129A1
WO2020233129A1 PCT/CN2019/130536 CN2019130536W WO2020233129A1 WO 2020233129 A1 WO2020233129 A1 WO 2020233129A1 CN 2019130536 W CN2019130536 W CN 2019130536W WO 2020233129 A1 WO2020233129 A1 WO 2020233129A1
Authority
WO
WIPO (PCT)
Prior art keywords
resolution
module
feature map
layer
image
Prior art date
Application number
PCT/CN2019/130536
Other languages
English (en)
French (fr)
Inventor
邬晶晶
张涌
周阳
姜元爽
Original Assignee
深圳先进技术研究院
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 深圳先进技术研究院 filed Critical 深圳先进技术研究院
Publication of WO2020233129A1 publication Critical patent/WO2020233129A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T11/00—2D [Two Dimensional] image generation
    • G06T11/001—Texturing; Colouring; Generation of texture or colour
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4046—Scaling of whole images or parts thereof, e.g. expanding or contracting using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution

Definitions

  • This application belongs to the field of image processing technology, and in particular relates to an image super-resolution and coloring method, system and electronic equipment.
  • Image super-resolution technology refers to the restoration of high-resolution images from a low-resolution image or image sequence. This technology was first proposed in the field of optics, and is now widely used in the field of image compression, medical imaging, and remote sensing imaging.
  • Traditional super-resolution techniques include: interpolation-based methods (such as: nearest neighbor interpolation, bilinear interpolation, bicubic interpolation, etc.), reconstruction-based methods (such as: non-uniform interpolation, iterative back projection method, maximum Posterior probability method, etc.) etc. At present, the best effect is the deep learning method.
  • Image coloring is the process of pseudo-coloring black and white grayscale images to make the image more visually perceptive and attractive.
  • Image coloring technology has applications in industrial production, medical image processing, early film and television works and old photos. Since the changes from grayscale to color images are diverse, this also makes this task a difficult task.
  • Common image coloring methods can be divided into image coloring methods based on local colors and coloring image methods based on color transfer. Deep learning techniques are constantly being applied to the image coloring process, and more and more models have emerged, including the article [Wang F Y, Zhang J, Zheng X, et al.
  • the article First calculate the chromaticity diagram, which adjusts the spatial position of the chromaticity samples provided by the low-resolution input image. The chromaticity diagram is then used to color the final result based on the super-resolution luminance channel.
  • the article When applying this method to learning-based super-resolution technology, the article also introduces a back-projection step to first normalize the luminance channel before the image is colored.
  • This application provides an image super-resolution and coloring method, system, and electronic device, which aim to solve at least one of the above technical problems in the prior art to a certain extent.
  • An image super-resolution and coloring method including the following steps:
  • Step a Design a new network model combining image super-resolution and coloring, the network model including an encoding module and a decoding module;
  • Step b Input the low-resolution grayscale image into the network model
  • Step c Extract the semantic feature map of the low-resolution grayscale image through the encoding module, and pass the semantic feature map to the decoding module, and the decoding module superimposes the semantic feature maps of each level and outputs the high-resolution color Figure.
  • the encoding module includes a 13-layer convolutional layer, a 5-layer pooling layer, and a 3-layer fully connected layer, and the convolution kernel of each convolutional layer Respectively 3*3, the convolution output channels of the first and second layers are 64, the convolution output channels of the third and fourth layers are 128, and the convolution output channels of the fifth, sixth, and seventh layers are 256.
  • the convolution output channels of the 9th and 10th layers are 512, and the convolutional output channels of the 11th, 12th, and 13th layers are 512; the downsampling rate of the pooling layer is 2, and the first four pooling layers are the maximum pooling.
  • the final pooling layer is global average pooling; the numbers of nodes in the 3 fully connected layers are 4096, 4096, and 1000 respectively.
  • the decoding module includes an up-sampling module, a hole residual module, and a mixed up-sampling module, the number of the up-sampling module and the hole residual module
  • the hole residual module includes two multi-branch convolution modules with varying void rates
  • each convolution module includes 4 branches, and each branch is compressed by a common convolution first.
  • the hole rates of the four branch hole convolutions are 1, 2, 3, and 5 respectively
  • the hybrid upsampling module includes a bilinear difference branch with fixed parameters and a parameter learnable inverse Convolution branch.
  • the technical solution adopted in the embodiment of the application further includes: in the step c, the encoding module extracts the semantic feature map of the low-resolution grayscale image, and transmits the semantic feature map to the decoding module, the decoding module To superimpose the semantic feature maps of each level and output the high-resolution color image is specifically: extracting the semantic feature map of the low-resolution gray-scale image through the 13-layer convolutional layer, and passing through the 5-layer pooling layer and the 3-layer fully connected layer After that, the semantic feature maps A, B, C, D, and E output by the second, fourth, seventh, tenth, and 13th layers of convolution are retained and transmitted to the decoding module in turn.
  • the feature map E passes through the hole residual module and The feature map output after the mixed upsampling module is stacked with the feature map output by the feature map D after passing through the hole residual module.
  • the stacked feature map is output after the hole residual module and the mixed upsampling module, and then The feature maps of the upper layer of feature map C after passing through the hole residual module are stacked and proceeded in sequence, until finally the feature maps generated after the feature maps A, B, C, D, and E are passed through the decoding module and the low-resolution gray
  • the feature map generated after the degree map passes through the mixed up-sampling module is added to the output of the high-resolution color map.
  • an image super-resolution and coloring system including designing a new network model combining image super-resolution and coloring.
  • the network model includes an encoding module and a decoding module;
  • the image is input to the network model, the semantic feature map of the low-resolution grayscale image is extracted through the encoding module, and the semantic feature map is passed to the decoding module, which superimposes the semantic feature maps of each level, and Output high-resolution color images.
  • the encoding module includes a 13-layer convolutional layer, a 5-layer pooling layer, and a 3-layer fully connected layer.
  • the convolution kernel of each convolutional layer is 3*3.
  • the convolution output channel of layer 1, 2 is 64
  • the convolution output channel of layer 3, 4 is 128, the convolution output channel of layer 5, 6, and 7 is 256
  • the output channel is 512, and the convolution output channel of the 11th, 12th, and 13th layers is 512; the downsampling rate of the pooling layer is 2, the first four pooling layers are maximum pooling, and the last pooling layer is global Average pooling; the number of nodes in the 3 fully connected layers are 4096, 4096, and 1000 respectively.
  • the technical solution adopted in the embodiment of the present application further includes: the decoding module includes an up-sampling module, a hole residual module, and a hybrid up-sampling module, the number of the up-sampling module and the hole residual module is at least two;
  • the void residual module includes two multi-branch convolution modules with varying void rates.
  • Each convolution module includes 4 branches. Each branch is compressed by a common convolution first, and then undergoes a hole convolution.
  • the hole rates of the four branch hole convolutions are 1, 2, 3, and 5 respectively;
  • the hybrid up-sampling module includes a bilinear difference branch with fixed parameters and a deconvolution branch with learnable parameters.
  • the technical solution adopted in the embodiment of the application further includes: the image super-resolution and coloring method of the network model is specifically: extracting the semantic feature map of the low-resolution grayscale image through the 13-layer convolutional layer, and passing through the 5-layer pooling layer After the fully connected layer and the 3rd layer, the semantic feature maps A, B, C, D, and E output by the second, fourth, seventh, 10th, and 13th layers of convolution are retained and transmitted to the decoding module in turn, the feature map E
  • the feature map output after the hole residual module and the hybrid upsampling module is stacked with the feature map output after the feature map D passes through the hole residual module.
  • the stacked feature map is output after the hole residual module and the hybrid upsampling module.
  • the feature map of the previous layer is stacked with the feature map generated by the previous layer of feature map C after passing through the hole residual module, and proceeded in turn, until the feature map generated by the feature map A, B, C, D, E through the decoding module
  • the addition of the low-resolution gray-scale image to the feature map generated after the hybrid up-sampling module is the output of the high-resolution color image.
  • an electronic device including:
  • At least one processor At least one processor
  • a memory communicatively connected with the at least one processor; wherein,
  • the memory stores instructions that can be executed by the one processor, and the instructions are executed by the at least one processor, so that the at least one processor can perform the following operations of the above-mentioned image super-resolution and rendering method:
  • Step a Design a new network model combining image super-resolution and coloring, the network model including an encoding module and a decoding module;
  • Step b Input the low-resolution grayscale image into the network model
  • Step c Extract the semantic feature map of the low-resolution grayscale image through the encoding module, and pass the semantic feature map to the decoding module, and the decoding module superimposes the semantic feature maps of each level and outputs the high-resolution color Figure.
  • the beneficial effects produced by the embodiments of the present application are: the image super-resolution and coloring methods, systems and electronic devices of the embodiments of the present application design new network models for super-resolution and coloring problems, and perform super-resolution and coloring tasks.
  • Common processing direct mapping of low-resolution grayscale images to high-resolution color images, improve people's visual perception, save resources and time.
  • the network model of the present application can optimize all parameters together without biasing to a certain sub-model.
  • Fig. 1 is a flowchart of an image super-resolution and coloring method according to an embodiment of the present application
  • Figure 2 is a schematic structural diagram of a network model of an embodiment of the application
  • FIG. 3 is a structural diagram of a cavity residual module according to an embodiment of the application.
  • FIG. 4 is a schematic structural diagram of an image super-resolution and coloring system according to an embodiment of the present application.
  • FIG. 5 is a schematic diagram of the hardware device structure of an image super-resolution and coloring method provided by an embodiment of the present application.
  • FIG. 1 is a flowchart of an image super-resolution and coloring method according to an embodiment of the present application.
  • the image super-resolution and coloring method of the embodiment of the present application includes the following steps:
  • Step 100 Design a new network model combining image super-resolution and coloring
  • step 100 is a schematic structural diagram of a network model according to an embodiment of this application.
  • the network model of the embodiment of the present application includes an encoding module and a decoding module.
  • the encoding module is VGG16 (a network model).
  • the VGG16 has been pre-trained on a large amount of data on the ImageNet database and has a certain ability to extract semantic information.
  • the coding module includes a 13-layer convolutional layer, a 5-layer pooling layer, and a 3-layer fully connected layer.
  • the convolution kernel of each convolutional layer is 3*3, the convolution output channel of the first and second layers is 64, the convolution output channel of the third and fourth layers is 128, and the convolution of the fifth, sixth, and seventh layers
  • the output channel is 256, the convolution output channel of the 8, 9, and 10 layers is 512, and the convolution output channel of the 11, 12, and 13 layers is 512.
  • the downsampling rate of the pooling layer is 2, the first four pooling layers are the maximum pooling, and the last pooling layer is the global average pooling, and then through 3 fully connected layers, the number of nodes in the 3 fully connected layers are respectively It is 4096, 4096, 1000.
  • the decoding module includes a multi-layer up-sampling module (the specific number depends on the task), multiple hole residual modules (the specific number depends on the task), and a mixed up-sampling module.
  • the up-sampling module and the hole residual module The number is at least two, and the specific number can be set according to the actual task.
  • FIG. 3 together is a structural diagram of a cavity residual module according to an embodiment of the application.
  • the cavity residual module mainly includes two multi-branch convolution modules with varying cavity rates.
  • Each convolution module includes 4 branches, and each branch is compressed by a common convolution first, and then undergoes a hole convolution.
  • the hole rates of the four branch hole convolutions are 1, 2, 3, and 5 respectively.
  • the hybrid up-sampling module includes a bilinear difference branch with fixed parameters and a deconvolution branch with learnable parameters, and finally the feature map after the fusion of the two branches is input.
  • Step 200 Input the low-resolution grayscale image into the network model
  • Step 300 Extract the semantic feature map of the low-resolution grayscale image through the encoding module of the network model, and pass the semantic feature map to the decoding module.
  • the decoding module uses the hole residual module and the hybrid up-sampling module to analyze the semantic feature maps of each level Perform superimposition and output color high-resolution images.
  • step 300 it is assumed that the semantic feature maps output by the second, fourth, seventh, tenth, and 13th layers of convolution are A, B, C, D, and E, respectively.
  • the feature map overlay method is specifically: the semantic feature maps A, B, C, D, and E are sequentially transmitted to at least two void residual modules. Since the sizes of semantic feature maps A, B, C, D, and E are different, each time the feature maps are stacked through the mixed up-sampling module, the smaller The feature map is increased by a factor of 2 (all upsampling rates are 2).
  • Feature map E is the feature map output after passing through the hole residual module and the mixed up-sampling module, and is stacked with the feature map output by the feature map D after passing through the hole residual module.
  • the stacked feature map passes through the hole residual module and mixed up-sampling
  • the feature map output after the module is stacked with the feature map of the previous layer of feature map C after passing through the hole residual module, and then proceeding in turn, until the final feature map A, B, C, D, E are generated after the decoding module
  • the feature map and the input low-resolution grayscale image are added after the mixed up-sampling module to produce the feature map, which is the output of the high-resolution color image.
  • the network model continuously updates the network parameters after multiple training on the training set, and finally the network model can learn the mapping function from low-resolution grayscale image to high-resolution color image to a certain extent.
  • FIG. 4 is a schematic structural diagram of an image super-resolution and coloring system according to an embodiment of the present application.
  • the image super-resolution and coloring system of the embodiment of the application is a new network model designed by combining image super-resolution and coloring.
  • the network model includes an encoding module and a decoding module.
  • the low-resolution grayscale image is input to the network model through the encoding module Extract the semantic feature map of the low-resolution grayscale image, and pass the semantic feature map to the decoding module.
  • the decoding module uses the hole residual module and the hybrid up-sampling module to superimpose the semantic feature maps of each level, and output the color high-resolution image.
  • the coding module of the network model is VGG16 (a network model).
  • VGG16 has been pre-trained on a large amount of data on the ImageNet database and has a certain ability to extract semantic information.
  • the coding module includes a 13-layer convolutional layer, a 5-layer pooling layer, and a 3-layer fully connected layer.
  • the convolution kernel of each convolutional layer is 3*3, the convolution output channel of the first and second layers is 64, the convolution output channel of the third and fourth layers is 128, and the convolution of the fifth, sixth, and seventh layers
  • the output channel is 256, the convolution output channel of the 8, 9, and 10 layers is 512, and the convolution output channel of the 11, 12, and 13 layers is 512.
  • the downsampling rate of the pooling layer is 2, the first four pooling layers are the maximum pooling, and the last pooling layer is the global average pooling, and then through 3 fully connected layers, the number of nodes in the 3 fully connected layers are respectively It is 4096, 4096, 1000.
  • Extract the semantic feature map of the input image through the 13-layer convolutional layer, and after the 5-layer pooling layer and the 3-layer fully connected layer, the semantic feature maps of the second, fourth, seventh, 10th, and 13th convolutional output are retained respectively Down, and in turn lost to the decoding module.
  • the decoding module includes a multi-layer up-sampling module (the specific number depends on the task), multiple hole residual modules (the specific number depends on the task), and a mixed up-sampling module.
  • the up-sampling module and the hole residual module The number is at least two, and the specific number can be set according to the actual task.
  • the cavity residual module mainly includes two multi-branch convolution modules with varying cavity rates. Each convolution module includes 4 branches, and each branch is compressed by a common convolution first, and then undergoes a hole convolution.
  • the hole rates of the four branch hole convolutions are 1, 2, 3, and 5 respectively.
  • the hybrid up-sampling module includes a bilinear difference branch with fixed parameters and a deconvolution branch with learnable parameters, and finally the feature map after the fusion of the two branches is input.
  • the image super-resolution and coloring method of the network model is specifically as follows: assuming that the semantic feature maps output by the second, fourth, seventh, tenth, and thirteenth layers of convolution are A, B, C, D, and E, respectively, the semantics Feature maps A, B, C, D, and E are transferred to at least two hollow residual modules in turn. Since the sizes of semantic feature maps A, B, C, D, and E are different, each time the feature maps are stacked before they are mixed Sampling module to increase the small feature map by 2 times (all upsampling rates are 2).
  • Feature map E is the feature map output after passing through the hole residual module and the mixed up-sampling module, and is stacked with the feature map output by the feature map D after passing through the hole residual module.
  • the stacked feature map passes through the hole residual module and mixed up-sampling
  • the feature map output after the module is stacked with the feature map of the previous layer of feature map C after passing through the hole residual module, and then proceeding in turn, until the final feature map A, B, C, D, E are generated after the decoding module
  • the feature map and the input low-resolution grayscale image are added after the mixed up-sampling module to produce the feature map, which is the output of the high-resolution color image.
  • the network model continuously updates the network parameters after multiple training on the training set, and finally the network model can learn the mapping function from low-resolution grayscale image to high-resolution color image to a certain extent.
  • FIG. 5 is a schematic diagram of the hardware device structure of an image super-resolution and coloring method provided by an embodiment of the present application.
  • the device includes one or more processors and memory. Taking a processor as an example, the device may also include: an input system and an output system.
  • the processor, the memory, the input system, and the output system may be connected by a bus or other methods.
  • the connection by a bus is taken as an example.
  • the memory can be used to store non-transitory software programs, non-transitory computer executable programs, and modules.
  • the processor executes various functional applications and data processing of the electronic device by running non-transitory software programs, instructions, and modules stored in the memory, that is, realizing the processing methods of the foregoing method embodiments.
  • the memory may include a program storage area and a data storage area, where the program storage area can store an operating system and an application program required by at least one function; the data storage area can store data and the like.
  • the memory may include a high-speed random access memory, and may also include a non-transitory memory, such as at least one magnetic disk storage device, a flash memory device, or other non-transitory solid state storage devices.
  • the storage may optionally include storage remotely arranged with respect to the processor, and these remote storages may be connected to the processing system through a network. Examples of the aforementioned networks include, but are not limited to, the Internet, corporate intranets, local area networks, mobile communication networks, and combinations thereof.
  • the input system can receive input digital or character information, and generate signal input.
  • the output system may include display devices such as a display screen.
  • the one or more modules are stored in the memory, and when executed by the one or more processors, the following operations of any of the foregoing method embodiments are performed:
  • Step a Design a new network model combining image super-resolution and coloring, the network model including an encoding module and a decoding module;
  • Step b Input the low-resolution grayscale image into the network model
  • Step c Extract the semantic feature map of the low-resolution grayscale image through the encoding module, and pass the semantic feature map to the decoding module, and the decoding module superimposes the semantic feature maps of each level and outputs the high-resolution color Figure.
  • the embodiment of the present application provides a non-transitory (nonvolatile) computer storage medium, the computer storage medium stores computer executable instructions, and the computer executable instructions can perform the following operations:
  • Step a Design a new network model combining image super-resolution and coloring, the network model including an encoding module and a decoding module;
  • Step b Input the low-resolution grayscale image into the network model
  • Step c Extract the semantic feature map of the low-resolution grayscale image through the encoding module, and pass the semantic feature map to the decoding module, and the decoding module superimposes the semantic feature maps of each level and outputs the high-resolution color Figure.
  • the embodiment of the present application provides a computer program product, the computer program product includes a computer program stored on a non-transitory computer-readable storage medium, the computer program includes program instructions, when the program instructions are executed by a computer To make the computer do the following:
  • Step a Design a new network model combining image super-resolution and coloring, the network model including an encoding module and a decoding module;
  • Step b Input the low-resolution grayscale image into the network model
  • Step c Extract the semantic feature map of the low-resolution grayscale image through the encoding module, and pass the semantic feature map to the decoding module, and the decoding module superimposes the semantic feature maps of each level and outputs the high-resolution color Figure.
  • the image super-resolution and coloring method, system and electronic device of the embodiments of the application design a new network model for the super-resolution and coloring problem, and jointly process the super-resolution and coloring tasks, and directly map the low-resolution grayscale image to the high-resolution image. Color map, improve people's visual perception, save resources and time.
  • the network model of the present application can optimize all parameters together without biasing to a certain sub-model.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Image Processing (AREA)
  • Image Analysis (AREA)
  • Compression Of Band Width Or Redundancy In Fax (AREA)
  • Compression Or Coding Systems Of Tv Signals (AREA)

Abstract

一种图像超分辨和着色方法、系统及电子设备。所述方法包括：步骤a：结合图像超分辨和着色设计新的网络模型，所述网络模型包括编码模块和解码模块；步骤b：将低分辨灰度图输入网络模型；步骤c：通过所述编码模块提取低分辨灰度图像的语义特征图，并将所述语义特征图传入解码模块，所述解码模块对各个层级的语义特征图进行叠加，并输出高分辨彩色图。该方法针对超分辨与着色问题设计新的网络模型，对超分辨和着色任务共同处理，将低分辨的灰度图直接映射到高分辨的彩色图，提高人们视觉观感，节省资源和时间。

Description

一种图像超分辨和着色方法、系统及电子设备 技术领域
本申请属于图像处理技术领域，特别涉及一种图像超分辨和着色方法、系统及电子设备。
背景技术
在图像处理领域，图像的超分辨和着色一直是两个热门的研究领域。图像超分辨率技术是指由一幅低分辨率图像或图像序列恢复出高分辨率图像。该技术最早在光学领域被提出，现在广泛应用于图像压缩领域、医学成像领域、遥感成像领域等。传统的超分辨技术包括：基于插值的方法(如：最近邻插值法，双线性插值法，双三次插值法等)、基于重建的方法(如：非均匀插值法、迭代反投影法、最大后验概率法等)等。目前效果最好的是深度学习的方法，参考文献[Dong C,Loy C C,He K,et al.Image super-resolution using deep convolutional networks[J].IEEE transactions on pattern analysis and machine intelligence,2016,38(2):295-307.]首次将卷积神经网络应用于图像超分辨领域，仅仅用了一个三层的卷积神经网络模型进行学习，就在重建质量和重建效率上远远的超越了现有的方法。后来越来越多的网络结构被提出，其中包括：FSRCNN[Dong C,Loy C C,Tang X.Accelerating the super-resolution convolutional neural network[C]//European conference on computer vision.Springer,Cham,2016:391-407]、SRDenseNet[Tong T,Li G,Liu X,et al.Image super-resolution using dense skip connections[C]//Proceedings of the IEEE International Conference on  Computer Vision.2017:4799-4807.]、SRGAN[Ledig C,Theis L,Huszár F,et al.Photo-realistic single image super-resolution using a generative adversarial network[C]//Proceedings of the IEEE conference on computer vision and pattern recognition.2017:4681-4690.]等。
图像着色是对黑白灰度图像进行伪彩色化的过程，使图像更具视觉感知力和吸引力。图像着色技术在工业生产、医学影像处理、早期的影视作品及老旧照片方面都有应用。由于灰度图到彩色图的变化是多样的，这也使得该任务成为一个难题。常见的图像着色方法可以分为基于局部颜色图像着色方法和基于颜色传递着色图像方法，深度学习的技术也不断被应用到图像的着色过程中，涌现越来越多的模型，包括文章[Wang F Y,Zhang J J,Zheng X,et al.Where does AlphaGo go:From church-turing thesis to AlphaGo thesis and beyond[J].IEEE/CAA Journal of Automatica Sinica,2016,3(2):113-120]、[Zhang R,Isola P,Efros A A.Colorful image colorization[C]//European conference on computer vision.Springer,Cham,2016:649-666]、[He M,Chen D,Liao J,et al.Deep exemplar-based colorization[J].ACM Transactions on Graphics(TOG),2018,37(4):47]等。
文章[Liu S,Brown M S,Kim S J,et al.Colorization for single image super resolution[C]//European Conference on Computer Vision.Springer,Berlin,Heidelberg,2010:323-336.]提出一种处理单图像超分辨率的新方法，与色度上采样一样，该方法只对亮度通道应用超分辨率。该方法的不同之处在于使用图像着色来分配色度值。首先计算一个色度图，它调整低分辨输入图像提供的色度样本的空间位置，然后使用色度图对基于超分辨亮度通道的最终结果进行着色。首先计算色度图，该图调整低分辨输入图像提供的色度样本的空间位置。然后使用色度图来基于超分辨亮度通道对最终结果着色。在将这种方法应用于 基于学习的超分辨技术时，文章还引入了反投影步骤，以在图像着色之前首先标准化亮度通道。
综上所述，现有技术中，对于一些既需要对图像进行超分辨，也要对其着色的任务时(如：处理一些老旧照片及视频)，需要分别对图像超分辨和着色单独建模，训练出两个网络模型，将这两个任务分开处理，既浪费资源、又浪费时间。同时，无论先超分辨再着色，还是先着色再超分辨，由于前一个任务产的误差，对于后一个任务而言，难度都很大，不利于恢复出高分辨的彩色图。
发明内容
本申请提供了一种图像超分辨和着色方法、系统及电子设备，旨在至少在一定程度上解决现有技术中的上述技术问题之一。
为了解决上述问题，本申请提供了如下技术方案：
一种图像超分辨和着色方法，包括以下步骤：
步骤a：结合图像超分辨和着色设计新的网络模型，所述网络模型包括编码模块和解码模块；
步骤b：将低分辨灰度图输入网络模型；
步骤c：通过所述编码模块提取低分辨灰度图像的语义特征图，并将所述语义特征图传入解码模块，所述解码模块对各个层级的语义特征图进行叠加，并输出高分辨彩色图。
本申请实施例采取的技术方案还包括：在所述步骤a中，所述编码模块包括13层卷积层、5层池化层和3层全连接层，每层卷积层的卷积核分别为3*3，第1、2层的卷积输出通道为64，第3、4层的卷积输出通道为128，第5、6、 7层的卷积输出通道为256，第8、9、10层的卷积输出通道为512，第11、12、13层的卷积输出通道为512；所述池化层的下采样率为2，前四个池化层为最大池化，最后的池化层为全局平均池化；所述3个全连接层的节点个数分别为4096、4096、1000。
本申请实施例采取的技术方案还包括：在所述步骤a中，所述解码模块包括上采样模块、空洞残差模块和混合上采样模块，所述上采样模块和空洞残差模块的个数为至少两个；所述空洞残差模块包括两个多分支结构的空洞率变化的卷积模块，每个卷积模块分别包括4个分支，每个分支分别由一个普通卷积先压缩维度，再经过一个空洞卷积，四个分支空洞卷积的空洞率分别为1、2、3、5；所述混合上采样模块包括一个固定参数的双线性差值分支和一个参数可学习的反卷积分支。
本申请实施例采取的技术方案还包括：在所述步骤c中，所述通过编码模块提取低分辨灰度图像的语义特征图，并将所述语义特征图传入解码模块，所述解码模块对各个层级的语义特征图进行叠加，并输出高分辨彩色图具体为：通过所述13层卷积层提取低分辨灰度图的语义特征图，经过5层池化层和3层全连接层后，分别将第2、4、7、10、13层卷积输出的语义特征图A、B、C、D、E保留下来，并依次传输至解码模块，特征图E经过空洞残差模块和混合上采样模块之后输出的特征图，与特征图D经过空洞残差模块后输出的特征图相堆叠，堆叠后的特征图经过空洞残差模块和混合上采样模块之后输出的特征图，再和上一层特征图C经过空洞残差模块后产生的特征图相堆叠，依次进行，直到最后将特征图A、B、C、D、E经过解码模块后产生的特征图与所述低分辨灰度图经过混合上采样模块之后产生的特征图相加，即为高分辨率彩色图的输出。
本申请实施例采取的另一技术方案为：一种图像超分辨和着色系统，包括结合图像超分辨和着色设计新的网络模型，所述网络模型包括编码模块和解码模块；将低分辨灰度图输入所述网络模型，通过所述编码模块提取低分辨灰度图像的语义特征图，并将所述语义特征图传入解码模块，所述解码模块对各个层级的语义特征图进行叠加，并输出高分辨彩色图。
本申请实施例采取的技术方案还包括：所述编码模块包括13层卷积层、5层池化层和3层全连接层，每层卷积层的卷积核分别为3*3，第1、2层的卷积输出通道为64，第3、4层的卷积输出通道为128，第5、6、7层的卷积输出通道为256，第8、9、10层的卷积输出通道为512，第11、12、13层的卷积输出通道为512；所述池化层的下采样率为2，前四个池化层为最大池化，最后的池化层为全局平均池化；所述3个全连接层的节点个数分别为4096、4096、1000。
本申请实施例采取的技术方案还包括：所述解码模块包括上采样模块、空洞残差模块和混合上采样模块，所述上采样模块和空洞残差模块的个数为至少两个；所述空洞残差模块包括两个多分支结构的空洞率变化的卷积模块，每个卷积模块分别包括4个分支，每个分支分别由一个普通卷积先压缩维度，再经过一个空洞卷积，四个分支空洞卷积的空洞率分别为1、2、3、5；所述混合上采样模块包括一个固定参数的双线性差值分支和一个参数可学习的反卷积分支。
本申请实施例采取的技术方案还包括：所述网络模型的图像超分辨和着色方式具体为：通过所述13层卷积层提取低分辨灰度图的语义特征图，经过5层池化层和3层全连接层后，分别将第2、4、7、10、13层卷积输出的语义特征图A、B、C、D、E保留下来，并依次传输至解码模块，特征图E经过空洞 残差模块和混合上采样模块之后输出的特征图，与特征图D经过空洞残差模块后输出的特征图相堆叠，堆叠后的特征图经过空洞残差模块和混合上采样模块之后输出的特征图，再和上一层特征图C经过空洞残差模块后产生的特征图相堆叠，依次进行，直到最后将特征图A、B、C、D、E经过解码模块后产生的特征图与所述低分辨灰度图经过混合上采样模块之后产生的特征图相加，即为高分辨率彩色图的输出。
本申请实施例采取的又一技术方案为：一种电子设备，包括：
至少一个处理器；以及
与所述至少一个处理器通信连接的存储器；其中，
所述存储器存储有可被所述一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行上述的图像超分辨和着色方法的以下操作：
步骤a：结合图像超分辨和着色设计新的网络模型，所述网络模型包括编码模块和解码模块；
步骤b：将低分辨灰度图输入网络模型；
步骤c：通过所述编码模块提取低分辨灰度图像的语义特征图，并将所述语义特征图传入解码模块，所述解码模块对各个层级的语义特征图进行叠加，并输出高分辨彩色图。
相对于现有技术，本申请实施例产生的有益效果在于：本申请实施例的图像超分辨和着色方法、系统及电子设备针对超分辨与着色问题设计新的网络模型，对超分辨和着色任务共同处理，将低分辨的灰度图直接映射到高分辨的彩色图，提高人们视觉观感，节省资源和时间。同时，本申请的网络模型可以对所有参数共同优化，不会偏向某一个子模型。
附图说明
图1是本申请实施例的图像超分辨和着色方法的流程图；
图2为本申请实施例的网络模型的结构示意图；
图3为本申请实施例的空洞残差模块的结构图；
图4是本申请实施例的图像超分辨和着色系统的结构示意图；
图5是本申请实施例提供的图像超分辨和着色方法的硬件设备结构示意图。
具体实施方式
为了使本申请的目的、技术方案及优点更加清楚明白，以下结合附图及实施例，对本申请进行进一步详细说明。应当理解，此处所描述的具体实施例仅用以解释本申请，并不用于限定本申请。
请参阅图1，是本申请实施例的图像超分辨和着色方法的流程图。本申请实施例的图像超分辨和着色方法包括以下步骤：
步骤100：结合图像超分辨和着色设计新的网络模型；
步骤100中，请一并参阅图2，为本申请实施例的网络模型的结构示意图。本申请实施例的网络模型包括编码模块和解码模块，其中，编码模块为VGG16(一种网络模型)，VGG16已经在ImageNet数据库上经过大量数据进行预训练，具备一定的提取语义信息的能力。具体的，编码模块包括13层卷积层、5层池化层和3层全连接层。每层卷积层的卷积核为3*3，第1、2层的卷积输出通道为64，第3、4层的卷积输出通道为128，第5、6、7层的卷积输出通道为256，第8、9、10层的卷积输出通道为512，第11、12、13层的卷积输出通道为512。池化层的下采样率为2，前四个池化层为最大池化，最后的池化层 为全局平均池化，然后经过3个全连接层，3个全连接层的节点个数分别为4096、4096、1000。通过13层卷积层提取输入图像的语义特征图，并经过5层池化层和3层全连接层后，分别将第2、4、7、10、13层卷积输出的语义特征图保留下来，并依次输给解码模块。
解码模块包含多层上采样模块(具体个数视任务而定)、多个空洞残差模块(具体个数视任务而定)和混合上采样模块，其中，上采样模块和空洞残差模块的个数为至少两个，具体个数可根据实际任务进行设定。具体的，请一并参阅图3，为本申请实施例的空洞残差模块的结构图。空洞残差模块主要包括两个多分支结构的空洞率变化的卷积模块。每个卷积模块分别包括4个分支，每个分支分别由一个普通卷积先压缩维度，再经过一个空洞卷积，四个分支空洞卷积的空洞率分别为1、2、3、5。多个变化空洞率的分支结构增加了卷积网络模型的特征图的多样性，能够在不同尺度上进行特征提取，有助于网络得到更好的性能。混合上采样模块包括一个固定参数的双线性差值分支和一个参数可学习的反卷积分支，最后输入两个分支融合过后的特征图。
步骤200：将低分辨灰度图输入网络模型；
步骤300：通过网络模型的编码模块提取低分辨灰度图像的语义特征图，并将该语义特征图传入解码模块，解码模块利用空洞残差模块和混合上采样模块对各个层级的语义特征图进行叠加，并输出彩色的高分辨图像。
步骤300中，假设第2、4、7、10、13层卷积输出的语义特征图分别为A、B、C、D、E，特征图叠加方式具体为：将语义特征图A、B、C、D、E依次传至至少两个空洞残差模块，由于语义特征图A、B、C、D、E的大小是不同的，每次堆叠特征图之前经过混合上采样模块，使小的特征图增大2倍(所有的上采样率为2)。特征图E经过空洞残差模块和混合上采样模块之后输出的 特征图，与特征图D经过空洞残差模块后输出的特征图相堆叠，堆叠后的特征图经过空洞残差模块和混合上采样模块之后输出的特征图，再和上一层特征图C经过空洞残差模块后产生的特征图相堆叠，依次进行，直到最后特征图A、B、C、D、E经过解码模块后产生的特征图与输入的低分辨灰度图经过混合上采样模块之后产生的特征图相加，即为高分辨率彩色图的输出。本申请实施例中，网络模型在训练集上经过多次训练不断更新网络参数，最终网络模型能够一定程度的学习到低分辨灰度图到高分辨率彩色图的映射函数。
请参阅图4，是本申请实施例的图像超分辨和着色系统的结构示意图。本申请实施例的图像超分辨和着色系统为结合图像超分辨和着色设计的新的网络模型，所述网络模型包括编码模块和解码模块，将低分辨灰度图输入该网络模型，通过编码模块提取低分辨灰度图像的语义特征图，并将该语义特征图传入解码模块，解码模块利用空洞残差模块和混合上采样模块对各个层级的语义特征图进行叠加，并输出彩色的高分辨图像。
具体的，网络模型的编码模块为VGG16(一种网络模型)，VGG16已经在ImageNet数据库上经过大量数据进行预训练，具备一定的提取语义信息的能力。具体的，编码模块包括13层卷积层、5层池化层和3层全连接层。每层卷积层的卷积核为3*3，第1、2层的卷积输出通道为64，第3、4层的卷积输出通道为128，第5、6、7层的卷积输出通道为256，第8、9、10层的卷积输出通道为512，第11、12、13层的卷积输出通道为512。池化层的下采样率为2，前四个池化层为最大池化，最后的池化层为全局平均池化，然后经过3个全连接层，3个全连接层的节点个数分别为4096、4096、1000。通过13层卷积层提取输入图像的语义特征图，并经过5层池化层和3层全连接层后，分别将第2、4、7、10、13层卷积输出的语义特征图保留下来，并依次输给解码模 块。
解码模块包含多层上采样模块(具体个数视任务而定)、多个空洞残差模块(具体个数视任务而定)和混合上采样模块，其中，上采样模块和空洞残差模块的个数为至少两个，具体个数可根据实际任务进行设定。空洞残差模块主要包括两个多分支结构的空洞率变化的卷积模块。每个卷积模块分别包括4个分支，每个分支分别由一个普通卷积先压缩维度，再经过一个空洞卷积，四个分支空洞卷积的空洞率分别为1、2、3、5。多个变化空洞率的分支结构增加了卷积网络模型的特征图的多样性，能够在不同尺度上进行特征提取，有助于网络得到更好的性能。混合上采样模块包括一个固定参数的双线性差值分支和一个参数可学习的反卷积分支，最后输入两个分支融合过后的特征图。
本申请实施例，网络模型的图像超分辨和着色方式具体为：假设第2、4、7、10、13层卷积输出的语义特征图分别为A、B、C、D、E，将语义特征图A、B、C、D、E依次传至至少两个空洞残差模块，由于语义特征图A、B、C、D、E的大小是不同的，每次堆叠特征图之前经过混合上采样模块，使小的特征图增大2倍(所有的上采样率为2)。特征图E经过空洞残差模块和混合上采样模块之后输出的特征图，与特征图D经过空洞残差模块后输出的特征图相堆叠，堆叠后的特征图经过空洞残差模块和混合上采样模块之后输出的特征图，再和上一层特征图C经过空洞残差模块后产生的特征图相堆叠，依次进行，直到最后特征图A、B、C、D、E经过解码模块后产生的特征图与输入的低分辨灰度图经过混合上采样模块之后产生的特征图相加，即为高分辨率彩色图的输出。本申请实施例中，网络模型在训练集上经过多次训练不断更新网络参数，最终网络模型能够一定程度的学习到低分辨灰度图到高分辨率彩色图的映射函数。
图5是本申请实施例提供的图像超分辨和着色方法的硬件设备结构示意图。如图5所示，该设备包括一个或多个处理器以及存储器。以一个处理器为例，该设备还可以包括：输入系统和输出系统。
处理器、存储器、输入系统和输出系统可以通过总线或者其他方式连接，图5中以通过总线连接为例。
存储器作为一种非暂态计算机可读存储介质，可用于存储非暂态软件程序、非暂态计算机可执行程序以及模块。处理器通过运行存储在存储器中的非暂态软件程序、指令以及模块，从而执行电子设备的各种功能应用以及数据处理，即实现上述方法实施例的处理方法。
存储器可以包括存储程序区和存储数据区，其中，存储程序区可存储操作系统、至少一个功能所需要的应用程序；存储数据区可存储数据等。此外，存储器可以包括高速随机存取存储器，还可以包括非暂态存储器，例如至少一个磁盘存储器件、闪存器件、或其他非暂态固态存储器件。在一些实施例中，存储器可选包括相对于处理器远程设置的存储器，这些远程存储器可以通过网络连接至处理系统。上述网络的实例包括但不限于互联网、企业内部网、局域网、移动通信网及其组合。
输入系统可接收输入的数字或字符信息，以及产生信号输入。输出系统可包括显示屏等显示设备。
所述一个或者多个模块存储在所述存储器中，当被所述一个或者多个处理器执行时，执行上述任一方法实施例的以下操作：
步骤a：结合图像超分辨和着色设计新的网络模型，所述网络模型包括编码模块和解码模块；
步骤b：将低分辨灰度图输入网络模型；
步骤c：通过所述编码模块提取低分辨灰度图像的语义特征图，并将所述语义特征图传入解码模块，所述解码模块对各个层级的语义特征图进行叠加，并输出高分辨彩色图。
上述产品可执行本申请实施例所提供的方法，具备执行方法相应的功能模块和有益效果。未在本实施例中详尽描述的技术细节，可参见本申请实施例提供的方法。
本申请实施例提供了一种非暂态(非易失性)计算机存储介质，所述计算机存储介质存储有计算机可执行指令，该计算机可执行指令可执行以下操作：
步骤a：结合图像超分辨和着色设计新的网络模型，所述网络模型包括编码模块和解码模块；
步骤b：将低分辨灰度图输入网络模型；
步骤c：通过所述编码模块提取低分辨灰度图像的语义特征图，并将所述语义特征图传入解码模块，所述解码模块对各个层级的语义特征图进行叠加，并输出高分辨彩色图。
本申请实施例提供了一种计算机程序产品，所述计算机程序产品包括存储在非暂态计算机可读存储介质上的计算机程序，所述计算机程序包括程序指令，当所述程序指令被计算机执行时，使所述计算机执行以下操作：
步骤a：结合图像超分辨和着色设计新的网络模型，所述网络模型包括编码模块和解码模块；
步骤b：将低分辨灰度图输入网络模型；
步骤c：通过所述编码模块提取低分辨灰度图像的语义特征图，并将所述语义特征图传入解码模块，所述解码模块对各个层级的语义特征图进行叠加，并输出高分辨彩色图。
本申请实施例的图像超分辨和着色方法、系统及电子设备针对超分辨与着色问题设计新的网络模型，对超分辨和着色任务共同处理，将低分辨的灰度图直接映射到高分辨的彩色图，提高人们视觉观感，节省资源和时间。同时，本申请的网络模型可以对所有参数共同优化，不会偏向某一个子模型。
对所公开的实施例的上述说明，使本领域专业技术人员能够实现或使用本申请。对这些实施例的多种修改对本领域的专业技术人员来说将是显而易见的，本申请中所定义的一般原理可以在不脱离本申请的精神或范围的情况下，在其它实施例中实现。因此，本申请将不会被限制于本申请所示的这些实施例，而是要符合与本申请所公开的原理和新颖特点相一致的最宽的范围。

Claims (9)

  1. 一种图像超分辨和着色方法，其特征在于，包括以下步骤：
    步骤a：结合图像超分辨和着色设计新的网络模型，所述网络模型包括编码模块和解码模块；
    步骤b：将低分辨灰度图输入网络模型；
    步骤c：通过所述编码模块提取低分辨灰度图像的语义特征图，并将所述语义特征图传入解码模块，所述解码模块对各个层级的语义特征图进行叠加，并输出高分辨彩色图。
  2. 根据权利要求1所述的图像超分辨和着色方法，其特征在于，在所述步骤a中，所述编码模块包括13层卷积层、5层池化层和3层全连接层，每层卷积层的卷积核分别为3*3，第1、2层的卷积输出通道为64，第3、4层的卷积输出通道为128，第5、6、7层的卷积输出通道为256，第8、9、10层的卷积输出通道为512，第11、12、13层的卷积输出通道为512；所述池化层的下采样率为2，前四个池化层为最大池化，最后的池化层为全局平均池化；所述3个全连接层的节点个数分别为4096、4096、1000。
  3. 根据权利要求1或2所述的图像超分辨和着色方法，其特征在于，在所述步骤a中，所述解码模块包括上采样模块、空洞残差模块和混合上采样模块，所述上采样模块和空洞残差模块的个数为至少两个；所述空洞残差模块包括两个多分支结构的空洞率变化的卷积模块，每个卷积模块分别包括4个分支，每个分支分别由一个普通卷积先压缩维度，再经过一个空洞卷积，四个分支空洞卷积的空洞率分别为1、2、3、5；所述混合上采样模块包括一个固定参数的双线性差值分支和一个参数可学习的反卷积分支。
  4. 根据权利要求3所述的图像超分辨和着色方法，其特征在于，在所述步骤c中，所述通过编码模块提取低分辨灰度图像的语义特征图，并将所述语义特征图传入解码模块，所述解码模块对各个层级的语义特征图进行叠加，并输出高分辨彩色图具体为：通过所述13层卷积层提取低分辨灰度图的语义特征图，经过5层池化层和3层全连接层后，分别将第2、4、7、10、13层卷积输出的语义特征图A、B、C、D、E保留下来，并依次传输至解码模块，特征图E经过空洞残差模块和混合上采样模块之后输出的特征图，与特征图D经过空洞残差模块后输出的特征图相堆叠，堆叠后的特征图经过空洞残差模块和混合上采样模块之后输出的特征图，再和上一层特征图C经过空洞残差模块后产生的特征图相堆叠，依次进行，直到最后将特征图A、B、C、D、E经过解码模块后产生的特征图与所述低分辨灰度图经过混合上采样模块之后产生的特征图相加，即为高分辨率彩色图的输出。
  5. 一种图像超分辨和着色系统，其特征在于，包括结合图像超分辨和着色设计新的网络模型，所述网络模型包括编码模块和解码模块；将低分辨灰度图输入所述网络模型，通过所述编码模块提取低分辨灰度图像的语义特征图，并将所述语义特征图传入解码模块，所述解码模块对各个层级的语义特征图进行叠加，并输出高分辨彩色图。
  6. 根据权利要求5所述的图像超分辨和着色系统，其特征在于，所述编码模块包括13层卷积层、5层池化层和3层全连接层，每层卷积层的卷积核分别为3*3，第1、2层的卷积输出通道为64，第3、4层的卷积输出通道为128，第5、6、7层的卷积输出通道为256，第8、9、10层的卷积输出通道为512，第11、12、13层的卷积输出通道为512；所述池化层的下采样率为2，前四个池化层为最大池化，最后的池化层为全局平均池化；所述3个全连接层的节点个数分别为 4096、4096、1000。
  7. 根据权利要求5或6所述的图像超分辨和着色系统，其特征在于，所述解码模块包括上采样模块、空洞残差模块和混合上采样模块，所述上采样模块和空洞残差模块的个数为至少两个；所述空洞残差模块包括两个多分支结构的空洞率变化的卷积模块，每个卷积模块分别包括4个分支，每个分支分别由一个普通卷积先压缩维度，再经过一个空洞卷积，四个分支空洞卷积的空洞率分别为1、2、3、5；所述混合上采样模块包括一个固定参数的双线性差值分支和一个参数可学习的反卷积分支。
  8. 根据权利要求7所述的图像超分辨和着色系统，其特征在于，所述网络模型的图像超分辨和着色方式具体为：通过所述13层卷积层提取低分辨灰度图的语义特征图，经过5层池化层和3层全连接层后，分别将第2、4、7、10、13层卷积输出的语义特征图A、B、C、D、E保留下来，并依次传输至解码模块，特征图E经过空洞残差模块和混合上采样模块之后输出的特征图，与特征图D经过空洞残差模块后输出的特征图相堆叠，堆叠后的特征图经过空洞残差模块和混合上采样模块之后输出的特征图，再和上一层特征图C经过空洞残差模块后产生的特征图相堆叠，依次进行，直到最后将特征图A、B、C、D、E经过解码模块后产生的特征图与所述低分辨灰度图经过混合上采样模块之后产生的特征图相加，即为高分辨率彩色图的输出。
  9. 一种电子设备，包括：
    至少一个处理器；以及
    与所述至少一个处理器通信连接的存储器；其中，
    所述存储器存储有可被所述一个处理器执行的指令，所述指令被所述至少一个处理器执行，以使所述至少一个处理器能够执行上述1至4任一项所述的图像 超分辨和着色方法的以下操作：
    步骤a：结合图像超分辨和着色设计新的网络模型，所述网络模型包括编码模块和解码模块；
    步骤b：将低分辨灰度图输入网络模型；
    步骤c：通过所述编码模块提取低分辨灰度图像的语义特征图，并将所述语义特征图传入解码模块，所述解码模块对各个层级的语义特征图进行叠加，并输出高分辨彩色图。
PCT/CN2019/130536 2019-05-17 2019-12-31 一种图像超分辨和着色方法、系统及电子设备 WO2020233129A1 (zh)

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201910411068.1A CN110163801B (zh) 2019-05-17 2019-05-17 一种图像超分辨和着色方法、系统及电子设备
CN201910411068.1 2019-05-17

Publications (1)

Publication Number Publication Date
WO2020233129A1 true WO2020233129A1 (zh) 2020-11-26

Family

ID=67631101

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2019/130536 WO2020233129A1 (zh) 2019-05-17 2019-12-31 一种图像超分辨和着色方法、系统及电子设备

Country Status (2)

Country Link
CN (1) CN110163801B (zh)
WO (1) WO2020233129A1 (zh)

Cited By (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN112967293A (zh) * 2021-03-04 2021-06-15 首都师范大学 一种图像语义分割方法、装置及存储介质
US20210209732A1 (en) * 2020-06-17 2021-07-08 Beijing Baidu Netcom Science And Technology Co., Ltd. Face super-resolution realization method and apparatus, electronic device and storage medium
CN113255675A (zh) * 2021-04-13 2021-08-13 西安邮电大学 基于扩张卷积和残差路径的图像语义分割网络结构及方法
CN113505792A (zh) * 2021-06-30 2021-10-15 中国海洋大学 面向非均衡遥感图像的多尺度语义分割方法及模型

Families Citing this family (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110163801B (zh) * 2019-05-17 2021-07-20 深圳先进技术研究院 一种图像超分辨和着色方法、系统及电子设备
CN111179283A (zh) * 2019-12-30 2020-05-19 深圳市商汤科技有限公司 图像语义分割方法及装置、存储介质
CN111353940B (zh) * 2020-03-31 2021-04-02 成都信息工程大学 一种基于深度学习迭代上下采样的图像超分辨率重建方法
CN111654721A (zh) * 2020-04-17 2020-09-11 北京奇艺世纪科技有限公司 视频处理方法、系统、电子设备及存储介质
CN111950469A (zh) * 2020-08-14 2020-11-17 上海云从汇临人工智能科技有限公司 一种道路标识检测方法、系统、设备和介质
CA3195077A1 (en) * 2020-10-07 2022-04-14 Dante DE NIGRIS Systems and methods for segmenting 3d images

Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106855996A (zh) * 2016-12-13 2017-06-16 中山大学 一种基于卷积神经网络的灰阶图像着色方法及其装置
CN107657586A (zh) * 2017-10-13 2018-02-02 深圳市唯特视科技有限公司 一种基于深度残差网络的单照片超分辨增强方法
CN107833183A (zh) * 2017-11-29 2018-03-23 安徽工业大学 一种基于多任务深度神经网络的卫星图像同时超分辨和着色的方法
CN108830912A (zh) * 2018-05-04 2018-11-16 北京航空航天大学 一种深度特征对抗式学习的交互式灰度图像着色方法
CN109118491A (zh) * 2018-07-30 2019-01-01 深圳先进技术研究院 一种基于深度学习的图像分割方法、系统及电子设备
CN110163801A (zh) * 2019-05-17 2019-08-23 深圳先进技术研究院 一种图像超分辨和着色方法、系统及电子设备

Family Cites Families (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN103489174B (zh) * 2013-10-08 2016-06-29 武汉大学 一种基于残差保持的人脸超分辨率方法
WO2017029681A1 (en) * 2015-08-14 2017-02-23 Siddhartha Bhattacharyya A led based colorimeter device
CN105844589B (zh) * 2016-03-21 2018-12-21 深圳市未来媒体技术研究院 一种基于混合成像系统的实现光场图像超分辨的方法
CN107767343B (zh) * 2017-11-09 2021-08-31 京东方科技集团股份有限公司 图像处理方法、处理装置和处理设备
CN108596841B (zh) * 2018-04-08 2021-01-19 西安交通大学 一种并行实现图像超分辨率及去模糊的方法
CN108717698A (zh) * 2018-05-28 2018-10-30 深圳市唯特视科技有限公司 一种基于深度卷积生成对抗网络的高质量图像生成方法

Patent Citations (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN106855996A (zh) * 2016-12-13 2017-06-16 中山大学 一种基于卷积神经网络的灰阶图像着色方法及其装置
CN107657586A (zh) * 2017-10-13 2018-02-02 深圳市唯特视科技有限公司 一种基于深度残差网络的单照片超分辨增强方法
CN107833183A (zh) * 2017-11-29 2018-03-23 安徽工业大学 一种基于多任务深度神经网络的卫星图像同时超分辨和着色的方法
CN108830912A (zh) * 2018-05-04 2018-11-16 北京航空航天大学 一种深度特征对抗式学习的交互式灰度图像着色方法
CN109118491A (zh) * 2018-07-30 2019-01-01 深圳先进技术研究院 一种基于深度学习的图像分割方法、系统及电子设备
CN110163801A (zh) * 2019-05-17 2019-08-23 深圳先进技术研究院 一种图像超分辨和着色方法、系统及电子设备

Non-Patent Citations (1)

* Cited by examiner, † Cited by third party
Title
LIU, SHUAICHENG: "Colorization for Single Image Super Resolution", ECCV 2010, 31 December 2010 (2010-12-31), XP055755147, DOI: 20200303145631A *

Cited By (7)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20210209732A1 (en) * 2020-06-17 2021-07-08 Beijing Baidu Netcom Science And Technology Co., Ltd. Face super-resolution realization method and apparatus, electronic device and storage medium
US11710215B2 (en) * 2020-06-17 2023-07-25 Beijing Baidu Netcom Science And Technology Co., Ltd. Face super-resolution realization method and apparatus, electronic device and storage medium
CN112967293A (zh) * 2021-03-04 2021-06-15 首都师范大学 一种图像语义分割方法、装置及存储介质
CN113255675A (zh) * 2021-04-13 2021-08-13 西安邮电大学 基于扩张卷积和残差路径的图像语义分割网络结构及方法
CN113255675B (zh) * 2021-04-13 2023-10-10 西安邮电大学 基于扩张卷积和残差路径的图像语义分割网络结构及方法
CN113505792A (zh) * 2021-06-30 2021-10-15 中国海洋大学 面向非均衡遥感图像的多尺度语义分割方法及模型
CN113505792B (zh) * 2021-06-30 2023-10-27 中国海洋大学 面向非均衡遥感图像的多尺度语义分割方法及模型

Also Published As

Publication number Publication date
CN110163801A (zh) 2019-08-23
CN110163801B (zh) 2021-07-20

Similar Documents

Publication Publication Date Title
WO2020233129A1 (zh) 一种图像超分辨和着色方法、系统及电子设备
Li et al. MDCN: Multi-scale dense cross network for image super-resolution
WO2023092813A1 (zh) 一种基于通道注意力的Swin-Transformer图像去噪方法及系统
CN111275618B (zh) 一种基于双支感知的深度图超分辨率重建网络构建方法
Jiang et al. A progressively enhanced network for video satellite imagery superresolution
CN102142137B (zh) 基于高分辨率字典的稀疏表征图像超分辨重建方法
CN111861961A (zh) 单幅图像超分辨率的多尺度残差融合模型及其复原方法
CN110610526B (zh) 一种基于wnet对单目人像进行分割和景深渲染的方法
Chen et al. Single image super-resolution using deep CNN with dense skip connections and inception-resnet
CN109359527B (zh) 基于神经网络的头发区域提取方法及系统
CN109934771B (zh) 基于循环神经网络的无监督遥感图像超分辨率重建方法
CN112070664A (zh) 一种图像处理方法以及装置
CN113222819B (zh) 一种基于深度卷积神经网络的遥感图像超分辨重建方法
CN113837946B (zh) 一种基于递进蒸馏网络的轻量化图像超分辨率重建方法
CN112991231B (zh) 单图像超分与感知图像增强联合任务学习系统
CN112489050A (zh) 一种基于特征迁移的半监督实例分割算法
CN114581347B (zh) 无参考影像的光学遥感空谱融合方法、装置、设备及介质
CN110930306A (zh) 一种基于非局部感知的深度图超分辨率重建网络构建方法
CN111414988B (zh) 基于多尺度特征自适应融合网络的遥感影像超分辨率方法
CN114841859A (zh) 基于轻量神经网络和Transformer的单图像超分辨率重建方法
CN116934592A (zh) 一种基于深度学习的图像拼接方法、系统、设备及介质
CN116958534A (zh) 一种图像处理方法、图像处理模型的训练方法和相关装置
CN116596822A (zh) 基于自适应权重与目标感知的像素级实时多光谱图像融合方法
CN116524121A (zh) 一种单目视频三维人体重建方法、系统、设备及介质
Hua et al. Dynamic scene deblurring with continuous cross-layer attention transmission

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 19930053

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 19930053

Country of ref document: EP

Kind code of ref document: A1

122 Ep: pct application non-entry in european phase

Ref document number: 19930053

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 09.06.2022)

122 Ep: pct application non-entry in european phase

Ref document number: 19930053

Country of ref document: EP

Kind code of ref document: A1