WO2017045374A1 - 图像升频系统、其训练方法及图像升频方法 - Google Patents

图像升频系统、其训练方法及图像升频方法 Download PDF

Info

Publication number
WO2017045374A1
WO2017045374A1 PCT/CN2016/075338 CN2016075338W WO2017045374A1 WO 2017045374 A1 WO2017045374 A1 WO 2017045374A1 CN 2016075338 W CN2016075338 W CN 2016075338W WO 2017045374 A1 WO2017045374 A1 WO 2017045374A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
neural network
recombiner
convolutional neural
network module
Prior art date
Application number
PCT/CN2016/075338
Other languages
English (en)
French (fr)
Inventor
那彦波
张丽杰
何建民
Original Assignee
京东方科技集团股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 京东方科技集团股份有限公司 filed Critical 京东方科技集团股份有限公司
Priority to US15/518,404 priority Critical patent/US10019642B1/en
Publication of WO2017045374A1 publication Critical patent/WO2017045374A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting
    • G06T3/4053—Scaling of whole images or parts thereof, e.g. expanding or contracting based on super-resolution, i.e. the output image resolution being higher than the sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F18/00—Pattern recognition
    • G06F18/20—Analysing
    • G06F18/25—Fusion techniques
    • G06F18/251—Fusion techniques of input or preprocessed data
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformations in the plane of the image
    • G06T3/40—Scaling of whole images or parts thereof, e.g. expanding or contracting

Definitions

  • the present invention relates to the field of image signal processing technologies, and in particular, to an image upscaling system, a training method thereof, and an image upscaling method.
  • the resolution of an image is generally improved by using standard upscaling (increasing image resolution) such as bicubic and linear.
  • standard upscaling increasing image resolution
  • FIG. 1 a 2x up-conversion mode is shown, using four different filters F1, F2, F3 and F4 for each pixel of the input image (plus neighboring pixels).
  • the filters produce a quarter of the pixels of the output image.
  • This process can be thought of as applying a 4-filter (convolution) to the input image and then interleaving or multiplexing to create a single output image that doubles in width and height.
  • the current image up-conversion system has a large amount of data calculation, and the up-conversion multiple cannot be flexibly adjusted.
  • an embodiment of the present invention provides an image upscaling system, a training method thereof, and an image upscaling method, which are used to implement high-quality up-conversion of image resolution and reduce up-calculation calculation based on a convolutional neural network. Improve the flexibility of the upscaling adjustment.
  • an image upscaling system comprising: a cascade of at least one first convolutional neural network module and at least one recombiner; wherein a signal input of the image upscaling system is a signal input end of the first first convolutional neural network module of the at least one first convolutional neural network module, the signal output end of the image upscaling system and the last recombinator of the at least one recombiner a signal output terminal; a signal input end of each of the at least one recombiner and a signal of the first convolutional neural network module of the at least one first convolutional neural network module located at a level before the recombiner
  • the output is connected or connected to a signal output of another multiplexer located at a stage of the multiplexer; the first convolutional neural network module is configured to convert an image input to its signal input into a plurality of feature images and Output to a signal input of a recombiner connected thereto; the recombiner is
  • the number of the recombiners is two or three.
  • the signal inputs of the respective combiners are respectively connected to the signal outputs of a corresponding first convolutional neural network module of the at least one first convolutional neural network module.
  • each recombiner is a recombiner with the same up-conversion multiple.
  • the recombiner is a recombiner with a multiplication frequency and a prime number.
  • the recombiner is a recombiner with a multiplication ratio of two.
  • the synthesizer is an adaptive interpolation filter.
  • the image upscaling system further includes: a second convolutional neural network module, wherein a signal input end is connected to a signal output end of the last recombinator of the at least one recombiner, and a signal output end thereof Connected to a signal output end of the image upscaling system; the second convolutional neural network module is configured to perform image quality optimization on a feature image output by the recombiner.
  • the first convolutional neural network module and the second convolutional neural network module comprise at least one convolution layer consisting of a plurality of filtering units.
  • a display device comprising the above image upscaling system provided by an embodiment of the present invention.
  • a training method for an image upscaling system comprising: initializing each parameter in the image upscaling system; using an original image signal as an output signal of an image upscaling system, using the The down-converted image signal of the original image signal is used as an input signal of the image up-conversion system, and each parameter in the image up-conversion system is adjusted to perform up-conversion processing on the down-converted image signal by using the adjusted parameters. It is the same as the original image signal.
  • initializing each parameter in the image upscaling system includes: initializing a first convolutional neural network module and a second convolutional neural network module in an image upscaling system according to the following formula The weight W ij of each filtering unit of each convolutional layer:
  • n represents the number of feature images input to the filtering unit
  • the offset of each filter unit is initialized to zero.
  • initializing each parameter in the image upscaling system includes: initializing a first convolutional neural network module and a second convolutional neural network module in an image upscaling system according to the following formula The weight W ij of each filtering unit of each convolutional layer:
  • m represents the number of feature images input to the filtering unit; uniform(-1, 1) represents a random number selected between (-1, 1);
  • the offset of each filter unit is initialized to zero.
  • a method for image up-conversion using the image upscaling system comprising: a first convolutional neural network module inputting to the first convolutional nerve The input image of the network module is converted into a plurality of feature images and output; the synthesizer combines each n*n feature images input into the feature image of the composite device into a resolution of the feature image whose resolution is input n* The feature image of n times is output and output; the number of feature images input to the compositor is a multiple of n*n, and n is an integer greater than one.
  • the image up-conversion system, the training method thereof, and the image up-scaling method use a convolutional neural network module to acquire a feature image of an image, and a composite device is used to perform an up-conversion process of the image, and each input signal is n*
  • the n feature images are combined into a feature image with a resolution of n*n times.
  • the information of the input feature images is not lost in the generated feature image, so the image is elapsed by one liter.
  • the image resolution can be increased by n*n times.
  • each recombiner can perform an up-multiple up-conversion function, so that the system can flexibly adjust the up-conversion multiple according to needs, and realize a different target.
  • a frequency up multiple can be used for general purpose upsampling systems.
  • each recombiner enlarges the resolution of the feature image by n*n times while reducing the number of feature images output by the recombiner, the cascaded lower level recombiner or the first convolutional nerve can be reduced.
  • the input semaphore of the network module which simplifies the calculation of the up-conversion.
  • 1 is a schematic diagram of an up-conversion of 2x in the prior art
  • FIGS. 2a-2e are schematic structural diagrams of an image upscaling system according to an embodiment of the present invention.
  • FIG. 3 is a schematic diagram of frequency up-conversion of a synthesizer in an image upscaling system according to an embodiment of the present invention
  • FIG. 4 is a schematic structural diagram of a convolutional neural network module in an image upscaling system according to an embodiment of the present invention
  • FIG. 5 is a schematic flowchart of a training method of an image upscaling system according to an embodiment of the present invention.
  • Convolutional neural network is a kind of artificial neural network, which has become a research hotspot in the field of speech analysis and image recognition.
  • the weight-sharing network structure of the convolutional neural network makes it more similar to the biological neural network, which reduces the complexity of the network model and reduces the number of weights. This advantage is more obvious when the input of the network is a multi-dimensional image, so that the image can be directly used as the input of the network, avoiding the complicated feature extraction and data reconstruction process in the traditional recognition algorithm.
  • a convolutional neural network is a multi-layer perceptron specifically designed to recognize two-dimensional shapes. This network structure can be applied to translation, scaling, tilting, or other forms of deformation.
  • the present invention Based on a convolutional neural network, the present invention provides an image upscaling system, a method of training the image upscaling system, and a method of upconverting an input image according to the trained image upscaling system.
  • the image up-conversion system specifically uses a convolutional neural network to up-convert the image, and can effectively convert the low-resolution image into a high-resolution image without guaranteeing loss of image information.
  • An image upscaling system includes: a cascade of at least one first convolutional network module (CN) and at least one recombiner (Muxer Layer) , ML); wherein the signal input end of the image up-conversion system is connected to the signal input end of a first convolutional neural network module, and the signal output end of the image up-conversion system is connected to the signal output end of a recombiner.
  • CN convolutional network module
  • Muxer Layer recombiner
  • the signal input of the combiner is connected to the signal output of a first convolutional neural network module or to the signal output of another combiner.
  • the first convolutional neural network module is configured to convert an image input to its signal input into a plurality of feature images and output to a signal input of the combiner.
  • the composite device is configured to synthesize each n*n feature images in the feature image input to the signal input end into a feature image with a resolution of n*n times the resolution of the input feature image and output the image of the feature image input to the composite device.
  • the number is a multiple of n*n, and n is an integer greater than one.
  • the image is up-converted by the composite device, and each n*n feature image input to the composite device is synthesized into a feature image with a resolution enlarged by n*n times.
  • the information of the input feature images is recorded into the generated feature image without loss, so the image is passed through a recombiner with an up-multiple multiple of n.
  • the resolution can be increased by n*n times.
  • each recombiner can be set for successive frequency up-up, and each recombiner can perform an up-multiple up-conversion function, so that the system can flexibly adjust the up-conversion multiple according to needs, and realize a different target.
  • Up-conversion multiples are common image upscaling systems. Further, since each recombiner enlarges the resolution of the feature image by n*n times while reducing the number of feature images output by the recombiner, the cascaded lower level recombiner or the first convolutional nerve can be reduced. The input semaphore of the network module, which simplifies the calculation of the up-conversion.
  • the image resolution can be increased by (n*n)*(n*n) times after the picture is up-converted by the system.
  • the resolution of the image is increased by 4*4 times after passing through the two recombiners; if the system includes three recombiners with upsampling multiples of 2x, After the images pass through the three synthesizers, the resolution is increased by 4*4*4 times.
  • the specific implementation manner of the above image upscaling system provided by the embodiment of the present invention may be various according to the required upscaling multiple.
  • a recombiner may be provided as shown in FIG. 2a, or two composites may be set as shown in FIG. 2b and FIG. 2c. It is also possible to set up three recombiners as shown in Figure 2d.
  • the image upscaling system provided by the embodiment of the present invention includes a first convolutional nerve, when the upsampling multiple is required to be upscaled, such as 2x, 3x, or 5x.
  • the image up-conversion system provided by the embodiment of the present invention may include two 2x-fold up-converter combiners.
  • the up-conversion multiple is required to be 8x
  • the image up-conversion system provided by the embodiment of the present invention as shown in FIG.
  • 2d may include three 2x times up-converter combiners.
  • high quality feature images can be used to synthesize high quality resolution as input.
  • the characteristic image of the feature image has n*n times the feature image, as shown in FIG. 2c and FIG. 2d.
  • each composite device is connected with the signal output end of a first convolutional neural network module, first adopting the first A convolutional neural network module to acquire the feature image and then acquire the acquired feature map
  • the first convolutional neural network module and the recombiner are arranged in pairs.
  • the multiplication ratios of each recombiner may be the same or different.
  • the upsampling multiple n of each recombiner is generally set to be the same.
  • the smaller the up-multiplier of each recombiner the smaller the calculation amount, and the better the up-conversion effect. Therefore, if the required up-multiplier is large, multiple up-conversions are generally used, and the up-multiplier n of each recombiner is generally set to a prime number such as 2, 3, 5 or 7.
  • the up-multiplier n of each recombiner is generally set to two.
  • the method further includes: a second convolutional neural network module, wherein the signal input end is connected to the signal output end of the recombiner, and the signal output thereof is The end is connected to the signal output end of the image up-conversion system; the second convolutional neural network module is used for image quality optimization of the feature image output by the recombiner.
  • the second convolutional neural network module can be used to enhance the image quality of the output image according to actual needs, thereby improving the quality of the output image.
  • the first convolutional neural network module and the second convolutional neural network module may each include at least one convolution layer composed of a plurality of filtering units.
  • the number of convolution layers included in the first convolutional neural network module and the second convolutional neural network module can be set as needed.
  • the number of filtering units included in each convolutional layer may be the same or different.
  • the number of layers of convolutional layers in each convolutional neural network module is typically set to no more than 10 layers.
  • the first-stage first convolutional neural network module connected to the signal input end of the image up-conversion system is composed of four layers of convolution layers, each convolution layer comprising 128 filter units, each filter unit consisting of 3*3 A filter consisting of setting the filter at the [1, 1] position as the center pixel.
  • 128 feature images are generated and output to the next convolutional layer until the last convolutional layer outputs 128 feature images to the next level (second stage) recombiner.
  • the second-stage multiplexer After receiving the 128 feature images sent by the first-level first convolutional neural network module, the second-stage multiplexer combines each of the input feature images into a feature image of 4 ⁇ pixel resolution (2 ⁇ up-amplitude), ie The 128 input feature images pass through the second level recombiner and output 32 feature images to the next (third level) first convolutional neural network module.
  • the third-stage first convolutional neural network module consists of four layers of convolutional layers, each convolutional layer consisting of 32 filtering units, each of which consists of 3*3 filters, where [1,1 The position filter is set to the center pixel. After the 32 feature images input from the second level of the composite device pass through the first convolutional layer, 32 feature images are generated and output to the next convolution layer until the last convolution layer outputs 32 feature images to the next level. (fourth level) composite.
  • the fourth-stage multiplexer After receiving the 32 feature images transmitted by the first convolutional neural network module of the third stage, the fourth-stage multiplexer combines each of the input feature images into a feature image of 4x pixel resolution (2x up-conversion). That is, the 32 input feature images are outputted through the fourth level recombiner to output 8 feature images to the next (fifth level) second convolutional neural network module.
  • the fifth-level second convolutional neural network module consists of four layers of convolutional layers, the first two convolutional layers contain eight filtering units, the third convolutional layer contains four filtering units, and the fourth convolutional layer contains A filter unit, each filter unit consisting of 3*3 filters, wherein the filter at the [1, 1] position is set as the center pixel.
  • each compositor is substantially equivalent to an adaptive interpolation filter.
  • the pixel values of every four feature images in the input feature image are interlaced and combined to generate a 4 ⁇ pixel feature. image.
  • the working principle of the recombiner is to matrix-arrange the pixel values of the same pixel position in the four input feature images in the output feature image, and therefore, will not be modified during the up-conversion process ( Loss or increase) any pixel information in the feature image.
  • both the first convolutional neural network module and the second convolutional neural network module can be regarded as a neural network structure using images as input and output, and each neural network structure contains multiple convolutions. Layers, and each convolution layer is composed of multiple filters.
  • the working principle of the neural network structure of the two-layer convolutional layer in FIG. 4 is briefly introduced as an example.
  • each has a scalar weight
  • the box is equivalent to a filter (such as a 3x3 or 5x5 core filter), biased Represents the increment of the screen added to the convolution output.
  • k denotes a convolutional layer number
  • i and j denote an input image number and an output image number, respectively.
  • the embodiment of the present invention further provides a training method for the image upscaling system, as shown in FIG. , including the following steps:
  • each parameter in the image up-conversion system is actually a parameter of all convolutional neural network modules.
  • Step S501 initializes each parameter in the image up-conversion system, and may set a weight W ij of each filter unit of each convolutional layer of all convolutional neural network modules to a small random number by using a conventional initialization manner. And initialize all offsets to zero.
  • the conventional initialization method does not cause any problem when applied to a small multiple up-conversion such as 2x, but there are some problems when applied to a high-multiplier up-conversion such as 4x combined by several convolutional neural network modules, therefore,
  • two new methods are also provided for initializing each parameter in the image upscaling system, as follows:
  • n represents the number of feature images input to the filtering unit.
  • m represents the number of feature images input to the filtering unit; uniform(-1,1) represents A random number chosen between (-1, 1).
  • the second initialization method adds a small uniform distributed noise value to the weight W ij of each filtering unit relative to the first initialization mode, which facilitates the image upscaling system to have the ability to recognize noise after training.
  • an embodiment of the present invention further provides a method for image up-conversion using the image up-conversion system described above. Since the method for solving the problem is similar to the foregoing image up-conversion system, the implementation of the method may be See the implementation of the system, and the repetitions are not repeated here.
  • the first convolutional neural network module converts the input image input to the first convolutional neural network module into a plurality of feature images having specific features and outputs the same.
  • the composite unit synthesizes each n*n feature images input into the feature image of the composite device into a feature image whose resolution is n*n times the resolution of the input feature image and outputs the feature image input to the composite device
  • the number is a multiple of n*n, and n is an integer greater than one.
  • each recombiner receives the feature image and then up-converts the feature image, and then outputs the image to the next recombiner, and the next recombinator performs the received feature image.
  • the up-converting process continues until the last remixer outputs the final up-converted image.
  • the image upscaling system may be implemented by a group of central processing units (CPUs), may also be implemented by a set of image processing units (GPUs), or may also be implemented by a field programmable gate array (FPGA). .
  • CPUs central processing units
  • GPUs graphics processing units
  • FPGA field programmable gate array
  • an embodiment of the present invention further provides a display device, including the image upscaling system provided by the embodiment of the present invention, which may be: a mobile phone, a tablet computer, a television, a display, a notebook computer, or a digital device. Any product or component that has a display function, such as a photo frame, wearable device, and navigator.
  • a display device including the image upscaling system provided by the embodiment of the present invention, which may be: a mobile phone, a tablet computer, a television, a display, a notebook computer, or a digital device. Any product or component that has a display function, such as a photo frame, wearable device, and navigator.
  • the display device reference may be made to the embodiment of the image upscaling system described above, and the repeated description is omitted.
  • An image upscaling system, a training method thereof and an image upscaling method provided by the embodiments of the present invention use a convolutional neural network module to acquire a feature image of an image, and a composite device is used for up-converting the image to input an n* in the input signal.
  • the n feature images are synthesized to a resolution image with a resolution of n*n times.
  • the information of each feature image in the input signal is not lost in the generated feature image, so each image passes through After a synthesizer with a multiplication ratio of n, the image resolution can be increased by n*n times.
  • each recombiner can perform a single multiplication up-conversion function, so that the system can be flexible according to needs.
  • an up-converting system that can be used for different up-conversion multiples is realized.
  • each recombiner enlarges the resolution of the feature image by n*n times while reducing the number of feature images output by the recombiner, the cascaded lower level recombiner or the first convolutional nerve can be reduced.
  • the input semaphore of the network module which simplifies the calculation of the up-conversion.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • General Physics & Mathematics (AREA)
  • Data Mining & Analysis (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Evolutionary Computation (AREA)
  • General Engineering & Computer Science (AREA)
  • Artificial Intelligence (AREA)
  • Computing Systems (AREA)
  • Mathematical Physics (AREA)
  • Computational Linguistics (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Biophysics (AREA)
  • Biomedical Technology (AREA)
  • Health & Medical Sciences (AREA)
  • Software Systems (AREA)
  • Bioinformatics & Cheminformatics (AREA)
  • Bioinformatics & Computational Biology (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Evolutionary Biology (AREA)
  • Image Processing (AREA)

Abstract

一种图像升频系统、其训练方法及图像升频方法，涉及图像信号处理技术领域。采用卷积神经网络模块获取输入图像的特征图像，采用复合器进行图像的升频处理将特征图像中每n*n个特征图像合成一分辨率放大n*n倍的特征图像。在复合器的升频过程中，输入的各特征图像的信息不损失的记载到生成的特征图像中，因此经过复合器后可以得到分辨率提升n*n倍的高质量图像。并且，在图像升频系统中可设置多个复合器进行逐次升频，每个复合器都可以执行单独倍数的升频功能，使系统可以根据需要灵活调整升频倍数。且由于复合器在将特征图像的分辨率放大n倍的同时，减少了复合器输出的特征图像的数量，从而减少下一级复合器或第一卷积神经网络模块的输入信号量，简化其计算量。

Description

图像升频系统、其训练方法及图像升频方法 技术领域
本发明涉及图像信号处理技术领域，尤其涉及一种图像升频系统、其训练方法及图像升频方法。
背景技术
目前，在图像信号处理的过程中，一般是利用标准的诸如双三次(bicubic)和线性等标准升频(提高图像分辨率)方式对图像进行分辨率的提升。如图1所示，示出了一个2x的升频方式，对输入图像的各像素(加上邻像素)使用四个不同的滤波器F1、滤波器F2、滤波器F3和滤波器F4，每个滤波器产生四分之一的输出图像的像素，这个过程可以看作是对输入图像应用4个滤波器(卷积)后交错或复用以创建宽度和高度翻倍的单一输出图像。
但是，目前的图像升频系统数据计算量较大，升频倍数无法灵活调节。
发明内容
有鉴于此，本发明实施例提供了一种图像升频系统、其训练方法及图像升频方法，用以基于卷积神经网络实现对图像分辨率高品质的升频，降低升频计算量，提高升频倍数调节的灵活度。
根据本发明一方面，提供了一种图像升频系统，包括：级联的至少一个第一卷积神经网络模块和至少一个复合器；其中，所述图像升频系统的信号输入端与所述至少一个第一卷积神经网络模块中的第一个第一卷积神经网络模块的信号输入端连接，所述图像升频系统的信号输出端与所述至少一个复合器中最后一个复合器的信号输出端连接；所述至少一个复合器中每个复合器的信号输入端与所述至少一个第一卷积神经网络模块中位于该复合器前一级的第一卷积神经网络模块的信号输出端连接，或与位于该复合器前一级的另一复合器的信号输出端连接；所述第一卷积神经网络模块用于将输入到其信号输入端的图像转换为多个特征图像并输出至与其连接的复合器的信号输入端；所述复合器用于将输入到其信号输入端的特征图像中每n*n个特征图像合成一个分辨率为输入的特征图像的分辨率n*n倍的特征图像并输出；输入到所述复合器的特征图像的数量为n*n的倍数，n为大于1的整数。
根据本发明实施例，所述复合器的数量为两个或三个。
根据本发明实施例，各复合器的信号输入端分别与所述至少一个第一卷积神经网络模块中一个相应的第一卷积神经网络模块的信号输出端连接。
根据本发明实施例，在所述复合器为多个时，各复合器为升频倍数相同的复合器。
根据本发明实施例，所述复合器为升频倍数为质数的复合器。
根据本发明实施例，所述复合器为升频倍数为2的复合器。
根据本发明实施例，所述复合器为自适应插值滤波器。
根据本发明实施例，所述图像升频系统还包括：第二卷积神经网络模块，其信号输入端与所述至少一个复合器中最后一个复合器的信号输出端连接，并且其信号输出端与所述图像升频系统的信号输出端连接；所述第二卷积神经网络模块用于对所述复合器输出的特征图像进行画质优化。
根据本发明实施例，所述第一卷积神经网络模块和第二卷积神经网络模块包括至少一层由多个滤波单元组成的卷积层。
根据本发明另一方面，还提供了一种显示装置，包括本发明实施例提供的上述图像升频系统。
根据本发明又一方面，还提供了一种图像升频系统的训练方法，包括：初始化所述图像升频系统中的各参数；采用原始图像信号作为图像升频系统的输出信号，采用所述原始图像信号经降频后的图像信号作为图像升频系统的输入信号，调整所述图像升频系统中的各参数，以使采用调整后的各参数对降频后的图像信号进行升频处理后与原始图像信号相同。
根据本发明实施例，在上述训练方法中，初始化所述图像升频系统中的各参数包括：按照以下公式初始化图像升频系统中第一卷积神经网络模块和第二卷积神经网络模块的各卷积层的各滤波单元的权值Wij：
Figure PCTCN2016075338-appb-000001
其中，m表示输入所述滤波单元的特征图像的数量；
将各滤波单元的偏置初始化为0。
根据本发明实施例，在上述训练方法中，初始化所述图像升频系统中的各参数包括：按照以下公式初始化图像升频系统中第一卷积神经网络模块和第二卷积神经网络模块的各卷积层的各滤波单元的权值Wij：
Figure PCTCN2016075338-appb-000002
Figure PCTCN2016075338-appb-000003
其中，m表示输入所述滤波单元的特征图像的数量；uniform(-1,1)表示在(-1,1)之间选取的随机数；
将各滤波单元的偏置初始化为0。
根据本发明再一方面，还提供了一种采用本发明实施例提供的上述图像升频系统进行图像升频的方法，包括：第一卷积神经网络模块将输入到所述第一卷积神经网络模块的输入图像转换为多个特征图像并输出；复合器将输入到所述复合器的特征图像中每n*n个特征图像合成一分辨率为输入的所述特征图像的分辨率n*n倍的特征图像并输出；输入到所述复合器的特征图像的数量为n*n的倍数，n为大于1的整数。
根据本发明实施例，所述图像升频系统、其训练方法及图像升频方法，采用卷积神经网络模块获取图像的特征图像，采用复合器进行图像的升频处理将输入信号中每n*n个特征图像合成一分辨率放大n*n倍的特征图像，在复合器的升频过程中，输入的各特征图像的信息不损失的记载到生成的特征图像中，因此图像每经过一个升频倍数为nx的复合器后，图像分辨率可提升n*n倍。并且，在图像升频系统中可设置不止一个复合器进行逐次升频，每个复合器都可以执行单独倍数的升频功能，使系统可以根据需要灵活调整升频倍数，实现了一种针对不同升频倍数可通用的升频系统。进一步地，由于每个复合器在将特征图像的分辨率放大n*n倍的同时，减少了复合器输出的特征图像的数量，可以减少级联的下一级复合器或第一卷积神经网络模块的输入信号量，从而简化升频计算量。
附图说明
图1为现有技术中2x的升频示意图；
图2a-图2e分别为本发明实施例提供的图像升频系统的结构示意图；
图3为本发明实施例提供的图像升频系统中复合器的升频示意图；
图4为本发明实施例提供的图像升频系统中卷积神经网络模块的结构示意图；
图5为本发明实施例提供的图像升频系统的训练方法的流程示意图。
具体实施方式
卷积神经网络是人工神经网络的一种，已成为当前语音分析和图像识别领域的研究热点。卷积神经网络的权值共享网络结构使之更类似于生物神经网络，降低了网络模型的复杂度，减少了权值的数量。该优点在网络的输入是多维图像时表现的更为明显，使图像可以直接作为网络的输入，避免了传统识别算法中复杂的特征提取和数据重建过程。卷积神经网络是为识别二维形状而特殊设计的一个多层感知器，这种网络结构可以应用于平移、比例缩放、倾斜或者其他形式的变形。
基于卷积神经网络，本发明提供一种图像升频系统、对该图像升频系统进行训练的方法，以及根据训练后的图像升频系统对输入图像进行升频的方法。该图像升频系统具体采用卷积神经网络对图像进行升频，在保证不丢失图像信息的前提下，可有效的将低分辨率的图像转换为高分辨率的图像。
下面结合附图，对本发明实施例提供的图像升频系统、其训练方法及图像升频方法的具体实施方式进行详细地说明。
本发明实施例提供的一种图像升频系统，如图2a至图2d所示，包括：级联的至少一个第一卷积神经网络模块(Convolutional Network，CN)和至少一个复合器(Muxer Layer，ML)；其中，图像升频系统的信号输入端与一第一卷积神经网络模块的信号输入端连接，图像升频系统的信号输出端与一复合器的信号输出端连接。
复合器的信号输入端与一第一卷积神经网络模块的信号输出端连接，或与另一复合器的信号输出端连接。
第一卷积神经网络模块用于将输入到其信号输入端的图像转换为多个特征图像并输出至复合器的信号输入端。复合器用于将输入到其信号输入端的特征图像中每n*n个特征图像合成一个分辨率为输入的特征图像的分辨率n*n倍的特征图像并输出；输入到复合器的特征图像的数量为n*n的倍数，n为大于1的整数。
在本发明实施例提供的上述图像升频系统中，采用复合器进行图像的升频处理将输入到复合器的每n*n个特征图像合成为一个分辨率放大n*n倍的特征图像，在复合器的升频过程中，输入的各特征图像的信息不损失地记载到生成的特征图像中，因此图像每经过一个升频倍数为n的复合器后，图像 分辨率可提升n*n倍。并且，在图像升频系统中可设置不止一个复合器进行逐次升频，每个复合器都可以执行单独倍数的升频功能，使系统可以根据需要灵活调整升频倍数，实现了一种针对不同升频倍数可通用的图像升频系统。进一步地，由于每个复合器在将特征图像的分辨率放大n*n倍的同时，减少了复合器输出的特征图像的数量，可以减少级联的下一级复合器或第一卷积神经网络模块的输入信号量，从而简化升频计算量。
需要说明的是，若系统包含多个放大倍数为n倍的复合器，则图片经过该系统升频后，图像分辨率可提高(n*n)*(n*n)倍。
比如，若系统包括两个升频倍数为2x的复合器时，图像经过这两个复合器之后，分辨率提高为4*4倍；若系统包括三个升频倍数为2x的复合器时，图像经过这三个复合器之后，分辨率提高为4*4*4倍。
在具体实施时，根据所需的升频倍数，本发明实施例提供的上述图像升频系统的具体实施方式可以有多种。例如，根据所需的升频倍数，可以在本发明实施例提供的上述图像升频系统中，可以如图2a所示设置一个复合器，也可以如图2b和图2c所示设置两个复合器，还可以如图2d所示设置三个复合器。
具体地，一般当需要升频倍数为诸如2x、3x或5x等质数倍的升频时，本发明实施例提供的上述图像升频系统，如图2a所示，包含一个第一卷积神经网络模块和一个将特征图像进行对应的2x、3x或5x倍升频的复合器。当需要升频倍数为4x时，本发明实施例提供的上述图像升频系统，如图2b和图2c所示，可以包含两个2x倍升频的复合器。当需要升频倍数为8x时，本发明实施例提供的上述图像升频系统，如图2d所示，可以包含三个2x倍升频的复合器。以此类推，当需要的倍频数越大，所需的复合器的数量也就对应的越多，对应的系统所进行的数据计算量也就越大。因此，较佳地，在本发明实施例提供的上述图像升频系统中，一般设置两个或三个复合器，进行两次或三次升频。
进一步地，在本发明实施例提供的上述图像升频系统中设置多个复合器时，为了使每个复合器在进行升频处理时可以采用高质量的特征图像合成高质量的分辨率为输入的特征图像的分辨率n*n倍的特征图像，如图2c和图2d所示，一般各复合器的信号输入端均与一第一卷积神经网络模块的信号输出端连接，先采用第一卷积神经网络模块来获取特征图像再将所获取的特征图 像输入到对应的复合器的信号输入端，即在图像升频系统中，第一卷积神经网络模块和复合器成对设置。
进一步地，在本发明实施例提供的上述图像升频系统中设置多个复合器时，每个复合器的升频倍数可以相同，也可以不同。一般地，在复合器为多个时，一般将各复合器的升频倍数n设置为相同。而且，每个复合器的升频倍数越小，其计算量越小，升频效果越好。因此，若所需升频倍数较大时，一般采用多次升频，而每个复合器的升频倍数n一般均设置为诸如2、3、5或7等质数。较佳地，一般每个复合器的升频倍数n设置为2。
进一步地，本发明实施例提供的上述图像升频系统中，如图2e所示，还包括：第二卷积神经网络模块，其信号输入端与复合器的信号输出端连接，并且其信号输出端与图像升频系统的信号输出端连接；该第二卷积神经网络模块用于对复合器输出的特征图像进行画质优化。在最后一级的复合器输出最终的升频特征图像之前，可以利用第二卷积神经网络模块根据实际需要对输出画质进行画质增强，提高输出图像的质量。
具体地，本发明实施例提供的上述图像升频系统中，第一卷积神经网络模块和第二卷积神经网络模块均可以包括至少一层由多个滤波单元组成的卷积层。第一卷积神经网络模块和第二卷积神经网络模块所包含的卷积层数可以根据需要设定。且每个卷积层中包含的滤波单元个数可以相同也可以不同。一般地，为了便于系统优化参数，每个卷积神经网络模块中的卷积层的层数一般设置为不大于10层。
下面以图2e所示的结构，且采用两个2x的复合器进行4x升频为例说明本发明实施例提供的上述图像升频系统。
具体地，与图像升频系统的信号输入端连接的首级第一卷积神经网络模块由四层卷积层组成，每个卷积层包含128个滤波单元，每个滤波单元由3*3个滤波器组成，其中，将[1,1]位置的滤波器设置为中心像素。在输入图像经过第一卷积层后会生成128个特征图像输出到下一卷积层，直至最后一层卷积层输出128个特征图像至下一级(第二级)的复合器。
第二级的复合器接收到首级第一卷积神经网络模块发送的128个特征图像之后，将输入的每4个特征图像合成一个4倍像素分辨率(2x升频)的特征图像，即128个输入的特征图像经过第二级的复合器后输出32个特征图像到下一级(第三级)的第一卷积神经网络模块。
第三级的第一卷积神经网络模块由四层卷积层组成，每个卷积层包含32个滤波单元，每个滤波单元由3*3个滤波器组成，其中，将[1,1]位置的滤波器设置为中心像素。从第二级的复合器输入的32个特征图像经过第一卷积层后会生成32个特征图像输出到下一卷积层，直至最后一层卷积层输出32个特征图像至下一级(第四级)的复合器。
第四级的复合器接收到第三级的第一卷积神经网络模块发送的32个特征图像之后，将输入的每4个特征图像合成一个4倍像素分辨率(2x升频)的特征图像，即32个输入的特征图像经过第四级的复合器后输出8个特征图像到下一级(第五级)的第二卷积神经网络模块。
第五级的第二卷积神经网络模块由四层卷积层组成，前两个卷积层包含8个滤波单元，第三个卷积层包含4个滤波单元，第四个卷积层包含1个滤波单元，每个滤波单元由3*3个滤波器组成，其中，将[1,1]位置的滤波器设置为中心像素。在输入信号的图像经过第一卷积层后会生成8个特征图像输出到第二卷积层，经过第二卷积层后会生成8个特征图像输入到第三卷积层，经过第三卷积层后会有4个特征图像输入到第四卷积层，最后，第四卷积层输出1个特征图像至图像升频系统的输出端。
在上述过程中，每个复合器实质上相当于一个自适应插值滤波器，如图3所示，将输入的特征图像中每4个特征图像的像素值交错组合，生成一4倍像素的特征图像。如图3所示，复合器的工作原理是将4个输入的特征图像中各相同像素点位置的像素值进行矩阵排列在输出的特征图像中，因此，在此升频过程中不会修改(丢失或增加)特征图像中的任何像素信息。
在上述过程中，不论是第一卷积神经网络模块还是第二卷积神经网络模块均可以看成是一个采用图像作为输入和输出的神经网络结构，每个神经网络结构中包含多个卷积层，且每个卷积层均由多个滤波器组成。下面以图4中的两层卷积层的神经网络结构为例简要的介绍下其工作原理。
在图4中左侧的具有四个输入图像，经过第一卷积层的各滤波器后生成三个特征图像，经过第二卷积层的各滤波器后生成两个特征图像输出。其中，每个写有标量权重
Figure PCTCN2016075338-appb-000004
的框相当于一个滤波器(例如一个3x3或5x5内核的滤波器)，偏置
Figure PCTCN2016075338-appb-000005
表示添加到卷积输出的画面增量。k表示卷积层编号，i和j分别表示输入图像编号和输出图像编号。
在系统运行过程中标量权重
Figure PCTCN2016075338-appb-000006
和偏置
Figure PCTCN2016075338-appb-000007
的数值相对固定，在系统运行前 需要采用一系列的标准输入输出图像来对系统进行训练，并且依靠应用程序调整到适合某些优化准则。因此，在本发明实施例提供的上述图像升频系统运行之前，需要进行一系列的训练，基于同一发明构思，本发明实施例还提供了上述图像升频系统的训练方法，如图5所示，包括以下步骤：
S501、初始化图像升频系统中的各参数；由于复合器不会引入任何参数，因此图像升频系统中的各参数实际上为所有卷积神经网络模块的参数。
S502、采用原始图像信号作为图像升频系统的输出信号，采用原始图像信号经降频后的图像信号作为图像升频系统的输入信号，调整图像升频系统中的各参数，以使采用调整后的各参数对降频后的图像信号进行升频处理后与原始图像信号相同。之后，采用调整后的各参数作为升频系统的升频参数，对低分辨率的图像进行升频。
其中，步骤S501初始化图像升频系统中的各参数，可以采用传统的初始化方式，将所有卷积神经网络模块的各卷积层的各滤波单元的权值Wij设置为一小的随机数，并将所有偏置初始化为0。传统的初始化方式在应用于诸如2x的小倍数升频时不会出现任何问题，但是在应用于由几个卷积神经网络模块结合的诸如4x的高倍数升频时会出现一些问题，因此，本发明实施例提供的上述训练方法中对于图像升频系统中的各参数的初始化还提供了两种新的方式，具体如下：
第一种：将各滤波单元的偏置初始化为0；并按照以下公式初始化图像升频系统中第一卷积神经网络模块和第二卷积神经网络模块的各卷积层的各滤波单元的权值Wij：
Figure PCTCN2016075338-appb-000008
其中，m表示输入至该滤波单元的特征图像的数量。
第二种：将各滤波单元的偏置初始化为0；并按照以下公式初始化图像升频系统中第一卷积神经网络模块和第二卷积神经网络模块的各卷积层的各滤波单元的权值Wij：
Figure PCTCN2016075338-appb-000009
Figure PCTCN2016075338-appb-000010
其中，m表示输入至该滤波单元的特征图像数量；uniform(-1,1)表示 在(-1,1)之间选取的随机数。
第二种初始化的方式相对于第一种初始化的方式，在各滤波单元的权值Wij中添加了小的均匀分布噪声值，这样利于图像升频系统在训练之后具有识别噪声的能力。
基于同一发明构思，本发明实施例还提供了一种采用上述图像升频系统进行图像升频的方法，由于该方法解决问题的原理与前述一种图像升频系统相似，因此该方法的实施可以参见系统的实施，重复之处不再赘述。
本发明实施例提供的一种采用图像升频系统进行图像升频的方法，包括：
第一卷积神经网络模块将输入到该第一卷积神经网络模块的输入图像转换为多个具有特定特征的特征图像并输出。
复合器将输入到该复合器的特征图像中每n*n个特征图像合成一分辨率为输入的特征图像的分辨率n*n倍的特征图像并输出；输入到该复合器的特征图像的数量为n*n的倍数，n为大于1的整数。
具体地，在系统存在多个复合器时，每一个复合器接收到特征图像后都会对特征图像进行升频处理，然后输出给下一个复合器，该下一个复合器对接收到的特征图像进行升频处理直至最后一个复合器输出最终的升频图像。
本发明实施例所述的图像升频系统可以由一组中央处理器(CPU)实现、一也可由一组图像处理器(GPU)实现，或者还可以由现场可编程门阵列(FPGA)实现。
基于同一发明构思，本发明实施例还提供了一种显示装置，包括本发明实施例提供的上述图像升频系统，该显示装置可以为：手机、平板电脑、电视机、显示器、笔记本电脑、数码相框、可穿戴设备、导航仪等任何具有显示功能的产品或部件。该显示装置的实施可以参见上述图像升频系统的实施例，重复之处不再赘述。
本发明实施例提供的一种图像升频系统、其训练方法及图像升频方法，采用卷积神经网络模块获取图像的特征图像，采用复合器进行图像的升频处理将输入信号中每n*n个特征图像合成至一分辨率放大n*n倍的特征图像，在复合器的升频过程中，输入信号中各特征图像的信息不损失的记载到生成的特征图像中，因此图像每经过一个升频倍数为n的复合器后，图像分辨率可提升n*n倍。并且，在图像升频系统中可设置不止一个复合器进行逐次升频，每个复合器都可以执行单独倍数的升频功能，使系统可以根据需要灵活 调整升频倍数，实现了一种针对不同升频倍数可通用的升频系统。进一步地，由于每个复合器在将特征图像的分辨率放大n*n倍的同时，减少了复合器输出的特征图像的数量，可以减少级联的下一级复合器或第一卷积神经网络模块的输入信号量，从而简化升频计算量。
显然，本领域的技术人员可以对本发明进行各种改动和变型而不脱离本发明的精神和范围。这样，倘若本发明的这些修改和变型属于本发明权利要求及其等同技术的范围之内，则本发明也意图包含这些改动和变型在内。
本申请要求2015年09月17日提交的申请号为201510595656.7且发明名称为“一种图像升频系统、其训练方法及图像升频方法”的中国优先申请的优先权，通过引用将其全部内容并入于此。

Claims (14)

  1. 一种图像升频系统，包括：级联的至少一个第一卷积神经网络模块和至少一个复合器；其中，
    所述图像升频系统的信号输入端与所述至少一个第一卷积神经网络模块中的第一个第一卷积神经网络模块的信号输入端连接，所述图像升频系统的信号输出端与所述至少一个复合器中最后一个复合器的信号输出端连接；
    所述至少一个复合器中每个复合器的信号输入端与所述至少一个第一卷积神经网络模块中位于该复合器前一级的第一卷积神经网络模块的信号输出端连接，或与所述至少一个复合器中位于该复合器前一级的另一复合器的信号输出端连接；
    所述第一卷积神经网络模块用于将输入到其信号输入端的图像转换为多个特征图像并输出至与其连接的复合器的信号输入端；
    所述复合器用于将输入到其信号输入端的特征图像中每n*n个特征图像合成一个分辨率为输入的特征图像的分辨率n*n倍的特征图像并输出；输入到所述复合器的特征图像的数量为n*n的倍数，n为大于1的整数。
  2. 如权利要求1所述的图像升频系统，其中，所述复合器的数量为两个或三个。
  3. 如权利要求2所述的图像升频系统，其中，各复合器的信号输入端分别与所述至少一个第一卷积神经网络模块中一个相应的第一卷积神经网络模块的信号输出端连接。
  4. 如权利要求1所述的图像升频系统，其中，在所述复合器为多个时，各复合器为升频倍数相同的复合器。
  5. 如权利要求1所述的图像升频系统，其中，所述复合器为升频倍数为质数的复合器。
  6. 如权利要求5所述的图像升频系统，其中，所述复合器为升频倍数为2的复合器。
  7. 如权利要求1所述的图像升频系统，其中，所述复合器为自适应插值滤波器。
  8. 如权利要求1所述的图像升频系统，还包括：第二卷积神经网络模块，其信号输入端与所述至少一个复合器中最后一个复合器的信号输出端连接， 并且其信号输出端与所述图像升频系统的信号输出端连接；
    所述第二卷积神经网络模块用于对所述复合器输出的特征图像进行画质优化。
  9. 如权利要求8所述的图像升频系统，其中，所述第一卷积神经网络模块和第二卷积神经网络模块包括至少一层由多个滤波单元组成的卷积层。
  10. 一种显示装置，包括如权利要求1-9任一项所述图像升频系统。
  11. 一种如权利要求1-9任一项所述的图像升频系统的训练方法，其特征在于，包括：
    初始化所述图像升频系统中的各参数；
    采用原始图像信号作为图像升频系统的输出信号，采用所述原始图像信号经降频后的图像信号作为图像升频系统的输入信号，调整所述图像升频系统中的各参数，以使采用调整后的各参数对降频后的图像信号进行升频处理后与原始图像信号相同。
  12. 如权利要求11所述的训练方法，其中，初始化所述图像升频系统中的各参数包括：
    按照以下公式初始化图像升频系统中第一卷积神经网络模块和第二卷积神经网络模块的各卷积层的各滤波单元的权值Wij：
    Figure PCTCN2016075338-appb-100001
    其中，m表示输入所述滤波单元的特征图像的数量；
    将各滤波单元的偏置初始化为0。
  13. 如权利要求11所述的训练方法，其中，初始化所述图像升频系统中的各参数包括：
    按照以下公式初始化图像升频系统中第一卷积神经网络模块和第二卷积神经网络模块的各卷积层的各滤波单元的权值Wij：
    Figure PCTCN2016075338-appb-100002
    Figure PCTCN2016075338-appb-100003
    其中，m表示输入所述滤波单元的特征图像的数量；uniform(-1,1)表示在(-1,1)之间选取的随机数；
    将各滤波单元的偏置初始化为0。
  14. 一种采用如权利要求1-9任一项所述的图像升频系统进行图像升频的方法，包括：
    第一卷积神经网络模块将输入到所述第一卷积神经网络模块的输入图像转换为多个特征图像并输出；
    复合器将输入到所述复合器的特征图像中每n*n个特征图像合成一分辨率为输入的所述特征图像的分辨率n*n倍的特征图像并输出；输入到所述复合器的特征图像的数量为n*n的倍数，n为大于1的整数。
PCT/CN2016/075338 2015-09-17 2016-03-02 图像升频系统、其训练方法及图像升频方法 WO2017045374A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US15/518,404 US10019642B1 (en) 2015-09-17 2016-03-02 Image upsampling system, training method thereof and image upsampling method

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201510595656.7A CN105120130B (zh) 2015-09-17 2015-09-17 一种图像升频系统、其训练方法及图像升频方法
CN201510595656.7 2015-09-17

Publications (1)

Publication Number Publication Date
WO2017045374A1 true WO2017045374A1 (zh) 2017-03-23

Family

ID=54668007

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2016/075338 WO2017045374A1 (zh) 2015-09-17 2016-03-02 图像升频系统、其训练方法及图像升频方法

Country Status (3)

Country Link
US (1) US10019642B1 (zh)
CN (1) CN105120130B (zh)
WO (1) WO2017045374A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107729885A (zh) * 2017-11-23 2018-02-23 中电科新型智慧城市研究院有限公司 一种基于多重残差学习的人脸增强方法
CN109903219A (zh) * 2019-02-28 2019-06-18 深圳市商汤科技有限公司 图像处理方法及装置、电子设备、计算机可读存储介质
CN110770784A (zh) * 2017-06-21 2020-02-07 佳能株式会社 图像处理装置、成像装置、图像处理方法、程序、以及存储介质

Families Citing this family (26)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2016132150A1 (en) 2015-02-19 2016-08-25 Magic Pony Technology Limited Enhancing visual data using and augmenting model libraries
WO2016156864A1 (en) 2015-03-31 2016-10-06 Magic Pony Technology Limited Training end-to-end video processes
GB201604345D0 (en) * 2016-03-14 2016-04-27 Magic Pony Technology Ltd Super resolution using fidelity transfer
GB201604672D0 (en) * 2016-03-18 2016-05-04 Magic Pony Technology Ltd Generative methods of super resolution
CN105120130B (zh) 2015-09-17 2018-06-29 京东方科技集团股份有限公司 一种图像升频系统、其训练方法及图像升频方法
CN108475331B (zh) * 2016-02-17 2022-04-05 英特尔公司 用于对象检测的方法、装置、系统和计算机可读介质
CN105611303B (zh) * 2016-03-07 2019-04-09 京东方科技集团股份有限公司 图像压缩系统、解压缩系统、训练方法和装置、显示装置
CN105847968B (zh) 2016-03-21 2018-12-21 京东方科技集团股份有限公司 基于深度学习的解像方法和系统
CN107231566B (zh) * 2016-03-25 2020-12-18 阿里巴巴集团控股有限公司 一种视频转码方法、装置和系统
WO2017210174A1 (en) * 2016-05-31 2017-12-07 Linkedin Corporation Training a neural network using another neural network
AU2017324069B2 (en) * 2016-09-06 2019-12-19 Elekta, Inc. Neural network for generating synthetic medical images
CN108074215B (zh) * 2016-11-09 2020-04-14 京东方科技集团股份有限公司 图像升频系统及其训练方法、以及图像升频方法
CN108229455B (zh) 2017-02-23 2020-10-16 北京市商汤科技开发有限公司 物体检测方法、神经网络的训练方法、装置和电子设备
US10410322B2 (en) * 2017-04-05 2019-09-10 Here Global B.V. Deep convolutional image up-sampling
US10489887B2 (en) * 2017-04-10 2019-11-26 Samsung Electronics Co., Ltd. System and method for deep learning image super resolution
CN107124609A (zh) * 2017-04-27 2017-09-01 京东方科技集团股份有限公司 一种视频图像的处理系统、其处理方法及显示装置
CN107169927B (zh) * 2017-05-08 2020-03-24 京东方科技集团股份有限公司 一种图像处理系统、方法及显示装置
CN107610140A (zh) * 2017-08-07 2018-01-19 中国科学院自动化研究所 基于深度融合修正网络的精细边缘检测方法、装置
US10979718B2 (en) * 2017-09-01 2021-04-13 Apple Inc. Machine learning video processing systems and methods
CN107895174B (zh) 2017-11-09 2020-01-07 京东方科技集团股份有限公司 图像分类和转换方法、装置以及图像处理系统
CN110956575B (zh) * 2018-09-26 2022-04-12 京东方科技集团股份有限公司 转变图像风格的方法和装置、卷积神经网络处理器
CN109191382B (zh) * 2018-10-18 2023-12-05 京东方科技集团股份有限公司 图像处理方法、装置、电子设备及计算机可读存储介质
CN110084309B (zh) * 2019-04-30 2022-06-21 北京市商汤科技开发有限公司 特征图放大方法、装置和设备及计算机可读存储介质
CN110426560B (zh) * 2019-06-28 2020-08-25 北京大学 一种脉冲阵列信号时空上采样器生成方法
CN110717405B (zh) * 2019-09-17 2023-11-24 平安科技（深圳）有限公司 人脸特征点定位方法、装置、介质及电子设备
CN112788338B (zh) * 2020-12-31 2022-08-26 展讯通信(天津)有限公司 图像压缩及解压缩方法、设备、装置和存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7187811B2 (en) * 2003-03-18 2007-03-06 Advanced & Wise Technology Corp. Method for image resolution enhancement
CN101639937A (zh) * 2009-09-03 2010-02-03 复旦大学 一种基于人工神经网络的超分辨率方法
CN105120130A (zh) * 2015-09-17 2015-12-02 京东方科技集团股份有限公司 一种图像升频系统、其训练方法及图像升频方法
CN204948182U (zh) * 2015-09-17 2016-01-06 京东方科技集团股份有限公司 一种图像升频系统及显示装置

Family Cites Families (11)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US4416288A (en) * 1980-08-14 1983-11-22 The Regents Of The University Of California Apparatus and method for reconstructing subsurface electrophysiological patterns
US5138695A (en) * 1989-10-10 1992-08-11 Hnc, Inc. Systolic array image processing system
US7028271B2 (en) * 2002-11-06 2006-04-11 Canon Kabushiki Kaisha Hierarchical processing apparatus
JP4546157B2 (ja) * 2004-06-03 2010-09-15 キヤノン株式会社 情報処理方法、情報処理装置、撮像装置
US9202144B2 (en) * 2013-10-30 2015-12-01 Nec Laboratories America, Inc. Regionlets with shift invariant neural patterns for object detection
CN104573731B (zh) * 2015-02-06 2018-03-23 厦门大学 基于卷积神经网络的快速目标检测方法
US10726328B2 (en) * 2015-10-09 2020-07-28 Altera Corporation Method and apparatus for designing and implementing a convolution neural net accelerator
US11074492B2 (en) * 2015-10-07 2021-07-27 Altera Corporation Method and apparatus for performing different types of convolution operations with the same processing elements
US10614354B2 (en) * 2015-10-07 2020-04-07 Altera Corporation Method and apparatus for implementing layers on a convolutional neural network accelerator
US10997496B2 (en) * 2016-08-11 2021-05-04 Nvidia Corporation Sparse convolutional neural network accelerator
AU2017338783B2 (en) * 2016-10-04 2022-02-10 Magic Leap, Inc. Efficient data layouts for convolutional neural networks

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7187811B2 (en) * 2003-03-18 2007-03-06 Advanced & Wise Technology Corp. Method for image resolution enhancement
CN101639937A (zh) * 2009-09-03 2010-02-03 复旦大学 一种基于人工神经网络的超分辨率方法
CN105120130A (zh) * 2015-09-17 2015-12-02 京东方科技集团股份有限公司 一种图像升频系统、其训练方法及图像升频方法
CN204948182U (zh) * 2015-09-17 2016-01-06 京东方科技集团股份有限公司 一种图像升频系统及显示装置

Non-Patent Citations (2)

* Cited by examiner, † Cited by third party
Title
CHAO, DONG ET AL.: "Learning a Deep Convolutional Network for Image Super-Resolution", COMPUTER VERSION-ECCV 2014, 13TH EUROPEAN CONFERENCE, ZURICH, SWITZERLAND , SEPTEMBER 6-12, 2014 , PROCEEDINGS, PART IV, 12 September 2014 (2014-09-12), pages 184 - 199, XP047296566 *
LIU, NA ET AL.: "Single Image Super-Resolution Reconstruction via Deep Convolutional Neural Network", CHINA SCIENCEPAPER, vol. 10, no. 2, 31 January 2015 (2015-01-31), pages 201 - 206, ISSN: 2095-2783 *

Cited By (6)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110770784A (zh) * 2017-06-21 2020-02-07 佳能株式会社 图像处理装置、成像装置、图像处理方法、程序、以及存储介质
CN110770784B (zh) * 2017-06-21 2023-09-01 佳能株式会社 图像处理装置、成像装置、图像处理方法、程序、以及存储介质
CN107729885A (zh) * 2017-11-23 2018-02-23 中电科新型智慧城市研究院有限公司 一种基于多重残差学习的人脸增强方法
CN107729885B (zh) * 2017-11-23 2020-12-29 中电科新型智慧城市研究院有限公司 一种基于多重残差学习的人脸增强方法
CN109903219A (zh) * 2019-02-28 2019-06-18 深圳市商汤科技有限公司 图像处理方法及装置、电子设备、计算机可读存储介质
CN109903219B (zh) * 2019-02-28 2023-06-30 深圳市商汤科技有限公司 图像处理方法及装置、电子设备、计算机可读存储介质

Also Published As

Publication number Publication date
CN105120130B (zh) 2018-06-29
US20180197037A1 (en) 2018-07-12
CN105120130A (zh) 2015-12-02
US10019642B1 (en) 2018-07-10

Similar Documents

Publication Publication Date Title
WO2017045374A1 (zh) 图像升频系统、其训练方法及图像升频方法
US10515435B2 (en) Apparatus for upscaling an image, method for training the same, and method for upscaling an image
US10311547B2 (en) Image upscaling system, training method thereof, and image upscaling method
JP4465112B2 (ja) Ｌｃｄパネルにおける画像表示に適したｄｗｔに基づくアップサンプリング・アルゴリズム
CN107169927B (zh) 一种图像处理系统、方法及显示装置
CN204948182U (zh) 一种图像升频系统及显示装置
US10943340B2 (en) Blending images
He et al. FPGA-based real-time super-resolution system for ultra high definition videos
CN110047038B (zh) 一种基于层级递进网络的单图像超分辨重建方法
Manabe et al. FPGA implementation of a real-time super-resolution system using a convolutional neural network
US20180082149A1 (en) Clustering method with a two-stage local binary pattern and an iterative image testing system thereof
CN116485651A (zh) 一种图像超分辨率重建方法
JP5203824B2 (ja) 画像処理装置及び撮像システム
CN105469399B (zh) 一种面向混合噪声的人脸超分辨率的重建方法及装置
US6678708B1 (en) Method and apparatus for two-dimensional separable symmetric filtering
US20230153944A1 (en) Super resolution image generating device
JP2020064557A (ja) 画像処理装置、画像処理システム、撮像装置、画像処理方法、および、プログラム
US20240135506A1 (en) Adaptive sharpening for blocks of pixels
CN111724292A (zh) 图像处理方法、装置、设备以及计算机可读介质
Azgin et al. A high performance alternating projections image demosaicing hardware
KR100300338B1 (ko) 2차원 이산 웨이브렛 변환을 위한 초고밀도 집적회로 구조
CN111461987B (zh) 一种网络构建方法、图像超分辨率重建方法及系统
Park et al. Image enhancement by recurrently-trained super-resolution network
CN104301584B (zh) 图像处理系统
CN112488916B (zh) 一种图像超分重建模型的训练方法和计算机设备

Legal Events

Date Code Title Description
WWE Wipo information: entry into national phase

Ref document number: 15518404

Country of ref document: US

121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 16845490

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 16845490

Country of ref document: EP

Kind code of ref document: A1

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 23.08.2018)

122 Ep: pct application non-entry in european phase

Ref document number: 16845490

Country of ref document: EP

Kind code of ref document: A1