WO2018205627A1 - 一种图像处理系统、方法及显示装置 - Google Patents

一种图像处理系统、方法及显示装置 Download PDF

Info

Publication number
WO2018205627A1
WO2018205627A1 PCT/CN2017/117116 CN2017117116W WO2018205627A1 WO 2018205627 A1 WO2018205627 A1 WO 2018205627A1 CN 2017117116 W CN2017117116 W CN 2017117116W WO 2018205627 A1 WO2018205627 A1 WO 2018205627A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
resolution
feature
feature image
processing system
Prior art date
Application number
PCT/CN2017/117116
Other languages
English (en)
French (fr)
Inventor
刘瀚文
那彦波
Original Assignee
京东方科技集团股份有限公司
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by 京东方科技集团股份有限公司 filed Critical 京东方科技集团股份有限公司
Priority to US16/073,712 priority Critical patent/US11216910B2/en
Publication of WO2018205627A1 publication Critical patent/WO2018205627A1/zh

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4046—Scaling the whole image or part thereof using neural networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/048—Activation functions
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4007—Interpolation-based scaling, e.g. bilinear interpolation
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T3/00—Geometric image transformation in the plane of the image
    • G06T3/40—Scaling the whole image or part thereof
    • G06T3/4053—Super resolution, i.e. output image resolution higher than sensor resolution
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/50—Image enhancement or restoration by the use of more than one image, e.g. averaging, subtraction
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/0002—Inspection of images, e.g. flaw detection
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/06—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons
    • G06N3/063—Physical realisation, i.e. hardware implementation of neural networks, neurons or parts of neurons using electronic means
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10024—Color image
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20212—Image combination
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/30—Subject of image; Context of image processing
    • G06T2207/30168—Image quality inspection

Definitions

  • the present disclosure relates to the field of image signal processing technologies, and in particular, to an image processing system, method, and display device.
  • the resolution of an image is generally improved by using standard upscaling (increasing image resolution) such as bicubic and linear.
  • standard upscaling increasing image resolution
  • the image signal processing method in the related art has a problem that the flexibility of image processing is poor.
  • an embodiment of the present disclosure provides an image processing system including at least one resolution conversion subsystem, the resolution conversion subsystem including a concatenated convolutional neural network module, a composite, and Activate the module.
  • the convolutional neural network module is configured to perform convolution processing on the input signal to obtain a plurality of first feature images having a first resolution.
  • the multiplexer is configured to synthesize a second feature image having a second resolution greater than the first resolution using the first feature image.
  • the activation module is coupled to the recombiner for selecting the second feature image using an activation function.
  • embodiments of the present disclosure also provide a display device including an image processing system including at least one resolution conversion subsystem.
  • the resolution conversion subsystem includes a concatenated convolutional neural network module, a recombiner, and an activation module.
  • the convolutional neural network module is configured to perform convolution processing on the input signal to obtain a plurality of first feature images having a first resolution.
  • the multiplexer is configured to synthesize a second feature image having a second resolution greater than the first resolution using the first feature image.
  • the activation module is coupled to the recombiner for selecting the second feature image using an activation function.
  • an embodiment of the present disclosure provides an image processing method, including: performing convolution processing on an input signal to obtain a plurality of first feature images having a first resolution; and synthesizing the image by using the first feature image a second feature image of the second resolution of the first resolution; the second feature image is selected using an activation function.
  • an embodiment of the present disclosure provides an electronic device, including: one or more processors; a memory; and one or more programs. Wherein the one or more programs are stored in the memory and are configured to be executed by the one or more processors, the steps of the image processing method described above being implemented when the program is executed.
  • an embodiment of the present disclosure provides a computer readable storage medium having stored thereon a computer program that, when executed by a processor, implements the steps in the image processing method described above.
  • 1 is a schematic diagram of an up-conversion of 2x in the related art
  • FIG. 2 is a schematic diagram of frequency up-conversion of a combiner in an image upscaling system in the related art
  • FIG. 3 is a structural diagram of an image processing system according to an embodiment of the present disclosure.
  • FIG. 4 is a structural diagram of a resolution conversion subsystem in an image processing system according to an embodiment of the present disclosure
  • FIG. 5 is a second structural diagram of a resolution conversion subsystem in an image processing system according to an embodiment of the present disclosure
  • FIG. 6 is a third structural diagram of a resolution conversion subsystem in an image processing system according to an embodiment of the present disclosure.
  • FIG. 7a is a fourth structural diagram of a resolution conversion subsystem in an image processing system according to an embodiment of the present disclosure.
  • FIG. 7b is a fifth structural diagram of a resolution conversion subsystem in an image processing system according to an embodiment of the present disclosure.
  • FIG. 7c is a sixth structural diagram of a resolution conversion subsystem in an image processing system according to an embodiment of the present disclosure.
  • FIG. 8 is a second structural diagram of an image processing system according to an embodiment of the present disclosure.
  • FIG. 9 is a third structural diagram of an image processing system according to an embodiment of the present disclosure.
  • FIG. 10 is a fourth structural diagram of an image processing system according to an embodiment of the present disclosure.
  • FIG. 11 is a schematic diagram of a recombining process performed by a synthesizer in a resolution conversion subsystem of an image processing system according to an embodiment of the present disclosure
  • FIG. 12 is a flowchart of an image processing method according to an embodiment of the present disclosure.
  • FIG. 13 is a structural diagram of an electronic device according to an embodiment of the present disclosure.
  • FIG. 1 is a schematic diagram of an up-conversion of 2x in the related art.
  • a 2x up-conversion mode four different filters F1, F2, F3, and F4 are used for each pixel of the input image (plus neighboring pixels), each of which The filter produces a quarter of the pixels of the output image.
  • This process can be thought of as applying a 4-filter (convolution) to the input image and then interleaving or multiplexing to create a single output image that doubles the width and height;
  • the image processing method makes the data calculation amount large, and the frequency up multiple cannot be flexibly adjusted.
  • a method for improving the resolution of a video image by using a convolutional neural network is proposed in the related art; as shown in FIG. 2, the input feature image is input.
  • Each of the four feature images is a group, and the pixel values of the image are intersected to generate a feature image of 4 times pixels; obviously, the processing method can only generate the feature image in a fixed pixel value arrangement manner.
  • the image signal processing method in the related art has a problem that the flexibility of image processing is poor.
  • embodiments of the present disclosure provide an image processing system, method, and display device.
  • Embodiments of the present disclosure provide an image processing system, as shown in FIG. 3, that includes at least one resolution conversion subsystem 10.
  • the resolution conversion subsystem 10 includes a concatenated convolutional neural network module 110, a recombiner 120, and an activation module 130.
  • the convolutional neural network module 110 is configured to perform convolution processing on the input signal to obtain a plurality of first feature images having a first resolution.
  • the recombiner 120 is configured to synthesize a second feature image having a second resolution greater than the first resolution by using the first feature image.
  • the activation module 130 is coupled to the recombiner 120 for selecting the second feature image using an activation function.
  • one or more concatenated convolutional neural network subsystems 20 may also be included in the image processing system.
  • the connection relationship between the convolutional neural network subsystem 20 and the resolution conversion subsystem 10 can be set according to actual conditions.
  • each input of the resolution conversion subsystem 10 is cascaded with two convolutional neural network subsystems 20, and the output is cascaded with two convolutional neural network subsystems 20.
  • the number and location of the convolutional neural network subsystems 20 may be other ways, which are not enumerated here.
  • the convolutional neural network subsystem 20 includes a concatenated convolutional neural network module and an activation module.
  • an image processed with an image processing system may be a continuously played video image.
  • the input signal includes not only an image to be processed but also a noise image; the noise image may manually generate details, and an input image with a noise greater than 1 More details can be generated to help with more detailed processing of the video image; not only that, for video images, adding a noise image to the input signal also helps to input low-resolution frames and several past frames. To keep the video image stable.
  • the activation module 130 is provided in the resolution conversion subsystem 10, and the activation module 130 can adaptively select the second feature image output by the recombiner 120, The second feature image of different pixel arrangement manners can be generated, thereby improving the flexibility of image processing.
  • the synthesizer 120 directly performs the enlargement processing by using the first feature image outputted by the convolutional neural network module 110, it is possible to ensure that as many feature images as possible participate in the enlargement process, thereby improving the enlargement effect.
  • the convolutional neural network module 110 and the recombiner 120 in the resolution conversion subsystem 10 are connected in the following manners, as shown in FIG. 4 to FIG. 6 .
  • the resolution conversion subsystem 10 may include a convolutional neural network module 110, a recombiner 120, and an activation module 130, and convolute the signal output of the neural network module 110 with the signal of the recombiner 120.
  • the input is connected, and the signal output of the combiner 120 is connected to the input of the activation module 130.
  • the resolution conversion subsystem 10 may include a convolutional neural network module 110, two recombiners 120, and an activation module 130, and the signal output of the convolutional neural network module 110 and the first stage recombiner
  • the signal input end of 120 is connected
  • the signal output end of the first stage combiner 120 is connected to the signal input end of the second stage combiner 120
  • the resolution conversion subsystem 10 may include two convolutional neural network modules 110, two recombiners 120, and two activation modules 130, and the convolutional neural network module 110, the recombiner 120, and the activation module 130 in turn Interleaved connections.
  • a high quality first feature image may be used in order to perform amplification processing for each recombiner.
  • a high-quality second feature image having a resolution of n times is synthesized.
  • the signal input end of each recombiner 120 is connected to the output end of the convolutional neural network module 110, and the signal output of each recombiner 120 is output.
  • the terminal is connected to the input end of the activation module 130.
  • the convolutional neural network module 110 is used to obtain the first feature image and then input to the input end of the corresponding recombiner 120, that is, convolutional neural network module in the resolution conversion subsystem 10. 110.
  • the combiner 120 and the activation module 130 can be arranged in pairs.
  • magnifications of the resolution conversion subsystems 10 may be the same or different, and are not limited herein.
  • the recombination numbers of the recombiners 120 may be the same or different, and are not limited herein.
  • the magnification of the two recombiners 120 is M times
  • the resolution of the image will be enlarged to the original M 2 times after the picture passes through the resolution conversion subsystem 10;
  • the recombiner may include a plurality of sub-compositors; and, according to a multiple of the input of the first feature image, the amplification factor may be factorized, that is, the larger magnification is decomposed into multiple A smaller magnification allows each sub-compositor to process the input first feature image at a smaller magnification. Therefore, the number of sub-combiners is generally set to the number of factorizations of the resolution magnification of the output second feature image with respect to the input first feature image.
  • a vertical sub-combiner for enlarging the column direction of the input first feature image may be set according to the resolution of the input first feature image in the row direction and the resolution in the column direction, And a horizontal sub-combiner for enlarging the row direction of the input feature image, so that the vertical sub-compositor and the horizontal sub-compositor respectively perform amplification processing on the input first feature image in the column direction and the row direction. Therefore, the sub-compositor may include a vertical sub-compositor for enlarging the column direction of the input first feature image, and a horizontal sub-combiner for enlarging the row direction of the input feature image.
  • the recombiner is a recombiner with a resolution of M times, and the recombiner includes:
  • the sub-combiner can be set in the following two ways.
  • the magnification 36x can be factorized into an amplification of 4x (2 x 2) and an amplification of 9x (3 x 3), that is, the larger magnification factor is factorized into two smaller magnifications.
  • the number of sub-combiners is set to two, respectively, a sub-combiner 121 for amplifying the resolution of the input first feature image by 4x, and a The resolution of a feature image is amplified by a 9x sub-combiner 122.
  • the sub-compositor may be provided with a vertical sub-combiner for enlarging the column direction of the input first feature image, and a horizontal sub-combiner for enlarging the row direction of the input first feature image.
  • the sub-compositor may include: a first vertical sub-combiner 123 with a magnification of 2x (1 ⁇ 2), and a magnification of 3x.
  • each vertical sub-compositor and each horizontal sub-compositor can also be arranged crosswise, as shown in Figure 7c.
  • each second feature image output by the multiplexer 120 since the pixel arrangement manner of each second feature image output by the multiplexer 120 is different, that is, multiple second feature images can be generated by using multiple pixel value arrangement manners, so that not only the input signal can be made.
  • the information of each of the first feature images is not lost in the generated second feature image, and the next layer connected to the signal output end of the activation module 130 can be adaptively selected. The flexibility of image processing is improved while keeping image information intact.
  • the recombiner deployed in the resolution conversion subsystem 10 of the first stage is a horizontal subcomplexer for amplifying the row direction of the input first feature image, deployed in the resolution conversion subsystem 10 of the second stage.
  • the recombiner is a horizontal subcombiner for amplifying the column direction of the input first feature image.
  • the image processing system may further include:
  • the conversion module is configured to convert the received input image of the RGB format into a YUV format, and output a Y channel signal, a U channel signal, and a V channel signal.
  • the at least one resolution conversion subsystem 10 performs resolution amplification processing on the Y channel signal, the U channel signal, and/or the V channel signal.
  • the YUV format is a picture format composed of Y, U, and V, where Y represents brightness, and U and V represent color chromaticity.
  • the Y-channel signal, the U-channel signal, and/or the V-channel signal may be subjected to resolution amplification processing using the above-described resolution conversion subsystem.
  • the amplification factors of the Y channel signal, the U channel signal, and the V channel signal are all implemented by the resolution conversion subsystem in this embodiment.
  • the amplification factor of the Y channel signal is implemented by the resolution conversion subsystem in this embodiment.
  • the amplification factor of the U channel signal and the V channel signal is implemented by using the resolution conversion subsystem in the embodiment and the resolution amplification subsystem in the related art.
  • the magnification of the resolution of the Y channel signal by the at least one resolution conversion subsystem is N times;
  • the U channel signal is amplified by a resolution amplification subsystem by at least one resolution conversion subsystem by N1 times, and N1 is less than N;
  • the V channel signal is amplified by a resolution of at least one resolution conversion subsystem by a factor of N2, and N2 is less than N.
  • the size of N1 and N2 can be set according to actual needs, and is not further limited herein. For example, if the above N is 100, and both N1 and N2 are 50, in the embodiment of the present disclosure, the U channel signal and the V signal are processed by setting at least one resolution conversion subsystem to perform resolution amplification 10 times, and the related technology is set.
  • the resolution amplification subsystem performs a 10x magnification process. It should be understood that, in order to reduce the amount of calculation of the amplification, in general, after the amplification is performed 10 times by at least one resolution conversion subsystem in the present disclosure, the resolution amplification subsystem in the related art is used to perform the amplification 10 times.
  • the calculation algorithm of the resolution amplification subsystem in the related art may be selected according to actual needs. For example, the pixels of the image may be amplified by a standard up-conversion method such as bicubic and linear.
  • the at least one resolution conversion subsystem performs resolution amplification processing only on the Y channel signal.
  • the U channel signal and the V channel signal are amplified by a resolution amplification subsystem in the related art.
  • the resolution conversion subsystem in the embodiment of the present disclosure needs to perform the selection of the second feature image after the first feature image is enlarged to the second feature image, so the calculation amount of the enlargement is large.
  • the amplification process is implemented by any of the methods of FIG. 8 to FIG.
  • the image it includes two aspects: color and grayscale, and corresponding to the amplification technique, in which the processing of the grayscale information has a greater influence on the magnification effect than the color information. Therefore, according to the above recognition of the amplification, in the specific embodiment of the present disclosure, after converting the RGB image into the YUV format, using the amplification mode shown in FIG. 9 and FIG. 10, the resource overhead of the system can be reduced, and the amplification effect can be ensured. .
  • one or more resolution conversion subsystems may be used for each channel signal, and the number of resolution conversion subsystems set for each channel may be set according to actual needs, and is not further limited herein.
  • the manner in which the activation module 130 selects the second feature image may be set according to actual needs.
  • the activation module 130 is specifically configured to output the second feature image or the second feature image after the offset is applied when the condition is satisfied, or discard the second feature image.
  • the second feature image may be compared with -b (negative number of the offset parameter b), and whether the second feature image is output is determined according to the result of the comparison. And how to output the second feature image. Specifically, when the second feature image is greater than -b, the second feature image may be directly output or the second feature image after the offset b is increased. When the second feature image is less than or equal to -b, the first feature image is directly discarded. Two feature images, the output is 0.
  • the second feature image has a weight parameter a in the convolution process.
  • the weight parameter a and the offset parameter b are generally included, and in the specific embodiment of the present disclosure, the present disclosure finds through continuous experiments that only the weight parameter can be used in the convolution process. At the same time, the amplification effect of the weight parameter a and the offset parameter b is used, and the training and debugging difficulty is also reduced.
  • the recombiner is a recombiner with a resolution magnification of M times, specifically for: combining the pixel values of each M first feature image to synthesize M second feature images. And output.
  • the amplification factor Mx of the input first feature image relative to the output second feature image in the row direction and the amplification factor My in the column direction are determined, Mx and My are positive integers.
  • the composite device is configured to: after the pixel values of the M first input feature images in the input first feature image are intersected, synthesize the M feature resolutions into a second feature image of the input first feature image M times and output The number of input first feature images is an integer multiple of M. Wherein, the second feature image is magnified Mx times in the row direction and My times in the column direction as compared with the first feature image.
  • the recombiner 120 in the resolution conversion subsystem 10 shown in FIG. 6 is taken as an example.
  • the magnification of the recombiner 120 is 4, that is, the amplification factor of the input first feature image relative to the output second feature image in the row direction is Mx is 2, and the input first feature image is relative to the output.
  • the magnification factor of the second feature image in the column direction is My, so that the recombiner 120 crosses the pixel values of the input four first feature images to synthesize 4 resolutions and enlarges the input into the first feature image 4 times.
  • the feature image is output to the activation module 130 for selective adaptive selection, and then output to the next level for amplification, and adaptive selection is performed again, thereby improving the flexibility of video image processing.
  • each of the recombiners may be an adaptive interpolation filter.
  • each of the first first feature images in the input first feature image is a set of pixel values interleaved with the image, and then generates a second feature that is 4 times the resolution of the first feature image.
  • the image is outputted; as shown in FIG. 11 , the working principle of the recombiner is to matrix-arrange the pixel values of the same pixel position in the four input first feature images in the output second feature image, and therefore, Any pixel information in the feature image is not modified (lost or added) during this zooming process.
  • the composite device synthesizes the second feature image by using the following formula:
  • i has a value range of (0...H-1)
  • j has a value range of (0...W-1)
  • p has a value range of (0...MyH-1).
  • the value range of q is (0...MxW-1)
  • H represents the height of the second feature image
  • W represents the width of the second feature image
  • c represents the number of the first feature image
  • Mx represents the first feature image.
  • My represents the magnification factor of the first feature image relative to the second feature image in the column direction
  • % is a remainder operator.
  • the training of the convolutional neural network module and the activation module is determined based on a cost function of a structural similarity SSIM criterion.
  • the cost function can be:
  • L1(W) is the L1 norm of the weight parameter
  • L1(b) is the L1 norm of the offset parameter
  • eps is a constant
  • the parameters involved only include the convolutional neural network.
  • the weight parameter and the offset parameter of the activator are the parameters involved in the image processing system of the embodiment of the present disclosure.
  • training images are taken from a database with a large number of sample images, such as 500 images with a resolution of 480*320.
  • the training image may be a randomly selected partial sample image, and then randomly select a certain number of 80*80 image blocks, such as 30,000, from the selected sample image.
  • These randomly selected image blocks constitute the training image of the embodiment of the present disclosure.
  • the original image is input to the parameter-initialized image processing system, and the parameters of the image processing system are adjusted to cause the parameter-adjusted image processing system to perform an X-magnification operation on the original image.
  • the parameter adjustment can be performed by various existing algorithms, such as a standard random gradient descent SGD algorithm.
  • Whether the parameter is suitable or not can be determined by the loss function COST of the target image and the training image.
  • the degree of difference between the two is small, it indicates that the image processing system has a better amplification effect, and the COST is smaller, and vice versa.
  • the loss function COST can be processed using various criteria, such as the commonly used mean square error MSE.
  • the inventors have studied hard in the process of implementing the embodiments of the present disclosure, and found that the SSIM (Structural Similarity) criterion can better train the convolutional neural network and the activator.
  • SSIM Structuretural Similarity
  • SSIM is a full reference image quality evaluation index, which measures image similarity from three aspects of brightness, contrast and structure, as defined below:
  • u O and u R represent the mean values of the target image and the original image, respectively
  • ⁇ O and ⁇ R represent the variance of the target image and the original image, respectively
  • ⁇ OR represents the covariance of the target image and the original image Y, and C1 and C2 are non-zero. constant.
  • the sliding window can be used to divide the image into blocks, so that the total number of blocks is N.
  • the mean, variance and covariance of each window are calculated by Gaussian weighting, and then the corresponding block is calculated.
  • the loss function COST can be expressed as 1-SSIM, which identifies the degree of difference between the target image and the original image, namely:
  • the COST function may also add other parameters, such as the L1 norm of the weight parameter.
  • INPUT is the original image
  • Downscale (Output (INPUT) is the training image
  • MSE is the image mean square error calculation function
  • SSIM is the structural similarity calculation function.
  • the neuroconvolution network (including the neuroconvolution module and activator) includes weight parameters and offset parameters. Ideally, the offset parameters should be exhausted relative to the weight parameters. May be big.
  • a specific embodiment of the present disclosure defines a function weight offset ratio function as follows:
  • L1(W) is the L1 norm of the weight parameter
  • L1(B) is the L1 norm of the offset parameter as follows:
  • 1 is the layer number of the neural network
  • f is the feature number
  • the filter size is N*M.
  • eps is a very small constant, such as le-6.
  • Cost 1-SSIM(Output(INPUT)+REFRENCE)+ ⁇ 1 DownReg(INPUT)+ ⁇ 2 WBratioReg(W,b)
  • ⁇ 1 is the weight coefficient of DownReg(INPUT)
  • ⁇ 2 is the weight of WBratioReg(W,b)
  • the coefficient, its specific value depends on its importance.
  • an embodiment of the present disclosure further provides a display device including the above image processing system as provided by an embodiment of the present disclosure.
  • the display device can be any product or component having a display function, such as a mobile phone, a tablet computer, a television, a display, a notebook computer, a digital photo frame, a navigator, and the like.
  • a display function such as a mobile phone, a tablet computer, a television, a display, a notebook computer, a digital photo frame, a navigator, and the like.
  • the display device reference may be made to the embodiment of the image processing system described above, and the repeated description is omitted. Since the above image processing system is employed in the display device, it has the same advantageous effects as the image processing system described above.
  • an embodiment of the present disclosure further provides an image processing method, which may be executed in the above image processing system.
  • an image processing method provided by an embodiment of the present disclosure includes steps. 1201-1203.
  • step 1201 convolution processing is performed on the input signal to obtain a plurality of first feature images having the first resolution.
  • Step 1202 Synthesize a second feature image having a second resolution greater than the first resolution by using the first feature image.
  • Step 1203 Select the second feature image by using an activation function.
  • step 1201 may be performed by a convolutional neural network module of the resolution conversion subsystem
  • step 1202 may be performed by a recombiner of the resolution conversion subsystem
  • step 1203 may be performed by an activation module of the resolution conversion subsystem.
  • the specific structure of the convolutional neural network module, the compositor, and the activation module can be referred to the foregoing embodiment, and details are not described herein again.
  • An image processing method provided by an embodiment of the present disclosure first performs convolution processing on an input signal to obtain a plurality of first feature images having a first resolution; and then synthesizes a second image having a greater resolution than the first resolution by using the first feature image. a second feature image of resolution; finally selecting the second feature image using an activation function.
  • the activation module is configured to adaptively select the second feature image output by the multiplexer in the resolution conversion subsystem, the second feature image of different pixel arrangement manners can be generated, thereby improving the flexibility of image processing.
  • the recombiner directly uses the first feature image outputted by the convolutional neural network module to perform amplification processing, it is possible to ensure that as many feature images as possible participate in the amplification process, thereby improving the amplification effect.
  • FIG. 13 is a structural diagram of an electronic device according to an embodiment of the present disclosure.
  • the electronic device includes: at least one processor 1301, a memory 1302, at least one network interface 1304, and a user interface 1303.
  • the various components in the electronic device are coupled together by a bus system 1305.
  • the bus system 1305 is used to implement connection communication between these components.
  • the bus system 1305 includes a power bus, a control bus, and a status signal bus in addition to the data bus.
  • various buses are labeled as the bus system 1305 in FIG.
  • the user interface 1303 may include a display, a keyboard, or a pointing device (eg, a mouse, a track ball, a touch pad, or a touch screen, etc.).
  • a pointing device eg, a mouse, a track ball, a touch pad, or a touch screen, etc.
  • the memory 1302 in the embodiments of the present disclosure may be a volatile memory or a non-volatile memory, or may include both volatile and non-volatile memory.
  • the non-volatile memory may be a read-only memory (ROM), a programmable read only memory (PROM), an erasable programmable read only memory (Erasable PROM, EPROM), or an electric Erase programmable read-only memory (EEPROM) or flash memory.
  • the volatile memory can be a Random Access Memory (RAM) that acts as an external cache.
  • RAM Random Access Memory
  • many forms of RAM are available, such as static random access memory (SRAM), dynamic random access memory (DRAM), synchronous dynamic random access memory (Synchronous D RAM).
  • Memory 1302 described herein is intended to comprise, without being limited to, these and any other suitable types of memory.
  • the memory 1302 stores elements, executable modules or data structures, or a subset thereof, or their extended set: an operating system 13021 and an application 13022.
  • the operating system 13021 includes various system programs, such as a framework layer, a core library layer, a driver layer, and the like, for implementing various basic services and processing hardware-based tasks.
  • the application 13022 includes various applications, such as a Media Player, a browser, etc., for implementing various application services.
  • a program implementing the method of the embodiments of the present disclosure may be included in the application 13022.
  • the program or the instruction stored in the memory 1302 is specifically a program or an instruction stored in the application 13022.
  • the processor 1301 is configured to: perform convolution processing on the input signal to obtain the first a plurality of first feature images of resolution; synthesizing a second feature image having a second resolution greater than the first resolution using the first feature image; selecting the second feature image using an activation function.
  • the embodiment of the present disclosure further provides a computer readable storage medium having stored thereon a computer program, the computer program being executed by a processor to implement the steps in the image processing method in any one of the above method embodiments.
  • the disclosed apparatus and method may be implemented in other manners.
  • the device embodiments described above are merely illustrative.
  • the division of the unit is only a logical function division.
  • there may be another division manner for example, multiple units or components may be combined or Can be integrated into another system, or some features can be ignored or not executed.
  • the mutual coupling or direct coupling or communication connection shown or discussed may be an indirect coupling or communication connection through some interface, device or unit, and may be in an electrical, mechanical or other form.
  • the units described as separate components may or may not be physically separated, and the components displayed as units may or may not be physical units, that is, may be located in one place, or may be distributed to multiple network units. Some or all of the units may be selected according to actual needs to achieve the objectives of the embodiments of the present disclosure.
  • each functional unit in various embodiments of the present disclosure may be integrated into one processing unit, or each unit may exist physically separately, or two or more units may be integrated into one unit.
  • the functions may be stored in a computer readable storage medium if implemented in the form of a software functional unit and sold or used as a standalone product. Based on such understanding, the portion of the technical solution of the present disclosure that contributes in essence or to the prior art or the portion of the technical solution may be embodied in the form of a software product stored in a storage medium, including The instructions are used to cause a computer device (which may be a personal computer, server, or network device, etc.) to perform all or part of the steps of the methods described in various embodiments of the present disclosure.
  • the foregoing storage medium includes various media that can store program codes, such as a USB flash drive, a mobile hard disk, a ROM, a RAM, a magnetic disk, or an optical disk.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • Artificial Intelligence (AREA)
  • Evolutionary Computation (AREA)
  • Biomedical Technology (AREA)
  • Computing Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Software Systems (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Molecular Biology (AREA)
  • Mathematical Physics (AREA)
  • General Engineering & Computer Science (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Quality & Reliability (AREA)
  • Image Processing (AREA)

Abstract

本公开提供一种图像处理系统、方法及显示装置，该图像处理系统包括至少一个分辨率转换子系统。所述分辨率转换子系统包括级联的卷积神经网络模块、复合器和激活模块。所述卷积神经网络模块用于对输入信号进行卷积处理，得到具有第一分辨率的多个第一特征图像。所述复合器用于利用所述第一特征图像合成具有大于第一分辨率的第二分辨率的第二特征图像。所述激活模块与所述复合器连接，用于利用激活函数对所述第二特征图像进行选择。

Description

一种图像处理系统、方法及显示装置
相关申请的交叉引用
本申请主张在2017年5月8日在中国提交的中国专利申请号No.201710324036.9的优先权，其全部内容通过引用包含于此。
技术领域
本公开涉及图像信号处理技术领域，尤其涉及一种图像处理系统、方法及显示装置。
背景技术
目前，在图像信号处理的过程中，一般是利用标准的诸如双三次(bicubic)和线性等标准升频(提高图像分辨率)方式对图像进行分辨率的提升。然而，相关技术中图像信号处理方式存在图像处理的灵活度较差的问题。
发明内容
第一方面，本公开实施例提供了一种图像处理系统，所述图像处理系统包括至少一个分辨率转换子系统，所述分辨率转换子系统包括级联的卷积神经网络模块、复合器和激活模块。所述卷积神经网络模块用于对输入信号进行卷积处理，得到具有第一分辨率的多个第一特征图像。所述复合器用于利用所述第一特征图像合成具有大于第一分辨率的第二分辨率的第二特征图像。所述激活模块与所述复合器连接，用于利用激活函数对所述第二特征图像进行选择。
第二方面，本公开实施例还提供了一种显示装置，该显示装置包括图像处理系统，该图像处理系统包括至少一个分辨率转换子系统。所述分辨率转换子系统包括级联的卷积神经网络模块、复合器和激活模块。所述卷积神经网络模块用于对输入信号进行卷积处理，得到具有第一分辨率的多个第一特征图像。所述复合器用于利用所述第一特征图像合成具有大于第一分辨率的第二分辨率的第二特征图像。所述激活模块与所述复合器连接，用于利用激 活函数对所述第二特征图像进行选择。
第三方面，本公开实施例提供了一种图像处理方法，包括：对输入信号进行卷积处理，得到具有第一分辨率的多个第一特征图像；利用所述第一特征图像合成具有大于第一分辨率的第二分辨率的第二特征图像；利用激活函数对所述第二特征图像进行选择。
第四方面，本公开实施例提供了一种电子设备，包括：一个或多个处理器；存储器；以及一个或多个程序。其中，所述一个或多个程序被存储在所述存储器中，并且被配置成由所述一个或多个处理器执行，所述程序被执行时实现上述的图像处理方法中的步骤。
第五方面，本公开实施例提供了一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现上述图像处理方法中的步骤。
附图说明
为了更清楚地说明本公开实施例的技术方案，下面将对本公开实施例描述中所需要使用的附图作简单地介绍，显而易见地，下面描述中的附图仅仅是本公开的一些实施例，对于本领域普通技术人员来讲，在不付出创造性劳动性的前提下，还可以根据这些附图获得其他的附图。
图1为相关技术中2x的升频示意图；
图2为相关技术中的图像升频系统中复合器的升频示意图；
图3是本公开实施例提供的图像处理系统的结构图之一；
图4是本公开实施例提供的图像处理系统中分辨率转换子系统的结构图之一；
图5是本公开实施例提供的图像处理系统中分辨率转换子系统的结构图之二；
图6是本公开实施例提供的图像处理系统中分辨率转换子系统的结构图之三；
图7a是本公开实施例提供的图像处理系统中分辨率转换子系统的结构图之四；
图7b是本公开实施例提供的图像处理系统中分辨率转换子系统的结构图之五；
图7c是本公开实施例提供的图像处理系统中分辨率转换子系统的结构图之六；
图8是本公开实施例提供的图像处理系统的结构图之二；
图9是本公开实施例提供的图像处理系统的结构图之三；
图10是本公开实施例提供的图像处理系统的结构图之四；
图11是本公开实施例提供的图像处理系统的一分辨率转换子系统中复合器进行放大处理的示意图；
图12是本公开实施例提供的图像处理方法的流程图；及
图13是本公开实施例提供的电子设备的结构图。
具体实施方式
图1为相关技术中2x的升频示意图。如图1所示，在一个2x的升频方式中，对输入图像的各像素(加上邻像素)使用四个不同的滤波器F1、滤波器F2、滤波器F3和滤波器F4，每个滤波器产生四分之一的输出图像的像素，这个过程可以看作是对输入图像应用4个滤波器(卷积)后交错或复用以创建宽度和高度翻倍的单一输出图像；但该图像处理方法使得数据计算量较大，无法灵活调节升频倍数。
为了减少数据计算量，提高升频倍数调节的灵活度，在相关技术中提出了一种通过使用卷积神经网络来提高视频图像的分辨率的方法；如图2所示，将输入的特征图像中的每四个特征图像为一组，将其图像的像素值交叉后，生成一个4倍像素的特征图像；显然，该处理方法只能以固定的像素值排布方式生成特征图像。
可见，相关技术中图像信号处理方式存在图像处理的灵活度较差的问题。
为了解决图像处理的灵活度较差的问题，本公开实施例提供一种图像处理系统、方法及显示装置。
下面将结合本公开实施例中的附图，对本公开实施例中的技术方案进行清楚、完整地描述，显然，所描述的实施例是本公开一部分实施例，而不是 全部的实施例。基于本公开中的实施例，本领域普通技术人员在没有作出创造性劳动前提下所获得的所有其他实施例，都属于本公开保护的范围。
本公开实施例提供了一种图像处理系统，如图3所示，该图像处理系统包括至少一个分辨率转换子系统10。所述分辨率转换子系统10包括级联的卷积神经网络模块110、复合器120和激活模块130。其中，所述卷积神经网络模块110用于对输入信号进行卷积处理，得到具有第一分辨率的多个第一特征图像。所述复合器120用于利用所述第一特征图像合成具有大于第一分辨率的第二分辨率的第二特征图像。激活模块130与所述复合器120连接，用于利用激活函数对所述第二特征图像进行选择。
具体实施例中，图像处理系统中还可以包括一个或者多个级联的卷积神经网络子系统20。其中，卷积神经网络子系统20和分辨率转换子系统10的连接关系可以根据实际情况进行设置。例如，每一分辨率转换子系统10的输入端级联有两个卷积神经网络子系统20，输出端级联有两个卷积神经网络子系统20。当然在其他的实施例中，卷积神经网络子系统20的数量和位置还可以采用其他方式，在此不再一一列举。在本实施例中，卷积神经网络子系统20包括级联的卷积神经网络模块和激活模块。
应理解，利用图像处理系统处理的图像可以是持续播放的视频图像。值得注意的是，在本公开实施例提供的上述处理系统中，输入信号中不仅包括待处理的图像，还可以包括噪声图像；该噪声图像可以人工生成细节，并且，一个噪声大于1的输入图像可以产生更多的细节，有助于对视频图像进行更为细致的处理；不仅如此，针对视频图像，在输入信号中加入噪声图像还有助于输入低分辨率帧和几个过去的帧，以保持视频图像的稳定。
在本公开实施例提供的上述图像处理系统中，由于在分辨率转换子系统10中设置了激活模块130，且激活模块130可对复合器120输出的第二特征图像进行自适应的选择，因此可以生成不同像素排布方式的第二特征图像，从而提高了图像处理的灵活度。此外，由于复合器120直接利用卷积神经网络模块110输出的第一特征图像进行放大处理，能够保证尽可能多的特征图像参与到放大处理过程中，从而提高放大效果。
在具体实施时，在本公开实施例提供的上述图像处理系统中，分辨率转 换子系统10中的卷积神经网络模块110与复合器120的连接方式有以下几种，如图4至图6所示。其中，在图4中，分辨率转换子系统10可以包括一个卷积神经网络模块110、一个复合器120和一个激活模块130，且卷积神经网络模块110的信号输出端与复合器120的信号输入端连接，复合器120的信号输出端与激活模块130的输入端连接。在图5中，分辨率转换子系统10可以包括一个卷积神经网络模块110、两个复合器120和一个激活模块130，且卷积神经网络模块110的信号输出端与第一级的复合器120的信号输入端连接，第一级的复合器120的信号输出端与第二级的复合器120的信号输入端连接，第二级的复合器120的信号输出端与激活模块130的输入端连接。在图6中，分辨率转换子系统10可以包括两个卷积神经网络模块110、两个复合器120和两个激活模块130，且卷积神经网络模块110、复合器120和激活模块130依次交错连接。
具体地，在本公开实施例提供的上述图像处理系统中，分辨率转换子系统10中设置多个复合器120时，为了使每个复合器进行放大处理时可以采用高质量的第一特征图像合成高质量的分辨率为n倍的第二特征图像，如图6所示，一般各复合器120的信号输入端与卷积神经网络模块110的输出端连接，各复合器120的信号的输出端与激活模块130的输入端连接，先采用卷积神经网络模块110来获取第一特征图像后输入到对应的复合器120的输入端，即在分辨率转换子系统10中卷积神经网络模块110、复合器120和激活模块130可以成对设置。
进一步地，在本公开提供的上述图像处理系统中，设置多个分辨率转换子系统10时，各分辨率转换子系统10的放大倍数可以相同也，可以不同，在此不做限定。同一分辨率转换子系统10设置多个复合器120时，各复合器120的放大倍数可以相同，也可以不同，在此不做限定。以图5和图6为例，若两个复合器120的放大倍数均为M倍，那么在图片经过分辨率转换子系统10后，图像的分辨率将放大到原来的M 2倍；若图像处理系统存在两个分辨率转换子系统10，且两个分辨率转换子系统10放大的倍数相同，那么在图片经过图像处理系统后，图像的分辨率将放大到原来的M 4倍。
在具体实施时，在一个分辨率转换子系统中，当需要对输入的第一特征图像进行较大倍数的放大处理时，为了不影响每个复合器的计算精度和处理 速度，在本公开实施例提供的上述处理系统中，复合器可以包括多个子复合器；并且，可以根据对输入的第一特征图像需要放大的倍数，将放大倍数进行因式分解，即将较大的放大倍数分解为多个较小的放大倍数，使各子复合器分别以较小的放大倍数对输入的第一特征图像进行处理。因此，子复合器的个数一般设置为输出的第二特征图像相对于输入的第一特征图像的分辨率放大倍数的因式分解的个数。
此外，在一实施例中，可以根据输入的第一特征图像在行方向上的分辨率和在列方向上的分辨率，设置用于放大输入的第一特征图像的列方向的垂直子复合器，和用于放大输入的特征图像的行方向的水平子复合器，使得垂直子复合器和水平子复合器分别对输入的第一特征图像在列方向和行方向进行放大处理。因此，子复合器可以包括：用于放大输入的第一特征图像的列方向的垂直子复合器，以及用于放大输入的特征图像的行方向的水平子复合器。
也可就是说，在一实施例中，上述复合器为分辨率放大M倍的复合器，所述复合器包括：
级联的多个在行列方向同时进行在行和列方向进行放大处理的子复合器，所述子复合器的放大倍数的乘积等于所述M；
或者
级联的用于在行方向进行放大处理的水平子复合器和用于在列方向进行放大处理的垂直子复合器，所述水平子复合器在行方向的放大倍数与所述垂直子复合器在列方向的放大倍数的乘积等于所述M。
具体地，以图4所示的分辨率转换子系统输出的第二特征图像相对于输入的第一特征图像的分辨率放大36x为例，可以按照以下两种方式设置子复合器。在第一种方式中，可以将放大倍数36x因式分解为放大4x(2×2)和放大9x(3×3)，即将较大地放大倍数因式分解成两个较小的放大倍数。相应地，如图7a所示，将子复合器的个数设置为两个，分别为用于将输入的第一特征图像的分辨率放大4x的子复合器121，和用于将输入的第一特征图像的分辨率放大9x的子复合器122。在第二种方式中，可以使子复合器包括用于放大输入的第一特征图像的列方向的垂直子复合器，以及用于放大输入的第一特征图像 的行方向的水平子复合器。相应地，为了使输入的第一特征图像放大36x，如图7b所示，在子复合器中可以包括：放大倍数为2x(1×2)的第一垂直子复合器123、放大倍数为3x(1×3)的第二垂直子复合器124、放大倍数为2x(2×1)的第一水平子复合器125和放大倍数为3x(3×1)的第二水平子复合器126。当然，各垂直子复合器和各水平子复合器还可以交叉设置，如图7c所示。通过以上两种设置方式，可以对图片进行多层次分布式的处理，有助于对每个处理过程进行优化，以提高视频图像处理的效率。
本实施例中，由于复合器120输出的每个第二特征图像的像素排布方式均不相同，即可以采用多种像素值排布方式生成多个第二特征图像，这样不仅可以使输入信号中各第一特征图像的信息不损失的记载到生成的第二特征图像中，还可以使与激活模块130的信号输出端相连接的下一层级对第二特征图像进行自适应地选择，在保持不丢失图像信息的前提下，提高了图像处理的灵活度。
应当说明的，还可以在不同的分辨率转换子系统10中部署不同类型的复合器实现对图像的分辨率进行放大。例如，在第一级的分辨率转换子系统10中部署的复合器为用于放大输入的第一特征图像的行方向的水平子复合器，在第二级的分辨率转换子系统10中部署的复合器为用于放大输入的第一特征图像的列方向的水平子复合器。应理解，上述第一特征图像和第二特征图像仅仅是用于说明通过复合器转换之前的特征图像，以及通过复合器转换之后的特征图像，并不用于限定图像本身。也就是说，针对所有的复合器，其输入的特征图像为第一特征图像，输出的图像为第二特征图像。
应理解，在具体实施时，上述图像处理系统还可以包括：
转换模块，用于将接收的RGB格式的输入图像转换为YUV格式，输出Y通道信号、U通道信号和V通道信号。
所述至少一个分辨率转换子系统10对所述Y通道信号、U通道信号和/或V通道信号进行分辨率放大处理。
本实施例中，YUV格式为一种图片格式，由Y、U和V组成，其中，Y表示亮度，U和V表示颜色的色度。
参照图8至图10所示，在图像处理系统中，可以采用上述分辨率转换子系 统对所述Y通道信号、U通道信号和/或V通道信号进行分辨率放大处理。
如图8所示，Y通道信号、U通道信号和V通道信号的放大倍数均采用本实施例中的分辨率转换子系统实现。
如图9所示，Y通道信号的放大倍数采用本实施例中的分辨率转换子系统实现。U通道信号和V通道信号的放大倍数采用本实施例中的分辨率转换子系统和相关技术中的分辨率放大子系统配合实现。
例如，上述图像处理系统对所述输入图像的分辨率的放大倍数为N倍，则有Y通道信号通过至少一个分辨率转换子系统进行分辨率放大的放大倍数为N倍；
U通道信号通过至少一个分辨率转换子系统进行分辨率放大的放大倍数为N1倍，N1小于N；
V通道信号通过至少一个分辨率转换子系统进行分辨率放大的放大倍数为N2倍，N2小于N。
其中，N1和N2的大小可以根据实际需要进行设置，在此不做进一步的限定。例如，上述N为100，N1和N2均为50，则在本公开实施例中，U通道信号和V信号通过设置至少一个分辨率转换子系统进行分辨率放大10倍处理，同时设置相关技术中的分辨率放大子系统进行放大10倍处理。应理解，为了减少放大的计算量，通常的，首先通过本公开中的至少一个分辨率转换子系统进行放大10倍之后，再采用相关技术中的分辨率放大子系统进行放大10倍处理。相关技术中的分辨率放大子系统的计算算法可以根据实际需要进行选择，例如可以采用双三次(bicubic)和线性等标准升频的方式对图像的像素进行放大处理。
如图10所示，所述至少一个分辨率转换子系统仅对所述Y通道信号进行分辨率放大处理。U通道信号和V通道信号采用相关技术中的分辨率放大子系统进行放大处理。
由于本公开实施例中的分辨率转换子系统需要在对第一特征图像放大到第二特征图像后，进行第二特征图像的选择，因此放大的计算量较大，在本实施例中，可以综合考虑系统的资源开销，采用图8至图10任一种方式实现放大处理。对于图像而言，都包括两方面的信息：色彩和灰阶，而对应于放大 技术而言，其中灰阶信息的处理对放大效果的影响大于色彩信息。因此，根据上述对放大的认知，本公开具体实施例中，将RGB图像转换为YUV格式之后，采用图9和图10所示的放大方式，可以减小系统的资源开销，同时保证放大效果。
具体的，对于每一通道信号均可以采用一个或者多个分辨率转换子系统，具体的每一通道设置分辨率转换子系统的数量可以根据实际需要进行设置，在此不做进一步的限定。
应当说明的是，上述激活模块130选择第二特征图像的方式可以根据实际需要进行设置。可选的，所述激活模块130具体用于在条件满足时，输出所述第二特征图像或施加偏移量后的第二特征图像，否则丢弃所述第二特征图像。
上述条件可以根据实际需要进行设置，例如，在本实施例中，可以将第二特征图像与-b(偏移量参数b的负数)进行比较，根据比较的结果确定是否输出第二特征图像，以及如何输出第二特征图像。具体的，当第二特征图像大于-b时，可以直接输出第二特征图像或者增加偏移量b之后的第二特征图像，当第二特征图像小于或等于-b时，则直接丢弃该第二特征图像，输出为0。其中，第二特征图像具有卷积处理中的权重参数a。
相关技术中的卷积处理中，通常都包括权重参数a和偏移量参数b，而本公开具体实施例中，本公开通过不断的实验发现，在卷积处理中仅使用权重参数也可达到同时使用权重参数a和偏移量参数b的放大效果，同时还能降低训练和调试难度。
进一步的，上述在分辨率转换子系统10中，上述复合器为分辨率放大M倍的复合器，具体用于：将每M个第一特征图像的像素值交叉后合成M个第二特征图像并输出。
在具体实施时，当分辨率转换子系统10放大的倍数为M倍，确定输入的第一特征图像相对于输出的第二特征图像在行方向上的放大因子Mx和列方向上的放大因子My，Mx和My为正整数。复合器，具体用于将输入的第一特征图像中每M个输入的第一特征图像的像素值交叉后合成M个分辨率放大为输入的第一特征图像M倍的第二特征图像并输出；输入的第一特征图像的数量为M的整数倍。其中，第二特征图像和第一特征图像相比，第二特征图像 在行方向上放大了Mx倍，在列方向上放大了My倍。
具体的，以图6所示的分辨率转换子系统10中的复合器120为例。在图11中，复合器120的放大倍数为4，即输入的第一特征图像相对于输出的第二特征图像在行方向上的放大因子为Mx为2，输入的第一特征图像相对于输出的第二特征图像在列方向上的放大因子为My为2，因此，复合器120将输入的4个第一特征图像的像素值交叉后合成4个分辨率放大为输入的第一特征图像4倍的特征图像，并输出至激活模块130进行选择自适应选择，然后输出到下一层级进行放大，并进行再次自适应选择，因此提高了视频图像处理的灵活度。
在具体实施时，在本公开实施例提供的上述图像处理系统中，每个复合器可以为自适应插值滤波器。如图11所示，将输入的第一特征图像中每4个第一特征图像为一组交错其图像的像素值后生成4个分辨率放大为输入的第一特征图像4倍的第二特征图像并输出；如图11所示，复合器的工作原理是将4个输入的第一特征图像中各相同像素点位置的像素值进行矩阵排列记载在输出的第二特征图像中，因此，在此放大过程中不会修改(丢失或增加)特征图像中的任何像素信息。
具体地，为了确定复合器合成输出的第二特征图像，在本公开实施例提供的上述图像处理系统中，上述复合器采用如下公式合成第二特征图像：
Figure PCTCN2017117116-appb-000001
其中，i的取值范围为(0...H-1)，j的取值范围为(0...W-1)，p的取值范围为(0...MyH-1)，q的取值范围为(0...MxW-1)，H表示第二特征图像的高度，W表示第二特征图像的宽度，c表示第一特征图像的个数，Mx表示第一特征图像相对于第二特征图像在行方向的放大因子，My表示第一特征图像相对于第二特征图像在列方向的放大因子，％为取余运算符，
Figure PCTCN2017117116-appb-000002
表示小于(n-1)/My的最大整数，
Figure PCTCN2017117116-appb-000003
表示第一特征图像，
Figure PCTCN2017117116-appb-000004
均表示的第二特征图像，n表示第二特征图像的编号。
可选的，所述卷积神经网络模块和所述激活模块的训练基于结构相似性SSIM准则的成本函数进行判断。
本实施例中，成本函数可以为：
Cost(INPUT，REFRENCE)
＝1-SSIM(Output(INPUT)+REFRENCE)+λ 1DownReg(INPUT)+λ 2WBratioReg(W，b)其中，INPUT为原始图像，SSIM为结构相似性计算函数，DownReg(INPUT)为描述原始图像和训练图像的相似度函数，λ1为DownReg(INPUT)的权重系数和λ2为WBratioReg(W，b)的权重系数，WBratioReg(W，b)为权重偏移比函数；
Figure PCTCN2017117116-appb-000005
其中L1(W)为权重参数的L1范数，L1(b)为偏移参数的L1范数，eps为常数。
在具体实施时，在系统运行过程中，由于每个滤波器的标量权重，以及偏置的数值相对固定，在系统运行前需要采用一系列的标准输入输出图像来对系统进行训练，并且依靠应用程序调整到满足某些优化准则。因此，在本公开实施例提供的上述处理系统运行之前，需要进行一系列的训练。本公开实施例还对神经网络以及激活器的训练进行了优化，对此详细说明如下。
结合图3所示的子系统结构以及子系统中各模块的说明可知，本公开实施例的图像处理系统中，由于复合器不会引入任何的参数，因此涉及到的参数仅包括卷积神经网络的权重参数和激活器的偏移参数。
在训练过程中，首先需要准备一系列的训练图像，对训练图像的取得简要说明如下。
这些训练图像取自于一个数据库，该数据库中具有大量的样本图像，如具有分辨率为480*320的500个图像。该训练图像可以是随机选择部分样本图像，然后从选择样本图像中随机选择一定数量的80*80的图像块，如30000个。
这些随机选择的图像块就构成了本公开实施例的训练图像。
在得到训练图像之后，可以采用各种标准的分辨率压缩算法，对该训练图像进行分辨率缩小X倍的操作后，得到输入到图像处理系统的原始图像。
在得到原始图像之后，将原始图像输入到参数初始化的图像处理系统，并调整图像处理系统的各参数，以使得参数调整后的图像处理系统对原始图像进行X倍放大操作得到的目标图像。
而参数的调整可以采用各种已有的算法来进行，如标准随机梯度下降SGD算法等。
而参数是否合适可以以目标图像和训练图像的损失函数COST进行判决， 当二者之间的差异度较小时，表明图像处理系统的放大效果较好，则COST较小，反之较差。
相关技术中，损失函数COST可以采用多种准则来进行处理，如常用的均方差MSE。
然而，发明人在实现本公开实施例的过程中发现，相关技术中的判断准则都与人类视觉系统的相关度较差，举例说明如下。
相关技术中，对于图像而言，都包括两方面的信息：色彩和灰阶，而人类视觉系统对不同的参数的敏感度并不相同。而相关技术中的参数设置是否合理的准则把所有的信息同等对待，与人类视觉系统的相关度较差。
发明人在实现本公开实施例的过程中潜心研究，发现采用SSIM(Structural Similarity，结构相似性)准则能够更好地训练卷积神经网络和激活器。
对此进一步说明如下。
SSIM，是一种全参考的图像质量评价指标，它分别从亮度、对比度、结构三方面度量图像相似性，定义如下：
Figure PCTCN2017117116-appb-000006
其中u O、u R分别表示目标图像和原始图像的均值，σ O、σ R分别表示目标图像和原始图像的方差，σ OR表示目标图像和原始图像Y的协方差，C1和C2为非零常数。
SSIM取值范围[0，1]，值越大，表示图像失真越小，即当SSIM等于1时，表示目标图像和原始图像完全相同。
在实际应用中，可以利用滑动窗将图像分块，令分块总数为N，考虑到窗口形状对分块的影响，采用高斯加权计算每一窗口的均值、方差以及协方差，然后计算对应块的结构相似度SSIM，最后将平均值作为两图像的结构相似性度量，即平均结构相似性MSSIM：
Figure PCTCN2017117116-appb-000007
而损失函数COST则可以表示为1-SSIM，标识目标图像和原始图像的差异程度，即：
COST＝1-SSIM。
本公开具体实施例中，为进一步提高卷积神经网络和激活器的收敛程度，该COST函数也可以附加其他的参数，如权重参数的L1范数等。
本公开具体实施例中进一步增加如下的两种新的参数：DownReg(INPUT)以及权重偏移比函数。
其中，
DownReg(INPUT)＝MSE(Downscale(Output(INPUT))，INPUT)
或者，
DownReg(INPUT)＝1-SSIM(Downscale(Output(INPUT))，INPUT)
其中，INPUT为原始图像，Downscale(Output(INPUT)为训练图像，MSE为图像均方差计算函数，SSIM为结构相似性计算函数。
对于神经卷积网络而言，为了将特征进行分类，神经卷积网络(包括神经卷积模块和激活器)包括权重参数和偏移参数，理想情况下，偏移参数相对权重参数而言应该尽可能大。
为此，本公开具体实施例定义如下函数权重偏移比函数，如下：
Figure PCTCN2017117116-appb-000008
其中L1(W)为权重参数的L1范数，L1(B)为偏移参数的L1范数如下：
Figure PCTCN2017117116-appb-000009
Figure PCTCN2017117116-appb-000010
其中：1为神经网络的层序号，f为特征序号，滤波器大小为N*M。其中eps为非常小的常数，如le-6。
其中：
Cost＝1-SSIM(Output(INPUT)+REFRENCE)+λ 1DownReg(INPUT)+λ 2WBratioReg(W，b)其中λ1为DownReg(INPUT)的权重系数和λ2为WBratioReg(W，b)的权重系数，其具体取值取决于其重要程度。
基于同一发明构思，本公开实施例还提供了一种显示装置，包括如本公开实施例提供的上述图像处理系统。该显示装置可以为：手机、平板电脑、 电视机、显示器、笔记本电脑、数码相框、导航仪等任何具有显示功能的产品或部件。该显示装置的实施可以参见上述图像处理系统的实施例，重复之处不再赘述。由于在显示装置中采用了上述图像处理系统，因此具有与上述图像处理系统相同的有益效果。
基于同一发明构思，本公开实施例还提供了一种图像处理方法，该图像处理方法可以在上述图像处理系统中被执行，具体的，参照图12，本公开实施例提供的图像处理方法包括步骤1201-1203。
步骤1201，对输入信号进行卷积处理，得到具有第一分辨率的多个第一特征图像。
步骤1202，利用所述第一特征图像合成具有大于第一分辨率的第二分辨率的第二特征图像。
步骤1203，利用激活函数对所述第二特征图像进行选择。
具体的，步骤1201可以由上述分辨率转换子系统的卷积神经网络模块执行，步骤1202可以由上述分辨率转换子系统的复合器执行，步骤1203可以由上述分辨率转换子系统的激活模块执行，该卷积神经网络模块、复合器和激活模块的具体结构可以参照上述实施例，在此不再赘述。
本公开实施例提供的图像处理方法首先对输入信号进行卷积处理，得到具有第一分辨率的多个第一特征图像；然后利用所述第一特征图像合成具有大于第一分辨率的第二分辨率的第二特征图像；最后利用激活函数对所述第二特征图像进行选择。由于在分辨率转换子系统中设置了激活模块对复合器输出的第二特征图像进行自适应的选择，因此可以生成不同像素排布方式的第二特征图像，从而提高了图像处理的灵活度。此外，由于复合器直接利用卷积神经网络模块输出的第一特征图像进行放大处理，能够保证尽可能多的特征图像参与到放大处理过程中，从而提高放大效果。
图13是本公开实施例提供的电子设备的结构图。如图13所示，电子设备包括：至少一个处理器1301、存储器1302、至少一个网络接口1304和用户接口1303。电子设备中的各个组件通过总线系统1305耦合在一起。可理解，总线系统1305用于实现这些组件之间的连接通信。总线系统1305除包括数据总线之外，还包括电源总线、控制总线和状态信号总线。但是为了清楚说明起 见，在图13中将各种总线都标为总线系统1305。
其中，用户接口1303可以包括显示器、键盘或者点击设备(例如，鼠标，轨迹球(track ball)、触感板或者触摸屏等。
可以理解，本公开实施例中的存储器1302可以是易失性存储器或非易失性存储器，或可包括易失性和非易失性存储器两者。其中，非易失性存储器可以是只读存储器(Read-Only Memory，ROM)、可编程只读存储器(Programmable ROM，PROM)、可擦除可编程只读存储器(Erasable PROM，EPROM)、电可擦除可编程只读存储器(Electrically EP ROM，EEPROM)或闪存。易失性存储器可以是随机存取存储器(Random Access Memory，RAM)，其用作外部高速缓存。通过示例性但不是限制性说明，许多形式的RAM可用，例如静态随机存取存储器(Static RAM，SRAM)、动态随机存取存储器(Dynamic RAM，DRAM)、同步动态随机存取存储器(Synchronous D RAM，SDRAM)、双倍数据速率同步动态随机存取存储器(Double Data Rate SD RAM，DDRSDRAM)、增强型同步动态随机存取存储器(Enhanced SD RAM，ESDRAM)、同步连接动态随机存取存储器(Synch link D RAM，SLDRAM)和直接内存总线随机存取存储器(Direct Rambus RAM，DRRAM)。本文描述的存储器1302旨在包括但不限于这些和任意其它适合类型的存储器。
在一些实施方式中，存储器1302存储了如下的元素，可执行模块或者数据结构，或者他们的子集，或者他们的扩展集：操作系统13021和应用程序13022。
其中，操作系统13021包含各种系统程序，例如框架层、核心库层、驱动层等，用于实现各种基础业务以及处理基于硬件的任务。应用程序13022包含各种应用程序，例如媒体播放器(Media Player)、浏览器(Browser)等，用于实现各种应用业务。实现本公开实施例方法的程序可以包含在应用程序13022中。
在本公开实施例中，通过调用存储器1302存储的程序或指令，具体的，可以是应用程序13022中存储的程序或指令，处理器1301用于：对输入信号进行卷积处理，得到具有第一分辨率的多个第一特征图像；利用所述第一特征图像合成具有大于第一分辨率的第二分辨率的第二特征图像；利用激活函数 对所述第二特征图像进行选择。
本公开实施例还提供一种计算机可读存储介质，其上存储有计算机程序，所述计算机程序被处理器执行时实现上述任意一个方法实施例中的图像处理方法中的步骤。
本领域普通技术人员可以意识到，结合本文中所公开的实施例描述的各示例的单元及算法步骤，能够以电子硬件、或者计算机软件和电子硬件的结合来实现。这些功能究竟以硬件还是软件方式来执行，取决于技术方案的特定应用和设计约束条件。专业技术人员可以对每个特定的应用来使用不同方法来实现所描述的功能，但是这种实现不应认为超出本公开的范围。
所属领域的技术人员可以清楚地了解到，为描述的方便和简洁，上述描述的系统、装置和单元的具体工作过程，可以参考前述方法实施例中的对应过程，在此不再赘述。
在本申请所提供的实施例中，应该理解到，所揭露的装置和方法，可以通过其它的方式实现。例如，以上所描述的装置实施例仅仅是示意性的，例如，所述单元的划分，仅仅为一种逻辑功能划分，实际实现时可以有另外的划分方式，例如多个单元或组件可以结合或者可以集成到另一个系统，或一些特征可以忽略，或不执行。另一点，所显示或讨论的相互之间的耦合或直接耦合或通信连接可以是通过一些接口，装置或单元的间接耦合或通信连接，可以是电性，机械或其它的形式。
所述作为分离部件说明的单元可以是或者也可以不是物理上分开的，作为单元显示的部件可以是或者也可以不是物理单元，即可以位于一个地方，或者也可以分布到多个网络单元上。可以根据实际的需要选择其中的部分或者全部单元来实现本公开实施例方案的目的。
另外，在本公开各个实施例中的各功能单元可以集成在一个处理单元中，也可以是各个单元单独物理存在，也可以两个或两个以上单元集成在一个单元中。
所述功能如果以软件功能单元的形式实现并作为独立的产品销售或使用时，可以存储在一个计算机可读取存储介质中。基于这样的理解，本公开的技术方案本质上或者说对现有技术做出贡献的部分或者该技术方案的部分可以 以软件产品的形式体现出来，该计算机软件产品存储在一个存储介质中，包括若干指令用以使得一台计算机设备(可以是个人计算机，服务器，或者网络设备等)执行本公开各个实施例所述方法的全部或部分步骤。而前述的存储介质包括：U盘、移动硬盘、ROM、RAM、磁碟或者光盘等各种可以存储程序代码的介质。
以上所述，仅为本公开的具体实施方式，但本公开的保护范围并不局限于此，任何熟悉本技术领域的技术人员在本公开揭露的技术范围内，可轻易想到变化或替换，都应涵盖在本公开的保护范围之内。因此，本公开的保护范围应以权利要求的保护范围为准。

Claims (16)

  1. 一种图像处理系统，包括至少一个分辨率转换子系统；
    其中，所述分辨率转换子系统包括级联的卷积神经网络模块、复合器和激活模块；
    所述卷积神经网络模块用于对输入信号进行卷积处理，得到具有第一分辨率的多个第一特征图像；
    所述复合器用于利用所述第一特征图像合成具有大于第一分辨率的第二分辨率的第二特征图像；
    所述激活模块与所述复合器连接，用于利用激活函数对所述第二特征图像进行选择。
  2. 根据权利要求1所述的图像处理系统，还包括：转换模块；
    其中，所述转换模块用于将接收的RGB格式的输入图像转换为YUV格式，输出Y通道信号、U通道信号和V通道信号；
    所述至少一个分辨率转换子系统对所述Y通道信号、U通道信号和V通道信号中的至少一个进行分辨率放大处理。
  3. 根据权利要求2所述的图像处理系统，其中，所述图像处理系统对所述输入图像的分辨率的放大倍数为N倍；
    所述Y通道信号通过至少一个分辨率转换子系统进行分辨率放大的放大倍数为N倍；
    所述U通道信号通过至少一个分辨率转换子系统进行分辨率放大的放大倍数为N1倍，N1小于N；
    所述V通道信号通过至少一个分辨率转换子系统进行分辨率放大的放大倍数为N2倍，N2小于N。
  4. 根据权利要求1所述的图像处理系统，还包括：转换模块；
    其中，所述至少一个分辨率转换子系统仅对所述Y通道信号进行分辨率放大处理。
  5. 根据权利要求1所述的图像处理系统，其中，所述激活模块具体用于在条件满足时，输出所述第二特征图像或施加偏移量后的第二特征图像，否 则丢弃所述第二特征图像。
  6. 根据权利要求5所述的图像处理系统，其中，所述条件包括所述第二特征图像大于所述偏移量的负数。
  7. 如权利要求1-6中任意一项所述的图像处理系统，其中，所述复合器为分辨率放大M倍的复合器；所述复合器具体用于：将每M个第一特征图像的像素值交叉后合成M个第二特征图像并输出。
  8. 如权利要求7所述的图像处理系统，其中，所述复合器为自适应插值滤波器。
  9. 如权利要求8所述的图像处理系统，其中，所述复合器采用如下公式合成第二特征图像：
    Figure PCTCN2017117116-appb-100001
    其中，i的取值范围为(0...H-1)，j的取值范围为(0...W-1)，p的取值范围为(0...MyH-1)，q的取值范围为(0...MxW-1)，H表示第二特征图像的高度，W表示第二特征图像的宽度，c表示第一特征图像的个数，Mx表示第一特征图像相对于第二特征图像在行方向的放大因子，My表示第一特征图像相对于第二特征图像在列方向的放大因子，
    Figure PCTCN2017117116-appb-100002
    表示小于(n-1)/My的最大整数，
    Figure PCTCN2017117116-appb-100003
    表示第一特征图像，
    Figure PCTCN2017117116-appb-100004
    和
    Figure PCTCN2017117116-appb-100005
    均表示的第二特征图像，n表示第二特征图像的编号。
  10. 如权利要求1-6中任意一项所述的图像处理系统，其中，所述复合器为分辨率放大M倍的复合器，所述复合器包括：
    级联的多个在行列方向同时进行在行和列方向进行放大处理的子复合器，所述子复合器的放大倍数的乘积等于所述M；
    或者
    级联的用于在行方向进行放大处理的水平子复合器和用于在列方向进行放大处理的垂直子复合器，所述水平子复合器在行方向的放大倍数与所述垂直子复合器在列方向的放大倍数的乘积等于所述M。
  11. 如权利要求1-6中任意一项所述的图像处理系统，其中，所述卷积神经网络模块和所述激活模块的训练基于结构相似性SSIM准则的成本函数进行判断。
  12. 如权利要求11所述的图像处理系统，其中，所述成本函数为：
    Cost(INPUT，REFRENCE)
    ＝1-SSIM(Output(INPUT)+REFRENCE)+λ 1DownReg(INPUT)+λ 2WBratioReg(W，b)其中，INPUT为原始图像，SSIM为结构相似性计算函数，DownReg(INPUT)为描述原始图像和训练图像的相似度的函数，λ1为DownReg(INPUT)的权重系数和λ2为WBratioReg(W，b)的权重系数，WBratioReg(W，b)为权重偏移比函数；
    Figure PCTCN2017117116-appb-100006
    其中L1(W)为权重参数的L1范数，L1(b)为偏移参数的L1范数，eps为常数。
  13. 一种显示装置，包括如权利要求1-12中任一项所述图像处理系统。
  14. 一种图像处理方法，包括：
    对输入信号进行卷积处理，得到具有第一分辨率的多个第一特征图像；
    利用所述第一特征图像合成具有大于第一分辨率的第二分辨率的第二特征图像；及
    利用激活函数对所述第二特征图像进行选择。
  15. 一种电子设备，包括：
    一个或多个处理器；
    存储器；以及
    一个或多个程序，其中，所述一个或多个程序被存储在所述存储器中，并且被配置成由所述一个或多个处理器执行，其中，所述程序被执行时实现权利要求14所述的图像处理方法中的步骤。
  16. 一种计算机可读存储介质，其上存储有计算机程序，其中，所述计算机程序被处理器执行时实现权利要求14所述的图像处理方法中的步骤。
PCT/CN2017/117116 2017-05-08 2017-12-19 一种图像处理系统、方法及显示装置 WO2018205627A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/073,712 US11216910B2 (en) 2017-05-08 2017-12-19 Image processing system, image processing method and display device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710324036.9A CN107169927B (zh) 2017-05-08 2017-05-08 一种图像处理系统、方法及显示装置
CN201710324036.9 2017-05-08

Publications (1)

Publication Number Publication Date
WO2018205627A1 true WO2018205627A1 (zh) 2018-11-15

Family

ID=59812839

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2017/117116 WO2018205627A1 (zh) 2017-05-08 2017-12-19 一种图像处理系统、方法及显示装置

Country Status (3)

Country Link
US (1) US11216910B2 (zh)
CN (1) CN107169927B (zh)
WO (1) WO2018205627A1 (zh)

Families Citing this family (9)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107169927B (zh) * 2017-05-08 2020-03-24 京东方科技集团股份有限公司 一种图像处理系统、方法及显示装置
CN107122826B (zh) 2017-05-08 2019-04-23 京东方科技集团股份有限公司 用于卷积神经网络的处理方法和系统、和存储介质
US10643306B2 (en) * 2017-10-11 2020-05-05 Qualcomm Incoporated Image signal processor for processing images
US11263782B2 (en) 2017-10-11 2022-03-01 Qualcomm Incorporated Image signal processor for processing images
CN108876737B (zh) * 2018-06-06 2021-08-03 武汉大学 一种联合残差学习和结构相似度的图像去噪方法
CN110298810A (zh) * 2019-07-24 2019-10-01 深圳市华星光电技术有限公司 图像处理方法及图像处理系统
CN110827211B (zh) * 2019-10-10 2023-11-21 杭州智团信息技术有限公司 一种图像细节增强方法
US20230237620A1 (en) * 2022-01-27 2023-07-27 Sonic Star Global Limited Image processing system and method for processing image
CN114723608B (zh) * 2022-04-14 2023-04-07 西安电子科技大学 基于流体粒子网络的图像超分辨率重建方法

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105120130A (zh) * 2015-09-17 2015-12-02 京东方科技集团股份有限公司 一种图像升频系统、其训练方法及图像升频方法
CN204948182U (zh) * 2015-09-17 2016-01-06 京东方科技集团股份有限公司 一种图像升频系统及显示装置
CN106600553A (zh) * 2016-12-15 2017-04-26 华中科技大学 一种基于卷积神经网络的dem超分辨率方法
CN107169927A (zh) * 2017-05-08 2017-09-15 京东方科技集团股份有限公司 一种图像处理系统、方法及显示装置

Family Cites Families (8)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
TWI490818B (zh) * 2013-01-02 2015-07-01 Chunghwa Picture Tubes Ltd 權重式影像強化方法以及系統
WO2016019324A2 (en) * 2014-08-01 2016-02-04 The Regents Of The University Of California Device and method for iterative phase recovery based on pixel super-resolved on-chip holography
CN106204467B (zh) * 2016-06-27 2021-07-09 深圳市未来媒体技术研究院 一种基于级联残差神经网络的图像去噪方法
US10803378B2 (en) * 2017-03-15 2020-10-13 Samsung Electronics Co., Ltd System and method for designing efficient super resolution deep convolutional neural networks by cascade network training, cascade network trimming, and dilated convolutions
US11354577B2 (en) * 2017-03-15 2022-06-07 Samsung Electronics Co., Ltd System and method for designing efficient super resolution deep convolutional neural networks by cascade network training, cascade network trimming, and dilated convolutions
US10776904B2 (en) * 2017-05-03 2020-09-15 Samsung Electronics Co., Ltd. Method and apparatus for processing image
US10733714B2 (en) * 2017-11-09 2020-08-04 Samsung Electronics Co., Ltd Method and apparatus for video super resolution using convolutional neural network with two-stage motion compensation
US10547823B2 (en) * 2018-09-25 2020-01-28 Intel Corporation View interpolation of multi-camera array images with flow estimation and image super resolution using deep learning

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN105120130A (zh) * 2015-09-17 2015-12-02 京东方科技集团股份有限公司 一种图像升频系统、其训练方法及图像升频方法
CN204948182U (zh) * 2015-09-17 2016-01-06 京东方科技集团股份有限公司 一种图像升频系统及显示装置
CN106600553A (zh) * 2016-12-15 2017-04-26 华中科技大学 一种基于卷积神经网络的dem超分辨率方法
CN107169927A (zh) * 2017-05-08 2017-09-15 京东方科技集团股份有限公司 一种图像处理系统、方法及显示装置

Also Published As

Publication number Publication date
US20210209730A1 (en) 2021-07-08
CN107169927B (zh) 2020-03-24
CN107169927A (zh) 2017-09-15
US11216910B2 (en) 2022-01-04

Similar Documents

Publication Publication Date Title
WO2018205627A1 (zh) 一种图像处理系统、方法及显示装置
US10311547B2 (en) Image upscaling system, training method thereof, and image upscaling method
CN107924554B (zh) 图像处理流水线中对图像数据的多速率处理
US10515435B2 (en) Apparatus for upscaling an image, method for training the same, and method for upscaling an image
WO2017045374A1 (zh) 图像升频系统、其训练方法及图像升频方法
JP4621733B2 (ja) イメージを見てエンハンスする方法及びシステム
US20130315499A1 (en) Content-adaptive image resizing method and related apparatus thereof
JP2007067917A (ja) 画像データ処理装置
US8861846B2 (en) Image processing apparatus, image processing method, and program for performing superimposition on raw image or full color image
WO2021163844A1 (en) Super resolution using convolutional neural network
WO2011092696A1 (en) Method and system for generating an output image of increased pixel resolution from an input image
US8422767B2 (en) Method and apparatus for transforming signal data
US9462189B2 (en) Piecewise perspective transform engine
CN111402139A (zh) 图像处理方法、装置、电子设备和计算机可读存储介质
US20120155550A1 (en) Auto-regressive edge-directed interpolation with backward projection constraint
JP2009159171A (ja) 画像高解像度化方法および装置
US11922598B2 (en) Image processing apparatus, image processing method, and storage medium
JP2014119997A (ja) 画像処理装置およびその制御方法
JP6102938B2 (ja) 画像処理装置、及び、画像処理方法
WO2011108144A1 (ja) 画像拡大装置、画像拡大プログラム、画像拡大プログラムを記憶した記憶媒体、及び表示装置
CN102842111B (zh) 放大图像的补偿方法及装置
US9928577B2 (en) Image correction apparatus and image correction method
JPWO2014077024A1 (ja) 画像処理装置、画像処理方法および画像処理プログラム
JP7183015B2 (ja) 画像処理装置、画像処理方法、及びプログラム
CN111724292B (zh) 图像处理方法、装置、设备以及计算机可读介质

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 17909114

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

32PN Ep: public notification in the ep bulletin as address of the adressee cannot be established

Free format text: NOTING OF LOSS OF RIGHTS PURSUANT TO RULE 112(1) EPC (EPO FORM 1205A DATED 17.03.2020)

122 Ep: pct application non-entry in european phase

Ref document number: 17909114

Country of ref document: EP

Kind code of ref document: A1