WO2019037487A1 - 基于混合框架的图像位深度扩展方法及装置 - Google Patents

基于混合框架的图像位深度扩展方法及装置 Download PDF

Info

Publication number
WO2019037487A1
WO2019037487A1 PCT/CN2018/087424 CN2018087424W WO2019037487A1 WO 2019037487 A1 WO2019037487 A1 WO 2019037487A1 CN 2018087424 W CN2018087424 W CN 2018087424W WO 2019037487 A1 WO2019037487 A1 WO 2019037487A1
Authority
WO
WIPO (PCT)
Prior art keywords
image
bit depth
flat
bit
local
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/CN2018/087424
Other languages
English (en)
French (fr)
Inventor
赵洋
王荣刚
高文
王振宇
王文敏
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Peking University Shenzhen Graduate School
Original Assignee
Peking University Shenzhen Graduate School
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Peking University Shenzhen Graduate School filed Critical Peking University Shenzhen Graduate School
Priority to US16/640,511 priority Critical patent/US11087439B2/en
Publication of WO2019037487A1 publication Critical patent/WO2019037487A1/zh
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/50—Depth or shape recovery
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/46—Colour picture communication systems
    • H04N1/56—Processing of colour picture signals
    • H04N1/58—Edge or detail enhancement; Noise or error suppression, e.g. colour misregistration correction
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/045—Combinations of networks
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/04—Architecture, e.g. interconnection topology
    • G06N3/0464—Convolutional networks [CNN, ConvNet]
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06N—COMPUTING ARRANGEMENTS BASED ON SPECIFIC COMPUTATIONAL MODELS
    • G06N3/00—Computing arrangements based on biological models
    • G06N3/02—Neural networks
    • G06N3/08—Learning methods
    • G06N3/09—Supervised learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T5/00—Image enhancement or restoration
    • G06T5/90—Dynamic range modification of images or parts thereof
    • G06T5/94—Dynamic range modification of images or parts thereof based on local image properties, e.g. for local contrast enhancement
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T7/00—Image analysis
    • G06T7/10—Segmentation; Edge detection
    • G06T7/11—Region-based segmentation
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N1/00—Scanning, transmission or reproduction of documents or the like, e.g. facsimile transmission; Details thereof
    • H04N1/46—Colour picture communication systems
    • H04N1/56—Processing of colour picture signals
    • H04N1/60—Colour correction or control
    • H04N1/6072—Colour correction or control adapting to different types of images, e.g. characters, graphs, black and white image portions
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/10—Image acquisition modality
    • G06T2207/10028—Range image; Depth image; 3D point clouds
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20004—Adaptive image processing
    • G06T2207/20012—Locally adaptive
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20081—Training; Learning
    • G—PHYSICS
    • G06—COMPUTING OR CALCULATING; COUNTING
    • G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
    • G06T2207/00—Indexing scheme for image analysis or image enhancement
    • G06T2207/20—Special algorithmic details
    • G06T2207/20084—Artificial neural networks [ANN]

Definitions

  • the invention belongs to the technical field of image processing, and relates to image bit depth expansion and enhancement processing technology, in particular to a method and device for image bit depth expansion based on a hybrid framework, which combines adaptive pixel value adjustment and a convolutional neural network. Reconstruction technology.
  • Bit-Depth Expansion refers to the restoration of a high-depth image by a low-bit depth image.
  • the bit depth of the image (Bit-Depth) is the number of binary bits of the value of each pixel in the image.
  • the 8-bit depth (8-bits) image has a value ranging from 0 to 255.
  • the higher the bit depth of the image the more subtle the brightness changes.
  • the depth of the human eye can be perceived as 12 to 14 bits. Therefore, we often feel that the image seen through the 8-bit display is different from the image seen by the eye.
  • the insufficient depth of the image and image display is an important reason. .
  • image bit depth expansion has potential application prospects in image display, image editing, high definition television and other fields.
  • the most classical image bit depth expansion algorithms include: Zero Padding method, which increases the bit depth by shifting the image pixel values by bits, and the new bits generated by the displacement all add 0; Ideal Gain In the method, the pixel value of the low bit depth is amplified by multiplying the factor of the bit depth enhancement to increase the bit depth; the Bit Replication method is similar to the zero extension method, and the bit shift is performed first, but the new bit generated by the displacement The bit is obtained by copying the previous bit. But these classical methods produce some unnatural effects, such as the Banding effect in flat areas, the Ringing effect, and flat area noise.
  • bit depth expansion In recent years, some methods of focusing on the bit depth expansion focusing on the elimination of the band effect have somewhat reduced the unnatural effects of the flat region.
  • the current method of bit depth expansion is mostly a traditional non-learning method, which uses a fixed strategy to fill the missing bit information, and the real value of the missing bit is less reconstructed, and at the same time the de-banding effect is often Causes loss of high frequency detail in non-flat areas.
  • the existing method is difficult to realize a high-depth image with higher fidelity without high-natural effects and high-frequency texture details recovered by low-bit depth images.
  • the present invention provides a method and apparatus for image bit depth expansion based on a hybrid framework, which can better remove image flat regions by combining a traditional de-banding algorithm and a deep network-based learning algorithm.
  • the unnatural effect while more realistically recovering the numerical information of the missing bits.
  • a method for image bit depth expansion based on hybrid frame including extraction process of image flat region, flat region bit depth expansion process based on local adaptive pixel value adjustment, and non-flat region bit depth expansion process based on convolutional neural network; Including the following steps:
  • the first step the extraction of the flat area of the image, the image is divided into flat areas and texture (non-flat) areas;
  • the flat region of the image is extracted according to the degree of local numerical change, and the image is divided into a flat region and a texture (non-flat) region, so that the flat region and the texture (non-flat) region are separately processed; the local average pixel value difference is specifically used.
  • Information to measure the extent of local numerical changes calculated as Equation 1:
  • the second step the flat region bit depth expansion process based on the local adaptive pixel value adjustment, eliminating the unnatural effect of the flat region of the low bit depth image;
  • the bit-changing pixel with a pixel value difference of 1 is located, and then the value of the pixel is adaptively adjusted according to the bit depth information of the pixel around the pixel, so that the bit changes. More natural, eliminating unnatural effects such as band effects;
  • the present invention uses a method based on local adaptive pixel value adjustment to eliminate unnatural effects such as flat region band effects and noise in low bit depth images. It is assumed that the input is a low-order depth image Y having a bit depth of l-bits, and the bit depth of the high-depth image X obtained after the bit depth is expanded is h-bits.
  • 8 pixels adjacent to a pixel point are used as their neighboring pixels.
  • the lifting factor ⁇ + and the suppression factor ⁇ - can be calculated by Equation 2 and Equation 3:
  • ZP(g c ) is to increase the bit depth by using the traditional zero expansion method for the pixel value g c
  • ⁇ is the adjustment value parameter
  • the default is 0.125
  • the parameter can be adjusted to increase or decrease the degree of denoising.
  • the third step the depth expansion process of the non-flat area based on the convolutional neural network.
  • the process first trains a convolutional neural network with amplified residual learning using a large number of low-bit images and corresponding high-bit images, and then reconstructs more accurate missing bit value information by using the network.
  • the present invention uses a convolutional neural network that amplifies residual learning to learn the missing bit depth reconstruction of an image. Since in the bit depth extension, the missing and added bits are the last bits, the value of the missing amount is small. In order to effectively train the network to reconstruct the missing bit position information, in the process of deep network training, the present invention uses the low bit depth image Y as an input, and the residual (XY) of the corresponding high bit depth image X and Y is amplified by the factor ⁇ . The value ⁇ (XY) is used as the calibration true value.
  • Equation 5 For input Y, the reconstruction result of the convolutional neural network F is defined as F(Y), and the final reconstructed high-depth image is expressed as Equation 5:
  • X * is the final reconstructed high bit depth image.
  • the adaptive pixel value adjustment method can better suppress the unnatural effects of flat regions, but introduces non-flat region ambiguity and further loses high-frequency detail information.
  • the method based on convolutional neural network can recover missing bit values more accurately.
  • the subjective quality of the flat region is still affected by the band effect and noise.
  • the result based on the adaptive pixel adjustment method is used in the flat region, and the result of the reconstruction using the convolutional neural network in the non-flat non-flat region, and finally the high-depth image is obtained, thereby obtaining a high-depth image.
  • the technical effect of combining the respective advantages of the above two methods is achieved.
  • the invention also provides an image bit depth expansion device based on a hybrid frame. As shown in FIG. 1, it comprises three modules, an image flat region extraction module, and a flat region bit depth expansion module based on local adaptive pixel value adjustment. A non-flat area bit depth extension module based on a convolutional neural network.
  • Image flat area extraction module For the bit depth expansion problem, the main goal in the flat area is to eliminate unnatural effects such as band effects and improve the visual quality. The main goal in the non-flat area is to restore high-fidelity high-frequency details as much as possible. . Thus, first we use the local average pixel value difference information to extract a flat region of the image, divide the image into flat regions and texture (non-flat) regions, and then process them separately.
  • Flat area bit depth expansion module based on local adaptive pixel value adjustment: in order to eliminate the unnatural effect of low bit depth image in a flat area, firstly position a bit change pixel with a difference of a neighbor pixel value of 1, and then according to the pixel around the pixel
  • the bit depth information adaptively adjusts the value of the pixel to make the bit change more natural, eliminating unnatural effects such as band effects.
  • Non-flat area bit depth extension module based on convolutional neural network the image bit depth will lose the numerical information of the discarded bits in the process of decreasing, so restoring the value of these bits is a morbid problem, and the traditional bit depth expansion
  • the method is mostly based on simple strategies and lacks the use of efficient learning-based reconstruction algorithms. We first train a convolutional neural network with amplified residual learning using a large number of low-bit images and corresponding high-bit images, and then use this network to reconstruct more accurate missing bit value information.
  • the current image bit depth expansion method is mostly based on the traditional method of focusing on how to eliminate the band effect of the flat region, and it is difficult to solve the technical problem of how to recover the missing bit with high fidelity.
  • the invention provides a method and device for image bit depth expansion based on a hybrid framework. By combining a traditional de-banding effect algorithm and a deep network-based learning algorithm, the unnatural effect of the flat region of the image can be better removed, and the image is more realistic. The value information of the missing bits is restored.
  • the present invention uses a learning-based approach to solve the ill-conditioned problem of realistically recovering missing bits by training an effective deep network. Meanwhile, for a flat region, the present invention uses a simple and robust local adaptive pixel value adjustment method, which can effectively suppress unnatural effects such as band effect, ringing effect, and flat noise of a flat region, thereby improving a flat region. Subjective visual quality.
  • Figure 1 is a block flow diagram of the method of the present invention.
  • FIG. 2 is a block flow diagram of a convolutional neural network method employed by the present invention.
  • 3 is a comparison diagram of effects of reconstructing image subjective quality when reconstructing images from 6 bits to 8 bits using the present invention and several existing methods;
  • the image in the upper left corner is indicated by the position of the flat region selected in the image, and the results of the bit depth expansion using the different methods in the flat image region are respectively (a) a zero expansion method, and (b) a bit copy method, (c) ) is an ideal gain method, (d) is the method of the invention, and (e) is a real image.
  • the invention provides an image bit depth expansion method based on a hybrid framework.
  • the unnatural effect of the flat region of the image can be better removed, and the restoration is more realistic.
  • the numeric information of the missing bits can be more realistic.
  • FIG. 1 is a flow chart of a method according to the present invention, including an extraction process of an image flat region, a flat region bit depth expansion process based on local adaptive pixel value adjustment, and a non-flat region bit depth expansion process based on a convolutional neural network, specifically including The following steps:
  • the first step the extraction of the flat area of the image.
  • the present invention uses local pixel value differences to measure the degree of local numerical variation, which is calculated as Equation 1:
  • the second step flat region bit depth expansion based on local adaptive pixel value adjustment.
  • the present invention uses a method based on local adaptive pixel value adjustment to eliminate unnatural effects such as flat region band effects and noise in low bit depth images. It is assumed that the input is a low-order depth image Y having a bit depth of l-bits, and the bit depth of the high-depth image X obtained after the bit depth is expanded is h-bits.
  • Equation 4 the pixel value of the contour height increase of the contour line by Equation 4, which is:
  • ZP(g c ) is to increase the bit depth by using the traditional zero expansion method for the pixel value g c
  • ⁇ is the adjustment value parameter
  • the default is 0.125
  • the parameter can be adjusted to increase or decrease the degree of denoising.
  • the third step the depth expansion of the non-flat region based on the convolutional neural network.
  • the present invention uses a convolutional neural network that amplifies residual learning to learn the missing bit depth reconstruction of an image. Since in the bit depth extension, the missing and added bits are the last bits, the value of the missing amount is small. In order to effectively train the network to reconstruct the missing bit position information, in the process of deep network training, the present invention uses the low bit depth image Y as an input, and the residual (XY) of the corresponding high bit depth image X and Y is amplified by the factor ⁇ . The value ⁇ (XY) is used as the calibration true value.
  • Equation 5 For input Y, the reconstruction result of the convolutional neural network F is defined as F(Y), and the final reconstructed high-depth image is expressed as Equation 5:
  • X * is a high bit depth image acquired using a convolutional network reconstruction.
  • the structure of the convolutional neural network used in the present invention is as shown in FIG. 2, and the structure is as follows:
  • a convolution layer composed of a 3 ⁇ 3 convolution kernel, and the output is 64 feature images
  • the convolution layer composed of 3 ⁇ 3 convolution kernels, the output is 64 feature images;
  • the convolution layer composed of 3 ⁇ 3 convolution kernels, the output is 64 feature images;
  • a convolution layer composed of a k.1 ⁇ 1 convolution kernel, and the output is 64 feature images
  • An output layer composed of a 3 ⁇ 3 convolution kernel, and the output is a reconstructed residual image F(Y).
  • the adaptive pixel value adjustment method can better suppress the unnatural effects of flat regions, but introduces non-flat region ambiguity and further loses high-frequency detail information.
  • the method based on convolutional neural network can recover missing bit values more accurately. However, the subjective quality of the flat region is still affected by the band effect and noise.
  • Table 1 shows the comparison of the peak signal-to-noise ratio (PSNR) of the proposed method with three conventional methods (zero extension, bit replica, ideal gain) on several data sets from 6bits to 8bits image reconstruction, Set5, Set14, Kodak and B100 are four image test sets; by numerical comparison, it can be seen that by using the convolutional neural network-based learning method, the numerical information of the missing bits can be recovered more accurately, thereby restoring the more realistic texture high-frequency details. .
  • PSNR peak signal-to-noise ratio
  • Figure 3 shows the comparison of the subjective quality of the reconstructed image of the present invention and several existing conventional methods from 6bits to 8bits image reconstruction. It can be seen that the band effect and noise of the flat region of the reconstructed image of the method of the present invention are very good. Good inhibition, which improves subjective visual quality.

Landscapes

  • Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Theoretical Computer Science (AREA)
  • General Physics & Mathematics (AREA)
  • General Health & Medical Sciences (AREA)
  • General Engineering & Computer Science (AREA)
  • Biophysics (AREA)
  • Computational Linguistics (AREA)
  • Data Mining & Analysis (AREA)
  • Evolutionary Computation (AREA)
  • Artificial Intelligence (AREA)
  • Molecular Biology (AREA)
  • Computing Systems (AREA)
  • Biomedical Technology (AREA)
  • Life Sciences & Earth Sciences (AREA)
  • Mathematical Physics (AREA)
  • Software Systems (AREA)
  • Health & Medical Sciences (AREA)
  • Multimedia (AREA)
  • Signal Processing (AREA)
  • Computer Vision & Pattern Recognition (AREA)
  • Image Processing (AREA)

Abstract

一种基于混合框架的图像位深度扩展方法及装置，通过融合传统去带效应算法和基于深度网络的学习算法，可较好的移除图像平坦区域的不自然效应，同时更逼真的恢复所缺失的比特位的数值信息；包括图像平坦区域的提取、基于局部自适应像素值调整的平坦区域位深度扩展和基于卷积神经网络的非平坦区域位深度扩展。本发明采用基于学习的方法，通过训练有效的深度网络来解决逼真的恢复缺失比特位问题；同时，针对平坦区域使用简单而鲁棒的局部自适应像素值调整的方法，有效抑制平坦区域的带效应、振铃效应、以及平坦噪声等不自然效应，提升平坦区域的主观视觉质量。

Description

基于混合框架的图像位深度扩展方法及装置 技术领域
本发明属于图像处理技术领域，涉及图像位深度扩展与增强处理技术，尤其涉及一种基于混合框架的图像位深度扩展方法及装置，该混合框架融合了自适应像素值调整与基于卷积神经网络的重建技术。
背景技术
图像位深度扩展(Bit-Depth Expansion)指的是通过低位深度的额图像恢复高位深度的图像。图像的位深度(Bit-Depth)即图像中每个像素的值的二进制位的个数，例如8位深(8-bits)的图像中像素的取值范围为0～255。图像的位深度越高，所能体现的亮度变化即更为细腻。人眼所能感知的位深度在12～14bits，因而我们常常会感到通过8bits显示器看到的图像和眼睛所见到的画面有所差异，图像及图像显示的位深度不足是其中的一个重要原因。随着图像显示技术的不断发展，图像位深度扩展在图像显示、图像编辑、高清电视等领域有着潜在的广泛应用前景。
现有最经典的图像位深度扩展算法包括：零扩展(Zero Padding)方法，通过图像像素值按比特位进行位移来提升位深度，位移产生的新比特位全部补0；理想增益(Ideal Gain)方法，通过乘以位深度提升的因子来对低位深的像素值进行放大，以提升位深度；位复制(Bit Replication)方法，与零扩展方法相似先进行比特位位移，但位移产生的新比特位通过复制之前的比特位来得到。但是这些经典方法会产生一些不自然效应，例如在平坦区域的带(Banding)效应、振铃(Ringing)效应、以及平坦区域噪声，等。
近年来，一些聚焦于消除带效应的位深度扩展方法的提出，一定程度上降低了平坦区域的不自然效应。但是，目前的位深度扩展方法多为传统非学习方法，使用固定策略来填补缺失的比特位信息，而对缺失的比特位的真实数值的重建能力较弱，同时在去带效应的同时又常常导致非平坦区域高频细节的损失。现有方法难以实现通过低位深度图像恢复出无不自然效应、高频纹理细节有更高保真度的高位深度图像。
发明内容
为了克服上述现有技术的不足，本发明提供一种基于混合框架的图像位深度扩展方法及装置，通过融合传统去带效应算法和基于深度网络的学习算法，可以较好的移除图像平坦区 域的不自然效应，同时更逼真的恢复所缺失的比特位的数值信息。
本发明提供的技术方案是：
一种基于混合框架的图像位深度扩展方法，包括图像平坦区域的提取过程、基于局部自适应像素值调整的平坦区域位深度扩展过程、基于卷积神经网络的非平坦区域位深度扩展过程；具体包括如下步骤：
第一步：图像平坦区域的提取，将图像划分为平坦区域和纹理(非平坦)区域；
根据局部数值变化的程度提取图像的平坦区域，将图像划分为平坦区域和纹理(非平坦)区域，以便于后续对平坦区域和纹理(非平坦)区域进行分别处理；具体使用局部平均像素值差异信息来衡量局部数值变化的程度，计算方式如式1：
Figure PCTCN2018087424-appb-000001
其中，P为局部总的像素个数，g i(i＝1,2,…,P)为局部的每个像素，
Figure PCTCN2018087424-appb-000002
为局部平均像素值。完成计算每个局部像素值差异D之后，可以得到整副图像的平均局部像素值差异
Figure PCTCN2018087424-appb-000003
因为平坦区域的局部像素值变化较为缓慢，即差异D数值较小，所以通过引入系数λ，如果局部区域满足
Figure PCTCN2018087424-appb-000004
则该局部区域为平坦区域，反之则为非平坦的区域。该系数λ的取值范围为0.1～0.2。
第二步：基于局部自适应像素值调整的平坦区域位深度扩展过程，消除低位深度图像平坦区域的不自然效应；
为消除低位深度图像在平坦区域的不自然效应，首先定位与某邻居像素值差异为1的位变化像素，再依据该像素周围像素的位深度信息自适应的调整该像素的值，使得位变化更加自然，消除带效应等不自然效应；
在图像的平坦区域，本发明使用基于局部自适应像素值调整的方法来消除低位深度图像中的平坦区域带效应和噪声等不自然效应。假定输入为位深度为l-bits的低位深度图像Y，位深度扩展后获取的高位深度图像X的位深度为h-bits。
因为平坦区域的带效应存在于比特位数值差异为1的像素处，因而将满足|g i-g c|＝1的像素点g c称为等高线点(Contour Pixel)，而g i为该点的邻居像素。本发明中使用与像素点相邻的8个像素作为其邻居像素。
对于每个等高线点，可以通过式2、式3计算其提升因子σ +和抑制因子σ -：
Figure PCTCN2018087424-appb-000005
Figure PCTCN2018087424-appb-000006
然后，可以通过式4得到该等高线点位深度提升后的像素值：
Figure PCTCN2018087424-appb-000007
其中，ZP(g c)为对像素值g c使用传统零扩展方法提高位深度，α为调整值参数，默认为0.125，可以通过调节该参数以提高或降低去噪程度。
第三步：基于卷积神经网络的非平坦区域位深度扩展过程。
该过程首先使用大量的低比特位图像和对应的高比特位图像训练一个放大残差学习的卷积神经网络，再通过使用该网络重建更加准确的缺失比特位数值信息。
本发明使用放大残差学习的卷积神经网络来学习图像的缺失位深度重建。因为在位深度扩展中，所缺失和增加的比特位均为最末尾的比特位，因而缺失量的数值较小。为了有效训练网络以重建缺失的比特位信息，在深度网络训练的过程中，本发明使用低位深度图像Y作为输入，将对应的高位深度图像X与Y的残差(X-Y)经过因子β放大后的值β(X-Y)作为标定真实值。
对于输入Y，该卷积神经网络F的重建结果定义为F(Y)，则最终重建的高位深度图像表示为式5：
Figure PCTCN2018087424-appb-000008
其中，
Figure PCTCN2018087424-appb-000009
X *为最终重建的高位深度图像。
自适应像素值调整的方法可以更好的抑制平坦区域不自然效应，但会引入非平坦区域模糊，进一步损失高频细节信息；基于卷积神经网络的方法可以更准确的恢复缺失的比特位数值，但平坦区域的主观质量仍受到带效应和噪声的影响。通过对平坦区域和非平坦区域采用不同方法，在平坦区域使用基于自适应像素调整方法的结果，而在非平坦的非平坦区域使用卷积神经网络重建的结果，最终得到高位深度图像，由此达到综合上述两种方法各自优势的技术效果。
本发明还提供一种基于混合框架的图像位深度扩展装置，如图1所示，它包括三个模块，一个图像平坦区域提取模块，一个基于局部自适应像素值调整的平坦区域位深度扩展模块，一个基于卷积神经网络的非平坦区域位深度扩展模块。
图像平坦区域提取模块：对于位深度扩展问题，在平坦区域的主要目标是消除带效应等不自然效应，提高直观视觉质量，在非平坦区域的主要目标是尽可能的恢复高逼真的高频细 节。因而，首先我们使用局部平均像素值差异信息来提取图像的平坦区域，将图像划分为平坦区域和纹理(非平坦)区域，然后分别处理。
基于局部自适应像素值调整的平坦区域位深度扩展模块：为消除低位深度图像在平坦区域的不自然效应，首先定位与某邻居像素值差异为1的位变化像素，再依据该像素周围像素的位深度信息自适应的调整该像素的值，使得位变化更加自然，消除带效应等不自然效应。
基于卷积神经网络的非平坦区域位深度扩展模块：图像位深度在降低的过程会丢失舍弃的比特位的数值信息，因而恢复这些比特位的数值是一个病态问题，而传统的比特位深度扩展的方法多基于简单策略，缺乏使用有效的基于学习的重建算法。我们首先使用大量的低比特位图像和对应的高比特位图像训练一个放大残差学习的卷积神经网络，再通过使用该网络重建更加准确的缺失比特位数值信息。
与现有技术相比，本发明的有益效果是：
目前的图像位深度扩展方法多基于传统的聚焦于如何消除平坦区域的带效应的方法，难以解决如何高逼真度的恢复缺失的比特位的技术问题。本发明提供一种基于混合框架的图像位深度扩展方法及其装置，通过融合传统去带效应算法和基于深度网络的学习算法，可以较好的移除图像平坦区域的不自然效应，同时更逼真的恢复所缺失的比特位的数值信息。
本发明使用了基于学习的方法，通过训练有效的深度网络来解决逼真的恢复缺失比特位这一病态问题。同时，针对平坦区域，本发明使用了简单而鲁棒的局部自适应像素值调整的方法，可以有效的抑制平坦区域的带效应、振铃效应、以及平坦噪声等不自然效应，以提升平坦区域的主观视觉质量。
附图说明
图1是本发明方法的流程框图。
图2是本发明采用的卷积神经网络方法的流程框图。
图3是采用本发明和几种现有方法从6bits到8bits图像重建时，重建图像主观质量的效果对比图；
其中，左上角图像为图像中所选取的平坦区域位置示意，该平坦图像区域使用不同方法进行位深度扩展的结果分别为，(a)为零扩展方法，(b)为位复制方法，(c)为理想增益方法，(d)为本发明方法，(e)为真实图像。
具体实施方式
下面结合附图，通过实施例进一步描述本发明，但不以任何方式限制本发明的范围。
本发明提供一种基于混合框架的图像位深度扩展方法，通过融合传统去带效应算法和基于深度网络的学习算法，可以较好的移除图像平坦区域的不自然效应，同时更逼真的恢复所缺失的比特位的数值信息。
图1所示是本发明方法的流程，包括图像平坦区域的提取过程、基于局部自适应像素值调整的平坦区域位深度扩展过程、基于卷积神经网络的非平坦区域位深度扩展过程，具体包括如下步骤：
第一步：图像平坦区域的提取。
本发明使用局部像素值差异来衡量局部数值变化的程度，其计算方式如式1：
Figure PCTCN2018087424-appb-000010
其中，P为局部总的像素个数，g i(i＝1,2,…,P)为局部的每个像素，
Figure PCTCN2018087424-appb-000011
为局部平均像素值。完成计算每个局部像素值差异D之后，可以得到整副图像的平均局部像素值差异
Figure PCTCN2018087424-appb-000012
因为平坦区域的局部像素值变化较为缓慢，即差异D数值较小，所以通过引入系数λ，如果局部区域满足
Figure PCTCN2018087424-appb-000013
则该局部区域为平坦区域，反之则为非平坦的区域。
第二步：基于局部自适应像素值调整的平坦区域位深度扩展。
在图像的平坦区域，本发明使用基于局部自适应像素值调整的方法来消除低位深度图像中的平坦区域带效应和噪声等不自然效应。假定输入为位深度为l-bits的低位深度图像Y，位深度扩展后获取的高位深度图像X的位深度为h-bits。
因为平坦区域的带效应存在于比特位数值差异为1的像素处，因而我们将满足|g i-g c|＝1的像素点g c称为等高线点(Contour Pixel)，而g i为该点的邻居像素。本发明中使用与像素点相邻的8个像素作为其邻居像素。
对于每个等高线点，我们可以计算其提升因子σ +和抑制因子σ -，具体计算公式如下：
Figure PCTCN2018087424-appb-000014
Figure PCTCN2018087424-appb-000015
然后，我们可以通过式4得到该等高线点位深度提升后的像素值，为:
Figure PCTCN2018087424-appb-000016
其中，ZP(g c)为对像素值g c使用传统零扩展方法提高位深度，α为调整值参数，默认为 0.125，可以通过调节该参数以提高或降低去噪程度。
第三步：基于卷积神经网络的非平坦区域位深度扩展。
本发明使用放大残差学习的卷积神经网络来学习图像的缺失位深度重建。因为在位深度扩展中，所缺失和增加的比特位均为最末尾的比特位，因而缺失量的数值较小。为了有效训练网络以重建缺失的比特位信息，在深度网络训练的过程中，本发明使用低位深度图像Y作为输入，将对应的高位深度图像X与Y的残差(X-Y)经过因子β放大后的值β(X-Y)作为标定真实值。
对于输入Y，该卷积神经网络F的重建结果定义为F(Y)，则最终重建的高位深度图像表示为式5：
Figure PCTCN2018087424-appb-000017
其中，
Figure PCTCN2018087424-appb-000018
X *为使用卷积网络重建获取的高位深度图像。
本发明所采用的卷积神经网络的结构如图2所示，其结构依次为：
a.3×3卷积核构成的输入层，该输入层将输入图像输出为64张特征图像；
b.3×3卷积核构成的卷积层，输出为64张特征图像；
c.批正则化层；
d.ReLU激励函数层；
e.3×3卷积核构成的卷积层，输出为64张特征图像；
f.批正则化层；
g.ReLU激励函数层；
h.3×3卷积核构成的卷积层，输出为64张特征图像；
i.批正则化层；
j.ReLU激励函数层；
k.1×1卷积核构成的卷积层，输出为64张特征图像；
l.批正则化层；
m.ReLU激励函数层；
n.3×3卷积核构成的输出层，输出为1张重建残差图像F(Y)。
自适应像素值调整的方法可以更好的抑制平坦区域不自然效应，但会引入非平坦区域模糊，进一步损失高频细节信息；基于卷积神经网络的方法可以更准确的恢复缺失的比特位数值，但平坦区域的主观质量仍受到带效应和噪声的影响。通过综合这两种方法各自的优势， 我们在平坦区域使用基于自适应像素调整方法的结果，而在非平坦的区域使用卷积神经网络重建的结果，融合两种方法最终得到高位深度图像。
表1给出了从6bits到8bits图像重建时在若干数据集上本发明方法与三种传统方法(零扩展、位复制、理想增益)的峰值信噪比(PSNR)效果比较，Set5、Set14、Kodak和B100为4个图像测试集；通过数值比较可以看出，通过使用基于卷积神经网络的学习方法，可以更准确的恢复出缺失位的数值信息，从而还原更为逼真的纹理高频细节。
表1 不同测试集上6bit到8bit重建的平均峰值信噪比(PSNR)对比
零扩展方法 位复制方法 理想增益方法 本发明方法
Set5 43.26 45.53 46.29 46.64
Set14 42.87 45.58 46.18 46.22
Kodak 42.61 45.49 46.10 46.36
B100 42.64 45.17 45.87 46.32
图3给出了从6bits到8bits图像重建时本发明和几种现有传统方法的重建图像主观质量的对比，可以看出，本发明的方法重建图像的平坦区域的带效应和噪声都得到很好的抑制，从而提高了主观视觉质量。
需要注意的是，公布实施例的目的在于帮助进一步理解本发明，但是本领域的技术人员可以理解：在不脱离本发明及所附权利要求的精神和范围内，各种替换和修改都是可能的。因此，本发明不应局限于实施例所公开的内容，本发明要求保护的范围以权利要求书界定的范围为准。

Claims (7)

  1. 一种基于混合框架的图像位深度扩展方法，包括图像平坦区域的提取过程、基于局部自适应像素值调整的平坦区域位深度扩展过程、基于卷积神经网络的非平坦区域位深度扩展过程；包括如下步骤：
    第一步：提取图像的平坦区域，将图像划分为平坦区域和非平坦区域；
    第二步：对平坦区域，基于局部自适应像素值调整进行位深度扩展；包括：
    首先定位与邻居像素值有差异的位变化像素；
    再依据该像素周围像素的位深度信息，使用基于局部自适应像素值调整的方法，自适应调整该像素的值，由此消除低位深度图像平坦区域的不自然效应；
    第三步：对非平坦区域，基于卷积神经网络进行位深度扩展；包括：
    首先使用大量的低比特位图像和对应的高比特位图像训练一个放大残差学习的卷积神经网络；
    再通过使用训练好的卷积神经网络重建更加准确的缺失比特位数值信息；
    完成基于卷积神经网络的非平坦区域位深度扩展过程；
    通过上述步骤实现图像的位深度扩展。
  2. 如权利要求1所述基于混合框架的图像位深度扩展方法，其特征是，第一步中将图像划分为平坦区域和非平坦区域具体包括如下步骤；
    11)使用局部平均像素值差异信息来衡量局部数值变化的程度，计算方式如式1：
    Figure PCTCN2018087424-appb-100001
    其中，D为每个局部像素值差异；P为局部总的像素个数；g i(i＝1,2,…,P)为局部的每个像素；
    Figure PCTCN2018087424-appb-100002
    为局部平均像素值；
    12)根据每个局部像素值差异D得到整副图像的平均局部像素值差异
    Figure PCTCN2018087424-appb-100003
    13)平坦区域的局部像素值变化较为缓慢，即差异D数值较小；引入系数λ，如果局部区域满足
    Figure PCTCN2018087424-appb-100004
    则该局部区域为平坦区域，反之则为非平坦的区域。
  3. 如权利要求1所述基于混合框架的图像位深度扩展方法，其特征是，第二步针对图像的平坦区域，使用基于局部自适应像素值调整的方法，以消除低位深度图像中的平坦区域的不自然效应；具体地，假定输入为位深度为l-bits的低位深度图像Y，位深度扩展后获取的高位深度图像X的位深度为h-bits；
    平坦区域的带效应存在于比特位数值差异为1的像素处，将满足|g i-g c|＝1的像素点g c 称为等高线点，g i为该点的邻居像素；使用与像素点相邻的8个像素作为该像素点的邻居像素；
    对每个等高线点，通过式2、式3计算等高线点的提升因子σ +和抑制因子σ -：
    Figure PCTCN2018087424-appb-100005
    Figure PCTCN2018087424-appb-100006
    然后，通过式4得到该等高线点位深度提升后的像素值：
    Figure PCTCN2018087424-appb-100007
    其中，ZP(g c)对像素值g c使用零扩展方法提高位深度；α为调整值参数，可通过调节该参数以提高或降低去噪程度。
  4. 如权利要求3所述基于混合框架的图像位深度扩展方法，其特征是，所述调整值参数α取值为0.125。
  5. 如权利要求1所述基于混合框架的图像位深度扩展方法，其特征是，第三步在卷积神经网络训练的过程中，使用低位深度图像Y作为输入，将对应的高位深度图像X与Y的残差(X-Y)经过因子β放大后的值β(X-Y)作为标定真实值；对于输入Y，将该卷积神经网络F的重建结果定义为F(Y)，则最终重建的高位深度图像表示为式5：
    Figure PCTCN2018087424-appb-100008
    其中，
    Figure PCTCN2018087424-appb-100009
    X *为使用卷积网络重建的高位深度图像。
  6. 如权利要求5所述基于混合框架的图像位深度扩展方法，其特征是，卷积神经网络的结构依次包括：
    a.3×3卷积核构成的输入层，该输入层将输入图像输出为64张特征图像；
    b.3×3卷积核构成的卷积层，输出为64张特征图像；
    c.批正则化层；
    d.ReLU激励函数层；
    e.3×3卷积核构成的卷积层，输出为64张特征图像；
    f.批正则化层；
    g.ReLU激励函数层；
    h.3×3卷积核构成的卷积层，输出为64张特征图像；
    i.批正则化层；
    j.ReLU激励函数层；
    k.1×1卷积核构成的卷积层，输出为64张特征图像；
    l.批正则化层；
    m.ReLU激励函数层；
    n.3×3卷积核构成的输出层，输出为1张重建残差图像F(Y)。
  7. 利用权利要求1～6所述基于混合框架的图像位深度扩展方法实现的基于混合框架的图像位深度扩展装置，其特征是，包括图像平坦区域提取模块、基于局部自适应像素值调整的平坦区域位深度扩展模块和基于卷积神经网络的非平坦区域位深度扩展模块；
    所述图像平坦区域提取模块用于将图像划分为平坦区域和非平坦区域；
    所述基于局部自适应像素值调整的平坦区域位深度扩展模块用于针对平坦区域，基于局部自适应像素值调整，消除低位深度图像在平坦区域的不自然效应；
    所述基于卷积神经网络的非平坦区域位深度扩展模块用于针对非平坦区域，基于卷积神经网络进行位深度扩展。
PCT/CN2018/087424 2017-08-21 2018-05-18 基于混合框架的图像位深度扩展方法及装置 Ceased WO2019037487A1 (zh)

Priority Applications (1)

Application Number Priority Date Filing Date Title
US16/640,511 US11087439B2 (en) 2017-08-21 2018-05-18 Hybrid framework-based image bit-depth expansion method and device

Applications Claiming Priority (2)

Application Number Priority Date Filing Date Title
CN201710717259.1 2017-08-21
CN201710717259.1A CN107481278B (zh) 2017-08-21 2017-08-21 基于混合框架的图像位深度扩展方法及装置

Publications (1)

Publication Number Publication Date
WO2019037487A1 true WO2019037487A1 (zh) 2019-02-28

Family

ID=60601809

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/CN2018/087424 Ceased WO2019037487A1 (zh) 2017-08-21 2018-05-18 基于混合框架的图像位深度扩展方法及装置

Country Status (3)

Country Link
US (1) US11087439B2 (zh)
CN (1) CN107481278B (zh)
WO (1) WO2019037487A1 (zh)

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN111325781A (zh) * 2020-02-17 2020-06-23 合肥工业大学 一种基于轻量化网络的位深度提升方法及其系统
CN111798381A (zh) * 2020-05-20 2020-10-20 北京迈格威科技有限公司 图像转换方法、装置、计算机设备和存储介质
US12236676B2 (en) 2019-05-24 2025-02-25 Google Llc Image extension neural networks

Families Citing this family (10)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN107481278B (zh) 2017-08-21 2019-06-28 北京大学深圳研究生院 基于混合框架的图像位深度扩展方法及装置
KR102516366B1 (ko) * 2018-05-30 2023-03-31 삼성전자주식회사 특징 데이터 획득 방법 및 장치
CN109784148A (zh) * 2018-12-06 2019-05-21 北京飞搜科技有限公司 活体检测方法及装置
US11183155B2 (en) * 2019-05-16 2021-11-23 Apple Inc. Adaptive image data bit-depth adjustment systems and methods
CN110852964A (zh) * 2019-10-30 2020-02-28 天津大学 一种基于深度学习的图像比特增强方法
CN112235571B (zh) * 2020-08-31 2023-04-07 咪咕文化科技有限公司 视频位深扩展方法、装置、电子设备及存储介质
CN112788338B (zh) * 2020-12-31 2022-08-26 展讯通信(天津)有限公司 图像压缩及解压缩方法、设备、装置和存储介质
CN112833818B (zh) * 2021-01-07 2022-11-15 南京理工大学智能计算成像研究院有限公司 一种单帧条纹投影三维面型测量方法
US11778305B2 (en) * 2021-06-18 2023-10-03 Qualcomm Incorporated Composite image signal processor
CN116416168A (zh) * 2021-12-28 2023-07-11 北京小米移动软件有限公司 图像处理方法、装置、电子设备和存储介质

Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1241748A (zh) * 1997-12-02 2000-01-19 数字设备公司 通过位复制使位深度增加
EP1094420A2 (en) * 1999-10-22 2001-04-25 Sharp Kabushiki Kaisha Bit-depth extension of digital displays using noise
CN106663311A (zh) * 2014-08-06 2017-05-10 谷歌公司 用于增加图像的位深的系统和方法
CN107481278A (zh) * 2017-08-21 2017-12-15 北京大学深圳研究生院 基于混合框架的图像位深度扩展方法及装置

Family Cites Families (2)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
KR102146560B1 (ko) * 2014-02-17 2020-08-20 삼성전자주식회사 영상 보정 방법 및 장치
US10460231B2 (en) * 2015-12-29 2019-10-29 Samsung Electronics Co., Ltd. Method and apparatus of neural network based image signal processor

Patent Citations (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN1241748A (zh) * 1997-12-02 2000-01-19 数字设备公司 通过位复制使位深度增加
EP1094420A2 (en) * 1999-10-22 2001-04-25 Sharp Kabushiki Kaisha Bit-depth extension of digital displays using noise
CN106663311A (zh) * 2014-08-06 2017-05-10 谷歌公司 用于增加图像的位深的系统和方法
CN107481278A (zh) * 2017-08-21 2017-12-15 北京大学深圳研究生院 基于混合框架的图像位深度扩展方法及装置

Cited By (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US12236676B2 (en) 2019-05-24 2025-02-25 Google Llc Image extension neural networks
CN111325781A (zh) * 2020-02-17 2020-06-23 合肥工业大学 一种基于轻量化网络的位深度提升方法及其系统
CN111798381A (zh) * 2020-05-20 2020-10-20 北京迈格威科技有限公司 图像转换方法、装置、计算机设备和存储介质

Also Published As

Publication number Publication date
US11087439B2 (en) 2021-08-10
US20200364833A1 (en) 2020-11-19
CN107481278A (zh) 2017-12-15
CN107481278B (zh) 2019-06-28

Similar Documents

Publication Publication Date Title
WO2019037487A1 (zh) 基于混合框架的图像位深度扩展方法及装置
CN110599409B (zh) 基于多尺度卷积组与并行的卷积神经网络图像去噪方法
CN110175964B (zh) 一种基于拉普拉斯金字塔的Retinex图像增强方法
CN110473154B (zh) 一种基于生成对抗网络的图像去噪方法
CN103606137B (zh) 保持背景和细节信息的直方图均衡化方法
CN110517327B (zh) 一种基于颜色校正和对比度拉伸的水下图像增强方法
CN109671022B (zh) 一种基于深度特征翻译网络的图片纹理增强超分辨率方法
CN113222875B (zh) 一种基于色彩恒常性的图像和谐化合成方法
CN111583123A (zh) 一种基于小波变换的融合高低频信息的图像增强算法
CN104574337B (zh) 基于双边伽马校正和多尺度图像融合的图像增强方法
CN106204447A (zh) 基于总变差分和卷积神经网络的超分辨率重建方法
CN118822908A (zh) 基于曲线小波注意力和傅里叶进行低光照图像增强方法
CN103295194A (zh) 亮度可控与细节保持的色调映射方法
CN111105376B (zh) 基于双分支神经网络的单曝光高动态范围图像生成方法
CN102063712B (zh) 基于子带结构的多曝光图像融合方法
CN102968772A (zh) 一种基于暗通道信息的图像去雾方法
CN103093433A (zh) 基于区域划分和字典学习的自然图像去噪方法
CN114897694A (zh) 基于混合注意力和双层监督的图像超分辨率重建方法
CN115034979B (zh) 一种基于双路径联合校正的深度学习水下图像增强方法
CN106570831A (zh) 一种灰度图像对比度均衡增强的方法
CN113096029A (zh) 基于多分支编解码器神经网络的高动态范围图像生成方法
CN118351040A (zh) 一种基于RetinexNet改进的低光照图像增强方法
CN110728643A (zh) 一种基于卷积神经网络的低照度带噪声图像优化方法
CN111462002A (zh) 一种基于卷积神经网络的水下图像增强与复原方法
CN108305232A (zh) 一种单帧高动态范围图像生成方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 18848351

Country of ref document: EP

Kind code of ref document: A1

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 18848351

Country of ref document: EP

Kind code of ref document: A1