JP7004351B2 - Equipment for color banding processing, image processing system, processing program, and storage medium - Google Patents

Description

The present invention relates to an apparatus, an image processing system, a processing program, and a storage medium related to color debanding processing.

Bhagavathy et al. Suppressed color banding by probabilistic dithering based on multiscale analysis. In addition, Mizuno et al. Resolved the two-dimensional interpolation problem by reducing it to the one-dimensional Poisson's equation in order to perform gradation interpolation of color banding. Baugh et al. Detected the banding region from the number of pixels in the block divided into regions by the RGB pixel level, and removed the color banding by weighting and smoothing the banding region according to the distance from the neighboring block. The above matters are described in Non-Patent Document 1 to Non-Patent Document 3 below, respectively.

G. Baugh, A. Kokaram and F. Pitie, Advanced video debanding, Proceedings of the 11th European Conference on Visual Media Production (CVMP'14), London, United Kingdom, (November 2014). S.Bhagavathy, J. Llach and J. Zhai, Multiscale probabilistic dithering for suppressing contour artifacts in digital images, IEEE Transactions on Image Processing, 18-9 (September 2009), 1936-1945. Akira Mizuno, Masaki Igarashi, Masayuki Ikebe, Smooth gradation restoration of images using Poisson's equation, Journal of the Institute of Image Information and Television Engineers, 67-8 (2013), J326-J333.

Most of the conventional methods are optimization processing that involves repetition and global processing that uses the entire image, and are not local processing that is performed while raster scanning pixels, so the processing time is relatively long. , Not suitable for video processing that requires continuous processing of many images.

A further problem is how to determine adjustment parameters in image processing. In image processing, there are usually some adjustment parameters, and how to determine the adjustment parameters depends on the content of each processing and the image / video content, and is an empirical value based on subjective evaluation. Has been used conventionally. There is a similar problem in the color debanding process.

In the present invention, while raster-scanning an image, local processing using pixels in a nearby block region is performed. The filter parameter is a debanding processed image as a result of generating a pseudo color banding image from an image collected from the Internet or the like, extracting a very small amount of the banding, and processing an original image in which no banding exists and a pseudo color banding image. Estimate the optimal parameter that minimizes the amount of Calback-Liber information between and. In addition, the optimum model for predicting the optimum parameters for the banding limit is determined by multiple regression analysis and variable selection (model selection) using a plurality of Internet-collected images.

According to the present invention, color banding can be removed by local processing using pixels in a nearby block region. By using a model predetermined from images collected from the Internet, it is possible to predict the optimum parameters for the color banding limit and realize the color banding process. This is a method for automatically determining parameters that can obtain extremely reasonably reasonable results, as opposed to subjective and empirical parameter determination in conventional image processing.

It is a color debanding processing block diagram. It is a block diagram of the optimum parameter learning of a color debanding process. It is a block diagram of the color debanding process by the optimum parameter prediction. Hybrid Log-Gamma (Hybrid Log-Gamma), PQ (Perfectual Quantizer) OETF standard dynamic range (BT.709 (Recommendation ITU-R BT.709-5, Parameter values for the HDTV standards for production and international programme exchange) (2002) )) / BT.2020 (Recommendation ITU-R BT.2020-1, Parameter values for ultra-high definition television systems for production and international programme exchange (2014))) with OETF (commer correction function). It is a figure explaining the automatic optimization of a color debanding processing parameter, (a) shows the block diagram of the optimum parameter learning of a color debanding processing, and (b) shows the block diagram of the color debanding processing by the optimum parameter prediction. It shows. It is a figure explaining an example of the image simulation of a color debanding process. It is a figure which shows the image used for learning the optimum parameter of a color debanding processing filter. It is a figure which shows the result of the multiple regression analysis and the variable selection (model selection) of the optimum parameter of a debanding processing filter. It is a figure which shows the example of the debanding processing image by the optimum parameter learning.

(Outline 1 of the invention)
The present invention proposes a method of correcting "color banding", which is a pseudo-contour-like interference that tends to occur in HDR video, by simple image processing. First, the banding candidate pixel is detected from the standard deviation of the luminance value in the block region in the vicinity of the pixel of interest. Banding is removed by using block pixels in the vicinity of the banding candidate pixels, increasing the weight for pixels having similar colors and similar brightness as possible, and performing weighted averaging of the luminance signals.

Next, the banding extra-trace amount of the pseudo color banding image generated from a plurality of images collected from the Internet or the like is extracted. For the banding limit, a model that predicts the optimum parameter that minimizes the objective function based on the amount of Kullback-Leibler information between the original image and the debanding processed image is determined by learning.

More specifically, the optimum parameters are automatically determined, such as a structure for removing color banding by local processing using pixels in the neighboring block region by raster scanning, a structure for detecting color banding candidate pixels and a structure for color banding removal processing. To collect Internet search images, generate pseudo-color banding images, estimate optimal parameters that minimize the amount of Calback-Liblar information between the original image, and from a combination of banding traces and optimal parameters. A structure that determines the optimum model by predicting the optimum parameter by multiple regression analysis and model selection, and when removing color banding, the banding limit is extracted, the optimum parameter is predicted from the learned model, and color debanding is performed. An image processing device including a structure for performing processing.

Further, the banding removal is performed by weighting and averaging the brightness values in the neighboring block area with the product of the banding candidate pixel detection processing unit and the weights of the banding candidate pixels due to the level difference between the brightness value and the color difference value in the neighboring block area as the final weight. A pseudo-color banding image generator that generates a pseudo-color banding image from an Internet search image, and an optimal parameter estimation method that optimally estimates parameters for removing banding from a pseudo-color banding image. The banding extraordinary trace extraction unit that extracts the It shall be provided with an optimum parameter prediction unit for predicting the optimum parameter by using it.

(Outline 2 of the invention)
As a realization method, it can be realized by a hardware device that processes a baseband video signal, or it can be realized by a software that processes an MXF file and a computer-based device that executes the software. However, any configuration can be realized by using a device that converts the MXF file into a baseband video signal or reversely converts it.

It is also possible to perform processing on the cloud by transmitting a compressed image of a camera image or an MXF file by IP (Internet Protocol). Various system forms are conceivable, such as decoding the compressed video transmitted by IP into a baseband video signal, compressing the result of color debanding processing again, and distributing the stream.

As a model estimation method (learning method) for estimating the optimum model from the combination of the banding limit and the optimum parameter, not only the multiple regression analysis but also the principal component analysis and the application of the nonlinear model can be considered. As the image used for learning the optimum parameters, not only the image obtained by Internet search but also the image from various media may be used as long as the image does not have banding.

FIG. 1 is a color debanding processing block diagram. As shown in FIG. 1, the procedure of the color debanding process is as follows.

1. Convert the color RGB input to luminance color difference _{YC BCR} (RGB → _{YC BCR} ).
2. The standard deviation in the vicinity block region of the converted Y pixel value is calculated, and the pixel equal to or less than the threshold value (Threshold) is set as a banding candidate pixel (Band Vector). Further, a condition is added in which the pixel value of Y is equal to or higher than another threshold value (Threshold 2). A band mask (Band Mask) is obtained by mask-shaping the banding candidate pixels obtained in this manner by a binary median filter.

3. _Weights of the pixel values of _CBCR and Y in the neighboring block region according to the level distance are calculated (Weight Calc), and Y is weighted and averaged using the weight (Weight). However, the weighted averaging is performed only for the banding candidate pixels by the band mask. At the time of weighting averaging other than the banding candidate pixels, the weight of the pixels other than the banding candidate pixels in the neighboring block region is set to 0.

4. The _{Y'and CBC R} weighted and averaged in this way are converted to RGB ( _YCBCR → RGB) and output. In the case of luminance color difference _{YC BCR} input / output, mutual conversion between RGB and _{YC BCR} in input / output is not necessary.

The weighted average of the luminance value Y is calculated as follows using the color difference signal _{CBC R and} the weighting coefficient w calculated from the luminance signal Y.

The weighting coefficient w is the product of the weight w _rC based on the level distance of the color difference signal _C = (CB, _CR ) ^T and the weight w _rY based on the level distance of the luminance signal Y, respectively, as follows by the Gaussian function. Define.

σ _rC and σ _rY are parameters for adjusting the permissible range of the level distances of the color difference signal and the luminance signal, respectively.

Further, FIG. 2 is a block diagram of optimal parameter learning for color debanding processing. As shown in FIG. 2, the procedure of prior parameter learning for predicting the optimum parameters of the debanding processing filter that removes color banding is as follows.
1. Perform an image search on the Internet and download multiple appropriate images.
2. Generate a pseudo color banding image using the downloaded image as the original image. For the pseudo color banding image, the brightness value Y of the image is used.

It is generated by multiplying it by s. Where s is an arbitrary positive number,

Is a floor function that truncates the fractional part.
3. A small amount of banding is detected from the banding candidate region in the pseudo color banding image.
4. The debanding processing parameter is the sliding simplex method so that the amount of Calback-Libler information calculated from the brightness value histogram of the banding candidate pixels between the original image and the image obtained by debanding the pseudo-color banding image is minimized. Optimized by (Nelder-Med method).
5. Multiple regression analysis is performed on the combination of the above-mentioned banding limit and the optimum debanding processing parameter, and the optimum model for predicting the optimum debanding processing parameter is determined by variable selection (model selection).

FIG. 3 is a block diagram of color debanding processing by optimal parameter prediction. As shown in FIG. 3, the procedure for performing the color debanding process using the training result between the banding limit amount and the optimum parameter in the training image is as follows.
1. Extract a very small amount of banding from the banding image.
2. Predict the optimum parameters using the learning results for the banding limit.
3. Debanding processing is performed using the predicted optimum parameters.
Those skilled in the art can understand the method of extracting the banding extraordinary amount and the learning method by multiple regression analysis and variable selection for estimating the model for predicting the optimum parameter for the banding extraordinary amount by referring to the following papers. Is.

(Detailed explanation: Color debanding processing for HDR video: Optimal parameter learning using Internet search images)

(Abstract)
"Color banding", which is a pseudo-contour-like interference that occurs in HDR video, is corrected by simple image processing. Banding candidate pixels are detected from the brightness value in the neighboring block area of the pixel of interest and its standard deviation, and the weight of the pixels in the neighboring block area with the same color as possible and the brightness is increased to increase the weight of the brightness signal. Banding is removed by performing a weighted average.

In order to automatically optimize the parameters of the debanding processing filter, a pseudo color banding image is generated from a plurality of images collected from the Internet, and a small amount of banding is extracted by simple image processing. For the pseudo-color banding image, the optimum parameter of the debanding processing filter that minimizes the amount of Kullback-Leibler information between the original image is estimated by the sliding simplex method, and the optimum parameter of the debanding processing filter for the banding limit is estimated. The optimum model for predicting is determined by multiple regression analysis and variable selection (model selection) by Akaike's Information Criterion (AIC).

(1 Introduction)
4K test broadcasting as next-generation television broadcasting started on June 2, 2014 on CS (Communication Satellite) and cable television (Next Generation Broadcasting Promotion Forum (NexTV-F), http: // www. .nextv-f.jP /). We are accelerating toward the start of practical broadcasting in 2018 (as soon as possible), including 8K (Super Hi-Vision (Reference [17])) (Ministry of Internal Affairs and Communications "Follow-up Meeting on 4K / 8K Roadmap (6th)" Meeting) Handouts ”, July 2015, http://www.soumu.go.jp/main_sosiki/kenkyu/4k8kroadmap/02ryutsu11_03000046.html).

4K / 8K ultra-high-definition video has not only resolution but also wide color range, high frame rate, and high bit depth ITU-R Recommendation BT.2020 standard (Recommendation ITU-R BT.2020-1, Parameter values for ultra- Although it is defined as high definition television systems for production and international programme exchange (2014)), in recent years, attention has been further focused on high dynamic range (HDR), which expands the brightness of images. With the improvement of display device technology, it has become possible to increase the maximum brightness (peak brightness) of the display device (extend the dynamic range) without changing the display brightness of "black". By using the area where the reproduction range is expanded for highlight reproduction, it is possible to provide a new viewing experience, enable near-realistic highlight reproduction (specular reflection and gloss reproduction), and improve highlight areas such as overexposure. It is said to be effective (Ministry of Internal Affairs and Communications "Information and Communication Council Information and Communication Technology Subcommittee ITU Subcommittee Broadcasting Business Committee (19th) Handouts", held on September 2, 2015 http: // www. soumu.go.jp/main_sosiki/joho_tsusin/policyreports/joho_tsusin/docs_Broadcast/02ryutsu08_03000224.html).

With the latest cameras, the evolution of image sensors has made it possible to take images beyond the dynamic range of the past, and each company independently proposes a non-linear transfer function for compressing the expanded signal level. (S-Log: A new LUT for digital production mastering and interchange applications, https://pro.Sony.com/bbsccms/assets/files/mkt/cinema/solutions/slog_manual.pdf, V-Log / V -Gamut REFERENCE MANUAL, http://Pro-av.Panasonic.net/jp/varicam/common/pdf/VARICAM_V-Log_V-Gamut.pdf, Canon-Log Cine Optoelectronic Transfer Function, http://learn.usa.canon .com / app / pdfs / white_papers / White_Paper_Clog_optoelectronic.pdf).

Such a transfer function is called an optical-electric transfer function (Opto-Electrical Transfer Function, OETF) and an electric-optical transfer function (EOTF) as its inverse function, and is mainly an exponent. It is defined by a logarithmic function.

What is called Hybrid Log Gamma (HLG) (ARIB STD-B67 Version 1.0, Essential parameter values for the extended image dynamic range television (EIRTV) system for programme production (2015)) is NHK (Japan Broadcasting Corporation) and BBC (UK). It is a method jointly proposed by the Broadcasting Corporation), and its OETF is a non-linear transfer function that combines a comma correction function and a logarithmic function in the conventional standard dynamic range (SDR) [References 3 and 13]. What is called PQ (Perfectual Quantizer) (SMPTE ST2084: 2014, Dynamic range electro-optical transfer function of mastering reference displays (2014)) has been adopted as the standard HDR method in 4K (UHD) Blu-ray Disc ^TM (registered trademark). (White Paper Blu-ray Disc ^TM Read-Only Format, Coding constraints on HEVC video streams for BD-ROM Version 3.0, June 2015. http://www.blu-raydisc.com/Assets/Downloadablefile/BD -ROM-AV-WhitePaper_HEVC_150608a-clean.pdf).

FIG. 4 shows a hybrid Log gamma (Hybrid Log-Gamma) and a PQ (Perfectual Quantizer) OETF with a standard dynamic range (BT.709 (Recommendation ITU-R BT.709-5, Parameter values for the HDTV standards for production and). With OETF (commercial correction function) of international programme exchange (2002)) / BT.2020 (Recommendation ITU-R BT.2020-1, Parameter values for ultra-high definition television systems for production and international programme exchange (2014)) It is a figure which shows. As shown in FIG. 4, Hybrid Log-Gamma (SMPTE ST2084: 2014, Dynamic range electro-optical transfer function of mastering reference displays (2014)) [Reference 3, Document 13], PQ (Perceptual Quantizer). (White Paper Blu-ray Disc ^TM Read-Only Format, Coding constraints on HEVC video streams for BD-ROM Version 3.0, June 2015. http://www.blu-raydisc.com/Assets/Downloadablefile/BD-ROM-AV -WhitePaper_HEVC_150608a-clean.pdf), standard dynamic range (BT.709 (Recommendation ITU-R BT.709-5, Parameter values for the HDTV standards for production and international programme exchange (2002)) / BT.2020 (Recommendation ITU-) R BT.2020-1, Parameter values for ultra-high definition television systems for production and international programme exchange (2014))) optical transfer function (OETF). As shown in FIG. 4, the PQ is adjusted so that the maximum level input that can be expressed by the hybrid Log gamma is 12.0 (1200%) and the output is 1.0 (100%). The standard dynamic range OETF (gamma correction function) clips an output of 1.0 or higher to 1.0.

HDR video is a high-brightness display of signal levels exceeding 100% that have been level-converted by various non-linear OETFs, compressed and transmitted, and then returned to the original signal levels (linear RGB) on the receiver side. However, in some scenes, pseudo-contoured obstruction appears. In particular, it tends to occur in a region such as the sky where the gradation changes smoothly with high brightness and low contrast (for example, it is displayed as if a striped pattern appears in the sky).

Such a phenomenon is known as color banding, and the calculation error associated with the calculation process of EOTF / OETF that performs level conversion, various process processes such as color correction in linear RGB, and the level by OETF. The cause is considered to be compression / decompression of the converted HDR video. It can be said to be a form of distortion in image / video compression.

Bhagavathy et al. Suppressed color banding by probabilistic dithering based on multiscale analysis [Reference 2]. Mizuno et al. Solved the two-dimensional interpolation problem by reducing it to the one-dimensional Poisson's equation in order to perform gradation interpolation of color banding [Reference 16]. Baugh et al. Detected the banding area from the number of pixels in the block divided into areas at the RGB pixel level, and removed the color banding by weighting and smoothing the banding area according to the distance from the neighboring block [Reference 1]. .. Most of the conventional methods are optimization processing involving repetition and global processing using the entire image, and are not local processing performed while rusk-scanning pixels.

The purpose of this proposal is to correct color banding, which is a pseudo-contour-like obstruction that occurs in HDR video, by simple image processing. Specifically, the banding candidate pixel is detected from the luminance value in the block area near the pixel of interest and its standard deviation. The smaller the standard deviation, the closer the image content is to flatness, and color banding may occur. Furthermore, banding is likely to occur in the region of high luminance value. Then, the banding is removed by weighting and averaging the luminance signals of the pixels in the neighboring block region by increasing the weight of the pixels having the same color as possible and having similar brightness.

The problem is how to determine the parameters in such a debanding filter. In image processing, there are usually some adjustment parameters, and how to determine the adjustment parameters depends on the content of each processing and the image / video content, and is an empirical value based on subjective evaluation. Is used. In this proposal, we try to automatically determine the optimum parameters of the debanding processing filter by learning using images collected by Internet search. A pseudo color banding image is generated using the Internet search image as the original image, and a small amount of banding is extracted. The optimum parameter that minimizes the amount of Kullback-Leibler information between the original image and the pseudo-color banding image is estimated by the downhill simplex method, and the optimum parameter for the banding limit is determined by multiple regression analysis and variable selection (model selection). ..

Attempts to recognize image content by means of extraordinary amounts obtained from images have been actively studied in the 2000s [Reference 4]. This is a method of learning and discriminating SIFT feature quantities [Reference 14] using a support vector machine (Support Vector Machine, SVM) [Reference 21]. Research has also been conducted on image restoration using a large number of images obtained from the Internet. Hays et al. Automatically complemented missing images using similar scene images [Reference 7], and Kevin et al. Performed image restoration by white balance correction, contrast enhancement, and exposure correction [Reference 11].

On the other hand, what has been showing a big boom in recent years is a method called "Deep Learning" in which learning is performed using an image as it is instead of using some image feature [Reference 12]. The essence of deep learning processing is a "filter called a convolutional neural network. It is used not only for image recognition but also for various image processing [Reference 9, Document 6, Document 5, Document 22]. Matsunaga. For the purpose of focus correction in ultra-high-definition video, the deblur restoration performance of a defocused image by a convolutional neural network with a minimum configuration was experimentally evaluated [Reference 15].

The structure of this proposal describes the procedure of color debanding processing in Chapter 2. In Chapter 3, the optimum parameter learning using the Internet search image for automatically optimizing the parameters of the debanding processing filter will be explained. In Chapter 4, a pseudo color banding image is generated from an image without color banding by image simulation, and color debanding processing is performed. The color debanding processed image is quantitatively evaluated by the amount of Kullback-Leibler information. Then, the evaluation results of the debanding process by the optimum parameter learning using the Internet search image are shown and summarized in Chapter 5.

(2 color debanding processing)
FIG. 1 is a block diagram of a color debanding process. The procedure for color debanding processing is as follows.
1. Convert the color RGB input to luminance color difference _{YC BCR} (RGB → _{YC BCR} ).
2. The standard deviation in the vicinity block region of the converted Y pixel value is calculated, and the pixel equal to or less than the threshold value (Threshold 1) is set as a banding candidate pixel (Band Vector). Further, a condition is added in which the pixel value of Y is equal to or higher than another threshold value (Threshold 2). A band mask (Band Mask) is obtained by mask-shaping the banding candidate pixels thus obtained by a binary median filter.

3. _Weights of the pixel values of _CBCR and Y in the neighboring block region according to the level distance are calculated (Weight Calc.), And Y is weighted and averaged using the weights (Weight). However, the weighted averaging is performed only for the banding candidate pixels by the band mask. At the time of weighting averaging of the banding candidate pixels, the weights of the pixels other than the banding candidate pixels in the neighboring block territory are set to 0.
4. Convert such weighted average _Y'and CBC _R to RGB ( _{YCBC R} → RGB) and output.

The weighted average of the luminance value Y is calculated as follows using the color difference signal _{CBC R and} the weighting coefficient w calculated from the luminance signal Y.

The weighting coefficient w is the product of the weight w _rC based on the level distance of the color difference signal _C = (CB, _CR ) ^T and the weight w _rY based on the level distance of the luminance signal Y, and is defined as follows by the Gaussian function. ..

σ _rC and σ _rY are parameters for adjusting the permissible range of the level distances of the color difference signal and the luminance signal, respectively. This is a bilateral filter [Reference 20] that is region-selective with respect to a luminance signal and has a "only" weight based on the level distance of the pixel value (Yaroslavsky filter [Reference 23], and is a bilateral filter. It corresponds to the weight by the spatial distance as the box function.).

In order to obtain a wide range of pixels in the neighboring block region used for the weighted averaging and the binary median filter for mask shaping of the band mask, an atrous algorithm [Reference 8] that acquires pixels in a discrete manner may be used.

(3 Automatic optimization of debanding processing parameters)
The procedure for learning the optimum parameters in advance for predicting the optimum parameters of the debanding processing filter that removes the color banding is as follows (see FIG. 5A). Here, FIG. 5 is a diagram illustrating automatic optimization of color debanding processing parameters, FIG. 5A shows a block diagram of optimum parameter learning for color debanding processing, and FIG. 5B is optimum. It shows a block diagram of color debanding processing by parameter prediction.
1. Perform an image search on the Internet and download multiple appropriate images. (At this time, image acquisition by search and download may be performed automatically).
2. Generate a pseudo color banding image using the downloaded image as the original image. For the pseudo color banding image, the brightness value Y of the image is used.

It is generated by multiplying it by s. Where s is an arbitrary positive number,

Is a floor function that truncates the fractional part.
3. Extract a small amount of banding from the banding candidate pixel region in the pseudo color banding image.
4. The Nelder simplex method (Nelder) so that the Kullback-Leibler information amount [Reference 19] calculated from the luminance value histogram of the banding candidate pixels between the original image and the banding-processed image for the debanding processing parameter is minimized. -Med method) Estimated by [Reference 18].
5. Multiple regression analysis is performed for the combination of the banding extraordinary amount and the optimum debanding processing parameter, and the optimum debanding processing parameter is predicted by the variable selection (model selection) [Reference 19] by the Akaike Information Criterion (AIC). Determine the optimal model for this.

(3.1 Extraction of a very small amount of banding)
As the banding feature amount, in the banding candidate pixel area B: ・ Banding gap BG having a brightness value Y
・ Average banding edge strength with brightness value Y

・ Standard deviation of one-dimensional luminance value Y σ _{Y
-Dispersion} covariance matrix V [CB, _CR ] of two _- dimensional color difference value CBC _R
Can be considered. Since the number of parameters is 1, 1, 1, 3 respectively, it is a 6-dimensional vector (since the variance _- covariance matrix is a symmetric matrix, the number of parameters of V [CB, _CR ] is 3). When the variance-covariance matrix V [ _CB , _CR ] of the two-dimensional color difference value C _{B CR} is used as the standard form by eigenvalue decomposition [Reference 10], and the eigenvalues λ ₁ and λ ₂ (λ ₁ ≧ λ ₂ ) are used. Then, it becomes a five-dimensional vector.

For the banding gap extraordinary amount of the luminance value Y, the histogram is binarized by the threshold value determined from the maximum frequency value of the luminance value histogram of the banding candidate pixels. Then, the median value at a plurality of intervals depending on the number of 0 values between the 1 value and the 1 value in the binarized luminance value histogram is set as an extraordinary amount corresponding to the banding gap BG.

The banding edge strength average extraordinary amount is the average value of the edge strength in the banding candidate pixel region B.

Is calculated as follows.

However, Y (i, j) is a luminance value in the banding candidate pixel region B of the pixel coordinates (i, j), and γ is a gain constant of the edge intensity value.

When calculating, use pixels whose γe is equal to or greater than the threshold T _γe . Therefore, M is the number of pixels in which γe is T _γe or more among the banding candidate pixels.
The (m, n) element of the variance-covariance matrix V [C] with the two-dimensional color difference value _C = (CB, _CR ) ^T is as follows.

Is the solution of [Reference 10].
(3.2 Optimal parameter estimation and multiple regression analysis for debanding processing)
The debanding processing parameters are
_{-Parameter σ rC that} allows the level distance of the color difference signal CBC _R
-Parameter σ rY that allows the level distance of the luminance signal Y
There are two.
The debanding processing parameter is the downhill simplex method (Nelder-Mead) so that the Kullback-Leibler information amount [Reference 19] calculated from the luminance value histogram of the banding candidate pixel between the original image and the debanding processed image is minimized. Method) Estimated according to [Reference 18]. However, in order to estimate σ _rC and σ _rY as small as possible, the final objective function J is defined as follows.

Here, the first term is the amount of Kullback-Leibler information, and the luminance value histogram P (Y) of the original image in the banding candidate pixel region B and the luminance value histogram Q (Y) of the debanding processed image by the debanding processing parameter θ. Calculated from θ). The second term is a regularization term. ν is a regularization parameter that adjusts the trade-off between the closeness between the original image and the debanding processed image in the estimation parameter and the magnitude of the estimation parameter.

The block area in the debanding process is a square block, and its block size is fixed. Therefore, the optimum parameter of the debanding process is determined from the five-dimensional vector of the banding extraordinary amount.

Optimal model for predicting the optimum parameters of debanding processing by applying the partial models of equations (10) and (11) and selecting variables (model selection) by Akaike's Information Criterion (AIC) [Reference 19]. To determine.

In the actual banding image, the extracted banding extraordinary amount is debanded according to the optimum parameters predicted from the optimum model of the equations (10) and (11) (see FIG. 5 (b)).

(4 image simulation)
(4.1 Debanding processing result)
FIG. 6 is a diagram illustrating an example of an image simulation of color debanding processing. FIG. 6A is an original image (400 × 265 pixel size), and FIG. 6B is a pseudo color banding image generated from the original image. It was generated by multiplying the brightness value of the original image by 1/6 and rounding down the fractional part and multiplying it by 6 (luminance value scale s = 6). In FIG. 6 (c), the standard deviation of the luminance value in the 5 × 5 block region of the pseudo color banding image is calculated for each pixel, and the threshold value of 5 or less and the luminance value of 100 or more are the color banding candidate pixels (white). It is a banding mask image. Since the isolated pixels are conspicuous, the mask shaped by the binary median filter processing is shown in FIG. 6 (d). The result of the binary median filter having a block size of 7 × 7 was further processed by the binary median filter having a block size of 11 × 11. FIG. 6 (e) shows the result of debanding processing on the banding candidate pixels based on the banding mask image (d). The debanding processing parameters were σ _rC = 5, σ _rY = 7, and the block size was 11 × 11 (one pixel skip), and the debanding processing was performed. FIG. 6 (f) is a luminance value histogram of the banding candidate pixel region. It is the result of the original image (Original), the pseudo color banding image (Banding), and the color debanding processed image (Debanding), respectively. The amount of Kullback-Leibler information for each original image calculated from the histogram has decreased from 1.788016 to 0.292420, and the debanding processed image is closer to the original image.

(4.2 Optimal parameter learning result of debanding process)
For learning the optimum parameters of the color debanding processing filter, 100 images including the sky where the gradation, which is expected to cause banding, is smoothly changing, were searched from the Internet and used. Figure 7 shows the image used for learning the optimum parameters of the color debanding processing filter (using the image of Flickr, http://www.flickr.com/, Commercial use & mods allowed).

A pseudo color banding image is generated with a brightness value scale s = 5, 6, 7, 8 for each training image, and an equation (9) based on the amount of Kullback-Leibler information between the debanding processed image and the original image. The debanding processing parameters σ _rC and σ _rY that minimize the objective function J in) were estimated by the sliding simplex method. Since the downhill simplex method shows the dependence on the initial value by local optimization, the final result was the one with the smallest J value among the results of estimation by changing the initial value. The initial values of the optimization parameters are uniform random numbers in the intervals {σ _rC | 5 ≤ σ _rC ≤ 20}, {σ _rY | 5 ≤ σ _rY ≤ 20}, and the minimum and maximum values of the parameters that can be taken are 1, 100, respectively. Estimated 20 times each.

The debanding candidate pixel detects pixels having a standard deviation of 5 or less for the brightness value in the 5 × 5 block area, an average of the brightness values in the upper 40% area of the image, and a threshold value or more determined from the standard deviation, and the block size. The mask was shaped by performing the 11 × 11 binary median filter processing twice. The block size in the debanding process was fixed at 19 × 19 (one pixel skip), and the regularization parameter was optimized with ν = 0.008.

Optimal parameters of the debanding processing filter estimated in this way and banding features of the pseudo-color banding image

Was subjected to multiple regression analysis. R language (The R Project for Statistical Computing, http://www.r-project.org/) was used for multiple regression analysis and variable selection (model selection). FIG. 8 shows the results of multiple regression analysis and variable selection (model selection) of the optimum parameters of the debanding processing filter.

Table 1 shows the evaluation results of the color debanding processing filter by learning the optimum parameters. Color debanding processing Color debanding processing was performed using 100 evaluation images different from those used for learning the optimum parameters of the filter. For the banding limit of the pseudo color banding image generated from the evaluation image as the original image, the optimum parameters of the evaluation image were predicted from the result of the optimum parameter learning by the learning image, and the debanding process was performed. The debanding process was performed using the optimum parameters estimated by the sliding down simplex method. The average amount of Kullback-Leibler information between the original image and the pseudo-color banding image, and the original image and the debanding processed image, respectively (standard deviation in parentheses). It can be seen that even when the optimum parameter is predicted from the result of the optimum parameter learning by the learning image, the debanding processing result by the optimum parameter estimated by the downhill simplex method is a good approximation.

FIG. 9 is an example of a debanding processed image by optimal parameter learning. Pseudo-color banding image generated by using one of the evaluation images as the original image FIG. 9 (a) The banding extra-trace amount is extracted from FIG. 9 (b), and the optimum parameters of the debanding process are obtained from the result of the optimum parameter learning. I predicted.

FIG. 9C is a composite display of the results of calculating the edge strength γe of the equation (5) in the banding candidate pixel region. With an edge strength gain γ = 5.0, the average value of edge strength with a γe value of 10 or more.

Was 44.968127.

FIG. 9D is a histogram (Histogram) of the luminance value Y in the banding candidate pixel and a binarized luminance value histogram (BinHistogram) thereof. The banding gap BG based on the median values at multiple intervals of 0 values between 1 value and 1 value in the histogram obtained by binarizing 40% of the maximum frequency value as a threshold value was 6. The standard deviation σ _Y of the brightness value was 26.806264, and the eigenvalues λ ₁ and λ ₂ of the variance-covariance matrix of the color difference values were 86.252465 and 1.037179, respectively.

9 (e) and 9 (f) are a debanding processed image with the optimum parameters predicted by the optimum parameter learning and a debanding processed image with the optimum parameters estimated by the downhill simplex method, respectively. The predicted parameter by the optimum parameter learning is (σ _rC , σ _rY ) = (1.848697, 9.340442), and the optimum parameter by the downhill simplex method is (σ _rC , σ _rY ) = (2.215408, 9). It was .954125). The amount of Kullback-Leibler information between the original image was 0.069092 and 0.057761, respectively, and the amount of Kullback-Leibler information between the pseudo-color banding image and the original image was 1.818045.

(5 Summary)
"Color banding", which is a pseudo-contour-like interference generated in HDR video, was corrected by simple image processing. Specifically, the banding candidate pixel was detected from the luminance value in the block territory in the vicinity of the pixel of interest and its standard deviation. Then, in the pixels of the neighboring block region, the banding was removed by weighting and averaging the luminance signals by increasing the weight for the pixels having the same color as possible and having similar brightness.

To automatically optimize the parameters of the debanding process filter
-A pseudo color banding image was generated from multiple images collected from the Internet, and a small amount of banding was extracted by simple image processing.
-For the pseudo-color banding image, the optimum parameters of the debanding processing filter that minimizes the amount of Kullback-Leibler information between the original image and the original image were estimated by the downhill simplex method.
-The optimum model for predicting the optimum parameters of the banding processing filter for the banding limit was determined by multiple regression analysis and variable selection (model selection) using the Akaike Information Criterion (AIC).

As for future issues,
・ Hyperparameters such as block size and regularization parameters in optimization, as well as parameters of debanding processing filter, and automatic optimization of various thresholds for detecting banding candidate pixel area ・ Multi-core CPU / GPU / FPGA Real-time processing of HDR video can be mentioned. It is also interesting to explore the feasibility of debanding processing by deep learning.

(Reference)
[1] G. Baugh, A. Kokaram, and F. Pitie, Advanced videodebanding, Proceedings of the 11th European Conference on Visual Media Production (CVMP'14), London, United Kingdom, (November 2014).
[2] S. Bhagavathy, J. Llach and J. Zhai, Multiscale probabilistic dithering for suppressing contour artifacts in digital images, IEEE Transactions on Image Processing, 18-9 (September 2009), 1936-1945.
[3] T. Borer and A. Cotton, A "display independent" high dynamic range television system, IBC2015 Conference, RAI, Amsterdam, the Netherlands, (September 2015).
[4] G. Csurka, C. Bray, C. Dance, and L. Fan, Visual categorization with bags of keypoints, Proceedings of ECCV Workshop on Statistical Learning in Computer Vision (SLCV), pp. 59-74 (2004).
[5] C. Dong, C.-C. Loy, K. He, and X. Tang, Learning a deep convolutional network for image superresolution, Proceedings of 13th European Conference on Computer Vision (ECCV2014), Part IV, Zurich, Switzerland , pp. 184-199 (September 2014).
[6] D. Eigen, D. Krishnan, and R. Fergus, Restoring an image taken through a window covered with dirt or rain, IEEE International Conference on Computer Vision (ICCV2013), Sydney, Australia, pp. 633-640 (December) 2013).
[7] J. Hays, and AA Efros, Scene completion using millions of photographs, ACM SIGGRAPH (SIGGRAPH '07), New York, NY, USA, Article 4 (August 2007).
[8] M. Holschneider, R. Kronland-Martinet, J. Morlet, and P. Tchamitchian, A real-time algorithm for signal analysis with the help of the wavelet transform, In Wavelets, Time-Frequency Methods and Phase Space, 289 -297, Springer-Verlag, 1989.
[9] V. Jain and HS Seung, Natural image denoising with convolutional networks, Proceedings of Advances in Neural Information Processing Systems 21 (NIPS2008), pp. 769-776 (2008).
[10] Kenichi Kanatani, "Applied Mathematics Classes That You Can Understand: From the Minimum Square Method to Wavelet", Kyoritsu Shuppan, June 2003.
[11] D. Kevin, MK Johnson, K. Sunkavalli, W. Matusik, and H. Pfister, Image restoration using online photo collections, Proceedings IEEE 12th International Conference on Computer Vision (ICCV), Kyoto, Japan, pp. 2217- 2224 (September --October 2009).
[12] A. Krizhevsky, I. Sutskever, and GE Hinton, ImageNet classification with deep convolutional neural networks, Proceedings of Advances in Neural Information Processing Systems 25 (NIPS2012), pp. 1106-1114 (2012).
[13] Y. Kusakabe, Y. Ikeda, N. Shirai, K. Masaoka, T. Yamashita, Y. Nishida, T. Ikeda, and M. Sugawara, Extended image dynamic range system for UHDTV broadcasting, IBC2015 Conference, RAI, Amsterdam, the Netherlands, (September 2015).
[14] D. Lowe, Distinctive image features from scaleinvariant keypoints, International Journal of Computer Vision, 60-2 (January 2004), 91-110.
[15] Riki Matsunaga, Focus correction by convolutional neural network for ultra-high-definition video, Proceedings of the 22nd Image Sensing Symposium (SSII2016), Yokohama (Pacifico Yokohama), June 2016.
[16] Akira Mizuno, Masaki Igarashi, Masayuki Ikebe, Smooth gradation restoration of images using Poisson's equation, Journal of the Institute of Image Information and Television Engineers, 67-8 (2013), J326-J333.
[17] Japan Broadcasting Corporation, "Super Hi-Vision Video Technology" Special Issue, NHK Science & Technical Research Laboratories R & D, No. 137, January 2013.
[18] WH Press, SA Teukolsky, WT Vetterling, and BP Flannery, Numerical Recipes 3rd Edition: The Art of Scientific Computing, Cambridge University Press, September 2007.
[19] Yoshiyuki Sakamoto, Makio Ishiguro, Genshiro Kitagawa, "Information Amount Statistics", Kyoritsu Shuppan, 1983.
[20] C. Tomasi and R. Manduchi, Bilateral filtering forgray and color images, Proceedings of the Sixth IEEE International Conference on Computer Vision (ICCV'98), Bombay, India, (January 1998).
[21] VN Vapnik, Statistical Learning Theory, Wiley, 1998.
[22] L. Xu, JS Ren, C. Liu, and J. Jia, Deep convolutional neural network for image deconvolution, Proceedings of Advances in Neural Information Processing Systems 27 (NIPS2014), pp. 1790-1798 (2014).
[23] LP Yaroslavsky, Digital Picture Processing: An Introduction, Springer Verlag, 1985.

The present invention is also suitable for 4K / 8K (super high-definition) ultra-high-definition video.

Claims (10)

In a color debanding processing device that performs local processing using pixels in a neighboring block area while raster-scanning an image.
A pixel detection unit that detects banding candidate pixels from the standard deviation of the luminance value in the vicinity block area of the pixel of interest, and
A banding removing unit that removes banding by using the block pixels in the vicinity of the banding candidate pixels, increasing the weight for pixels having similar colors as possible and having similar brightness, and performing weighted averaging of luminance signals. A device characterized by being equipped. In the optimum parameter learning device used for color debanding processing
A pseudo color banding image generator that generates a pseudo color banding image from the original image collected from the Internet,
A feature amount extraction unit that extracts a very small amount of banding in the generated pseudo-color banding image, and a feature amount extraction unit.
Optimal parameter estimation to estimate the optimum filter parameter that minimizes the amount of Kullback-Leibler information between the original image without banding and the debanding processed image resulting from the processing of the pseudo-color banding image. Department and
With a multiple regression analysis and an optimum model determination unit that determines the optimum model for predicting the optimum filter parameters for the banding limit by multiple regression analysis and variable selection or model selection using a plurality of original images collected from the Internet. A device characterized by: In an image processing system in which image processing is performed by the color debanding processing apparatus according to claim 1 using the optimum parameters acquired by the optimum parameter learning apparatus according to claim 2.
The banding removing part is
The banding feature amount of the banding candidate pixel is extracted, and the banding feature amount is extracted.
An image processing system characterized in that the optimum parameter corresponding to the extracted banding feature amount is calculated based on the model. In a program that causes a computer to execute a color debanding processing method that performs local processing using pixels in a nearby block area while raster-scanning an image.
A step of detecting a banding candidate pixel from the standard deviation of the luminance value in the neighboring block region of the pixel of interest, and
Using the block pixels in the vicinity of the banding candidate pixels, the step of removing the banding by increasing the weight for the pixels having the same color as possible and having similar brightness and performing the weighted averaging of the luminance signals is executed. A program characterized by that. In a program that causes a computer to execute the optimum parameter learning method used for color debanding processing.
Steps to generate a pseudo color banding image from the original image collected from the internet,
The step of extracting the banding extraordinary amount of the generated pseudo color banding image and
A step of estimating the optimum filter parameter that minimizes the amount of Kullback-Leibler information between the original image in which no banding is present and the debanding processed image obtained by processing the pseudo color banding image.
Perform multiple regression analysis and a step of determining by variable selection or model selection using a plurality of original images collected from the Internet to determine the optimum model for predicting the optimum filter parameter for the banding limit. A program characterized by letting you do it. In a program for causing a computer to perform image processing by the color debanding processing method according to claim 4, using the optimum parameters acquired by the optimum parameter learning method according to claim 5.
The step of removing the banding is
The step of extracting the banding feature amount of the banding candidate pixel and
A program characterized by further executing a step of calculating the optimum parameter corresponding to the extracted banding feature amount based on the model. A storage medium for storing the program according to any one of claims 4 to 6. The device according to claim 1 or 2.
A device characterized by being a hardware device that processes a baseband video signal, or a device that converts or reverses an MXF file into a baseband video signal. The image processing system according to claim 3 is a system characterized in that it is a hardware system that processes a baseband video signal. The program according to any one of claims 4 to 6 is
A program characterized by being a software program that processes MXF files.

Images