JP7303661B2 - Image processing device and program - Google Patents

Description

The present invention relates to an image processing apparatus and program for inputting a moving image and performing image processing using wavelet degeneration based on wavelet packet decomposition.

Conventionally, as a technique for removing noise by inputting a moving image and using wavelet degeneration based on wavelet packet decomposition, among the frequency decomposition coefficients obtained by wavelet transforming the input image, noise is generated by coring the components below a predetermined noise level. It is known to remove (see, for example, Non-Patent Document 1).

Also, as an image processing device using wavelet degeneration, a high-frequency noise image and a low-frequency noise image are extracted, and these images are divided and integrated according to whether they are images of luminance components or images of color difference components. A technique for reducing the noise level is known (see Patent Document 1, for example).

Patent No. 5352942 specification

David L. Donoho, Iain M. Johnstone, Gerard Kerkyacharian and Dominique Picard: “Wavelet Shrinkage: Asymptopia?”, Journal of the Royal Statistical Society. Series B (Methodological), Vol. 57, No. 2, pp. 301-369 , 1995.

As described above, techniques for removing noise from an image using wavelet degeneration are conventionally known, such as in Non-Patent Document 1 and Patent Document 1. There are also techniques for bandlimiting using wavelet shrinkage.

On the other hand, in recent years, for example, as next-generation terrestrial broadcasting, transmission of large-capacity content services such as 4K and 8K SHV (Super Hi-Vision) as ultra-high-definition moving images exceeding high-definition has been considered. When performing high-compression encoding for the transmission of such ultra-high-definition video, it is assumed that video images that are difficult to high-compression-encode due to reasons such as the power of spatial high-frequency band components being large may be input. be. In this case, since the encoding bit rate may not be sufficient for the enormous amount of information that the input moving image has as transmission capacity, image corruption may occur due to encoding related to compression or transmission.

Therefore, prior to high-compression encoding processing for ultra-high-definition video, conventional noise removal processing and band-limiting processing such as those described in Non-Patent Document 1 and Patent Document 1 are performed to reduce the amount of information contained in the input video. can do.

However, when performing noise removal processing and band-limiting processing using conventional technology, there are various linear filter processing and nonlinear Fvarta processing. There is a problem that it is not possible to adaptively perform noise reduction processing and band limiting processing as preprocessing of compression coding processing so as to suppress image corruption caused by coding related to transmission and prevent deterioration of moving image quality. be.

For this reason, when reducing the amount of information by using wavelet degeneracy in the preceding stage of compression encoding processing for moving images such as ultra-high-definition moving images, noise removal processing is performed so as not to deteriorate the quality of the moving image while suppressing image corruption. and filter processing (degeneration function used for wavelet degeneration) that performs band-limiting processing is desired.

Therefore, an object of the present invention is to input a moving image as preprocessing for compression encoding processing, and apply wavelet degeneration based on wavelet packet decomposition to this input moving image to suppress image corruption and prevent deterioration of moving image quality. It is an object of the present invention to provide an image processing apparatus and a program that perform image processing that adaptively performs noise removal processing and band limiting processing as described above, and that improve image quality related to compression encoding processing of moving images.

An image processing apparatus according to one aspect of the present invention predicts the encoding difficulty of an input moving image by spatio-temporal analysis when noise removal and band limitation are performed on the input moving image before compression encoding processing of the moving image. Meanwhile, a degeneration function for performing noise removal and band limitation for each spatial phase position in the frequency domain in each frame image of the input moving image is set, and spatial wavelet degeneration processing based on spatial wavelet packet decomposition is performed. The degeneracy function is based on the magnitude of temporal power fluctuation for each spatial phase position, and more preferably, the magnitude of temporal power fluctuation and the temporal power corresponding to the reference picture set related to the subsequent compression encoding process. Noise removal and band limitation are performed based on the degree of matching (time power matching amount).

That is, the image processing apparatus of the present invention is an image processing apparatus that receives a moving image and performs image processing using wavelet degeneration based on wavelet packet decomposition. a spatial wavelet packet decomposition unit that performs spatial wavelet packet decomposition processing of (n≧2) and extracts frequency decomposition coefficients of each frequency band group in the spatial direction; The frequency resolution coefficient of the highest spatial diagonal frequency band is extracted, and the power exceeding a predetermined ratio in the power of all frequency resolution coefficients in the spatial diagonal highest frequency band is regarded as a noise component and detected as noise power. A power detection unit, based on each frequency band group in the spatial direction related to the picture to be processed, detects a picture at a corresponding spatial position in one frame image immediately adjacent to the frame image to which the picture to be processed belongs. a temporal power variation detection unit that performs first-order wavelet packet decomposition in the temporal direction using a temporal power variation detection unit that detects the temporal power variation for each spatial phase position of the picture to be processed; Based on the temporal power fluctuation amount for each spatial phase position, the noise power defines the input value range of the degeneracy function for noise removal, and the larger the temporal power fluctuation amount, the more the degeneracy corresponds to the band limit amount. A degeneracy function setting unit that sets a degeneracy function for each spatial phase position in all frequency band groups related to the picture to be processed, and a frequency decomposition of each frequency band group related to the picture to be processed, as a function that reduces the slope of the function. a spatial wavelet degeneration unit that performs spatial wavelet degeneration processing based on the spatial wavelet packet decomposition processing by applying the degeneration function to the coefficients and performing the spatial wavelet degeneration processing; and the spatial wavelet degeneration unit for the picture to be processed. Performing a spatial wavelet reconstruction process using the frequency resolution coefficients of each frequency band band-limited and noise-removed by the process to generate a band-limited and noise-removed output video for each picture in the video. and a reconstructing unit.

Further, in the image processing apparatus of the present invention, the temporal power fluctuation amount detection unit detects the temporal power fluctuation for each spatial phase position regarding the picture to be processed based on a block of a predetermined block size centered at each spatial phase position. It is characterized by detecting quantity.

Further, in the image processing apparatus of the present invention, one or more reference pictures to be encoded-referenced to the picture to be processed are selected based on the information of the reference picture set related to the predetermined compression encoding process. a second spatial wavelet packet for generating a reference picture represented by frequency resolution coefficients of each frequency band group in the spatial direction by performing n-th order (n≧2) spatial wavelet packet decomposition processing on the reference picture A decomposing unit, for each spatial phase position in each spatial direction frequency band group regarding the picture to be processed, for each block of a predetermined block size centered at each spatial phase position, for each spatial direction frequency band group. Perform block matching with respect to the reference picture represented by the frequency resolution coefficient, search for the matching amount showing the minimum similarity evaluation index, and perform temporal power matching for each spatial phase position on the picture to be processed for the minimum matching amount. a temporal power matching amount detection unit for detecting as a quantity, wherein the degeneracy function setting unit detects the noise power for the picture to be processed, the temporal power fluctuation amount for each spatial phase position, and each spatial phase position Based on the time power matching amount, the noise power defines the input value range of the degeneracy function for noise removal, and the larger the value of the time power fluctuation amount, the more the slope of the degeneracy function corresponding to the band limit amount. Assuming that the greater the value of the temporal power matching amount, the smaller the slope of the degeneracy function corresponding to the band limit amount, degeneracy is performed for each spatial phase position in all frequency band groups related to the picture to be processed. It is characterized by setting a function.

Furthermore, the program of the present invention constitutes a program for causing a computer to function as the image processing apparatus of the present invention.

According to the present invention, by using the image processing apparatus according to the present invention as pre-processing for compression encoding processing of a moving image, while predicting the encoding difficulty of the input moving image by spatio-temporal analysis, input moving image Since it is possible to adaptively set a degeneration function that performs noise removal and band limitation for each spatial phase position in the frequency domain in each frame image of the image, it is possible to suppress image corruption in the subsequent compression encoding process, It is possible to improve the video coding quality.

1 is a block diagram showing a schematic configuration of an image processing apparatus according to one embodiment of the present invention; FIG. FIG. 4 is a diagram showing frequency bands obtained by second-order spatial wavelet packet decomposition related to image processing of one example in the image processing apparatus of one embodiment according to the present invention; 4 is a flow chart showing image processing of an example in the image processing apparatus of one embodiment according to the present invention; FIG. 5 is an explanatory diagram showing detection of temporal power variation in the image processing apparatus according to one embodiment of the present invention; FIG. 5 is an explanatory diagram showing detection of a temporal power matching amount in the image processing apparatus of one embodiment according to the present invention; (a) is a diagram illustrating a reference picture set, and (b) and (c) are explanatory diagrams illustrating block matching related to temporal power matching amount detection in an image processing apparatus according to an embodiment of the present invention. be. 4(a) and 4(b) are explanatory diagrams of degeneration functions related to image processing of an example and a modified example in the image processing apparatus of one embodiment according to the present invention, respectively. FIG. 4 is a diagram showing frequency bands obtained by third-order spatial wavelet packet decomposition related to image processing of one example in the image processing apparatus of one embodiment according to the present invention;

An image processing apparatus 1 according to an embodiment of the present invention will be described in detail below with reference to the drawings.

〔Device configuration〕
FIG. 1 is a block diagram showing a schematic configuration of an image processing apparatus 1 of one embodiment according to the present invention. The image processing apparatus 1 of this embodiment is configured as a device that performs image processing using spatial wavelet degeneracy based on spatial wavelet packet decomposition by inputting a moving image as preprocessing for compression encoding processing, and includes a spatial wavelet packet decomposition unit 11, a noise power detector 12, a temporal power variation detector 13, a spatial wavelet packet decomposing unit 14, a temporal power matching amount detector 15, a degeneration function setting unit 16, a spatial wavelet degeneracy unit 17, and a reconstruction unit 18. . In the example shown in FIG. 1, the wavelet packet decomposition order number (nth order) can be externally specified for the image processing apparatus 1, and each necessary component in the image processing apparatus 1 can be specified externally. , it is also possible to have a configuration in which processing is performed while being transmitted as parameters.

The spatial wavelet packet decomposition unit 11 receives a picture P(t) to be processed in the input moving image, and according to the set wavelet packet decomposition order (nth order) (n≧2), the picture to be processed in the input moving image is Spatial wavelet packet decomposition processing is performed on P(t) (t: time of frame image to be processed), and the frequency decomposition coefficients of each frequency band group in the spatial direction extracted by this are output to the spatial wavelet reduction unit 17. do. For the picture P(t) to be processed, the noise power detector 12 extracts the frequency resolution coefficients within the spatial diagonal maximum frequency band among the spatial direction frequency bands obtained by the n-th order spatial wavelet packet decomposition. be. Further, the frequency resolution coefficients of each frequency band group in the spatial direction obtained by the n-th order spatial wavelet packet decomposition for the picture P(t) to be processed are obtained by the temporal power fluctuation amount detection unit 13 and the temporal power matching amount detection unit 15. , they are used to detect a temporal power fluctuation amount and a temporal power matching amount, which will be described later in detail.

Spatial wavelet packet decomposition for a picture P(t) to be processed will be described more specifically with reference to FIG. FIG. 2 is a diagram showing frequency bands obtained by second-order spatial wavelet packet decomposition for a picture P(t) to be processed according to one example of image processing in the image processing apparatus 1 according to one embodiment of the present invention. This is an example in which a so-called 8K/60P moving image (horizontal resolution 8K (7680 pixels)×vertical resolution 4K (4320 pixels)) captured by a camera is processed. In this example, the entire frame image at time t is the picture P(t), but a slice image or tile image (HEVC standard) showing part of the frame image at time t can be treated as the picture P(t). .

As shown in FIG. 2, in the second-order spatial wavelet packet decomposition in which only the spatial direction is processed with n=2 for the picture P(t) to be processed in the input video, the picture P(t) to be processed in the input video Second-order spatial wavelet packet decomposition is performed on (t) in the horizontal and vertical directions, respectively. In this case, a picture P'(i,j,t) decomposed into the frequency domain as shown in FIG. 2 represented by horizontal frequency*vertical frequency is obtained. , the second-order spatial low-frequency band LL ^m (i,j,t), the horizontal high-frequency band LH ^m (i,j,t), the vertical high-frequency band HL ^m (i,j,t), and the diagonal ( (horizontal/vertical) high frequency band HH ^m (i, j, t) is decomposed into each frequency band group (m is the group number of each frequency band group, i is the spatial phase coordinate of the horizontal frequency resolution coefficient and j indicates the spatial phase coordinate of the frequency-resolved coefficients in the vertical direction). Parseval's law holds between each level in the n-level spatial wavelet Hackett decomposition.

As shown in FIG. 1, the noise power detection unit 12 detects each frequency band of the picture P′(i, j, t), which is the output value from the spatial wavelet packet decomposition unit 11, according to the wavelet packet decomposition order (nth order). A component (frequency resolution coefficient) of the spatial diagonal maximum frequency band (in this example, n=2, space HH ⁴ (i, j, t)) is extracted from the group. Then, the noise power detection unit 12 detects the power (upper %th power) is regarded as a noise component, detected as a noise power _PN , and output to the degeneracy function setting unit 16 .

The temporal power fluctuation amount detection unit 13 is based on each frequency band group of the picture P′ (i, j, t), which is the output value from the spatial wavelet packet decomposition unit 11, according to the wavelet packet decomposition order (nth order), Using a temporal wavelet filter, a picture P(ta) (a is the time t For example, when a = 1, the picture P (t-1) at the corresponding spatial position in the previous frame image is shown.) is used to perform first-order wavelet packet decomposition in the temporal direction, Obtain the frequency resolution coefficients of each frequency band group of the picture P'(i,j,ta). Subsequently, the temporal power fluctuation amount detection unit 13 corresponds to each frequency band group of the picture P′(i, j, t) and each frequency band group of the picture P′(i, j, t−a). At the spatial phase position (position (i, j) indicating the spatial phase of the frequency resolution coefficient in each frequency band), the externally specified block size {Bx, By }, the average value or median value in the block area is obtained as the power of the spatial phase position, and the picture P'(i, j, t) and the picture P'(i, j, t-a) The value obtained by subtracting the power at each spatial phase position (i, j) between the two is detected as the temporal power fluctuation amount for each spatial phase position (i, j) in the picture P′(i, j, t), and the degeneracy function Output to the setting unit 16 . When detecting the temporal power fluctuation amount for each spatial phase position (i, j), the accuracy of the power value and the accuracy of the set value of the degeneration function based on this are increased by processing in the block area as described above. .

The spatial wavelet packet decomposing unit 14 externally designates reference picture set (RPS) information used by a subsequent encoding device (not shown) that performs compression encoding processing on the output moving image of the image processing device 1 of this embodiment. be done. Then, based on the information of this reference picture set (RPS), the spatial wavelet packet decomposition unit 14 extracts one or more pictures P(t) to be processed in the input moving picture to be referenced for encoding. input the reference picture P(k), and perform n-th order spatial wavelet packet decomposition processing on the reference picture P(k) according to the set wavelet packet decomposition order (nth order) (n≧2), The frequency resolution coefficients of each frequency band group in the spatial direction thus obtained are output to the temporal power matching amount detection unit 15 . For example, the spatial wavelet packet decomposing unit 14, regarding the reference picture P(k) in the encoding order one before the picture P(t) to be processed, which is determined based on the information of the reference picture set (RPS), The frequency decomposition coefficients of the frequency band group obtained by applying the second-order spatial wavelet packet decomposition are output to the temporal power matching amount detection unit 15 .

The temporal power matching amount detection unit 15 detects each spatial For the phase position (i, j), the output value from the spatial wavelet packet decomposition unit 14 for each block of the externally specified block size {Bx, By} centered on each spatial phase position (i, j) Perform block matching for a certain reference picture P′ (i, j, k), search for the matching amount that indicates the minimum similarity evaluation index, and find the minimum matching amount in the picture P′ (i, j, t) A temporal power matching amount for each spatial phase position (i, j) is detected and output to the degenerate function setting unit 16 . Here, the similarity evaluation index may be an SSD (Sum of Squared Difference) value, or, in addition to the SSD value, an SAD (Sum of Absolute Difference) value, NCC (Normalized Cross-Correlation), ZNCC (Zero-means Normalized Cross-Correlation) etc. can be used. In detecting the temporal power matching amount for each spatial phase position (i, j), processing in the block region as described above increases the accuracy of the matching amount and the set value of the degeneracy function based thereon. .

The degeneracy function setting unit 16 sets the noise power P _N in the picture P′(i,j,t) obtained from the noise power detection unit 12 and the picture P′(i,j,t) obtained from the temporal power fluctuation amount detection unit 13 . t) for each spatial phase position (i, j) and for each spatial phase position (i, j) for picture P′ (i, j, t) obtained from the temporal power matching amount detection unit 15 Based on the amount of temporal power matching, a degeneration function to be used in the spatial wavelet degeneration unit 17 is set for each spatial phase position (i, j) in all frequency band groups in the picture P'(i, j, t). In setting the degeneracy function in the degeneracy function setting unit 16, the input value range of the degeneracy function for noise removal is defined for the noise power _PN , and the larger the value of the time power fluctuation amount, the degeneracy function corresponding to the band limit amount. Further, the larger the value of the time power matching amount, the smaller the slope of the degeneracy function corresponding to the band limiting amount.

The spatial wavelet degeneracy unit 17 applies the degeneracy function set by the degeneracy function setting unit 16 to the frequency resolution coefficients of each frequency band group in the picture P′(i, j, t) input from the spatial wavelet packet decomposer 11. is applied to perform the spatial wavelet reduction process. As a result, the spatial wavelet degeneracy unit 17 predicts the encoding difficulty of the input moving image by spatio-temporal analysis, and for each spatial phase position in the frequency domain in each frame image (each picture to be exact) of the input moving image. Adaptively sets a degeneration function that performs noise removal and band limitation to the spatial wavelet degeneration process based on spatial wavelet packet decomposition, and reconstructs the frequency decomposition coefficients in each frequency band after band limitation and noise removal Output to 18.

The reconstruction unit 18 uses the frequency decomposition coefficients of each frequency band group in the picture P′(i, j, t) band-limited and noise-removed by the spatial wavelet degeneration process according to the wavelet packet decomposition order (nth order). By performing spatial wavelet reconstruction processing and similarly processing each picture in the input moving image, a band-limited and noise-removed output moving image is generated and output to the outside.

[Device operation]
Hereinafter, as the operation of the image processing apparatus 1 according to the present embodiment shown in FIGS. 1 and 2, image processing according to an embodiment will be described with reference to FIGS. 4 to 6 and based on FIG. .

FIG. 3 is a flow chart showing an example of image processing in the image processing apparatus 1 of one embodiment according to the present invention. 4 is an explanatory diagram showing detection of the temporal power fluctuation amount in the image processing apparatus 1 of this embodiment, and FIG. 5 is an explanatory diagram showing detection of the temporal power matching amount in the image processing apparatus 1 of this embodiment. be. FIG. 6(a) is a diagram illustrating a reference picture set, and FIGS. 6(b) and 6(c) respectively illustrate block matching related to detection of the temporal power matching amount in the image processing apparatus 1 of this embodiment. It is an explanatory diagram for.

First, the image processing apparatus 1 inputs a picture P(t) to be processed in the input moving image by the spatial wavelet packet decomposition unit 11, and according to the set wavelet packet decomposition order (n order) (n≧2), A picture P(t) to be processed in an input moving image is subjected to spatial wavelet packet decomposition processing, and frequency decomposition coefficients of each frequency band group in the spatial direction obtained by this processing are obtained (step S1).

Subsequently, the image processing apparatus 1 causes the noise power detection unit 12 to detect the highest spatial diagonal frequency band (n=2 in this example) among the frequency band groups of the picture P′(i, j, t) shown in FIG. , the components (frequency resolution coefficients) of the space HH ⁴ (i, j, t)) are extracted, and all components (all frequencies The power that is equal to or higher than a predetermined ratio in the power of the resolution coefficient) (upper α%-th power) is regarded as a noise component and detected as a noise power _PN (step S2).

In general, the power of signal components is unevenly distributed in the low frequency band. On the other hand, the power of noise components such as thermal noise is almost constant over the entire frequency band group. For this reason, noise components are dominant in the components within the spatial diagonal maximum frequency band due to the n-th order spatial wavelet packet decomposition. Therefore, the noise power detection unit 12 sorts the power of all components (all frequency resolution coefficients) in the spatial diagonal highest frequency band, and if α=95, for example, the top 95% value is the noise power P _N and

Subsequently, the image processing apparatus 1 causes the temporal power fluctuation amount detection unit 13 to use the temporal wavelet filter F based on each frequency band group of the picture P'(i, j, t) to , j, t) to which the picture P' ( Obtain the frequency resolution coefficients of each frequency band group of i, j, ta).

For example, when a=1, the picture P(t− For 1), a picture P'(i, j, t-1) is obtained that is frequency-resolved in the temporal direction at the spatial phase position with reference to each frequency band group of the picture P'(i, j, t). That is, if the temporal wavelet filter F that performs first-order wavelet packet decomposition is a Haar wavelet filter, its tap length is 2, and as shown in FIG. At the corresponding spatial phase position, a temporally frequency-resolved picture P'(i,j,t-1) is obtained.

Then, the temporal power fluctuation amount detection unit 13 corresponds to each frequency band group of the picture P′(i, j, t) and each frequency band group of the picture P′(i, j, t−a). At the spatial phase position (i, j), the average value or the median value in the block area of the externally specified block size {Bx, By} centered at the spatial phase position (i, j) The power of each spatial phase position (i, j) between the picture P' (i, j, t) and the picture P' (i, j, t-a) is obtained by subtracting the power of the picture P' It is detected as a temporal power fluctuation amount for each spatial phase position (i, j) at (i, j, t) (step S3).

For example, as illustrated in FIG. 4, the temporal power fluctuation amount detection unit ¹³ detects a block size When 3×3 {Bx=3, By=3}, the power indicating the average or median value in the block area of the picture P′(i, j, t) at the spatial phase position (i, j); The difference between the power indicating the average value or the median value in the block area of the picture P'(i, j, t) at the same spatial phase position (i, j) is It is detected as the temporal power fluctuation amount P _HL ³ (i, j, t) of the spatial phase position (i, j). In this way, the temporal power fluctuation amount detection unit 13 detects the temporal power fluctuation amount at all spatial phase positions (i, j) in all frequency bands.

Subsequently, when the spatial wavelet packet decomposition unit 14 externally designates the information of the reference picture set (RPS) used in the subsequent encoding device (not shown) that performs the compression encoding process, the image processing device 1 specifies this information. inputting one or more reference pictures P(k) to be referenced for encoding with respect to the picture P(t) to be processed in the input video based on the information of the reference picture set (RPS); The reference picture P(k) is subjected to n-th order spatial wavelet packet decomposition processing, and the reference picture P′(i,j,k) represented by the frequency decomposition coefficients of each frequency band group in the spatial direction obtained by this to generate

For example, as illustrated in FIG. 5, the spatial wavelet packet decomposing unit 14 uses the coding order of the picture P(t) to be processed, which is determined based on the information of the reference picture set (RPS). For the reference picture P(k), a reference picture P'(i,j,k) represented by the frequency decomposition coefficients of the frequency band group obtained by performing second-order spatial wavelet packet decomposition is generated.

Subsequently, the image processing apparatus 1 causes the temporal power matching amount detection unit 15 to detect the picture P′(i, j, t ), for each block of externally specified block size {Bx,By} centered at that spatial phase position (i,j), spatial wavelet Block matching is performed with respect to the reference picture P′ (i, j, k), which is the output value from the packet decomposing unit 14, the matching amount indicating the minimum similarity evaluation index (for example, SSD value) is searched, and the minimum is detected as the temporal power matching amount for each spatial phase position (i, j) in the picture P'(i, j, t) (step S4).

For example, as exemplified in FIG. ⁵ , the temporal power matching amount detection unit 15 detects a block size Assuming 3×3 {Bx=3, By=3}, the spatial phase position (i, j) of the picture P′(i, j, t) is determined based on the information of the reference picture set (RPS) Block matching is performed with respect to the reference picture P'(i, j, k), the matching amount indicating the minimum similarity evaluation index (for example, SSD value) is searched, and the minimum matching amount is obtained from the picture P'(i , j, t) at the _spatial ^phase position (i, j). Thus, the temporal power matching amount detection unit 15 detects the temporal power matching amount at all spatial phase positions (i, j) in all frequency bands.

Note that reference picture sets and block matching related to detection of the temporal power matching amount in the image processing apparatus 1 of the present embodiment will be described with reference to FIG. As a moving image compression encoding technique, for example, H.264. 264/MPEG-4 AVC, H.264/MPEG-4 AVC. H.265/MPEG-H HEVC, etc., a predetermined group is set by dividing a moving image before predictive coding processing into a plurality of screen groups, and within the predetermined group, an inter-screen predictive code selected in advance as a predictive coding method is applied. encoding processing and intra-screen predictive coding processing.

H. In H.264/MPEG-4 AVC, a GOP (Group of Pictures) obtained by dividing a moving image before predictive coding into a plurality of screen groups is set as the predetermined group. Then, for each screen (frame or slice screen) in the GOP, an I picture for which intra-frame predictive coding processing is performed, a P picture for which unidirectional inter-frame predictive coding processing is performed from a temporally past screen, or a temporally A B-picture to be subjected to bidirectional inter-frame predictive coding processing is selected in advance from the screens located before and after the current picture. In addition, H. In H.264/MPEG-4 AVC, all pictures in a GOP can be compressed by a compression coding method that only performs intra-picture predictive coding.

Also, H.I. Also in H.265/MPEG-H HEVC, a sequence obtained by dividing a moving image before predictive coding into a plurality of screen groups is set as the predetermined group. Although the HEVC standard does not introduce the concept of GOP, this sequence corresponds to GOP. Then, for each screen (frame screen, slice screen, or tile screen) in the sequence, a processing unit that performs intra-screen predictive encoding processing and inter-screen predictive encoding processing is selected in advance. In addition, H. Also in H.265/MPEG-H HEVC, all the pictures in the sequence can be compressed by the compression coding method of only the intra-picture predictive coding process.

Then, as illustrated in FIG. 6A, in the moving image to be compression-encoded, each frame image indicated by its frame number (here, the entire frame image is exemplified as a picture) refers to any picture. It is determined in advance whether compression encoding is performed as a set, and can be identified as a reference picture set.

Therefore, the image processing apparatus 1 uses the temporal power matching amount detection unit 15 to determine the block size {Bx, By }, for example, for one reference picture P'(i,j,k), as shown in FIG. For P'(i,j,k2), block matching is performed within a predetermined search range as shown in FIG. SSD1, SSD2, . . . ), the smallest matching amount is detected as the temporal power matching amount. Note that when the reference picture P'(i, j, k) in the reference picture set corresponds to the I picture of the picture P'(i, j, t) to be processed, the temporal power matching amount detection unit 15 The processing can be omitted or configured to treat the I picture in the previous reference picture set or the last picture in coding order as the reference picture P'(i,j,k).

Subsequently, the image processing apparatus 1 causes the degeneration function setting unit 16 to set the noise power P _N in the picture P′(i, j, t) obtained from the noise power detection unit 12 and the noise power P N obtained from the temporal power fluctuation amount detection unit 13 . The amount of temporal power variation for each spatial phase position (i, j) in the picture P′(i, j, t) obtained from the obtained picture P′(i, j, t) and the space in the picture P′(i, j, t) obtained from the temporal power matching amount detection unit 15 Based on the amount of temporal power matching for each phase position (i, j), for each spatial phase position (i, j) in all frequency band groups in the picture P' (i, j, t), a spatial wavelet degeneration unit 17 is set (step S5).

In particular, the degeneracy function setting unit 16 defines the input value range of the degeneracy function for noise removal by the noise power _PN , and the larger the value of the time power fluctuation amount, the more the slope of the degeneracy function corresponding to the band limit amount. Further, the degeneracy function is set so that the larger the value of the time power matching amount, the smaller the slope of the degeneracy function corresponding to the band limit amount.

For example, FIG. 7A is an explanatory diagram of a degeneration function related to image processing of one example in the image processing apparatus 1 of this embodiment. The degeneracy function indicated by the output y for the input x shown in FIG. 7(a) defines the input value range of the degeneracy function so that the input value determined below the absolute value of the noise power _PN is denoised to zero value. , for an input value exceeding the absolute value of the noise power P _N , the temporal power fluctuation amount P _HL ³ (i, j, t) at the spatial phase position (i, j) shown in FIG. The band limit is determined by the temporal power matching amount M _HL ³ (i, j, t) at the same spatial phase position (i, j) shown in , and the degenerate function corresponds to the band limiting amount as the value of the temporal power fluctuation amount increases. Further, the slope of the degeneracy function corresponding to the band limitation amount is made smaller as the value of the amount of time power matching becomes larger. This is because it is considered that the larger the value of the temporal power fluctuation amount, and furthermore the larger the value of the temporal power matching amount, the more difficult compression encoding becomes and the more likely image deterioration due to encoding occurs. β and γ are constants that are determined arbitrarily.

(Modification)
In the example of the embodiment described above, an example of setting the degeneracy function using the noise power, the amount of temporal power fluctuation, and the amount of temporal power matching has been described. and the temporal power matching amount detection unit 15 may be omitted, and the degeneracy function may be set using the noise power and the temporal power fluctuation amount. The degeneration function of this modification is illustrated in FIG. 7(b). In the example shown in FIG. 7B as well, the slope of the degeneracy function is made smaller as the value of the temporal power fluctuation amount increases. It is possible to suppress the phenomenon in which image deterioration is likely to occur.

Subsequently, the spatial wavelet degeneracy unit 17 of the image processing apparatus 1 applies the degeneracy function By executing spatial wavelet degeneration processing by applying the degeneration function set by the setting unit 16, each frame image of the input video (accurate , adaptively sets a degeneration function for noise removal and band limitation for each spatial phase position in the frequency domain in each picture), and performs spatial wavelet degeneration processing based on spatial wavelet packet decomposition (step S6).

Finally, the image processing apparatus 1 performs frequency decomposition of each frequency band group in the picture P′(i,j,t) band-limited and noise-removed by the spatial wavelet degeneration process according to the wavelet packet decomposition order (nth order). Spatial wavelet reconstruction processing is performed using the coefficients, and each picture in the input moving image is similarly processed to generate a band-limited and noise-removed output moving image and output it to the outside (step S7).

As described above, the image processing apparatus 1 of the present embodiment is used as a pre-process for compressing and encoding a moving image, and predicts the encoding difficulty of the input moving image by spatio-temporal analysis. Since it is possible to adaptively set a degeneration function that performs noise removal and band limitation for each spatial phase position in the frequency domain in each frame image, it is possible to suppress image corruption in the subsequent compression encoding process. Image coding quality can be improved.

(Application example)
In the above-described example of the embodiment, the spatial wavelet packet decomposing unit 11 and the spatial wavelet packet decomposing unit 14 mainly explained an example when operating in second-order spatial wavelet packet decomposition with n=2, but n≧3. can also be For example, as shown in FIG. 8, it is also possible to operate with a 3rd order spatial wavelet packet decomposition with n=3. FIG. 8 shows an example in which a so-called 8K/60P moving image (horizontal resolution 8K (7680 pixels)×vertical resolution 4K (4320 pixels)) captured by a broadcast camera is the object of processing. In the example shown in FIG. 8, the noise power detection unit 12 detects the spatial diagonal highest frequency band (n=3 in this example) among the frequency band groups of the picture P′(i, j, t). Noise power is detected based on the component (frequency resolution coefficient) of HH ¹⁶ (i, j, t)). In the image processing device 1 using third-order spatial wavelet packet decomposition, which targets the spatial direction with n = 3, the noise power, the temporal power fluctuation amount, and the temporal power matching amount are set more finely. noise cancellation and bandlimiting can be performed on the

The image processing apparatus 1 in the above embodiment can be configured by a computer, and a program for functioning each processing unit of the image processing apparatus 1 can be preferably used. Specifically, a control unit for controlling each processing unit of the image processing apparatus 1 can be configured by a central processing unit (CPU) in a computer, and a program necessary for operating each processing unit can be At least one memory can be used as a storage unit that stores data as appropriate. That is, the functions of each processing unit of the image processing apparatus 1 can be realized by causing the CPU of such a computer to execute the program. Furthermore, a program for realizing the function of each processing unit of the image processing apparatus 1 can be stored in a predetermined area of the storage unit (memory) described above. Such a storage unit can be configured with a RAM or ROM inside the device, or can be configured with an external storage device (eg, hard disk). Also, such a program can be made up of a part of software (stored in a ROM or an external storage device) on an OS used in a computer. Furthermore, a program for causing such a computer to function as each processing unit of the image processing apparatus 1 can be recorded on a computer-readable recording medium. Also, each processing unit of the image processing apparatus 1 can be configured as a part of hardware or software, and can be realized by combining them.

Although the present invention has been described with reference to specific embodiments, the present invention is not limited to the above-described embodiments, and can be modified in various ways without departing from the technical idea thereof. For example, in the examples of the embodiments described above, examples were mainly described in which noise removal and band limitation were performed using spatial wavelet degeneration based on second or third order spatial wavelet packet decomposition. It can also be a process that denoises and bandlimits using spatial wavelet shrinkage based on . Therefore, the image processing apparatus 1 according to the present invention is not limited to the above-described embodiment examples, but is limited only by the description of the claims.

According to the present invention, by using the image processing apparatus according to the present invention as pre-processing for compression encoding processing of a moving image, while predicting the encoding difficulty of the input moving image by spatio-temporal analysis, input moving image It is possible to adaptively set a degeneration function that performs noise removal and band limitation for each spatial phase position in the frequency domain in each frame image of an image, suppressing image corruption in the subsequent compression coding process, and moving image coding. Since it is possible to improve the quality, it is particularly useful for applications requiring video coding.

1 Image processing device 11 Spatial wavelet packet decomposing unit 12 Noise power detecting unit 13 Temporal power fluctuation amount detecting unit 14 Spatial wavelet packet decomposing unit 15 Temporal power matching amount detecting unit 16 Degenerate function setting unit 17 Spatial wavelet degenerating unit 18 Reconstructing unit

Claims (4)

An image processing device for inputting a moving image and performing image processing using wavelet degeneration based on wavelet packet decomposition,
a spatial wavelet packet decomposition unit that inputs a picture to be processed in the moving image, performs n-th order (n≧2) spatial wavelet packet decomposition processing, and extracts frequency decomposition coefficients of each frequency band group in the spatial direction;
extracting a frequency resolution coefficient of a spatial diagonal maximum frequency band from among the spatial direction frequency band groups for the picture to be processed, and extracting a predetermined ratio of the power of all frequency resolution coefficients in the spatial diagonal maximum frequency band; a noise power detection unit that detects power equal to or higher than the noise component as noise power;
Based on each frequency band group in the spatial direction related to the picture to be processed, in the temporal direction using a picture at a corresponding spatial position in a frame image immediately adjacent to the frame image to which the picture to be processed belongs a temporal power fluctuation amount detection unit that performs first-order wavelet packet decomposition and detects the temporal power fluctuation amount for each spatial phase position regarding the picture to be processed;
Based on the noise power for the picture to be processed and the temporal power fluctuation amount for each spatial phase position, the noise power shall define the input value range of the degeneration function for noise removal, and the temporal power fluctuation amount shall be the a degeneracy function setting unit that sets a degeneracy function for each spatial phase position in all frequency band groups related to the picture to be processed so that the slope of the degeneracy function corresponding to the amount of band limitation decreases as the value increases;
Spatial wavelet degeneration that performs spatial wavelet degeneration processing based on the spatial wavelet packet decomposition processing by applying the degeneration function to the frequency resolution coefficients of each frequency band group related to the picture to be processed and executing the spatial wavelet degeneration processing. Department and
By performing spatial wavelet reconstruction processing using frequency resolution coefficients of each frequency band group band-limited and noise-removed by the spatial wavelet degeneration processing for the picture to be processed, band-limiting and noise-removing for each picture in the moving image a reconstructor that generates a denoised output video image;
An image processing device comprising: The temporal power fluctuation amount detection unit detects the temporal power fluctuation amount for each spatial phase position regarding the picture to be processed based on a block of a predetermined block size centered on each spatial phase position, The image processing apparatus according to claim 1. Based on the information of the reference picture set related to the predetermined compression encoding process, one or more reference pictures to be encoded and referenced for the picture to be processed are input, and the n-th order is applied to the reference picture. a second spatial wavelet packet decomposition unit that performs (n≧2) spatial wavelet packet decomposition processing and generates a reference picture represented by frequency decomposition coefficients of each frequency band group in the spatial direction;
For each spatial phase position in each frequency band group in the spatial direction regarding the picture to be processed, for each block of a predetermined block size centered at each spatial phase position, with frequency resolution coefficients of each frequency band group in the spatial direction Perform block matching on the represented reference picture, search for the matching amount showing the minimum similarity evaluation index, and detect the minimum matching amount as the temporal power matching amount for each spatial phase position regarding the picture to be processed. and a time power matching amount detection unit,
The degeneracy function setting unit removes noise from the noise power based on the noise power for the picture to be processed, the temporal power fluctuation amount for each spatial phase position, and the temporal power matching amount for each spatial phase position. The input value range of the degeneracy function is defined, and the greater the value of the temporal power fluctuation amount, the smaller the slope of the degeneracy function corresponding to the band limitation amount. 3. The method according to claim 1 or 2, wherein a degeneration function is set for each spatial phase position in all frequency band groups related to the picture to be processed, as a slope of the degeneration function corresponding to the amount of band limitation is reduced. The described image processing device. A program for causing a computer to function as the image processing apparatus according to any one of claims 1 to 3.

Images