JP4843528B2 - Moving image space resolution enlargement apparatus and moving image space resolution enlargement program - Google Patents

Description

  The present invention relates to a moving image spatial resolution enlarging apparatus and a moving image spatial resolution enlarging program, and in particular, effectively compensates for a spatial high-frequency component of a moving region on a moving image frame lost due to an integration effect in a storage-type image sensor. In addition, the present invention relates to a moving image spatial resolution enlargement apparatus and a moving image spatial resolution enlargement program capable of performing natural spatial resolution enlargement using a contrast sensitivity function related to human visual characteristics.

  In recent years, with the widespread use of digital high-definition televisions, for example, conventional NTSC (National Television System Committee) displays low-resolution moving images on high-resolution digital high-definition televisions. At this time, in order to perform video format conversion from a video format having a low spatial resolution to a video format having a high spatial resolution, a technique for expanding the spatial resolution of moving images is required.

  As a conventional spatial resolution enlarging method, an image enlarging method for still images has already been proposed (for example, see Non-Patent Document 1).

  The image enlarging method disclosed in Non-Patent Document 1 is an n-th order two-dimensional discrete wavelet decomposition of an input image, and the horizontal, vertical and diagonal high-frequency components of the n-th order wavelet decomposition component, and the horizontal of the n + 1-th order wavelet decomposition component, Using the vertical and diagonal high-frequency components, the horizontal, vertical, and diagonal high-frequency components of the unknown n−1-th wavelet decomposition component that is a component of the enlarged image are estimated.

Also, an image enlargement method for moving images has already been proposed (see, for example, Patent Document 1).
The image enlargement method disclosed in Patent Document 1 detects an edge from a high-frequency component extracted by performing multi-resolution analysis on an input image using wavelet transform, performs frequency analysis on the edge, and displays the edge on the enlarged image. By generating the image, an enlarged image having both sharpness and texture can be obtained.
Japanese Patent Laying-Open No. 2005-011191 ([0018], FIG. 1) Akira Tanaka, 3 others, “Enlargement of digital images using multi-resolution analysis”, IEICE Transactions, 1996, J79-D-II, No. 5, p. 819-825

  However, in a moving image, a moving image frame increases as the moving speed of a subject increases due to an integration effect in a storage type imaging device such as a complementary metal oxide semiconductor (CMOS) or a charge coupled device (CCD) and a storage type recording medium such as a film. High-frequency components in the inner moving area are lost, and motion blur occurs.

  In the conventionally proposed image enlargement method, since the resolution enlargement is performed on the image including the motion blur, there is a problem that the high frequency component in the moving region cannot be accurately estimated.

  Furthermore, since the conventional image enlargement method does not consider the spatial frequency characteristics in the human visual system for the estimated high-frequency component, the high-frequency component is excessively emphasized when viewed from the same viewing distance as before the image enlargement. There was a fear of becoming an image.

  The present invention has been made to solve the conventional problems, and effectively compensates for the spatial high-frequency component of the moving region on the moving image frame sequence lost due to the integration effect in the storage-type imaging device, and It is an object of the present invention to provide a moving image spatial resolution enlarging apparatus and a moving image spatial resolution enlarging program capable of performing natural spatial resolution expansion using a contrast sensitivity function related to human visual characteristics.

A moving image spatial resolution enlarging apparatus according to the present invention includes a spatial resolution enlarged image sequence generating means for generating a spatial resolution enlarged image sequence by performing spatial resolution expansion of a moving image frame sequence, and each n-th wavelet of the spatial resolution enlarged image sequence. the pixel values of the cracking component is corrected based on the contrast sensitivity function, seen including a correction image sequence generating means for generating a corrected image sequence by reconstructing each n floors wavelet decomposition component after correction, and the spatial resolution larger The image sequence generation means extracts a motion area image sequence based on a high frequency component obtained by performing first-order one-dimensional discrete wavelet decomposition on the moving image frame sequence in the time axis direction. Then, n-dimensional two-dimensional discrete wavelet decomposition is performed on the moving region image sequence in a spatial direction, and the moving region image sequence is decomposed into each n-th wavelet decomposition component. Region image sequence decomposition means, motion vector detection means for detecting a motion vector based on a low-frequency component of each n-th wavelet decomposition component of the motion region image sequence, and the motion image frame sequence using the motion vector and tracks the pixel positions before and after ± q frames based on the moving image frame at time t ₀ of the horizontal high-frequency component of each n floors wavelet decomposition components of the moving picture frame of the time t _0, the vertical high-frequency component and a diagonal The high-frequency component is replaced with the horizontal high-frequency component, vertical high-frequency component, and diagonal high-frequency component of each n-th wavelet decomposition component of the moving image frame in which the absolute value of the moving vector corresponding to the tracked pixel position is minimized. Dynamic region compensation image sequence generation means for reconstructing each n-th order wavelet decomposition component and generating a dynamic region compensation image sequence; and spatial resolution expansion of the dynamic region compensation image sequence It was carried out, and has a structure comprising a spatial resolution larger means for generating the spatial resolution magnified image sequence.

With this configuration, natural spatial resolution expansion can be performed using a contrast sensitivity function related to human visual characteristics.
In addition, with this configuration, it is possible to effectively compensate for the spatial high-frequency component of the moving region on the moving image frame sequence lost due to the integration effect in the storage type image pickup device.

  The present invention uses moving vector information on a moving image frame sequence and compensates for a spatial high-frequency component in a high-speed moving region from a spatial high-frequency component in a low-speed moving region, thereby moving the moving image lost due to the integration effect in the storage type imaging device. A moving image spatial resolution enlarging device and a moving image spatial resolution enlarging device that can effectively compensate for spatial high-frequency components in a moving region on a frame sequence and perform natural spatial resolution expansion using a contrast sensitivity function related to human visual characteristics. A program can be provided.

  Hereinafter, a moving image spatial resolution enlarging apparatus according to an embodiment of the present invention will be described with reference to the drawings. In this specification, it is assumed that the moving area includes a low speed moving area and a high speed moving area. The low speed moving area refers to a moving area on the moving image frame in which the absolute value of the moving vector corresponding to the tracked pixel position is minimized.

Further, the moving image frame at time t is set to the moving image frame F (t), the pixel value at the pixel position (i, j) of the moving image frame F (t) is set to F _{i, j} (t), and 0 ≦ t ≦ T. A set of moving image frames F (t) to be transferred is described as a moving image frame sequence {F (t)}.

FIG. 1 is a block diagram showing a functional configuration of a moving image spatial resolution enlarging apparatus according to an embodiment of the present invention.
That is, the moving image spatial resolution enlarging apparatus 1 according to the present invention performs spatial resolution enlarging of the moving image frame sequence {F (t)} and generates a spatial resolution enlarged image sequence 11, A pixel value of each nth-order wavelet decomposition component of the spatial resolution enlarged image sequence is corrected based on the contrast sensitivity function, and each corrected nth-order wavelet decomposition component is reconstructed to obtain a corrected image sequence {SU ⁰ (t)}. And a corrected image sequence generating means 12 to be generated.

  Then, the spatial resolution enlarged image sequence generating unit 11 includes a high frequency component compensating unit 111 that compensates a high frequency component in the high speed moving region based on a high frequency component in the low speed moving region of the moving image frame sequence {F (t)}, and a high frequency component. And spatial resolution enlarging means 112 for enlarging the spatial resolution of the compensated moving image frame sequence {F (t)}.

  Further, the high frequency component compensation unit 111 uses the moving region image sequence {M (t)} based on the high frequency component obtained by performing the one-dimensional discrete wavelet decomposition for each pixel of the moving image frame sequence {F (t)}. The moving region image sequence extracting means 113 extracts the moving region image sequence {M (t)} from the moving region image sequence extracting means 113 and the moving region image sequence {M (t)}. A moving area image sequence decomposing means 114 for decomposing into decomposed components, and a moving vector detecting means 115 for detecting a moving vector based on the low frequency components of each n-th wavelet decomposed component of the moving area image sequence {M (t)} including.

Furthermore, the high-frequency component compensation unit 111 tracks pixel positions for ± q frames before and after the moving image frame sequence {F (t)} at the time t _{0 based on} the moving image frame F (t ₀ ). The horizontal high-frequency component F_LH ⁿ _{i, j} (t ₀ ), the vertical high-frequency component F_HL ⁿ _{i, j} (t ₀ ), and the diagonal high-frequency component F_HH ⁿ of each n-th wavelet decomposition component of the ₀ moving image frame F (t ₀ ) _{i, j} (t ₀ ) is the horizontal high-frequency component F_LH ⁿ _{k, m} ( ⁿ ) of each n-th wavelet decomposition component of the moving image frame F (t _s ) that minimizes the absolute value of the moving vector corresponding to the tracked pixel position. t _s ), vertical high-frequency component F_HL ⁿ _{k, m} (t _s ) and diagonal high-frequency component F_HH ⁿ _{k, m} (t _s ) are replaced, and each n-th wavelet decomposition component after replacement is reconstructed to be a moving region Dynamic region compensation image sequence generator for generating compensation image sequence {CF (t)} And a 116.

  FIG. 2 is a block diagram showing a hardware configuration of the moving image spatial resolution enlarging apparatus 1 according to the present invention. The moving image spatial resolution enlarging apparatus 1 according to the present invention is configured by a microprocessor 2. .

The microprocessor 2 executes a process according to a moving image spatial resolution enlargement program, a memory 21 for storing a moving image spatial resolution enlargement program and a processing result, and a moving image frame sequence signal reading interface for reading a moving image frame sequence signal ( I / F) 22, a moving image frame sequence storage device 23 that stores a moving image frame sequence {F (t)}, and a corrected image sequence output I / F 24 that outputs a corrected image sequence {SU ⁰ (t)}. The peripheral device I / F 25 to which peripheral devices for operating the moving image spatial resolution enlarging device 1 are connected is connected to each other by a bus 26.

  A display panel 27, a keyboard 28, and a mouse 29 are connected to the peripheral device I / F 25. An operation panel can be applied instead of the display panel 27, the keyboard 28, and the mouse 29.

  Next, the operation of the moving image spatial resolution enlarging apparatus according to the embodiment of the present invention will be described with reference to the flowchart of the moving image spatial resolution enlarging program installed in the memory 21.

FIG. 3 is a flowchart of the moving image spatial resolution enlargement program. First, the CPU 20 reads a moving image frame sequence {F (t)} (0 ≦ t ≦ T) (step S30).
Next, the CPU 20 extracts a moving area image sequence {M (t)} in a moving area image sequence extraction routine described later (step S31).

Next, the CPU 20 decomposes the moving area image sequence {M (t)} into each n-th order wavelet decomposition component in a moving area image sequence decomposition routine described later (step S32).
Next, the CPU 20 detects a motion vector in a motion vector detection routine described later (step S33).

Next, the CPU 20 generates a moving region compensation image sequence {CF (t)} in a moving region compensation image sequence generation routine described later (step S34).
Next, the CPU 20 expands the spatial resolution of the dynamic region compensated image sequence {CF (t)} in a spatial resolution expansion routine described later (step S35).

Finally, the CPU 20 generates a corrected image sequence {SU ⁰ (t)} in a corrected image sequence generation routine described later (step S36), and ends the program.

  Next, the moving region image sequence extraction routine executed in step S31 of the moving image spatial resolution enlarging program in FIG. 3 will be described using the flowchart in FIG. 4 and the schematic diagrams in FIGS. 5 (a) and 5 (b).

First, the CPU 20 initializes the time t to “0” (step S40).
Next, the CPU 20 initializes an index i indicating the position of the pixel in the row direction to “1” (step S41), and initializes an index j indicating the position of the pixel in the column direction to “1” (step S42). .

Next, the CPU 20 uses, for example, a Daubechies secondary wavelet (support length = 4), as shown in the schematic diagrams of FIGS. 5A and 5B, for the moving image frame sequence {F (t)}. Time-series data in the time axis direction including pixel values F _{i, j} (t), F _{i, j} (t + 1), F _{i, j} (t + 2), F _{i, j} (t + 3) at the pixel position (i, j) First-order one-dimensional discrete wavelet decomposition is performed on the signal, and MA _{i, j} (t) _, which is the high frequency component, is calculated (step S43). That is, MA _{i, j} (t) includes a time-series change component for a support length of 4 frames.

Next, the CPU 20 performs step S43 from the time-series data in the time axis direction including F _{i, j} (t + 2), F _{i, j} (t + 3), F _{i, j} (t + 4), F _{i, j} (t + 5). MA _{i, j} (t + 2) is calculated in the same manner as (Step S44).
Next, the CPU 20 calculates MA _{i, j} (even) by binarizing MA _{i, j} (t) and MA _{i, j} (t + 2) with a predetermined threshold as a boundary and performing AND calculation. (Step S45).

Next, the CPU 20 performs step S43 from time-series data in the time axis direction including F _{i, j} (t + 1), F _{i, j} (t + 2), F _{i, j} (t + 3), and F _{i, j} (t + 4). MA _{i, j} (t + 1) is calculated in the same manner as (Step S46).
Next, the CPU 20 performs step S43 from the time-series data in the time axis direction including F _{i, j} (t + 3), F _{i, j} (t + 4), F _{i, j} (t + 5), F _{i, j} (t + 6). MA _{i, j} (t + 3) is calculated in the same manner as (Step S47).

Next, the CPU 20 calculates MA _{i, j} (odd) from MA _{i, j} (t + 1) and MA _{i, j} (t + 3) as in step S45 (step S48).

Next, the CPU 20 performs an AND calculation of MA _{i, j} (even) and MA _{i, j} (odd) to calculate Ma _{i, j} (t), and F _{i, j} (t + 3) and Ma _{i, j} Multiplying (t), the moving region pixel value M _{i, j} (t) for one frame is calculated (step S49).

  Next, the CPU 20 determines whether or not the index j indicating the position of the pixel in the column direction has reached the maximum value J (step S50). If the determination is negative, the index j is incremented (step S51). The process returns to step S43.

  When determining that the index j has reached the maximum value J in step S50, the CPU 20 determines whether or not the index i indicating the position of the pixel in the row direction has reached the maximum value I (step S52). If a negative determination is made, the index i is incremented (step S53), and the process returns to step S42.

If the CPU 20 determines in step S52 that the index i has reached the maximum value I, the CPU 20 generates a moving area image M (t) from the set of moving area pixel values M _{i, j} (t) (step S54). .

  Next, the CPU 20 determines whether or not the time t has reached the time T, which is the time of the last moving image frame (step S55). If the determination is negative, the time t is incremented (step S56). The process returns to step S41.

  If the CPU 20 determines in step S55 that the time t has reached the time T, the CPU 20 ends the routine.

  Next, the moving region image sequence decomposition routine executed in step S32 of the moving image spatial resolution enlarging program in FIG. 3 will be described using the flowchart in FIG. 6 and the schematic diagram in FIG.

First, the CPU 20 initializes the time t to “0” (step S60).
Next, the CPU 20 initializes an index n indicating the wavelet decomposition hierarchy to “1” (step S61).

Next, the CPU 20 performs n-order two-dimensional discrete wavelet decomposition on the moving region image M (t) using the Haar wavelet in the spatial direction, and performs low frequency component M_LL ⁿ (t) and horizontal high frequency component M_LH ⁿ ( t) The vertical high frequency component M_HL ⁿ (t) and the diagonal high frequency component M_HH ⁿ (t) are calculated (step S62).

  Here, FIG. 7 shows (a) the moving area image M (t), (b) the first-order wavelet decomposition image of the moving area image M (t), and (c) the second-order wavelet decomposition image of the moving area image M (t). The schematic diagram of is shown. As shown in the figure, the two-dimensional discrete wavelet decomposition on the second floor or higher performs two-dimensional discrete wavelet decomposition on the low frequency components on the first floor.

  Next, the CPU 20 determines whether or not the index n indicating the wavelet decomposition hierarchy has reached the maximum value N (step S63). If the determination is negative, the index n is incremented (step S64), and step S62 is performed. Return to the process.

  If it is determined in step S63 that the index n has reached the maximum value N, the CPU 20 determines whether or not the time t has reached the time T (step S65), and if negative, the time t is incremented. Then (step S66), the process returns to step S61.

  If the CPU 20 determines in step S65 that the time t has reached the time T, the CPU 20 ends the routine.

  Next, the moving vector detection routine executed in step S33 of the moving image spatial resolution enlarging program in FIG. 3 will be described with reference to the flowchart in FIG.

First, the CPU 20 initializes the time t to “0” (step S80).
Next, the CPU 20 initializes an index n indicating the wavelet decomposition hierarchy to “1” (step S81).

Next, the CPU 20 uses a block matching method between the low frequency component M_LL ⁿ (t) and the low frequency component M_LL ⁿ (t + 1) to set the pixel position (i, j) of the low frequency component M_LL ⁿ (t). The corresponding motion vector V ⁿ _{i, j} (t) is detected (step S82).

Next, the CPU 20 determines the pixel positions (i, j) of the low frequency component M_LL ⁿ (t), the horizontal high frequency component M_LH ⁿ (t), the vertical high frequency component M_HL ⁿ (t), and the diagonal high frequency component M_HH ⁿ (t). Pixel values M_LL ⁿ _{i, j} (t), M_LH ⁿ _{i, j} (t), M_HL ⁿ _{i, j} (t), and M_HH ⁿ _{i, j} (t) at V ⁿ _{i, j} (t) The pixel value at the position is replaced, and the motion correction low frequency component M′_LL ⁿ (t), the motion correction horizontal high frequency component M′_LH ⁿ (t), the motion correction vertical high frequency component M′_HL ⁿ (t), and the motion correction pair An angular high frequency component M′_HH ⁿ (t) is calculated (step S83).

Next, the CPU 20 moves the motion correction low frequency component M′_LL ⁿ (t), the motion correction horizontal high frequency component M′_LH ⁿ (t), the motion correction vertical high frequency component M′_HL ⁿ (t), and the motion correction diagonal high frequency. The component M′_HH ⁿ (t) is reconstructed to the first floor, and the motion correction low-frequency component M′_LL ⁿ⁻¹ (t) is calculated (step S84).

Next, the CPU 20 replaces the low frequency component M_LL ^n-1 (t) with the motion correction low frequency component M′_LL ^n-1 (t) (step S85).
Next, the CPU 20 determines whether or not the index n indicating the wavelet decomposition hierarchy has reached the maximum value N (step S86). If a negative determination is made, the index n is incremented (step S87), and step S82 is performed. Return to the process.

If the CPU 20 determines in step S86 that the index n has reached the maximum value N, the motion vector V ⁰ _{i, j} (t) for the original motion area image M (t) is calculated from [Equation 1]. (Step S88).

Next, the CPU 20 determines whether or not the time t has reached the time T (step S89). If the negative determination is made, the CPU 20 increments the time t (step S90) and returns to the process of step S81.
If the CPU 20 determines in step S89 that the time t has reached the time T, the CPU 20 ends the routine.

  Next, the moving region compensation image sequence generation routine executed in step S34 of the moving image spatial resolution enlarging program in FIG. 3 will be described using the flowchart in FIG. 9 and the schematic diagram in FIG.

First, since the start time of the moving image frame sequence {F (t)} is t = 0, the CPU 20 performs processing for ± q frames before and after the moving image frame F (t ₀ ) at time t _0. Therefore, the time t ₀ is initialized to “q” (step S120).
Next, the CPU 20 sets time t to “t ₀ ” (step S121).

  Next, the CPU 20 initializes an index i indicating the position of the pixel in the row direction to “1” (step S122), and initializes an index j indicating the position of the pixel in the column direction to “1” (step S123). .

Next, the CPU 20 uses the moving vector V ⁰ _{i, j} (t ′) (1 ≦ i ≦ I, 1 ≦ j ≦ J, t ₀ −q ≦ t ′) to generate the moving image frame F (t). The movement of the pixel position (i, j) is tracked in the minus direction of the time axis from time t ₀ to t ₀ -q. Further, using the motion vector V ⁰ _{i, j} (t ′) (1 ≦ i ≦ I, 1 ≦ j ≦ J, t ′ ≦ t ₀ + q), the pixel position (i, The movement of j) is traced in the time axis plus direction from time t ₀ to t ₀ + q (step S124).

Next, the CPU 20 determines the motion vector V ⁰ _{i, j} (t ′) (t ₀ −q ≦ t ′ ≦ t ₀ ) based on the tracking result of the pixel position (i, j) of the moving image frame F (t). + absolute value of q) is determined and the time t _s which minimizes the tracking position (k, m) (step S125).

  Next, the CPU 20 initializes an index n indicating the wavelet decomposition hierarchy to “1” (step S126).

Next, as shown in FIG. 10, the CPU 20 uses the horizontal high-frequency component F_LH ⁿ _{i, j} (t) and the vertical high-frequency component F_HL ⁿ _{i, j} (t) of the n-th wavelet decomposition component of the moving image frame F (t). and diagonal high-frequency component F_HH ⁿ _i, j a (t), horizontal high-frequency component of the n-floor wavelet decomposition components of the moving picture frame _{^{F (t s) F_LH n k}} , m (t s), a vertical high-frequency component F_HL ⁿ _k, Replacement with _m (t _s ) and diagonal high-frequency component F_HH ⁿ _{k, m} (t _s ) (step S127).

  Next, the CPU 20 determines whether or not the index n indicating the wavelet decomposition hierarchy has reached the maximum value N (step S128). If the determination is negative, the index n is incremented (step S129), and step S127 is performed. Return to the process.

  When determining that the index n has reached the maximum value N in step S128, the CPU 20 determines whether or not the index j indicating the position of the pixel in the column direction has reached the maximum value J (step S130). If a negative determination is made, the index j is incremented (step S131), and the process returns to step S124.

  When determining that the index j has reached the maximum value J in step S130, the CPU 20 determines whether or not the index i indicating the position of the pixel in the row direction has reached the maximum value I (step S132). If a negative determination is made, the index i is incremented (step S133), and the process returns to step S123.

If the CPU 20 determines in step S132 that the index i has reached the maximum value I, the low frequency component F_LL ⁿ (t), the horizontal high frequency component F_LH ⁿ (t), the vertical high frequency component F_HL ⁿ (t) and the pair The high-frequency angular component F_HH ⁿ (t) (1 ≦ n ≦ N) is reconstructed to the nth order to generate the motion region compensated image CF (t) (step S134).

Next, since the end time of the moving image frame sequence {F (t)} is t = T, the CPU 20 performs processing for ± q frames before and after the moving image frame F (t ₀ ) at time t _0. To do so, it is determined whether or not the time t ₀ has reached Tq (step S135). When a negative determination is made, the time t ₀ is incremented (step S136), and the process returns to step S121.

CPU20 at step S135, the time t ₀ is when it is determined to have reached the T-q, and terminates the routine.

  Next, the spatial resolution enlargement routine executed in step S35 of the moving image spatial resolution enlargement program of FIG. 3 will be described using the flowchart of FIG. 11 and the schematic diagram of FIG.

First, the CPU 20 initializes the time t to “0” (step S140).
Next, the CPU 20 calculates regression coefficients α ^LH , α ^HL , and α ^HH (step S141), which will be described in detail below.

As shown in FIG. 12, when the horizontally and vertically double-enlarged image of the moving region compensated image CF (t) is a spatial resolution enlarged image U ⁰ (t), the spatial resolution enlarged image U ⁰ (t) is decomposed on the first floor. The low frequency component U_LL ¹ (t) can be regarded as the moving region compensated image CF (t).

Here, the operator to perform downsampling with respect to n floor wavelet decomposition component of the dynamic domain compensation image CF (t) and I _n, when the Moore-Penrose generalized inverse and I _n ^+, moving areas compensated image CF (t) The first-order wavelet decomposition component is expressed by a linear regression model as shown in [Equation 2] using the second-order wavelet decomposition component and the coefficients β ^LH , β ^HL , and β ^HH .

At this time, coefficients β ^LH , β ^HL , β ^HH satisfying the least square criterion shown in [Equation 3] are determined as regression coefficients α ^LH , α ^HL , α ^HH .

Next, the CPU 20 calculates the horizontal high-frequency component U_LH ¹ (t), the vertical high-frequency component U_HL ¹ (t), and the diagonal high-frequency component U_HH ¹ (t) of the spatial resolution enlarged image U ⁰ (t) from [Equation 4]. Estimate (step S142).

Next, the CPU 20 reconstructs the low-frequency component U_LL ¹ (t), the horizontal high-frequency component U_LH ¹ (t), the vertical high-frequency component U_HL ¹ (t), and the diagonal high-frequency component U_HH ¹ (t) to the first floor. A resolution-enlarged image U ⁰ (t) is generated (step S143).

  Next, the CPU 20 determines whether or not the time t has reached the time T (step S144). If the negative determination is made, the CPU 20 increments the time t (step S145) and returns to the process of step S141.

  If the CPU 20 determines in step S144 that the time t has reached the time T, the CPU 20 ends the routine.

  Next, the corrected image sequence generation routine executed in step S36 of the moving image spatial resolution enlarging program of FIG. 3 will be described using the flowchart of FIG. 13 and the graph of FIG.

First, the CPU 20 initializes the time t to “0” (step S150).
Next, the CPU 20 initializes an index n indicating the wavelet decomposition hierarchy to “1” (step S151).

Next, the CPU 20 performs n-order two-dimensional discrete wavelet decomposition on the spatial resolution enlarged image U ⁰ (t), and performs low frequency component U ⁰ _LL ⁿ (t), horizontal high frequency component U ⁰ _LH ⁿ (t), The vertical high frequency component U ⁰ _HL ⁿ (t) and the diagonal high frequency component U ⁰ _HH ⁿ (t) are calculated (step S152).

Next, the CPU 20 determines the contrast sensitivities CS ⁿ _LH , CS ⁿ _HL , and CS ⁿ _HH from the graph of the contrast sensitivity function (CSF) in FIG. 14 (step S153), which will be described in detail below.

  The contrast sensitivity function (CSF) shown in FIG. 14 gives the spatial frequency dependence of the contrast sensitivity at a certain retinal illuminance [trd]. Here, the contrast sensitivity is the reciprocal of the contrast threshold, which is the minimum contrast that can be perceived by humans. The spatial frequency [cycle / deg] indicates the number of cycles per viewing angle of 1 °.

Using FIG. 14, the horizontal high-frequency component U ⁰ _LH ⁿ (t), the vertical high-frequency component U ⁰ _HL ⁿ (t), and the diagonal high-frequency component U ⁰ _HH ⁿ (t) of the spatial resolution enlarged image U ⁰ (t), respectively. The contrast sensitivities CS ⁿ _LH , CS ⁿ _HL , and CS ⁿ _HH are determined from the spatial frequency corresponding to the frequency band of.

Next, the CPU 20 sets a target pixel value ratio P _max / P _min (CS ⁿ _LH ) indicating a ratio between the maximum pixel value and the minimum pixel value based on the contrast sensitivity function for each of RGB colors (hereinafter collectively referred to as P). , P _max / P _min (CS ⁿ _HL ) and P _max / P _min (CS ⁿ _HH ) are calculated (step S154), the details of which will be described below.

[Expression 5] is the contrast sensitivity CS ⁿ _LH , CS ⁿ _HL , CS ⁿ _HH , the maximum pixel values P _max (CS ⁿ _LH ), P _max (CS ⁿ _HL ), P _max (CS ⁿ _HH ) and the minimum pixel. It is an expression showing the relationship between the values P _min (CS ⁿ _LH ), P _min (CS ⁿ _HL ), and P _min (CS ⁿ _HH ). Using the contrast sensitivities CS ⁿ _LH , CS ⁿ _HL , and CS ⁿ _HH that have already been obtained, the target pixel value ratios P _max / P _min (CS ⁿ _LH ), P _max / P _min (CS ⁿ _HL ) are obtained from [Equation 5]. ), P _max / P _min (CS ⁿ _HH ) is determined.

Next, the CPU 20 sets the horizontal high-frequency component U ⁰ _LH ⁿ (t), the vertical high-frequency component U ⁰ _HL ⁿ (t), and the diagonal high-frequency component U ⁰ _HH ⁿ (t) of the spatial resolution enlarged image U ⁰ (t). The ratio between the actual maximum pixel value and the minimum pixel value (hereinafter referred to as pixel value ratio) is the target pixel value ratio P _max / P _min (CS ⁿ _LH ), P _max / P _min (CS ⁿ _HL ), P If it is smaller than _max / P _min (CS ⁿ _HH ), pixel value correction is performed to expand the pixel value ratio to the target pixel value ratio (step S155).

  Next, the CPU 20 determines whether or not the index n indicating the wavelet decomposition hierarchy has reached the maximum value N (step S156). If a negative determination is made, the index n is incremented (step S157), and step S152 is performed. Return to the process.

If the CPU 20 determines in step S156 that the index n has reached the maximum value N, the low frequency component U ⁰ _LL ⁿ (t) and the horizontal high frequency component U ⁰ _LH ⁿ (t) after pixel value correction are vertical. The corrected image SU ⁰ (t) is generated by reconstructing the high-frequency component U ⁰ _HL ⁿ (t) and the diagonal high-frequency component U ⁰ _HH ⁿ (t) (1 ≦ n ≦ N) by n-th order (step S158).

  Next, the CPU 20 determines whether or not the time t has reached the time T (step S159). If the negative determination is made, the CPU 20 increments the time t (step S160) and returns to the process of step S151.

  If the CPU 20 determines in step S159 that the time t has reached the time T, the CPU 20 ends the routine.

  As described above, the moving image spatial resolution enlarging apparatus and the moving image spatial resolution enlarging program according to the embodiment of the present invention use the moving vector information on the moving image frame sequence to convert the high-speed moving region from the spatial high-frequency component of the low-speed moving region. By compensating the spatial high-frequency component of the storage type image sensor, it effectively compensates for the spatial high-frequency component of the moving region on the moving image frame sequence lost due to the integration effect in the storage type image sensor, and the contrast sensitivity function related to human visual characteristics. It is possible to perform natural spatial resolution expansion.

  As described above, the moving image spatial resolution enlarging apparatus and the moving image spatial resolution enlarging program according to the present invention effectively reduce the spatial high-frequency component of the moving region on the moving image frame sequence lost due to the integration effect in the storage type imaging device. It is effective as an image processing apparatus because it has the effect of being able to compensate for the natural spatial resolution by the contrast sensitivity function related to human visual characteristics.

The block diagram which shows the function structure of the moving image spatial resolution expansion apparatus which concerns on this invention The block diagram which shows the hardware constitutions of the moving image spatial resolution expansion apparatus which concerns on this invention Flowchart of moving image space resolution enlargement program executed by CPU of moving image space resolution enlargement device according to the present invention Flowchart of a moving region image sequence extraction routine executed by the CPU of the moving image spatial resolution enlarging device according to the present invention. Schematic diagram showing the processing flow of the moving region image sequence extraction routine Flowchart of a moving region image sequence decomposition routine executed by the CPU of the moving image spatial resolution enlarging device according to the present invention. Schematic diagram of moving area image M (t), first-order wavelet decomposition image of moving area image M (t), and second-order wavelet decomposition image of moving area image M (t) Flowchart of a motion vector detection routine executed by the CPU of the video image spatial resolution enlarging apparatus according to the present invention. Flowchart of a moving region compensation image sequence generation routine executed by the CPU of the moving image spatial resolution enlarging device according to the present invention. Schematic diagram showing the flow of processing of the moving region compensation image sequence generation routine The flowchart of the spatial resolution expansion routine which CPU of the moving image spatial resolution expansion apparatus which concerns on this invention performs Schematic diagram showing the configuration of the spatial resolution enlarged image The flowchart of the correction | amendment image sequence generation routine which CPU of the moving image spatial resolution expansion device which concerns on this invention performs Contrast sensitivity function (CSF) graph

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 Moving image spatial resolution expansion apparatus 11 Spatial resolution expansion image sequence production | generation means 12 Correction | amendment image sequence production | generation means 111 High frequency component compensation means 112 Spatial resolution expansion means 113 Dynamic area image sequence extraction means 114 Dynamic area image sequence decomposition means 115 116 Moving region compensation image sequence generation means

Claims (2)

A spatial resolution enlarged image sequence generating means for performing spatial resolution expansion of a moving image frame sequence and generating a spatial resolution enlarged image sequence;
Correction image sequence generation that corrects the pixel value of each n-th order wavelet decomposition component of the spatial resolution enlarged image sequence based on a contrast sensitivity function and reconstructs each corrected n-th order wavelet decomposition component to generate a corrected image sequence and means, only including,
The spatial resolution enlarged image sequence generation means includes:
A moving region image sequence extracting means for extracting a moving region image sequence based on a high frequency component obtained by performing first-order one-dimensional discrete wavelet decomposition in the time axis direction on the moving image frame sequence;
Moving region image sequence decomposition means for performing n-dimensional two-dimensional discrete wavelet decomposition on the moving region image sequence in a spatial direction, and decomposing the moving region image sequence into each n-th order wavelet decomposition component;
Motion vector detection means for detecting a motion vector based on a low frequency component of each n-th wavelet decomposition component of the motion region image sequence;
Using the motion vector, and tracks the pixel positions before and after ± q frames based on the moving image frame at time t ₀ of the moving image frame sequence, the n floors wavelet decomposition of the moving picture frame of the time t ₀ Horizontal high-frequency component, vertical high-frequency component, vertical high-frequency component, horizontal high-frequency component, vertical high-frequency component of each nth-order wavelet decomposition component of the moving image frame that minimizes the absolute value of the moving vector corresponding to the tracked pixel position And dynamic region compensation image sequence generation means for generating a motion region compensation image sequence by reconstructing each nth-order wavelet decomposition component after replacement with a diagonal high frequency component,
A moving image spatial resolution enlarging device , comprising: spatial resolution enlarging means for enlarging the spatial resolution of the moving area compensated image sequence and generating the spatial resolution enlarged image sequence . On the computer,
Spatial resolution enlarged image sequence generation processing for performing spatial resolution expansion of a moving image frame sequence and generating a spatial resolution enlarged image sequence;
Correction image sequence generation that corrects the pixel value of each n-th order wavelet decomposition component of the spatial resolution enlarged image sequence based on a contrast sensitivity function and reconstructs each corrected n-th order wavelet decomposition component to generate a corrected image sequence Process, and
The spatial resolution enlarged image sequence generation process includes:
A moving region image sequence extraction process for extracting a moving region image sequence based on a high-frequency component obtained by performing first-order one-dimensional discrete wavelet decomposition in the time axis direction on the moving image frame sequence;
A dynamic region image sequence decomposition process that performs n-dimensional two-dimensional discrete wavelet decomposition on the dynamic region image sequence in a spatial direction, and decomposes the dynamic region image sequence into each n-th order wavelet decomposition component;
A motion vector detection process for detecting a motion vector based on a low frequency component of each n-th order wavelet decomposition component of the motion region image sequence;
Using the moving vector, time t of the moving image frame sequence ₀ The pixel positions of ± q frames before and after are tracked with reference to the moving image frame of ₀ Each n-th wavelet of the moving image frame in which the absolute value of the moving vector corresponding to the tracked pixel position is minimized for the horizontal high-frequency component, vertical high-frequency component and diagonal high-frequency component of each n-th wavelet decomposition component of the moving image frame A dynamic region compensation image sequence generating process for generating a dynamic region compensation image sequence by reconstructing each n-th order wavelet decomposition component after replacement with a horizontal high frequency component, a vertical high frequency component and a diagonal high frequency component of the decomposition component;
And a spatial resolution expansion process for performing spatial resolution expansion of the dynamic region compensated image sequence and generating the spatial resolution expanded image sequence.

Images