JP4655213B2 - Image processing apparatus and method, program, and recording medium - Google Patents

Description

  The present invention relates to an image processing apparatus and method, a program, and a recording medium, and more particularly, to an image processing apparatus and method, a program, and a recording medium for processing an image.

  Conventionally, a frame memory for storing a television signal, a motion vector detection circuit for detecting the motion of the television signal read from the frame memory, a motion vector averaging circuit for obtaining an average of a plurality of motion vector data therefrom, An image conversion apparatus including a position correction circuit that performs frame interpolation processing on average motion vector data and a television signal read from a frame memory, a multiplication circuit, an addition circuit, and a correction signal generation circuit has been proposed (for example, Patent Document 1).

  The motion part / motion amount detection circuit detects the motion part and the motion amount from the MUSE signal and the motion vector, and the interframe interpolation circuit interpolates a frame in consideration of the motion vector into the still image of the MUSE signal, and the LPF. In the inter-field interpolation circuit via the sampling frequency conversion circuit, the field is interpolated between the fields, and in the class classification adaptive interpolation circuit, the class classification is performed from the level distribution pattern and edge data of the peripheral pixels to be interpolated, Using the read coefficient data, field interpolation and / or frame interpolation processing is performed and supplied to the mixing circuit via the sampling frequency conversion circuit. Some ratios are determined so that still images and motion images are mixed ( Patent reference 2).

JP-A-6-121290

Japanese Patent Laid-Open No. 9-172621

  However, when the vector used for interpolation is incorrect, an original edge or the like may occur in the generated image, and the generated image may break down. Such a failure is noticeable as image degradation, and the subjective image quality of the image is degraded.

  The present invention has been made in view of such a situation, and prevents the generated image from becoming inconspicuous.

  An image processing apparatus according to an aspect of the present invention detects an input motion vector on a first frame, assigns the detected motion vector to a pixel on a second frame, and assigns the assigned motion vector. An image processing device that generates a pixel value of a pixel on the second frame based on the second frame, the pixel on the second frame from which the pixel value is generated, and a pixel of interest A value indicating a variation between the motion vector assigned to the target pixel and the motion vector assigned to the first peripheral pixel that is a pixel in a predetermined first range around the target pixel is calculated. First calculating means, a pixel value of the target pixel, a pixel on the second frame, and a second peripheral pixel that is a pixel in a predetermined second range around the target pixel Pixel value of Second calculation means for calculating a value indicating a change in the pixel value in the spatial direction, and a value indicating the variation in the motion vector and a value indicating the change in the pixel value in the spatial direction based on the second calculation means. The pixel of the pixel on the input third frame, wherein the second frame is arranged between the determination unit for determining whether the image of the second frame is broken and the first frame The time from the time of the first frame to the time of the second frame from the value and the pixel value of the pixel on the first frame at the same position as the pixel on the third frame; Generating means for generating an interpolation value corresponding to a pixel on the second frame in accordance with a ratio of a time from the time of the second frame to a time of the third frame; Picture of the second frame Is determined to have failed, the pixel value of the pixel of interest is mixed with the interpolation value corresponding to the pixel of interest according to a predetermined ratio, and the pixel on the second frame, Mixing means for mixing the interpolation value corresponding to the third peripheral pixel with the pixel value of the third peripheral pixel which is a pixel in a predetermined third range around the pixel of interest.

  The first calculation means is a motion vector that matches the motion vector assigned to the target pixel as a value indicating variation, and that is the motion vector assigned to the first peripheral pixel. The number can be calculated.

  The first calculating means is a motion vector having a magnitude that falls within a range defined by a component of the motion vector assigned to the target pixel as a value indicating variation, and It is possible to calculate the number of motion vectors assigned.

  The second calculating means calculates a maximum absolute value of absolute values of differences between a pixel value of the target pixel and a pixel value of the second peripheral pixel as a value indicating a change in a pixel value in a spatial direction. It can be calculated.

  The determination means determines whether the image of the second frame at the target pixel is a failed image candidate based on a value indicating the variation in the motion vector and a predetermined first threshold. The image of the second frame at the pixel of interest is broken based on a first broken image candidate determining means for determining whether or not a value indicating a change in the pixel value in the spatial direction and a predetermined second threshold value. Second failed image candidate determining means for determining whether or not the image is a candidate for the image, determination results in the first failed image candidate determining means, and determination in the second failed image candidate determining means From the above result, it is possible to provide a failure location determination unit that determines whether or not the image of the second frame has failed in the target pixel.

  The maximum absolute value of the absolute values of the difference between the pixel value of the target pixel and the pixel value of the second peripheral pixel when each pixel on the second frame is sequentially set as the target pixel. Threshold calculating means for calculating the second threshold from the distribution can be further provided.

  A determination unit may be further provided that determines the ratio of mixing the interpolation value corresponding to the target pixel or the third peripheral pixel into the pixel value of the target pixel or the pixel value of the third peripheral pixel. .

  The determining means may determine the ratio based on ratio data stored in advance corresponding to the distance from the target pixel.

  In the case where two or more of the ratios are obtained for one of the target pixel or the third peripheral pixel, the determining unit determines the final ratio as a larger one of the obtained ratios. Can be determined.

  The image processing method according to one aspect of the present invention detects an input motion vector on a first frame, assigns the detected motion vector to a pixel on a second frame, and assigns the assigned motion vector. An image processing method for generating a pixel value of a pixel on the second frame based on the pixel, the pixel on the second frame from which the pixel value is generated, and a pixel of interest A value indicating a variation between the motion vector assigned to the target pixel and the motion vector assigned to the first peripheral pixel that is a pixel in a predetermined first range around the target pixel is calculated. Then, from the pixel value of the target pixel and the pixel value of the second peripheral pixel that is a pixel on the second frame and is a pixel in a predetermined second range around the target pixel, Pixels in the spatial direction Whether or not the image of the second frame is broken at the target pixel based on the value indicating the variation of the motion vector and the value indicating the change in the pixel value in the spatial direction. The pixel value of the pixel on the input third frame, in which the second frame is arranged between the first frame and the position of the pixel on the third frame The time from the time of the first frame to the time of the second frame from the pixel value of the pixel on the first frame at the position, and the time of the third frame from the time of the second frame When an interpolation value corresponding to a pixel on the second frame is generated according to a ratio to the time to the time, and it is determined that the image of the second frame is broken at the target pixel, According to a predetermined ratio, The interpolation value corresponding to the pixel of interest is mixed with the pixel value of the pixel, and the pixels on the second frame are pixels in a predetermined third range around the pixel of interest. Mixing the interpolated values corresponding to the third peripheral pixels with the pixel values of the three peripheral pixels.

  A program according to an aspect of the present invention detects an input motion vector on a first frame, assigns the detected motion vector to a pixel on a second frame, and based on the assigned motion vector A program for causing a computer to perform image processing for generating a pixel value of a pixel on the second frame, the pixel on the second frame from which the pixel value is generated, A variation between the motion vector assigned to the target pixel that is a certain pixel and the motion vector assigned to the first peripheral pixel that is a pixel in a predetermined first range around the target pixel. A pixel value of the target pixel and a second peripheral pixel which is a pixel on the second frame and which is a pixel in a predetermined second range around the target pixel. A value indicating a change in the pixel value in the spatial direction is calculated from the pixel value, and the second pixel is calculated for the target pixel based on the value indicating the variation in the motion vector and the value indicating the change in the pixel value in the spatial direction. It is determined whether the image of the frame is broken, and the pixel value of the pixel on the input third frame in which the second frame is arranged between the first frame and the first frame The time from the time of the first frame to the time of the second frame from the pixel value of the pixel on the first frame at the same position as the position of the pixel on the third frame, and the second An interpolation value corresponding to a pixel on the second frame is generated according to a ratio of a time from a frame time to a time from the third frame, and an image of the second frame is generated at the target pixel. When it is determined to be bankrupt The pixel value of the target pixel is mixed with the interpolation value corresponding to the target pixel according to a predetermined ratio, and the pixel on the second frame is a predetermined first area around the target pixel. A step of mixing the interpolation value corresponding to the third peripheral pixel with the pixel value of the third peripheral pixel which is a pixel in the range of 3.

  The recording medium of one aspect of the present invention detects an input motion vector on the first frame, assigns the detected motion vector to a pixel on the second frame, and assigns the assigned motion vector to the motion vector. A program for causing a computer to perform image processing for generating a pixel value of a pixel on the second frame based on the pixel on the second frame from which the pixel value is generated. Variation between the motion vector assigned to the pixel of interest that is a current pixel and the motion vector assigned to the first peripheral pixel that is a pixel in a predetermined first range around the pixel of interest And a pixel value of the target pixel and a pixel of the second peripheral pixel which is a pixel on the second frame and which is a pixel in a predetermined second range around the target pixel. A value indicating a change in the pixel value in the spatial direction is calculated from the prime value, and the second pixel in the target pixel is calculated based on the value indicating the variation in the motion vector and the value indicating the change in the pixel value in the spatial direction. It is determined whether the image of the frame is broken, and the pixel value of the pixel on the input third frame in which the second frame is arranged between the first frame and the first frame The time from the time of the first frame to the time of the second frame from the pixel value of the pixel on the first frame at the same position as the position of the pixel on the third frame, and the second An interpolation value corresponding to a pixel on the second frame is generated according to a ratio of a time from a frame time to a time from the third frame, and an image of the second frame is generated at the target pixel. When it is determined that the bankruptcy According to a predetermined ratio, the interpolated value corresponding to the target pixel is mixed with the pixel value of the target pixel, and the pixel on the second frame is a predetermined third around the target pixel. A program including a step of mixing the interpolation value corresponding to the third peripheral pixel with the pixel value of the third peripheral pixel which is a pixel in the range of is recorded.

  In one aspect of the present invention, the motion vector assigned to the pixel of interest that is a pixel on the second frame in which the pixel value is generated and is the pixel of interest, and the surroundings of the pixel of interest A value indicating a variation with the motion vector allocated to the first peripheral pixel that is a pixel in the first range determined in advance is calculated, and the pixel value of the target pixel and the pixel on the second frame A value indicating a change in a pixel value in a spatial direction is calculated from a pixel value of a second peripheral pixel that is a pixel in a predetermined second range around the pixel of interest, and the motion vector Based on the value indicating the variation and the value indicating the change in the pixel value in the spatial direction, it is determined whether or not the image of the second frame is broken at the pixel of interest, and between the first frame and The second frame is arranged From the input pixel value of the pixel on the third frame and the pixel value of the pixel on the first frame at the same position as the pixel position on the third frame. Depending on the ratio of the time from the time of the frame to the time of the second frame and the time from the time of the second frame to the time of the third frame, the pixels on the second frame When a corresponding interpolation value is generated and it is determined that the image of the second frame is broken at the target pixel, the pixel value of the target pixel is set to the pixel value of the target pixel according to a predetermined ratio. The interpolation value is mixed, and the pixel value of the third peripheral pixel which is a pixel on the second frame and which is a pixel in a predetermined third range around the target pixel is set to the third value. The interpolation values corresponding to the surrounding pixels are mixed. It is.

  As described above, according to one aspect of the present invention, an image can be generated.

  Also, according to one aspect of the present invention, it is possible to make the generated image inconspicuous.

  Embodiments of the present invention will be described below. Correspondences between the configuration requirements of the present invention and the embodiments described in the detailed description of the present invention are exemplified as follows. This description is to confirm that the embodiments supporting the present invention are described in the detailed description of the invention. Accordingly, although there are embodiments that are described in the detailed description of the invention but are not described here as embodiments corresponding to the constituent elements of the present invention, It does not mean that the embodiment does not correspond to the configuration requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  An image processing apparatus according to an aspect of the present invention detects an input motion vector on a first frame, assigns the detected motion vector to a pixel on a second frame, and assigns the assigned motion vector. An image processing device that generates a pixel value of a pixel on the second frame based on the second frame, the pixel on the second frame from which the pixel value is generated, and a pixel of interest A value indicating a variation between the motion vector assigned to the target pixel and the motion vector assigned to the first peripheral pixel that is a pixel in a predetermined first range around the target pixel is calculated. First calculation means (for example, the vector coincidence calculation unit 111 in FIG. 9), the pixel value of the target pixel, and the pixels on the second frame, which are predetermined around the target pixel. Second Second calculation means (for example, the maximum gradient calculation unit 113 in FIG. 9) that calculates a value indicating a change in the pixel value in the spatial direction from the pixel values of the second peripheral pixels that are pixels in the range; Based on a value indicating the variation of the vector and a value indicating the change in the pixel value in the spatial direction, determination means for determining whether or not the image of the second frame is broken in the target pixel (for example, FIG. 9 The pixel value of the pixel on the input third frame and the position of the pixel on the third frame, in which the second frame is arranged between the determination unit 112) and the first frame The time from the time of the first frame to the time of the second frame, and the time from the time of the second frame to the third value from the pixel value of the pixel on the first frame at the same position as Depending on the ratio of time to frame time Generating means for generating an interpolation value corresponding to the pixel on the second frame (for example, the 0 vector interpolation image generation unit 116 in FIG. 9), and the image of the second frame is broken at the target pixel. If it is determined that the pixel value of the target pixel is mixed with the interpolation value corresponding to the target pixel according to a predetermined ratio, the pixel on the second frame is the target pixel. 9 is a mixing unit that mixes the interpolation value corresponding to the third peripheral pixel with the pixel value of the third peripheral pixel that is a pixel in a predetermined third range around the pixel (for example, the image mixing unit in FIG. 9). 118).

  The determination means determines whether the image of the second frame at the target pixel is a failed image candidate based on a value indicating the variation in the motion vector and a predetermined first threshold. Based on first failure image candidate determination means (for example, failure location candidate determination unit 131 in FIG. 9) for determining whether or not a value indicating a change in the pixel value in the spatial direction and a predetermined second threshold value are used. Second failed image candidate determining means (for example, a failed part candidate determining unit 132 in FIG. 9) for determining whether or not the image of the second frame at the target pixel is a failed image candidate; Whether or not the image of the second frame has failed in the target pixel is determined based on the determination result in the first failed image candidate determination unit and the determination result in the second failed image candidate determination unit. Defect location judgment Means (e.g., collapse point determination unit 133 of FIG. 9) and can be provided.

  The image processing apparatus according to one aspect of the present invention provides a difference between a pixel value of the target pixel and a pixel value of the second peripheral pixel when each pixel on the second frame is the target pixel in order. Threshold value calculation means (for example, the gradient threshold value calculation unit 115 in FIG. 9) for calculating the second threshold value from the distribution of the maximum absolute value among the absolute values of the absolute value can be further provided.

  The image processing apparatus according to one aspect of the present invention is configured to mix the interpolation value corresponding to the target pixel or the third peripheral pixel into the pixel value of the target pixel or the pixel value of the third peripheral pixel. Further, a determination unit (for example, a mixture ratio determination unit 117 in FIG. 9) can be provided.

  The image processing method or program according to one aspect of the present invention is the pixel on the second frame in which the pixel value is generated, and the motion vector assigned to the pixel of interest that is the pixel of interest; A value indicating a variation with the motion vector assigned to the first peripheral pixel which is a pixel in a predetermined first range around the pixel of interest is calculated (for example, step S102 in FIG. 20), From the pixel value of the pixel of interest and the pixel value of the second peripheral pixel which is a pixel on the second frame and which is a pixel in a predetermined second range around the pixel of interest, A value indicating a change in pixel value is calculated (for example, step S103 in FIG. 20), and the second pixel is calculated for the target pixel based on the value indicating the variation in the motion vector and the value indicating the change in the pixel value in the spatial direction. of It is determined whether or not the frame image is broken (for example, step S104 in FIG. 20), and the second frame is arranged between the first frame and the input third frame. From the time of the first frame to the time of the second frame from the pixel value of the first frame and the pixel value of the pixel on the first frame at the same position as the pixel on the third frame And an interpolation value corresponding to the pixel on the second frame is generated according to the ratio of the time until the time of the second frame to the time of the third frame (for example, FIG. 20 Step S101), when it is determined that the image of the second frame is broken at the target pixel, the interpolation value corresponding to the target pixel is set to the pixel value of the target pixel according to a predetermined ratio. And mix Further, a pixel value of a third peripheral pixel which is a pixel on the second frame and which is a pixel in a predetermined third range around the target pixel corresponds to the third peripheral pixel. The step includes mixing the interpolation values (for example, step S106 in FIG. 20).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a configuration example of an image processing apparatus 1 to which the present invention is applied. The image processing apparatus 1 is composed of, for example, a personal computer. In FIG. 1, a CPU (Central Processing Unit) 11 executes various processes according to a program stored in a ROM (Read Only Memory) 12 or a storage unit 18. A RAM (Random Access Memory) 13 appropriately stores programs executed by the CPU 11 and data. The CPU 11, ROM 12, and RAM 13 are connected to each other by a bus 14.

  An input / output interface 15 is also connected to the CPU 11 via the bus 14. The input / output interface 15 is connected to an input unit 16 including a keyboard, a mouse, a microphone, and the like, and an output unit 17 including a display and a speaker. The CPU 11 executes various processes in response to commands input from the input unit 16. Then, the CPU 11 outputs the image and sound obtained as a result of the processing to the output unit 17.

  The storage unit 18 connected to the input / output interface 15 is composed of, for example, a hard disk and stores programs executed by the CPU 11 and various data. The communication unit 19 communicates with an external device via the Internet or other networks. A program may be acquired via the communication unit 19 and stored in the storage unit 18.

  The drive 20 connected to the input / output interface 15 drives a removable medium 21 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the programs and data recorded therein. Get etc. The acquired program and data are transferred to and stored in the storage unit 18 as necessary.

  The image processing apparatus 1 can be, for example, a television receiver, a hard disk player, an optical disk player, a portable player (cellular phone), or the like, or an image processing unit thereof.

  FIG. 2 is a block diagram illustrating an example of a functional configuration of the image processing apparatus 1. The image processing apparatus 1 includes a frame memory 51, a vector detection unit 52, a detection vector memory 53, a vector allocation unit 54, an allocation vector memory 55, an allocation flag memory 56, an allocation compensation unit 57, an image interpolation unit 58, and an image quality improvement processing unit. 59.

  It does not matter whether each function of the image processing apparatus 1 is realized by hardware or software. That is, each block diagram in this specification may be considered as a hardware block diagram or a software functional block diagram.

  In the image processing apparatus 1 having the configuration shown in FIG. 2, for example, an image of a progressive image signal (hereinafter referred to as a 24P signal) with a frame frequency of 24 Hz is input, and the input image (input image) has a frame frequency of 60 Hz. It is converted into an image of a progressive image signal (hereinafter referred to as 60P signal) and output.

  The input image of the 24P signal input to the image processing apparatus 1 is supplied to the frame memory 51, the vector detection unit 52, the vector allocation unit 54, the allocation compensation unit 57, the image interpolation unit 58, and the image quality improvement processing unit 59. The frame memory 51 stores input images in units of frames. The frame memory 51 stores a frame at time t one before the input image at time t + 1. The frame at time t stored in the frame memory 51 is supplied to the vector detection unit 52, the vector allocation unit 54, the allocation compensation unit 57, the image interpolation unit 58, and the image quality improvement processing unit 59. Hereinafter, the frame at time t on the frame memory 51 is referred to as frame t, and the frame of the input image at time t + 1 is referred to as frame t + 1.

  The vector detection unit 52 detects a motion vector between the target block of the frame t on the frame memory 51 and the target block of the frame t + 1 of the input image, and stores the detected motion vector in the detection vector memory 53. As a method for detecting the motion vector between the two frames, a gradient method or a block matching method is used. The detection vector memory 53 stores the motion vector detected by the vector detection unit 52 in the frame t.

  The vector allocation unit 54 distinguishes the motion vector obtained on the frame t of the 24P signal from the frame of the 60P signal to be interpolated in the allocation vector memory 55 (hereinafter, the frame of the 60P signal is distinguished from the frame of the 24P signal). , Which is also referred to as an interpolation frame), and the allocation flag of the allocation flag memory 56 of the pixel to which the motion vector is allocated is rewritten to 1 (True).

  The allocation vector memory 55 stores the motion vector allocated by the vector allocation unit 54 in association with each pixel of the interpolation frame. The allocation flag memory 56 stores an allocation flag indicating the presence / absence of a motion vector to be allocated for each pixel of the interpolation frame. For example, an assignment flag of True (1) indicates that a motion vector is assigned to the corresponding pixel, and an assignment flag of False (0) indicates that a motion vector is not assigned to the corresponding pixel. Show.

  The allocation compensation unit 57 refers to the allocation flag of the allocation flag memory 56, and supplements the motion vector of the peripheral pixel of the target pixel with respect to the target pixel for which the motion vector has not been allocated by the vector allocation unit 54. Allocate on 55 interpolation frames. At this time, the allocation compensator 57 rewrites the allocation flag of the target pixel to which the motion vector is allocated to 1 (True).

  The image interpolation unit 58 interpolates and generates the pixel value of the interpolation frame using the motion vector allocated to the interpolation frame in the allocation vector memory 55 and the pixel values of the frame t and the next frame t + 1. Then, the image interpolation unit 58 outputs the generated interpolation frame to the image quality improvement processing unit 59, and then outputs the frame t + 1 to the image quality improvement processing unit 59 as necessary. Hereinafter, the image output from the image interpolation unit 58 is also referred to as an interpolation image.

  The image quality improvement processing unit 59 uses the motion vector allocated to the interpolation frame stored in the allocation vector memory 55, the predetermined threshold value, and the pixel values of the frame t and the next frame t + 1 to interpolate. A process for improving the image quality is applied to the broken image among the images included in the frame, and the 60P signal image is output to a subsequent stage (not shown).

  Image corruption that occurs when interpolating the pixel value of the interpolation frame is empirically high when the motion vector assigned to the pixel of the interpolation frame is rampant. Sometimes it can happen.

  The image quality improvement processing unit 59 considers that a broken (degraded) image has occurred in a portion where the allocated motion vector is unclear and the change in the pixel value in the spatial direction is large. The image quality improvement processing unit 59 improves the image quality of the failed image by mixing the pixels of the zero vector interpolation image with the portion that is considered to have failed. Thereby, the failure of the image becomes inconspicuous.

  The 0 vector interpolation image is an image obtained by interpolating the frame t and the frame t + 1 with the 0 vector. Here, the zero vector is a motion vector whose spatial component is zero.

  Further, the threshold value may be supplied from the outside or stored inside. “Predetermined” means that it is predetermined before the execution of each process, and means that it is determined at least when the process is executed. Therefore, changing the threshold before the process in each block is executed corresponds to the predetermined.

  FIG. 3 is a diagram for explaining the principle of processing for interpolating frames in the image processing apparatus 1 according to the present invention. In the example of FIG. 3, a dotted line represents a frame of a 24P signal that is input to the image processing apparatus 1 at times t, t + 1, and t + 2, and a solid line is represented by the image processing apparatus 1 from the input 24P signal. , The 60P signal interpolation frames at the generated times t, t + 0.4, t + 0.8, t + 1.2, t + 1.6, and t + 2.

  Generally, in order to convert a 24P signal into a 60P signal, 5/2 times as many frames are required. That is, five 60P signal images must be generated from two 24P signal images. At this time, the generated interpolated frame of the 60P signal has a time phase of 0.0, 0.4, 0.8, 1.2, and 1 on the 24P signal in order to equalize the frame interval. 6 is arranged at a position. Among these, 4 frames (frames of t + 0.4, t + 0.8, t + 1.2, and t + 1.6) except for one frame at time t with a time phase of 0.0 do not exist on the 24P signal. It is an image. Therefore, when the image of the 24P signal is input, the image processing apparatus 1 generates four interpolation frames from the two frames at time t and time t + 1 of the 24P signal. Therefore, the image processing apparatus 1 outputs a 60P signal image including five frames at times t, t + 0.4, t + 0.8, t + 1.2, and t + 1.6.

  As described above, the image processing apparatus 1 executes the process of converting the frame frequency from the 24P signal image to the 60P signal image.

  In principle, as described above, from the two frames at time t and time t + 1 of the 24P signal, five frames at times t, t + 0.4, t + 0.8, t + 1.2, and t + 1.6 are obtained. A frame of 60P signal is newly generated. Actually, in the example of FIG. 3, 60P of t, t + 0.4, and t + 0.8 is based on two frames at time t and time t + 1 of the 24P signal. A frame of the signal is generated, and a frame of the 60P signal of t + 1.2, t + 1.6, and t + 2 is generated based on the two frames at the times t + 1 and t + 2 of the 24P signal.

  Also, hereinafter, an interpolation frame at time t + k such as time t + 0.4, t + 0.8, t + 1.2, and t + 1.6 is referred to as an interpolation frame t + k.

  FIG. 4 is a diagram for more specifically explaining the process of interpolating a frame. In the example of FIG. 4, a thick arrow represents a transition to each state, and an arrow T represents a time passage direction in the states 81 to 85. In states 81 to 85, the frame t at the time t of the 24P signal, the frame t + 1 at the time t + 1 next to the time t, or the frame t and the frame t + 1 at the time of input / output to each unit constituting the image processing apparatus 1 The state of the interpolation frame F of the 60P signal produced | generated during is shown notionally. That is, actually, for example, a frame in which a motion vector is detected as shown in the state 82 is not input, and the frame and the motion vector are input separately.

  The state 81 represents the state of the frame t and the frame t + 1 of the 24P signal input to the vector detection unit 52. A black dot on the frame t in the state 81 represents a pixel on the frame t. The vector detection unit 52 detects the position where the pixel on the frame t in the state 81 moves in the frame t + 1 of the next time, and the movement is indicated as shown on the frame t in the state 82. It outputs as a motion vector corresponding to each pixel. As a method for detecting the motion vector between the two frames, a block matching method or a gradient method is used. At this time, for example, when a plurality of motion vectors are detected in the pixel, the vector detection unit 52 obtains an evaluation value for each motion vector, and selects a motion vector based on the evaluation value.

  A state 82 represents the state of the frame t and the frame t + 1 input to the vector allocation unit 54. In the state 82, the arrow of each pixel of the frame t represents the motion vector detected by the vector detection unit 52.

  The vector allocating unit 54 extends the motion vector detected for each pixel of the frame t in the state 82 to the next frame t + 1 and sets it to a preset time phase (for example, t + 0.4 in FIG. 3). A position on a certain interpolation frame F is obtained. This is because, assuming that there is a constant motion between the frame t and the frame t + 1, the point where the motion vector passes through the interpolation frame F is the pixel position in that frame. Therefore, the vector allocating unit 54 allocates the passing motion vector to four neighboring pixels on the interpolation frame F in the state 83. At this time, depending on the pixel of the interpolation frame, there may be a case where no motion vector exists, or a plurality of motion vectors may be candidates for allocation. In the latter case, for example, like the vector detection unit 52, the vector allocation unit 54 obtains an evaluation value for each motion vector and selects a motion vector to be allocated based on the evaluation value.

  A state 83 represents the state of the frame t and the frame t + 1 and the interpolation frame F to which the motion vector is assigned, which is input to the assignment compensation unit 57. In the interpolation frame F in the state 83, the pixels to which the motion vector is assigned by the vector assigning unit 54 and the pixels to which the motion vector is not assigned are shown.

  The allocation compensation unit 57 compensates for a pixel to which a motion vector in the state 83 is not allocated by using a motion vector allocated to a peripheral pixel of the pixel. This is because the motion vector of the peripheral pixel of the pixel of interest and the motion vector of the pixel of interest are similar if the assumption that the neighboring region of the pixel of interest has the same motion holds. As a result, a motion vector that is accurate to some extent is also given to a pixel to which no motion vector is assigned, and a motion vector is assigned to all the pixels on the interpolation frame F in the state 84. In this case as well, since motion vectors of a plurality of surrounding pixels exist as candidates, the allocation compensation unit 57 obtains an evaluation value for each motion vector, for example, similarly to the vector allocation unit 54, and the evaluation value A motion vector to be allocated is selected based on

  A state 84 represents the state of the frame t and the frame t + 1 input to the image interpolation unit 58 and the state of the interpolation frame F in which motion vectors are assigned to all the pixels. Based on the motion vectors assigned to all these pixels, the image interpolation unit 58 can determine the positional relationship between the pixels on the interpolation frame F and the two frames t and t + 1. Therefore, the image interpolation unit 58 uses the motion vector allocated on the interpolation frame F and the pixel values of the frame t and the frame t + 1 to interpolate as indicated by the black dot of the interpolation frame F in the state 85. Pixel values on the frame F are generated by interpolation. Then, the image interpolation unit 58 outputs the generated interpolated frame, and then outputs a frame t + 1 as necessary, thereby outputting an interpolated image of the 60P signal to the subsequent stage.

  Next, the process of converting the frame frequency of the image processing apparatus 1 will be described with reference to the flowchart of FIG.

  In step S1, the vector detection unit 52 inputs the pixel value of the frame t + 1 of the input image at the time t + 1 and the pixel t of the frame t at the time t immediately before the input image of the frame memory 51, and proceeds to step S2. At this time, the vector allocating unit 54, the allocation compensating unit 57, the image interpolating unit 58, and the image quality improvement processing unit 59 perform the frame t + 1 of the input image at the time t + 1 and the time immediately before the input image in the frame memory 51. The pixel value of frame t of t is input.

  In step S2, the vector detection unit 52 detects a motion vector and proceeds to step S3. That is, the vector detection unit 52 detects a motion vector between the target block of the frame t on the frame memory 51 and the target block of the next frame t + 1 that is the input image, and the detected motion vector is detected by the detection vector memory 53. And proceed to step S3. As a method for detecting the motion vector between the two frames, a gradient method or a block matching method is used.

  In step S3, the vector allocation unit 54 allocates a motion vector to the pixel of interest on the interpolation frame to be interpolated, and the process proceeds to step S4. That is, in step S3, the vector allocation unit 54 allocates the motion vector obtained on the frame t to the pixel of interest on the interpolation frame to be interpolated in the allocation vector memory 55, and the pixel to which the motion vector is allocated. The allocation flag in the allocation flag memory 56 is rewritten to 1 (True). For example, an allocation flag that is True indicates that a motion vector is allocated to the corresponding pixel, and an allocation flag that is False indicates that a motion vector is not allocated to the corresponding pixel.

  In step S4, the allocation compensator 57 compensates the allocation and proceeds to step S5. That is, the allocation compensation unit 57 refers to the allocation flag in the allocation flag memory 56 in step S4, and for the pixel of interest for which no motion vector has been allocated by the vector allocation unit 54, the motion vector of the surrounding pixels of the pixel of interest. Is allocated on the interpolation frame of the allocation vector memory 55. At this time, the allocation compensator 57 compensates the motion vector and rewrites the allocation flag of the allocated pixel of interest to 1 (True).

  In step S5, the image interpolation unit 58 executes an image interpolation process. That is, in step S5, the image interpolation unit 58 uses the motion vector allocated to the interpolation frame of the allocation vector memory 55 and the pixel values of the frame t and the frame t + 1 to generate an interpolation frame pixel value. The process proceeds to step S6. Details of the image interpolation processing in step S5 will be described later. The image interpolation unit 58 outputs the generated interpolation frame, and then outputs a frame t + 1 as necessary, thereby outputting an interpolation image of a 60P signal.

  In step S6, the image quality improvement processing unit 59 executes image quality improvement processing for the interpolated frame, and the process proceeds to step S7. Details of the image quality improvement processing will be described later.

  In step S7, the image quality improvement processing unit 59 outputs an interpolation image of a 60P signal with improved image quality.

  In step S8, the vector detection unit 52 determines whether or not all the frames have been processed. If it is determined that all the frames have not been processed, the vector detection unit 52 returns to step S1 and performs the subsequent processing. repeat. On the other hand, when the vector detection unit 52 determines in step S8 that all the frames have been processed, the vector detection unit 52 ends the process of converting the frame frequency.

  As described above, the image processing apparatus 1 according to the present invention detects a motion vector from a frame of an input image of a 24P signal, assigns the detected motion vector to a pixel on a frame of a 60P signal, and assigns the assigned motion vector. Based on the above, the pixel value on the frame of the 60P signal is generated. Then, the image processing apparatus 1 applies an image quality improvement process to the interpolated frame, and outputs an image that is less noticeable.

  Next, details of the configuration of the image interpolation unit 58 will be described.

  FIG. 6 is a block diagram illustrating a configuration of the image interpolation unit 58. 6 interpolates and generates the pixel value of the interpolation frame using the motion vector allocated to the interpolation frame in the allocation vector memory 55 and the pixel values of the frame t and the frame t + 1. , 60P signal interpolation processing is performed.

  In the example of FIG. 6, the frame t of the image at time t is input to the spatial filter 92-1, and the frame t + 1 of the image at time t + 1 is input to the spatial filter 92-2 and the buffer 95.

  The interpolation control unit 91 selects the pixel of the interpolation frame in the allocation vector memory 55, and based on the motion vector assigned to the selected pixel, the pixel on the interpolation frame, the two frames t and t + 1 The positional relationship (spatial shift amount) with each pixel is obtained. That is, the interpolation control unit 91 uses, based on the pixels of the interpolation frame, the position on the frame t associated with the motion vector and the position of the pixel on the frame t corresponding to the pixel of the interpolation frame, The spatial shift amount is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-1. Similarly, the interpolation control unit 91 uses, based on the pixels of the interpolation frame, the position on the frame t + 1 associated with the motion vector and the position of the pixel on the frame t + 1 corresponding to the pixel of the interpolation frame. Is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-2.

  Further, the interpolation control unit 91 obtains an interpolation weight between the frame t and the frame t + 1 based on a preset time phase (time) of the interpolation frame, and uses the obtained interpolation weight as a multiplier 93-1. And 93-2. For example, when the time of the interpolation frame is a time that is “k” away from the time t + 1 of the frame t + 1 and a time that is “1−k” away from the time t of the frame t (ie, the time of the interpolation frame is the time t and time t + 1 is “1−k”: generated at a time that is internally divided into “k”), the interpolation control unit 91 sets an interpolation weight of “1-k” in the multiplier 93-1. An interpolation weight of “k” is set in the multiplier 93-2.

  Spatial filters 92-1 and 92-2 are constituted by cubic filters, for example. The spatial filter 92-1 is a pixel value on the frame t corresponding to the pixel of the interpolation frame based on the input pixel value of the pixel on the frame t and the spatial shift amount supplied from the interpolation control unit 91. And the obtained pixel value is output to the multiplier 93-1. The spatial filter 92-2, based on the input pixel value of the pixel on the frame t + 1 and the spatial shift amount supplied from the interpolation control unit 91, the pixel value on the frame t + 1 corresponding to the pixel of the interpolation frame And the obtained pixel value is output to the multiplier 93-2.

  When the position of the pixel in the interpolation frame does not match the position of the pixel on frame t or frame t + 1 (that is, when the position of the pixel in the interpolation frame is a component equal to or less than the pixel in frame t or frame t + 1). The spatial filters 92-1 and 92-2 obtain the sum of the inverse ratios of the distances of the four surrounding pixels by using the pixel values of the four surrounding pixels at the position of the pixel of the interpolation frame in the frame t or the frame t + 1. Then, the pixel value on the frame corresponding to the pixel of the interpolation frame is obtained. That is, the pixel values at the positions below the pixel are obtained by linear interpolation based on the distance from the surrounding four pixels.

  The multiplier 93-1 multiplies the pixel value on the frame t input from the spatial filter 92-1 by the interpolation weight “1-k” set by the interpolation control unit 91, and the weighted pixel value is The result is output to the adder 94. The multiplier 93-2 multiplies the pixel value on the frame t + 1 input from the spatial filter 92-2 by the interpolation weight “k” set by the interpolation control unit 91, and adds the weighted pixel value to the adder. Output to 94.

  The adder 94 generates the pixel value of the pixel of the interpolation frame by adding the pixel value input from the multiplier 93-1 and the pixel value input from the multiplier 93-2. The pixel value of the interpolation frame is output to the buffer 95. The buffer 95 buffers the input frame t + 1. The buffer 95 outputs the generated interpolation frame, and then outputs a buffered frame t + 1 as necessary based on a preset time phase (time) of the 60P frame. Thus, the 60P signal interpolated image is output to the image quality improvement processing unit 59.

  Details of the image interpolation processing of the image interpolation unit 58 configured as described above will be described with reference to the flowchart of FIG.

  In step S51, the interpolation control unit 91 calculates interpolation weights (for example, “k” and “1-k”) of the interpolation frame between the frame t and the frame t + 1 based on the time phase of the interpolation frame to be processed. The obtained interpolation weights are set in the multipliers 93-1 and 93-2, respectively, and the process proceeds to step S52.

  For example, as illustrated in FIG. 8, when generating an interpolation frame at time t + k between a frame t at time t and a frame t + 1 at time t + 1, the interpolation control unit 91 uses interpolation weights of k and 1-k. And the obtained interpolation weights are set in the multipliers 93-1 and 93-2, respectively. In the example of FIG. 8, the dotted line represents the frame of the 24P signal input to the image processing apparatus 1 at time t and t + 1, and the solid line is represented by the image processing apparatus 1 from the input 24P signal. It shows an interpolation frame of a 60P signal at time t + k to be generated.

  In step S52, the interpolation control unit 91 selects a pixel of the interpolation frame in the allocation vector memory 55, and proceeds to step S53. Note that the pixels on the interpolation frame are selected in the raster scan order from the upper left pixel of the frame.

  In step S53, based on the motion vector assigned to the selected pixel, the interpolation control unit 91 determines the positional relationship between the pixels on the interpolation frame and the two frames t and t + 1 (space shift amount). ) And the obtained spatial shift amounts are supplied to the spatial filters 92-1 and 92-2, respectively, and the process proceeds to step S54.

  Specifically, in step S53, the interpolation control unit 91 uses the pixel of the interpolation frame as a reference, the position on the frame t associated with the motion vector, and the frame t corresponding to the pixel of the interpolation frame. The spatial shift amounts are obtained from the pixel positions, and the obtained spatial shift amounts are supplied to the spatial filter 92-1. Similarly, the interpolation control unit 91 uses, based on the pixels of the interpolation frame, the position on the frame t + 1 associated with the motion vector and the position of the pixel on the frame t + 1 corresponding to the pixel of the interpolation frame. Is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-2.

  The pixel value of the frame t of the image at time t is input to the spatial filter 92-1, and the pixel value of the frame t + 1 of the image at time t + 1 is input to the spatial filter 92-2. In step S54, the spatial filters 92-1 and 92-2 determine the pixels of the interpolation frame based on the pixel values of the pixels on the input frames t and t + 1 and the spatial shift amount supplied from the interpolation control unit 91. The pixel values on each frame corresponding to are obtained, the obtained pixel values are output to the multipliers 93-1 and 93-2, and the process proceeds to step S55.

  In Step S55, the multipliers 93-1 and 93-2 weight the interpolation weight set by the interpolation control unit 91 to the pixel value on each frame input from the spatial filter 92-1 or 92-2, The weighted pixel value is output to the adder 94, and the process proceeds to step S56. That is, the multiplier 93-1 multiplies the pixel value on the frame t input from the spatial filter 92-1 by the interpolation weight “1-k” set by the interpolation control unit 91, and weighted pixel values. Is output to the adder 94. The multiplier 93-2 multiplies the pixel value on the frame t + 1 input from the spatial filter 92-2 by the interpolation weight “k” set by the interpolation control unit 91, and adds the weighted pixel value to the adder. Output to 94.

  In step S56, the adder 94 adds the pixel value weighted by the multiplier 93-1 and the pixel value weighted by the multiplier 93-2 to generate the pixel value of the pixel of the interpolation frame. The generated pixel value is output to the buffer 95, and the process proceeds to step S57.

  For example, as shown in FIG. 8 by the processing of step S52 to step S56, the pixel value of the pixel q on the frame t associated with the motion vector v assigned to the selected pixel p in the interpolation frame and From the pixel value of the pixel r on the frame t + 1, the pixel value of the pixel p on the new interpolation frame t + k between the frame t and the frame t + 1 is generated.

  In step S57, the interpolation control unit 91 determines whether or not the processing for all the pixels on the interpolation frame is completed, and determines that the processing for all the pixels on the interpolation frame is not completed. If so, the process returns to step S52, and the subsequent processing is repeated. If the interpolation control unit 91 determines in step S57 that the processing for all the pixels on the interpolation frame has been completed, the interpolation control unit 91 ends the image interpolation processing.

  As described above, the pixel value of the interpolation frame is generated based on the motion vector assigned to the interpolation frame, and the interpolation frame is output by the buffer 95 in step S5 of FIG. In addition, if necessary, the frame t + 1 is output, so that an interpolated image of the 60P signal is output to the image quality improvement processing unit 59.

  Note that the processing described with reference to the flowchart of FIG. 7 is also referred to as interpolation frame generation processing.

  Next, the image quality improvement processing unit 59 will be described.

  FIG. 9 is a block diagram illustrating an example of the configuration of the image quality improvement processing unit 59. The image quality improvement processing unit 59 includes a vector matching number calculation unit 111, a determination unit 112, a maximum gradient calculation unit 113, a gradient histogram generation unit 114, a gradient threshold calculation unit 115, a 0 vector interpolation image generation unit 116, a mixture ratio determination unit 117, And an image mixing unit 118.

  The vector coincidence calculation unit 111 obtains variation in motion vectors. That is, the vector match number calculation unit 111 is a pixel on the interpolation frame in which the pixel value is generated, and the motion vector assigned to the target pixel that is the target pixel and the surroundings of the target pixel in advance. A value indicating variation with a motion vector assigned to surrounding pixels that are pixels in a predetermined range is calculated.

  Empirically, when the motion vector assigned to the pixel of the interpolation frame is unclear, there is a high possibility that the interpolation frame is broken, so the motion vector calculated by the vector match number calculation unit 111 It can be said that the value indicating the variation in the number indicates the possibility of the failure.

  For example, the vector coincidence calculation unit 111 calculates the number of motion vectors that are the same as the motion vectors assigned to the target pixel and that are assigned to the peripheral pixels, as values indicating variation.

  More specifically, the vector coincidence calculation unit 111 reads the motion vector assigned to the target image and the motion vector assigned to the pixels around the target pixel from the assigned vector memory 55. For example, the vector coincidence number calculation unit 111 assigns a motion vector assigned to a pixel in a range that is two pixels up and down and two pixels left and right away from the target pixel, or a motion vector assigned to the target pixel. Read from.

  The vector coincidence calculation unit 111 determines whether or not the motion vector assigned to the target image matches the motion vector assigned to the surrounding pixels.

  For example, the vector coincidence calculation unit 111 determines a range to be used for coincidence determination from a component having a large absolute value among Vx that is the x component and Vy that is the y component of the motion vector V assigned to the target image.

  When the Vx that is the x component is larger than the Vy that is the y component of the motion vector V assigned to the image of interest, the vector match number calculation unit 111 selects Vx that is the x component. Then, the vector coincidence calculation unit 111 sets the center position of the range used for coincidence determination as the position indicated by the end point of the motion vector V arranged so that the start point coincides with the origin, and the width of the range used for coincidence determination and The height is set to a value obtained by multiplying the selected absolute value of Vx by a predetermined constant.

  As shown in the example of FIG. 10, 0.2 × | Vx | above the Vy centered on the position (Vx, Vy) indicated by the end point of the motion vector V arranged so that the start point coincides with the origin. , Vy is 0.2 × | Vx |, Vx is 0.2 × | Vx |, and Vx is 0.2 × | Vx |. (Hereinafter, referred to as a matching allowable range). In this case, the matching allowable range is a rectangle having one side of 0.4 × | Vx |. In FIG. 10, the hatched portion indicates the matching allowable range.

That is, the width and height of the coincidence allowable range can be obtained by the equation (1).
Matchable range = ± α × (absolute value of maximum component of motion vector of target pixel)
= ± α × max (| Vx |, | Vy |)
... (1)
The constant α is, for example, 0.2. Note that max (a, b) is a function that selects the larger of a or b.

  That is, when the start point of the motion vector assigned to the pixels around the target pixel is arranged so as to coincide with the origin, when the end point of the motion vector indicates the matching allowable range, the motion vector is It is determined that it matches the motion vector assigned to the image. On the other hand, when the end point of the motion vector indicates that it is outside the allowable matching range, it is determined that the motion vector does not match the motion vector assigned to the target image.

In addition, as shown in FIG. 11, the matching allowable range may be circular. In this case, the radius of the coincidence allowable range is obtained by, for example, Expression (2).
Matchable range = ± α × (size of motion vector of target pixel)
= ± α × | V |
... (2)
Also in this case, the constant α is, for example, 0.2.

  The vector matching number calculation unit 111 determines whether each of the motion vectors assigned to the pixels around the target pixel matches the motion vector assigned to the target image, Among the assigned motion vectors, the number of motion vectors that match the motion vector assigned to the target image is obtained.

  The vector coincidence calculation unit 111 supplies a value indicating the variation of the motion vector to the determination unit 112.

  The maximum gradient calculation unit 113 obtains the activity of the interpolation frame. That is, the maximum gradient calculating unit 113 calculates the spatial direction from the pixel value of the target pixel and the pixel values of the peripheral pixels that are pixels in the interpolation frame and are in a predetermined range around the target pixel. A value indicating a change in pixel value is calculated.

  Empirically, when the activity in the generated interpolation frame is high, there is a high possibility that the interpolation frame has failed, so the value indicating the change in the pixel value in the spatial direction calculated by the maximum gradient calculation unit 113 Can be said to indicate the possibility of bankruptcy.

  For example, the maximum gradient calculation unit 113 calculates the maximum absolute value among the absolute values of the difference between the pixel value of the target pixel and the pixel values of the surrounding pixels as a value indicating the change in the pixel value in the spatial direction.

  More specifically, for example, as illustrated in FIG. 12, the maximum gradient calculating unit 113 determines the pixel value of the target pixel and the pixel value of the peripheral pixel for the peripheral pixels in the range of 3 × 3 pixels centering on the target pixel. Find the absolute value of the difference. That is, the maximum gradient calculating unit 113 obtains an absolute value of a difference between the pixel value of the target pixel and the pixel values of neighboring pixels that are adjacent to the target pixel in the up, down, left, and right directions.

  The maximum gradient calculation unit 113 sets the maximum value of the absolute values of the difference between the pixel value of the target pixel and the pixel values of the surrounding pixels as the maximum gradient of the target pixel.

  Note that when obtaining the maximum gradient, the gradient of the pixel at an oblique position with respect to the target pixel may not be considered.

  The maximum gradient calculation unit 113 supplies a maximum gradient, which is an example of a value indicating a change in the pixel value in the spatial direction, to the gradient histogram generation unit 114 and the determination unit 112.

  The gradient histogram generation unit 114 generates a histogram of the maximum gradient supplied from the maximum gradient calculation unit 113. For example, as illustrated in FIG. 13, when the number of pixels in the vertical direction of the image is 480, the gradient histogram generation unit 114 includes the upper end or lower end of the image among the maximum gradients supplied from the maximum gradient calculation unit 113. A histogram of the maximum gradient excluding the maximum gradient of 100 pixels is generated. That is, the gradient histogram generation unit 114 generates a histogram of the maximum gradient when the pixel of the target region is the target pixel, with the region of 280 pixels in the vertical direction and 720 pixels in the horizontal direction of the center of the image as the target region.

  Excluding the top or bottom of the image, a telop or banner is often displayed at the top or bottom of the image, and when the telop or banner is displayed, the gradient becomes larger, so avoid this effect Because.

  Note that although it has been described that the target area is 280 pixels in the vertical direction of the center of the image, the target area is not limited to 280 pixels. Further, the height (number of pixels) of the target region may be changed according to the number of pixels in the vertical direction of the entire image (entire frame).

  Further, a histogram having the maximum gradient may be generated by thinning out pixels in the target area. For example, a maximum gradient histogram may be generated for every other pixel in the target region.

  The gradient histogram generation unit 114 supplies data indicating the generated histogram to the gradient threshold value calculation unit 115.

  The gradient threshold value calculation unit 115 calculates the absolute value of the difference between the pixel value of the target pixel and the pixel values of the peripheral pixels that are the peripheral pixels of the target pixel when each pixel on the interpolation frame is the target pixel in order. Among these, a threshold value (gradient threshold value) is calculated from the maximum absolute value distribution. For example, the gradient threshold value calculation unit 115, based on the data indicating the histogram supplied from the gradient histogram generation unit 114, has reached 95% of the total number of frequencies from the smaller maximum gradient in the histogram. The maximum gradient of the part is set as a threshold value.

  Note that the threshold value is not limited to the maximum gradient of the portion that has reached 95%. In the experiment by the present inventor, a favorable result was obtained by setting the maximum gradient of the portion reaching 90% to 95% as a threshold value.

  FIG. 14 is a diagram illustrating a histogram indicated by data supplied from the gradient histogram generator 114. In FIG. 14, the horizontal axis indicates the maximum gradient, and the vertical axis indicates the frequency. In the example of the histogram shown in FIG. 14, the frequency for the interval of the maximum gradient of 8 steps is shown. That is, in the example of the histogram shown in FIG. 14, the maximum gradient section of 0 to 7, the maximum gradient section of 8 to 15, the maximum gradient section of 16 to 23, the maximum gradient section of 24 to 31, The frequency of each section of 39 maximum gradient sections, 40 to 47 maximum gradient sections... Is shown.

  For example, as shown in FIG. 13, when the target area is 280 pixels in the vertical direction and 720 pixels in the horizontal direction in the center of the image, the number of pixels in the target area is 201600. Since 95% of 2011600 is 191520, the minimum value of the maximum gradient in the next section in which the integrated value of frequencies exceeds 191520 is set as the threshold value.

  For example, when the integrated value of the frequency from the maximum gradient section of 0 to 7 to the maximum gradient section of 32 to 39 exceeds 191520, the gradient threshold value calculation unit 115 is the next maximum gradient of the 39 maximum gradient. 40 is the threshold value.

  Note that one section of the histogram is not limited to 8 steps, and may be an arbitrary number. By setting one section of the histogram in increments of 2 or more (n is an integer), the processing can be simplified. Of course, one maximum gradient may be set as one section.

  The gradient threshold value calculation unit 115 supplies the calculated threshold value (gradient threshold value) to the determination unit 112.

  The determination unit 112 determines whether or not the image of the interpolated frame has failed at the target pixel, based on the value indicating the motion vector variation and the value indicating the change in the pixel value in the spatial direction. The determination unit 112 includes a failure location candidate determination unit 131, a failure location candidate determination unit 132, and a failure location determination unit 133.

  The failure location candidate determination unit 131 determines whether the image of the interpolation frame at the target pixel is a failed image candidate based on the value indicating the variation of the motion vector and a predetermined threshold.

  The failure location candidate determination unit 132 is based on a value indicating a change in the pixel value in the spatial direction and a predetermined threshold (gradient threshold), and the image of the interpolated frame at the target pixel is a candidate for an image that has failed. It is determined whether or not.

  The failure location determination unit 133 determines whether the image of the interpolation frame has failed at the target pixel from the determination result of the failure location candidate determination unit 131 and the determination result of the failure location candidate determination unit 132. .

For example, the failure location candidate determination unit 131 indicates that the number of motion vectors matching the motion vector assigned to the target image indicated by the value supplied from the vector match number calculation unit 111 is less than a predetermined threshold TH ₁ . In some cases, it is determined that the image of the interpolated frame at the target pixel is a candidate for a failed image, and the number of motion vectors that match the motion vector assigned to the target image is greater than or equal to a predetermined threshold TH ₁ If it is, it is determined that the image of the interpolated frame at the pixel of interest is not a candidate for a broken image.

That is, the failure location candidate determination unit 131 determines that the image of the interpolated frame at the target pixel is a failed image candidate when the condition represented by Equation (3) is satisfied, and Equation (3) When the condition indicated by is not satisfied, it is determined that the image of the interpolation frame at the target pixel is not a failed image candidate.
Number of motion vector matches <TH ₁ (3)

Empirically, it is preferable that the threshold TH ₁ is half of the number of motion vectors to be determined for matching in the vector matching number calculation unit 111. In the vector coincidence calculation unit 111, when a motion vector assigned to a pixel in a range that is 2 pixels up and down and 2 pixels left and right from the target pixel is a match determination target, the match determination target Since the number of motion vectors is 8 (excluding the motion vector to which the pixel of interest is assigned), the threshold TH ₁ is preferably 4. Therefore, in this case, when the number of motion vectors matching the motion vector assigned to the pixel of interest is less than 4, the failure location candidate determination unit 131 has failed the image of the interpolation frame at the pixel of interest. It is determined that the image is a candidate.

For example, when the maximum gradient supplied from the maximum gradient calculation unit 113 is equal to or greater than the threshold TH ₂ that is the gradient threshold, the failure location candidate determination unit 132 determines that the image of the interpolation frame at the target pixel is a broken image. is determined to be a candidate, if the maximum gradient supplied from the maximum gradient calculating unit 113 is less than the threshold TH _2, it determines that the image of the interpolation frame in the target pixel is not a candidate for an image that is broken.

In other words, the failure location candidate determination unit 132 determines that the image of the interpolation frame at the target pixel is a failed image candidate when the condition represented by Equation (4) is satisfied, and Equation (4) When the condition indicated by is not satisfied, it is determined that the image of the interpolation frame at the target pixel is not a failed image candidate.
Maximum gradient> = TH ₂ (4)

  For example, the failure location determination unit 133 determines that the target pixel is a candidate for a failed image in the failure location candidate determination unit 131 and the failure location candidate determination unit 132 determines that the image of the failure has occurred. When it is determined as a candidate, it is determined that the image of the interpolation frame at the target pixel is broken. The failure location determination unit 133 determines that the target pixel is not a candidate for a failed image in the failure location candidate determination unit 131, or a failed image candidate in the failure location candidate determination unit 132. If it is determined that it is not, it is determined that the image of the interpolation frame at the target pixel has not failed.

That is, as shown in Expression (5), the failure location determination unit 133 includes an image determined to be a candidate for an image that has failed due to the number of motion vector matches and an image that has failed due to the maximum gradient. A product set with an image determined to be a candidate is determined to be a failed image.
Failure location = failure location candidate based on the number of motion vector matches ∩ failure location candidate due to maximum gradient
... (5)

Note that, as shown in Expression (6), the failure location determination unit 133 includes an image determined to be a candidate for an image that has failed due to the number of matching motion vectors, and an image that has failed due to the maximum gradient. You may make it determine with the union with the image determined to be a candidate the image which has failed.
Failure location = failure location candidate based on the number of motion vector matches ∪ failure location candidate due to maximum gradient
... (6)

  The failure location determination unit 133 supplies data indicating an image determined to have failed (for example, data indicating a position) to the mixture ratio determination unit 117.

  The zero vector interpolated image generation unit 116 has the interpolated frame t + k arranged between the frame t and the input pixel value of the pixel on the frame t + 1 and the frame t at the same position as the pixel on the frame t + 1. A ratio between the pixel value of the upper pixel and the time k from the time t of the frame t to the time t + k of the interpolation frame t + k and the time 1-k from the time t + k of the interpolation frame t + k to the time t + 1 of the frame t + 1 Accordingly, an interpolation value corresponding to the pixel on the interpolation frame t + k is generated.

  FIG. 15 is a diagram for explaining generation of an interpolation value of a 0 vector interpolation image in the 0 vector interpolation image generation unit 116. In the example of FIG. 15, the dotted line represents the frame of the 24P signal input to the image processing apparatus 1 at time t and t + 1, and the solid line is represented by the image processing apparatus 1 from the input 24P signal. It shows an interpolation frame of a 60P signal at time t + k to be generated. The motion vector v is assigned to the pixel p of the interpolation frame, and the pixel r on the frame t + 1 and the pixel q on the frame t are associated with each other by the motion vector v.

  The zero vector whose spatial component in the figure is 0 is the pixel p on the frame t at the same position as the position of the target pixel p on the interpolation frame and the frame t + 1 at the same position as the position of the target pixel p. The pixel p is associated.

As illustrated in FIG. 15, when paying attention to the pixel p on the interpolation frame, the 0 vector interpolation image generation unit 116 has a pixel value F _t of the pixel p on the frame t at the same position as the position of the pixel p. (P) and the pixel value F _{t + 1} (p) of the pixel p on the frame t + 1 at the same position as the position of the pixel p are acquired. Then, the 0 vector interpolated image generation unit 116 multiplies the pixel value F _{t + 1} (p) by a weight k corresponding to the time from time t to time t + k, and the pixel value F _t (p) at time t + k. Is multiplied by a weight 1-k according to the time from to time t + 1. Further, the zero vector interpolation image generation unit 116 adds these multiplication results to generate an interpolation value G _{t + k} (p).

That is, the zero vector interpolation image generation unit 116 obtains the interpolation value G _{t + k} (p) _represented by the equation (7).
G _{t + k} (p) = (1−k) F _t (p) + kF _{t + 1} (p) (7)

  The zero vector interpolation image generation unit 116 generates an interpolation value obtained by the calculation represented by Expression (7) for all the pixels of the interpolation frame. The zero vector interpolation image generation unit 116 supplies a zero vector interpolation image including the generated interpolation value to the image mixing unit 118.

  The mixing ratio determining unit 117 determines a ratio for mixing the pixel value of the target pixel or the pixel value of the peripheral pixel with the interpolation value corresponding to the target pixel or the peripheral pixel. For example, the mixture ratio determination unit 117 determines the ratio based on the ratio data stored in advance corresponding to the distance from the target pixel.

  Hereinafter, the ratio of mixing the pixel value and the interpolation value is also referred to as a mixing ratio.

  The mixing ratio determination unit 117 supplies the determined mixing ratio to the image mixing unit 118.

  When the target pixel is a failure location, the image mixing unit 118 mixes the zero vector interpolation image in a certain range centered on the target pixel. That is, when it is determined that the image of the interpolation frame has failed at the target pixel, the image mixing unit 118 sets the target pixel to the pixel value of the target pixel according to the ratio (mixing ratio) from the mixing ratio determination unit 117. And the interpolation value corresponding to the peripheral pixel is mixed with the pixel value of the peripheral pixel which is a pixel in the interpolation frame and is in a predetermined range around the pixel of interest. .

For example, when the mixing ratio γ is input for the pixel p from the mixing ratio determination unit 117, the image mixing unit 118 corresponds to the pixel p included in the 0 vector interpolation image supplied from the 0 vector interpolation image generation unit 116. The interpolation value G _{t + k} (p) is multiplied by the mixture ratio γ, and the pixel value F _{t + k} (p) of the pixel p of the interpolation frame t + k is multiplied by (1−γ). Then, the image mixing unit 118 sets a value obtained by adding these multiplication results as a pixel value F ′ _{t + k} (p) of the pixel p of the interpolation frame t + k.

That is, the interpolation value G generated by the 0 vector interpolation image generation unit 116 is converted by the image mixing unit 118 into the pixel value F _{t + k} (p) of the pixel p of the interpolation frame t + k according to the mixing ratio γ for the pixel p. _By mixing _{t + k} (p), the pixel value F ′ _{t + k} (p) of the pixel p of the interpolation frame t + k, which is represented by Expression (8), is obtained.
F ′ _{t + k} (p) = (1−γ) F _{t + k} (p) + γG _{t + k} (p) (8)

  The image mixing unit 118 outputs an image obtained as a result of the mixing as an output image.

  Here, a specific example of the mixing ratio will be described.

  For example, the mixture ratio determination unit 117 determines the mixture ratio for the pixel of interest, and determines the mixture ratio for peripheral pixels that are pixels around a predetermined range centered on the pixel of interest. For example, as shown in FIG. 16, the mixture ratio determination unit 117 determines the mixture ratio for pixels in the range of 15 × 15 pixels centered on the target pixel.

  For example, the maximum mixture ratio for the target pixel is 75%, and the minimum mixture ratio is 40%. Since the target pixel is a broken part, the mixture ratio between the target pixel and the interpolation value is maximized, and the mixture ratio is gradually decreased toward the periphery of the target pixel. That is, the mixture ratio of the peripheral pixels is set to a smaller value as the peripheral pixels are separated from the target pixel.

  For example, as shown in FIG. 16, the mixing ratio is determined by a gaussian function. In this case, for example, the mixture ratio determination unit 117 stores a look-up table (LUT) shown in FIG. 17 in advance.

  The lookup table illustrated in FIG. 17 has an array of 23 rows x 23 columns. Each element of the lookup table corresponds to each pixel. That is, an example in which the mixture ratio is set in a range of 23 horizontal pixels × 23 vertical pixels is shown. For the sake of simplicity, the example in FIG. 17 shows a numerical value when the precision of 3 bits after the decimal point (corresponding to decimal number 0.125) is maintained.

  For example, in the lookup table, values determined by the gaussian function are arranged in the elements of the array as a mixture ratio.

The gaussian function is expressed by Equation (9), where σ and λ are constants.
G ^ex (x, y) = λ ² exp [− (x ² + y ² ) / 2σ ² ] (9)
In Expression (9), (x, y) indicates a pixel position.

In the example shown in FIG. 17, with λ ² = 1 / 2πσ ² , the coefficient in the case where the target pixel is the center is obtained as a mixing ratio by Expression (10).
G ^ex (x, y) = (1 / 2πσ ² ) exp [− (x ² + y ² ) / 2σ ² ] (10)

  When the mixture ratio is arranged in the elements of the lookup table illustrated in FIG. 17, the element at the center of the lookup table array (the element in the 12th row from the left and the 12th column from the top) is: The mixing ratio of the target pixel is used. In this case, the mixture ratio of the target pixel is 1.000.

  Further, since the elements in the 11th to 13th rows from the left and the 11th to 13th columns from the top are 1.000, the mixture ratio of the peripheral pixels adjacent to the target pixel is also 1.000. .

  For example, the elements in the 10th row from the left and the 10th to 14th columns from the top, the elements in the 11th row from the left and the 10th column from the top, the elements in the 12th row from the left and the 10th column from the top The element of the eye, the 13th row from the left and the 10th column from the top, the 11th row from the left and the 14th column from the top, the 12th row from the left and the 14th column from the top Since the element, the element in the 13th row from the left and the 14th column from the top, and the element in the 14th row from the left and the 10th to 14th columns from the top are 0.875, 2 pixels from the target pixel The mixing ratio of the distant peripheral pixels is set to 0.875.

  Similarly, for example, since the element in the second row from the left and the twelfth column from the top is 0.125, the mixture ratio of the peripheral pixels 10 pixels to the left from the target pixel is 0.125.

  Further, a coefficient for multiplying the maximum value of the mixture ratio may be arranged in the lookup table.

  An example will be described in which the maximum value of the mixing ratio is set to 0.75, and the look-up table in FIG. 17 arranges the coefficient for multiplying the maximum value of the mixing ratio as an element.

  Since the element in the 12th row from the left and the 12th column from the top is 1.0, the mixture ratio of the target pixel is 0.75 (1.0 × 0.75 (maximum value of the mixture ratio)). The Since the elements in the 11th to 13th rows from the left and the 11th to 13th columns from the top are 1.0, the mixture ratio of the peripheral pixels adjacent to the target pixel is 0.75.

  For example, the elements in the 10th row from the left and the 10th to 14th columns from the top, the elements in the 11th row from the left and the 10th column from the top, the elements in the 12th row from the left and the 10th column from the top The element of the eye, the 13th row from the left and the 10th column from the top, the 11th row from the left and the 14th column from the top, the 12th row from the left and the 14th column from the top The element, the element in the 13th row from the left and the 14th column from the top, and the element in the 14th row from the left and the 10th to 14th columns from the top are 0.875, so 0.875 × 0 .75, the mixture ratio of the peripheral pixels that are two pixels away from the target pixel is 0.65625.

  Similarly, for example, the element in the second row from the left and the twelfth column from the top is 0.125, so the mixture ratio of the surrounding pixels 10 pixels to the left from the target pixel from 0.125 × 0.75 Is 0.09375.

  As described above, the mixture ratio determination unit 117 can determine the mixture ratio for the target pixel and the peripheral pixels based on the lookup table stored in advance.

  The function for determining the mixing ratio is not limited to the gaussian function, and may be other functions as long as the value decreases as the distance from the center increases. For example, as shown in FIG. 18, the mixing ratio may be determined based on a linear function.

  In addition, as shown in FIG. 19, when a failure occurs in an adjacent location (when failure location A and failure location B are close), neighboring pixels mixed with the interpolation value In some cases, an overlapping region is generated, and a plurality of mixing ratios are obtained for one pixel. In this case, the mixture ratio determination unit 117 employs the larger one of the plurality of mixture ratios obtained for one pixel as the mixture ratio of the pixel. That is, when two or more ratios are obtained for one of the target pixel or the peripheral pixels, the mixture ratio determination unit 117 determines the final ratio as a larger one of the obtained ratios.

  Next, the details of the image quality improvement processing in step S6 in FIG. 5 will be described with reference to the flowchart in FIG. In step S101, the 0 vector interpolation image generation unit 116 generates a 0 vector interpolation image from the frame t and the frame t + 1. In other words, in step S101, the 0 vector interpolation image generation unit 116 inputs the pixel value of the pixel on the frame t + 1 and the position of the pixel on the frame t + 1 in which the interpolation frame t + k is arranged between the frame t. The time from the time t of the frame t to the time t + k of the interpolation frame t + k, the time from the time t + k of the interpolation frame t + k to the time t + 1 of the frame t + 1, The interpolation value corresponding to the pixel on the interpolation frame t + k is generated according to the ratio.

  In step S102, the vector matching number calculation unit 111 calculates the number of motion vectors that match the motion vector assigned to the target image, from the motion vectors assigned to the pixels around the target image. That is, in step S102, the vector coincidence calculation unit 111 calculates a motion vector assigned to a target pixel that is a pixel on the interpolation frame in which the pixel value is generated and is a target pixel, and the target pixel. A value indicating a variation from a motion vector assigned to a peripheral pixel which is a pixel in a predetermined range around the pixel is calculated.

  In step S103, the maximum gradient calculation unit 113 calculates the maximum gradient between the pixel value of the image of interest and the pixel values of surrounding pixels. In other words, in step S103, the maximum gradient calculating unit 113 calculates the pixel value of the target pixel and the pixel values of the peripheral pixels that are pixels on the interpolation frame and are pixels in a predetermined range around the target pixel. Then, a value indicating a change in the pixel value in the spatial direction is calculated. For example, in step S <b> 103, the maximum gradient calculating unit 113 uses the maximum absolute value (the absolute value of the difference between the pixel value of the target pixel and the pixel value of the surrounding pixels as a value indicating the change in the pixel value in the spatial direction ( Maximum slope) is calculated.

  In step S104, the determination unit 112 determines a determination location based on a value indicating variation in motion vectors and a value indicating a change in pixel value in the spatial direction. That is, in step S104, the determination unit 112 determines whether the image of the interpolated frame has failed in the target pixel based on the value indicating the variation in the motion vector for the target pixel and the value indicating the change in the pixel value in the spatial direction. Determine whether or not.

For example, in step S104, the determination unit 112 is smaller than the value the threshold TH ₁ showing a variation in motion vectors, when the value indicating the change of the pixel values in the spatial direction is the threshold value TH ₂ or more, the interpolation frame in the pixel of interest It is determined that the image of is broken.

  In step S <b> 105, the mixture ratio determination unit 117 determines a mixture ratio that is a ratio of mixing the pixel value of the target pixel or the pixel value of the peripheral pixel with the interpolation value corresponding to the target pixel or the peripheral pixel. For example, in step S105, the mixture ratio determining unit 117 determines the mixture ratio based on a lookup table stored in advance.

  In step S106, the image mixing unit 118 mixes the interpolation image and the zero vector interpolation image. That is, in step S106, when the image mixing unit 118 determines that the image of the interpolation frame has failed at the target pixel, the pixel value of the target pixel is determined according to the ratio (mixing ratio) from the mixing ratio determination unit 117. In addition, the interpolation value corresponding to the target pixel is mixed, and the pixel corresponding to the peripheral pixel is the pixel value of the peripheral pixel that is a pixel on the interpolation frame and is in a predetermined range around the target pixel. Mix values.

  In step S107, the image quality improvement processing unit 59 determines whether or not the process has been completed for all the pixels of the interpolation frame. If it is determined that the process has not been completed, the process returns to step S102 to interpolate. The process from step S102 to step S106 is repeated for the next pixel of the frame. Note that the pixel (target pixel) that is the target of the processing in steps S102 to S106 is selected in the raster scan order, for example.

  If it is determined in step S107 that the processing has been completed for all the pixels in the interpolation frame, the process proceeds to step S108, where the gradient threshold value calculation unit 115 determines a gradient determination threshold value, and the image quality improvement processing ends.

  That is, for example, in step S108, the gradient histogram generation unit 114 generates a histogram of the maximum gradient supplied from the maximum gradient calculation unit 113. For example, in step S108, the gradient histogram generation unit 114 determines the pixel value of the target pixel and the pixel values of the peripheral pixels that are the peripheral pixels of the target pixel when each pixel on the interpolation frame is the target pixel in order. A histogram showing the distribution of the maximum absolute value among the absolute values of the differences is generated. Then, the gradient threshold value calculation unit 115 calculates a threshold value from a histogram indicating the maximum absolute value distribution.

  Note that the threshold value calculated in step S108 is used for the determination process in step S104 to be executed next. That is, the threshold value used in the determination process in step S104 is a value calculated in the previous frame. In this way, it is not necessary to hold the determination process until the threshold value is calculated, and the process can be executed easily and quickly.

  As described above, since the zero vector interpolation image is mixed with the portion where the interpolation frame has failed, the failure of the image can be made inconspicuous without impairing the smoothness of the motion of the image. Since the failure of the image is less noticeable, the user viewing the image feels that the image quality of the image has been improved.

  Based on the variation of the motion vector and the change in the spatial direction of the image, the broken image can be detected more accurately and with higher accuracy.

  In this way, the motion vector on the first frame is detected, the detected motion vector is allocated to the pixel on the second frame, and the first frame and the third frame are assigned based on the allocated motion vector. When the pixel value is calculated from the frame to generate the pixel on the second frame, an image can be generated. In addition, the motion vector assigned to the pixel of interest that is a pixel on the second frame in which the pixel value is generated, and the pixel of interest, and a predetermined first range around the pixel of interest A value indicating a variation with the motion vector assigned to the first peripheral pixel that is a pixel is calculated, and a pixel value of the target pixel and a pixel on the second frame that are predetermined around the target pixel are determined in advance. The value indicating the change in the pixel value in the spatial direction is calculated from the pixel values of the second peripheral pixels that are the pixels in the second range, and the value indicating the variation in the motion vector and the change in the pixel value in the spatial direction are calculated. Based on the indicated value, it is determined whether or not the image of the second frame is broken at the target pixel, and the second frame is arranged between the first frame and the input third frame The pixel value of the upper pixel and on the third frame From the pixel value of the pixel on the first frame at the same position as the pixel position, the time from the time of the first frame to the time of the second frame, and the time of the second frame from the time of the second frame An interpolation value corresponding to the pixel on the second frame is generated according to the ratio to the time until the time, and when it is determined that the image of the second frame is broken at the target pixel, a predetermined ratio And the pixel value of the target pixel is mixed with the interpolation value corresponding to the target pixel, and the pixels on the second frame, which are pixels in a predetermined third range around the target pixel. When the interpolated value corresponding to the third peripheral pixel is mixed with the pixel value of the peripheral pixel, the failure of the generated image can be made inconspicuous.

  Further, in the present embodiment, the area where each process is performed is described as being configured by, for example, 3 pixels × 3 pixels, but these are only examples, and the pixels constituting the area where each process is performed Is not limited to this number of pixels.

  Furthermore, in the present embodiment, the signal conversion from 24P signal to 60P signal has been described as an example. However, the present invention can be applied to interlace signal or other frame rate conversion as frame frequency conversion of moving images, for example. Can also be applied.

  The input image may be an image using either an analog signal or a digital signal as a medium. Further, an image read from a recording medium, an image transmitted by broadcasting or communication, an image acquired via a network, etc. Any image may be used, and the present invention is not limited thereto.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  As shown in FIG. 1, a program recording medium for storing a program that is installed in a computer and can be executed by the computer is a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only). Memory), DVD (including Digital Versatile Disc), magneto-optical disk), or removable media 21 that is a package medium made of semiconductor memory or the like, or ROM 12 in which a program is temporarily or permanently stored, The storage unit 18 is configured by a hard disk or the like. The program is stored in the program recording medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 19 that is an interface such as a router or a modem as necessary. Done.

  In the present specification, the step of describing the program stored in the program recording medium is not limited to the processing performed in time series in the order described, but is not necessarily performed in time series. Or the process performed separately is also included.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a block diagram showing the example of a structure of the image processing apparatus to which this invention is applied. It is a block diagram which shows the example of a function structure of an image processing apparatus. It is a figure explaining the principle of the process which interpolates a flame | frame. It is a figure explaining the process which interpolates a frame more concretely. It is a flowchart explaining the process which converts a frame frequency. It is a block diagram which shows the structure of an image interpolation part. It is a flowchart explaining the detail of an image interpolation process. It is a figure explaining an interpolation frame. It is a block diagram which shows the example of a structure of an image quality improvement process part. It is a figure which shows the example of the tolerance | permissible_range of a motion vector matching. It is a figure which shows the example of the tolerance | permissible_range of a motion vector matching. It is a figure explaining the maximum gradient. It is a figure explaining the area | region used as the object of the production | generation of the histogram of the maximum gradient. It is a figure which shows the example of a histogram. It is a figure explaining the production | generation of an interpolation value. It is a figure explaining the example of mixing ratio. It is a figure which shows the example of a look-up table. It is a figure explaining the example of mixing ratio. It is a figure explaining determination of a mixture ratio. 10 is a flowchart illustrating details of image quality improvement processing.

Explanation of symbols

  11 CPU, 12 ROM, 13 RAM, 18 storage unit, 21 removable media, 51 frame memory, 52 vector detection unit, 53 detection vector memory, 54 vector allocation unit, 55 allocation vector memory, 56 allocation flag memory, 57 allocation compensation unit , 58 image interpolation unit, 59 image quality improvement processing unit, 111 vector matching number calculation unit, 112 determination unit, 113 maximum gradient calculation unit, 114 gradient histogram generation unit, 115 gradient threshold calculation unit, 1160 vector interpolation image generation unit, 117 Mixing ratio determination unit, 118 image mixing unit, 131 failed location candidate determination unit, 132 failed location candidate determination unit, 133 failed location determination unit

Claims (12)

A motion vector on the input first frame is detected, the detected motion vector is allocated to a pixel on the second frame, and on the second frame based on the allocated motion vector In an image processing device that generates pixel values of pixels,
The motion vector assigned to the pixel of interest that is the pixel on the second frame in which the pixel value is generated and is the pixel of interest, and a predetermined first range around the pixel of interest First calculation means for calculating a value indicating variation with the motion vector assigned to the first peripheral pixel which is a pixel of
From the pixel value of the target pixel and the pixel value of a second peripheral pixel that is a pixel on the second frame and that is a pixel in a predetermined second range around the target pixel, in the spatial direction Second calculating means for calculating a value indicating a change in the pixel value of
Determination means for determining whether or not the image of the second frame is broken in the target pixel based on a value indicating the variation of the motion vector and a value indicating a change in the pixel value in the spatial direction;
The second frame is arranged between the first frame and the input pixel value of the pixel on the third frame, and the position of the pixel at the same position as the position of the pixel on the third frame. The time from the time of the first frame to the time of the second frame, and the time from the time of the second frame to the time of the third frame from the pixel value of the pixel on one frame Generating means for generating an interpolated value corresponding to a pixel on the second frame according to the ratio of:
When it is determined that the image of the second frame is broken at the target pixel, the interpolation value corresponding to the target pixel is mixed with the pixel value of the target pixel according to a predetermined ratio, and A pixel value of a third peripheral pixel that is a pixel on a second frame and is a pixel in a predetermined third range around the pixel of interest, and the interpolation value corresponding to the third peripheral pixel An image processing apparatus comprising: a mixing unit that mixes. The first calculation means is a motion vector that matches the motion vector assigned to the target pixel as a value indicating variation, and that is the motion vector assigned to the first peripheral pixel. The image processing apparatus according to claim 1, wherein the number is calculated. The first calculating means is a motion vector having a magnitude that falls within a range defined by a component of the motion vector assigned to the target pixel as a value indicating variation, and The image processing apparatus according to claim 2, wherein the number of motion vectors assigned is calculated. The second calculating means calculates a maximum absolute value of absolute values of differences between a pixel value of the target pixel and a pixel value of the second peripheral pixel as a value indicating a change in a pixel value in a spatial direction. The image processing apparatus according to claim 1. The determination means includes
First determining whether or not the image of the second frame at the pixel of interest is a failed image candidate based on a value indicating the variation of the motion vector and a predetermined first threshold. A failed image candidate determination means,
Based on a value indicating a change in the pixel value in the spatial direction and a predetermined second threshold value, it is determined whether or not the image of the second frame at the target pixel is a failed image candidate. A second failed image candidate determination means;
Whether or not the image of the second frame has failed in the target pixel is determined based on the determination result in the first failed image candidate determination unit and the determination result in the second failed image candidate determination unit. The image processing apparatus according to claim 1, further comprising: a failure location determination unit that determines. The maximum absolute value of the absolute values of the difference between the pixel value of the target pixel and the pixel value of the second peripheral pixel when each pixel on the second frame is sequentially set as the target pixel. The image processing apparatus according to claim 5, further comprising a threshold value calculation unit that calculates the second threshold value from a distribution. The apparatus further comprises a determining unit that determines the ratio of mixing the interpolation value corresponding to the target pixel or the third peripheral pixel into the pixel value of the target pixel or the pixel value of the third peripheral pixel. An image processing apparatus according to 1. The image processing apparatus according to claim 7, wherein the determination unit determines the ratio based on ratio data stored in advance corresponding to the distance from the target pixel. In the case where two or more of the ratios are obtained for one of the target pixel or the third peripheral pixel, the determining unit determines the final ratio as a larger one of the obtained ratios. The image processing apparatus according to claim 8. A motion vector on the input first frame is detected, the detected motion vector is allocated to a pixel on the second frame, and on the second frame based on the allocated motion vector In an image processing method for generating a pixel value of a pixel,
The motion vector assigned to the pixel of interest that is the pixel on the second frame in which the pixel value is generated and is the pixel of interest, and a predetermined first range around the pixel of interest A value indicating variation with the motion vector assigned to the first peripheral pixel that is a pixel of
From the pixel value of the target pixel and the pixel value of a second peripheral pixel that is a pixel on the second frame and that is a pixel in a predetermined second range around the target pixel, in the spatial direction A value indicating the change in the pixel value of
Based on the value indicating the variation of the motion vector and the value indicating the change in the pixel value in the spatial direction, it is determined whether or not the image of the second frame is broken in the target pixel,
The second frame is arranged between the first frame and the input pixel value of the pixel on the third frame, and the position of the pixel at the same position as the position of the pixel on the third frame. The time from the time of the first frame to the time of the second frame, and the time from the time of the second frame to the time of the third frame from the pixel value of the pixel on one frame And generating an interpolated value corresponding to the pixel on the second frame according to the ratio of
When it is determined that the image of the second frame is broken at the target pixel, the interpolation value corresponding to the target pixel is mixed with the pixel value of the target pixel according to a predetermined ratio, and A pixel value of a third peripheral pixel that is a pixel on a second frame and is a pixel in a predetermined third range around the pixel of interest, and the interpolation value corresponding to the third peripheral pixel An image processing method including the step of mixing. A motion vector on the input first frame is detected, the detected motion vector is allocated to a pixel on the second frame, and on the second frame based on the allocated motion vector In a program for causing a computer to perform image processing for generating pixel values of pixels,
The motion vector assigned to the pixel of interest that is the pixel on the second frame in which the pixel value is generated and is the pixel of interest, and a predetermined first range around the pixel of interest A value indicating variation with the motion vector assigned to the first peripheral pixel that is a pixel of
From the pixel value of the target pixel and the pixel value of a second peripheral pixel that is a pixel on the second frame and that is a pixel in a predetermined second range around the target pixel, in the spatial direction A value indicating the change in the pixel value of
Based on the value indicating the variation of the motion vector and the value indicating the change in the pixel value in the spatial direction, it is determined whether or not the image of the second frame is broken in the target pixel,
The second frame is arranged between the first frame and the input pixel value of the pixel on the third frame, and the position of the pixel at the same position as the position of the pixel on the third frame. The time from the time of the first frame to the time of the second frame, and the time from the time of the second frame to the time of the third frame from the pixel value of the pixel on one frame And generating an interpolated value corresponding to the pixel on the second frame according to the ratio of
When it is determined that the image of the second frame is broken at the target pixel, the interpolation value corresponding to the target pixel is mixed with the pixel value of the target pixel according to a predetermined ratio, and A pixel value of a third peripheral pixel that is a pixel on a second frame and is a pixel in a predetermined third range around the pixel of interest, and the interpolation value corresponding to the third peripheral pixel A program that includes a step of mixing.   A recording medium on which the program according to claim 11 is recorded.

Images