JP4650684B2 - Image processing apparatus and method, program, and recording medium - Google Patents

Description

  The present invention relates to an image processing apparatus and method, a program, and a recording medium, and more particularly, to an image processing apparatus and method, a program, and a recording medium for processing an image.

  Conventionally, a frame memory for storing a television signal, a motion vector detection circuit for detecting the motion of the television signal read from the frame memory, a motion vector averaging circuit for obtaining an average of a plurality of motion vector data therefrom, An image conversion apparatus including a position correction circuit that performs frame interpolation processing on average motion vector data and a television signal read from a frame memory, a multiplication circuit, an addition circuit, and a correction signal generation circuit has been proposed (for example, Patent Document 1).

  The motion part / motion amount detection circuit detects the motion part and the motion amount from the MUSE signal and the motion vector, and the interframe interpolation circuit interpolates a frame in consideration of the motion vector into the still image of the MUSE signal, and the LPF. In the inter-field interpolation circuit via the sampling frequency conversion circuit, the field is interpolated between the fields, and in the class classification adaptive interpolation circuit, the class classification is performed from the level distribution pattern and edge data of the peripheral pixels to be interpolated, Using the read coefficient data, field interpolation and / or frame interpolation processing is performed and supplied to the mixing circuit via the sampling frequency conversion circuit. Some ratios are determined so that still images and motion images are mixed ( Patent reference 2).

JP-A-6-121290

Japanese Patent Laid-Open No. 9-172621

  However, when the vector used for interpolation is incorrect, an original edge or the like may occur in the generated image, and the generated image may break down. Such a failure is noticeable as image degradation, and the subjective image quality of the image is degraded.

  The present invention has been made in view of such a situation, and prevents the generated image from becoming inconspicuous.

  An image processing apparatus according to an aspect of the present invention detects an input motion vector on a first frame, assigns the detected motion vector to a pixel on a second frame, and assigns the assigned motion vector. An image processing device that generates a pixel value of a pixel on the second frame based on the second frame, the pixel on the second frame from which the pixel value is generated, and a pixel of interest First calculation means for calculating a value indicating a variation between the motion vector allocated to the target pixel and the motion vector allocated to the peripheral pixel that is a pixel in a predetermined range around the target pixel. Determining means for determining whether or not the image of the second frame is broken at the target pixel based on the value indicating the variation of the motion vector; and the image of the second frame If is it is determined that the collapse, from an image of a predetermined range including the target pixel in the second frame, and a removal means for removing high-frequency components of the image.

  The second frame is disposed between the first frame and a pixel on the input third frame, the second position being the same as the position of the target pixel on the second frame. A second value that calculates a change in the pixel value in the time direction from the pixel value of the pixel on the third frame and the pixel value of the pixel on the first frame at the same position as the position of the target pixel; A calculating means is further provided, wherein the determining means determines whether the image of the second frame has failed at the target pixel based on a value indicating variation in the motion vector and a value indicating change in the pixel value in the time direction. It can be determined whether or not.

  Selection means for selecting a filter to be applied to the image in the range including the target pixel in the second frame, based on a value indicating variation in the motion vector, from a plurality of filters that remove high frequency components of the image. Further, the removing means may remove the high frequency component of the image by applying the selected filter to the image in the range including the target pixel in the second frame.

  The selection unit may select a filter that removes more high-frequency components of an image when the motion vectors are more varied.

  When the motion vector is assigned to every n pixels on the second frame, the calculating means assigns the motion vector assigned to the surrounding pixels separated from the target pixel by n pixels. Then, a value indicating a variation from the motion vector assigned to the target pixel can be calculated.

  Based on the motion vector assigned to the detection means for detecting the motion vector on the first frame, the assignment means for assigning the detected motion vector to the pixels on the second frame, Generation means for generating pixel values of pixels on the second frame from the first frame and the third frame may be further provided.

  The image processing method according to one aspect of the present invention detects an input motion vector on a first frame, assigns the detected motion vector to a pixel on a second frame, and assigns the assigned motion vector. An image processing method for generating a pixel value of a pixel on the second frame based on the pixel, the pixel on the second frame from which the pixel value is generated, and a pixel of interest Calculating a value indicating a variation between the motion vector assigned to the target pixel and the motion vector assigned to a peripheral pixel that is a pixel in a predetermined range around the target pixel; Based on a value indicating variation, it is determined whether or not the image of the second frame has failed in the target pixel, and if it is determined that the image of the second frame has failed , From an image of a predetermined range including the target pixel in the second frame, comprising the steps of removing a high-frequency component of the image.

  A program according to an aspect of the present invention detects an input motion vector on a first frame, assigns the detected motion vector to a pixel on a second frame, and based on the assigned motion vector A program for causing a computer to perform image processing for generating a pixel value of a pixel on the second frame, the pixel on the second frame from which the pixel value is generated, Calculating a value indicating a variation between the motion vector assigned to the pixel of interest that is a pixel and the motion vector assigned to a peripheral pixel that is a pixel in a predetermined range around the pixel of interest; Based on the value indicating the variation of the motion vector, it is determined whether the image of the second frame is broken at the target pixel, and the image of the second frame is determined. If is it is determined that the collapse, from an image of a predetermined range including the target pixel in the second frame, comprising the steps of removing a high-frequency component of the image.

  The recording medium of one aspect of the present invention detects an input motion vector on the first frame, assigns the detected motion vector to a pixel on the second frame, and assigns the assigned motion vector to the motion vector. A program for causing a computer to perform image processing for generating a pixel value of a pixel on the second frame based on the pixel on the second frame from which the pixel value is generated. Calculating a value indicating a variation between the motion vector assigned to the target pixel that is a target pixel and the motion vector assigned to a peripheral pixel that is a pixel in a predetermined range around the target pixel. And determining whether the image of the second frame is broken at the target pixel based on the value indicating the variation of the motion vector, and determining whether the image of the second frame If it is determined that the broken, the image of a predetermined range including the target pixel in the second frame, the program comprising the step of removing high frequency components of the image are recorded.

  In one aspect of the present invention, the motion vector assigned to a pixel of interest that is a pixel on the second frame in which a pixel value is generated and is a pixel of interest, and the periphery of the pixel of interest A value indicating a variation with the motion vector assigned to a peripheral pixel that is a pixel in a predetermined range is calculated, and based on the value indicating the variation in the motion vector, the second frame at the target pixel is calculated. If it is determined whether or not the image of the second frame is broken, and it is determined that the image of the second frame is broken, an image of a predetermined range including the target pixel in the second frame Is removed.

  As described above, according to one aspect of the present invention, an image can be generated.

  Also, according to one aspect of the present invention, it is possible to make the generated image inconspicuous.

  Embodiments of the present invention will be described below. Correspondences between the configuration requirements of the present invention and the embodiments described in the detailed description of the present invention are exemplified as follows. This description is to confirm that the embodiments supporting the present invention are described in the detailed description of the invention. Accordingly, although there are embodiments that are described in the detailed description of the invention but are not described here as embodiments corresponding to the constituent elements of the present invention, It does not mean that the embodiment does not correspond to the configuration requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  In the image processing apparatus according to the aspect of the present invention, firstly, the motion vector assigned to a target pixel that is a pixel on the second frame in which a pixel value is generated and is a target pixel. And a first calculation means for calculating a value indicating a variation between the motion vector assigned to the peripheral pixels that are pixels in a predetermined range around the target pixel (for example, the vector feature amount calculation in FIG. 20) Unit 141) and determination means for determining whether or not the image of the second frame has failed in the pixel of interest based on the value indicating the variation of the motion vector (for example, the failure point determination unit in FIG. 20) 161), when it is determined that the image of the second frame is broken, the removal of removing the high frequency component of the image from the image in a predetermined range including the target pixel in the second frame And a stage (e.g., the filter processing unit 146 in FIG. 20).

  An image processing apparatus according to an aspect of the present invention is secondly provided with pixels on the input third frame in which the second frame is arranged between the first frame and the first frame. From the pixel value of the pixel on the third frame at the same position as the position of the target pixel on the second frame and the pixel value of the pixel on the first frame at the same position as the position of the target pixel A second calculation unit (for example, the image feature amount calculation unit 143 in FIG. 12) that calculates a value indicating the change in the pixel value in the direction is further provided, and the determination unit (for example, the failure location determination unit 142 in FIG. 12) includes Determining whether or not the image of the second frame is broken at the target pixel based on the value indicating the variation in the motion vector and the value indicating the change in the pixel value in the time direction. it can.

  Thirdly, the image processing apparatus according to one aspect of the present invention is configured to detect the target pixel in the second frame based on a value indicating the variation in the motion vector from a plurality of filters that remove a high frequency component of the image. The image processing apparatus further includes a selection unit (for example, a filter selection unit 145 in FIG. 12) that selects a filter to be applied to the image in the range including the second frame. By applying the selected filter to the image in the range including the pixel of interest in, the high frequency component of the image can be removed.

  Fourthly, an image processing apparatus according to one aspect of the present invention includes a detection unit (for example, the vector detection unit 52 in FIG. 2) that detects the motion vector on the first frame, and the second frame. From the first frame and the third frame based on the allocation means (for example, the vector allocation unit 54 in FIG. 2) that allocates the detected motion vector to the pixel, and the allocated motion vector, Generation means (for example, the image interpolation unit 58 in FIG. 2) for generating pixel values of pixels on the second frame can be further provided.

  The image processing method or program according to one aspect of the present invention is the pixel on the second frame in which the pixel value is generated, and the motion vector assigned to the pixel of interest that is the pixel of interest; A value indicating variation with the motion vector allocated to the peripheral pixels that are pixels in a predetermined range around the target pixel is calculated (for example, step S141 in FIG. 18), and the variation in the motion vector is indicated. Based on the value, it is determined whether or not the image of the second frame has failed in the target pixel (for example, step S143 in FIG. 18), and it is determined that the image of the second frame has failed. If so, the high frequency component of the image is removed from the image in the predetermined range including the target pixel in the second frame (for example, step S145 in FIG. 18). Including the flop.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a configuration example of an image processing apparatus 1 to which the present invention is applied. The image processing apparatus 1 is composed of, for example, a personal computer. In FIG. 1, a CPU (Central Processing Unit) 11 executes various processes according to a program stored in a ROM (Read Only Memory) 12 or a storage unit 18. A RAM (Random Access Memory) 13 appropriately stores programs executed by the CPU 11 and data. The CPU 11, ROM 12, and RAM 13 are connected to each other by a bus 14.

  An input / output interface 15 is also connected to the CPU 11 via the bus 14. The input / output interface 15 is connected to an input unit 16 including a keyboard, a mouse, a microphone, and the like, and an output unit 17 including a display and a speaker. The CPU 11 executes various processes in response to commands input from the input unit 16. Then, the CPU 11 outputs the image and sound obtained as a result of the processing to the output unit 17.

  The storage unit 18 connected to the input / output interface 15 is composed of, for example, a hard disk and stores programs executed by the CPU 11 and various data. The communication unit 19 communicates with an external device via the Internet or other networks. A program may be acquired via the communication unit 19 and stored in the storage unit 18.

  The drive 20 connected to the input / output interface 15 drives a removable medium 21 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the programs and data recorded therein. Get etc. The acquired program and data are transferred to and stored in the storage unit 18 as necessary.

  The image processing apparatus 1 can be, for example, a television receiver, a hard disk player, an optical disk player, a portable player (cellular phone), or the like, or an image processing unit thereof.

  FIG. 2 is a block diagram illustrating an example of a functional configuration of the image processing apparatus 1. The image processing apparatus 1 includes a frame memory 51, a vector detection unit 52, a detection vector memory 53, a vector allocation unit 54, an allocation vector memory 55, an allocation flag memory 56, an allocation compensation unit 57, an image interpolation unit 58, an entire frame post-processing unit 59, and a failed part post-processing unit 60.

  It does not matter whether each function of the image processing apparatus 1 is realized by hardware or software. That is, each block diagram in this specification may be considered as a hardware block diagram or a software functional block diagram.

  In the image processing apparatus 1 having the configuration shown in FIG. 2, for example, an image of a progressive image signal (hereinafter referred to as a 24P signal) with a frame frequency of 24 Hz is input, and the input image (input image) has a frame frequency of 60 Hz. It is converted into an image of a progressive image signal (hereinafter referred to as 60P signal) and output.

  The input image of the 24P signal input to the image processing apparatus 1 includes a frame memory 51, a vector detection unit 52, a vector allocation unit 54, an allocation compensation unit 57, an image interpolation unit 58, an entire frame post-processing unit 59, and a failure location It is supplied to the processing unit 60. The frame memory 51 stores input images in units of frames. The frame memory 51 stores a frame at time t one before the input image at time t + 1. The frame at time t stored in the frame memory 51 is supplied to the vector detection unit 52, the vector allocation unit 54, the allocation compensation unit 57, the image interpolation unit 58, the entire frame post-processing unit 59, and the failed part post-processing unit 60. The Hereinafter, the frame at time t on the frame memory 51 is referred to as frame t, and the frame of the input image at time t + 1 is referred to as frame t + 1.

  The vector detection unit 52 detects a motion vector between the target block of the frame t on the frame memory 51 and the target block of the frame t + 1 of the input image, and stores the detected motion vector in the detection vector memory 53. As a method for detecting the motion vector between the two frames, a gradient method or a block matching method is used. The detection vector memory 53 stores the motion vector detected by the vector detection unit 52 in the frame t.

  The vector allocation unit 54 distinguishes the motion vector obtained on the frame t of the 24P signal from the frame of the 60P signal to be interpolated in the allocation vector memory 55 (hereinafter, the frame of the 60P signal is distinguished from the frame of the 24P signal). , Which is also referred to as an interpolation frame), and the allocation flag of the allocation flag memory 56 of the pixel to which the motion vector is allocated is rewritten to 1 (True).

  The allocation vector memory 55 stores the motion vector allocated by the vector allocation unit 54 in association with each pixel of the interpolation frame. The allocation flag memory 56 stores an allocation flag indicating the presence / absence of a motion vector to be allocated for each pixel of the interpolation frame. For example, an assignment flag of True (1) indicates that a motion vector is assigned to the corresponding pixel, and an assignment flag of False (0) indicates that a motion vector is not assigned to the corresponding pixel. Show.

  The allocation compensation unit 57 refers to the allocation flag of the allocation flag memory 56, and supplements the motion vector of the peripheral pixel of the target pixel with respect to the target pixel for which the motion vector has not been allocated by the vector allocation unit 54. Allocate on 55 interpolation frames. At this time, the allocation compensator 57 rewrites the allocation flag of the target pixel to which the motion vector is allocated to 1 (True).

  The image interpolation unit 58 interpolates and generates the pixel value of the interpolation frame using the motion vector allocated to the interpolation frame in the allocation vector memory 55 and the pixel values of the frame t and the next frame t + 1. Then, the image interpolation unit 58 outputs the generated interpolation frame to the entire frame post-processing unit 59, and then outputs the frame t + 1 to the entire frame post-processing unit 59 as necessary. Hereinafter, the image output from the image interpolation unit 58 is also referred to as an interpolation image.

  The whole frame post-processing unit 59 uses the predetermined mixing ratio and the pixel values of the frame t and the next frame t + 1 to perform post-processing that prevents the failure from becoming noticeable. Applies to the entire image interpolation frame. The entire frame post-processing unit 59 supplies the image to which the post-processing is applied to the failed portion post-processing unit 60.

  The failure location post-processing unit 60 uses the motion vector assigned to the interpolation frame in the assignment vector memory 55, the predetermined threshold value, and the pixel values of the frame t and the next frame t + 1 to be included in the interpolation frame. The post-processing is applied to the broken image, and the 60P signal image is output to a subsequent stage (not shown).

  Note that the mixture ratio and the threshold value may be supplied from the outside or may be stored inside. “Predetermined” means that it is predetermined before the execution of each process, and means that it is determined at least when the process is executed. Therefore, changing the mixture ratio and the threshold before the processing in each block is executed corresponds to the predetermined.

  FIG. 3 is a diagram for explaining the principle of processing for interpolating frames in the image processing apparatus 1 according to the present invention. In the example of FIG. 3, a dotted line represents a frame of a 24P signal that is input to the image processing apparatus 1 at times t, t + 1, and t + 2, and a solid line is represented by the image processing apparatus 1 from the input 24P signal. , The 60P signal interpolation frames at the generated times t, t + 0.4, t + 0.8, t + 1.2, t + 1.6, and t + 2.

  Generally, in order to convert a 24P signal into a 60P signal, 5/2 times as many frames are required. That is, five 60P signal images must be generated from two 24P signal images. At this time, the generated interpolated frame of the 60P signal has a time phase of 0.0, 0.4, 0.8, 1.2, and 1 on the 24P signal in order to equalize the frame interval. 6 is arranged at a position. Among these, 4 frames (frames of t + 0.4, t + 0.8, t + 1.2, and t + 1.6) except for one frame at time t with a time phase of 0.0 do not exist on the 24P signal. It is an image. Therefore, when the image of the 24P signal is input, the image processing apparatus 1 generates four interpolation frames from the two frames at time t and time t + 1 of the 24P signal. Therefore, the image processing apparatus 1 outputs a 60P signal image including five frames at times t, t + 0.4, t + 0.8, t + 1.2, and t + 1.6.

  As described above, the image processing apparatus 1 executes the process of converting the frame frequency from the 24P signal image to the 60P signal image.

  In principle, as described above, from the two frames at time t and time t + 1 of the 24P signal, five frames at times t, t + 0.4, t + 0.8, t + 1.2, and t + 1.6 are obtained. A frame of 60P signal is newly generated. Actually, in the example of FIG. 3, 60P of t, t + 0.4, and t + 0.8 is based on two frames at time t and time t + 1 of the 24P signal. A frame of the signal is generated, and a frame of the 60P signal of t + 1.2, t + 1.6, and t + 2 is generated based on the two frames at the times t + 1 and t + 2 of the 24P signal.

  Also, hereinafter, an interpolation frame at time t + k such as time t + 0.4, t + 0.8, t + 1.2, and t + 1.6 is referred to as an interpolation frame t + k.

  FIG. 4 is a diagram for more specifically explaining the process of interpolating a frame. In the example of FIG. 4, a thick arrow represents a transition to each state, and an arrow T represents a time passage direction in the states 81 to 85. In states 81 to 85, the frame t at the time t of the 24P signal, the frame t + 1 at the time t + 1 next to the time t, or the frame t and the frame t + 1 at the time of input / output to each unit constituting the image processing apparatus 1 The state of the interpolation frame F of the 60P signal produced | generated during is shown notionally. That is, actually, for example, a frame in which a motion vector is detected as shown in the state 82 is not input, and the frame and the motion vector are input separately.

  The state 81 represents the state of the frame t and the frame t + 1 of the 24P signal input to the vector detection unit 52. A black dot on the frame t in the state 81 represents a pixel on the frame t. The vector detection unit 52 detects the position where the pixel on the frame t in the state 81 moves in the frame t + 1 of the next time, and the movement is indicated as shown on the frame t in the state 82. It outputs as a motion vector corresponding to each pixel. As a method for detecting the motion vector between the two frames, a block matching method or a gradient method is used. At this time, for example, when a plurality of motion vectors are detected in the pixel, the vector detection unit 52 obtains an evaluation value for each motion vector, and selects a motion vector based on the evaluation value.

  A state 82 represents the state of the frame t and the frame t + 1 input to the vector allocation unit 54. In the state 82, the arrow of each pixel of the frame t represents the motion vector detected by the vector detection unit 52.

  The vector allocating unit 54 extends the motion vector detected for each pixel of the frame t in the state 82 to the next frame t + 1 and sets it to a preset time phase (for example, t + 0.4 in FIG. 3). A position on a certain interpolation frame F is obtained. This is because, assuming that there is a constant motion between the frame t and the frame t + 1, the point where the motion vector passes through the interpolation frame F is the pixel position in that frame. Therefore, the vector allocating unit 54 allocates the passing motion vector to four neighboring pixels on the interpolation frame F in the state 83. At this time, depending on the pixel of the interpolation frame, there may be a case where no motion vector exists, or a plurality of motion vectors may be candidates for allocation. In the latter case, for example, like the vector detection unit 52, the vector allocation unit 54 obtains an evaluation value for each motion vector and selects a motion vector to be allocated based on the evaluation value.

  A state 83 represents the state of the frame t and the frame t + 1 and the interpolation frame F to which the motion vector is assigned, which is input to the assignment compensation unit 57. In the interpolation frame F in the state 83, the pixels to which the motion vector is assigned by the vector assigning unit 54 and the pixels to which the motion vector is not assigned are shown.

  The allocation compensation unit 57 compensates for a pixel to which a motion vector in the state 83 is not allocated by using a motion vector allocated to a peripheral pixel of the pixel. This is because the motion vector of the peripheral pixel of the pixel of interest and the motion vector of the pixel of interest are similar if the assumption that the neighboring region of the pixel of interest has the same motion holds. As a result, a motion vector that is accurate to some extent is also given to a pixel to which no motion vector is assigned, and a motion vector is assigned to all the pixels on the interpolation frame F in the state 84. In this case as well, since motion vectors of a plurality of surrounding pixels exist as candidates, the allocation compensation unit 57 obtains an evaluation value for each motion vector, for example, similarly to the vector allocation unit 54, and the evaluation value A motion vector to be allocated is selected based on

  A state 84 represents the state of the frame t and the frame t + 1 input to the image interpolation unit 58 and the state of the interpolation frame F in which motion vectors are assigned to all the pixels. Based on the motion vectors assigned to all these pixels, the image interpolation unit 58 can determine the positional relationship between the pixels on the interpolation frame F and the two frames t and t + 1. Therefore, the image interpolation unit 58 uses the motion vector allocated on the interpolation frame F and the pixel values of the frame t and the frame t + 1 to interpolate as indicated by the black dot of the interpolation frame F in the state 85. Pixel values on the frame F are generated by interpolation. Then, the image interpolation unit 58 outputs the generated interpolated frame, and then outputs a frame t + 1 as necessary, thereby outputting an interpolated image of the 60P signal to the subsequent stage.

  Next, the process of converting the frame frequency of the image processing apparatus 1 will be described with reference to the flowchart of FIG.

  In step S1, the vector detection unit 52 inputs the pixel value of the frame t + 1 of the input image at the time t + 1 and the pixel t of the frame t at the time t immediately before the input image of the frame memory 51, and proceeds to step S2. At this time, the vector allocating unit 54, the allocation compensating unit 57, the image interpolating unit 58, the entire frame post-processing unit 59, and the failed part post-processing unit 60 The pixel value of frame t at time t immediately before the input image is input.

  In step S2, the vector detection unit 52 detects a motion vector and proceeds to step S3. That is, the vector detection unit 52 detects a motion vector between the target block of the frame t on the frame memory 51 and the target block of the next frame t + 1 that is the input image, and the detected motion vector is detected by the detection vector memory 53. And proceed to step S3. As a method for detecting the motion vector between the two frames, a gradient method or a block matching method is used.

  In step S3, the vector allocation unit 54 allocates a motion vector to the pixel of interest on the interpolation frame to be interpolated, and the process proceeds to step S4. That is, in step S3, the vector allocation unit 54 allocates the motion vector obtained on the frame t to the pixel of interest on the interpolation frame to be interpolated in the allocation vector memory 55, and the pixel to which the motion vector is allocated. The allocation flag in the allocation flag memory 56 is rewritten to 1 (True). For example, an allocation flag that is True indicates that a motion vector is allocated to the corresponding pixel, and an allocation flag that is False indicates that a motion vector is not allocated to the corresponding pixel.

  In step S4, the allocation compensator 57 compensates the allocation and proceeds to step S5. That is, the allocation compensation unit 57 refers to the allocation flag in the allocation flag memory 56 in step S4, and for the pixel of interest for which no motion vector has been allocated by the vector allocation unit 54, the motion vector of the surrounding pixels of the pixel of interest. Is allocated on the interpolation frame of the allocation vector memory 55. At this time, the allocation compensator 57 compensates the motion vector and rewrites the allocation flag of the allocated pixel of interest to 1 (True).

  In step S5, the image interpolation unit 58 executes an image interpolation process. That is, in step S5, the image interpolation unit 58 uses the motion vector allocated to the interpolation frame of the allocation vector memory 55 and the pixel values of the frame t and the frame t + 1 to generate an interpolation frame pixel value. The process proceeds to step S6. Details of the image interpolation processing in step S5 will be described later. The image interpolation unit 58 outputs the generated interpolation frame, and then outputs a frame t + 1 as necessary, thereby outputting an interpolation image of a 60P signal.

  In step S6, the entire frame post-processing unit 59 performs post-processing of the entire frame for the interpolated frame, and the process proceeds to step S7. Details of post-processing of the entire frame will be described later.

  In step S7, the failed part post-processing unit 60 performs post-processing of the failed part for the interpolated frame, and proceeds to step S8. Details of post-processing of the failed part will be described later.

  In step S8, the failed location post-processing unit 60 outputs an interpolated image of the post-processed 60P signal.

  In step S9, the vector detection unit 52 determines whether or not all the frames have been processed. If it is determined that all the frames have not been processed, the vector detection unit 52 returns to step S1 and performs the subsequent processing. repeat. On the other hand, when the vector detection unit 52 determines in step S9 that all the frames have been processed, the vector detection unit 52 ends the process of converting the frame frequency.

  As described above, the image processing apparatus 1 according to the present invention detects a motion vector from a frame of an input image of a 24P signal, assigns the detected motion vector to a pixel on a frame of a 60P signal, and assigns the assigned motion vector. Based on the above, the pixel value on the frame of the 60P signal is generated. Then, the image processing apparatus 1 applies post-processing to the entire frame or the failure location of the interpolated frame, and outputs an image that is less prominent of the failure.

  Next, details of the configuration of the image interpolation unit 58 will be described.

  FIG. 6 is a block diagram illustrating a configuration of the image interpolation unit 58. 6 interpolates and generates the pixel value of the interpolation frame using the motion vector allocated to the interpolation frame in the allocation vector memory 55 and the pixel values of the frame t and the frame t + 1. , 60P signal interpolation processing is performed.

  In the example of FIG. 6, the frame t of the image at time t is input to the spatial filter 92-1, and the frame t + 1 of the image at time t + 1 is input to the spatial filter 92-2 and the buffer 95.

  The interpolation control unit 91 selects the pixel of the interpolation frame in the allocation vector memory 55, and based on the motion vector assigned to the selected pixel, the pixel on the interpolation frame, the two frames t and t + 1 The positional relationship (spatial shift amount) with each pixel is obtained. That is, the interpolation control unit 91 uses, based on the pixels of the interpolation frame, the position on the frame t associated with the motion vector and the position of the pixel on the frame t corresponding to the pixel of the interpolation frame, The spatial shift amount is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-1. Similarly, the interpolation control unit 91 uses, based on the pixels of the interpolation frame, the position on the frame t + 1 associated with the motion vector and the position of the pixel on the frame t + 1 corresponding to the pixel of the interpolation frame. Is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-2.

  Further, the interpolation control unit 91 obtains an interpolation weight between the frame t and the frame t + 1 based on a preset time phase (time) of the interpolation frame, and uses the obtained interpolation weight as a multiplier 93-1. And 93-2. For example, when the time of the interpolation frame is a time that is “k” away from the time t + 1 of the frame t + 1 and a time that is “1−k” away from the time t of the frame t (ie, the time of the interpolation frame is the time t and time t + 1 is “1−k”: generated at a time that is internally divided into “k”), the interpolation control unit 91 sets an interpolation weight of “1-k” in the multiplier 93-1. An interpolation weight of “k” is set in the multiplier 93-2.

  Spatial filters 92-1 and 92-2 are constituted by cubic filters, for example. The spatial filter 92-1 is a pixel value on the frame t corresponding to the pixel of the interpolation frame based on the input pixel value of the pixel on the frame t and the spatial shift amount supplied from the interpolation control unit 91. And the obtained pixel value is output to the multiplier 93-1. The spatial filter 92-2, based on the input pixel value of the pixel on the frame t + 1 and the spatial shift amount supplied from the interpolation control unit 91, the pixel value on the frame t + 1 corresponding to the pixel of the interpolation frame And the obtained pixel value is output to the multiplier 93-2.

  When the position of the pixel in the interpolation frame does not match the position of the pixel on frame t or frame t + 1 (that is, when the position of the pixel in the interpolation frame is a component equal to or less than the pixel in frame t or frame t + 1). The spatial filters 92-1 and 92-2 obtain the sum of the inverse ratios of the distances of the four surrounding pixels by using the pixel values of the four surrounding pixels at the position of the pixel of the interpolation frame in the frame t or the frame t + 1. Then, the pixel value on the frame corresponding to the pixel of the interpolation frame is obtained. That is, the pixel values at the positions below the pixel are obtained by linear interpolation based on the distance from the surrounding four pixels.

  The multiplier 93-1 multiplies the pixel value on the frame t input from the spatial filter 92-1 by the interpolation weight “1-k” set by the interpolation control unit 91, and the weighted pixel value is The result is output to the adder 94. The multiplier 93-2 multiplies the pixel value on the frame t + 1 input from the spatial filter 92-2 by the interpolation weight “k” set by the interpolation control unit 91, and adds the weighted pixel value to the adder. Output to 94.

  The adder 94 generates the pixel value of the pixel of the interpolation frame by adding the pixel value input from the multiplier 93-1 and the pixel value input from the multiplier 93-2. The pixel value of the interpolation frame is output to the buffer 95. The buffer 95 buffers the input frame t + 1. The buffer 95 outputs the generated interpolation frame, and then outputs a buffered frame t + 1 as necessary based on a preset time phase (time) of the 60P frame. Thus, the interpolated image of the 60P signal is output to the entire frame post-processing unit 59.

  Details of the image interpolation processing of the image interpolation unit 58 configured as described above will be described with reference to the flowchart of FIG.

  In step S51, the interpolation control unit 91 calculates interpolation weights (for example, “k” and “1-k”) of the interpolation frame between the frame t and the frame t + 1 based on the time phase of the interpolation frame to be processed. The obtained interpolation weights are set in the multipliers 93-1 and 93-2, respectively, and the process proceeds to step S52.

  For example, as illustrated in FIG. 8, when generating an interpolation frame at time t + k between a frame t at time t and a frame t + 1 at time t + 1, the interpolation control unit 91 uses interpolation weights of k and 1-k. And the obtained interpolation weights are set in the multipliers 93-1 and 93-2, respectively. In the example of FIG. 8, the dotted line represents the frame of the 24P signal input to the image processing apparatus 1 at time t and t + 1, and the solid line is represented by the image processing apparatus 1 from the input 24P signal. It shows an interpolation frame of a 60P signal at time t + k to be generated.

  In step S52, the interpolation control unit 91 selects a pixel of the interpolation frame in the allocation vector memory 55, and proceeds to step S53. Note that the pixels on the interpolation frame are selected in the raster scan order from the upper left pixel of the frame.

  In step S53, based on the motion vector assigned to the selected pixel, the interpolation control unit 91 determines the positional relationship between the pixels on the interpolation frame and the two frames t and t + 1 (space shift amount). ) And the obtained spatial shift amounts are supplied to the spatial filters 92-1 and 92-2, respectively, and the process proceeds to step S54.

  Specifically, in step S53, the interpolation control unit 91 uses the pixel of the interpolation frame as a reference, the position on the frame t associated with the motion vector, and the frame t corresponding to the pixel of the interpolation frame. The spatial shift amounts are obtained from the pixel positions, and the obtained spatial shift amounts are supplied to the spatial filter 92-1. Similarly, the interpolation control unit 91 uses, based on the pixels of the interpolation frame, the position on the frame t + 1 associated with the motion vector and the position of the pixel on the frame t + 1 corresponding to the pixel of the interpolation frame. Is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-2.

  The pixel value of the frame t of the image at time t is input to the spatial filter 92-1, and the pixel value of the frame t + 1 of the image at time t + 1 is input to the spatial filter 92-2. In step S54, the spatial filters 92-1 and 92-2 determine the pixels of the interpolation frame based on the pixel values of the pixels on the input frames t and t + 1 and the spatial shift amount supplied from the interpolation control unit 91. The pixel values on each frame corresponding to are obtained, the obtained pixel values are output to the multipliers 93-1 and 93-2, and the process proceeds to step S55.

  In Step S55, the multipliers 93-1 and 93-2 weight the interpolation weight set by the interpolation control unit 91 to the pixel value on each frame input from the spatial filter 92-1 or 92-2, The weighted pixel value is output to the adder 94, and the process proceeds to step S56. That is, the multiplier 93-1 multiplies the pixel value on the frame t input from the spatial filter 92-1 by the interpolation weight “1-k” set by the interpolation control unit 91, and weighted pixel values. Is output to the adder 94. The multiplier 93-2 multiplies the pixel value on the frame t + 1 input from the spatial filter 92-2 by the interpolation weight “k” set by the interpolation control unit 91, and adds the weighted pixel value to the adder. Output to 94.

  In step S56, the adder 94 adds the pixel value weighted by the multiplier 93-1 and the pixel value weighted by the multiplier 93-2 to generate the pixel value of the pixel of the interpolation frame. The generated pixel value is output to the buffer 95, and the process proceeds to step S57.

  For example, as shown in FIG. 8 by the processing of step S52 to step S56, the pixel value of the pixel q on the frame t associated with the motion vector v assigned to the selected pixel p in the interpolation frame and From the pixel value of the pixel r on the frame t + 1, the pixel value of the pixel p on the new interpolation frame t + k between the frame t and the frame t + 1 is generated.

  In step S57, the interpolation control unit 91 determines whether or not the processing for all the pixels on the interpolation frame is completed, and determines that the processing for all the pixels on the interpolation frame is not completed. If so, the process returns to step S52, and the subsequent processing is repeated. If the interpolation control unit 91 determines in step S57 that the processing for all the pixels on the interpolation frame has been completed, the interpolation control unit 91 ends the image interpolation processing.

  As described above, the pixel value of the interpolation frame is generated based on the motion vector assigned to the interpolation frame, and the interpolation frame is output by the buffer 95 in step S5 of FIG. In addition, if necessary, the frame t + 1 is output, whereby an interpolated image of the 60P signal is output to the entire frame post-processing unit 59.

  Note that the processing described with reference to the flowchart of FIG. 7 is also referred to as interpolation frame generation processing.

  Next, the entire frame post-processing unit 59 will be described.

  FIG. 9 is a block diagram illustrating an example of the configuration of the entire frame post-processing unit 59. The entire frame post-processing unit 59 includes a zero vector interpolation image generation unit 111 and an image mixing unit 112.

  The zero vector interpolation image generation unit 111 indicates the pixel value of the pixel of the frame t and the pixel value of the pixel of the frame t + 1 of the input image adjacent to the interpolation frame t + k indicated by the zero vector assigned to the pixel on the interpolation frame t + k. To a pixel value is generated according to the ratio of the time from the time t of the frame t to the time t + k of the interpolation frame t + k and the time from the time t + k of the interpolation frame t + k to the time t + 1 of the frame t + 1.

  Here, the zero vector is a motion vector whose spatial component is zero.

  That is, the 0 vector interpolation image generation unit 111 is a pixel on the frame t at the same position as the pixel of interest that is a pixel on the interpolation frame t + k and is a pixel of interest on the interpolation frame t + k. The pixel value is determined according to the ratio between the time from time t to time t + k and the time from time t + k to time t + 1 from the value and the pixel value of the pixel on the frame t + 1 at the same position as the target pixel. Generate.

  FIG. 10 is a diagram for explaining generation of pixel values in the 0 vector interpolation image generation unit 111. In the example of FIG. 10, the dotted line represents the frame of the 24P signal input to the image processing apparatus 1 at time t and t + 1, and the solid line is represented by the image processing apparatus 1 from the input 24P signal. It shows an interpolation frame of a 60P signal at time t + k to be generated. The motion vector v is assigned to the pixel p of the interpolation frame, and the pixel r on the frame t + 1 and the pixel q on the frame t are associated with each other by the motion vector v.

  A zero vector (0 vector) having a spatial component of 0 in the figure is the pixel p on the frame t at the same position as the position of the target pixel p on the interpolation frame and the same position as the position of the target pixel p. The pixel p on the frame t + 1 is associated.

As illustrated in FIG. 10, when paying attention to the pixel p on the interpolation frame, the 0 vector interpolation image generation unit 111 has the pixel value F _t of the pixel p on the frame t at the same position as the position of the pixel p. (P) and the pixel value F _{t + 1} (p) of the pixel p on the frame t + 1 at the same position as the position of the pixel p are acquired. Then, 0 vector interpolation image generating unit 111, the pixel value F _t (p), multiplied by the weight k corresponding to the period from time t to time t + k, the pixel value F _{t +} 1 (p), a time t + k Is multiplied by a weight 1-k according to the time from to time t + 1. Further, the 0 vector interpolation image generation unit 111 adds the results of these multiplications to generate a pixel value G _{t + k} (p).

That is, the pixel value G _{t + k} (p) _represented by Expression (1) is obtained by the 0 vector interpolation image generation unit 111.
G _{t + k} (p) = (1−k) F _t (p) + kF _{t + 1} (p) (1)

  The zero vector interpolation image generation unit 111 generates pixel values obtained by the calculation represented by Expression (1) for all the pixels of the interpolation frame. The 0 vector interpolation image generation unit 111 supplies a 0 vector interpolation image including pixels having the generated pixel values to the image mixing unit 112.

  The image mixing unit 112 mixes the pixel value generated by the 0 vector interpolation image generation unit 111 with the pixel value of the target pixel according to a predetermined mixing ratio (ratio). For example, the mixing ratio is determined in advance.

For example, when a predetermined mixture ratio γ is input from the outside, the image mixing unit 112 focuses on the pixel p and the pixel p of the 0 vector interpolation image supplied from the 0 vector interpolation image generation unit 111 The pixel value G _{t + k} (p) is multiplied by the mixture ratio γ, and the pixel value F _{t + k} (p) of the pixel p in the interpolation frame is multiplied by (1−γ). Then, the image mixing unit 112 sets a value obtained by adding these multiplication results as a pixel value F ′ _{t + k} (p) of the pixel p of the interpolation frame. Note that the mixing ratio γ is larger than 0 and smaller than 1.

That is, the image mixing unit 112 mixes the pixel value G _{t + k} (p) generated in the 0 vector interpolation image generation unit 111 with the pixel value of the target pixel p in accordance with the mixing ratio γ, thereby obtaining the expression ( The pixel value F ′ _{t + k} (p) of the pixel p of the interpolation frame shown in 2) is obtained.
F ′ _{t + k} (p) = (1−γ) F _{t + k} (p) + γG _{t + k} (p) (2)

  The image mixing unit 112 mixes the pixels of the 0 vector interpolation image according to the mixing ratio for all the pixels of the interpolation frame, as shown in Expression (2). The image mixing unit 112 outputs an interpolated image obtained as a result of mixing.

If the ratio of the pixel value G _{t + k} (p) generated by the 0 vector interpolation image generation unit 111 is increased, the motion of the image becomes unnatural, so 0.0 <γ <= 0.5. It is preferable to use the mixing ratio γ.

  More preferably, a mixing ratio γ of 0.25 <= γ <= 0.5 is used. In the experiment of the present inventor, it has been confirmed that when a mixture ratio γ of 0.25 <= γ <= 0.5 is used, it is possible to obtain an image that is more natural and less prone to image failure.

Note that when the image interpolation unit 58 calculates F _{t + k} (p), since the frame t and the frame t + 1 are stored, the entire frame post-processing unit 59 receives the frame t and the frame t + 1 from the image interpolation unit 58. If reading is performed, it is not necessary to prepare a new memory.

  Moreover, since the pixels of the 0 vector interpolation image are uniformly mixed with all the pixels of the interpolation frame, hardware or software can be realized more easily.

FIG. 11 is a flowchart for explaining details of post-processing of the entire frame corresponding to step S6 of FIG. In step S101, the 0 vector interpolation image generation unit 111 generates pixels of a 0 vector interpolation image. That is, for example, in step S101, the 0 vector interpolation image generation unit 111 calculates the pixel value G _{t + k} (p) of the pixel of the 0 vector interpolation image according to Equation (1), thereby Generate a pixel.

  In step S102, the image mixing unit 112 mixes the input interpolation image pixels and the zero vector interpolation image pixels in accordance with a predetermined mixture ratio γ. For example, in step S102, the image mixing unit 112 mixes the pixels of the 0 vector interpolation image with the pixels of the interpolation frame in accordance with the mixing ratio γ, as shown in Expression (2).

  In step S103, the entire frame post-processing unit 59 determines whether or not the process has been completed for all the pixels of the interpolated frame. If it is determined that the process has not been completed, the process returns to step S101. The process of step S101 and step S102 is repeated for the next pixel of the insertion frame. Note that the pixels to be processed in steps S101 and S102 are selected, for example, in raster scan order.

  If it is determined in step S103 that the processing has been completed for all the pixels in the interpolated frame, the post-processing for the entire frame ends.

  As described above, since the zero vector interpolation image is mixed for the entire interpolation frame, it is possible to make the image inconspicuous with a simple process without impairing the smoothness of the motion of the image. Since the failure of the image is less noticeable, the user viewing the image feels that the image quality of the image has been improved.

  Note that the interpolation image and the zero vector interpolation image may be mixed after the entire zero vector interpolation image is generated.

  Next, the failure location post-processing unit 60 will be described.

  FIG. 12 is a block diagram illustrating an example of the configuration of the failed part post-processing unit 60. The failed part post-processing unit 60 includes a vector feature amount calculating unit 141, a failed part determining unit 142, an image feature amount calculating unit 143, a filter coefficient ROM 144, a filter selecting unit 145, and a filter processing unit 146.

  The vector feature amount calculation unit 141 is a pixel on the interpolation frame in which the pixel value is generated by the image interpolation unit 58, and the motion vector assigned to the target pixel that is the target pixel and the target pixel A value indicating a variation from a motion vector assigned to a peripheral pixel that is a pixel on an interpolation frame in a predetermined peripheral range is calculated.

  An image collapse (deterioration) that occurs when generating an interpolation frame is empirically likely to occur in a portion where the motion vector assigned to the pixel is unclear (varied).

  The vector feature amount calculation unit 141 calculates a value indicating the variation of the motion vector assigned to the pixel of the generated image as an evaluation value indicating the possibility of image failure (deterioration).

For example, the vector feature amount calculation unit 141 calculates an irregularity evaluation value, which is an example of a value indicating the variation of the motion vector, for the target pixel px _{, y} in the interpolation frame t + k. Here, the irregularity evaluation value is a result of evaluating the degree of variation when the motion vector assigned to the pixel of interest px _{, y} is compared with the motion vector assigned to the surrounding pixels. It is shown.

  Hereinafter, a case where the vector feature amount calculation unit 141 calculates VDAS (Vector Difference Absolute Summation) as the irregularity evaluation value will be described as an example.

For example, as illustrated in FIG. 13, when the pixel of interest on the interpolation frame t + k is the pixel of interest _{px, y} , the vector feature amount calculation unit 141 is centered on the pixel of interest _{px, y} . A motion vector assigned to pixels in a predetermined range indicated by a one-dot chain line is read from the assigned vector memory 55.

More specifically, for example, when the vector allocating unit 54 allocates a motion vector for every two pixels, the vector feature amount calculating unit 141 focuses on the pixel of interest on the interpolation frame as shown in FIG. When the pixel is p _{x, y} , the motion vector assigned to the pixel in the range of 2 pixels up and down and 2 pixels left and right from the target pixel p _{x, y} is read from the assigned vector memory 55. That is, the vector feature amount calculation unit 141 has two pixels above the pixel of interest p _{x, y} , two pixels above and two pixels left of the pixel p _{x-2, y-2} , and two pixels of the pixel of interest p _{x, y} . For the upper pixel p _{x, y-2} and the target pixel p _{x, y} , two pixels on the right and two pixels p _{x + 2, y−2 above} the target pixel p _{x, y} and two for the _target pixel p _{x, y} left pixel p _{x-2, y,} the pixel of interest p _x, with respect to _y, the pixel of the 2 pixels to the right p _{x + 2, y,} the pixel of interest p _x, with respect to _y, 2 pixels left two pixels lower pixel p _{x-2, y + 2,} the pixel of interest p _x, 2 against _y, the pixel p _x 2 pixels _{lower, y + 2,} the pixel of interest p _x, with respect to _y, in two pixels lower The motion vector assigned to each of the pixel p _{x + 2, y + 2} on the right side of the pixel and the pixel of interest p _{x, y} is read from the assigned vector memory 55.

The vector feature amount calculation unit 141 calculates VDAS _{t + k} (p _{x, y} ) _, which is VDAS for the target pixel px _{, y} on the interpolation frame t + k, based on Expression (3).
VDAS _{t + k} (p _{x, y} ) = max (VDAS _{t + k} (p _{x, y} ) _x , VDAS _{t + k} (p _{x, y} ) _y )
... (3)
In equation (3), max (a, b) is a function that selects the larger of a and b.

VDAS _{t + k} (p _{x, y} ) _x is calculated by equation (4), and VDAS _{t + k} (p _{x, y} ) _y is calculated by equation (5).

・・・（４）

... (4)

・・・（５）
式（４）および式（５）において、ｉ＝−２，０，２であり、ｊ＝−２，０，２である。

... (5)
In the equations (4) and (5), i = −2, 0, 2 and j = −2, 0, 2.

  Thus, for example, when a motion vector is assigned to every n pixels on the interpolation frame, the vector feature amount calculation unit 141 is assigned to peripheral pixels that are separated from the target pixel by n pixels. A value indicating the variation between the motion vector currently assigned and the motion vector assigned to the target pixel is calculated.

  As described above, the vector feature amount calculation unit 141 calculates VDAS as the irregularity evaluation value.

  The vector feature amount calculation unit 141 supplies a value indicating variation in motion vectors, such as an irregularity evaluation value, to the failure location determination unit 142.

When VDAS is calculated for the chroma (color difference signal) of the so-called 4: 2: 2 image signal, it is assigned to pixels in the range of 4 pixels vertically and 4 pixels horizontally from the pixel of interest px _{, y.} The motion vector being read is read from the assigned vector memory 55, and the equation (3) is applied to the read motion vector to calculate VDAS. In this case, i = −4,0,4 and j = −4,0,4.

  In the case of a color difference signal, the VDAS value calculated at the position of the corresponding pixel in the luminance signal may be used.

  Further, in the equations (4) and (5), a square value may be used instead of the absolute value.

  Furthermore, the sum of Euclidean distances between motion vectors may be used for calculating the value of VDAS.

  In calculating the value of VDAS, the pixel range may be changed according to the type of input image. For example, the value of VDAS may be calculated on the basis of 81 motion vectors assigned to all the pixels in the range of 9 pixels vertically and 9 pixels horizontally, centered on the pixel of interest. .

  Next, the image feature amount calculation unit 143 has the same position as the pixel of interest on the input frame t + k, which is a pixel on the input frame t + 1 where the interpolation frame t + k is arranged between the frame t. A value indicating a change in the pixel value in the time direction is calculated from the pixel value of the pixel on the frame t + 1 at the position and the pixel value of the pixel on the frame t at the same position as the position of the target pixel.

  This is the case when the image is not broken, considering the case where the boundary of the object with different motion is included in the range of the pixel to which the motion vector for which the value indicating the variation is calculated is assigned. However, since variations in motion vectors occur, if it is determined whether or not an image has failed only by variations in motion vectors, the failure location determination unit 142 erroneously determines that the image has failed (false detection). There is a time.

  For example, as shown in FIG. 15, when nine pixels (circles in the figure) every other pixel in a 5 × 5 pixel region are considered, three pixels in the upper horizontal row and the lower horizontal The three pixels in this row correspond to the moving background image object, and the middle row is sandwiched between the three pixels in the upper horizontal row and the three pixels in the lower horizontal row. When three pixels correspond to a still foreground image object, the upper pixel and the lower pixel are assigned the same motion vector, but the middle pixel is assigned the zero vector.

  In this case, as shown in FIG. 16, when the central pixel among the nine pixels of every other pixel in the 5 × 5 pixel region is the target pixel (pixel indicated by a black circle in the figure), the motion vector Is determined by the magnitudes of the motion vectors assigned to the three pixels in the upper horizontal row and the three pixels in the lower horizontal row, and becomes large. Although the image shown in the figure is not broken, if it is determined whether or not the image is broken only by a value indicating the variation of the motion vector, it is erroneously determined that the image is broken. Become.

  Therefore, in order to reduce such erroneous detection and improve the accuracy of the detection of the broken part from the detection of the broken part of the image, it is considered to use the information of the image in the time direction.

As shown in FIG. 17, when attention is paid to the pixel p, the pixel value F _t (p) of the pixel on the frame t at the same position (same phase) as the position of the pixel p is the same as the position of the pixel p. The absolute difference value D (p) from the pixel value F _{t + 1} (p) of the pixel on the frame t + 1 at the position is used as an evaluation value for indicating stability in the time direction (indicating a change in the pixel value in the time direction). Value). The absolute difference value D (p) is calculated by equation (6).
D (p) = | F _t (p) −F _{t + 1} (p) | (6)

  For example, the image feature amount calculation unit 143 calculates the absolute difference value D (p) as a value indicating a change in the pixel value in the time direction by the calculation represented by Expression (6).

  The image feature amount calculation unit 143 supplies a value indicating the calculated change in the pixel value in the time direction to the failure location determination unit 142.

  The failure location determination unit 142 determines the pixel of interest from the value indicating the variation in the motion vector supplied from the vector feature amount calculation unit 141 and the value indicating the change in the pixel value in the time direction supplied from the image feature amount calculation unit 143. It is determined whether the image of the interpolation frame is broken.

For example, the failure location determination unit 142 compares the VDAS supplied from the vector feature amount calculation unit 141 with a predetermined threshold value TH _pf0 for the target pixel px _{, y} , and is supplied from the image feature amount calculation unit 143. comparing the threshold value TH _pf1 determined in advance and the difference absolute value D (p), VDAS is greater than the threshold value TH _pf0, and, when the difference absolute value D (p) is greater than the threshold value TH _pf1, the pixel of interest p _x, It is determined that the image of the interpolation frame is broken at _y . The failure location determination unit 142 causes the image of the interpolation frame to fail at the pixel of interest _{px , y} when _VDAS is equal to or less than the threshold value TH _pf0 or the absolute difference value D (p) is _{equal to} or less than the threshold value TH _pf1. Judge that it is not.

In other words, the failure location determination unit 142 determines that the image of the interpolation frame has failed when the condition represented by Equation (7) is satisfied, and if the condition indicated by Equation (7) is not satisfied, It is determined that the image of the insertion frame has not failed.
VDAS _{t + k} (p _{x, y} )> TH _pf0∩D (p)> TH _pf1 (7)

  In this way, it is possible to detect the failure in the spatial direction of the image from the variation of the motion vector, but by further using the information in the time direction of the image, it is possible to reduce the erroneous detection of the failure location, more accurately, more It is possible to reliably determine whether or not the image is broken.

  The failure location determination unit 142 supplies the filter selection unit 145 with data indicating the result of determination as to whether or not the image of the interpolation frame has failed at the target pixel, along with a value indicating the variation from the motion vector.

  The filter coefficient ROM 144 stores in advance a plurality of filter coefficients for determining the characteristics of the filter used for a filter (low-pass filter) that removes the high-frequency component of the image.

  The filter selection unit 145 applies one filter to be applied to an image in a predetermined range including the target pixel in the interpolation frame, based on a value indicating motion vector variation, from a plurality of filters that remove high frequency components of the image. select.

  When it is determined that the image of the interpolation frame has failed at the target pixel, the filter processing unit 146 determines the high frequency component of the image from an image in a predetermined range including the target pixel in the interpolation frame of the interpolated image. Remove. The filter processing unit 146 outputs an image from which the high frequency component has been removed as an output image. For example, the filter processing unit 146 applies the processing of the filter that removes the high-frequency component of the image to an image in a predetermined range including the target pixel in the interpolation frame, which is indicated by a one-dot chain line in FIG. Removes the high frequency component of the image.

  Note that the range of pixels to which a motion vector for calculating a value indicating variation is assigned may be the same as or different from the range of an image from which high-frequency components are removed.

  More specifically, for example, the filter coefficient ROM 144 stores in advance a plurality of filter coefficients for use in the filter processing unit 146 to process a filter that removes high-frequency components of an image and for determining filter characteristics. .

  The filter selection unit 145 reads a predetermined filter coefficient among the filter coefficients stored in the filter coefficient ROM 144 in accordance with VDAS which is an example of a value indicating the variation of the motion vector. The filter selection unit 145 supplies the filter coefficient read from the filter coefficient ROM 144 to the filter processing unit 146. The filter processing unit 146 applies the filter processing using the filter coefficient supplied from the filter selection unit 145 to the image in a predetermined range including the target pixel in the interpolation frame, thereby removing the high frequency component of the image. .

  That is, for example, the filter selection unit 145 selects one filter coefficient for realizing a predetermined filter characteristic from among a plurality of filter coefficients based on the value indicating the variation of the motion vector, and the filter processing unit 146. By supplying the selected filter coefficient, a filter to be applied to an image in a predetermined range including the target pixel in the interpolation frame is selected.

More specifically, the filter selection unit 145 selects the filter 1 when the condition represented by Expression (8) is satisfied, and selects the filter 2 when the condition represented by Expression (9) is satisfied.
TH _pf00 <VDAS _{t + k} (p _{x, y} ) <= TH _pf01 (8)
TH _pf01 <VDAS _{t + k} (p _{x, y} ) <= TH _pf02 (9)
However, TH _pf00 <TH _pf01 <TH _pf02 .

  In this case, it is known that the filter 2 should be a filter that removes more high-frequency components of the image than the filter 1.

  That is, when the condition expressed by Expression (8) is satisfied, the filter selection unit 145 selects a filter coefficient for realizing the characteristics of the filter 1, reads the filter coefficient from the filter coefficient ROM 144, and supplies the filter coefficient to the filter processing unit 146. To do. On the other hand, the filter selection unit 145 selects a filter coefficient for realizing the characteristics of the filter 2 when the condition represented by Expression (9) is satisfied, reads the filter coefficient from the filter coefficient ROM 144, and supplies the filter coefficient to the filter processing unit 146. To do.

  The filter processing unit 146 applies the filter 1 or the filter 2 to the broken image according to the filter coefficient supplied from the filter selection unit 145. When the variation of the motion vector is larger, more high frequency components of the image are removed.

  That is, when the variation of the motion vector is larger, it is estimated that the degree to which the image is broken is large, so that more high frequency components of the image are removed, and the image is made inconspicuous.

  Since the failed part post-processing unit 60 removes the high-frequency component of the image from the broken image among the images of the interpolation frame, the broken image is not noticeable without impairing the smoothness of the movement of the image. Can be output. Since the failure of the image is less noticeable, the user viewing the image feels that the image quality of the image has been improved.

  In addition, by using the motion vector and the input image (image information), it is possible to detect a failed part of the image accurately and accurately.

  In this way, assuming that there is a possibility of image degradation in a portion where the motion vector is rampant, a low-pass filter is applied to this portion. By applying the low-pass filter, it is possible to make the deterioration generated in the interpolated image inconspicuous.

  Next, details of the post-processing of the failed part in step S7 of FIG. 5 will be described with reference to the flowchart of FIG.

  In step S141, the vector feature amount calculation unit 141 calculates the irregularity evaluation value of the motion vector allocated to the pixels in a predetermined range around the target pixel that is the target pixel on the interpolation frame. . For example, in step S141, the vector feature amount calculation unit 141 calculates VDAS as the irregularity evaluation value based on Expression (3). The vector feature amount calculation unit 141 supplies the irregularity evaluation value to the failure location determination unit 142.

  In step S142, the image feature amount calculation unit 143 calculates an image feature amount indicating a change in the pixel value in the time direction. For example, in step S142, the image feature amount calculation unit 143 calculates the absolute difference value D (p) as the image feature amount based on Expression (6). The image feature amount calculation unit 143 supplies the image feature amount to the failure location determination unit 142.

  In step S143, the failure location determination unit 142 determines whether the image of the interpolation frame has failed at the target pixel based on the irregularity evaluation value and the image feature amount. If it is determined in step S143 that the image of the interpolation frame has failed at the target pixel, the process proceeds to step S144, and the filter selection unit 145 performs irregularity from the plurality of filters that remove the high frequency components of the image. Based on the evaluation value, a filter to be applied to an image in a range including the target pixel in the interpolation frame is selected. For example, in step S144, the filter selection unit 145 reads a predetermined filter coefficient corresponding to the irregularity evaluation value from the filter coefficients stored in the filter coefficient ROM 144, and filters the filter coefficient read from the filter coefficient ROM 144. By supplying to the processing unit 146, a filter to be applied to an image in a range including the target pixel in the interpolation frame is selected.

  In step S145, the filter processing unit 146 applies the selected filter to the image in the range including the target pixel in the interpolation frame, and the process proceeds to step S146. That is, in step S144, the filter processing unit 146 removes the high frequency component of the image from the image in a predetermined range including the target pixel in the interpolation frame. For example, the filter processing unit 146 applies a filter whose characteristics are determined by the filter coefficient supplied from the filter selection unit 145 to an image in a range including the target pixel in the interpolation frame.

  FIG. 19 is a diagram for explaining a filter applied to an image in a range including the target pixel in the interpolation frame in the filter processing unit 146. The horizontal direction in FIG. 19 indicates the frequency of the image, and the vertical direction in FIG. 19 indicates the output level. In FIG. 19, dotted lines indicate characteristics of the filter 1 that is an example of a filter applied in the filter processing unit 146 and removes a high frequency component of an image, and a solid line indicates an example of a filter applied in the filter processing unit 146. In comparison with the filter 1, the characteristic of the filter 2 that removes more high frequency components of the image is shown.

  For example, when the condition represented by Expression (8) is satisfied in step S144, the filter selection unit 145 selects the filter 1, reads the filter coefficient of the filter 1 from the filter coefficient ROM 144, and supplies the filter coefficient to the filter processing unit 146. To do. Further, when the condition represented by Expression (9) is satisfied, the filter selection unit 145 selects the filter 2, reads the filter coefficient of the filter 2 from the filter coefficient ROM 144, and supplies the filter coefficient to the filter processing unit 146.

  Then, in step S145, the filter processing unit 146 to which the filter coefficient is supplied from the filter selection unit 145 adds either the filter 1 or the filter 2 to the image in the range including the target pixel in the interpolation frame according to the filter coefficient. Either one is applied to remove the high frequency component of the image from the image in a predetermined range including the target pixel in the interpolation frame.

  If it is determined in step S143 that the image of the interpolation frame has not failed at the target pixel, steps S144 and S145 are skipped, and the procedure proceeds to step S146.

  In step S146, the failed part post-processing unit 60 determines whether or not the process has been completed for all the pixels of the interpolation frame. If it is determined that the process has not been completed, the process returns to step S141. The above-described processing is repeated for the next pixel of the insertion frame. Note that the pixels to be processed in steps S141 to S145 are selected in the raster scan order, for example.

  If it is determined in step S146 that the process has been completed for all the pixels of the interpolation frame, the post-processing of the failed part ends.

  As described above, since the high-frequency component of the image is removed from the broken image in the image of the interpolation frame, the non-broken image is not affected, and the image motion is smooth. It is possible to make the image inconspicuous (masking) without impairing the thickness. Further, since the filter is adaptively used according to the degree of image failure, the image failure can be made inconspicuous (masking) without removing the high frequency components of the image more than necessary.

  Since the failure of the image is less noticeable, the user viewing the image feels that the image quality of the image has been improved.

  In addition, you may make it perform each process of step S141 thru | or step S145 about the whole interpolation frame. That is, for example, if a motion vector irregularity evaluation value is calculated for each pixel selected from the interpolation frame in the raster scan order, and the motion vector irregularity evaluation value is calculated for the entire interpolation frame, For each pixel selected in the raster scan order from the insertion frame, the processing of each step may be executed for the entire interpolation frame, such as calculating the image feature amount.

  In addition, the failure location post-processing unit 60 may determine whether or not the image has failed without using only the value indicating the variation of the motion vector and the value indicating the change in the pixel value in the time direction. Good.

  FIG. 20 is a block diagram showing another example of the configuration of the failed part post-processing unit 60 in this case. The failure location post-processing unit 60 whose configuration is shown in FIG. 20 includes a vector feature amount calculation unit 141, a filter coefficient ROM 144, a filter selection unit 145, a filter processing unit 146, and a failure location determination unit 161.

  The same parts as those shown in FIG. 12 are denoted by the same reference numerals, and the description thereof is omitted.

  The vector feature amount calculation unit 141 calculates a variation between the motion vector assigned to the pixel of interest on the interpolation frame and the motion vector assigned to the peripheral pixel that is a pixel in a predetermined range around the pixel of interest. A value indicating the deviation from the calculated motion vector is supplied to the failure location determination unit 161.

  The failure location determination unit 161 determines whether the image of the interpolation frame has failed at the target pixel from the value indicating the variation of the motion vector supplied from the vector feature amount calculation unit 141, and indicates the determination result. Data is supplied to the filter selection unit 145.

  FIG. 21 is a flowchart for explaining another example of details of the post-failure location post-processing in step S7 of FIG. 5 by the failure location post-processing unit 60 whose configuration is shown in FIG.

  In step S161, the vector feature amount calculation unit 141 calculates the irregularity evaluation value of the motion vector assigned to the pixels in a predetermined range around the target pixel that is the target pixel on the interpolation frame. Then, the irregularity evaluation value is supplied to the failure location determination unit 161.

  In step S162, the failure location determination unit 161 determines whether the image of the interpolation frame has failed in the target pixel based on the irregularity evaluation value.

For example, the failure location determination unit 161 determines that the image of the interpolated frame has failed when the condition indicated by Equation (10) is satisfied, and if the condition indicated by Equation (10) is not satisfied, It is determined that the image of the insertion frame has not failed.
VDAS _{t + k} (p _{x, y} )> TH _pf0 (10)

  The processing from step S163 to step S165 is the same as that from step S144 to step S146 in FIG.

  In this way, it is possible to determine whether or not the image is broken only by the value indicating the variation of the motion vector. In this case, it is possible to output an image in which the failure of the image is not noticeable with a simpler process without impairing the smoothness of the motion of the image.

  In addition, after the post-processing is applied to the broken image among the interpolated images output from the image interpolation unit 58, the post-processing may be applied to the entire interpolation frame.

  FIG. 22 is a block diagram illustrating another example of the functional configuration of the image processing apparatus 1. The image processing apparatus 1 includes a frame memory 51, a vector detection unit 52, a detection vector memory 53, a vector allocation unit 54, an allocation vector memory 55, an allocation flag memory 56, an allocation compensation unit 57, an image interpolation unit 58, an entire frame post-processing unit 59, and a failed part post-processing unit 60. The same parts as those shown in FIG. 2 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The image interpolation unit 58 supplies the interpolated image to the failed part post-processing unit 60. The failure location post-processing unit 60, among the images included in the interpolation frame of the interpolation image supplied from the image interpolation unit 58, the motion vector assigned to the interpolation frame of the allocation vector memory 55 to the failed image, Post-processing is applied using a predetermined threshold value and pixel values of the frame t and the next frame t + 1. The failed part post-processing unit 60 supplies an interpolated image obtained by applying post-processing to the failed image to the entire frame post-processing unit 59.

  The entire frame post-processing unit 59 uses a predetermined mixing ratio and pixel values of the frame t and the next frame t + 1 for the entire interpolation frame of the interpolated image supplied from the failed part post-processing unit 60, Apply post-processing that prevents the failure from becoming noticeable. The entire frame post-processing unit 59 outputs a 60P signal image obtained by applying post-processing to the entire interpolated frame to a subsequent stage (not shown).

  Furthermore, only one of the entire frame post-processing unit 59 and the failed part post-processing unit 60 may be provided.

  FIG. 23 is a block diagram illustrating still another example of the functional configuration of the image processing apparatus 1. The image processing apparatus 1 includes a frame memory 51, a vector detection unit 52, a detection vector memory 53, a vector allocation unit 54, an allocation vector memory 55, an allocation flag memory 56, an allocation compensation unit 57, an image interpolation unit 58, and an entire frame post-processing. A portion 59 is provided. The same parts as those shown in FIG. 2 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The whole frame post-processing unit 59 uses the predetermined mixture ratio and the pixel values of the frame t and the next frame t + 1 for the entire interpolated frame of the interpolated image supplied from the image interpolation unit 58, so that the failure occurs. Apply post-processing to make it inconspicuous. The entire frame post-processing unit 59 outputs a 60P signal image obtained by applying post-processing to the entire interpolated frame to a subsequent stage (not shown).

  FIG. 24 is a block diagram illustrating still another example of the functional configuration of the image processing apparatus 1. The image processing apparatus 1 includes a frame memory 51, a vector detection unit 52, a detection vector memory 53, a vector allocation unit 54, an allocation vector memory 55, an allocation flag memory 56, an allocation compensation unit 57, an image interpolation unit 58, and a failed part post-processing. The unit 60 is provided. The same parts as those shown in FIG. 2 are denoted by the same reference numerals, and description thereof will be omitted as appropriate.

  The failure location post-processing unit 60, among the images included in the interpolation frame of the interpolation image supplied from the image interpolation unit 58, the motion vector assigned to the interpolation frame of the allocation vector memory 55 to the failed image, Post-processing is applied using a predetermined threshold value and pixel values of the frame t and the next frame t + 1. The failed part post-processing unit 60 outputs a 60P signal image obtained by applying post-processing to the failed image to a subsequent stage (not shown).

  In this way, the motion vector on the first frame is detected, the detected motion vector is allocated to the pixels on the second frame, and the first frame and the third frame are assigned based on the allocated motion vector. When the pixel values of the pixels on the second frame are generated from the frame, an image can be generated. Further, the pixels on the second frame in which the pixel value is generated are the motion vector assigned to the target pixel that is the target pixel and the pixels in a predetermined range around the target pixel. Calculating a value indicating variation with the motion vector assigned to the peripheral pixels, and determining whether the image of the second frame is broken at the target pixel based on the value indicating variation in the motion vector; When it is determined that the image of the second frame is broken, it is generated when the high frequency component of the image is removed from the image of the predetermined range including the target pixel in the second frame. It is possible to make the image corruption inconspicuous.

  Further, in the present embodiment, the area where each process is performed is described as being configured by, for example, 5 pixels × 5 pixels, but these are only examples, and the pixels constituting the area where each process is performed Is not limited to this number of pixels.

  Furthermore, in the present embodiment, the signal conversion from the 24P signal to the 60P signal has been described as an example. However, the present invention can be applied to, for example, interlaced signals and other frame rates as frame frequency conversion of moving images. It can also be applied to conversion.

  The input image may be an image using either an analog signal or a digital signal as a medium. Further, an image read from a recording medium, an image transmitted by broadcasting or communication, an image acquired via a network, etc. Any image may be used, and the present invention is not limited thereto.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  As shown in FIG. 1, a program recording medium for storing a program that is installed in a computer and can be executed by the computer is a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only). Memory), DVD (including Digital Versatile Disc), magneto-optical disk), or removable media 21 that is a package medium made of semiconductor memory or the like, or ROM 12 in which a program is temporarily or permanently stored, The storage unit 18 is configured by a hard disk or the like. The program is stored in the program recording medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 19 that is an interface such as a router or a modem as necessary. Done.

  In the present specification, the step of describing the program stored in the program recording medium is not limited to the processing performed in time series in the order described, but is not necessarily performed in time series. Or the process performed separately is also included.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

It is a block diagram showing the example of a structure of the image processing apparatus to which this invention is applied. It is a block diagram which shows the example of a function structure of an image processing apparatus. It is a figure explaining the principle of the process which interpolates a flame | frame. It is a figure explaining the process which interpolates a frame more concretely. It is a flowchart explaining the process which converts a frame frequency. It is a block diagram which shows the structure of an image interpolation part. It is a flowchart explaining the detail of an image interpolation process. It is a figure explaining an interpolation frame. It is a block diagram which shows the example of a structure of the whole frame post-processing part. It is a figure explaining the production | generation of the pixel value in a 0 vector interpolation image generation part. It is a flowchart explaining the detail of the post-process of the whole flame | frame. It is a block diagram which shows the example of a structure of a failure location post-processing part. It is a figure which shows the range of the pixel to which the motion vector used as the object of calculation of the value which shows dispersion | variation is allocated. It is a figure which shows the range of the pixel to which the motion vector used for calculation of VDAS is allocated. It is a figure explaining the erroneous detection of the failure of an image. It is a figure explaining the erroneous detection of the failure of an image. It is a figure explaining calculation of difference absolute value D (p). It is a flowchart explaining the detail of the post-process of a failure location. It is a figure explaining the characteristic of a filter. It is a block diagram which shows the other example of a structure of a failure location post-processing part. It is a flowchart explaining the other example of the detail of the post-process of a failure location. It is a block diagram which shows the other example of a function structure of an image processing apparatus. FIG. 10 is a block diagram illustrating still another example of the functional configuration of the image processing apparatus. FIG. 10 is a block diagram illustrating still another example of the functional configuration of the image processing apparatus.

Explanation of symbols

  11 CPU, 12 ROM, 13 RAM, 18 storage unit, 21 removable media, 51 frame memory, 52 vector detection unit, 53 detection vector memory, 54 vector allocation unit, 55 allocation vector memory, 56 allocation flag memory, 57 allocation compensation unit , 58 image interpolation unit, 59 whole frame post-processing unit, 60 failure location post-processing unit, 1100 vector interpolation image generation unit, 112 image mixing unit, 141 vector feature value calculation unit, 142 failure location determination unit, 143 image feature value Calculation unit, 144 Filter coefficient ROM, 145 Filter selection unit, 146 Filter processing unit, 161 Failure location determination unit

Claims (9)

A motion vector on the input first frame is detected, the detected motion vector is allocated to a pixel on the second frame, and on the second frame based on the allocated motion vector In an image processing device that generates pixel values of pixels,
A pixel on the second frame in which a pixel value is generated, the motion vector assigned to the pixel of interest that is the pixel of interest, and pixels in a predetermined range around the pixel of interest First calculation means for calculating a value indicating variation with the motion vector assigned to a certain peripheral pixel;
Determination means for determining whether or not the image of the second frame is broken in the target pixel based on a value indicating variation in the motion vector;
An image comprising: removal means for removing a high frequency component of an image from an image in a predetermined range including the target pixel in the second frame when it is determined that the image of the second frame has failed Processing equipment. The second frame is disposed between the first frame and a pixel on the input third frame, the second position being the same as the position of the target pixel on the second frame. A second value that calculates a change in the pixel value in the time direction from the pixel value of the pixel on the third frame and the pixel value of the pixel on the first frame at the same position as the position of the target pixel; A calculation means,
The determination unit determines whether or not the image of the second frame is broken at the target pixel based on a value indicating variation in the motion vector and a value indicating change in the pixel value in the time direction. Item 8. The image processing apparatus according to Item 1. Selection means for selecting a filter to be applied to the image in the range including the target pixel in the second frame, based on a value indicating variation in the motion vector, from a plurality of filters that remove high frequency components of the image. In addition,
The image processing apparatus according to claim 1, wherein the removing unit removes a high-frequency component of the image by applying a selected filter to the image in the range including the target pixel in the second frame. The image processing apparatus according to claim 3, wherein the selection unit selects a filter that removes more high-frequency components of the image when the motion vector is more varied. When the motion vector is assigned to every n pixels on the second frame, the calculating means assigns the motion vector assigned to the surrounding pixels separated from the target pixel by n pixels. The image processing apparatus according to claim 1, wherein a value indicating variation between the motion vector assigned to the target pixel is calculated. Detecting means for detecting the motion vector on the first frame;
Allocating means for allocating the detected motion vector to pixels on the second frame;
The image according to claim 1, further comprising: generating means for generating a pixel value of a pixel on the second frame from the first frame and the third frame based on the allocated motion vector. Processing equipment. A motion vector on the input first frame is detected, the detected motion vector is allocated to a pixel on the second frame, and on the second frame based on the allocated motion vector In an image processing method for generating a pixel value of a pixel,
A pixel on the second frame in which a pixel value is generated, the motion vector assigned to the pixel of interest that is the pixel of interest, and pixels in a predetermined range around the pixel of interest Calculate a value indicating a variation with the motion vector assigned to a certain peripheral pixel,
Based on the value indicating the variation of the motion vector, it is determined whether or not the image of the second frame has failed in the pixel of interest,
An image processing method including a step of removing a high frequency component of an image from an image in a predetermined range including the target pixel in the second frame when it is determined that the image of the second frame is broken . A motion vector on the input first frame is detected, the detected motion vector is allocated to a pixel on the second frame, and on the second frame based on the allocated motion vector In a program for causing a computer to perform image processing for generating pixel values of pixels,
A pixel on the second frame in which a pixel value is generated, the motion vector assigned to the pixel of interest that is the pixel of interest, and pixels in a predetermined range around the pixel of interest Calculate a value indicating a variation with the motion vector assigned to a certain peripheral pixel,
Based on the value indicating the variation of the motion vector, it is determined whether or not the image of the second frame has failed in the pixel of interest,
A program including a step of removing a high frequency component of an image from an image of a predetermined range including the target pixel in the second frame when it is determined that the image of the second frame is broken.   A recording medium on which the program according to claim 8 is recorded.

Images