JP4655217B2 - Image processing apparatus and method, recording medium, and program - Google Patents

Description

  The present invention relates to an image processing apparatus and method, a recording medium, and a program, and more particularly to an image processing apparatus and method, a recording medium, and a program that can suppress deterioration in image quality.

  When converting the frame frequency of an image, motion vectors are assigned (for example, Patent Document 1).

  For example, when a predetermined pixel (for example, foreground pixel) of a frame at time t (hereinafter also referred to as frame t. The same applies hereinafter) corresponds to a pixel at a predetermined position on the frame at time t + 1 (frame t + 1). , The vector connecting the two is the motion vector Va of the pixel. When converting the frame frequency (when generating a frame at time t + p between frame t and frame t + 1 (frame t + p)), for a pixel G on frame t + p, the frame t passing through the vicinity of the pixel G A motion vector is allocated. For example, when the motion vector Va passes near the pixel G on the frame t + p, the motion vector Va is assigned to the pixel G.

  However, for example, when a foreground that moves relatively slowly and a background that moves at high speed are mixed in the screen, not only the motion vector Va of the foreground pixels but also the background pixels are located in the vicinity of one pixel G. May also pass through the motion vector Vb.

In such a case, a difference absolute value sum (DFD (Displaced Frame Difference)) representing a correlation value between blocks shifted by the vector amount of interest of two frames is calculated, and 1 corresponding to the pixel G is calculated based on the value. One motion vector was determined. For example, DFDa, which is DFD based on the motion vector Va, and DFDb, which is DFD based on the motion vector Vb, are calculated, and the motion vector corresponding to the smaller one of the two values is assigned as the motion vector of the pixel G.
Japanese Patent Laid-Open No. 9-172621

  However, the foreground is originally an object, and foreground motion vectors should be assigned to all foreground pixels. In practice, however, background motion vectors may be assigned to some foreground pixels. is there.

  For example, as shown in FIG. 1, when the foreground 1002 exists in front of the background 1001, as shown in FIG. 2, the foreground 1002 has a relatively small (slow) speed in the right direction in the figure. Of the area of the background 1001 that moves to the left in the figure at a high speed when moving, it is difficult to obtain a motion vector for an area that is covered (covered) by the foreground 1002 after the foreground 1002 has moved. Covered region 1003.

  For example, when the motion of the background 1001 is V1 (for example, 80 pixels), the covered region 1003 appears at a position separated from the foreground 1002 by 80 pixels in the opposite direction to the motion V1 of the background 1001.

  In the covered region 1003, the motion vector is detected as 0 (whether it is a 0 vector) or as a very small motion.

  In such a case, as shown in FIG. 3, when the motion vector V1 of the dangerous area 1004 formed between the foreground 1002 and the covered area 1003 is assigned to the foreground 1002, the movement is the movement of the foreground 1002. Therefore, a deteriorated portion 1005 is formed in the foreground 1002. In the penetrating degradation portion 1005, as shown in FIG. 3, an image in which a part of the foreground 1002 is missing is obtained.

  As described above, the image on the frame t + p has a problem that the image quality deteriorates due to a part of the image being lost.

  The present invention has been made in view of such a situation, and is intended to suppress deterioration in image quality.

  One aspect of the present invention provides an acquisition unit that acquires a motion vector of a target point of the first frame in an image processing apparatus that allocates a motion vector of the first frame to a second frame; Detection means for detecting a size and a region on the first frame to which the motion vector of the target point belongs, and the second frame of the motion vector of the target point based on the detected size and region An image processing apparatus including prohibiting means for prohibiting assignment to the image processing apparatus.

  The detection means can detect that the region to which the motion vector of the point of interest belongs is a region in which penetration degradation may occur.

  The detection means includes a third region sandwiched between a first region in which a motion vector of the point of interest belongs is a foreground and a second region in which no motion is obtained because the region is hidden by the foreground It can be detected.

  The acquisition means acquires a first motion vector that is a motion vector of the point of interest, and a second motion vector of a first reference point that is located in the direction of the first motion vector from the point of interest. , Obtaining a third motion vector of a second reference point located in a direction opposite to the direction of the first motion vector from the point of interest, and the detecting means includes the first motion vector and the second motion vector. Detecting the third region based on a difference in magnitude of a motion vector and a reference value of a difference in magnitude between the first motion vector and the third motion vector, and detecting the first motion The magnitude of the vector relative to the second motion vector and the third motion vector can be detected.

  The first reference point is a pixel in the direction of the first motion vector from the point of interest in the first frame, and a position that internally divides the magnitude of the first motion vector into n to m , And the second reference point is a pixel in a direction opposite to the direction of the first motion vector from the point of interest of the first frame, The magnitude of the first motion vector can be any one of N pixels in the vicinity of a position that internally divides m by n.

  The N pixels may be pixels within a range of ± α times or ± β times the size of the x-coordinate component and the y-coordinate component of the first motion vector with reference to the internally divided position. .

  The internal ratio n: m can be 1: 1, 1: 3, or 3: 1.

  The image processing apparatus may further include a calculation unit that calculates an evaluation value of the motion vector, and a selection unit that selects a motion vector to be assigned to the second frame based on the evaluation value.

  According to another aspect of the present invention, in an image processing method, program, or recording medium that allocates a motion vector of a first frame to a second frame, a motion vector of a target point of the first frame is acquired, and the target The size of the motion vector of the point and the region on the first frame to which the motion vector of the target point belongs are detected, and the second of the motion vectors of the target point is detected based on the detected size and region. An image processing method, a program, or a recording medium including a step of prohibiting allocation to a frame.

  In one aspect of the present invention, the magnitude of the motion vector of the point of interest in the first frame, and the region on the first frame to which the motion vector of the point of interest belongs are detected, and based on the detected size and region The allocation of the motion vector of the point of interest to the second frame is prohibited.

  As described above, according to one aspect of the present invention, a high-quality image can be obtained. In particular, according to one aspect of the present invention, it is possible to suppress partial loss of an image when the frame frequency of the image is converted.

  Embodiments of the present invention will be described below. Correspondences between the configuration requirements of the present invention and the embodiments described in the detailed description of the present invention are exemplified as follows. This description is to confirm that the embodiments supporting the present invention are described in the detailed description of the invention. Accordingly, although there are embodiments that are described in the detailed description of the invention but are not described here as embodiments corresponding to the constituent elements of the present invention, It does not mean that the embodiment does not correspond to the configuration requirements. Conversely, even if an embodiment is described here as corresponding to a configuration requirement, that means that the embodiment does not correspond to a configuration requirement other than the configuration requirement. It's not something to do.

  One aspect of the present invention is an image processing apparatus (for example, an image signal in FIG. 4) that allocates a motion vector of a first frame (for example, frame t1 in FIG. 37) to a second frame (for example, frame F in FIG. 37). In the recording / reproducing apparatus 1), an acquisition unit (for example, step of FIG. 24) that acquires a motion vector (for example, the motion vector Vc of FIG. 27) of the point of interest (for example, the target pixel 281 of FIG. 27) of the first frame. The penetration improvement processing unit 706 in FIG. 16 that executes the processing of S252, and detection means for detecting the size of the motion vector of the target point and the region on the first frame to which the motion vector of the target point belongs ( For example, the focus improvement processing unit 706 in FIG. 16 that executes the processes in steps S259 and S260 in FIG. The inhibiting means for inhibiting the allocation for said second frame of motion vectors (e.g., penetrate improvement processing unit 706 in FIG. 16 for executing the processing of step S262 in FIG. 24) is an image processing apparatus and a.

  The detection means includes a region (for example, in FIG. 39) in which a region to which a motion vector (for example, the motion vector V1 in FIG. 39) of the point of interest (for example, the pixel 1015 in FIG. 39) belongs may occur. It is possible to detect that it is a dangerous area 1014).

  The detection means includes a first region (for example, foreground in FIG. 37) in which a region to which a motion vector (for example, motion vector Vc in FIG. 37) of the point of interest (for example, target pixel 301 in FIG. 37) belongs is a foreground. 302) and a third area (for example, area 391 in FIG. 37) sandwiched by a second area (for example, covered area 321 in FIG. 37) that is hidden by the foreground and does not require movement. Can be detected.

The acquisition unit acquires a first motion vector (for example, the motion vector Vc in FIG. 27) that is a motion vector of the point of interest (for example, the target pixel 281 in FIG. 27), and the first motion vector from the point of interest. The second motion vector (for example, the motion vector Vl in FIG. 27) of the first reference point (for example, the reference point 282 in FIG. 27) located in the direction of the motion vector and the first motion from the point of interest A third motion vector (for example, motion vector Vr in FIG. 27) of a second reference point (for example, reference point 283 in FIG. 27) located in the direction opposite to the vector direction is obtained, and the detection means A difference in magnitude between the first motion vector and the second motion vector (eg, M _{L in} equation (15)), and a difference in magnitude between the first motion vector and the third motion vector (eg, , M _R in equation (16) (for example, , 28) to detect the third region, and the magnitude of the first motion vector relative to the second motion vector and the third motion vector (e.g., Equation (17), Equation (18)) can be detected.

  The first reference point is a pixel in the direction of the first motion vector (for example, the motion vector Vc in FIG. 34) from the point of interest (for example, the target pixel 281 in FIG. 34) of the first frame. Thus, N (for example, nine in FIG. 34) pixels (for example, FIG. 34) in the vicinity of a position (for example, the reference point 282 in FIG. 34) that internally divides the magnitude of the first motion vector into n to m. 34 reference points 282 and surrounding reference points 282-1 to 282-8), and the second reference point is determined from the point of interest of the first frame to the first motion vector. N pixels (for example, FIG. 34) in the direction opposite to the direction and in the vicinity of a position (for example, reference point 283 in FIG. 34) that internally divides the magnitude of the first motion vector into m to n. Reference point 283 and surrounding reference points 283-1 to 283-8) Door can be.

  The N pixels have the x-coordinate component (for example, | Vcx | in FIG. 34) and the y-coordinate component (for example, | Vcy | in FIG. 34) of the first motion vector with reference to the internally divided position. The pixel can be in the range of ± α times or ± β times the size of.

  The internal ratio n to m can be 1 to 1 (eg, FIG. 34), 1 to 3 (eg, FIG. 36), or 3 to 1 (eg, FIG. 35).

  Calculation means for calculating an evaluation value (for example, DFD, difference) of the motion vector (for example, the evaluation value calculation unit 702 in FIG. 16 that executes the process in step S203 in FIG. 22 and the process in step S781 in FIG. 48) 43 and the selecting means for selecting a motion vector to be allocated to the second frame based on the evaluation value (for example, executing the processing of step S749 in FIG. 42). The vector selection unit 705 of FIG. 43 and the vector selection unit 754 of FIG. 43 that executes the process of step S787 of FIG. 48 can be further provided.

  An aspect of the present invention provides an image processing method, program, or recording medium that allocates a motion vector of a first frame (for example, frame t1 in FIG. 37) to a second frame (for example, frame F in FIG. 37). Then, a motion vector (for example, the motion vector Vc in FIG. 27) of the target point (for example, the target pixel 281 in FIG. 27) of the first frame is acquired (for example, step S252 in FIG. 24). And the region on the first frame to which the motion vector of the target point belongs (for example, steps S259 and S260 in FIG. 24), and based on the detected size and region, the target point Image processing including a step (for example, step S262 in FIG. 24) of prohibiting allocation of a motion vector of the second motion vector to the second frame Law, a program or a recording medium.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 4 is a block diagram showing a configuration of an image signal recording / reproducing apparatus as an image processing apparatus to which the present invention is applied. The image signal recording / reproducing apparatus 1 includes an image signal input unit 2, a signal processing device 3, and an image signal output unit 4.

  The image signal input unit 2 includes, for example, a tuner or a disk driver, and outputs a received image signal or an image signal reproduced from a recording medium to the signal processing device 3. The signal processing device 3 executes a frame frequency conversion process of the image signal supplied from the image signal input unit 2. The image signal converted from the frame frequency output from the signal processing device 3 is supplied to the image signal output unit 4. The image signal output unit 4 outputs the image signal input from the signal processing device 3 to a display unit such as an LCD (Liquid Crystal Display) or a CRT (cathode-ray tube), and displays it in addition to an instruction from the user. In response, recording is performed on a recording medium (not shown).

  FIG. 5 is a block diagram illustrating a configuration of the signal processing device 3.

  It does not matter whether each function of the signal processing device 3 is realized by hardware or software. That is, each block diagram in this specification may be considered as a hardware block diagram or a software functional block diagram.

  In the signal processing device 3 having the configuration shown in FIG. 5, for example, an image of a progressive image signal (hereinafter referred to as a 24P signal) having a frame frequency of 24 Hz is input, and the input image (input image) has a frame frequency of It is converted into an image of a 60 Hz progressive image signal (hereinafter referred to as 60P signal) and output.

  The input image of the 24P signal input to the signal processing device 3 is supplied to the frame memory 51, the vector detection unit 52, the vector allocation unit 54, the allocation compensation unit 57, and the image interpolation unit 58. The frame memory 51 stores and outputs the input image in units of frames (outputs with a delay of one frame time). The frame memory 51 stores a frame at time t one before the input image at time t + 1. The frame at time t stored in the frame memory 51 is supplied to the vector detection unit 52, the vector allocation unit 54, the allocation compensation unit 57, and the image interpolation unit 58. Hereinafter, the frame at time t on the frame memory 51 is referred to as frame t, and the frame of the input image at time t + 1 is referred to as frame t + 1.

  The vector detection unit 52 detects a motion vector between the target block of the frame t on the frame memory 51 and the target block of the frame t + 1 of the input image, and stores the detected motion vector in the detection vector memory 53. As a method for detecting the motion vector between the two frames, a gradient method or a block matching method is used. Details of the configuration of the vector detection unit 52 will be described later with reference to FIG. The detection vector memory 53 stores the motion vector detected by the vector detection unit 52 in the frame t.

  The vector allocation unit 54 distinguishes the motion vector obtained on the frame t of the 24P signal from the frame of the 60P signal to be interpolated in the allocation vector memory 55 (hereinafter, the frame of the 60P signal is distinguished from the frame of the 24P signal). , Which is also referred to as an interpolation frame), and the allocation flag of the allocation flag memory 56 of the pixel to which the motion vector is allocated is rewritten to 1 (true). Details of the configuration of the vector allocation unit 54 will be described later with reference to FIGS. 16 and 43.

  The allocation vector memory 55 stores the motion vector allocated by the vector allocation unit 54 in association with each pixel of the interpolation frame. The allocation flag memory 56 stores an allocation flag indicating the presence / absence of a motion vector to be allocated for each pixel of the interpolation frame. For example, an assignment flag of true (1) indicates that a motion vector is assigned to the corresponding pixel, and an assignment flag of false (0) indicates that a motion vector is not assigned to the corresponding pixel. Show.

  The allocation compensation unit 57 refers to the allocation flag of the allocation flag memory 56, and supplements the motion vector of the peripheral pixel of the target pixel with respect to the target pixel for which the motion vector has not been allocated by the vector allocation unit 54. Allocate on 55 interpolation frames. At this time, the allocation compensator 57 rewrites the allocation flag of the target pixel to which the motion vector is allocated to 1 (true). Details of the configuration of the allocation compensator 57 will be described later with reference to FIG.

  The image interpolation unit 58 interpolates and generates the pixel value of the interpolation frame using the motion vector allocated to the interpolation frame in the allocation vector memory 55 and the pixel values of the frame t and the next frame t + 1. Then, the image interpolation unit 58 outputs the generated interpolation frame, and then outputs a frame t + 1 as necessary, thereby outputting the 60P signal image to the subsequent image signal output unit 4. To do. In the following, the pixel value is also referred to as a luminance value as appropriate.

  FIG. 6 is a diagram for explaining the principle of processing in the signal processing device 3. In the example of FIG. 6, a dotted line represents a frame of a 24P signal input at time t, t + 1, and t + 2 input to the signal processing device 3, and a solid line is represented by the signal processing device 3 from the input 24P signal. , The 60P signal interpolation frames at the generated times t, t + 0.4, t + 0.8, t + 1.2, t + 1.6, and t + 2.

  Generally, in order to convert a 24P signal into a 60P signal, 5/2 times as many frames are required. That is, five 60P signal images must be generated from two 24P signal images. At this time, the generated interpolated frame of the 60P signal has a time phase of 0.0, 0.4, 0.8, 1.2, and 1 on the 24P signal in order to equalize the frame interval. 6 is arranged at a position. Among these, 4 frames (frames of t + 0.4, t + 0.8, t + 1.2, and t + 1.6) except for one frame at time t with a time phase of 0.0 do not exist on the 24P signal. It is an image. Therefore, when the image of the 24P signal is input, the signal processing device 3 generates four interpolation frames from the two frames at the time t and the time t + 1 of the 24P signal. Therefore, the signal processing device 3 outputs a 60P signal image composed of five frames at times t, t + 0.4, t + 0.8, t + 1.2, and t + 1.6.

  As described above, the signal processing device 3 executes the process of converting the frame frequency from the 24P signal image to the 60P signal image.

  In principle, as described above, from the two frames at time t and time t + 1 of the 24P signal, five frames at times t, t + 0.4, t + 0.8, t + 1.2, and t + 1.6 are obtained. A frame of 60P signal is newly generated. Actually, in the case of the example of FIG. 6, 60P of t, t + 0.4, and t + 0.8 is based on two frames at time t and time t + 1 of the 24P signal. A frame of the signal is generated, and a frame of the 60P signal of t + 1.2, t + 1.6, and t + 2 is generated based on the two frames at the times t + 1 and t + 2 of the 24P signal.

  FIG. 7 is a diagram for explaining the processing of the signal processing device 3 more specifically. In the example of FIG. 7, a thick arrow indicates a transition to each state, and an arrow T indicates a time passage direction in the states 81 to 85. The states 81 to 85 are the frame t at the time t of the 24P signal, the frame t + 1 at the time t + 1 next to the time t, or the frame t and the frame t + 1 at the time of input / output to each unit constituting the signal processing device 3. The state of the interpolation frame F of the 60P signal produced | generated during is shown notionally. That is, in practice, for example, a frame in which a motion vector is detected as shown in the state 82 is not input to the vector allocation unit 54, but the frame and the motion vector are separately input to the vector allocation unit 54. The

  The state 81 represents the state of the frame t and the frame t + 1 of the 24P signal input to the vector detection unit 52. A black dot on the frame t in the state 81 represents a pixel on the frame t. The vector detection unit 52 detects the position where the pixel on the frame t in the state 81 moves in the frame t + 1 of the next time, and the movement is indicated as shown on the frame t in the state 82. It outputs as a motion vector corresponding to each pixel. As a method for detecting the motion vector between the two frames, a block matching method or a gradient method is used. At this time, when a plurality of motion vectors are detected in the pixel, the vector detection unit 52 obtains an evaluation value to be described later with reference to FIG. 8 for each motion vector, and calculates a motion vector based on the evaluation value. select.

  A state 82 represents the state of the frame t and the frame t + 1 input to the vector allocation unit 54. In the state 82, the arrow of each pixel of the frame t represents the motion vector detected by the vector detection unit 52.

  The vector allocating unit 54 extends the motion vector detected for each pixel of the frame t in the state 82 to the next frame t + 1 and sets it to a preset time phase (for example, t + 0.4 in FIG. 6). A position on a certain interpolation frame F is obtained. This is because, assuming that there is a constant motion between the frame t and the frame t + 1, the point where the motion vector passes through the interpolation frame F is the pixel position in that frame. Therefore, the vector allocating unit 54 allocates the passing motion vector to four neighboring pixels on the interpolation frame F in the state 83. At this time, depending on the pixel of the interpolation frame, there may be a case where no motion vector exists, or a plurality of motion vectors may be candidates for allocation. In the latter case, like the vector detection unit 52, the vector allocation unit 54 obtains an evaluation value for each motion vector and selects a motion vector to be allocated based on the evaluation value.

  A state 83 represents the state of the frame t and the frame t + 1 and the interpolation frame F to which the motion vector is assigned, which is input to the assignment compensation unit 57. In the interpolation frame F in the state 83, the pixels to which the motion vector is assigned by the vector assigning unit 54 and the pixels to which the motion vector is not assigned are shown.

  The allocation compensation unit 57 compensates for a pixel to which a motion vector in the state 83 is not allocated by using a motion vector allocated to a peripheral pixel of the pixel. This is processing based on the fact that the motion vector of the peripheral pixel of the pixel of interest and the motion vector of the pixel of interest are similar if the assumption that the neighboring region of the pixel of interest has the same motion holds. As a result, a motion vector that is accurate to some extent is also given to a pixel to which no motion vector is assigned, and a motion vector is assigned to all the pixels on the interpolation frame F in the state 84. In this case as well, since motion vectors of a plurality of surrounding pixels exist as candidates, the allocation compensator 57 obtains an evaluation value for each motion vector in the same manner as the vector allocation unit 54, and based on the evaluation values Select the motion vector to be assigned.

  A state 84 represents the state of the frame t and the frame t + 1 input to the image interpolation unit 58 and the state of the interpolation frame F in which motion vectors are assigned to all the pixels. Based on the motion vectors assigned to all these pixels, the image interpolation unit 58 can determine the positional relationship between the pixels on the interpolation frame F and the two frames t and t + 1. Therefore, the image interpolation unit 58 uses the motion vector allocated on the interpolation frame F and the pixel values of the frame t and the frame t + 1 to interpolate as indicated by the black dot of the interpolation frame F in the state 85. Pixel values on the frame F are generated by interpolation. Then, the image interpolation unit 58 outputs the generated interpolation frame, and then outputs a frame t + 1 as necessary, thereby outputting the 60P signal image to the subsequent image signal output unit 4. To do.

  Next, an evaluation value of a motion vector used in the signal processing device 3 will be described with reference to FIG. As described above with reference to FIG. 7, in each unit of the signal processing device 3 (the vector detection unit 52, the vector allocation unit 54, and the allocation compensation unit 57), an optimal motion vector is selected for subsequent processing. At this time, in each part of the signal processing device 3, as an evaluation value for the motion vector, a sum of absolute differences (DFD (Displaced Frame Difference) representing a correlation value between blocks shifted by an amount corresponding to the vector of interest of the two frames). )) Is used.

  In the example of FIG. 8, the m × n block centered on the pixel position p on the frame t at the time t and the vector amount of the motion vector v of interest are shifted from the pixel position p on the frame t + 1 at the time t + 1. Two blocks of m × n blocks centered on the pixel position p + v are shown. The difference absolute value sum DFDt (p) obtained between these two blocks is expressed by the following equation (1).

・・・（１）

... (1)

  Here, Ft (p) represents the luminance value at the pixel position p at time t, and m × n represents the DFD calculation range (block) for obtaining the sum of absolute differences. Since this difference absolute value sum represents a correlation value between DFD calculation ranges (blocks) in two frames, generally, the smaller the difference absolute value sum, the more the waveform of the block between the frames matches. Thus, it is determined that the reliability of the motion vector v is higher as the difference absolute value sum is smaller. As a result, this sum of absolute differences is used when the most probable motion vector is selected from a plurality of candidates.

  Therefore, hereinafter, in each unit (vector detection unit 52, vector allocation unit 54, and allocation compensation unit 57) of signal processing device 3, as an evaluation value when a motion vector is selected, a difference is obtained unless otherwise specified. An absolute value sum (hereinafter referred to as an evaluation value DFD) is used.

  Next, the process of converting the frame frequency of the signal processing device 3 will be described with reference to the flowchart of FIG.

  In step S1, the vector detection unit 52 inputs the pixel value of the frame. That is, the pixel value of the frame t + 1 of the input image at the time t + 1 and the frame t at the time t immediately before the input image stored in the frame memory 51 are input. At this time, the vector allocating unit 54, the allocation compensating unit 57, and the image interpolating unit 58 also have the pixel values of the frame t + 1 of the input image at time t + 1 and the frame t of time t before the input image of the frame memory 51. Enter.

  In step S2, the vector detection unit 52 executes motion vector detection processing. That is, the vector detection unit 52 detects a motion vector between the target block of the frame t on the frame memory 51 and the target block of the next frame t + 1 that is the input image, and the detected motion vector is detected by the detection vector memory 53. To remember. As a method for detecting the motion vector between the two frames, a gradient method or a block matching method is used. When there are a plurality of motion vector candidates, an evaluation value DFD is obtained for each motion vector, and a highly reliable motion vector based on the obtained evaluation value DFD is detected. That is, in this case, the most probable motion vector is selected and detected in the target block for detecting the motion vector. Details of the motion vector detection process in step S2 will be described later with reference to FIG.

  In step S3, the vector allocation unit 54 executes a vector allocation process. That is, in step S3, the vector allocation unit 54 allocates the motion vector obtained on the frame t to the pixel of interest on the interpolation frame to be interpolated in the allocation vector memory 55, and the pixel to which the motion vector is allocated. The allocation flag in the allocation flag memory 56 is rewritten to 1 (true). For example, an assignment flag that is true indicates that a motion vector is assigned to the corresponding pixel, and an assignment flag that is false indicates that a motion vector is not assigned to the corresponding pixel. When there are a plurality of motion vector candidates in each pixel, an evaluation value DFD is obtained for each motion vector, and a highly reliable motion vector based on the obtained evaluation value DFD is assigned. That is, in this case, the most probable motion vector is selected and assigned in the target pixel to which the motion vector is assigned. Details of the vector allocation processing in step S3 will be described later with reference to FIGS.

  In step S4, the allocation compensator 57 executes an allocation compensation process. That is, the allocation compensation unit 57 refers to the allocation flag in the allocation flag memory 56 in step S4, and for the pixel of interest for which no motion vector has been allocated by the vector allocation unit 54, the motion vector of the surrounding pixels of the pixel of interest. Is allocated on the interpolation frame of the allocation vector memory 55. At this time, the allocation compensator 57 compensates the motion vector and rewrites the allocation flag of the allocated pixel of interest to 1 (true). When there are a plurality of motion vectors of peripheral pixels, an evaluation value DFD is obtained for each motion vector, and a highly reliable motion vector based on the obtained evaluation value DFD is assigned. That is, in this case, the most probable motion vector is selected and assigned in the target pixel to which the motion vector is assigned. Details of the allocation compensation processing in step S54 will be described later with reference to FIG.

  In step S5, the image interpolation unit 58 executes an image interpolation process. That is, in step S5, the image interpolation unit 58 uses the motion vector allocated to the interpolation frame in the allocation vector memory 55 and the pixel values of the frame t and the frame t + 1 to generate an interpolation frame pixel value. . Details of the image interpolation processing in step S5 will be described later with reference to FIG. In step S6, the image interpolation unit 58 outputs an image of the 60P signal (outputs the generated interpolation frame), and then outputs a frame t + 1 as necessary, thereby outputting an image of the 60P signal. Is output to the subsequent image signal output unit 4.

  In step S7, the vector detection unit 52 determines whether all the frames have been processed. If the vector detection unit 52 determines that all the frames have not been processed yet, the process returns to step S1 and the subsequent processing. Is repeated. On the other hand, when the vector detection unit 52 determines in step S7 that all the frames have been processed, the vector detection unit 52 ends the process of converting the frame frequency.

  As described above, the signal processing device 3 detects the motion vector from the frame of the input image of the 24P signal, allocates the detected motion vector to the pixel on the frame of the 60P signal, and based on the allocated motion vector, A pixel value on the frame of the 60P signal is generated. At this time, the signal processing device 3 selects a motion vector with higher reliability based on the evaluation value DFD (sum of absolute differences) in each process. Therefore, in the signal processing device 3, it is possible to suppress the failure of the movement and the like and to generate a more accurate image.

  Next, details of the configuration of the image interpolation unit 58 will be described.

  FIG. 10 is a block diagram illustrating a configuration of the image interpolation unit 58. The image interpolating unit 58 having the configuration shown in FIG. 10 interpolates and generates the pixel value of the interpolation frame using the motion vector allocated to the interpolation frame of the allocation vector memory 55 and the pixel values of the frame t and the frame t + 1. , 60P signal image processing is performed.

  In the example of FIG. 10, the frame t of the image at time t is input to the spatial filter 92-1, and the frame t + 1 of the image at time t + 1 is input to the spatial filter 92-2 and the buffer 95.

  The interpolation control unit 91 selects the pixel of the interpolation frame in the allocation vector memory 55, and based on the motion vector assigned to the selected pixel, the pixel on the interpolation frame, the two frames t and t + 1 The positional relationship (spatial shift amount) with each pixel is obtained. That is, the interpolation control unit 91 uses, based on the pixels of the interpolation frame, the position on the frame t associated with the motion vector and the position of the pixel on the frame t corresponding to the pixel of the interpolation frame, The spatial shift amount is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-1. Similarly, the interpolation control unit 91 uses, based on the pixels of the interpolation frame, the position on the frame t + 1 associated with the motion vector and the position of the pixel on the frame t + 1 corresponding to the pixel of the interpolation frame. Is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-2.

  Further, the interpolation control unit 91 obtains an interpolation weight between the frame t and the frame t + 1 based on a preset time phase (time) of the interpolation frame, and uses the obtained interpolation weight as a multiplier 93-1. And 93-2. For example, when the time of the interpolation frame is a time that is “k” away from the time t + 1 of the frame t + 1 and a time that is “1−k” away from the time t of the frame t (ie, the time of the interpolation frame is the time t and time t + 1 is “1−k”: generated at a time that is internally divided into “k”), the interpolation control unit 91 sets an interpolation weight of “1-k” in the multiplier 93-1. An interpolation weight of “k” is set in the multiplier 93-2.

  Spatial filters 92-1 and 92-2 are constituted by cubic filters, for example. The spatial filter 92-1 is a pixel value on the frame t corresponding to the pixel of the interpolation frame based on the input pixel value of the pixel on the frame t and the spatial shift amount supplied from the interpolation control unit 91. And the obtained pixel value is output to the multiplier 93-1. The spatial filter 92-2, based on the input pixel value of the pixel on the frame t + 1 and the spatial shift amount supplied from the interpolation control unit 91, the pixel value on the frame t + 1 corresponding to the pixel of the interpolation frame And the obtained pixel value is output to the multiplier 93-2.

  When the position of the pixel in the interpolation frame does not match the position of the pixel on the frame t or the frame t + 1 (that is, the position of the pixel in the interpolation frame is a component equal to or less than the pixel in the frame t or the frame t + 1 (pixel And the spatial filters 92-1 and 92-2 use the pixel values of the four surrounding pixels at the pixel position of the interpolation frame in the frame t or the frame t + 1. The pixel value on the frame corresponding to the pixel of the interpolation frame is obtained by calculating the sum of the inverse ratios of the distances. That is, the pixel values at the positions below the pixel are obtained by linear interpolation based on the distance to the surrounding four pixels (details will be described later with reference to FIG. 17).

  The multiplier 93-1 multiplies the pixel value on the frame t input from the spatial filter 92-1 by the interpolation weight “1-k” set by the interpolation control unit 91, and the weighted pixel value is The result is output to the adder 94. The multiplier 93-2 multiplies the pixel value on the frame t + 1 input from the spatial filter 92-2 by the interpolation weight “k” set by the interpolation control unit 91, and adds the weighted pixel value to the adder. Output to 94.

  The adder 94 generates the pixel value of the pixel of the interpolation frame by adding the pixel value input from the multiplier 93-1 and the pixel value input from the multiplier 93-2. The pixel value of the interpolation frame is output to the buffer 95. The buffer 95 buffers the input frame t + 1. The buffer 95 outputs the generated interpolation frame, and then outputs a buffered frame t + 1 as necessary based on a preset time phase (time) of the 60P frame. Thus, the 60P signal image is output to the subsequent stage.

  Details of the image interpolation processing of the image interpolation unit 58 configured as described above will be described with reference to the flowchart of FIG.

  In step S51, the interpolation control unit 91 obtains an interpolation weight of the interpolation frame to be processed. That is, based on the time phase of the interpolation frame to be processed, interpolation weights (for example, “k” and “1-k”) of the interpolation frame between frames t and t + 1 are obtained, and the obtained interpolation weights are obtained. Are set in multipliers 93-1 and 93-2, respectively. In step S52, the interpolation control unit 91 selects a pixel on the interpolation frame of the allocation vector memory 55. Note that the pixels on the interpolation frame are selected one by one in the raster scan order from the upper left pixel of the frame.

  In step S53, the interpolation control unit 91 obtains the spatial shift amount based on the motion vector of the selected pixel. That is, based on the motion vector assigned to the pixel selected in step S52, the positional relationship (spatial shift amount) between the pixel on the interpolation frame and the two frames t and t + 1 is obtained. The obtained spatial shift amounts are supplied to the spatial filters 92-1 and 92-2, respectively. Specifically, in step S53, the interpolation control unit 91 uses the pixel of the interpolation frame as a reference, the position on the frame t associated with the motion vector, and the frame t corresponding to the pixel of the interpolation frame. The spatial shift amounts are obtained from the pixel positions, and the obtained spatial shift amounts are supplied to the spatial filter 92-1. Similarly, the interpolation control unit 91 uses, based on the pixels of the interpolation frame, the position on the frame t + 1 associated with the motion vector and the position of the pixel on the frame t + 1 corresponding to the pixel of the interpolation frame. Is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-2.

  The pixel value of the frame t of the image at time t is input to the spatial filter 92-1, and the pixel value of the frame t + 1 of the image at time t + 1 is input to the spatial filter 92-2. In step S54, the spatial filters 92-1 and 92-2 obtain pixel values on each frame corresponding to the pixels of the interpolation frame. That is, the pixel value on each frame corresponding to the pixel of the interpolation frame is obtained based on the pixel value of the pixel on the input frames t and t + 1 and the spatial shift amount supplied from the interpolation control unit 91. The obtained pixel values are output to multipliers 93-1 and 93-2, respectively.

  Multipliers 93-1 and 93-2 weight the pixel values in step S55. That is, the pixel value on each frame input from the spatial filter 92-1 or 92-2 is weighted with the interpolation weight set by the interpolation control unit 91, and the weighted pixel value is output to the adder 94. The Specifically, the multiplier 93-1 multiplies the pixel value on the frame t input from the spatial filter 92-1 by the interpolation weight “1 -k” set by the interpolation control unit 91 and performs weighting. The obtained pixel value is output to the adder 94. The multiplier 93-2 multiplies the pixel value on the frame t + 1 input from the spatial filter 92-2 by the interpolation weight “k” set by the interpolation control unit 91, and adds the weighted pixel value to the adder. Output to 94.

  In step S56, the adder 94 generates a pixel value of the selected pixel. That is, by adding the pixel value weighted by the multiplier 93-1 and the pixel value weighted by the multiplier 93-2, the pixel value of the pixel of the interpolation frame is generated, and the generated pixel value is Are output to the buffer 95. When the interpolation control unit 91 determines in step S57 whether or not the processing for all the pixels on the interpolation frame has been completed, and determines that the processing for all the pixels on the interpolation frame has not been completed yet The processing returns to step S52, and the subsequent processing is repeated. If the interpolation control unit 91 determines in step S57 that the processing for all the pixels on the interpolation frame has been completed, the interpolation control unit 91 ends the image interpolation processing.

  As described above, the pixel value of the interpolation frame is generated based on the motion vector assigned to the interpolation frame, and the interpolation frame is output by the buffer 95 in step S6 of FIG. 9 described above. If necessary, the frame t + 1 is output, so that an image of the 60P signal is output to the subsequent stage. Therefore, since the most probable motion vector is assigned to the pixel of the interpolation frame, an accurate interpolation frame can be generated.

  Next, details of the configuration of the vector detection unit 52 will be described.

  FIG. 12 is a block diagram illustrating a configuration of the vector detection unit 52. The vector detection unit 52 having the configuration shown in FIG. 12 detects a motion vector on the frame t by using the input frame t of the image at time t and the frame t + 1 of the image at time t + 1. The process of detecting the motion vector is executed for each predetermined block composed of a plurality of pixels.

  The initial vector selection unit 101 uses, as an initial vector V0 that is an initial value used in the gradient method, a high-reliability motion vector obtained from the detection result of the past motion vector for each predetermined block, as an iterative gradient method calculation unit. To 103. Specifically, the initial vector selection unit 101 uses the motion vectors of the peripheral blocks obtained in the past stored in the detection vector memory 53 and the shifted initial vectors stored in the shifted initial vector memory 107 as initial vectors. As a candidate vector. Then, the initial vector selection unit 101 obtains the evaluation value DFD of the candidate vector using the frame t and the frame t + 1, and selects the candidate vector having the highest reliability based on the obtained evaluation value DFD. And output as an initial vector V0.

  The pre-filters 102-1 and 102-2 are configured by a low-pass filter and a Gaussian filter, respectively, and remove noise components of the frame t and the frame t + 1 of the input image, and output them to the iterative gradient method computing unit 103.

  The iterative gradient method computing unit 103 uses the initial vector V0 input from the initial vector selection unit 101 and the frame t and the frame t + 1 input via the prefilters 102-1 and 102-2, for each predetermined block. Then, the motion vector Vn is calculated by the gradient method. The iterative gradient method computing unit 103 outputs the initial vector V0 and the calculated motion vector Vn to the vector evaluation unit 104. In addition, the iterative gradient method computing unit 103 repeatedly performs the gradient method computation based on the motion vector evaluation result by the vector evaluation unit 104 to calculate the motion vector Vn.

  The vector evaluation unit 104 obtains the motion vector Vn−1 (or initial vector V0) from the iterative gradient method computing unit 103 and the evaluation value DFD of the motion vector Vn, and based on the obtained evaluation value DFD, the iterative gradient method The calculation unit 103 is controlled to repeatedly execute the calculation of the gradient method, and finally, a highly reliable one based on the evaluation value DFD is selected and stored in the detection vector memory 53 as a motion vector V. At this time, the vector evaluation unit 104 supplies the evaluation value DFD obtained for the motion vector V together with the motion vector V to the shifted initial vector allocation unit 105.

  When the motion vector V and the evaluation value DFD are supplied from the vector evaluation unit 104, the shifted initial vector allocation unit 105 shifts the motion vector passing through the target block on the next frame to the target block. Set as initial vector. In other words, the shifted initial vector allocation unit 105 shifts a motion vector of the same size and the same direction as the motion vector V, starting from the target block on the next frame at the same position as the end block of the motion vector V. Set as initial vector. Then, the shifted initial vector allocation unit 105 allocates the set shifted initial vector to the shifted initial vector memory 107 in association with the block of interest.

  Specifically, the shifted initial vector allocation unit 105 stores the evaluation value DFD of the motion vector V allocated as the shifted initial vector in the evaluation value memory 106 in association with the target block, and stores the same target block. It is compared with the evaluation value DFD of another motion vector V that passes (that is, a block of a past frame at the same position as the target block). Then, the shifted initial vector allocation unit 105 shifts the motion vector V determined to have high reliability based on the evaluation value DFD to the target block, and allocates it to the shifted initial vector memory 107 as the shifted initial vector of the target block.

Next, the principle of the gradient method used in the vector detection unit 52 will be described. First, a luminance value of a pixel represented by coordinates (x, y, t) using a horizontal, vertical, and time axis in a moving image is assumed to be g (x, y, t). Here, when the pixel of interest (x ₀ , y ₀ , t ₀ ) is displaced by (dx, dy, dt) during a minute time, the horizontal (vertical) and time axis gradients (difference differences) are respectively expressed as gx. When expressed as (x ₀ , y ₀ , t ₀ ), gy (x ₀ , y ₀ , t ₀ ), and gt (x ₀ , y ₀ , t ₀ ), the luminance value of the pixel after displacement is approximated by Taylor expansion. Is expressed by the following equation (2).

g (x ₀ + dx, y ₀ + dy, t ₀ + dt)
≒ g (x ₀ , y ₀ , t ₀ ) + gx (x ₀ , y ₀ , t ₀ ) dx
+ Gy (x ₀ , y ₀ , t ₀ ) dy + gt (x ₀ , y ₀ , t ₀ ) dt
... (2)

  Here, when a pixel of interest in a moving image moves by horizontal vx and vertical vy after one frame (hereinafter, expressed as (vx, vy)), the luminance value of the pixel is expressed by the following equation (3). Is done.

_{g (x 0 + vx, y} 0 + vy, t 0 +1) = g (x 0, y 0, t 0)
... (3)

  When Expression (2) is substituted into Expression (3), it is expressed by the following Expression (4).

gx (x ₀ , y ₀ , t ₀ ) vx + gy (x ₀ , y ₀ , t ₀ ) vy
+ Gt (x ₀ , y ₀ , t ₀ ) = 0
... (4)

  Since the equation (4) is a two-variable equation of vx and vy, the solution cannot be obtained with a single equation for the target pixel. Therefore, as described below, a block that is a peripheral region of the target pixel is considered as one processing unit, and it is assumed that all pixels in the block (peripheral region) have the same movement (vx, vy). Build a similar formula. Assuming an assumption, an expression for the number of neighboring pixels is obtained for two variables. Therefore, these equations are combined to obtain (vx, vy) that minimizes the sum of squares of motion compensation frame differences of all pixels in the block.

  When the pixel (x, y, t) moves by (vx, vy) by one frame, the motion compensation inter-frame difference d is expressed by the following equation (5).

d = g (x + vx, y + vy, t + 1) −g (x, y, t)
= Δxvx + Δyvy + Δt
... (5)

  In equation (5), Δx = gx (x, y, t), which represents a horizontal gradient, Δy = gy (x, y, t), which represents a vertical gradient, Δt = gt ( x, y, t), which represents the gradient in the time direction. Using these, assuming that the square sum of the difference between motion compensation frames is E, it is expressed by Equation (6).

E = Σd ²
= Σ (Δx ² vx ² + Δy ² vy ² + 2ΔxΔyvxvy
+ 2ΔxΔtvx + 2ΔyΔtvy + Δt ² )
= Vx ² ΣΔx ² + vy ² ΣΔy ² +2 vxvy ΣΔxΔy
+ 2vxΣΔxΔt + 2vyΣΔyΔt + ΣΔt ²
... (6)

  Here, E becomes the minimum (vx, vy) when the partial differential value in each variable becomes 0, that is, when the condition of δE / δvx = δE / δvy = 0 is satisfied. Thus, the following equations (7) and (8) are obtained.

vxΣΔx ² + vyΣΔxΔy + ΣΔxΔt = 0
... (7)
vyΣΔy ² + vxΣΔxΔy + ΣΔyΔt = 0
... (8)

  From these equations (7) and (8), (vx, vy) which is the desired motion can be obtained by calculating the following equation (9).

・・・（９）

... (9)

  The principle of the above gradient method will be specifically described with reference to FIG. In the example of FIG. 13, an arrow X indicates the horizontal direction, and an arrow Y indicates the vertical direction. In addition, an arrow T indicates the direction of time passage from the frame t at the time t in the back right to the frame t + 1 at the time t + 1 in the left front. In the example of FIG. 13, each frame shows only an area of 8 pixels × 8 pixels used for the gradient method calculation as a peripheral area (block) of the pixel of interest p.

  In the frame t, when the motion vector V (vx, vy) of the pixel of interest p, which is the fifth pixel down from the upper left pixel and the fifth pixel on the right, is obtained using the gradient method described above, the motion vector V (vx , Vy) is the difference in luminance (ie, gradient) Δx and Δy between adjacent pixels px and py obtained for the x and y directions of the pixel of interest p, and the same phase of the pixel of interest p obtained in frame t + 1. A difference in luminance (gradient) Δt in the time direction with respect to the pixel q located is obtained for all pixels in the peripheral region (8 pixels × 8 pixels) of the pixel of interest p, and the difference between them is expressed by Equation (9). It can obtain | require by calculating using.

  That is, the gradient method calculates gradients Δx, Δy, and Δt between two frames, and statistically calculates the motion vector V (vx, Δt from the obtained Δx, Δy, and Δt using the sum of squares of differences. vy).

  Generally, in a motion vector detection method using such a gradient method, a highly accurate result can be obtained for a minute motion. However, when trying to obtain a motion in an actual moving image, this gradient method is not practical because the amount of motion is too large. Correspondingly, a method of repeating this gradient method a plurality of times is conceivable. By repeatedly executing the gradient method, the amount of motion obtained by each calculation converges, so that a correct motion is gradually obtained.

  However, it is not practical from the viewpoint of calculation time when trying to perform real-time processing only by repeating the gradient method. Therefore, the vector detection unit 52 reduces the number of iterations of the gradient method by using, as an initial value, an initial vector obtained based on the motion of surrounding pixels in the past frame and the current frame. In other words, if a rough motion is calculated by adding an offset in advance from the pixel of interest that is the starting point of the motion to the destination indicated by the initial vector, an operation using the gradient method is performed from the position where the offset is added. Fine adjustment including movement below the pixel can be performed. Thereby, an accurate motion vector can be detected without increasing the calculation time.

  FIG. 14 is a diagram for specifically explaining the iterative gradient method executed using the initial vector. In the example of FIG. 14, the arrow T indicates the passage of time from the frame t at the time t on the left front side to the frame t + 1 at the time t + 1 on the right back in the drawing. A block centered on each pixel p, q0, q1, q2, and q3 represents a peripheral region (block) used for the gradient method calculation of that pixel.

  In the case of the example in FIG. 14, the initial vector v0 obtained in advance is offset (moved) in the frame t + 1, not the pixel q0 located in the same phase as the target pixel p in the frame t + 1. The first gradient method calculation is performed with the position (pixel) q1 calculated in this way as the starting point, and as a result, a motion vector v1 is obtained.

  Next, the second gradient method calculation is performed with the position (pixel) q2 calculated by offsetting v0 + v1 from the pixel q0 as a starting point, and as a result, a motion vector v2 is obtained. Thereby, the motion vector V is finally obtained as Expression (10).

V = v0 + v1 + v2
... (10)

  As described above, by performing the iterative gradient method using the initial vector, it is possible to obtain a highly accurate motion vector while reducing the calculation time.

  Next, details of the motion vector detection process will be described with reference to the flowchart of FIG. The vector detection unit 52 receives the input frame t of the image at time t and input frame t + 1 of the image at time t + 1.

  The initial vector selection unit 101 selects a block of interest in step S101. That is, one block to be processed on the frame t is selected as the target block. On the frame, the processing is executed in the raster scan order from the upper left block.

  In step S102, the initial vector selection unit 101 performs an initial vector selection process. In step S101, the initial vector selection unit 101 selects a motion vector with high reliability from the detection result of the past motion vector for each predetermined block, and uses the selected motion vector as an initial value used in the gradient method. The initial vector V0 is output to the iterative gradient method computing unit 103.

  That is, the initial vector selection unit 101 obtains the motion vector of the peripheral blocks obtained in the past gradient method calculation evaluation process (step S103 described later) and stored in the detection vector memory 53, and the past shifted initial vector allocation process (described later). In step S104), the shifted initial vector stored in the shifted initial vector memory 107 is selected as an initial vector candidate vector. Then, the initial vector selection unit 101 obtains the evaluation value DFD of the candidate vector using the frame t and the frame t + 1, and selects a candidate vector having a high reliability based on the obtained evaluation value DFD. The selected candidate vector is output as the initial vector V0.

  In step S103, the iterative gradient method calculation unit 103 and the vector evaluation unit 104 execute an iterative gradient method calculation evaluation process (also referred to as an iterative gradient method calculation process). Specifically, in step S103, the iterative gradient method computing unit 103 receives the initial vector V0 input from the initial vector selecting unit 101, the frame t and the frame input via the prefilters 102-1 and 102-2. Based on the motion vector evaluation result by the vector evaluation unit 104 using t + 1, the gradient method is repeatedly calculated to calculate the motion vector Vn. The vector evaluation unit 104 obtains the motion vector Vn−1 from the iterative gradient method computing unit 103 and the evaluation value DFD of the motion vector Vn, and selects the most reliable one based on the obtained evaluation value DFD. And stored in the detected vector memory 53 as a motion vector V. At this time, the vector evaluation unit 104 supplies the evaluation value DFD obtained for the motion vector V together with the motion vector V to the shifted initial vector allocation unit 105.

  In step S104, the shifted initial vector allocation unit 105 performs a shifted initial vector allocation process. When the motion vector V and its evaluation value DFD are supplied from the vector evaluation unit 104, the shifted initial vector allocation unit 105 shifts the motion vector passing through the target block on the next frame to the target block in step S104. The shifted initial vector is set. That is, in other words, a motion vector having the same size and the same direction as the motion vector V starting from the block of interest on the next frame at the same position as the end block of the motion vector V is set as the initial shift vector. . Then, the shifted initial vector allocation unit 105 allocates the set shifted initial vector to the shifted initial vector memory 107 in association with the block of interest.

  Specifically, the shifted initial vector allocation unit 105 stores the evaluation value DFD of the motion vector V allocated as the shifted initial vector in the evaluation value memory 106 in association with the target block, so that the same target Compared with the evaluation value DFD of another motion vector V that passes through the block (that is, the block of the past frame at the same position as the target block is the end point), the motion with high reliability based on the evaluation value DFD The vector V is shifted to that block and set as a shifted initial vector, and is allocated to the shifted initial vector memory 107 in correspondence with the shifted block.

  In step S105, the initial vector selection unit 101 determines whether or not all blocks have been processed in the frame t. If it is determined that all blocks have not been processed, the process proceeds to step S101. Return, and the subsequent processing is repeated. In step S105, the initial vector selection unit 101 determines that the motion vectors V have been detected in all the blocks on the frame t when it is determined that all the blocks have been processed in the frame t. Then, the motion vector detection process ends.

  As described above, an initial vector is selected from motion vectors detected in the past, an iterative gradient method is calculated based on the selected initial vector, and a repeated motion vector is calculated. Among them, a motion vector having a high reliability (that is, most likely) based on the evaluation value DFD is detected. As a result, the motion vector V corresponding to all the blocks on the frame t is stored in the detection vector memory 53.

  Next, details of the configuration of the vector allocation unit 54 will be described.

  FIG. 16 is a block diagram showing the configuration of the vector allocation unit 54. The vector allocating unit 54 having the configuration shown in FIG. 16 allocates motion vectors detected on the frame t using the frame t of the image at time t to which the 24P signal is input and the frame t + 1 of the image at time t + 1. A process of assigning to the pixels on the interpolation frame of the 60P signal to be interpolated on the vector memory 55 is performed.

  In the example of FIG. 16, the frame t of the image at time t and the frame t + 1 of the image at time t + 1 are input to the pixel information calculation unit 701, the evaluation value calculation unit 702, and the target pixel difference calculation unit 703.

  The pixel information calculation unit 701 acquires the motion vector detected by the pixel on the frame t in the detection vector memory 53 in the raster scan order from the upper left pixel, and extends the acquired motion vector in the frame t + 1 direction at the next time. The intersection of the extended motion vector and the interpolation frame is calculated. Then, the pixel information calculation unit 701 sets, from the intersection of the calculated motion vector and the interpolation frame, a pixel (hereinafter referred to as an allocation target pixel) to which the motion vector is assigned on the interpolation frame. The motion vector and the position information of the allocation target pixel are output to the vector selection unit 705. The image information calculation unit 701 calculates the position P of the frame t and the position Q on the frame t + 1 that are associated with the allocation target pixel and the motion vector, and the calculated position information on the frame t and the frame t + 1 The result is output to the evaluation value calculation unit 702 and the target pixel difference calculation unit 703.

  When the evaluation value calculation unit 702 receives from the pixel information calculation unit 701 position information on the frame t and the frame t + 1 associated with the allocation target pixel and the motion vector, the evaluation value calculation unit 702 receives the position P of the frame t and the position Q of the frame t + 1. In order to calculate the evaluation value DFD, a certain range of DFD calculation ranges (m × n) centering on the position P and the position Q are respectively set, and it is determined whether or not those DFD calculation ranges are within the image frame. . When the evaluation value calculation unit 702 determines that the DFD calculation range is within the image frame, the evaluation value calculation unit 702 calculates the evaluation value DFD of the allocation target pixel for the motion vector by calculating using the DFD calculation range, and the calculated evaluation value The DFD is output to the vector evaluation unit 704.

  When the pixel-of-interest difference calculation unit 703 receives from the pixel information calculation unit 701 position information on the frame t and the frame t + 1 associated with the allocation target pixel and the motion vector, the position P of the frame t and the position Q of the frame t + 1 Is used to obtain the absolute value of the luminance difference for the allocation target pixel, and the obtained absolute value of the luminance difference is output to the vector evaluation unit 704.

  The vector evaluation unit 704 includes a pixel difference determination unit 711 and an evaluation value determination unit 712. The pixel difference determination unit 711 determines whether the luminance difference absolute value for the allocation target pixel input from the target pixel difference calculation unit 703 is smaller than a predetermined threshold value. The evaluation value determination unit 712 calculates the evaluation value when the pixel difference determination unit 711 determines that the luminance difference absolute value for the allocation target pixel input from the target pixel difference calculation unit 703 is smaller than a predetermined threshold value. It is determined whether or not the evaluation value DFD of the allocation target pixel input from the unit 702 is smaller than the minimum evaluation value of the DFD table included in the vector selection unit 705. When the evaluation value determination unit 712 determines that the evaluation value DFD of the allocation target pixel is smaller than the minimum evaluation value of the DFD table, the evaluation value determination unit 712 determines that the reliability of the motion vector corresponding to the allocation target pixel is high, The evaluation value DFD of the allocation target pixel is output to the selection unit 705.

  The vector selection unit 705 has a DFD table that holds a minimum evaluation value at each pixel on the interpolation frame, and an evaluation value DFD0 when a 0 vector is assigned to each pixel on the interpolation frame. The minimum evaluation value for each pixel on the interpolation frame is stored in advance in the DFD table. When the vector selection unit 705 receives the evaluation value DFD of the allocation target pixel from the vector evaluation unit 704, the vector selection unit 705 sets the flag of the allocation flag memory 56 to 1 (based on the information on the position of the allocation target pixel from the pixel information calculation unit 701). true), and the minimum evaluation value in the DFD table of the allocation target pixel is rewritten to the evaluation value DFD of the allocation target pixel. Further, the vector selection unit 705 performs pixel information calculation on the allocation target pixel of the allocation vector memory 55 based on the position information of the allocation target pixel from the pixel information calculation unit 701 and the penetration risk flag from the penetration improvement processing unit 706. The motion vector from the unit 701 is allocated.

  The penetration improvement processing unit 706 detects a motion vector that causes the penetration degradation unit from among the motion vectors stored in the detection vector memory 53, and outputs the detection result to the vector selection unit 705 as a penetration risk flag. To do. When the penetration danger flag indicates that the penetration vector is a motion vector that causes the penetration degradation portion, the vector selection unit 705 prohibits the assignment of the motion vector. Details of the processing of the penetration improvement processing unit 706 will be described later with reference to FIGS. 24 and 40.

  Next, the accuracy below the pixel of the motion vector will be described. In the calculation of the DFD evaluation expressed by the above equation (1), the phase p + v on the previous frame t + 1 obtained by shifting the pixel position p of the frame t by the amount of the vector v is actually on the frame t + 1 of the 24p signal. In many cases, the luminance value is not defined. Therefore, in order to perform the calculation of the evaluation value DFD for the motion vector v having subpixel accuracy, it is necessary to generate a luminance value in a phase below the pixel by some method.

  Corresponding to this, there is a method of using the luminance value of the pixel closest to the phase p + v on the previous frame t + 1 obtained by shifting the pixel position p of the frame t by the vector v amount. However, in this method, the sub-pixel components of the motion vector to be evaluated are rounded, so the sub-pixel components of the motion vector are discarded, and the reliability of the obtained evaluation value DFD is thereby reduced. End up.

  Therefore, in the embodiment of the present invention, a four-point interpolation process using the luminance values of the surrounding four pixels is used. FIG. 17 is a diagram showing the concept of the four-point interpolation process of the present invention. In FIG. 17, the arrow X indicates the horizontal direction in the frame t + 1, and the arrow Y indicates the vertical direction in the frame t + 1. In this frame t + 1, the white circle represents the pixel position on the frame t + 1, and the black dot represents the position below the pixel. Further, the upper left black point p + v and its surrounding four pixels on the frame t + 1 are shown enlarged in the window E. In window E, the alphabet in the white circle indicates the luminance value of the surrounding four pixels.

If the black point p + v at the upper left in the frame t + 1 is the phase p + v after the pixel position p of the frame t is shifted by the vector v amount, the luminance value F _{t + 1} (p + v) of the phase p + v is the horizontal direction of the phase p + v. The pixel sub-component α and the pixel sub-component β in the vertical direction and the luminance values L0 to L3 of the four surrounding pixels of the phase p + v are obtained as the sum of the inverse ratios of the distances of the four surrounding pixels. That is, the luminance value F _{t + 1} (p + v) is expressed by the following equation (11).

F _{t + 1} (p + v) = (1-α) (1-β) L0 + α (1-β) L1
+ (1-α) βL2 + αβL3 (11)

As described above, the evaluation value DFD is calculated by using the luminance value F _{t + 1} (p + v) obtained by the four-point interpolation process, without increasing the cost in hardware implementation. A decrease in reliability can be suppressed. In the following, an example in which this four-point interpolation is applied in the calculation of the evaluation value DFD and the luminance difference absolute value at the time of vector assignment will be described. Of course, the initial vector selection process and the vector detection process described above, Alternatively, this four-point interpolation is also applied to the calculation of the evaluation value DFD in the case of evaluating a vector such as allocation compensation processing described later.

  Next, the concept of vector allocation processing will be described. For example, when it is assumed that an object is moving at a velocity v on a 24P signal and a constant velocity assumption is made for the motion of the object between any two frames, a new frame is interpolated between the frames of the 24P signal. Think about what to do. In this case, when the motion vector v is extended from the 24P signal object, the intersection of the motion vector v and the interpolation frame is the same object and has the same velocity v.

  Therefore, the motion vector of the frame of the 24P signal detected by the vector detection unit 53 (hereinafter also referred to as the original frame with respect to the interpolation frame) is represented by the motion vector and the interpolation frame of the 60P signal to be interpolated. By assigning them to the intersections, the movement of each pixel on the interpolation frame can be obtained. Conversely, from the allocated motion vector, it can be determined from which position the pixel on the interpolation frame has moved on the 24P signal frame.

  FIG. 18 shows one-dimensional examples of motion vectors detected in the original frame of the 24P signal and pixels on the interpolation frame of the 60P signal. In the example of FIG. 18, between the frame t at the time t and the frame t + 1 at the time t + 1 of the two 24P signals, for example, the interpolation frame F1 at the time t + 0.4 and the interpolation frame F2 at the time t + 0.8 are Two are inserted. As described above, the position of this interpolation frame is the position where the time phase on the 24P signal is 0.0, 0.4, 0.8, 1.2, and 1.6, and the 60P signal. Are set in advance in the signal processing device 3.

  A circle on each frame indicates each pixel. Motion vectors v1, v2, and v3 detected in the frame t by the preceding vector detection unit 52 are extended in the frame t + 1 direction. When these motion vectors are allocated to each pixel of the interpolation frames F1 and F2, the motion vector passing through the vicinity of each pixel on the interpolation frame is a candidate vector (hereinafter also referred to as allocation candidate vector) allocated to that pixel. Is done.

  Therefore, the motion vector v1 from the leftmost pixel of the frame t to the vicinity of the fourth and fifth pixels from the left of the frame t + 1 is between the second and third pixels from the left on the interpolation frame F1. It passes between the third and fourth pixels from the left on the insertion frame F2. Therefore, the motion vector v1 is a pixel included in the vicinity N1 of the point where the motion vector v1 intersects the interpolation frames F1 and F2 (second and third pixels from the left of the interpolation frame F1 and the left of the interpolation frame F2). To the third and fourth pixels).

  In addition, the motion vector v2 from the third pixel from the left of the frame t to the vicinity of the second and third pixels from the left of the frame t + 1 is between the second and third pixels from the left on the interpolation frame F1. , And passes between the second and third pixels from the left on the interpolation frame F2. Therefore, the motion vector v2 is a pixel included in the vicinity region N2 of the point where the motion vector v2 intersects the interpolation frames F1 and F2 (second and third pixels from the left of the interpolation frame F1 and the interpolation frame F2). Allocation candidate vectors to be allocated to the second and third pixels from the left).

  Further, the motion vector v3 from the fifth pixel from the left of the frame t to the vicinity of the fourth and fifth pixels from the left of the frame t + 1 is between the fourth and fifth pixels from the left on the interpolation frame F1. , And passes between the fourth and fifth pixels from the left on the interpolation frame F2. Therefore, the motion vector v3 includes the pixels included in the vicinity region N3 of the point where the motion vector v3 intersects the interpolation frames F1 and F2 (the fourth and fifth pixels from the left of the interpolation frame F1 and the interpolation frame F2). 4th and 5th pixels from the left).

  That is, the allocation candidate vector of the second pixel from the left of the interpolation frame F2 is the motion vector v2, and the second and third pixels from the left on the interpolation frame F1, and 3 from the left of the interpolation frame F2. The allocation candidate vectors of the th pixel are motion vectors v1 and v2. Further, the allocation candidate vectors of the fourth pixel from the left of the interpolation frame F2 are motion vectors v1 and v3, the fourth and fifth pixels from the left on the interpolation frame F1, and the left of the interpolation frame F2. The fifth pixel allocation candidate vector from is the motion vector v3.

  As described above, an allocation candidate vector to be allocated to each pixel on the interpolation frame is obtained from the motion vectors detected in the original frame. In the leftmost pixel and the rightmost pixel (black circles in the figure) of the interpolation frames F1 and F2, motion vectors passing through the vicinity are not shown. That is, there is no allocation candidate vector to be allocated to the leftmost pixel and the rightmost pixel of the interpolation frames F1 and F2. Therefore, an allocation compensation process is performed on these pixels in an allocation compensation unit 57 in a later stage described later.

Furthermore, with reference to FIG. 19, the motion vector detected in the original frame and the pixels on the interpolation frame of the 60P signal will be described in detail. In the example of FIG. 19, the arrow T indicates the direction of time passage from the frame t at the left front time t to the frame t + 1 at the right rear time t + 1. Further, interpolation frame F1 is placed at time t + pos _t between times t and time t + 1.

For example in FIG. 19, the pixel (x _a, y _a) of the frame t is detected by the motion vector _{v a (x va, y va} ) extend in frame t + 1 direction, the intersection of the interpolated frame F1 (x _ia , y _ia ) is calculated. Intersection, it is considered that pixels corresponding to end points of the motion vector v _a of the frame t of the 24P signal to a point moved, specifically, is expressed by the equation (12) and (13).

_{x ia = x a + pos t} x va ··· (12)
_{y ia = y a + pos t} y va ··· (13)

Here, as described above, when the motion vector v _a has a less precision pixels, the intersection of the motion vector v _a, pixel positions on the interpolation frame F1 does not always match. Otherwise, as shown in FIG. 19, the motion vector v _a is allocated with respect to the four neighboring pixels G1 to G4 of the intersection on the interpolated frame F1. That is, the motion vector v _a is shifted (parallel movement) on each of the pixels G1 to G4 in the neighborhood, is the allocation candidate vector to be allocated to each of the pixels, the allocation compensating process is executed.

  As described above, since one motion vector may be a candidate assigned to four neighboring pixels, a plurality of motion vectors may be assigned candidate vectors depending on the pixel. In this case, the vector allocating unit 54 calculates, for each motion vector, a pixel on the interpolation frame and an intersection point on the original frame associated with the motion vector, and evaluates each motion vector using the intersection point. Thus, finally, a motion vector to be assigned to the pixel on the interpolation frame is determined.

With reference to FIG. 20, the evaluation of the motion vector in the vector allocation part 55 is demonstrated. Figure 20 is a bottom, frame t, interpolated frame F1 at time t + pos _t at time t of FIG. 19, and the frame t + 1 at time t + 1, is one-dimensionally illustrated.

In the example of FIG. 20, the motion vector sv _a is the pixel (x _a, y _a) of the frame t detected motion vector v _a at is as allocation candidate vectors of the pixels in the neighborhood G4, shifts on the pixel G4 ( Translated). Here, the intersections of the motion vector sva, shifted onto the pixel G4, and the frame t and the frame t + 1 are defined as _a point P and a point Q, respectively.

The vector allocating unit 54, a first evaluation of motion vector sv _a, first, obtains a DFD operation range around the point P and point Q respectively, DFD operation range determined whether jutting out the image frame Judging. Therefore, when the DFD operation range around the point P and point Q had protruding a picture frame, the motion vector sv _a is excluded from the candidates.

Further, when the point P and the point Q belong to different objects, for example, the difference between the luminance value F _t (P) of the point P and the luminance value F _{t + 1} (Q) of the point Q becomes large. . Accordingly, the vector allocating unit 54, a second evaluation of motion vector sv _a, using the points P and Q, determine the luminance difference absolute value dp of the pixel G4, the luminance difference absolute value dp is greater than a predetermined value Determine whether or not. If the luminance difference absolute value dp is determined to be larger than the predetermined value is a low reliability of the motion vector sv _a in the pixel G4, a motion vector sv _a is excluded from the candidates. The luminance difference absolute value dp is expressed by the following equation (14).

dp = | F _t (P) − F _{t + 1} (Q) | (14)

Then, as the third evaluation of the motion vector sv _a, the vector allocating unit 54 performs evaluation determined by the difference absolute value that represents the correlation value DFD operation range around the point P and the point Q. That is, the vector allocating unit 54, using the DFD operation range around the point P and the point Q, obtains the evaluated value DFD of the motion vector sv _a (absolute differences) in the pixel G4, obtained evaluation value DFD is It is determined whether it is smaller than the minimum evaluation value of the DFD table. As a result of the above evaluation, the vector assigning unit 54 assigns a motion vector having the smallest evaluation value DFD among the obtained evaluation values DFD to the pixel G4.

  As described above, since the motion vector of the allocation candidate in the pixel of the interpolation frame is evaluated using not only the evaluation value DFD of the allocation target pixel but also the luminance difference absolute value in the allocation target pixel, the conventional evaluation A more reliable motion vector can be allocated to the allocation target pixel than when only the value DFD is used. As a result, the accuracy of vector allocation is improved.

  Further, when the penetration risk flag indicates that the penetration vector is a motion vector that causes the penetration deterioration portion, the vector assignment unit 54 prohibits the assignment of the motion vector. Thereby, it can suppress that a penetration deterioration part generate | occur | produces.

  As described above, in order to evaluate the motion vector of the allocation candidate in the pixel of the interpolation frame, the position on the original frame associated with the motion vector is used with reference to the pixel of the interpolation frame. When obtaining the absolute value of luminance difference dp and the evaluation value DFD, the motion vector of the allocation candidate is extended based on the pixel position of the interpolation frame, so that the intersection of the motion vector and the original frame is the pixel position of the original frame. The pixel values may not match, and the pixel value cannot be obtained as it is. In such a case, the four-point interpolation process described above with reference to FIG. 17 is executed.

  FIG. 21 shows an example of four-point interpolation in the vector allocation process. In FIG. 21, the parts corresponding to those in FIGS. 19 and 20 are denoted by the corresponding reference numerals, and the description thereof will be omitted to avoid repetition.

In the example of FIG. 21, the motion vector sv _a assignment candidate, because it is extended pixel position G4 in interpolated frame F1 as a reference, the point of intersection P between the motion vector sv _a frame t, in frame t does not coincide with the pixel position (white circle on the frame t), also intersecting point Q between the motion vector sv _a frame t + 1 also does not match the pixel positions on the frame t + 1 (open circles in frame t). Therefore, in the frame t and the frame t + 1, the above-described four-point interpolation calculation is performed using the four pixels in the vicinity E centered at the intersection point P and the intersection point Q (white circles on the frame t and the frame t + 1), respectively. Pixel values of P and intersection point Q are obtained.

  As described above, in this vector allocation process, the luminance difference absolute value dp and the evaluation value DFD are calculated using the pixel values of the intersection point P and the intersection point Q obtained by the four-point interpolation process. The luminance difference absolute value dp and the evaluation value DFD can be obtained with higher accuracy than the method of rounding the components.

  Next, the details of the vector allocation processing will be described with reference to the flowchart of FIG. The frame t of the image at time t and the frame t + 1 of the image at time t + 1, which are the original frames of the 24P signal, are input to the pixel information calculation unit 701, the evaluation value calculation unit 702, and the target pixel difference calculation unit 703.

  When a new original frame is input, the pixel information calculation unit 701 controls the vector selection unit 705 to initialize the allocation flag of the allocation flag memory 56 with 0 (false) in step S201. In step S202, the vector selection unit 705 is controlled by the pixel information calculation unit 701, and initializes the allocation vector memory 55 with the 0 vector. As a result, a 0 vector (a motion vector meaning that the motion is 0) is assigned to a pixel to which no motion vector is assigned.

  In step S203, the pixel information calculation unit 701 controls the evaluation value calculation unit 702 to calculate the evaluation value DFD0 using the 0 vector. That is, the evaluation value DFD0 is calculated for all the pixels on the interpolation frame using the 0 vector. Also, the image information calculation unit 701 controls the vector selection unit 705, and uses the zero vector evaluation value DFD0 calculated by the evaluation value calculation unit 702 as the minimum evaluation value for each pixel of the interpolation frame. Store in the DFD table. That is, in step S203, the evaluation value calculation unit 702 calculates the evaluation value DFD0 using the 0 vector for all pixels of the interpolation frame, and the calculated evaluation value DFD0 is passed through the vector evaluation unit 704. To the vector selection unit 705. Then, the vector selection unit 705 stores the evaluation value DFD0 input via the vector evaluation unit 704 as the minimum evaluation value of the corresponding pixel in the DFD table.

  The pixel information calculation unit 701 selects a pixel from the original frame on the detection vector memory 53 in step S204. In this case, pixels are selected one by one in the raster scan order from the upper left of the frame.

  In step S205, the penetration improvement processing unit 706 executes penetration improvement processing. The details thereof will be described later with reference to FIG. 24. As a result, a motion vector that may cause a penetration deterioration portion is detected, and the penetration danger flag of the motion vector is set to true. In step S206, the vector selection unit 705 determines whether the penetration risk flag of the pixel selected in step S204 is true.

  When the penetration risk flag of the selected pixel is not true (if false), the pixel information calculation unit 701 executes pixel position calculation processing in step S207. The details of this pixel position calculation processing will be described later with reference to FIG. 41, whereby the allocation target pixel on the interpolation frame to which the motion vector detected in the pixel selected in step S204 is allocated is calculated. Then, based on the calculated allocation target pixel, the position on the original frame associated with the motion vector is calculated.

  In step S208, the pixel information calculation unit 701 selects the calculated allocation target pixel, and outputs the selected allocation target pixel and its motion vector to the vector selection unit 705. At the same time, the pixel information calculation unit 701 outputs, to the evaluation value calculation unit 702 and the pixel-of-interest calculation unit 703, information on the position on the original frame associated with the motion vector based on the selected allocation target pixel. To do. In step S208, when there are a plurality of allocation target pixels, the pixel information calculation unit 701 selects from the upper left pixels.

  In step S209, the pixel information calculation unit 701 executes an allocation vector evaluation process for the selected allocation target pixel. The details of the allocation vector evaluation process will be described later with reference to FIG. 42, whereby the motion vector evaluation value DFD and the luminance difference absolute value in the allocation target pixel are obtained, and the reliability of the motion vector in the allocation target pixel is determined. As a result of these determinations, the motion vector in the allocation vector memory 55 is rewritten with the motion vector determined to have high reliability.

  In step S210, the pixel information calculation unit 701 determines whether or not the processing of all allocation target pixels has been completed. If all the allocation target pixels have not been processed yet, the process returns to step S208, the next allocation target pixel is selected, and the subsequent processing is repeated.

  If it is determined in step S210 that the processing for all the allocation target pixels has been completed, and if it is determined in step S206 that the penetration risk flag is true, the process proceeds to step S211. In step S <b> 211, the pixel information calculation unit 701 determines whether the processing of all the pixels in the original frame on the detection vector memory 53 has been completed. If it is determined in step S211 that the processing of all the pixels of the original frame on the detection vector memory 53 has not been completed, the process returns to step S204, and the next pixel of the original frame on the detection vector memory 53 is determined. Is selected, and the subsequent processing is repeated. If the pixel information calculation unit 701 determines in step S211 that the processing has been completed for all the pixels in the detection vector memory 53, the vector allocation processing ends.

  Next, the penetration improvement process will be described. Before that, the cause of the penetration deterioration portion will be described with reference to FIG. As shown in the figure, when the foreground 202 on the 24P frame t1 moves to the foreground 212 on the 24P frame t1 + 1, the line connecting the pixel 203 of the foreground 202 and the pixel 213 of the foreground 212 is A motion vector Vf is represented. Similarly, when the background pixel 201 on the frame t1 corresponds to the pixel 211 on the frame t1 + 1, the motion vector Vb represented by a line connecting the two becomes the motion vector of the pixel 201. The motion vector Vf moves in the right direction in the figure at a relatively small speed, whereas the motion vector Vb moves in the left direction in the figure at a relatively large speed. When a frame t + P of 60P is generated between the frame t1 and the frame t1 + 1, the pixel 204 needs to be assigned a motion vector.

  However, if the motion vector Vf of the foreground 202 is assigned to the pixel 204 through which both the motion vector Vf and the motion vector Vb pass, the pixel 204 is recognized as the pixel 203-3 of the foreground 202. When the motion vector Vb of the background pixel 201 is assigned, the pixel 204 is recognized as the pixel of the background pixel 201-3. Although the pixel 204 is originally a pixel of the foreground 202, when the background motion vector Vb is allocated thereto, the correct allocation is not performed. At this time, a penetrating deterioration portion is generated.

  Next, the details of the penetration improving process in step S205 of FIG. 22 will be described with reference to the flowchart of FIG.

  In step S251, the penetration improvement processing unit 706 initially sets the penetration danger flag to false. This penetration risk flag is left as it is when there is no possibility of the penetration deterioration part, but when there is a possibility of the penetration deterioration part, it is rewritten to true in step S262 described later.

  Next, in step S252, the penetration improvement processing unit 706 acquires the motion vector of the pixel of interest. That is, from the motion vector stored in the detection vector memory 53, the motion vector of the pixel of interest that is the current processing target is read out.

  As shown in FIG. 25, this reading process is sequentially selected one by one from the upper left to the lower right in a predetermined range of blocks in one frame (720 × 480 pixels). Then, as shown in FIG. 26, the motion vector Vc of one selected pixel is read. In FIG. 26, the motion vector Vc of the pixel of interest (x, y) is represented as (Vcx, Vcy).

  In step S253, the penetration improvement processing unit 706 selects the reference mode. As the reference mode, three modes are prepared as shown in FIG. 27, and one of them is selected here. As shown in FIG. 27A, the first reference mode is a pixel located in the direction of the motion vector Vc with reference to the pixel of interest 281 and located at a point that divides the motion vector Vc into one-to-one. In this mode, the pixel is the first reference point 282. At this time, the second reference point 283 is a pixel at a position where the vector in the direction opposite to the motion vector Vc is divided into one-to-one. In the second reference mode, as shown in FIG. 27B, the pixel in the direction of the motion vector Vc and the position at which the motion vector Vc is internally divided into 3 to 1 is set as the first reference point 282. A pixel in the reverse direction of the motion vector Vc and having a size that divides the size into a 1: 3 ratio is set as the second reference point 283. Further, as shown in FIG. 27C, in the third reference mode, pixels in the direction of the motion vector Vc from the pixel of interest 281 that are points that internally divide the size of the motion vector Vc into one to three. Is the first reference point 282, and the pixel in the reverse direction of the motion vector Vc, the pixel at a position that internally divides the size into 3 to 1, is the second reference point 283.

  In step S254, the penetration improvement processing unit 706 calculates the coordinates of the left and right reference points. That is, as shown in FIG. 27, when the first reference mode is selected, a position having a size that is ½ of the motion vector Vc in the direction of the motion vector Vc from the target pixel 281 is the first The coordinates are calculated as the position of the reference point (left reference point) 282, and the position separated by a half of the motion vector Vc in the opposite direction to the motion vector Vc is the second reference point (right side The coordinates are calculated as the position of (reference point) 283. Similarly, as shown in FIG. 27B or FIG. 27C, when the second reference mode or the third reference mode is selected, the coordinates of the reference points 282 and 283 are calculated here, respectively.

  The reason why the surrounding reference points are set in this way is to determine whether or not a foreground (or covered area) exists in the vicinity of the target pixel.

  That is, when an example in which a penetrating deterioration portion is observed is observed, the motion vector of the pixel of interest on the 24P image, which is the cause, corresponds to the next frame on the 24P image past the foreground (intersecting on the 60P image). It is often a motion vector of the background to be performed. In other words, when there is a foreground between the pixel of interest and a point that has moved by the amount of background movement, a penetration deterioration portion often occurs.

  For example, as shown in FIG. 28, when the pixel of interest 301 on the frame t1 moves greatly to the left in the figure and corresponds to the pixel 311 on the frame t1 + 1, the motion vector connecting the pixel of interest 301 and the pixel 311 is It becomes the motion vector Vc of the pixel of interest 301. Then, when the foreground 302 on the frame t1 moves slightly to the right in the drawing to the foreground 312 on the frame t1 + 1, the vector Vl connecting the pixel 303 of the foreground 302 of the frame t1 and the pixel 313 of the foreground 312 on the frame t1 + 1. Becomes the motion vector of the pixel 303.

  If the motion vector Vc and the motion vector Vl intersect and the pixel on the frame t1 corresponding to the pixel 311 on the frame t1 + 1 is the pixel 304, the foreground 302 is positioned between the pixel 304 and the target pixel 301 in the frame t1. To do. However, the width Wl of the foreground 312 is unknown.

  Therefore, as shown in FIG. 29, for example, when the width of the foreground 302 is equal to or greater than ½ of the magnitude of the motion vector Vc of the pixel of interest 301, the intermediate position between the pixel 304 and the pixel of interest 301 (motion The foreground 302 can be detected at the position P) where the vector Vc is divided into one-to-one.

  On the other hand, when the width of the foreground 303 is equal to or greater than ¼ of the magnitude of the motion vector Vc, at a position P that internally divides the magnitude of the motion vector Vc shown in FIG. Although there is no guarantee that the foreground 302 can be detected, as shown in FIG. 30 and FIG. 31, its position and a point P (FIG. 30) that internally divides the motion vector Vc into 1 to 3, or 3 to 1 If the search is performed also at the point P to be internally divided (FIG. 31), the foreground 302 can be detected at any position. That is, the foreground is searched in the form of a binary branch search. Therefore, as shown in FIG. 27, reference points are set at three internal positions within the range of the magnitude of the motion vector Vc (first to third reference modes are provided).

  In the above, the minimum unit of the search content is set to 1/4 the size of the motion vector Vc (although the number of times of internal division to 1/2 is set to 2), the length is further divided into small parts. If this is done (when the number of times is increased), a foreground having a smaller size can be detected.

  Further, detection with a length of 1/4 has an effect as shown in FIG. That is, as shown in FIG. 32A, when the size of the foreground 302 is not less than ½ of the motion vector Vc, the foreground 302 is relatively distant from the target pixel 301 as shown in FIG. In the case where the foreground 302 and the covered area 321 exist at positions separated from each other, the foreground 302 and the covered area 321 are detected at positions P1 and P2 of ½ inner parts. However, as shown in FIG. 32B, when the foreground 302 exists at a position relatively close to the pixel of interest 301, the foreground 302 can be detected at a half internal position P1, but the covered region 321 is It cannot be detected at the internal division position P2 of 1/2 (because the covered region 321 is separated from the foreground 302 by the magnitude of the motion vector Vc). However, as shown in FIG. 32C, if the motion vector Vc is detected at a position (1/4 position) where the motion vector Vc is internally divided (1/4 position), the foreground 302 is at the position P11 and is covered. The region 321 can be detected at the position P12.

  Referring back to FIG. 24, after the coordinates of the left reference point 282 and the right reference point 283 are calculated in step S254 as described above, the penetration improvement processing unit 706 selects surrounding reference points in step S255. To do.

  As shown in FIG. 33A, it is assumed that a left reference point 282 and a right reference point 283 exist for the pixel of interest 281. The pixel of interest 281 is a background pixel, and the left (first) reference point 282 is a pixel of the foreground 371 that is a human body. On the other hand, since the right (second) reference point 283 is a pixel on the covered region 372, it is difficult to detect the motion vector. In the case of this embodiment, it is desirable to detect a zero vector (a motion vector with zero motion) for the pixels on the covered region 372, but there are many cases where the zero vector is not detected because of difficulty. However, as shown in FIG. 33B, since the 0 vector is detected as a continuous area in the image of the covered area 372 as in the foreground object, not only the reference point 283 but also the surrounding reference points 283-1 in the vicinity thereof. The pixels from 283 to 283-8 should have almost the same motion vector as the reference point 283. Therefore, if the motion vector of the reference point 283 cannot be detected, the motion vectors of the surrounding reference points 283-1 to 283-8 in the vicinity thereof can be used instead.

  Therefore, as shown in FIG. 34 (first reference mode), FIG. 35 (second reference mode), and FIG. 36 (third reference mode), the reference point 282 and the surrounding reference points 282 in the vicinity thereof are shown. -1 to 281-8, and one of the reference point 283 and its neighboring peripheral reference points 283-1 to 283-8 are selected in step S255, and their coordinates are calculated in step S256. (However, the coordinates of the reference point have already been calculated in step S254). The correspondence of surrounding reference points is as follows. That is, reference point 283 for reference point 282, reference point 283-1 for reference point 282-1, reference point 283-2 for reference point 282-2, and so on. Reference points 283-n correspond to -n, respectively. The range in which the surrounding reference points are calculated is α times (0 <α <1) the x coordinate component | Vcx | of the motion vector Vc of the point of interest 281. Similarly, the range in the y coordinate direction is β times the y coordinate component | Vcy | of the motion vector Vc (0 <β <1).

  In step S257, the penetration improvement processing unit 706 determines whether the surrounding reference point is within the image frame. The surrounding reference points here include reference points. The same applies to cases where it is not necessary to distinguish between the two. When the surrounding reference points calculated in the processes of steps S254 and S256 are outside the image frame, the surrounding reference points cannot be referred to. Therefore, in this case, the process proceeds to step S263.

  If it is determined that the surrounding reference point is not outside the frame of the image frame (within the image frame), in step S258, the penetration improvement processing unit 706 acquires a motion vector of the surrounding reference point. That is, the motion vector of the surrounding reference point of the coordinates calculated in steps S254 and S256 is acquired from the detection vector memory 53. In step S259, the penetration improvement processing unit 706 calculates a relative motion amount. Specifically, the following equations (15) and (16) are calculated.

That is, the relative quantity of motion M _L includes a motion vector Vc of the target pixel 281, the motion vector Vl, around the reference point 282-1 to 281 of the motion vector (see point 282 around the reference points 282,282-1 to 281-8 This represents a difference in magnitude from the selected motion vector V _L among the −8 motion vectors Vl1 to Vl8). Similarly, the relative motion amount M _R is calculated based on the motion vector Vc of the pixel of interest 281 and the motion vectors of the reference points 283, 283-1 to 283-8 (reference point 283 or its surrounding reference points 283-1 to 283-83). 8 represents the difference in magnitude from the selected motion vector V _R among the eight motion vectors Vr, Vr1 to Vr8).

In step S260, the penetration improvement processing unit 706 calculates a motion amount (norm). That is, the absolute value | Vc | of the motion vector Vc of the pixel of interest 281, the absolute value | V _L | of the motion vector V _L of one selected from the surrounding reference points 282, 282-1 to 282-8, and the surrounding reference point 283,283-1 to the absolute value of the selected although motion vector V _R of 283-8 | V _R | is calculated.

  In step S <b> 261, the penetration improvement processing unit 706 determines whether the following expression is satisfied.

That is, among the motion amounts calculated in step S260, the motion amount | Vc | of the motion vector Vc of the pixel of interest 281 is the motion amount | V _L |, | V _R of the motion vectors V _L and V _R of the surrounding reference points. Is greater than |. Further, it is determined according to the following equation whether the relative motions M _L and M _R calculated in step S259 are larger than a reference value 28 set in advance by experiments or the like.

  If any of the expressions (17) to (20) is satisfied, the penetration improvement unit 706 sets true in the penetration danger flag in step S262. That is, the penetrating danger flag initially set to false in step S251 is rewritten to true here.

The conditions for setting the penetration danger flag to true will be described with reference to FIGS. In FIG. 37, the foreground 302 becomes the foreground 312 having the center pixel 313 when the center pixel 303 moves from the frame t1 to the frame t1 + 1 in the right direction in the drawing at a slow speed. The motion vector Vl of the foreground 302 is expressed by a line connecting the pixel 303 and the pixel 313. The width W _{302 of the} foreground 302 is unknown. The background pixel 301 of the frame t1 moves at a high speed in the left direction in the figure, and becomes the pixel 311 in the frame t1 + 1. Therefore, the motion vector of the pixel 301 is represented by the line Vc connecting the pixel 301 and the pixel 311. The motion vector Vc crosses the motion vector Vl in the vicinity of the frame t + P.

  The covered region 321 having the pixel 322 in the frame t1 as the center is the background, and should have a motion vector substantially similar to the motion vector Vc of the nearby background pixel 301. However, the foreground 312 is located in the area of the frame t1 + 1 corresponding to the motion vector Vc from the covered area 321. Therefore, it is difficult to detect the motion vector of the covered region 321. Therefore, in the case of this embodiment, for the pixel 322 in the covered region 321, the 0 vector (motion vector with motion 0) is set as the motion vector Vr of the pixel 322.

  The penetration deterioration portion often occurs when the pixel (target pixel) 301 is located in a region 391 that is sandwiched between the foreground 302 and the covered region 321 as shown in FIG. Therefore, if it can be detected that the pixel of interest 301 belongs to the region 391, the motion vector of the pixel located there is a motion vector that tends to cause penetration degradation. Therefore, the presence of the foreground 302 and the covered area 321 is detected by determining that Expression (19) and Expression (20) are satisfied. That is, it is detected that the pixel 301 belongs to the area 391 between the foreground 302 and the covered area 321.

In addition, motion vectors that may cause penetration degradation often have a large amount of motion themselves, and are larger than the motion of the surrounding foreground and larger than the motion applied to the covered area. Many. Therefore, this condition is determined by Expression (17) and Expression (18). That is, as shown in FIG. 38, it is determined as a condition that the motion vector Vc of the target pixel is larger than the motion vectors Vl and Vr of the reference points (motion vectors V _L and V _{R of} surrounding reference points).

  As described above, when all of the equations (17) to (20) are satisfied, it is determined that the motion vector is likely to cause penetration degradation. Therefore, in this case, in step S262, the motion vector penetration danger flag is set to true.

  In step S261, when it is determined that any one of the expressions (17) to (20) is not satisfied, the process of step S262 is skipped. That is, in this case, false initially set in step S251 is set as a penetration risk flag as it is (a flag indicating that the motion vector is a motion vector that is less likely to cause penetration deterioration). Will be set).

  In step S263, the penetration improvement processing unit 706 determines whether all surrounding reference points have been selected. If there is a surrounding reference point that has not yet been selected, the process returns to step S255, a new surrounding reference point is selected, and the subsequent processing is similarly performed with respect to the surrounding reference point. If it is determined in step S263 that all surrounding reference points have been selected, in step S264, the penetration improvement processing unit 706 determines whether all reference modes have been selected. If not all reference modes have been selected, the process returns to step S253, the next reference mode is selected, and the same process is executed for the selected reference mode.

  If it is determined in step S264 that all the reference modes have been selected, in step S265, the penetration improvement processing unit 706 outputs the penetration risk flag set in step S251 or step S262.

  When the penetrating danger flag is set to true, as described with reference to FIG. 22, it is determined YES in step S206, and the processes in steps S207 to S210 are skipped. That is, the motion vector is removed from the allocation process target (allocation is prohibited). Allocation processing is executed only for motion vectors for which the penetration risk flag is set to false.

  As a result, as shown in FIG. 39, the large motion vector V1 of the pixel 1015 in the dangerous area 1014 formed between the small motion foreground 1012 and the small motion or stationary state (0 vector) covered area 1013 It is less likely to be assigned to the foreground on t + P, and the occurrence of penetration deteriorated parts is suppressed.

  FIG. 40 shows another embodiment of the penetration improvement process. The processing from step S351 to step S365 in this embodiment is basically the same processing as the processing from step S251 to step S265 in FIG. However, in the penetration improvement process of FIG. 24, it is determined whether or not each of the nine surrounding reference point motion vectors sets a penetration danger flag. However, if the penetration risk flag is set to true for at least one surrounding reference point, even if the motion vectors of other surrounding reference points are less likely to penetrate, the result is true as a penetration danger flag. Is output. Therefore, in the embodiment of FIG. 40, if true is set in the penetration risk flag for the motion vector currently being processed in step S362, the process proceeds to step S363, and all surrounding reference points have been selected. Instead of performing such a determination process, the process immediately proceeds to step S365, and a process of outputting a penetration danger flag is executed. As a result, the process can be completed more quickly. The other processing is the same as the processing in FIG. 24, and the description thereof will be omitted because it is repeated.

  Next, details of the pixel position calculation process in step S207 of FIG. 22 will be described with reference to the flowchart of FIG.

  In step S721, the pixel information calculation unit 701 obtains, from the detection memory vector 53, a motion vector that is detected by the pixel selected by the process in step S204 and has a penetration risk flag of false. If the motion vector of the selected pixel is a 0 vector, the allocation vector memory 55 stores the 0 vector as an initial value in advance, so that the subsequent steps S722 to S724 and step S208 of FIG. 22 are performed. The process from S210 to S210 is skipped, and the process proceeds to step S211.

  In step S722, the pixel information calculation unit 701 calculates the intersection of the acquired motion vector and the interpolation frame. That is, the pixel information calculation unit 701 extends the acquired motion vector in the direction of the next frame t + 1, and calculates the intersection of the extended motion vector and the interpolation frame.

  In step S723, the pixel information calculation unit 701 sets a pixel to be allocated from the intersection calculated from the motion vector and the interpolation frame. At this time, if the intersection point coincides with the pixel position on the interpolation frame, the pixel information calculation unit 701 sets the intersection point as the allocation target pixel. On the other hand, when the intersection does not coincide with the pixel position on the interpolation frame, the pixel information calculation unit 701 sets four pixels near the intersection on the interpolation frame as allocation target pixels as described above.

  In step S724, the pixel information calculation unit 701 is acquired with reference to each allocation target pixel, which is necessary for the evaluation value calculation unit 702 and the target pixel difference calculation unit 703 to obtain the evaluation value DFD and the luminance difference absolute value. The position on the original frame associated with the motion vector is calculated. Specifically, in step S724, the pixel information calculation unit 701 shifts (translates) the acquired motion vector to the set allocation target pixel, and the shifted motion vector and the intersection of the original frame The position is obtained, and the pixel position calculation process ends. Thereafter, the processing returns to step S208 in FIG.

  Next, details of the allocation vector evaluation process in step S209 of FIG. 22 will be described with reference to the flowchart of FIG.

  In step S208 in FIG. 22, the position on the original frame associated with the motion vector is obtained with reference to the allocation target pixel selected by the pixel information calculation unit 701, and the obtained position information on the original frame is The evaluation value calculation unit 702 and the target pixel difference calculation unit 703 are input.

  When the position information on the original frame is input from the pixel information calculation unit 701 from the pixel information calculation unit 701, the evaluation value calculation unit 702 obtains the frame t and the frame in order to obtain the evaluation value DFD of the motion vector in the allocation target pixel in step S741. A DFD calculation range (m × n) centered on the position on t + 1 is obtained, and in step S742, it is determined whether the calculated DFD calculation range is within the image frame. If the evaluation value calculation unit 702 determines in step S742 that the DFD calculation range is outside the image frame, the evaluation value calculation unit 702 determines that the motion vector does not become an allocation candidate vector to be allocated to the allocation target pixel, and steps S743 to S749. This process is skipped, and the allocation vector evaluation process is terminated. Thereafter, the process returns to step S210 in FIG.

  If the evaluation value calculation unit 702 determines in step S742 that the calculated DFD calculation range is within the image frame, the evaluation value calculation unit 702 uses the DFD calculation range determined to be in the image frame in step S743 to assign The evaluation value DFD of the pixel is calculated, and the obtained evaluation value DFD is output to the evaluation value determination unit 712. At this time, when the position on the original frame is equal to or less than the pixel, the evaluation value DFD of the allocation target pixel is obtained by obtaining the luminance value of the intersection on the original frame using the above-described four-point interpolation. Calculated.

  On the other hand, when the position information on the original frame is input from the pixel information calculation unit 701, the target pixel difference calculation unit 703 obtains the luminance difference absolute value dp in the allocation target pixel in step S744, and the obtained luminance The difference absolute value dp is output to the pixel difference determination unit 711. At this time, if the position on the original frame is equal to or less than the pixel, the pixel-of-interest calculation unit 703 obtains the luminance value of the intersection on the original frame using the above-described four-point interpolation. The luminance difference absolute value dp in the allocation target pixel is calculated.

  In step S745, the pixel difference determination unit 711 determines whether the luminance difference absolute value dp of the allocation target pixel from the pixel-of-interest difference calculation unit 703 is equal to or less than a predetermined threshold value, and the luminance difference absolute of the allocation target pixel. When it is determined that the value dp is greater than the predetermined threshold, it is determined that the intersection of the frame t and the frame t + 1 is likely to belong to different objects, that is, the motion vector is the reliability of the allocation target pixel. Is low and does not become an allocation candidate vector to be allocated to the allocation target pixel, the processing of steps S746 to S749 is skipped, and the allocation vector evaluation processing is terminated. Thereafter, the process returns to step S210 in FIG.

  When the luminance difference absolute value dp of the allocation target pixel is equal to or smaller than the predetermined threshold value, the evaluation value determination unit 712 refers to the DFD table of the vector selection unit 705 in step S746 and outputs the evaluation value calculation unit 702 from the evaluation value calculation unit 702. It is determined whether the evaluation value DFD of the allocation target pixel is smaller than the minimum evaluation value of the allocation target pixel stored in the DFD table (in this case, the zero vector evaluation value DFD0). When it is determined that the evaluation value DFD of the allocation target pixel is equal to or higher than the minimum evaluation value of the allocation target pixel stored in the DFD table, the motion vector is determined not to have high reliability in the allocation target pixel. The process of steps S747 to S749 is skipped, and the allocation vector evaluation process ends. Thereafter, the process returns to step S210 in FIG.

  On the other hand, when the evaluation value determination unit 712 determines in step S746 that the evaluation value DFD of the allocation target pixel from the evaluation value calculation unit 702 is smaller than the minimum evaluation value of the allocation target pixel stored in the DFD table. The motion vector is determined to have the highest reliability based on the evaluation value DFD among the motion vectors compared so far in the allocation target pixel, and the evaluation value DFD of the allocation target pixel determined to have high reliability Is output to the vector selection unit 705.

  Upon receiving the evaluation value DFD of the allocation target pixel from the evaluation value determination unit 712, the vector selection unit 705 rewrites the allocation flag of the allocation target pixel in the allocation flag memory 56 to 1 (true) in step S747, and in step S748. The minimum evaluation value corresponding to the pixel to be allocated in the DFD table is rewritten to the evaluation value DFD determined to be highly reliable by the evaluation value determination unit 712.

  The vector selection unit 705 receives the allocation target pixel selected from the pixel information calculation unit 701 and its motion vector in step S208. Therefore, in step S749, the vector selection unit 705 rewrites the motion vector allocated to the allocation target pixel in the allocation vector memory 55 with the motion vector corresponding to the evaluation value DFD determined to have high reliability, and allocates the allocation vector. End the evaluation process. Thereafter, the process returns to step S210 in FIG.

  As described above, when selecting a motion vector to be allocated to the allocation target pixel of the interpolation frame, not only the evaluation value DFD but also a position on the original frame associated with the motion vector based on the allocation target pixel is obtained. Since the absolute value of the luminance difference of the allocation target pixel is handled separately and evaluated, the most probable motion vector is selected from the allocation candidate vectors rather than using the conventional evaluation value DFD. , It can be assigned to the pixel to be assigned. Thereby, the accuracy of vector allocation is improved, discontinuity of the image generated in the subsequent image interpolation processing, etc. can be suppressed, and the image quality can be improved.

  Furthermore, when obtaining the evaluation value DFD and the luminance difference absolute value, etc., if a pixel value at a position below the pixel is required, the value is obtained by linear interpolation based on the distance to the four neighboring pixels at the position below the pixel. As a result, it is possible to perform processing with subpixel accuracy, and it is possible to obtain the luminance difference absolute value dp and the evaluation value DFD with higher accuracy than the conventional method of rounding off the subpixel components. From the allocation candidate vectors, it is possible to allocate a more probable motion vector by the pixel of interest. That is, the accuracy of vector allocation processing is improved.

  In addition, an evaluation value DFD based on a zero vector is stored in advance in the DFD table as an initial value, and while the motion vector is sequentially processed, the evaluation value DFD based on a certain motion vector becomes the minimum evaluation value at that time. In this case, since the minimum evaluation value of the DFD table and the motion vector allocated to the allocation vector memory 55 are updated as needed, time and resources can be used efficiently.

  FIG. 43 is a block diagram showing another configuration of the vector allocating unit 54. 43 assigns a difference (variation) value (hereinafter also referred to as a difference value) between a motion vector detected on the frame t and a motion vector around the motion vector. By using the evaluation value as an evaluation value and evaluating the reliability, a process of assigning to the pixels on the interpolation frame of the 60P signal to be interpolated on the assignment vector memory 55 is performed.

  That is, unlike the vector allocation unit 54 of FIG. 16, the pixel values of the frames t and t + 1 are not input to the vector allocation unit 54 of FIG.

  The pixel information calculation unit 751 selects a pixel on the frame t in the detection vector memory 53 in the raster scan order from the upper left pixel, acquires a motion vector detected by the selected pixel, And the intersection of the extended motion vector and the interpolated frame is calculated. Then, the pixel information calculation unit 751 sets an allocation target pixel of the motion vector on the interpolation frame from the intersection of the calculated motion vector and the interpolation frame, and information on the position of the motion vector and the allocation target pixel Is output to the vector selection unit 754.

  When the pixel information calculation unit 751 acquires the motion vector, the evaluation value calculation unit 752 receives the motion vector of the pixel selected by the pixel information calculation unit 751 and pixels around the pixel of the motion vector (for example, the surrounding 8 The detected motion vector is input to (pixel). The evaluation value calculation unit 752 uses the motion vector of the pixel selected by the pixel information calculation unit 751 and the motion vector detected for pixels around the pixel of the motion vector (for example, surrounding 8 pixels), And the difference value of the obtained motion vector is output to the vector evaluation unit 753.

  The vector evaluation unit 753 determines whether or not the difference value of the motion vector input from the evaluation value calculation unit 752 is smaller than the minimum evaluation value of the allocation target pixel from the vector selection unit 754, and the difference of the motion vector is determined. If it is determined that the value is smaller than the minimum evaluation value of the allocation target pixel, it is determined that the reliability of the motion vector is high, and the difference value of the motion vector is output to the vector selection unit 754.

  The vector selection unit 754 has an evaluation value memory (not shown) that holds a minimum evaluation value for each pixel on the interpolation frame. The evaluation value memory has the same configuration as the DFD table included in the vector selection unit 705 in FIG. The vector selection unit 754 rewrites the flag of the allocation flag memory 56 corresponding to the allocation target pixel to which the motion vector is allocated to 1 (true) or based on the position information of the allocation target pixel from the pixel information calculation unit 751. The minimum evaluation value in the evaluation value memory of the allocation target pixel is rewritten with the difference value of the motion vector. In addition, the vector selection unit 754 controls the pixel evaluation unit 753 to assign the pixel information calculation unit to the allocation target pixel of the allocation vector memory 55 based on the position information of the allocation target pixel from the pixel information calculation unit 751. The motion vector from 751 is allocated.

  The penetration improvement unit 755 performs the same processing as the penetration improvement unit 706 of FIG. 16 and outputs a penetration danger flag to the vector selection unit 754. Therefore, the vector selection unit 754 also prohibits the allocation of a motion vector whose value is true based on the penetration risk flag.

  Next, with reference to FIG. 44, the motion vector evaluation in the vector allocation unit 54 of FIG. 43 will be described. FIG. 44 one-dimensionally shows a frame t at time t, an interpolation frame F1 between time t and time t + 1, and a frame t + 1 at time t + 1 from the bottom. A circle on each frame indicates a pixel on the frame.

  In the example of FIG. 44, the motion vectors v1 and v2 detected in the frame t by the preceding vector detection unit 52 are extended in the frame t + 1 direction.

  The motion vector v1 from the leftmost pixel of the frame t to the vicinity of the third pixel from the left of the frame t + 1 passes through the vicinity of the second pixel G from the left on the interpolation frame F1, and the left of the frame t The motion vector v2 from the third pixel to the vicinity of the second pixel from the left in the frame t + 1 passes through the vicinity of the second pixel G from the left on the interpolation frame F1. That is, since the motion vectors v1 and v2 pass through the vicinity of the pixel G of the interpolation frame F1, each of the motion vectors v1 and v2 is set as an allocation candidate vector of the pixel G of the interpolation frame F1.

  In order to compare the motion vector v1 and the motion vector v2, which are allocation candidate vectors, the vector allocation unit 54 obtains a difference value (variation) of each motion vector on the frame t by the following equation (21), Is used as an evaluation value.

Difference value of motion vector = Σ | v _i −v | (21)

In Expression (21), v is a target motion vector, and v _i is a motion detected in the surrounding pixels (for example, surrounding 8 pixels) of the pixel in which the motion vector v is detected on the frame t. A vector (ie, an ambient vector). In equation (21), a square value may be used instead of the absolute value.

  The vector allocating unit 54 obtains the difference values of the motion vectors v1 and v2 using the equation (21), and as a result of comparing the obtained differences, the difference from the surrounding vectors is small. A motion vector having a smaller difference from the surrounding vector is determined to be a motion vector that is stably detected on the frame t, and a motion vector having a smaller difference from the surrounding vector is allocated to the pixel G. .

  As described above, by evaluating the motion vector based on the difference of the motion vectors, the vector assignment has a low image dependency and may select a motion vector that is less accurate than the evaluation value DFD. It is suppressed. Thereby, for example, it is possible to suppress problems such as penetration degradation that may occur at the time of vector assignment as shown in FIG.

  In the example of FIG. 45, the image object of the object Ob passes the time t to the time t + 1 (that is, the frame t to the frame t + 1) with the motion vector v1 from the left side to the right side in the drawing. On the other hand, the background passes from time t to time t + 1 with the motion vector v2 from the right side to the left side in the figure.

  Therefore, originally, the motion vector v1 should be selected as the allocation vector for the pixel G at the position where the image object of the object Ob passes in the interpolation frame F1, but for example, from the image object of the object Ob When the evaluation value DFD is obtained using the pixels of the block B of m × n having a larger size, and the motion vectors v1 and v2 are evaluated based on the evaluation value DFD, the background is often flat. Since the background evaluation value DFD tends to be small, the background motion vector v2 may be erroneously selected. As a result, the background of the image object of the object Ob of the interpolated frame F1 generated thereby results.

  However, if the background is flat, the motion vector detection process often detects an erroneous motion vector rather than an object having details. In such a case, the difference between the motion vector v2 and the surrounding vector becomes a large value. On the other hand, since an object having details has details, there is a high possibility that a more reliable motion vector is detected than a flat background. Therefore, in this case, the difference between the motion vector v1 and its surrounding vector is smaller than the difference between the motion vector v2 and its surrounding vector.

  From the above, by using the difference between the motion vector and the surrounding vector as an evaluation value, the penetration degradation that the background penetrates the object Ob of the object Ob is suppressed in the pixel G of the interpolation frame F1. A probable motion vector is allocated as an allocation vector.

  Next, details of the vector allocation processing of the vector allocation unit 54 of FIG. 43 will be described with reference to the flowchart of FIG. That is, this vector allocation process is another example of the vector allocation process of FIG.

  When the motion vector detected in the frame t is stored in the detection vector memory 53 by the preceding vector detection unit 52, the pixel information calculation unit 751 controls the vector selection unit 754, and in step S761, the allocation flag memory 56 allocation flags are initialized with 0 (false).

  The pixel information calculation unit 751 selects a pixel from the original frame on the detection vector memory 53 in step S762. In this case, pixels are selected one by one in the raster scan order from the upper left of the frame.

  In step S763, the penetration improvement processing unit 755 executes a penetration improvement process. The details thereof are as described above with reference to FIG. 24 or FIG. 40, whereby a motion vector that may cause penetration deterioration is detected, and the penetration danger flag of the motion vector is set to true. In step S764, the vector selection unit 754 determines whether the penetration risk flag of the selected pixel is true.

  If the penetration risk flag of the selected pixel is not true (if false), the pixel information calculation unit 751 executes pixel position calculation processing in step S765.

  Details of this pixel position calculation processing will be described with reference to the flowchart of FIG. The pixel position calculation process in FIG. 47 is different from the pixel position calculation process in FIG. 41 except for the position calculation process in the original frame in step S724. The other processes are the same as those in FIG. This is basically the same process as the pixel position calculation process.

  In step S771 of FIG. 47, the pixel information calculation unit 751 detects a motion vector detected by the pixel selected by the process of step S762 of FIG. 46 and having a penetration risk flag of false as a detection memory vector. 53. At this time, the motion vector of the pixel selected by the pixel information calculation unit 751 and the motion vector detected in pixels around the pixel of the motion vector (for example, the surrounding eight pixels) are sent to the evaluation value calculation unit 752. Entered.

  In step S772, the pixel information calculation unit 751 calculates the intersection of the acquired motion vector and the interpolation frame. That is, the pixel information calculation unit 751 extends the acquired motion vector in the direction of the next frame t + 1, and calculates the intersection of the extended motion vector and the interpolation frame.

  In step S <b> 773, the pixel information calculation unit 751 sets a pixel to be allocated from the intersection calculated from the motion vector and the interpolation frame. At this time, when the intersection point coincides with the pixel position on the interpolation frame, the pixel information calculation unit 751 sets the intersection point as the allocation target pixel. On the other hand, when the intersection does not coincide with the pixel position on the interpolation frame, the pixel information calculation unit 751 sets the four pixels near the intersection on the interpolation frame as the allocation target pixels as described above. Then, the pixel information calculation unit 751 ends the pixel position calculation process. Thereafter, the processing returns to step S766 of FIG.

  In step S766, the pixel information calculation unit 751 executes an allocation vector evaluation process. The details of the allocation vector evaluation process will be described later with reference to FIG. 48. The allocation vector evaluation process obtains a motion vector difference value, and assigns the allocation based on the obtained motion vector difference value. The reliability of the motion vector in the target pixel is determined, and the motion vector in the allocation vector memory 55 is rewritten with the motion vector determined to have high reliability as a result of these determinations.

  In step S767, the pixel information calculation unit 751 determines whether the processing of all the pixels in the original frame on the detection vector memory 53 has been completed. If it is determined in step S767 that the processing of all the pixels of the original frame on the detection vector memory 53 has not been completed, the process returns to step S762, and the next pixel of the original frame on the detection vector memory 53 is determined. Is selected, and the subsequent processing is repeated. If the pixel information calculation unit 751 determines in step S767 that the processing for all the pixels in the detection vector memory 53 has been completed, the pixel allocation calculation unit 751 ends the vector allocation processing.

  Next, the details of the allocation vector evaluation process in step S766 of FIG. 46 will be described with reference to the flowchart of FIG.

  In step S781 of FIG. 48, the evaluation value calculation unit 752 calculates the difference value of the motion vector of the pixel selected in step S762 of FIG. That is, in step S771 of FIG. 47, when the pixel information calculation unit 751 acquires a motion vector, the evaluation value calculation unit 752 includes the motion vector of the pixel selected by the pixel information calculation unit 751 and the pixel of the motion vector. The detected motion vector is input to the surrounding pixels (for example, surrounding 8 pixels).

  Therefore, the evaluation value calculation unit 752 detects the motion vector of the pixel selected by the pixel information calculation unit 751 and the pixels around the pixel of the motion vector (for example, the surrounding 8 pixels) in step S781 of FIG. The motion vector difference value shown in Expression (21) is calculated using the obtained motion vector, and the obtained motion vector difference value is output to the vector evaluation unit 753.

  In step S782, the pixel information calculation unit 751 selects the allocation target pixel calculated in step S773 of FIG. 47, and outputs the selected allocation target pixel and its motion vector to the vector selection unit 754. In step S782, the pixel information calculation unit 751 selects from the upper left pixels when there are a plurality of allocation target pixels.

  In step S783, the vector selection unit 754 determines whether the flag of the allocation flag memory 56 corresponding to the allocation target pixel from the pixel information calculation unit 751 is 1 (true), and the flag of the allocation flag memory 56 is 1 ( If it is determined that it is not true, in step S784, the flag of the allocation flag memory 56 corresponding to the allocation target pixel is rewritten to 1 (true).

  That is, when the flag of the allocation flag memory 56 is 0 (false), the allocation vector memory 55 of the allocation target pixel has not yet been allocated, and the evaluation value memory built in the vector selection unit 754 No minimum evaluation value is stored. Accordingly, when the flag of the allocation flag memory 56 is 0 (false), the vector selection unit 754 rewrites the flag of the allocation flag memory 56 to 1 (true) and does not compare with the minimum evaluation value (that is, step S785). Without executing the process of step S786). The evaluation value memory may store the maximum value stored in the memory as an initial value.

  On the other hand, if it is determined in step S783 that the flag of the allocation flag memory 56 is 1 (true), the allocation vector memory 55 of the allocation target pixel has already been allocated with other allocation vectors. A corresponding minimum evaluation value is stored in the evaluation value memory.

  Therefore, when it is determined in step S783 that the flag of the allocation flag memory 56 is 1 (true), the vector selection unit 754 reads the minimum evaluation value corresponding to the allocation target pixel from the evaluation value memory, and performs vector evaluation. Part 753.

  When the minimum evaluation value is supplied from the vector selection unit 754, the vector evaluation unit 753 determines whether the difference value of the motion vector obtained in step S781 is smaller than the minimum evaluation value of the allocation target pixel from the vector selection unit 754. If the motion vector difference value is determined to be smaller than the minimum evaluation value of the allocation target pixel, it is determined that the reliability of the motion vector is high, and the motion vector difference value is determined as a vector selection unit. 754.

  In step S786, the vector selection unit 754 rewrites the corresponding minimum evaluation value in the evaluation value memory with the motion vector difference value from the vector evaluation unit 753, and assigns it to the allocation target pixel in the allocation vector memory 55 in step S787. The motion vector being rewritten is rewritten with the motion vector corresponding to the evaluation value determined to have high reliability.

  In step S785, the difference value of the motion vector is not smaller than the minimum evaluation value of the allocation target pixel in the evaluation value memory (the minimum evaluation value of the allocation target pixel in the evaluation value memory is the difference in motion vector). If it is determined that the motion vector is smaller than the value, the reliability of the motion vector is low, and the reliability of the motion vector already written in the allocation vector memory 55 is higher. Therefore, the processes of steps S786 and S787 are skipped. The process proceeds to step S788.

  In step S788, the pixel information calculation unit 751 determines whether or not the processing for all the allocation target pixels (for example, four pixels) has been completed. If all the allocation target pixels have not been processed yet, the process returns to step S782, the next allocation target pixel is selected, and the subsequent processing is repeated. That is, even when there are four allocation target pixels, once the difference value of the motion vector is obtained, it can be used for the four allocation target pixels. In addition, it is not necessary to perform the calculation process in step S781.

  If it is determined in step S788 that the processing for all the allocation target pixels has been completed, the pixel information calculation unit 751 ends the allocation vector evaluation processing. Thereafter, the processing returns to step S767 in FIG.

  As described above, when selecting the motion vector to be allocated to the allocation target pixel of the interpolation frame, the difference (variation) of the allocated motion vector is used as the evaluation value. Therefore, compared to the case where the evaluation value DFD is used. In particular, problems such as penetration degradation as described above with reference to FIG. 45 can be suppressed, and more likely motion vectors can be selected from the allocation candidate vectors and allocated to the allocation target pixels. Thereby, the accuracy of vector allocation is improved, discontinuity of the image generated in the subsequent image interpolation processing, etc. can be suppressed, and the image quality can be improved.

  Further, by using the difference (variation) of the motion vectors to be allocated as an evaluation value, the positions of the two original frames are obtained by linear interpolation as compared with the allocation vector evaluation executed by the vector allocation unit 54 in FIG. It is not necessary to execute the process of step S724 in FIG. 41, and it is not necessary to obtain the evaluation value DFD and the luminance difference absolute value dp for each allocation target pixel. Therefore, even if the allocation vector evaluation processing of FIG. 42 and the allocation vector evaluation processing of FIG. 48 are simply compared, the allocation vector evaluation processing of FIG. 48 requires fewer steps and processing time.

  Further, the image information calculation unit 751 eliminates the need to recalculate the positions of the two original frames necessary for the image information calculation unit 701 and linear interpolation, and the vector allocation unit 54 in FIG. 43 is not required for the evaluation value calculation unit 702 and the target pixel calculation unit 703, and the vector allocation unit 54 in FIG. 43 requires a memory for storing the pixel values. Since this is eliminated, the circuit scale can be considerably reduced.

  In the above description, the motion vector of the detection vector memory 53 is detected using the gradient method, but a motion vector obtained by another method such as a block matching method may be used.

  Next, details of the configuration of the allocation compensation unit 57 will be described.

  FIG. 49 is a block diagram showing a configuration of the allocation compensation unit 57. 49 is configured by an allocation vector determination unit 801 and a vector compensation unit 802. The allocation compensation unit 57 includes a vector on the interpolation frame of the 60P signal to which no motion vector is allocated by the vector allocation unit 54. A process is performed in which the motion vectors of surrounding pixels are supplemented.

  The motion vector is assigned to the pixel of the interpolation frame on the assigned vector memory 55 by the preceding vector assigning unit 54. Further, 1 (true) is written in the allocation flag of the pixel allocation flag memory 56 to which the motion vector is allocated by the vector allocation unit 54, and the allocation of the allocation flag memory 56 of the pixel to which the motion vector has not been allocated. 0 (false) is written in the flag.

  The allocation vector determination unit 801 refers to the allocation flag in the allocation flag memory 56 and determines whether or not a motion vector is allocated to the pixel of interest by the vector allocation unit 54. Then, the allocation vector determination unit 801 selects a pixel of interest for which no motion vector has been allocated by the vector allocation unit 54, controls the vector compensation unit 802 for the selected pixel of interest, and sets the surrounding pixels of the pixel of interest. A motion vector is selected and allocated on the interpolation frame of the allocation vector memory 55.

  The vector compensation unit 802 acquires the motion vector assigned to the peripheral pixel of the target pixel from the assigned vector memory 55, and uses the frame t of the image at time t and the frame t + 1 of the image at time t + 1 that are input. The obtained motion vector evaluation value DFD is obtained and compared, and the motion vector having the highest reliability based on the evaluation value DFD among the motion vectors assigned to the peripheral pixels of the target pixel is assigned to the assigned vector memory 55. Assigned to the pixel of interest. Further, the vector compensation unit 802 rewrites the allocation flag of the target pixel to which the motion vector is allocated to 1 (true).

  FIG. 50 is a block diagram showing the configuration of the vector compensation unit 802. The vector compensation unit 802 having the configuration shown in FIG. 50 includes a compensation processing unit 811 and an evaluation value calculation unit 812.

  The compensation processing unit 811 includes a memory 821 that stores the minimum evaluation value DFD and the motion vector of the minimum evaluation value DFD as candidate vectors (hereinafter also referred to as compensation candidate vectors), and is selected by the allocation vector determination unit 801. As the initial value of the target pixel, the evaluation value DFD of the zero vector is stored in the memory 821 as the minimum evaluation value, and the zero vector is stored in the memory 821 as the compensation candidate vector. The compensation processing unit 811 refers to the allocation flag memory 56 to determine the presence or absence of motion vectors of the peripheral pixels of the target pixel, acquires the motion vector allocated to the peripheral pixels from the allocation vector memory 55, and evaluates the evaluation value. The calculation unit 812 is controlled to calculate the motion vector evaluation value DFD.

  In addition, the compensation processing unit 811 determines whether or not the evaluation value DFD calculated by the evaluation value calculation unit 812 is smaller than the minimum evaluation value stored in the memory 821, and the calculated evaluation value DFD is the minimum evaluation. When it is determined that the value is smaller than the value, the compensation candidate vector and the minimum evaluation value in the memory 821 are rewritten to the calculated evaluation value DFD and its motion vector, and finally, the peripheral pixels for which the evaluation value DFD is determined to be the smallest Are assigned to the target pixel in the allocation vector memory 55 as the motion vector of the target pixel. Furthermore, the compensation processing unit 811 rewrites the allocation flag in the allocation flag memory 56 of the target pixel to which the motion vector is allocated to 1 (true).

  When the evaluation value calculation unit 812 acquires the motion vector of the surrounding pixels from the allocation vector memory 55, the evaluation vector calculation unit 812 uses the frame t of the 24P signal image at time t and the frame t + 1 of the image at time t + 1 to be input. The motion vector evaluation value DFD from 55 is calculated, and the calculated evaluation value DFD is output to the compensation processing unit 811.

  FIG. 51 is a diagram for explaining the principle of allocation compensation processing. In the example of FIG. 51, each pixel on the interpolation frame is shown. An arrow output from a pixel represents a motion vector assigned to each pixel, and a pixel without an arrow represents a pixel to which no motion vector is assigned.

  Here, based on the evaluation value DFD among the motion vectors assigned to the peripheral pixels in the vicinity of the target pixel P with respect to the central target pixel P to which the motion vector is not assigned by the vector allocating unit 54 in the previous stage. Select and assign the one with high reliability. In the case of the example in FIG. 51, the motion vector (thick line arrow) of the pixel above the target pixel P is selected and assigned to the target pixel P. This is a process executed based on the motion correlation described below.

  FIG. 52 is a diagram for explaining the principle of motion correlation. In the example of FIG. 52, an object O1 moving with a motion v1 and an object O2 moving with a motion v2 on a certain frame are shown. The target pixel P1 belonging to the object O1 and its vicinity K1 have substantially the same movement v1 as the object O1. Further, the target pixel P2 belonging to the object O2 and its vicinity K2 have substantially the same movement v2 as the object O2.

  As described above, the motion correlation represents that the motion of the pixels belonging to a certain object is often almost the same in a space at the same time (in the same frame). Therefore, for a pixel for which a motion vector could not be allocated, using such a motion correlation in a space at the same time (within the same frame), The corresponding motion vector for that pixel can be selected. In addition, although description is abbreviate | omitted, the correlation of a time direction is also the same.

  Next, with reference to FIGS. 53 to 59, motion vector compensation processing executed based on motion correlation will be described. This process is a process of selecting a motion vector from the motion vectors of surrounding pixels and supplementing it as the motion vector of the pixel of interest. In the example of FIG. 53, white circles represent pixels on the interpolation frame, and 8 peripheral pixels are shown around the pixel of interest P for which a motion vector is to be obtained. The motion vector of the pixel of interest P is obtained with reference to the motion vectors of the surrounding eight pixels.

  In the example of FIG. 54, the upper left pixel, the upper right pixel, and the lower right pixel (pixels indicated by black circles in the figure) among the eight pixels around the target pixel P are processed in the previous stage (for example, the above-described vector allocation). The motion vector (arrow) obtained by (processing) etc. is shown. That is, the compensation candidate vectors for the pixel of interest P in this case are the motion vectors of the upper left pixel, the upper right pixel, and the lower right pixel. On the frame, since the motion vector is obtained in the raster scan order from the upper left pixel on the frame, there may be a pixel for which the motion vector is not yet obtained among the surrounding 8 pixels. Since a motion vector has not yet been obtained, it cannot be a compensation candidate vector.

  Here, as shown in the example of FIG. 55, among the peripheral 8 pixels, in addition to the pixels having the motion vectors obtained in the previous process (pixels indicated by black circles), the pixels obtained by this process are obtained. There are also pixels with motion vectors (pixels indicated by hatched circles). That is, in this process, the result of the previous stage of this process itself is also used. Therefore, the compensation candidate vector of the pixel of interest P in the example of FIG. 55 is a pixel having a motion vector of a pixel in which a motion vector already exists (a pixel indicated by a black circle) and a motion vector obtained by this processing in the previous stage. (Pixels indicated by hatched circles) of motion vectors.

  Also, as shown in the example of FIG. 56, a 0 vector (still vector) S0 with a motion amount of 0 can also be used as a compensation candidate vector. In the example of FIG. 55, the pixel having the motion vector obtained by this process and the pixel in which the motion vector already exists are represented separately, but both are the same in that they have a motion vector. In FIG. 56 to FIG. 59, the pixel having the motion vector obtained by this process is also included in the pixels (pixels indicated by black circles) in which the motion vector already exists. Therefore, in the example of FIG. 56, the compensation candidate vector of the pixel of interest P is composed of a motion vector of a pixel (a pixel indicated by a black circle) in which a motion vector already exists and a zero vector S0.

  In order to compare the reliability (probability) of the compensation candidate vectors configured as described above, an evaluation value DFD, which is a motion vector evaluation method, is obtained as shown in FIGS. 57 to 59 below. It is done. FIG. 57 shows an example in which a zero vector S0 is used as a compensation candidate vector. FIG. 58 shows an example in which the motion vector VK1 of the upper left pixels of the surrounding eight pixels is used as the compensation candidate vector. FIG. 59 shows an example in which the motion vector VK2 of the upper center pixel of the surrounding 8 pixels is used as the compensation candidate vector.

  In the example of FIG. 57, an example is shown in which the zero vector S0 is selected from the compensation candidate vectors of the pixel of interest P shown on the left side of the drawing, and the evaluation value DFD of the selected zero vector S0 is obtained. That is, the evaluation value DFD for the 0 vector S0 is selected on the basis of the pixel of interest P on the interpolation frame on the frame t and the frame t + 1 of the 24P signal sandwiching the pixel of interest P (the interpolation frame of the 60P signal). The intersection with which the zero vector S0 is associated is obtained, and the DFD calculation ranges D1-1 (frame t) and D1-2 (frame t + 1) within a predetermined range (m × n) are calculated with the intersection as the center. Using the DFD calculation ranges D1-1 and D1-2, the above equation (1) is calculated.

  In the example of FIG. 58, the motion vector VK1 of the upper left pixel of the surrounding eight pixels is selected from the compensation candidate vectors of the target pixel P shown on the left side in the drawing, and the evaluation value DFD of the selected motion vector VK1 is obtained. An example is shown. In other words, the evaluation value DFD for the motion vector VK1 of the upper left pixel of the eight neighboring pixels is selected on the basis of the target pixel P on the interpolation frame on the frame t and the frame t + 1 sandwiching the target pixel P (interpolation frame). The intersection with which the motion vector VK1 is associated is obtained, and the DFD calculation range D2-1 (frame t) and D2-2 (frame t + 1) within a predetermined range (m × n) is calculated with the intersection as the center. It is calculated | required by calculating Formula (1) mentioned above using the performed DFD calculation range D2-1 and D2-2.

  In the example of FIG. 59, the motion vector VK2 of the upper center pixel of the surrounding 8 pixels is selected as the compensation candidate vector from the compensation candidate vectors of the target pixel P shown on the left side of the drawing, and the selected motion vector VK2 is selected. An example of obtaining the evaluation value DFD of the is shown. That is, the evaluation value DFD for the motion vector VK2 of the upper center pixel of the surrounding eight pixels is based on the target pixel P on the interpolation frame on the frame t and the frame t + 1 sandwiching the target pixel P (interpolation frame). An intersection point with which the selected motion vector VK2 is associated is obtained, and a DFD calculation range D3-1 (frame t) and D3-2 (frame t + 1) within a predetermined range (m × n) around the intersection point is calculated. It is calculated | required by calculating Formula (1) mentioned above using the calculated DFD calculation range D3-1 and D3-2.

  Note that the other compensation candidate vectors shown on the left side in the figure are basically the same processing, and thus the description thereof is omitted. However, as described above, all the compensation candidate vectors for the peripheral pixels of the pixel of interest P are as described above. Evaluation values DFD are obtained, and the obtained evaluation values DFD are compared. Among them, a compensation candidate vector having the smallest evaluation value DFD is assigned to the target pixel P as shown in FIG. Selected as the most reliable and probable motion vector.

  In the case of the example in FIG. 60, it is determined that the evaluation value DFD of the motion vector VK1 of the upper left pixel of the eight neighboring pixels is the smallest among the compensation candidate vectors of the peripheral pixels of the target pixel P, and the motion vector VK1 is Is selected and assigned as a motion vector.

  As described above, since the motion vector of the pixel that could not be allocated by the vector allocation unit 54 is compensated from the motion vector of the surrounding pixels using motion correlation, the motion vector is not allocated. , The disturbance of motion can be suppressed as compared with the case where the 0 vector is assigned. Also, the motion vector of the pixel compensated in this way can be used again as a compensation candidate vector for another pixel. That is, since not only motion vectors in the spatial direction but also motion vectors in the temporal direction can be used as compensation candidate vectors, pixels that perform substantially the same motion in the object have substantially the same motion vector. A stable motion vector that is selected and has few errors can be obtained. This improves the accuracy of vector assignment.

  Next, the details of the allocation compensation processing in step S4 in FIG. 9 will be described with reference to the flowchart in FIG. The motion vector is assigned to the pixel of the interpolation frame on the assigned vector memory 55 by the preceding vector assigning unit 54. Further, 1 (true) is written in the allocation flag of the pixel allocation flag memory 56 to which the motion vector is allocated by the vector allocation unit 54, and the allocation of the allocation flag memory 56 of the pixel to which the motion vector has not been allocated. 0 (false) is written in the flag.

  In step S801, the allocation vector determination unit 801 selects a pixel of the interpolation frame in the allocation flag memory 56 as a target pixel, and the process proceeds to step S802. At this time, the allocation vector determination unit 801 selects pixels from the upper left pixel of the frame in the raster scan order.

  In step S802, the allocation vector determination unit 801 determines whether the allocation flag of the target pixel in the allocation flag memory 56 is 0 (false), and the allocation flag of the target pixel in the allocation flag memory 56 is If it is determined that the motion vector is 0 (false), it is determined that no motion vector is allocated, and in step S803, the compensation processing unit 811 is controlled to execute vector compensation processing. As a result of this vector compensation processing, the smallest motion vector of the evaluation value DFD is stored in the memory 821 as a compensation candidate vector among the motion vectors assigned to the peripheral pixels.

  In step S804, the compensation processing unit 811 allocates the compensation candidate vector in the memory 821 as the motion vector of the target pixel to the allocation vector memory 55, and in step S805, sets the target pixel allocation flag in the allocation flag memory 56 to 1. Rewrite to (true).

  On the other hand, if the allocation vector determination unit 801 determines in step S802 that the allocation flag of the target pixel in the allocation flag memory 56 is not 0 (false) (1 (true)), the allocation pixel determination unit 801 It is determined that a motion vector has already been assigned, and the processing of steps S803 to S805 is skipped.

  In step S806, the allocation vector determination unit 801 determines whether all the pixels in the interpolation frame in the allocation flag memory 56 have been processed. If it is determined that all the pixels have not been processed yet, the process returns to step S801, the next pixel of the interpolation frame in the allocation flag memory 56 is selected as the pixel of interest, and the subsequent processing is executed. Is done. If it is determined in step S806 that the processing of all the pixels of the interpolation frame in the allocation flag memory 56 has been completed, the allocation compensation processing is ended.

  As described above, even in the case of a pixel that could not be allocated in the vector allocation process, the most reliable level based on the evaluation value DFD is selected from the motion vectors around the pixel by using the fact that there is a motion correlation. There is a certain motion vector. As a result, the accuracy of vector assignment is improved as compared with the case where a vector is not assigned and a 0 vector is assigned, and discontinuity of an image generated in subsequent image interpolation processing can be suppressed.

  Further, the allocation flag of the pixel to which the motion vector is allocated by the allocation compensation process described above is rewritten to 1 (true), and the motion vector allocated by the allocation compensation process is also used as a compensation candidate vector for the next pixel. Therefore, almost the same motion vector is selected for pixels that perform substantially the same motion in the object, and a stable motion vector with few errors can be obtained. As a result, it is possible to improve the quality by suppressing block noise, powder noise and the like of the image generated in the subsequent stage.

  In addition, for a pixel of interest for which a motion vector is to be calculated, an evaluation value DFD of 0 vector is calculated in advance and stored in a memory as a minimum evaluation value, whereby all compensation candidate vector evaluation values DFD are calculated at one time. Time and resources can be used more efficiently than when the smallest evaluation value DFD is selected.

  In the above description, the vector compensation processing is performed on the pixels that are not assigned by the vector assigning unit 54, but there are some kinds of pixels such as pixels that are not detected by the vector detecting unit 52 (the zero vector is detected). You may make it perform a vector compensation process with respect to the pixel in which the motion vector was not calculated | required in the process. Furthermore, vector compensation processing may be performed on pixels for which the detected motion vector or the assigned motion vector is uncertain (low reliability).

  In this embodiment, the evaluation value DFD that is the sum of absolute differences is used as the evaluation value when selecting a motion vector. However, the evaluation value DFD is not limited to the evaluation value DFD, and the reliability of the motion vector is evaluated. If it is a thing, you may make it use another thing.

  In the present embodiment, the block for performing each process has been described as being configured by, for example, 8 pixels × 8 pixels, 9 pixels × 9 pixels, and the like. However, these are examples, and each process is performed. The number of pixels constituting the block to be performed is not limited to the number of pixels.

  Furthermore, in the present embodiment, the signal conversion from the 24P signal to the 60P signal has been described as an example. However, the present invention can be applied to, for example, interlaced signals and other frame rates as frame frequency conversion of moving images. It can also be applied to conversion.

  FIG. 62 is a block diagram showing an example of the configuration of a personal computer that executes the above-described series of processing by a program. A CPU (Central Processing Unit) 921 executes various processes according to a program stored in a ROM (Read Only Memory) 922 or a storage unit 928. A RAM (Random Access Memory) 923 appropriately stores programs executed by the CPU 921 and data. The CPU 921, ROM 922, and RAM 923 are connected to each other via a bus 924.

  An input / output interface 925 is also connected to the CPU 921 via the bus 924. The input / output interface 925 is connected to an input unit 926 including a keyboard, a mouse, and a microphone, and an output unit 927 including a display and a speaker. The CPU 921 executes various processes in response to commands input from the input unit 926. Then, the CPU 921 outputs the processing result to the output unit 927.

  The storage unit 928 connected to the input / output interface 925 includes, for example, a hard disk, and stores programs executed by the CPU 921 and various data. The communication unit 929 communicates with an external device via a network such as the Internet or a local area network.

  Further, a program may be acquired via the communication unit 929 and stored in the storage unit 928.

  A drive 930 connected to the input / output interface 925 drives a removable medium 931 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the program or data recorded therein. Get etc. The acquired program and data are transferred to and stored in the storage unit 928 as necessary.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program storage medium in a general-purpose personal computer or the like.

  As shown in FIG. 62, a program storage medium for storing a program that is installed in a computer and can be executed by the computer is a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only). Memory), DVD (including Digital Versatile Disc), magneto-optical disk (including MD (Mini-Disc)), or removable media 931, which is a package media made of semiconductor memory, or the program is temporary or permanent ROM 922 stored in the hard disk, a hard disk constituting the storage unit 928, and the like. The program is stored in the program storage medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 929 which is an interface such as a router or a modem as necessary. Done.

  In the present specification, the step of describing the program stored in the program storage medium is not limited to the processing performed in chronological order according to the described order, but is not necessarily performed in chronological order. Or the process performed separately is also included.

  Further, in this specification, the system represents the entire apparatus constituted by a plurality of apparatuses.

  The embodiment of the present invention is not limited to the above-described embodiment, and various modifications can be made without departing from the gist of the present invention.

  In the present specification, the steps shown in the flowcharts include not only processes performed in time series according to the described order, but also processes executed in parallel or individually even if not necessarily performed in time series. Is included.

It is a figure of the display screen of the conventional foreground image. It is a figure of the display screen of the conventional covered area | region. It is a figure of the display screen explaining degradation of an image. It is a block diagram which shows the structure of the image signal recording / reproducing apparatus as embodiment of this invention. It is a block diagram which shows the structure of the signal processing apparatus as embodiment of this invention. It is a figure explaining the principle of the frame frequency conversion process of embodiment of this invention. It is a figure explaining the specific process of embodiment of this invention. It is a figure explaining the evaluation value of the motion vector used in a signal processor. It is a flowchart explaining the frame frequency conversion process of a signal processing apparatus. It is a block diagram which shows the structure of an image interpolation part. It is a flowchart explaining an image interpolation process. It is a block diagram which shows the structure of a vector detection part. It is a figure explaining the gradient method used in a vector detection part. It is a figure explaining the iterative gradient method using an initial vector. It is a flowchart explaining a motion vector detection process. It is a block diagram which shows the structure of a vector allocation part. It is a figure explaining the concept of 4-point interpolation processing. It is a figure explaining the outline | summary of a vector allocation process. It is a figure explaining the example of the neighborhood pixel of the intersection of a motion vector and an interpolation frame. It is a figure explaining the evaluation method of the motion vector of an allocation candidate. It is a figure explaining the example of 4-point interpolation in vector allocation. It is a flowchart explaining a vector allocation process. It is a figure explaining the relationship between the motion vector of a foreground and the motion vector of a background pixel. It is a flowchart explaining a penetration improvement process. It is a figure explaining selection of a pixel. It is a figure explaining the motion vector of a pixel. It is a figure explaining reference mode. It is a figure explaining the relationship between a foreground and a background. It is a figure explaining the detection in case internal division ratio is 1: 1. It is a figure explaining the detection in case internal division ratio is 1: 3. It is a figure explaining the detection in case internal division ratio is 3: 1. It is a figure explaining the detection in case an internal division ratio is 1: 3. It is a figure explaining a surrounding reference point. It is a figure explaining the surrounding reference point in case an internal division ratio is 1: 1. It is a figure explaining the reference point in case an internal ratio is 3: 1. It is a figure explaining the reference point in case an internal ratio is 1: 3. It is a figure explaining the relationship between penetration degradation and a field. It is a figure explaining the relationship of a motion vector. It is a figure explaining a dangerous area. It is a flowchart explaining other embodiment of a penetration improvement process. It is a flowchart explaining a pixel position calculation process. It is a flowchart explaining the allocation vector evaluation process. It is a block diagram which shows the structure of other embodiment of a vector allocation part. It is a figure explaining evaluation of the motion vector of the vector allocation part of FIG. It is a figure explaining penetration deterioration. It is a flowchart explaining the vector allocation process of the vector allocation part of FIG. 47 is a flowchart for describing pixel position calculation processing in step S765 of FIG. 46. It is a flowchart explaining the allocation vector evaluation process of step S766 of FIG. It is a block diagram which shows the structure of an allocation compensation part. It is a block diagram which shows the structure of a vector compensation part. It is a figure explaining the principle of an allocation compensation process. It is a figure explaining the principle of motion correlation. It is a figure explaining the structure of the surrounding pixel of a focused pixel. It is a figure explaining the example of the compensation candidate vector of the motion vector of the focused pixel. It is a figure explaining the example of the compensation candidate vector of the motion vector of the focused pixel. It is a figure explaining the example of the compensation candidate vector of the motion vector of the focused pixel. It is a figure explaining the example which evaluates a compensation candidate vector. It is a figure explaining the example which evaluates a compensation candidate vector. It is a figure explaining the example which evaluates a compensation candidate vector. It is a figure explaining the example which selects a compensation candidate vector as a motion vector of a focused pixel. It is a flowchart explaining an allocation compensation process. It is a block diagram which shows the structure of a personal computer.

Explanation of symbols

  3 signal processing device, 51 frame memory, 52 vector detection unit, 53 detection vector memory, 55 vector allocation unit, 56 allocation vector memory, 57 allocation flag memory, 58 allocation compensation unit, 59 image interpolation unit, 101 initial vector selection unit, 103 Iterative Gradient Method Calculation Unit, 104 Vector Evaluation Unit, 105 Shift Initial Vector Allocation Unit, 106 Evaluation Value Memory, 107 Shift Initial Vector Memory 701 Pixel Information Calculation Unit, 702 Evaluation Value Calculation Unit, 703 Pixel Difference Calculation Unit, 704 Vector Evaluation unit, 705 vector selection unit, 706 penetration improvement processing unit, 711 pixel difference determination unit, 712 evaluation value determination unit

Claims (11)

In an image processing apparatus that assigns a motion vector of a first frame to a second frame, an acquisition unit that acquires a motion vector of a point of interest of the first frame;
Detecting means for detecting the size of the motion vector of the point of interest and the region on the first frame to which the motion vector of the point of interest belongs;
An image processing apparatus comprising: prohibiting means for prohibiting allocation of the motion vector of the point of interest to the second frame based on the detected size and area. The image processing apparatus according to claim 1, wherein the detection unit detects that a region to which the motion vector of the point of interest belongs is a region in which penetration degradation may occur. The detection means includes a third region sandwiched between a first region in which a motion vector of the point of interest belongs is a foreground and a second region in which no motion is obtained because the region is hidden by the foreground The image processing device according to claim 1, wherein the image processing device is detected. The acquisition means acquires a first motion vector that is a motion vector of the point of interest, and a second motion vector of a first reference point that is located in the direction of the first motion vector from the point of interest. , Obtaining a third motion vector of a second reference point located in a direction opposite to the direction of the first motion vector from the point of interest,
The detection means is for comparing a difference between the magnitudes of the first motion vector and the second motion vector, and a reference value for a difference between the magnitudes of the first motion vector and the third motion vector. The image processing apparatus according to claim 3, wherein the third region is detected based on the size of the first motion vector and the magnitude of the first motion vector with respect to the second motion vector and the third motion vector is detected. The first reference point is a pixel in the direction of the first motion vector from the point of interest in the first frame, and a position that internally divides the magnitude of the first motion vector into n to m , And the second reference point is a pixel in a direction opposite to the direction of the first motion vector from the point of interest of the first frame, The image processing apparatus according to claim 4, wherein the image processing apparatus is any one of N pixels near a position that internally divides the magnitude of the first motion vector into m to n. 6. The N pixels are pixels in a range of ± α times or ± β times the magnitude of the x-coordinate component and y-coordinate component of the first motion vector with reference to the internally divided position. An image processing apparatus according to 1. The image processing apparatus according to claim 5, wherein the internal ratio n to m is 1: 1, 1: 3, or 3: 1. Computing means for computing an evaluation value of the motion vector;
The image processing apparatus according to claim 1, further comprising: a selection unit that selects a motion vector to be allocated to the second frame based on the evaluation value. In an image processing method for assigning a motion vector of a first frame to a second frame,
Obtaining a motion vector of a point of interest in the first frame;
Detecting the size of the motion vector of the point of interest, as well as the region on the first frame to which the motion vector of the point of interest belongs,
An image processing method comprising a step of prohibiting allocation of the motion vector of the point of interest to the second frame based on the detected size and area. In a program for assigning a motion vector of a first frame to a second frame,
Obtaining a motion vector of a point of interest in the first frame;
Detecting the size of the motion vector of the point of interest, as well as the region on the first frame to which the motion vector of the point of interest belongs,
A computer-executable program comprising a step of prohibiting allocation of the motion vector of the point of interest to the second frame based on the detected size and area.   A recording medium on which the program according to claim 10 is recorded.

Images