JP4512983B2 - Image processing apparatus and method, recording medium, and program - Google Patents

Description

  The present invention relates to an image processing apparatus and method, a recording medium, and a program, and more particularly to an image processing apparatus and method, a recording medium, and a program that can improve the quality of an image generated based on a motion vector. About.

  Conventionally, in an image processing apparatus that converts a frame frequency of an image, a motion vector is detected by using, for example, a gradient method or a block matching method described in Patent Document 1, and the detected motion vector is Assigned to the generated interpolation frame. However, in these motion vector detection methods, the motion vector is not always obtained, and the motion vector of the target pixel cannot be obtained, or the obtained motion vector is unstable. There was a case where there was a problem in the reliability of the vector.

  Further, even when the detected motion vector is allocated to the generated interpolation frame, the motion vector is not allocated to all the pixels on the interpolation frame.

  In such a case, the image processing apparatus has coped with the motion vector being left unobtained or by making the motion vector a zero vector (that is, a stationary state).

JP 60-158786 A

  However, a motion vector that has not been obtained or a motion vector that has been set to a zero vector is not necessarily a motion suitable for the object to which the pixel of interest belongs, and an image generated using these motion vectors For example, there has been a problem of causing visual deterioration such as discontinuity.

  In addition, the motion vector obtained by the conventional motion vector detection method as described above or the motion vector assigned to the pixel of the interpolation frame has no motion correlation even though it is motion in the same object. In many cases, it is obtained as a motion vector (not similar to the motion vectors of surrounding pixels), and there is a problem that this motion vector is likely to be obtained in error, that is, it is often not very accurate.

  The present invention has been made in view of such a situation, and makes it possible to allocate a probable motion vector to a pixel to which a motion vector has not been allocated in the previous processing with a smaller amount of computation. It is.

The image processing apparatus according to the present invention includes, for each target block that is a target block on the third frame, which includes a target pixel that is a plurality of target pixels adjacent to each other on the third frame. In addition, the pixels around the target block are assigned to the target pixel or the adjacent pixels in the order of the priority assigned to the adjacent pixel and the target pixel that are arranged adjacent to the target block. For the motion vector, the pixel value of the pixel of the first frame belonging to the first region centered on the start point of the motion vector in the first frame and the second of the motion vector centered on the end point of the second frame of the motion vector. Evaluation value calculation means for calculating an evaluation value indicating a correlation with the pixel value of the pixel of the second frame belonging to the region, and when the evaluation value is calculated for a predetermined number of motion vectors , Of the target pixel of the target block, the motion in the target pixel motion vector is not allocated, to the allocation means for allocating a motion vector evaluation value is computed indicating that correlation is strongest, attempts calculating an evaluation value Judgment means for judging whether or not the magnitude and direction of the vector are the same as the magnitude and orientation of the motion vector that has already been computed, and the evaluation value computing means tries to compute the evaluation value. When it is determined that the magnitude and direction of the motion vector to be evaluated are the same as the magnitude and direction of the motion vector that has already been calculated, the calculation of the evaluation value for the motion vector is skipped. To do.
The evaluation value calculation means can calculate the evaluation value for the motion vector assigned by the assignment means in the processing for the other block of interest.

  When the evaluation value calculating means moves the point where the motion vector assigned to the target pixel or the adjacent pixel and the third frame intersect in order of priority to a point in the target block, the motion vector at that position A pixel value of a pixel belonging to the first region centered on the start point of the first frame, and a pixel value of a pixel belonging to the second region centered on the end point of the second frame of the motion vector at that position An evaluation value indicating the correlation can be calculated.

  The evaluation value calculation means is a priority order in which the target pixel has priority over the adjacent pixel, and the priority order of the target pixel corresponding to the raster scan order of the target pixel and the priority of the adjacent pixel corresponding to the raster scan order of the adjacent pixel. Evaluation values can be calculated for motion vectors assigned to the target pixel or adjacent pixels in order of rank.

  The evaluation value calculation means can calculate an evaluation value that is a sum of absolute differences between a pixel value of a pixel belonging to the first area and a pixel value of a pixel belonging to the second area.

An image processing method , a recording medium program, or a program according to the present invention includes a target pixel on a third frame, which includes a target pixel that is a plurality of target pixels adjacent to each other on the third frame. For each target block that is a target block, the target pixel in the order of the priority assigned to the adjacent pixel and the target pixel that are pixels adjacent to the target block and that are adjacent to the target block Alternatively, for a motion vector assigned to an adjacent pixel, the pixel value of the pixel of the first frame belonging to the first region centered on the start point of the first frame of the motion vector and the second frame of the motion vector An evaluation value calculating step for calculating an evaluation value indicating a correlation with the pixel value of the pixel of the second frame belonging to the second region centered on the end point in When the evaluation value is calculated for the motion vector of, the allocation that allocates the motion vector for which the evaluation value indicating the strongest correlation is calculated to the target pixel to which the motion vector is not allocated among the target pixels of the target block And a determination step for determining whether or not the magnitude and direction of the motion vector for which the evaluation value is to be calculated are the same as the magnitude and direction of the motion vector that has already been calculated. In the value calculation step, when it is determined that the magnitude and direction of the motion vector for which the evaluation value is to be calculated are the same as the magnitude and direction of the motion vector for which the evaluation value has already been calculated, The calculation of the evaluation value is skipped .

In the image processing apparatus and method, the recording medium, and the program of the present invention, the attention on the third frame, which is composed of a plurality of attention pixels that are adjacent to each other on the third frame. For each target block that is a target block, the pixels in the vicinity of the target block that are adjacent to the target block and in the order of priority assigned to the target pixel For a motion vector assigned to a pixel or an adjacent pixel, the pixel value of the pixel of the first frame belonging to the first region centered on the start point of the first frame of the motion vector, and the second of the motion vector An evaluation value indicating a correlation with the pixel value of the pixel of the second frame belonging to the second region centering on the end point in the frame is calculated, and a predetermined number of motion vectors are calculated. If the evaluation value is computed for, among the target pixel of the target block, the target pixel motion vector is not allocated, allocated motion vector evaluation value is computed indicating that correlation is strongest. Further, it is determined whether or not the magnitude and direction of the motion vector for which the evaluation value is to be calculated are the same as the magnitude and direction of the motion vector for which the evaluation value has already been calculated, and the evaluation value is to be calculated. When it is determined that the magnitude and direction of the motion vector are the same as the magnitude and direction of the motion vector for which the evaluation value has already been calculated, the calculation of the evaluation value for the motion vector is skipped.

  According to the present invention, a motion vector can be assigned to a pixel to which no motion vector has been assigned in the previous processing. Further, according to the present invention, a motion vector can be allocated to a pixel to which a motion vector has not been allocated in the previous process with a smaller amount of calculation.

  Embodiments of the present invention will be described below. Correspondences between constituent elements described in the claims and specific examples in the embodiments of the present invention are exemplified as follows. This description is to confirm that specific examples supporting the invention described in the claims are described in the embodiments of the invention. Accordingly, although there are specific examples that are described in the embodiment of the invention but are not described here as corresponding to the configuration requirements, the specific examples are not included in the configuration. It does not mean that it does not correspond to a requirement. On the contrary, even if a specific example is described here as corresponding to a configuration requirement, this means that the specific example does not correspond to a configuration requirement other than the configuration requirement. not.

  Further, this description does not mean that all the inventions corresponding to the specific examples described in the embodiments of the invention are described in the claims. In other words, this description is an invention corresponding to the specific example described in the embodiment of the invention, and the existence of an invention not described in the claims of this application, that is, in the future, a divisional application will be made. Nor does it deny the existence of an invention added by amendment.

According to the present invention, an image processing apparatus is provided. This image processing apparatus is a motion vector detected based on a first frame (for example, frame t) and a second frame (for example, frame t + 1), and is a second frame starting from the first frame. A pixel on the third frame in the vicinity of a point where a motion vector whose end point is and a third frame (for example, an interpolation frame) arranged between the first frame and the second frame intersect An image processing apparatus for generating a pixel value of a pixel of a third frame based on the allocated motion vector, wherein a plurality of adjacent pixels on the third frame are adjacent to each other. For each target block (for example, the compensation processing target block 1021 in FIG. 48) that is a target block on the third frame, the target pixel is a target pixel that is a target pixel. The pixel of interest in the order of the priority (for example, the priority shown in FIG. 48) assigned to the adjacent pixel and the pixel of interest that are pixels around the lock and are arranged adjacent to the block of interest. Alternatively, for a motion vector assigned to an adjacent pixel, the pixel value of the pixel of the first frame belonging to the first region centered on the start point of the first frame of the motion vector and the second frame of the motion vector Evaluation value calculating means (for example, the evaluation value of FIG. 47) for calculating an evaluation value (for example, evaluation value DFD) indicating the correlation with the pixel value of the pixel of the second frame belonging to the second region centered on the end point in FIG. When the evaluation value is calculated for the calculation unit 1002) and a predetermined number of motion vectors, attention is not assigned to a motion vector among the target pixels of the target block. Based, allocation means for allocating a motion vector evaluation value is computed indicating that correlation is strongest (e.g., vector compensating unit 1001 in FIG. 47) and, the magnitude and direction of the motion vector to be computed evaluation value Judgment means (for example, the same vector judgment unit 1004 in FIG. 47) for judging whether or not the evaluation value is the same as the magnitude and direction of the motion vector that has already been computed. If it is determined that the magnitude and direction of the motion vector for which the value is to be calculated are the same as the magnitude and direction of the motion vector that has already been calculated, the calculation of the evaluation value for that motion vector is skipped. To do.

  The evaluation value calculating means determines a point in the target block (for example, the priority level 1 shown in FIG. 48) at the point where the motion vector assigned to the target pixel or the adjacent pixel and the third frame intersect in order of priority. The position of the pixel of interest marked with (), the pixel value of the pixel belonging to the first area centered on the start point of the first frame of the motion vector at that position, and the motion vector at that position An evaluation value indicating a correlation with a pixel value of a pixel belonging to the second region centered on the end point of the second frame can be calculated.

  The evaluation value calculation means is a priority order in which the target pixel has priority over the adjacent pixels, and is a priority order of the target pixel corresponding to the raster scan order of the target pixel (for example, any one of 1 to 4 in FIG. 48). Priority) and the priority of adjacent pixels corresponding to the raster scan order of the adjacent pixels (for example, the priority of any one of 5 to 16 in FIG. 48). An evaluation value can be calculated for.

  The evaluation value calculation means can calculate an evaluation value that is a sum of absolute differences between a pixel value of a pixel belonging to the first area and a pixel value of a pixel belonging to the second area.

According to the present invention, an image processing method or program is provided. This image processing method or program is a motion vector detected based on a first frame (for example, frame t) and a second frame (for example, frame t + 1), and the second frame starting from the first frame. On the third frame in the vicinity of the point where the motion vector whose end point is the third frame and the third frame (for example, the interpolation frame) arranged between the first frame and the second frame intersect An image processing method for assigning a motion vector to a pixel of the image and generating a pixel value of a pixel of the third frame based on the assigned motion vector, and adjacent to each other on the third frame A target block which is a target block on the third frame and includes a target pixel which is a plurality of target pixels (for example, the compensation processing target block 10 in FIG. 48). 1) For each neighboring pixel of the block of interest and adjacent pixels that are arranged adjacent to the block of interest and the priority assigned to the pixel of interest (for example, the priority shown in FIG. 48) ), The pixel values of the pixels of the first frame belonging to the first region centering on the start point of the first frame of the motion vector, and the motion of the motion vector assigned to the target pixel or the adjacent pixel An evaluation value calculation step (for example, an evaluation value (for example, evaluation value DFD)) that calculates a correlation with the pixel value of the pixel of the second frame belonging to the second region centered on the end point of the vector in the second frame , And the evaluation value is calculated for a predetermined number of motion vectors, the motion vector in the target pixel of the target block is The pixel of interest that are not Attach, motion vectors and allocating step of allocating a motion vector evaluation value is computed indicating that correlation is strongest (for example, step S1021 in FIG. 55), attempts to calculating an evaluation value Including a determination step (for example, the process of step S1011 in FIG. 54) for determining whether or not the evaluation value is the same as the magnitude and direction of the motion vector that has already been calculated. In the calculation step, when it is determined that the magnitude and direction of the motion vector for which the evaluation value is to be calculated are the same as the magnitude and direction of the motion vector that has already been calculated, evaluation of the motion vector is performed. Skip value operations.

  This program can be recorded on a recording medium (for example, the magnetic disk 31 in FIG. 1).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a configuration example of a signal processing apparatus 1 to which the present invention is applied. The signal processing device 1 is composed of, for example, a personal computer. In FIG. 1, a CPU (Central Processing Unit) 11 executes various processes according to a program stored in a ROM (Read Only Memory) 12 or a storage unit 18. A RAM (Random Access Memory) 13 appropriately stores programs executed by the CPU 11 and data. The CPU 11, ROM 12, and RAM 13 are connected to each other by a bus 14.

  An input / output interface 15 is also connected to the CPU 11 via the bus 14. The input / output interface 15 is connected to an input unit 16 including a keyboard, a mouse, a microphone, and the like, and an output unit 17 including a display and a speaker. The CPU 11 executes various processes in response to commands input from the input unit 16. Then, the CPU 11 outputs an image, sound, or the like obtained as a result of the processing to the output unit 17.

  The storage unit 18 connected to the input / output interface 15 is composed of, for example, a hard disk, and stores programs executed by the CPU 11 and various data. The communication unit 19 communicates with an external device via the Internet or other networks. A program may be acquired via the communication unit 19 and stored in the storage unit 18.

  The drive 20 connected to the input / output interface 15 drives the magnetic disk 31, the optical disk 32, the magneto-optical disk 33, or the semiconductor memory 34 when they are mounted, and programs and data recorded there. Get etc. The acquired program and data are transferred to and stored in the storage unit 18 as necessary.

  Note that the signal processing device 1 can be, for example, a television receiver, an optical disc player, or a signal processing unit thereof.

  FIG. 2 is a block diagram showing the signal processing apparatus 1.

  It does not matter whether each function of the signal processing device 1 is realized by hardware or software. That is, each block diagram in this specification may be considered as a hardware block diagram or a software functional block diagram.

  In the signal processing device 1 having the configuration shown in FIG. 2, for example, an image of a progressive image signal (hereinafter referred to as a 24P signal) with a frame frequency of 24 Hz is input, and the input image (input image) has a frame frequency of 60 Hz. It is converted into an image of a progressive image signal (hereinafter referred to as 60P signal) and output. That is, FIG. 2 is a diagram illustrating a configuration of a signal processing device that is an image processing device.

  The input image of the 24P signal input to the signal processing device 1 is supplied to the frame memory 51, the vector detection unit 52, the vector allocation unit 54, the allocation compensation unit 57, and the image interpolation unit 58. The frame memory 51 stores input images in units of frames. The frame memory 51 stores a frame at time t one before the input image at time t + 1. The frame at time t stored in the frame memory 51 is supplied to the vector detection unit 52, the vector allocation unit 54, the allocation compensation unit 57, and the image interpolation unit 58. Hereinafter, the frame at time t on the frame memory 51 is referred to as frame t, and the frame of the input image at time t + 1 is referred to as frame t + 1.

  The vector detection unit 52 detects a motion vector between the target block of the frame t on the frame memory 51 and the target block of the frame t + 1 of the input image, and stores the detected motion vector in the detection vector memory 53. As a method for detecting the motion vector between the two frames, a gradient method or a block matching method is used. Details of the configuration of the vector detection unit 52 will be described later. The detection vector memory 53 stores the motion vector detected by the vector detection unit 52 in the frame t.

  The vector allocation unit 54 distinguishes the motion vector obtained on the frame t of the 24P signal from the frame of the 60P signal to be interpolated in the allocation vector memory 55 (hereinafter, the frame of the 60P signal is distinguished from the frame of the 24P signal). , Which is also referred to as an interpolation frame), and the allocation flag of the allocation flag memory 56 of the pixel to which the motion vector is allocated is rewritten to 1 (True). Details of the configuration of the vector allocation unit 54 will be described later.

  The allocation vector memory 55 stores the motion vector allocated by the vector allocation unit 54 in association with each pixel of the interpolation frame. The allocation flag memory 56 stores an allocation flag indicating the presence / absence of a motion vector to be allocated for each pixel of the interpolation frame. For example, an assignment flag of True (1) indicates that a motion vector is assigned to the corresponding pixel, and an assignment flag of False (0) indicates that a motion vector is not assigned to the corresponding pixel. Show.

  The allocation compensation unit 57 refers to the allocation flag of the allocation flag memory 56, and supplements the motion vector of the peripheral pixel of the target pixel with respect to the target pixel for which the motion vector has not been allocated by the vector allocation unit 54. Allocate on 55 interpolation frames. At this time, the allocation compensator 57 rewrites the allocation flag of the target pixel to which the motion vector is allocated to 1 (True). Details of the configuration of the allocation compensator 57 will be described later.

  The image interpolation unit 58 interpolates and generates the pixel value of the interpolation frame using the motion vector allocated to the interpolation frame in the allocation vector memory 55 and the pixel values of the frame t and the next frame t + 1. Then, the image interpolation unit 58 outputs the generated interpolated frame, and then outputs the frame t + 1 as necessary, thereby outputting the 60P signal image to a subsequent stage (not shown). In the following, the pixel value is also referred to as a luminance value as appropriate.

  FIG. 3 is a diagram for explaining the principle of processing in the signal processing apparatus 1 according to the present invention. In the example of FIG. 3, a dotted line represents a frame of a 24P signal that is input to the signal processing device 1 at times t, t + 1, and t + 2, and a solid line is represented by the signal processing device 1 from the input 24P signal. , The 60P signal interpolation frames at the generated times t, t + 0.4, t + 0.8, t + 1.2, t + 1.6, and t + 2.

  Generally, in order to convert a 24P signal into a 60P signal, 5/2 times as many frames are required. That is, five 60P signal images must be generated from two 24P signal images. At this time, the generated interpolated frame of the 60P signal has a time phase of 0.0, 0.4, 0.8, 1.2, and 1 on the 24P signal in order to equalize the frame interval. 6 is arranged at a position. Among these, 4 frames (frames of t + 0.4, t + 0.8, t + 1.2, and t + 1.6) except for one frame at time t with a time phase of 0.0 do not exist on the 24P signal. It is an image. Therefore, when the image of the 24P signal is input, the signal processing device 1 generates four interpolation frames from the two frames at the time t and the time t + 1 of the 24P signal. Therefore, the signal processing apparatus 1 outputs a 60P signal image composed of five frames at times t, t + 0.4, t + 0.8, t + 1.2, and t + 1.6.

  As described above, the signal processing apparatus 1 executes the process of converting the frame frequency from the 24P signal image to the 60P signal image.

  In principle, as described above, from the two frames at time t and time t + 1 of the 24P signal, five frames at times t, t + 0.4, t + 0.8, t + 1.2, and t + 1.6 are obtained. A frame of 60P signal is newly generated. Actually, in the example of FIG. 3, 60P of t, t + 0.4, and t + 0.8 is based on two frames at time t and time t + 1 of the 24P signal. A frame of the signal is generated, and a frame of the 60P signal of t + 1.2, t + 1.6, and t + 2 is generated based on the two frames at the times t + 1 and t + 2 of the 24P signal.

  FIG. 4 is a diagram for more specifically explaining the processing of the present invention. In the example of FIG. 4, a thick arrow represents a transition to each state, and an arrow T represents a time passage direction in the states 81 to 85. The states 81 to 85 are the frame t at the time t of the 24P signal, the frame t + 1 at the time t + 1 next to the time t, or the frame t and the frame t + 1 at the time of input / output to each unit constituting the signal processing device 1. The state of the interpolation frame F of the 60P signal produced | generated during is shown notionally. That is, actually, for example, a frame in which a motion vector is detected as shown in the state 82 is not input, and the frame and the motion vector are input separately.

  The state 81 represents the state of the frame t and the frame t + 1 of the 24P signal input to the vector detection unit 52. A black dot on the frame t in the state 81 represents a pixel on the frame t. The vector detection unit 52 detects the position where the pixel on the frame t in the state 81 moves in the frame t + 1 of the next time, and the movement is indicated as shown on the frame t in the state 82. It outputs as a motion vector corresponding to each pixel. As a method for detecting the motion vector between the two frames, a block matching method or a gradient method is used. At this time, when a plurality of motion vectors are detected in the pixel, the vector detection unit 52 obtains an evaluation value to be described later for each motion vector, and selects a motion vector based on the evaluation value.

  A state 82 represents the state of the frame t and the frame t + 1 input to the vector allocation unit 54. In the state 82, the arrow of each pixel of the frame t represents the motion vector detected by the vector detection unit 52.

  The vector allocating unit 54 extends the motion vector detected for each pixel of the frame t in the state 82 to the next frame t + 1 and sets it to a preset time phase (for example, t + 0.4 in FIG. 3). A position on a certain interpolation frame F is obtained. This is because, assuming that there is a constant motion between the frame t and the frame t + 1, the point where the motion vector passes through the interpolation frame F is the pixel position in that frame. Therefore, the vector allocating unit 54 allocates the passing motion vector to four neighboring pixels on the interpolation frame F in the state 83. At this time, depending on the pixel of the interpolation frame, there may be a case where no motion vector exists, or a plurality of motion vectors may be candidates for allocation. In the latter case, like the vector detection unit 52, the vector allocation unit 54 obtains an evaluation value for each motion vector and selects a motion vector to be allocated based on the evaluation value.

  A state 83 represents the state of the frame t and the frame t + 1 and the interpolation frame F to which the motion vector is assigned, which is input to the assignment compensation unit 57. In the interpolation frame F in the state 83, the pixels to which the motion vector is assigned by the vector assigning unit 54 and the pixels to which the motion vector is not assigned are shown.

  The allocation compensation unit 57 compensates for a pixel to which a motion vector in the state 83 is not allocated by using a motion vector allocated to a peripheral pixel of the pixel. This is because the motion vector of the peripheral pixel of the pixel of interest and the motion vector of the pixel of interest are similar if the assumption that the neighboring region of the pixel of interest has the same motion holds. As a result, a motion vector that is accurate to some extent is also given to a pixel to which no motion vector is assigned, and a motion vector is assigned to all the pixels on the interpolation frame F in the state 84. In this case as well, since motion vectors of a plurality of surrounding pixels exist as candidates, the allocation compensator 57 obtains an evaluation value for each motion vector in the same manner as the vector allocation unit 54, and based on the evaluation values Select the motion vector to be assigned.

  A state 84 represents the state of the frame t and the frame t + 1 input to the image interpolation unit 58 and the state of the interpolation frame F in which motion vectors are assigned to all the pixels. Based on the motion vectors assigned to all these pixels, the image interpolation unit 58 can determine the positional relationship between the pixels on the interpolation frame F and the two frames t and t + 1. Therefore, the image interpolation unit 58 uses the motion vector allocated on the interpolation frame F and the pixel values of the frame t and the frame t + 1 to interpolate as indicated by the black dot of the interpolation frame F in the state 85. Pixel values on the frame F are generated by interpolation. Then, the image interpolation unit 58 outputs the generated interpolated frame, and then outputs the frame t + 1 as necessary, thereby outputting the 60P signal image to a subsequent stage (not shown).

  Next, with reference to FIG. 5, the evaluation value of the motion vector used in the signal processing apparatus 1 according to the present invention will be described. As described above with reference to FIG. 4, in each unit (the vector detection unit 52, the vector allocation unit 54, and the allocation compensation unit 57) of the signal processing device 1, an optimal motion vector is selected for subsequent processing. At this time, in each part of the signal processing device 1, as an evaluation value for the motion vector, a sum of absolute differences (DFD (Displaced Frame Difference)) representing a correlation value between the blocks shifted by the vector amount of interest of the two frames is used. Used.

In the example of FIG. 5, the m × n block centered on the pixel position p on the frame t at the time t and the vector amount of the motion vector v of interest are shifted from the pixel position p on the frame t + 1 at the time t + 1. Two blocks of m × n blocks centered on the pixel position p + v are shown. The difference absolute value sum DFD _t (p) obtained between these two blocks is expressed by the following equation (1).

・・・（１）

... (1)

Here, F _t (p) represents the luminance value at the pixel position p at time t, and m × n represents the DFD calculation range (block) for obtaining the sum of absolute differences. Since this difference absolute value sum represents a correlation value between DFD calculation ranges (blocks) in two frames, generally, the smaller the difference absolute value sum, the more the waveform of the block between the frames matches. Thus, it is determined that the reliability of the motion vector v is higher as the difference absolute value sum is smaller. Thereby, this sum of absolute differences is used when the most probable motion vector is selected from a plurality of candidates.

  Therefore, hereinafter, in each unit (vector detection unit 52, vector allocation unit 54, and allocation compensation unit 57) of signal processing device 1, as an evaluation value when a motion vector is selected, a difference is obtained unless otherwise specified. An absolute value sum (hereinafter referred to as an evaluation value DFD) is used.

  Next, the process of converting the frame frequency of the signal processing apparatus 1 will be described with reference to the flowchart of FIG.

  In step S1, the vector detection unit 52 inputs the pixel value of the frame t + 1 of the input image at the time t + 1 and the pixel t of the frame t at the time t immediately before the input image of the frame memory 51, and proceeds to step S2. At this time, the vector allocating unit 54, the allocation compensating unit 57, and the image interpolating unit 58 determine the pixel value of the frame t + 1 of the input image at the time t + 1 and the frame t at the time t immediately before the input image of the frame memory 51. Enter.

  In step S2, the vector detection unit 52 executes a motion vector detection process and proceeds to step S3. That is, the vector detection unit 52 detects a motion vector between the target block of the frame t on the frame memory 51 and the target block of the next frame t + 1 that is the input image, and the detected motion vector is detected by the detection vector memory 53. And proceed to step S3. As a method for detecting the motion vector between the two frames, a gradient method or a block matching method is used. When there are a plurality of motion vector candidates, an evaluation value DFD is obtained for each motion vector, and a highly reliable motion vector based on the obtained evaluation value DFD is detected. That is, in this case, the most probable motion vector is selected and detected in the target block for detecting the motion vector. Details of the motion vector detection process in step S2 will be described later.

  In step S3, the vector allocation unit 54 executes vector allocation processing, and proceeds to step S4. That is, in step S3, the vector allocation unit 54 allocates the motion vector obtained on the frame t to the pixel of interest on the interpolation frame to be interpolated in the allocation vector memory 55, and the pixel to which the motion vector is allocated. The allocation flag in the allocation flag memory 56 is rewritten to 1 (True). For example, an allocation flag that is True indicates that a motion vector is allocated to the corresponding pixel, and an allocation flag that is False indicates that a motion vector is not allocated to the corresponding pixel. When there are a plurality of motion vector candidates in each pixel, an evaluation value DFD is obtained for each motion vector, and a highly reliable motion vector based on the obtained evaluation value DFD is assigned. That is, in this case, the most probable motion vector is selected and assigned in the target pixel to which the motion vector is assigned. Details of the vector allocation processing in step S3 will be described later.

  In step S4, the allocation compensator 57 performs an allocation compensation process, and proceeds to step S5. That is, the allocation compensation unit 57 refers to the allocation flag in the allocation flag memory 56 in step S4, and for the pixel of interest for which no motion vector has been allocated by the vector allocation unit 54, the motion vector of the surrounding pixels of the pixel of interest. Is allocated on the interpolation frame of the allocation vector memory 55. At this time, the allocation compensator 57 compensates the motion vector and rewrites the allocation flag of the allocated pixel of interest to 1 (True). When there are a plurality of motion vectors of peripheral pixels, an evaluation value DFD is obtained for each motion vector, and a highly reliable motion vector is assigned based on the obtained evaluation value DFD. That is, in this case, the most probable motion vector is selected and assigned in the target pixel to which the motion vector is assigned. Details of the allocation compensation processing in step S4 will be described later.

  In step S5, the image interpolation unit 58 executes an image interpolation process. That is, in step S5, the image interpolation unit 58 uses the motion vector allocated to the interpolation frame of the allocation vector memory 55 and the pixel values of the frame t and the frame t + 1 to generate an interpolation frame pixel value. The process proceeds to step S6. Details of the image interpolation processing in step S5 will be described later. In step S6, the image interpolation unit 58 outputs the generated interpolation frame, and then outputs the frame t + 1 as necessary, thereby outputting the 60P signal image to a subsequent stage (not shown). The process proceeds to step S7.

  In step S7, the vector detection unit 52 determines whether or not all the frames have been processed. If the vector detection unit 52 determines that all the frames have not been processed, the process returns to step S1 and the subsequent processing is performed. repeat. On the other hand, when the vector detection unit 52 determines in step S7 that all the frames have been processed, the vector detection unit 52 ends the process of converting the frame frequency.

  As described above, the signal processing apparatus 1 according to the present invention detects a motion vector from a frame of a 24P signal input image, assigns the detected motion vector to a pixel on a frame of a 60P signal, and assigns the assigned motion vector. Based on the above, the pixel value on the frame of the 60P signal is generated. At this time, the signal processing apparatus 1 selects a motion vector with higher reliability based on the evaluation value DFD (sum of absolute differences) in each process, and outputs it to the subsequent stage. Therefore, in the signal processing device 1, it is possible to suppress a failure of the movement and the like and generate a more accurate image.

  Next, details of the configuration of the image interpolation unit 58 will be described.

  FIG. 7 is a block diagram illustrating a configuration of the image interpolation unit 58. The image interpolation unit 58 having the configuration shown in FIG. 7 interpolates and generates the pixel value of the interpolation frame using the motion vector allocated to the interpolation frame in the allocation vector memory 55 and the pixel values of the frame t and the frame t + 1. , 60P signal image processing is performed.

  In the example of FIG. 7, the frame t of the image at time t is input to the spatial filter 92-1, and the frame t + 1 of the image at time t + 1 is input to the spatial filter 92-2 and the buffer 95.

  The interpolation control unit 91 selects the pixel of the interpolation frame in the allocation vector memory 55, and based on the motion vector assigned to the selected pixel, the pixel on the interpolation frame, the two frames t and t + 1 The positional relationship (spatial shift amount) with each pixel is obtained. That is, the interpolation control unit 91 uses, based on the pixels of the interpolated frame, the position on the frame t associated with the motion vector and the position of the pixel on the frame t corresponding to the pixel of the interpolated frame. The spatial shift amount is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-1. Similarly, the interpolation control unit 91 uses, based on the pixels of the interpolation frame, the position on the frame t + 1 associated with the motion vector and the position of the pixel on the frame t + 1 corresponding to the pixel of the interpolation frame. Is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-2.

  Further, the interpolation control unit 91 obtains an interpolation weight between the frame t and the frame t + 1 based on a preset time phase (time) of the interpolation frame, and uses the obtained interpolation weight as a multiplier 93-1. And 93-2. For example, when the time of the interpolation frame is a time that is “k” away from the time t + 1 of the frame t + 1 and a time that is “1−k” away from the time t of the frame t (ie, the time of the interpolation frame is the time t and time t + 1 is “1−k”: generated at a time that is internally divided into “k”), the interpolation control unit 91 sets an interpolation weight of “1-k” in the multiplier 93-1. An interpolation weight of “k” is set in the multiplier 93-2.

  Spatial filters 92-1 and 92-2 are constituted by cubic filters, for example. The spatial filter 92-1 is a pixel value on the frame t corresponding to the pixel of the interpolation frame based on the input pixel value of the pixel on the frame t and the spatial shift amount supplied from the interpolation control unit 91. And the obtained pixel value is output to the multiplier 93-1. The spatial filter 92-2, based on the input pixel value of the pixel on the frame t + 1 and the spatial shift amount supplied from the interpolation control unit 91, the pixel value on the frame t + 1 corresponding to the pixel of the interpolation frame And the obtained pixel value is output to the multiplier 93-2.

  When the position of the pixel in the interpolation frame does not match the position of the pixel on frame t or frame t + 1 (that is, when the position of the pixel in the interpolation frame is a component equal to or less than the pixel in frame t or frame t + 1). The spatial filters 92-1 and 92-2 obtain the sum of the inverse ratios of the distances of the four surrounding pixels by using the pixel values of the four surrounding pixels at the position of the pixel of the interpolation frame in the frame t or the frame t + 1. Then, the pixel value on the frame corresponding to the pixel of the interpolation frame is obtained. That is, the pixel value at the position below the pixel is obtained by linear interpolation based on the distance to the surrounding four pixels (details will be described later).

  The multiplier 93-1 multiplies the pixel value on the frame t input from the spatial filter 92-1 by the interpolation weight “1-k” set by the interpolation control unit 91, and the weighted pixel value is The result is output to the adder 94. The multiplier 93-2 multiplies the pixel value on the frame t + 1 input from the spatial filter 92-2 by the interpolation weight “k” set by the interpolation control unit 91, and adds the weighted pixel value to the adder. Output to 94.

  The adder 94 generates the pixel value of the pixel of the interpolation frame by adding the pixel value input from the multiplier 93-1 and the pixel value input from the multiplier 93-2. The pixel value of the interpolation frame is output to the buffer 95. The buffer 95 buffers the input frame t + 1. The buffer 95 outputs the generated interpolation frame, and then outputs a buffered frame t + 1 as necessary based on a preset time phase (time) of the 60P frame. Thus, an image of the 60P signal is output to a subsequent stage (not shown).

  Details of the image interpolation processing of the image interpolation unit 58 configured as described above will be described with reference to the flowchart of FIG.

  In step S51, the interpolation control unit 91 calculates interpolation weights (for example, “k” and “1-k”) of the interpolation frame between the frame t and the frame t + 1 based on the time phase of the interpolation frame to be processed. The obtained interpolation weights are set in the multipliers 93-1 and 93-2, respectively, and the process proceeds to step S52. In step S52, the interpolation control unit 91 selects a pixel of the interpolation frame in the allocation vector memory 55, and proceeds to step S53. Note that the pixels on the interpolation frame are selected in the raster scan order from the upper left pixel of the frame.

  In step S53, based on the motion vector assigned to the selected pixel, the interpolation control unit 91 determines the positional relationship between the pixels on the interpolation frame and the two frames t and t + 1 (space shift amount). ) And the obtained spatial shift amounts are supplied to the spatial filters 92-1 and 92-2, respectively, and the process proceeds to step S54. Specifically, in step S53, the interpolation control unit 91 uses the pixel of the interpolation frame as a reference, the position on the frame t associated with the motion vector, and the frame t corresponding to the pixel of the interpolation frame. The spatial shift amounts are obtained from the pixel positions, and the obtained spatial shift amounts are supplied to the spatial filter 92-1. Similarly, the interpolation control unit 91 uses, based on the pixels of the interpolation frame, the position on the frame t + 1 associated with the motion vector and the position of the pixel on the frame t + 1 corresponding to the pixel of the interpolation frame. Is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-2.

  The pixel value of the frame t of the image at time t is input to the spatial filter 92-1, and the pixel value of the frame t + 1 of the image at time t + 1 is input to the spatial filter 92-2. In step S54, the spatial filters 92-1 and 92-2 determine the pixels of the interpolation frame based on the pixel values of the pixels on the input frames t and t + 1 and the spatial shift amount supplied from the interpolation control unit 91. The pixel values on each frame corresponding to are obtained, the obtained pixel values are output to the multipliers 93-1 and 93-2, and the process proceeds to step S55.

  In Step S55, the multipliers 93-1 and 93-2 weight the interpolation weight set by the interpolation control unit 91 to the pixel value on each frame input from the spatial filter 92-1 or 92-2, The weighted pixel value is output to the adder 94, and the process proceeds to step S56. That is, the multiplier 93-1 multiplies the pixel value on the frame t input from the spatial filter 92-1 by the interpolation weight “1-k” set by the interpolation control unit 91, and weighted pixel values. Is output to the adder 94. The multiplier 93-2 multiplies the pixel value on the frame t + 1 input from the spatial filter 92-2 by the interpolation weight “k” set by the interpolation control unit 91, and adds the weighted pixel value to the adder. Output to 94.

  In step S56, the adder 94 adds the pixel value weighted by the multiplier 93-1 and the pixel value weighted by the multiplier 93-2 to generate the pixel value of the pixel of the interpolation frame. The generated pixel value is output to the buffer 95, and the process proceeds to step S57. In step S57, the interpolation control unit 91 determines whether or not the processing for all the pixels on the interpolation frame is completed, and determines that the processing for all the pixels on the interpolation frame is not completed. If so, the process returns to step S52, and the subsequent processing is repeated. If the compatibility control unit 91 determines in step S57 that the processing for all the pixels on the interpolation frame has been completed, the compatibility control unit 91 ends the image interpolation processing.

  As described above, the pixel value of the interpolation frame is generated based on the motion vector assigned to the interpolation frame, and the interpolation frame is output by the buffer 95 in step S6 of FIG. If necessary, the frame t + 1 is output, so that an image of the 60P signal is output to the subsequent stage. Therefore, since the most probable motion vector is assigned to the pixel of the interpolation frame, an accurate interpolation frame can be generated.

  Next, details of the configuration of the vector detection unit 52 will be described.

  FIG. 9 is a block diagram showing the configuration of the vector detection unit 52. The vector detection unit 52 having the configuration shown in FIG. 9 detects a motion vector on the frame t by using the input frame t of the image at time t and the frame t + 1 of the image at time t + 1. The process of detecting the motion vector is executed for each predetermined block composed of a plurality of pixels.

  The initial vector selection unit 101 uses, as an initial vector V0 that is an initial value used in the gradient method, a high-reliability motion vector obtained from the detection result of the past motion vector for each predetermined block, as an iterative gradient method calculation unit. To 103. Specifically, the initial vector selection unit 101 uses a motion vector of a peripheral block obtained in the past stored in the detection vector memory 53 or a shift initial vector stored in the shift initial vector memory 107 as an initial vector. Select as a candidate vector. Then, the initial vector selection unit 101 obtains the evaluation value DFD of the candidate vector using the frame t and the frame t + 1, and selects the candidate vector having the highest reliability based on the obtained evaluation value DFD. And output as an initial vector V0. Details of the configuration of the initial vector selection unit 101 will be described later.

  The pre-filters 102-1 and 102-2 are configured by a low-pass filter and a Gaussian filter, respectively, and remove noise components of the frame t and the frame t + 1 of the input image, and output them to the iterative gradient method computing unit 103.

  The iterative gradient method computing unit 103 uses the initial vector V0 input from the initial vector selection unit 101 and the frame t and the frame t + 1 input via the prefilters 102-1 and 102-2, for each predetermined block. Then, the motion vector Vn is calculated by the gradient method. The iterative gradient method computing unit 103 outputs the initial vector V0 and the calculated motion vector Vn to the vector evaluation unit 104. In addition, the iterative gradient method computing unit 103 repeatedly performs the gradient method computation based on the motion vector evaluation result by the vector evaluation unit 104 to calculate the motion vector Vn. Details of the configuration of the iterative gradient method computing unit 103 will be described later.

  The vector evaluation unit 104 obtains the motion vector Vn−1 (or initial vector V0) from the iterative gradient method computing unit 103 and the evaluation value DFD of the motion vector Vn, and based on the obtained evaluation value DFD, the iterative gradient method The calculation unit 103 is controlled to repeatedly execute the calculation of the gradient method. Finally, a highly reliable one based on the evaluation value DFD is selected and stored in the detection vector memory 53 as a motion vector V. At this time, the vector evaluation unit 104 supplies the evaluation value DFD obtained for the motion vector V together with the motion vector V to the shifted initial vector allocation unit 105.

  When the motion vector V and the evaluation value DFD are supplied from the vector evaluation unit 104, the shifted initial vector allocation unit 105 shifts the motion vector passing through the target block on the next frame to the target block. Set as initial vector. In other words, the shifted initial vector allocation unit 105 shifts a motion vector of the same size and the same direction as the motion vector V, starting from the target block on the next frame at the same position as the end block of the motion vector V. Set as initial vector. Then, the shifted initial vector allocation unit 105 allocates the set shifted initial vector to the shifted initial vector memory 107 in association with the block of interest.

  Specifically, the shifted initial vector allocation unit 105 stores the evaluation value DFD of the motion vector V allocated as the shifted initial vector in the evaluation value memory 106 in association with the target block, and stores the same target block. It is compared with the evaluation value DFD of another motion vector V that passes (that is, the block of the past frame at the same position as the target block). Then, the shifted initial vector allocation unit 105 shifts the motion vector V determined to have high reliability based on the evaluation value DFD to the target block, and allocates it to the shifted initial vector memory 107 as the shifted initial vector of the target block. Details of the configuration of the shifted initial vector allocation unit 105 will be described later.

Next, the principle of the gradient method used in the vector detection unit 52 will be described. First, a luminance value of a pixel represented by coordinates (x, y, t) using a horizontal, vertical, and time axis in a moving image is assumed to be g (x, y, t). Here, when the pixel of interest (x ₀ , y ₀ , t ₀ ) is displaced by (dx, dy, dt) during a minute time, the horizontal (vertical) and time axis gradients (difference differences) are respectively expressed as gx. When expressed as (x ₀ , y ₀ , t ₀ ), gy (x ₀ , y ₀ , t ₀ ), and gt (x ₀ , y ₀ , t ₀ ), the luminance value of the pixel after displacement is approximated by Taylor expansion. Is expressed by the following equation (2).

g (x ₀ + dx, y ₀ + dy, t ₀ + dt)
≒ g (x ₀ , y ₀ , t ₀ ) + gx (x ₀ , y ₀ , t ₀ ) dx
+ Gy (x ₀ , y ₀ , t ₀ ) dy + gt (x ₀ , y ₀ , t ₀ ) dt
... (2)

  Here, when a pixel of interest in a moving image moves by horizontal vx and vertical vy after one frame (hereinafter, expressed as (vx, vy)), the luminance value of the pixel is expressed by the following equation (3). Is done.

_{g (x 0 + vx, y} 0 + vy, t 0 +1) = g (x 0, y 0, t 0)
... (3)

  When Expression (2) is substituted into Expression (3), it is expressed by the following Expression (4).

gx (x ₀ , y ₀ , t ₀ ) vx + gy (x ₀ , y ₀ , t ₀ ) vy
+ Gt (x ₀ , y ₀ , t ₀ ) = 0
... (4)

  Since the equation (4) is a two-variable equation of vx and vy, the solution cannot be obtained with a single equation for the target pixel. Therefore, as described below, a block that is a peripheral region of the target pixel is considered as one processing unit, and it is assumed that all pixels in the block (peripheral region) have the same movement (vx, vy). Build a similar formula. Assuming an assumption, an expression for the number of neighboring pixels is obtained for two variables. Therefore, these equations are combined to obtain (vx, vy) that minimizes the sum of squares of motion compensation frame differences of all pixels in the block.

  When the pixel (x, y, t) moves by (vx, vy) by one frame, the motion compensation inter-frame difference d is expressed by the following equation (5).

d = g (x + vx, y + vy, t + 1) −g (x, y, t)
= Δxvx + Δyvy + Δt
... (5)

  In equation (5), Δx = gx (x, y, t), which represents a horizontal gradient, Δy = gy (x, y, t), which represents a vertical gradient, Δt = gt ( x, y, t), which represents the gradient in the time direction. Using these, assuming that the square sum of the difference between motion compensation frames is E, it is expressed by Equation (6).

E = Σd ²
= Σ (Δx ² vx ² + Δy ² vy ² + 2ΔxΔyvxvy
+ 2ΔxΔtvx + 2ΔyΔtvy + Δt ² )
= Vx ² ΣΔx ² + vy ² ΣΔy ² +2 vxvy ΣΔxΔy
+ 2vxΣΔxΔt + 2vyΣΔyΔt + ΣΔt ²
... (6)

  Here, E becomes the minimum (vx, vy) when the partial differential value in each variable becomes 0, that is, when the condition of δE / δvx = δE / δvy = 0 is satisfied. Thus, the following equations (7) and (8) are obtained.

vxΣΔx ² + vyΣΔxΔy + ΣΔxΔt = 0
... (7)
vyΣΔy ² + vxΣΔxΔy + ΣΔyΔt = 0
... (8)

  From these equations (7) and (8), (vx, vy) which is the desired motion can be obtained by calculating the following equation (9).

・・・（９）

... (9)

  Here, it demonstrates concretely with reference to FIG. In the example of FIG. 10, an arrow X indicates the horizontal direction, and an arrow Y indicates the vertical direction. In addition, an arrow T indicates the direction of time passage from the frame t at the time t in the back right to the frame t + 1 at the time t + 1 in the left front. In the example of FIG. 10, each frame shows only an area of 8 pixels × 8 pixels used for the gradient method calculation as a peripheral area (block) of the pixel of interest p.

  In the frame t, when the motion vector V (vx, vy) of the pixel of interest p, which is the fifth pixel down from the upper left pixel and the fifth pixel on the right, is obtained using the gradient method described above, the motion vector V (vx , Vy) is the difference in luminance (ie, gradient) Δx and Δy between adjacent pixels px and py obtained for the x and y directions of the pixel of interest p, and the same phase of the pixel of interest p obtained in frame t + 1. A difference in luminance (gradient) Δt in the time direction with respect to the pixel q located is obtained for all pixels in the peripheral region (8 pixels × 8 pixels) of the pixel of interest p, and the difference between them is expressed by Equation (9). It can obtain | require by calculating using.

  That is, the gradient method calculates gradients Δx, Δy, and Δt between two frames, and statistically calculates the motion vector V (vx, Δt from the obtained Δx, Δy, and Δt using the sum of squares of differences. vy).

  Generally, in a motion vector detection method using such a gradient method, a highly accurate result can be obtained for a minute motion. However, when trying to obtain a motion in an actual moving image, this gradient method is not practical because the amount of motion is too large. Correspondingly, a method of repeating this gradient method a plurality of times is conceivable. By repeatedly executing the gradient method, the amount of motion obtained by each calculation converges, so that a correct motion is gradually obtained.

  However, it is not practical from the viewpoint of calculation time when trying to perform real-time processing only by repeating the gradient method. Therefore, the vector detection unit 52 reduces the number of iterations of the gradient method by using, as an initial value, an initial vector obtained based on the motion of surrounding pixels in the past frame and the current frame. In other words, if a rough motion is calculated by adding an offset in advance from the pixel of interest that is the starting point of the motion to the destination indicated by the initial vector, an operation using the gradient method is performed from the position where the offset is added. Fine adjustment including movement below the pixel can be performed. Thereby, an accurate motion vector can be detected without increasing the calculation time.

  FIG. 11 is a diagram specifically explaining the iterative gradient method executed using the initial vector. In the example of FIG. 11, the arrow T indicates the passage of time from the frame t at the time t on the left front side to the frame t + 1 at the time t + 1 on the right back in the drawing. A block centered on each pixel p, q0, q1, q2, and q3 represents a peripheral region (block) used for the gradient method calculation of that pixel.

  In the case of the example in FIG. 11, the initial vector v0 obtained in advance is offset (moved) instead of the pixel q0 located in the same phase of the target pixel p in the frame t + 1 with respect to the target pixel p in the frame t. The first gradient method calculation is performed with the position (pixel) q1 calculated in this way as the starting point, and as a result, a motion vector v1 is obtained.

  Next, the second gradient method calculation is performed with the position (pixel) q2 calculated by offsetting v0 + v1 from the pixel q0 as a starting point, and as a result, a motion vector v2 is obtained. Thereby, the motion vector V is finally obtained as Expression (10).

V = v0 + v1 + v2
... (10)

  As described above, by performing the iterative gradient method using the initial vector, it is possible to obtain a highly accurate motion vector while reducing the calculation time.

  Next, details of the motion vector detection process will be described with reference to the flowchart of FIG. The vector detection unit 52 receives the input frame t of the image at time t and input frame t + 1 of the image at time t + 1.

  In step S101, the initial vector selection unit 101 selects a block to be processed on the frame t as a target block, and the process proceeds to step S102. On the frame, the processing is executed in the raster scan order from the upper left block.

  In step S102, the initial vector selection unit 101 performs an initial vector selection process. In step S101, the initial vector selection unit 101 selects a motion vector with high reliability from the detection result of the past motion vector for each predetermined block, and uses the selected motion vector as an initial value used in the gradient method. The initial vector V0 is output to the iterative gradient method computing unit 103, and the process proceeds to step S103.

  That is, the initial vector selection unit 101 obtains the motion vector of the peripheral blocks obtained in the past gradient method calculation evaluation process (step S103 described later) and stored in the detection vector memory 53, and the past shifted initial vector allocation process (described later). In step S104), the shifted initial vector stored in the shifted initial vector memory 107 is selected as an initial vector candidate vector. Then, the initial vector selection unit 101 obtains the evaluation value DFD of the candidate vector using the frame t and the frame t + 1, and selects a candidate vector having a high reliability based on the obtained evaluation value DFD. The selected candidate vector is output as the initial vector V0. Details of the initial vector selection process in step S102 will be described later.

  In step S103, the iterative gradient method calculation unit 103 and the vector evaluation unit 104 execute an iterative gradient method calculation evaluation process (also referred to as an iterative gradient method calculation process), and the process proceeds to step S104. Specifically, in step S103, the iterative gradient method computing unit 103 receives the initial vector V0 input from the initial vector selecting unit 101, the frame t and the frame input via the prefilters 102-1 and 102-2. Based on the motion vector evaluation result by the vector evaluation unit 104 using t + 1, the gradient method is repeatedly calculated to calculate the motion vector Vn. Further, the vector evaluation unit 104 obtains the motion vector Vn−1 from the iterative gradient method computing unit 103 and the evaluation value DFD of the motion vector Vn, and selects the most reliable one based on the obtained evaluation value DFD. And stored in the detected vector memory 53 as a motion vector V. At this time, the vector evaluation unit 104 supplies the evaluation value DFD obtained for the motion vector V together with the motion vector V to the shifted initial vector allocation unit 105. Details of the iterative gradient method calculation evaluation process in step S103 will be described later.

  In step S104, the shifted initial vector allocation unit 105 executes a shifted initial vector allocation process, and proceeds to step S105. When the motion vector V and its evaluation value DFD are supplied from the vector evaluation unit 104, the shifted initial vector allocation unit 105 shifts the motion vector passing through the target block on the next frame to the target block in step S104. The shifted initial vector is set. That is, in other words, a motion vector having the same size and the same direction as the motion vector V is set as the initial shift vector, starting from the block of interest on the next frame at the same position as the end block of the motion vector V. . Then, the shifted initial vector allocation unit 105 allocates the set shifted initial vector to the shifted initial vector memory 107 in association with the block of interest.

  Specifically, the shifted initial vector allocation unit 105 stores the evaluation value DFD of the motion vector V allocated as the shifted initial vector in the evaluation value memory 106 in association with the target block, so that the same target Compared with the evaluation value DFD of another motion vector V that passes through the block (that is, the block of the past frame at the same position as the target block is the end point), the motion determined to have high reliability based on the evaluation value DFD The vector V is shifted to that block and set as a shifted initial vector, and is allocated to the shifted initial vector memory 107 in correspondence with the shifted block. Details of the configuration of the shifted initial vector allocation unit 105 will be described later.

  In step S105, the initial vector selection unit 101 determines whether or not all blocks have been processed in the frame t. If it is determined that all blocks have not been processed, the initial vector selection unit 101 returns to step S101. The subsequent processing is repeated. In step S105, the initial vector selection unit 101 determines that the motion vectors V have been detected in all the blocks on the frame t when it is determined that all the blocks have been processed in the frame t. Then, the motion vector detection process ends.

  As described above, an initial vector is selected from motion vectors detected in the past, an iterative gradient method is calculated based on the selected initial vector, and a repeated motion vector is calculated. Among them, a motion vector having a high reliability (that is, most likely) based on the evaluation value DFD is detected. As a result, the motion vector V corresponding to all the blocks on the frame t is stored in the detection vector memory 53.

  Next, the details of the configuration of the shifted initial vector allocation unit 105 will be described.

  FIG. 13 is a block diagram showing a configuration of the shifted initial vector allocating unit 105. The shifted initial vector allocation unit 105 having the configuration shown in FIG. 13 sets a shifted initial vector that is a candidate vector of the initial vector based on the motion vector V detected by the vector evaluation unit 104 in the previous (past) frame, A process of assigning to the shifted initial vector memory 107 is performed. The shifted initial vector allocation unit 105 receives the motion vector V detected by the vector evaluation unit 104 and the evaluation value DFD of the motion vector V.

  The allocation target position calculation unit 201 detects the position of the block through which the motion vector V detected by the vector evaluation unit 104 passes on the frame at the next time (that is, the end point of the motion vector V detected on the current frame). The position of the block on the next frame at the same position as the block) is calculated, and the calculated position of the block is supplied to the evaluation value memory 106 and the shifted initial vector replacement unit 203.

  When the evaluation value comparison unit 202 receives the motion vector V and the evaluation value DFD of the motion vector V, the evaluation value comparison unit 202 reads the evaluation value DFD of the block position from the allocation target position calculation unit 201 from the evaluation value memory 106. . Then, the evaluation value comparison unit 202 compares the evaluation value DFD read from the evaluation value memory 106 with the evaluation value DFD of the motion vector V detected by the vector evaluation unit 104, and detects the detected motion vector V When the evaluation value DFD is smaller (that is, the reliability is higher), the shifted initial vector replacement unit 203 is controlled to determine the block of the block supplied by the shifted initial vector allocation unit 105 of the shifted initial vector memory 107. The position initial shift vector is rewritten with the motion vector V determined to have high reliability based on the evaluation value DFD. At the same time, the evaluation value comparison unit 202 controls the evaluation value replacement unit 204, and the evaluation value memory 106 selects the evaluation value DFD of the position of the block selected by the allocation target position calculation unit 201 in the evaluation value memory 106. Rewrite with evaluation value DFD.

  The shift initial vector replacement unit 203 uses the motion vector V (that is, the motion) supplied from the evaluation value comparison unit 202 as the shift initial vector of the block position supplied from the allocation target position calculation unit 201 in the shift initial vector memory 107. The motion vector having the same magnitude as the vector V and the same direction) is rewritten. The evaluation value replacement unit 204 converts the evaluation value DFD of the position of the block selected by the allocation target position calculation unit 201 into the evaluation value DFD of the motion vector V in the evaluation value memory 106 under the control of the evaluation value comparison unit 202. Rewrite with.

  The evaluation value memory 106 stores, for each block, an evaluation value DFD of a shifted initial vector candidate assigned to each block on the next frame. The shifted initial vector memory 107 stores the motion vector having the smallest evaluation value DFD (that is, the most reliable) in each block in the next frame as a shifted initial vector in association with the block.

  FIG. 14 is a diagram for describing one-dimensionally the shifted initial vectors allocated by the shifted initial vector allocating unit 105. In the example of FIG. 14, a frame t−1 at time T = t−1, a frame t at time T = t, and a frame t + 1 at time T = t + 1 are shown in order from the top. The partition indicates a block boundary in each frame.

  In the case of the example in FIG. 14, the motion vector detected in the block B on the frame t−1 is the motion vector V (solid arrow in the figure), and the motion on the frame t by the motion vector V starting from the block B. A compensation destination (hereinafter also referred to as an offset destination) is a block Bt. A motion vector obtained by shifting the motion vector V on the frame t-1 to the block Bt on the frame t is defined as a shifted initial vector SV (dotted line arrow in the figure). In this case, the offset destination represents the block Bt on the frame t at the same position as the end block of the motion vector V detected on the frame t−1, and the shift means the frame t The start point of a motion vector having the same magnitude and the same direction as the motion vector V detected on -1 is a block Bt on the frame t at the same position as the end point block of the motion vector V on the frame t-1. That means. That is, the shifted initial vector SV is the same size and the same direction as the motion vector V, starting from the block Bt on the frame t at the same position as the end block of the motion vector V detected on the frame t−1. It is a motion vector.

  In general, there is a certain degree of continuity in the amount of movement of the moving object between consecutive frames, and the change in the amount of movement is considered to be small. Therefore, in the example of FIG. When the motion is at or close to it, the motion vector in the block Bt is often the motion vector V or a motion vector close thereto.

  Therefore, when this shifted initial vector SV is used as a candidate for an initial vector given to the gradient method calculation when detecting a motion vector of the block of interest Bt on the frame t, only the motion vectors of the peripheral blocks are used as initial vector candidates. It is possible to obtain a more appropriate initial vector for motion vector detection by the gradient method calculation than in the case of using as.

  The above will be specifically described with reference to FIGS. 15 and 16. In FIGS. 15 and 16, nine blocks are shown on each frame.

  FIG. 15 shows an example of a motion vector of a block through which a large motion object passes and its neighboring blocks. In the case of the example in FIG. 15, the image object of a certain object O1 consisting of one block is a motion vector V1 of a large motion, and is on the frame t from the rightmost block (the ninth from the left) on the frame t-1. It moves to the leftmost block on frame t + 1 via the fifth block from the left. On the other hand, the image object of the object O2 composed of four blocks is a motion vector V2 having a motion smaller than the motion vector V1, and the fourth to fourth blocks from the third to sixth from the left on the frame t−1 and 4 from the left on the frame t. Through the 4th to 7th blocks, the frame moves to the 5th to 8th 4 blocks from the left on the frame t + 1. That is, in the fifth block from the left on the frame t, the image object of the object O1 and the image object of the object O2 intersect each other with the image object of the object O1 in front.

  At this time, in order to detect the motion vector of the fifth block from the left on which the image object of the object O1 passes by the motion vector V1 on the frame t, the initial vector is extracted from the motion of the peripheral blocks. Since the image object of the object O2 passes through the block (the fourth or sixth block from the left) with a motion vector V2 different from that of the image object of the object O1, there is no appropriate motion vector. On the other hand, the shift initial vector SV1 (dotted arrow in the figure) is the fifth motion vector from the left on the frame t, even in such a case, with the motion vector V1 detected on the past frame T-1. Motion vector shifted to a block (that is, the same size and the same direction as the motion vector V1, starting from a block on the frame t at the same position as the end block of the motion vector V1 detected on the frame t-1) Therefore, the motion vector can be reliably acquired.

  FIG. 16 shows an example of a motion vector of a block at the boundary of a moving object and its surrounding blocks. In the case of the example of FIG. 16, the image object of a certain object O3 consisting of one block is a motion vector V3, and the sixth block from the left on the frame t-1 to the fifth block from the left on the frame t. Via, it moves to the fourth block from the left on the frame t + 1. On the other hand, the image object of the object O4 composed of 9 blocks or more is a motion vector V4 different from the motion vector V1, and from the 9 blocks on the frame t-1 to the 2nd to 9th 8 blocks from the left on the frame t. Via, it moves to the third to ninth seventh blocks from the left on the frame t + 1. That is, the image object of the object O3 is in front of the image object of the object O4 in the sixth block from the left of the frame t-1, the fifth block from the left on the frame t, and the fourth block from the left of the frame t + 1. Is going through.

  At this time, in order to detect the motion vector of the fifth block from the left on which the image object of the object O3 passes with the motion vector V3 on the frame t, the initial vector is extracted from the motion of the peripheral blocks. Since the image object of the object O4 passes through the block (the fourth or sixth block from the left) with a motion vector V4 different from that of the image object of the object O3, there is no appropriate motion vector. However, in this case as well, as in the case of FIG. 15, the shifted initial vector SV3 (indicated by a dotted arrow in the figure) represents the motion vector V3 detected on the past frame T-1 as the right on the frame t. To the fifth block (ie, the same size as the motion vector V3, starting from the block on the frame t at the same position as the end block of the motion vector V3 detected on the frame t-1) The motion vectors in the same direction in FIG.

  As described above, at the boundary of an object having a large movement as shown in FIG. 15 or an object moving as shown in FIG. There are many cases where the optimal motion vector does not exist as the initial vector of the block of interest in the peripheral block. On the other hand, the initial shift vector has the same size and the same size as the motion vector starting from the block on the frame t at the same position as the end block of the motion vector detected on the past frame t-1. Since it is a motion vector in the direction, it does not depend on the spatial distance between the blocks, so even if there is no appropriate motion vector as the initial vector of the block of interest in the surrounding blocks, it can be obtained reliably. It can be used as an optimal candidate.

  Here, returning to FIG. 14, a method for determining the shifted initial vector will be described. In the example of FIG. 14, with reference to the target block Bt on the frame t on which the motion vector detection process is performed, the motion vector passing from the frame t-1 through the target block Bt (that is, at the same position as the block Bt on the frame t). If an attempt is made to search for a motion vector whose end point is a block of a certain frame t-1, the motion vectors of all the blocks on the frame t-1 must be searched, and the processing becomes enormous. Therefore, in this case, when detecting a motion vector, it is necessary to deal with a huge amount of processing, and it is difficult to implement (hardware).

  Therefore, the shifted initial vector allocating unit 105 detects the position of the block through which the motion vector V detected in the block of interest Bt passes on the frame t at the next time (that is, when detecting the motion vector in the frame t−1). A block on the frame t at the same position as the end point block of the motion vector V) is obtained, and the motion vector V is assigned in advance as a shift initial vector SV for the block Bt. As a result, it is possible to reduce the amount of calculation for searching for the initial shift vector necessary for detecting the motion vector on the frame t, and to realize hardware implementation. It can be used as a candidate for an initial vector.

  With reference to FIG. 17, a method for determining the shifted initial vector by shifted initial vector allocating unit 105 will be specifically described. In the example of FIG. 17, the arrow T indicates the passage of time from the frame t-1 at time t-1 on the left front side to the frame t at time t on the right back in the drawing. A circle on each frame represents a pixel.

  On the frame t−1, the block B0 is composed of 4 × 4 pixels. Assuming that the motion vector Vb detected in the block B0, an offset destination (a block on the frame t at the same position as the end block of the motion vector Vb of the block B0) A0 obtained by offsetting (motion compensation) the block B0 on the frame t is In general, there is an overlap with four blocks Bt1 to Bt4 configured by 4 × 4 pixels on the frame t. Therefore, in the example of FIG. 17, all of the four blocks Bt1 to Bt4 that overlap with the offset destination A0 are set as target blocks to which the motion vector Vb is allocated as the shifted initial vector.

  The motion vector detected in the previous frame as described above is assigned as the shifted initial vector. In this case, the position of the block to be allocated depends on the magnitude and direction of the motion vector. Therefore, depending on the distribution of motion vectors detected on the frame t−1, as shown in FIG. 18, on the frame t, a block that is a target for allocation of a plurality of shift initial vectors, or conversely, a shift Blocks that do not have an initial vector assignment will occur.

  FIG. 18 shows an example of allocation of shifted initial vectors. In the example of FIG. 18, the partition of each frame indicates the boundary of the block, and is composed of blocks B1 to B5 on the frame t−1 from the left in the figure, and on the frame t on the left in the figure. Thus, blocks Bt1 to Bt5 are shown.

  In the block B1, the motion vector V1 is detected, and the offset destination A1 of the block B1 overlaps with the blocks Bt1 and Bt2 on the frame t. In block B2, motion vector V2 is detected, and offset destination A2 of block B2 overlaps with blocks Bt1 and Bt2 on frame t. In block B3, motion vector V3 is detected, and offset destination A3 of block B3 overlaps with blocks Bt4 and Bt5 on frame t. In block B4, motion vector V4 is detected, and offset destination A4 of block B4 overlaps with blocks Bt4 and Bt5 on frame t. In the block B5, the motion vector V5 is detected, and the offset destination A5 of the block B5 overlaps with the block Bt5 on the frame t and an adjacent block (not shown).

  That is, in the example of FIG. 18, one of the motion vectors V1 and V2 is allocated as the shifted initial vector to the blocks Bt1 and Bt2 on the frame t. Also, one of the motion vectors V3 and V4 is assigned as the initial shift vector to the block Bt4 on the frame t, and any one of the motion vectors V3, V4, and V5 is assigned to the block Bt5 on the frame t. Are assigned as shifted initial vectors. However, in the block Bt3 on the frame t, there is no motion vector that is a candidate for the shifted initial vector, and the shifted initial vector is not allocated.

  Therefore, the shifted initial vector allocating unit 105 allocates the zero vector as the shifted initial vector to a block such as the block Bt3 where allocation of the shifted initial vector does not occur. On the other hand, the shifted initial vector allocating unit 105 assigns a motion vector having high reliability based on the above-described evaluation value DFD to a block in which a plurality of motion vectors compete, such as the blocks Bt1, Bt2, Bt4, and Bt5. The selected motion vector is assigned to each block as a shifted initial vector.

  In the example of FIG. 18, for convenience of explanation, the offset destination A is shown slightly shifted from the frame t.

  FIG. 19 is a diagram illustrating an example of competing motion vectors. In the example of FIG. 19, the motion vectors V1 and V2 detected in the blocks B1 and B2 on the frame t-1 respectively pass the block Bt on the frame t. That is, the block Bt is a block on the frame t at the same position as the end point on the frame t-1 of the motion vector V1, and is a block on the frame t at the same position as the end point on the frame t-1 of the motion vector V2. is there. Thus, when a plurality of motion vectors compete for the same block, the evaluation value DFD1 for the motion vector V1 is compared with the evaluation value DFD2 for the motion vector V2, and the evaluation value DFD is smaller (that is, , With higher reliability) is selected as the shifted initial vector SV2 corresponding to the block Bt. As a result, a highly reliable shifted initial vector based on the evaluation value DFD can be obtained.

  Next, the details of the shifted initial vector allocation processing will be described with reference to the flowchart of FIG. In the preceding stage, when the vector evaluation unit 104 detects the motion vector V of the block of interest on the frame t−1, the vector evaluation unit 104 uses the detected motion vector V and the evaluation value DFD obtained for the motion vector V as the shifted initial vector. This is supplied to the allocation unit 105.

  In step S <b> 201, the evaluation value comparison unit 202 inputs the evaluation value DFD of the motion vector V together with the motion vector V from the vector evaluation unit 104. At this time, the allocation target position calculation unit 201 also receives the motion vector V and proceeds to step S202. The allocation target position calculation unit 201 obtains the position of the allocation target block that is the offset (motion compensation) destination of the motion vector V in the frame t. That is, the allocation target position calculation unit 201 obtains the position of the block on the frame t at the same position as the end point block of the motion vector V detected on the frame t−1, and proceeds to step S203. In step S203, the allocation target position calculation unit 201 selects one allocation target block from the obtained allocation target blocks, and sets the position of the selected allocation target block to the evaluation value memory 106 and the shifted initial vector replacement unit 203. To proceed to step S204. In step S203, the blocks to be allocated are selected in order from the upper left block on the frame t.

  In step S204, the evaluation value comparison unit 202 acquires the evaluation value DFD of the allocation target block selected by the allocation target position calculation unit 201 from the evaluation value memory 106, proceeds to step S205, and the motion input in step S201. Whether the evaluation value DFD of the vector V is smaller than the evaluation value DFD of the evaluation value memory 106 (that is, whether the evaluation value DFD of the motion vector V is more reliable than the evaluation value DFD of the evaluation value memory 106). If the evaluation value DFD of the motion vector V is determined to be smaller than the evaluation value DFD of the evaluation value memory 106, the process proceeds to step S206.

  In step S206, the evaluation value comparison unit 202 controls the shift initial vector replacement unit 203 to determine the shift initial vector of the allocation target block in the shift initial vector memory 107 selected by the allocation target position calculation unit 201 as the motion vector V ( That is, the motion vector V is rewritten to a motion vector having the same magnitude and the same direction as the motion vector V, and the process proceeds to step S207 where the evaluation value replacement unit 204 is controlled and the evaluation value of the allocation target block selected by the allocation target position calculation unit 201 The DFD is rewritten with the evaluation value DFD of the motion vector V, and the process proceeds to step S208.

  If the evaluation value comparison unit 202 determines in step S205 that the evaluation value DFD of the motion vector V input in step S201 is not smaller than the evaluation value DFD stored in the evaluation value memory 106, step S206 is performed. And the process of S207 is skipped and it progresses to step S208. That is, in this case, since the evaluation value DFD of the evaluation value memory 106 is determined to have higher reliability than the evaluation value DFD of the motion vector V, the values of the evaluation value memory 106 and the shifted initial vector memory 107 are rewritten. There is nothing.

  In step S208, the allocation target position calculation unit 201 determines whether all the allocation target blocks of the motion vector V have been processed, and if it determines that all the allocation target blocks have not been processed, step S203. Return to and repeat the subsequent processing. If the allocation target position calculation unit 201 determines in step S208 that all the allocation target blocks of the motion vector V have been processed, the shifted initial vector allocation process ends.

  In the first process, the shifted initial vector corresponding to the selected allocation target block is not yet stored in the shifted initial vector memory 107. Therefore, if the evaluation value DFD of the shifted initial vector is not yet stored in the corresponding allocation target block of the evaluation value memory 106, the evaluation value DFD is obtained from the selected allocation target block in step S204. Since it is not carried out, it will be judged as Yes in Step S205 and processing of Steps S206 and S207 will be performed.

  As described above, when the motion vector detection of the block of interest is processed, the motion amount of the moving object has a certain degree of continuity between successive frames, and the change in the motion amount is small. A motion vector that passes through the target block of the frame at the next time from the frame at the time (past) (that is, a block on the next frame at the same position as the end block of the motion vector V detected on the past frame) As a conventional method, a shift initial vector that is a motion vector having the same magnitude as the motion vector V and having the same direction as a starting point is used as a candidate vector of the initial vector, and thus is obtained in the past in the peripheral block as in the past. Compared to the case where only the motion vector is the candidate vector of the initial vector, especially at the boundary of the moving object, the gradient method It can give an appropriate motion vector as an initial value for calculation, which makes it possible to improve the accuracy of motion vector detection by computation of the gradient method.

  Further, when the initial shift vector is obtained, the motion vector detected in the frame at the previous time passes in the frame at the next time (that is, the block at the end of the motion vector V detected on the frame t−1). And a motion vector passing through the block of interest is assigned to all the frames of the previous time. Compared to the search from the motion vector of the block, the calculation amount of processing is reduced, so that it is possible to realize hardware that is difficult to realize due to the huge amount of calculation.

  Next, details of the configuration of the initial vector selection unit 101 will be described.

  FIG. 21 is a block diagram illustrating a configuration of the initial vector selection unit 101. The initial vector selection unit 101 having the configuration shown in FIG. 21 uses a motion vector with high reliability from a motion vector detected in the previous (past) frame and a candidate vector such as a shifted initial vector (hereinafter also referred to as an initial candidate vector). Is selected as the initial vector. The initial vector selection unit 101 receives the frame t of the image at time t and the frame t + 1 of the image at time t + 1.

  When the frame t is input, the candidate vector position calculation unit 251 selects a target block to be processed on the frame t, and obtains an initial candidate vector of the target block from the peripheral region of the target block. The position, the type of motion vector to be the initial candidate vector and the priority order are obtained, and the position information of the candidate block and the type information of the initial candidate vector are obtained in the order of the obtained priority order, and the detected vector acquisition unit 252 and the shifted initial vector acquisition unit 253. The candidate vector position calculation unit 251 also supplies the position information of the candidate block to the offset position calculation unit 254.

  In the signal processing device 1, the number of initial candidate vectors is set to a predetermined number based on the balance between the accuracy of the initial vector and the hardware capability, and further, the position of the candidate block and the initial candidate vector The type and priority are also set in advance as will be described later with reference to FIG. In addition, as a type of the initial candidate vector, a motion vector obtained by shifting a motion vector passing through a predetermined block in the past frame to the predetermined block (that is, a block at the end point of the motion vector detected on the past frame) A shift initial vector SV, which is a motion vector having the same magnitude and the same direction as the motion vector V, starting from a block on the next frame at the same position, and a motion vector detected in the past frame (hereinafter, past vector) PV), a motion vector detected in a block before the target block in the current frame (also referred to as current vector CV), and a zero vector.

  Accordingly, when the preset initial candidate vector type information is a past vector or a current vector, the candidate vector position calculation unit 251 displays the candidate block position information and the initial candidate vector type information as the detected vector acquisition unit 252. If the type information of the obtained initial candidate vector is a shifted initial vector, the position information of the candidate block and the type information of the initial candidate vector are supplied to the shifted initial vector acquisition unit 253, which is neither of them. In the case (for example, when the type information of the initial candidate vector is 0 vector), the 0 vector is set, and the position information of the candidate block is supplied to the offset position calculation unit 254 together with the 0 vector.

  The detection vector acquisition unit 252 acquires the motion vector corresponding to the position information of the candidate block and the type information of the initial candidate vector supplied from the candidate vector position calculation unit 251 from the detection vector memory 53, and the acquired motion vector is The initial candidate vector is output to the offset position calculation unit 254.

  The shifted initial vector acquisition unit 253 generates the shifted initial vector corresponding to the position information of the candidate block according to the position information of the candidate block and the type information of the initial candidate vector supplied by the candidate vector position calculating unit 251. Obtained from the memory 107 and output to the offset position calculation unit 254 as an initial candidate vector. In addition, the shifted initial vector acquisition unit 253 outputs a zero vector to the offset position calculating unit 254 when the shifted initial vector is not allocated to the position of the block designated by the candidate vector position calculating unit 251. If no shifted initial vector is assigned, the zero vector may be stored in advance in the shifted initial vector memory 107.

  When the initial position candidate vector (or 0 vector from the candidate vector position calculation section 251) is input from the detection vector acquisition section 252 or the shifted initial vector acquisition section 253, the offset position calculation section 254 is supplied by the candidate vector position calculation section 251. Based on the position information of the candidate block, an offset destination block position obtained by offsetting (motion compensation) the target block of frame t to frame t + 1 is calculated for each initial candidate vector. Then, the offset position calculation unit 254 outputs the candidate block position information and the offset destination block position information to the evaluation value calculation unit 255 together with the initial candidate vector.

  When the evaluation value calculation unit 255 inputs the position information of the candidate block and the information of the offset destination block position together with the initial candidate vector from the offset position calculation unit 254, the evaluation value of the initial candidate vector is used using the frame t and the frame t + 1. Find the DFD. Then, the evaluation value calculation unit 255 outputs the obtained evaluation value DFD together with the initial candidate vector to the evaluation value comparison unit 256.

  The evaluation value comparison unit 256 compares the evaluation value DFD input by the evaluation value calculation unit 255 with the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 257 and inputs the evaluation value DFD by the evaluation value calculation unit 255. When the evaluation value DFD of the initial candidate vector is smaller than the evaluation value DFD of the optimal candidate vector, that is, when it is determined that the initial candidate vector has higher reliability than the optimal candidate vector, the optimal candidate storage register 257 The optimal candidate vector and its evaluation value DFD are replaced with the initial candidate vector and its evaluation value DFD that are determined to have high reliability. Finally, the evaluation value comparison unit 256 controls the optimum candidate storage register 257 to select the optimum candidate vector determined to have the highest reliability based on the evaluation value DFD from among all candidate vectors. The iterative gradient method computing unit 103 outputs the initial vector V0.

  In the optimum candidate storage register 257, an initial candidate vector whose evaluation value DFD is made small (high reliability) by the evaluation value comparison unit 256 is stored as an optimum candidate vector together with the evaluation value DFD. Then, under the control of the evaluation value comparison unit 256, the optimal candidate storage register 257 outputs the finally stored optimal candidate vector to the iterative gradient method computing unit 103 as the initial vector V0.

  FIG. 22 shows a peripheral region that can be a candidate block of the initial vector. In the example of FIG. 22, the arrow T indicates the direction of time passage from the past (previous) frame t−1 at the time t−1 on the left front side to the current frame t at the time t on the right rear side. . In the example of FIG. 22, the peripheral region that can be a candidate for the initial vector is configured by 7 blocks × 7 blocks centered on the block Bt of interest, and each block is configured by 4 pixels × 4 pixels. Has been.

  As described above, the motion vector detection process is executed in the raster scan order from the upper left block of the frame. Therefore, the initial vector selection unit 101 can use the result of motion vector detection up to the immediately preceding block as an initial vector candidate in the motion vector detection processing of the block Bt of interest in the current frame t.

  That is, the peripheral area of the block of interest Bt is composed of a block CVB processed before the block of interest Bt and a block PVB processed after the block of interest Bt. Therefore, when obtaining the initial vector of the block of interest Bt, the initial candidate vector is detected on the motion vector (current vector CV) detected on the current frame t of the block CVB and on the past frame t-1 of the block PVB. Motion vectors (past vectors PV) can be selected. At this time, a shifted initial vector assigned to a block in the same peripheral area can also be a candidate for an initial candidate vector.

  FIG. 23 illustrates an example of initial vector candidate blocks set in the candidate vector position calculation unit 251. In the example of FIG. 23, eight predetermined blocks each having an “alphabet / number” symbol are set as candidate blocks from which an initial candidate vector is acquired in the peripheral area of the block of interest Bt. Yes. The symbols before and after the hatched lines shown in these eight predetermined blocks represent “initial candidate vector type” and “priority”, respectively. Of the types of initial candidate vectors, P represents the past vector PV detected in the past frame t−1, C represents the current vector detected in the current frame t, and S is The shifted initial vector SV is represented.

  “S / 1” of the target block Bt indicates that the shifted initial vector SV assigned to the target block Bt is used as the initial candidate vector in the priority “1”. “C / 2” of the block adjacent to the left of the block of interest Bt indicates that the current vector CV of the block detected in the current frame t is used as the initial candidate vector in the “2” priority order. “C / 3” in the upper right block of the block of interest Bt indicates that the current vector CV of the block detected in the current frame t is used as the initial candidate vector with the priority “3”. “P / 4” in the lower right block of the target block Bt indicates that the past vector PV of the block detected in the past frame t−1 is used as the initial candidate vector with the priority “4”. Yes.

  As for “C / 6” of the lower right block at the upper left corner of the peripheral area 7 × 7, the current vector CV of the block detected in the current frame t is used as the initial candidate vector with the priority “6”. Represents. “C / 5” of the lower left block in the upper right corner of the peripheral area 7 × 7 indicates that the current vector CV of the block detected in the current frame t is used as the initial candidate vector with the priority “5”. Represents. For the upper right block “P / 7” in the lower left corner of the peripheral area 7 × 7, the past vector PV of the block detected in the past frame t−1 is used as the initial candidate vector with the priority “7”. Represents that. In the upper left block “P / 8” in the lower right corner of the peripheral area 7 × 7, the past vector PV of the block detected in the past frame t−1 is used as the initial candidate vector with the priority “8”. Represents that.

  As described above, in the example of FIG. 23, the initial candidate vectors include the block of interest Bt, the block adjacent to the block of interest Bt, the block next to the block of interest Bt, the block adjacent to the left of the block of interest Bt, the upper right, and the lower right. Among the blocks that are not adjacent to the block of interest Bt, the block is selected from the blocks at positions one inside each from the four corners of the peripheral area 7 × 7.

  That is, in the case of the example in FIG. 23, a block adjacent to the block of interest Bt (a block that is spatially and temporally close) has a higher priority than a block slightly apart, and the current vector detected in the current frame t. The priority is set higher than the past vector detected in the past frame t-1. Note that the positions, types, and priorities of these initial vector candidates are not only in the vicinity of the block of interest Bt but also slightly apart so that a reliable initial candidate vector can be obtained with as few candidates as possible. These blocks are set on the basis of spatial correlation and temporal correlation so that there is no bias. Therefore, using the motion vector of a predetermined block set in advance as an initial candidate vector in this way prevents the number of initial candidate vectors from increasing and the amount of computation of processing from becoming enormous. .

  Note that the types of candidate blocks and the positions of the candidate blocks are not limited to the example of FIG. 23, and other types of vectors and block positions may be used. In consideration of the case where the optimum vector does not exist in the peripheral blocks, a zero vector that suppresses deterioration in such a case may be used. Also, the number of candidate blocks may be more or less than eight.

  Next, the details of the initial vector selection process will be described with reference to the flowchart of FIG.

  In step S251, the candidate vector position calculation unit 251 determines the position of the candidate block, the type of the initial candidate vector, and the priority order from which the initial candidate vector of the target block set in advance is acquired from the peripheral area of the selected target block. In step S252, it is determined whether or not the type of the initial candidate vector of the candidate block is a past vector or a current vector in the order of the obtained priority. In step S252, when the candidate vector position calculation unit 251 determines that the type of the initial candidate vector of the candidate block is the past vector or the current vector, the process proceeds to step S253, where the candidate block position information and the type of initial candidate vector are determined. Information is supplied to the detection vector acquisition unit 252, and the motion vector (past vector PV or current vector CV) according to the position information of the candidate block and the type information of the initial candidate vector is detected in the detection vector memory. 53, the acquired motion vector is output to the offset position calculation unit 254, and the process proceeds to step S257.

  In step S252, when the candidate vector position calculation unit 251 determines that the type of the initial candidate vector of the candidate block is not a past vector or a current vector, the process proceeds to step S254, where the candidate vector position calculation unit 251 It is determined whether or not the type of candidate vector is a shifted initial vector. In step S254, if the candidate vector position calculation unit 251 determines that the type of the initial candidate vector of the candidate block is a shifted initial vector, the process proceeds to step S255, where the candidate block position information and the initial candidate vector type information are obtained. The shifted initial vector acquisition unit 253 supplies the shifted initial vector acquisition unit 253 with the shifted initial vector corresponding to the position information of the candidate block from the shifted initial vector memory 107, and the acquired shifted initial vector is offset. The position calculation unit 254 is output, and the process proceeds to step S257.

In step S254, when the candidate vector position calculation unit 251 determines that the type of the initial candidate vector of the candidate block is not the shifted initial vector (that is, determines that the type of the initial candidate vector of the candidate block is the 0 vector). In step S256, a zero vector is set as the initial candidate vector, and the position information of the candidate block is supplied to the offset position calculation unit 254 together with the zero vector, and the process proceeds to step S257. Also in steps S253 and S255, the candidate vector position calculation unit 251 supplies the position information of the candidate block to the offset position calculation unit 254.

  In step S257, the offset position calculation unit 254 receives the initial candidate vector from the detection vector acquisition unit 252 or the shifted initial vector acquisition unit 253, and based on the position information of the candidate block supplied by the candidate vector position calculation unit 251, For each initial candidate vector, an offset destination block position obtained by offsetting the target block of frame t to frame t + 1 is calculated. Then, the offset position calculation unit 254 outputs the position information of the candidate block and the information of the offset destination block position together with the initial candidate vector to the evaluation value calculation unit 255, and proceeds to step S258.

  When the evaluation value calculation unit 255 inputs the position information of the candidate block and the information of the offset destination block position together with the initial candidate vector from the offset position calculation unit 254, in step S258, using the frame t and the frame t + 1, the initial candidate An evaluation value DFD of the vector is obtained, and the obtained evaluation value DFD is output to the evaluation value comparison unit 256 together with the initial candidate vector, and the process proceeds to S259.

  In step S259, the evaluation value comparison unit 256 determines whether or not the evaluation value DFD obtained by the evaluation value calculation unit 255 is smaller than the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 257. The evaluation value DFD obtained by the evaluation value calculator 255 is smaller than the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 257, that is, the initial candidate vector is more reliable than the optimal candidate vector. Is determined to be high, in step S260, the optimal candidate vector of the optimal candidate storage register 257 and its evaluation value DFD are rewritten with the initial candidate vector and its evaluation value DFD that are determined to have high reliability, and the process returns to step S261. move on.

  In step S259, the evaluation value comparison unit 256 determines that the evaluation value DFD obtained by the evaluation value calculation unit 255 is not smaller than the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 257. If so, the process of step S260 is skipped, and the process proceeds to step S261.

  In step S261, the candidate vector position calculation unit 251 determines whether or not the processing of all the initial candidate vectors (8 vectors in the case of FIG. 23) is finished, and the processing of all the initial candidate vectors is finished. If it is determined that it is not, the process returns to step S252 and the subsequent processing is repeated.

  When it is determined in step S261 that all the initial candidate vectors have been processed, the process proceeds to step S262, and the evaluation value comparison unit 256 controls the optimal candidate storage register 257 to control all the initial candidate vectors. The optimal candidate vector having the highest reliability based on the evaluation value DFD is output to the iterative gradient method computing unit 103 as the initial vector V0, and the initial vector selection process is terminated.

  As described above, in the target block, the evaluation value DFD of a plurality of initial candidate vectors is obtained, and the initial candidate vector having the smallest evaluation value DFD, that is, the highest reliability is selected as the initial vector. Therefore, it is possible to provide an optimal initial vector for the subsequent motion vector detection, and as a result, it is possible to improve the accuracy of the subsequent motion vector detection. In addition, since a predetermined motion vector of a predetermined block is used as an initial candidate vector, the number of initial candidate vectors is increased, and the amount of calculation of processing is suppressed from becoming enormous.

  Furthermore, based on the fact that there is a certain degree of continuity in the amount of movement of moving objects between consecutive frames and the change in the amount of movement is small, an initial shift vector that is a motion vector that passes through the block of interest from the previous frame Since the vector candidates are used, it is possible to perform motion detection with higher accuracy than in the case where only the motion vectors obtained in the past in the peripheral blocks are used as the initial vector candidates. This is particularly effective at the boundaries of moving objects.

  Next, details of the configuration of the iterative gradient method computing unit 103 will be described.

  FIG. 25 is a block diagram showing a configuration of the iterative gradient method computing unit 103. The iterative gradient method computing unit 103 having the configuration shown in FIG. 25 performs processing for detecting an optimal motion vector using the frame t of the image at time t and the frame t + 1 of the image at time t + 1. The process of detecting the motion vector is a process executed for each predetermined block composed of a plurality of pixels, and the iterative gradient method computing unit 103 performs gradients for each block or for each pixel in each block. By repeatedly executing the calculation using the method, an optimal motion vector with high reliability based on the evaluation value DFD is output.

  The iterative gradient method computing unit 103 receives the frame t of the image at time t and the frame t + 1 of the image at time t + 1 via the pre-filters 102-1 and 102-2. The initial vector V0 is input.

  The mode selection unit 401 selects a gradient method calculation processing mode for each predetermined block under the control of the effective pixel determination unit 403, and selects the initial vector V0 from the initial vector selection unit 102 with the selector 402 and the vector evaluation. Output to the unit 104. The gradient method processing mode consists of a block unit processing mode in which the motion vector detection target is a block and a pixel unit processing mode in which the motion vector detection target is a pixel, and the block unit processing mode is selected as an initial value. The Under the control from the vector evaluation unit 104, the selector 402 uses either the initial vector V0 from the mode selection unit 401 or the motion vector Vn output from the gradient method calculation unit 404 as an initial value of the gradient method calculation. The motion vector to be used (hereinafter referred to as an offset vector) is output to the effective pixel determination unit 403 and the gradient method calculation unit 404.

  When the effective pixel determination unit 403 receives the offset vector selected by the selector 402, the frame t of the image at time t and the frame t + 1 of the image at time t + 1 input via the pre-filters 102-1 and 102-2. , Using the position calculated by offsetting the offset vector as a starting point, it is determined whether or not the number of pixels effective for the calculation of the gradient method is greater than the threshold in the calculation block of each unit processing mode, The processing method is switched according to the result. That is, when the effective pixel determination unit 403 determines that the number of pixels effective for the calculation of the gradient method is equal to or less than the threshold value, the effective pixel determination unit 403 controls the mode selection unit 401 and sets the processing mode for the predetermined block. Then, change to the pixel unit processing mode or stop the calculation. When the effective pixel determination unit 403 determines that the number of pixels effective for gradient method calculation is greater than the threshold value in the calculation block of each unit processing mode, the effective pixel determination unit 403 controls the gradient method calculation unit 404 to The gradient method calculation process is executed.

  When the gradient vector calculation unit 404 receives the offset vector input from the selector 402, the frame t of the image at time t and the frame t + 1 of the image at time t + 1 input via the prefilters 102-1 and 102-2. Using the offset vector from the selector 402 as a starting point, the gradient method calculation of each unit processing mode is executed, and the calculated motion vector Vn is used as the delay unit 405 and the vector evaluation unit 104. Output to.

  The delay unit 405 holds the motion vector Vn output from the gradient method calculation unit 404 until the next processing cycle of the effective pixel determination unit 403 and the gradient method calculation unit 404, and according to the control of the selector 402, The held motion vector Vn is output to the effective pixel determination unit 403.

  FIG. 26 is a block diagram illustrating a detailed configuration of the effective pixel determination unit 403. In the example of FIG. 26, the effective pixel determination unit 403 includes a temporal pixel difference calculation unit 411, a pixel difference value determination unit 412, an effective pixel counter 413, and a gradient method continuation determination unit 414.

  When the offset vector selected by the selector 402 is input, the temporal pixel difference calculation unit 411 selects a pixel in an arithmetic block in units of blocks, and inputs the offset vector and the pre-filters 102-1 and 102-2. The pixel difference Δt in the time direction of the selected pixel is calculated using the frame t of the image at time t and the frame t + 1 of the image at time t + 1, and the calculated pixel difference Δt in the time direction is calculated as the pixel difference. The data is output to the value determination unit 412.

  The pixel difference value determination unit 412 determines whether or not the pixel difference Δt in the time direction calculated by the time pixel difference calculation unit 411 is smaller than a predetermined threshold (hereinafter referred to as a pixel difference value). When it is determined that the pixel difference Δt in the time direction is smaller than the predetermined pixel difference value, the number of effective pixels of the effective pixel number counter 413 is counted by one. The effective pixel number counter 413 counts the value of pixels determined to be effective by the pixel difference value determination unit 412 for each calculation block. The value of the effective pixel number counter 413 is reset for each calculation block.

  The gradient method continuation determination unit 414 determines whether or not the number of effective pixels for the gradient method calculation is greater than a threshold value in each unit processing mode calculation block, and the unit processing mode calculation block includes a gradient. If it is determined that the number of pixels effective for the arithmetic operation is greater than the threshold value, a flag (flg = 1) that causes the gradient operation unit 404 to execute the gradient operation for each unit processing mode is output. If the gradient method continuation determination unit 414 determines that the number of pixels effective for gradient method calculation is less than the threshold in the calculation block of each unit processing mode, the gradient method continuation determination unit 414 controls the mode selection unit 401 to perform other unit processing. A mode is selected, or a flag (flg = 0) for stopping the gradient method calculation is output to the gradient method calculation unit 404.

  FIG. 27 is a block diagram showing a detailed configuration of the gradient method computing unit 404. In the example of FIG. 27, the gradient method calculation unit 404 includes a time pixel difference calculation unit 421, a pixel difference value determination unit 422, a horizontal / vertical pixel difference calculation unit 423, a gradient integration unit 424, and a vector calculation unit 425.

  The temporal pixel difference calculation unit 421 controls each unit of the gradient method calculation unit 404 based on the flag input from the effective pixel determination unit 403. In other words, when the flag indicates 1 (flg = 1), the temporal pixel difference calculation unit 421 causes each unit of the gradient method calculation unit 404 to execute gradient method calculation processing. Specifically, when the time pixel difference calculation unit 421 receives a flag from the effective pixel determination unit 403, the frame t of the image at time t input via the prefilters 102-1 and 102-2, and the time t + 1 The pixel in the calculation block of each unit processing mode is selected around the pixel calculated by offsetting the offset vector from the selector 402 using the frame t + 1 of the image of the selected pixel, and the pixel in the time direction of the selected pixel is selected. The difference Δt is calculated, and the calculated pixel difference Δt and offset vector in the time direction are output to the pixel difference value determination unit 422. Also, the time pixel difference calculation unit 421 outputs the offset vector to the vector calculation unit 425 when the processing of the pixels in the calculation block is completed, and controls the vector calculation unit 425 to obtain the motion vector of the detection target block. Let it be calculated.

  On the other hand, when the flag indicates 0 (flg = 0), the time pixel difference calculation unit 421 does not calculate the pixel difference Δt in the time direction of the pixels in the calculation block. The gradient method calculation process is not executed, and the process is terminated. At this time, the temporal pixel difference calculation unit 421 controls the vector calculation unit 425 to set the motion vector V to 0 vector.

  The pixel difference value determination unit 422 determines whether or not the pixel difference Δt in the time direction calculated by the time pixel difference calculation unit 421 is smaller than a predetermined threshold (hereinafter referred to as a pixel difference value). When it is determined that the pixel difference Δt in the time direction is smaller than the predetermined pixel difference value, the pixel is subjected to the gradient method calculation, and the pixel difference Δt in the time direction and the offset vector are input to the horizontal / vertical pixel difference calculation unit 423. Output. That is, the pixel of the pixel difference Δt determined to be smaller than the predetermined pixel difference value is set as a calculation target in the horizontal / vertical pixel difference calculation unit 423 and the gradient integration unit 424. In addition, when the pixel difference value determination unit 422 determines that the pixel difference Δt in the time direction is equal to or greater than a predetermined pixel difference value, the pixel difference value determination unit 422 causes the horizontal / vertical pixel difference calculation unit 423 and the gradient integration unit 424 to process the pixel. Prohibit.

  When the horizontal / vertical pixel difference calculation unit 423 receives the offset vector input from the pixel difference value determination unit 422, the frame t of the image at time t input through the prefilters 102-1 and 102-2, and the time The pixel difference Δt in the time direction is smaller than a predetermined pixel difference value in the calculation block of each unit processing mode centered on the pixel calculated by offsetting the offset vector using the frame t + 1 of the t + 1 image. The pixel difference Δx in the horizontal direction and the pixel difference Δy in the vertical direction of the pixel determined to be calculated are calculated. The horizontal / vertical pixel difference calculation unit 423 outputs the pixel difference Δt in the time direction, the pixel difference Δx in the horizontal direction, and the pixel difference Δy in the vertical direction to the gradient integration unit 424. The gradient integration unit 424 integrates the gradients of pixels that are determined to have a pixel difference Δt in the time direction smaller than a predetermined pixel difference value. That is, the gradient integrating unit 424 is configured to calculate the temporal pixel difference Δt calculated by the temporal pixel difference calculating unit 421, the horizontal pixel difference Δx calculated by the horizontal / vertical pixel difference calculating unit 423, and the vertical pixel difference. Δy is integrated, and the integrated gradient value is output to the vector calculation unit 425.

  When the offset vector is input from the time pixel difference calculation unit 421, the vector calculation unit 425 uses the value of the gradient accumulated by the gradient accumulation unit 424 and the least square sum of Equation (9) described above to use the motion vector. vn is calculated. Further, the vector calculation unit 425 adds the offset vector from the temporal pixel difference calculation unit 421 to the calculated motion vector vn to obtain a motion vector Vn, and outputs it to the vector evaluation unit 104 and the delay unit 405.

  Next, the motion vector detection target block and the calculation block will be described. When considering the movement of an object in an image, the same object often moves the same. Therefore, the unit of detection processing for detecting a motion vector is usually performed in units of blocks as shown in FIG.

  In the example of FIG. 28, the arrow X indicates the horizontal direction, and the arrow Y indicates the vertical direction. In addition, an arrow T indicates the direction of time passage from the frame t at the time t in the back right to the frame t + 1 at the time t + 1 in the left front.

  In the case of the example of FIG. 28, an 8 × 8 pixel block (hereinafter referred to as an 8 × 8 block) is shown as the detection target block K of each frame. The detection target block K is a block in which the same motion is obtained for the 8 × 8 block when the motion vector is detected for the 8 × 8 block.

  On the other hand, motion vector detection (that is, calculation) by the gradient method has a feature that a motion vector can be detected for each pixel. However, when the motion vector is calculated in units of pixels, the gradient method using the least square method, which is a statistical solution, has a drawback that the detection accuracy is significantly reduced. Therefore, in general, when calculating a motion vector not only for each block but also for each pixel, the pixels used for the calculation of the gradient method are configured in units of blocks (calculation blocks). There are many.

  As described above, the pixels used for the calculation of the gradient method are the pixels of the calculation block of 8 × 8 blocks. That is, in the example of FIG. 28, the motion vector obtained as a result of the gradient method computation using the computation block (8 × 8) is the motion corresponding to all the pixels of the detection target block K (8 × 8). It will be a vector.

  However, as described above, as a problem that occurs when the pixels used for the calculation of the gradient method are handled in units of calculation blocks as described above, different motions are generated in the pixels constituting the detection target block as shown in FIG. An example is a case where pixels of objects that are held are mixed.

  In the example of FIG. 29, in the detection target block (= calculation block) K composed of 8 × 8 pixels (64 pixels), 28 pixels (thick line circles) from the upper left pixel to the diagonally upper left indicated by the arrow A. In the 8 × 8 detection target block K, 36 pixels (thin line circles) from the lower right pixel have a rightward movement indicated by an arrow B.

  Therefore, in the detection target block K, the assumption that the movements are almost the same as described above is lost, and if the calculation of the gradient method is performed using a calculation block that includes pixels with different movements, the movement is remarkably increased. The vector detection accuracy is reduced. That is, this is a problem of a decrease in detection accuracy regarding the disappearance region (covered background region) and the generation region (uncovered background region) of the object that occurs at the boundary between objects having different movements. The disappearance region is a mixed region at a position corresponding to all end portions in the advancing direction of the foreground object with respect to the foreground region, and is a region where the background component is covered with the foreground as time passes. On the other hand, the generation region is a mixed region at a position corresponding to the rear end of the foreground object in the traveling direction with respect to the foreground region, and refers to a region where a background component appears as time passes.

  FIG. 30 is a diagram for explaining the luminance state of the detection target block in the case of the same object. On the left side in the figure, the arrow X indicates the horizontal direction, and the arrow Y indicates the vertical direction. In addition, an arrow T indicates the direction of time passage from the frame t at the time t in the back right to the frame t + 1 at the time t + 1 in the left front. On the right side in the figure, the arrow X indicates the horizontal direction, and the arrow L indicates the luminance value. That is, the luminance values Lp1, Lp2, Lp3,..., Lpi (3 <i <7),..., Lp7 on the right side in the figure are, for example, the upper left in the 8 × 8 detection target block of the frame t on the left side in the figure. .., Pi (3 <i <7),..., P7,..., And similarly, the luminance values Lq1, Lq2, and Lq3. ,..., Lqi (3 <i <7),..., Lq7 are pixels p1, p2, p3,..., Pi (3 <i <7) in the 8 × 8 detection target block in the left frame t + 1 in the drawing. ,..., P7,... Are pixels located in the same phase as the pixels q1, q2, q3,..., Qi (3 <i <7),. Represents the luminance value. The arrows between the luminance values Lp1 to Lp7 and Lq1 to Lq7 indicate the time direction difference Δt between the pixel on the frame t and the pixel on the frame t + 1.

  Therefore, in the example of FIG. 30, the time direction difference Δt between the pixel p1 and the pixel q1 is represented by the difference between the luminance value Lp1 and the luminance value Lq1, and the time direction difference Δt between the pixel p2 and the pixel q2 is the luminance value Lp2. The time direction difference Δt between the pixel p3 and the pixel q3 is represented by the difference between the luminance value Lp3 and the luminance value Lq3, and the time direction difference Δt between the pixel pi and the pixel qi is represented by the luminance value Lpi. The time direction difference Δt between the pixel p7 and the pixel q7 is represented by the difference between the luminance value Lp7 and the luminance value Lq7, and each luminance value difference is a curve connecting the luminance values Lp1 to Lp7. And, the change in luminance is small so that the shape of the curve connecting the luminance values Lq1 to Lq7 is shown as substantially the same shape.

  As described above, when the pixels in the detection target block are the same object (that is, the same motion), the luminance change is not so large for the corresponding pixel in the time direction.

  FIG. 31 is a diagram illustrating the luminance state of a detection target block when pixels of objects having different movements are mixed. In FIG. 30, portions corresponding to those in FIG. 31 are denoted by the corresponding reference numerals, and the description thereof will be omitted to avoid repetition. However, in the example of FIG. 31, the pixel p7 on the frame t is , Has a different motion from other pixels.

  For example, when the pixel p7 on the frame t has the same movement as other pixels, the luminance value of the pixel q7 located in the same phase as the pixel p7 on the frame t + 1 is the luminance value Lq7-1. In the example of FIG. 31, the pixel p7 has a different motion from the other pixels. Therefore, the luminance value of the pixel q7 is represented by the luminance value Lq7-2. The luminance value Lq7-1 is greatly changed to the lower side in the figure. Therefore, the time direction difference Δt (the difference between the luminance value Lp7 and the luminance value Lq7-2) between the pixel p7 and the pixel q7 becomes large.

  Thus, for example, when pixels of an object having different movements are mixed in the detection target block due to the disappearance, generation, or deformation of the above-described object, the luminance value of the corresponding pixel greatly changes. This often leads to disturbances in movement. Therefore, in the effective pixel determination unit 403, a pixel with a small luminance change is effective even when used for motion vector detection, and conversely, a pixel with a large luminance change is determined to be ineffective when used for motion vector detection. Like that. That is, paying attention to the factor of the brightness change, and determining whether the pixel is valid or invalid by the brightness change, the pixel having a small brightness change is used for motion vector detection. Conversely, the brightness change is large. Pixels can be prevented from being used for motion vector detection. Thereby, the accuracy of motion vector detection can be increased.

  Specifically, the effective pixel determination unit 403 sets a predetermined pixel difference value (threshold value) for the pixel difference Δt in the time direction for each pixel in the calculation block, and the pixel difference Δt in the time direction is When the pixel difference value is smaller than the set predetermined pixel difference value, it is determined that the pixel contributes (is effective) to motion vector detection, and is used for the calculation of the gradient method when detecting the motion vector. If the pixel difference Δt in the direction is greater than or equal to the set predetermined pixel difference value, the pixel is excluded from the gradient method calculation when detecting a motion vector because there is a risk of disturbing motion. To do. Thereby, mixing of different motions is suppressed, and more stable gradient method calculation is executed. As a result, a probable motion vector is detected.

  However, as described above, since the gradient method calculation is based on the least square method, which is a statistical solution, it is used for the gradient method calculation by separating the pixels used for the gradient method calculation from the pixels not used. If the number of pixels is extremely small and the gradient method calculation is performed with these extremely small number of pixels, the reliability of the calculation is lowered, and the accuracy of motion vector detection may be reduced. is there.

  Therefore, when the number of pixels used for the gradient method calculation is small, the effective pixel determination unit 403 determines that the calculation in the processing unit of this detection target block (8 × 8) is unstable and is shown in FIG. As described above, the processing is performed by switching to the calculation unit for each pixel (64 pixels) in the 8 × 8 block.

  FIG. 32 shows an example of a calculation block in pixel unit processing. In the example of FIG. 32, the calculation block E of the pixel p at the upper left of the detection target block K when the detection target block (block-based calculation block) K (8 × 8) on the frame is switched to pixel-unit processing. (9 × 9 in the example of FIG. 32) is shown.

  That is, when the motion vector of the pixel p is obtained, a calculation block E (9 × 9 in the example of FIG. 32) centered on the pixel p is used. For example, when obtaining a motion vector of a pixel right next to the pixel p, an arithmetic block 9 × 9 (not shown) centered on the pixel right next to the pixel p is used.

  Therefore, for example, even if most of the detection target block K moves differently from the pixel p, in the detection target block K (8 × 8), the number of pixels used in the gradient method calculation is counted, and the number is When the number is small, the process is switched to the pixel unit process, and as shown in FIG. 32, the calculation of the gradient method can be executed using the calculation block E in the pixel unit process.

  As described above, when the pixels used for the gradient method calculation are separated from the pixels used for the gradient method calculation, and the number of pixels used for the gradient method calculation becomes extremely small, the motion vector for all the pixels in the detection target block K (8 × 8) block. Instead of stopping the detection of pixels, the gradient method calculation is performed by changing the pixel of the calculation block E (9 × 9) for each pixel, thereby mixing pixels that are not used in the calculation. As a result, motion vectors can be detected with higher accuracy depending on the pixel.

  FIG. 33 is a diagram illustrating an example in which pixel unit processing is applied to the pixels of the detection target block (arithmetic block) K in FIG. In the example of FIG. 33, the pixel has a pixel having a diagonally upper left movement indicated by arrow A (thick line circle) and a pixel having two different movements indicated by an arrow B (thin line circle). In the detection target block K (8 × 8) in FIG. 29, processing in units of pixels is performed. In this case, when detecting the motion vector of the detection target pixel p at the upper left of the detection target block K, the calculation block E (9 × 9 in the example of FIG. 33) centering on the detection target pixel p is used.

  The pixels in the computation block E have a diagonally upper left movement indicated by an arrow A except for the lower right three pixels (thin line circles) having a horizontal movement indicated by an arrow B. Here, as in the case of the block unit processing described above, for each pixel in the computation block E, a predetermined threshold value (pixel difference value) is set for the time direction pixel difference Δt, and the time direction pixel difference Δt is When the pixel difference value is smaller than the set predetermined pixel difference value, it is determined that the pixel contributes (is effective) to motion vector detection, and is used for gradient method calculation when detecting a motion vector. When the pixel difference Δt is larger than the set predetermined pixel difference value, the pixel is not used for the gradient method calculation when detecting a motion vector because it may cause a disturbance of motion.

  That is, in the example of FIG. 33, in the calculation block E, the pixels (thick line circles) excluding the lower right three pixels (thin line circles) are considered to be effective pixels for the gradient method calculation of the detection target pixel p. Since the effective pixel is used and the gradient method calculation is executed, the detection target block K (8 × 8) in FIG. 29 is more than the gradient method calculation of the block unit processing executed using the calculation block. A stable gradient method operation is performed and, as a result, a probable motion vector is detected.

  Even in the calculation block for each pixel, if the pixels used for the gradient method calculation are extremely reduced by separating the pixels used for the gradient method calculation from those not used, a motion vector is detected. The operation is aborted. That is, even if the motion vector detection is unstable, if a motion having some size is detected, the motion vector does not match the motion of the object in the actual image (not sure). Because there is a risk of becoming. Subsequent processing (for example, vector allocation processing or vector compensation processing) is executed after using such uncertain motion vectors, and the generated image may have motion discontinuities or block noise. In many cases, it causes a large visual deterioration such as the occurrence of the above.

  Accordingly, when the number of pixels used for pixel-based gradient method calculation becomes extremely small, for example, a zero vector (that is, a stationary state) is used as the detected motion vector. Thereby, in the subsequent processing, this motion vector is suppressed from having an adverse effect, and large visual deterioration of the generated image can be suppressed.

  Next, details of the iterative gradient method calculation processing will be described with reference to the flowchart of FIG. The initial vector V0 is input to the mode selection unit 401 from the previous stage.

  In step S401, the mode selection unit 401 selects a block unit processing mode, sets the block on the frame t as a detection target block (calculation block), and sets the initial vector V0 of the detection target block to the selector 402 and the vector evaluation unit 104. Output, and proceed to step S402. As a result, each unit of the iterative gradient method computing unit 103 executes processing in units of blocks. In step S402, the selector 402 selects the initial vector V0 input from the mode selection unit 401 as an offset vector in accordance with control from the vector evaluation unit 104, and selects the selected offset vector as a time pixel difference calculation unit 411 and The result is output to the time pixel difference calculation unit 421, and the process proceeds to step S403.

  In step S403, the temporal pixel difference calculation unit 411 and the pixel difference value determination unit 412 execute the effective pixel determination process in units of blocks using the selected offset vector (in this case, the initial vector V0 is offset). Then, the process proceeds to step S404. The details of this effective pixel determination process will be described later with reference to FIG. 35. With this effective pixel determination process, the effective pixel number counter 413 has a pixel difference value of a predetermined pixel difference within a block-by-block calculation block. The number of pixels determined to be smaller than the value (that is, determined to be an effective pixel in the subsequent gradient method calculation) is counted.

  In step S404, the gradient method continuation determination unit 414 determines whether or not the number of pixels counted by the effective pixel number counter 413 (effective pixel number) is greater than a predetermined threshold value α, and the effective pixel number is predetermined. If it is determined that the value is larger than the threshold value α, a flag (flg = 1) for executing the gradient method calculation is output to the gradient method calculation unit 404 for each block, and the process proceeds to step S405.

  In step S405, the gradient method calculation unit 404 uses the offset vector from the selector 402 (in this case, the initial vector V0) based on the flag from the gradient method continuation determination unit 414, and performs gradient method calculation processing in units of blocks. Is executed, and the process proceeds to step S406. The details of the gradient method calculation processing will be described later with reference to FIG. 36, but the motion vector Vn is obtained by this gradient method calculation processing and is output to the vector evaluation unit 104 and the delay unit 405. The motion vector Vn is obtained by adding the offset vector Vn−1 to the motion vector vn calculated by the first gradient method calculation (Vn = Vn−1 + vn). For example, the motion vector V1 is obtained by adding an offset vector (initial vector V0) to the motion vector v1 calculated by the first gradient method calculation (V1 = V0 + v1).

  In step S406, the vector evaluation unit 104 evaluates the motion vector Vn obtained by the gradient method calculation process DFD (n) and the evaluation value DFD (n-1) of the motion vector Vn-1 used as the offset vector. , And whether or not the evaluation value DFD (n) is smaller than the evaluation value DFD (n−1), that is, the motion vector obtained by the gradient method calculation processing than the motion vector Vn−1 used as the offset vector It is determined whether or not the reliability of Vn is high. Specifically, for example, in the first process, the obtained evaluation value DFD (1) of the motion vector V1 is compared with the evaluation value DFD (0) of the initial vector V0 used as an offset vector. In the second processing, the obtained evaluation value DFD (2) of the motion vector V2 is compared with the evaluation value DFD (1) of the motion vector V1 used as the offset vector.

  When the vector evaluation unit 104 determines in step S406 that the evaluation value DFD (n) is smaller than the evaluation value DFD (n-1), that is, the reliability of the motion vector Vn is higher than the motion vector Vn-1. In step S407, the calculated motion vector Vn (in the first case, the motion vector V1, in the second case, the motion vector V2) is set as the motion vector V of the detection target block. The process proceeds to S408. In step S408, the vector evaluation unit 104 counts the number of iterations n by one, proceeds to step S409, determines whether or not the number of iterations n has reached the set maximum number of iterations (for example, two times), and If it is determined that the number of iterations n is not the set maximum number of iterations, the selector 402 is controlled to return to step S402 to repeat the subsequent processing.

  That is, in step S402, the selector 402 selects the motion vector V1 output from the vector calculation unit 425 and held in the delay unit 405 as an offset vector, and in step S403, the time pixel difference calculation unit 411 and the pixel difference value are selected. The determination unit 412 executes the effective pixel determination process in units of blocks using the motion vector V1 as an offset vector, and repeats the subsequent processes. Therefore, the motion vector V2 calculated in step S405 is obtained by adding the motion vector V1 used as the offset vector to the motion vector v2 calculated by the second gradient method calculation (that is, V2 = V1 + v2 = V0 + v1 + v2). )

  When the vector evaluation unit 104 determines in step S406 that the evaluation value DFD (n) is not smaller than the evaluation value DFD (n−1), that is, the motion vector Vn−1 is more than the motion vector Vn. If it is determined that the reliability is high, the process proceeds to step S410, and the vector Vn−1 (the initial vector V0 in the first time, the motion vector V1 in the second time) is used as the offset of the gradient method calculation. The motion vector V is set and the process proceeds to step S411. If the vector evaluation unit 104 determines in step S409 that the number of iterations n has reached the set maximum number of iterations, the vector evaluation unit 104 proceeds to step S411.

  In step S411, the vector evaluation unit 104 determines whether or not the motion vector V is within a search area preset as a range for detecting a motion vector, and determines that the motion vector V is within the search area. If so, the process proceeds to step S415, the motion vector V is stored in the detection vector memory 53 in association with the detection target block, and the iterative gradient method process is terminated.

  If the vector evaluation unit 104 determines in step S411 that the motion vector V is not within the search area, the vector evaluation unit 104 proceeds to step S412, sets the motion vector V to 0 vector, proceeds to step S415, and proceeds to step S415. (0 vector) is stored in the detection vector memory 53 in association with the detection target block, and the iterative gradient method processing is terminated.

  On the other hand, if the gradient method continuation determination unit 414 determines in step S404 that the effective pixel number of the effective pixel number counter 413 is equal to or less than the predetermined threshold value α, the process proceeds to step S413 and controls the mode selection unit 401. The pixel unit processing mode is selected, and the process proceeds to step S414. As a result, each unit of the iterative gradient method computing unit 103 executes processing in units of pixels. In step S414, the mode selection unit 401 executes the iterative gradient method calculation process for each pixel, and the process proceeds to step S415. The details of the iterative gradient method computing process in the pixel unit processing mode will be described later with reference to FIG. 37, but the motion vector V of all pixels in the detection target block is obtained by the iterative gradient method computing process in the pixel unit processing mode. Therefore, in step S415, the vector evaluation unit 104 stores the motion vector V in the detection vector memory 53 in association with each pixel of the detection target block, and ends the iterative gradient method processing.

  In step S406 described above, the reliability of the motion vector is evaluated by evaluating the evaluation value DFD (n) of the motion vector Vn obtained by the gradient method processing and the motion vector Vn-1 used as the offset vector. Although the determination is made by comparing the value DFD (n−1), simultaneously with the comparison determination of the evaluation value DFD, the motion vector (motion vector before adding the offset vector) vn obtained by the gradient method calculation It may be determined that the value (| vn |) is smaller than a predetermined size (for example, 16 in the case of the calculation block 8 × 8).

  Here, the norm represents the magnitude of a vector in a certain vector space. For example, the norm of the vector v from the origin (0, 0) to (x, y) is expressed by the following equation (11). expressed.

・・・（１１）

(11)

  That is, in the gradient method calculation, a high-precision result is generally obtained for a minute motion, but the norm value | vn | of the motion vector obtained by the gradient method calculation exceeds a predetermined magnitude. In other words, a highly accurate result is not always obtained for the large motion vector vn. Accordingly, in parallel with the evaluation value DFD, it is determined whether or not the norm value | vn | of the motion vector obtained by the gradient method calculation exceeds a predetermined magnitude, and the norm value | vn | By excluding excess motion vectors, this gradient method can obtain highly accurate results for both small and large motions, and further improve the accuracy of motion vector detection. Can be made.

  In the case of the determination of the evaluation value DFD and the determination based on the norm value, in step S406, when | vn | ≦ 16, and evaluation value DFD (n−1)> evaluation value DFD (n). If there is, the process proceeds to step S407, where V = Vn is set, and if | vn |> 16 in step S406, or evaluation value DFD (n−1) ≦ evaluation value DFD (n). In this case, the process proceeds to step S410, and V = Vn−1 is set.

  Next, the details of the effective pixel determination process will be described with reference to the flowchart of FIG. FIG. 35 shows an example of the effective pixel determination process in step S403 of FIG.

  When the offset vector is input from the selector 402, the temporal pixel difference calculation unit 411 controls the pixel difference value determination unit 412 in step S431, resets the effective pixel number counter 413, proceeds to step S432, and proceeds to block unit. In the calculation block, a pixel is selected, and the process proceeds to step S433. The pixels are selected in the raster scan order from the upper left pixel of the calculation block.

  In step S433, the time pixel difference calculation unit 411 uses the offset vector and the frame t of the image at time t and the frame t + 1 of the image at time t + 1 input via the pre-filters 102-1 and 102-2. Then, the pixel difference Δt in the time direction of the selected pixel is calculated, and the calculated pixel difference Δt in the time direction is output to the pixel difference value determination unit 412 and the process proceeds to step S434.

  In step S434, the pixel difference value determination unit 412 determines whether or not the time direction pixel difference Δt calculated by the time pixel difference calculation unit 411 is smaller than a predetermined pixel difference value. It is determined whether or not the pixel is effective for the legal calculation, and if it is determined that the pixel difference Δt in the time direction is smaller than the predetermined pixel difference value, the process proceeds to step S435, and the effective pixel counter 413 determines the effective pixel number. The number is counted by one, and the process proceeds to step S436. Further, the pixel difference value determination unit 412 determines in step S434 that the time-direction pixel difference Δt is greater than or equal to a predetermined pixel difference value, that is, it is not a pixel that is effective for the subsequent gradient method calculation. Then, the process of step S435 is skipped, and the process proceeds to step S436.

  In step S436, the time pixel difference calculation unit 411 determines whether or not the processing of all the pixels in the calculation block has been completed, and determines that the processing of all the pixels in the calculation block has not ended. Returning to step S432, the next pixel is selected, and the subsequent processing is repeated. On the other hand, if the time pixel difference calculation unit 411 determines in step S436 that all the pixels in the calculation block have been processed, the time pixel difference calculation unit 411 ends the effective pixel determination process and returns to step S404 in FIG.

  Through the above processing, the effective pixel counter 413 counts the number of pixels in the calculation block whose pixel difference Δt is smaller than a predetermined pixel difference value and determined to be effective in the subsequent gradient method calculation. Yes. Therefore, the gradient method continuation determination unit 414 performs gradient method calculation in units of blocks or performs gradient method calculation in units of pixels by referring to the number of pixels of the effective pixel counter 413 in step S404 of FIG. Can be judged.

  Next, the details of the gradient method calculation processing in units of blocks will be described with reference to the flowchart of FIG. Note that FIG. 36 illustrates an example of the gradient method calculation processing in units of blocks in step S405 of FIG. Therefore, in the example of FIG. 36, the flag (flg = 1) for executing the gradient method calculation is input from the gradient method continuation determination unit 414 to the time pixel difference calculation unit 421.

  When the flag (flg = 1) is input from the gradient method continuation determination unit 414, the time pixel difference calculation unit 421 selects a pixel in the calculation unit in block units in step S451, and proceeds to step S452. The pixels are selected in the raster scan order from the upper left pixel of the calculation block.

  In step S452, the time pixel difference calculation unit 421 calculates the offset vector from the selector 402, the frame t of the image at time t input via the prefilters 102-1 and 102-2, and the image at time t + 1. The pixel difference Δt in the time direction of the selected pixel is calculated using the frame t + 1, and the calculated pixel difference Δt in the time direction and the offset vector are output to the pixel difference value determination unit 422, and the process proceeds to step S453.

  In step S453, the pixel difference value determination unit 422 determines whether or not the pixel difference Δt in the time direction calculated by the time pixel difference calculation unit 421 is smaller than a predetermined pixel difference value. It is determined whether or not the pixel is effective for legal calculation, and if it is determined that the pixel difference Δt in the time direction is smaller than the predetermined pixel difference value, the process proceeds to step S454 and the vertical horizontal pixel difference calculation unit 423 is controlled. Then, the pixel difference Δx in the horizontal direction of the pixel determined to be effective is calculated. Note that the predetermined pixel difference value in the pixel difference value determination unit 422 may be the same value as or different from the predetermined pixel difference value in the pixel difference value determination unit 412. At this time, the pixel difference value determination unit 422 outputs the pixel difference Δt in the time direction and the offset vector calculated by the time pixel difference calculation unit 421 to the vertical horizontal pixel difference calculation unit 423.

  In step S454, the vertical horizontal pixel difference calculation unit 423 calculates the offset vector and the frame t of the image at time t and the frame t + 1 of the image at time t + 1 input via the prefilters 102-1 and 102-2. The pixel difference Δx in the horizontal direction of the pixel determined to be effective is calculated, and the process proceeds to step S455, where the pixel difference Δy in the vertical direction of the pixel determined to be effective is calculated, and the pixel in the time direction is calculated. The difference Δt, the pixel difference Δx in the horizontal direction, and the pixel difference Δy in the vertical direction are output to the gradient integration unit 424, and the process proceeds to step S456. In step S456, the gradient integration unit 424 integrates the time-direction pixel difference Δt, the horizontal pixel difference Δx, and the vertical pixel difference Δy from the vertical horizontal pixel difference calculation unit 423, and calculates a vector result of the integration. The data is output to the unit 425, and the process proceeds to step S457.

  If the pixel difference value determination unit 422 determines in step S453 that the time-direction pixel difference Δt is greater than or equal to the predetermined pixel difference value, the pixel difference value determination unit 422 skips the processing of steps S454 to S456 and proceeds to step S457. That is, each pixel difference (gradient) of this pixel is not used for the calculation because it is not a pixel effective for the subsequent gradient method calculation.

  In step S457, the time pixel difference calculation unit 421 determines whether or not the processing of all the pixels in the calculation block has been completed, and determines that the processing of all the pixels in the calculation block has not ended. Returning to step S451, the next pixel is selected, and the subsequent processing is repeated. On the other hand, if the time pixel difference calculation unit 421 determines in step S457 that the processing of all the pixels in the calculation block has been completed, the time pixel difference calculation unit 421 outputs an offset vector to the vector calculation unit 425, and proceeds to step S458. When the offset vector is input from the time pixel difference calculation unit 421, the vector calculation unit 425 uses the gradient summation result from the gradient summation unit 424 and the least square sum of the above-described equation (9) to obtain the motion vector vn. Then, the process proceeds to step S459, the motion vector Vn is obtained by adding the calculated motion vector vn to the offset vector from the time pixel difference calculation unit 421, and the obtained motion vector Vn is sent to the vector evaluation unit 104. The gradient method calculation process is terminated, and the process returns to step S406 in FIG.

  As described above, when the pixel difference Δt in the time direction is equal to or larger than a predetermined pixel difference value, the gradient at the time of detecting a motion vector is assumed that the pixel may cause a disturbance in motion. Since it is excluded from the calculation of the method, a more stable gradient method calculation is executed, and as a result, a probable motion vector is detected, thereby improving the accuracy of the motion vector detection.

  Next, details of the iterative gradient method calculation processing in units of pixels will be described with reference to the flowchart of FIG. FIG. 37 shows an example of the iterative gradient method computing process for each pixel in step S414 of FIG. This process is executed for each pixel in the detection target block.

  When the pixel unit processing mode is set, the mode selection unit 401 selects the pixel of the detection target block as the detection target pixel in step S471, and selects the initial vector V0 of the calculation block (for example, 9 × 9) of the pixel as a selector. 402 and the vector evaluation unit 104, and the process proceeds to step S472. The pixels are selected in the raster scan order from the upper left pixel of the detection target block. In step S472, the selector 402 selects the initial vector V0 input from the mode selection unit 401 as an offset vector in accordance with the control from the vector evaluation unit 104, and selects the selected offset vector as a time pixel difference calculation unit 411 and The result is output to the time pixel difference calculation unit 421, and the process proceeds to step S473.

  In step S473, the temporal pixel difference calculation unit 411 and the pixel difference value determination unit 412 execute the effective pixel determination process in units of pixels using the selected offset vector (in this case, using the initial vector V0 as an offset). The process proceeds to step S474. The details of the effective pixel determination process are basically the same except that the effective pixel determination process in units of blocks described above with reference to FIG. 35 is different from the target calculation block (block range and pixels constituting the block). However, by this effective pixel determination process, the effective pixel number counter 413 has a pixel difference value smaller than a predetermined pixel difference value in the calculation unit in units of pixels. (That is, the number of pixels determined to be effective pixels in the subsequent gradient method calculation) is counted. It should be noted that the predetermined pixel difference value for determining the effective pixel in pixel units may be the same value as in the block unit or may be set to a different value.

  In step S474, the gradient method continuation determination unit 414 determines whether or not the number of pixels (effective pixel number) counted by the effective pixel number counter 413 is greater than a predetermined threshold value β, and the effective pixel number is predetermined. If it is determined that the value is larger than the threshold value β, a flag (flg = 1) for executing the gradient method calculation is output to the gradient method calculation unit 404, and the process proceeds to step S475. Note that the threshold value β may be the same value as the threshold value α in the case of a block unit, or a different value may be set.

  In step S475, based on the flag from the gradient method continuation determination unit 414, the gradient method calculation unit 404 uses the offset vector (initial vector V0) from the selector 402 to execute a gradient method calculation process for each pixel, The process proceeds to step S476. The details of the gradient method calculation process are basically the same as the gradient method calculation process in units of blocks described above with reference to FIG. 36 except that the target calculation block (range and pixels constituting the block) is different. Although detailed description thereof is omitted for the purpose of processing, a motion vector Vn of the detection target pixel is obtained by this gradient method calculation processing, and is output to the vector evaluation unit 104 and the delay unit 405. The motion vector Vn is obtained by adding the offset vector to the motion vector vn obtained by the first gradient method calculation (Vn = Vn−1 + vn).

  In step S476, the vector evaluation unit 104 obtains the evaluation value DFD (n) of the motion vector Vn calculated by the gradient method processing and the evaluation value DFD (n−1) of the motion vector Vn−1 used as the offset. , Whether the evaluation value DFD (n) is smaller than the evaluation value DFD (n−1), that is, the reliability of the motion vector Vn calculated by the gradient method processing than the motion vector Vn−1 used as the offset Judge whether or not is high.

  In step S476, the vector evaluation unit 104 determines that the evaluation value DFD (n) is smaller than the evaluation value DFD (n−1), that is, the reliability of the motion vector Vn is higher than the motion vector Vn−1. If it is determined that the value is high, in step S477, the calculated motion vector Vn (the first time, the motion vector V1, and the second time, the motion vector V2) is set as the motion vector V of the detection target pixel, The process proceeds to step S478. In step S478, the vector evaluation unit 104 counts the number of iterations n by one, proceeds to step S479, determines whether or not the number of iterations n has reached the set maximum number of iterations (for example, two times), and If it is determined that the number of iterations n is not the set maximum number of iterations, the selector 402 is controlled to return to step S472 to repeat the subsequent processing.

  That is, in step S472, the selector 402 selects the motion vector V1 output from the vector calculation unit 425 and held in the delay unit 405 as an offset vector, and in step S473, the temporal pixel difference calculation unit 411 and the pixel difference value determination The unit 412 offsets the motion vector V1, which is an offset vector, executes a valid pixel determination process in units of pixels, and repeats the subsequent processes. Therefore, the motion vector V2 obtained in step S475 is obtained by adding the offset motion vector V1 to the motion vector v2 calculated by the second gradient method calculation (that is, V2 = V0 + v1 + v2).

  In step S476, the vector evaluation unit 104 determines that the evaluation value DFD (n) is larger than the evaluation value DFD (n−1), that is, the motion vector Vn−1 is more reliable than the motion vector Vn. When it is determined that the degree is high, the process proceeds to step S480, and the vector Vn−1 (the initial vector V0 in the first case, the motion vector V1 in the second time) is used as the offset of the gradient method calculation. The motion vector V is set, and the process proceeds to step S481. If the vector evaluation unit 104 determines in step S479 that the number of iterations n has reached the set maximum number of iterations, the vector evaluation unit 104 proceeds to step S481.

  In step S481, the vector evaluation unit 104 determines whether or not the motion vector V is within a search area preset as a range for detecting a motion vector, and determines that the motion vector V is within the search area. If so, the process proceeds to step S483. If the vector evaluation unit 104 determines in step S481 that the motion vector V is not within the search area, the vector evaluation unit 104 proceeds to step S482, sets the motion vector V to 0 vector, and proceeds to step S483.

  On the other hand, in step S474, when the gradient method continuation determination unit 414 determines that the effective pixel number of the effective pixel number counter 413 is less than the predetermined threshold value β, the gradient method calculation unit 404 has a flag ( flg = 0) is output, and the flow advances to step S482. Based on the flag, the temporal pixel difference calculation unit 421 controls the vector calculation unit 425 to set the motion vector V to 0 vector, and the process proceeds to step S483.

  When the mode selection unit 401 determines in step S483 whether or not the processing of all the pixels in the detection target block has been completed, and determines that the processing of all the pixels in the detection target block has not been completed, Returning to step S471, the subsequent pixels of the detection target block are set as detection target pixels, and the subsequent processing is repeated. If the mode selection unit 401 determines in step S483 that the processing of all the pixels in the detection target block has been completed, the mode selection unit 401 ends the iterative gradient method calculation processing for each pixel, and returns to step S415 in FIG. That is, since the motion vector V is obtained for all the pixels in the detection target block by the iterative gradient method computing process in units of pixels, the process returns to step S415 in FIG. 34, and the vector evaluation unit 104 converts the motion vector V to the detection target block. Are stored in the detection vector memory 53 in correspondence with each of the pixels, and the iterative gradient method processing is terminated.

  Note that, in the evaluation of the reliability of the motion vector in step S476, the evaluation value DFD determination and the norm value determination may be used in combination, as in the case of step S406.

  As described above, by using only pixels whose pixel difference in the calculation block is smaller than the predetermined pixel difference value for the gradient method calculation, pixels having different motions can be excluded from the calculation target, and the gradient method calculation is performed. As a result, the certainty of the motion vector required by the method improves, and the accuracy of motion vector detection improves.

  In addition, when the pixel difference in the calculation block is larger than the predetermined pixel difference value and the number of pixels that are not used for the gradient method calculation is larger than the predetermined threshold value, the gradient method calculation processing is adaptively performed. Therefore, the accuracy of motion vector detection can be further improved especially at the boundary of the motion of the object, and as a result, the required motion vector can be made more probable.

  Specifically, when the pixel difference in the calculation block is larger than the predetermined pixel difference value and the number of pixels that are not used in the gradient method calculation is larger than the predetermined threshold value, the block unit mode is changed to the pixel unit. Since the calculation block is set for each pixel of the detection target block and the validity determination process and the gradient method calculation process are performed, the motion vector can be detected with higher accuracy depending on the pixel. In addition, when the pixel difference in the calculation block is larger than the predetermined pixel difference value and the number of pixels that are not used for the gradient method calculation is larger than the predetermined threshold value, the gradient method calculation processing is discontinued, Since the zero vector is used, detection of a violent motion vector can be suppressed and a safer motion vector can be detected.

  Next, details of the configuration of the vector allocation unit 54 will be described.

  FIG. 38 is a block diagram showing a configuration of the vector allocating unit 54. The vector allocating unit 54 having the configuration shown in FIG. 38 allocates motion vectors detected on the frame t using the frame t of the image at time t to which the 24P signal is input and the frame t + 1 of the image at time t + 1. A process of assigning to the pixels on the interpolation frame of the 60P signal to be interpolated on the vector memory 55 is performed.

  In the example of FIG. 38, the frame t of the image at time t and the frame t + 1 of the image at time t + 1 are input to the pixel information calculation unit 701, the evaluation value calculation unit 702, and the target pixel difference calculation unit 703.

  The pixel information calculation unit 701 acquires the motion vector detected by the pixel on the frame t in the detection vector memory 53 in the raster scan order from the upper left pixel, and extends the acquired motion vector in the frame t + 1 direction at the next time. The intersection of the extended motion vector and the interpolation frame is calculated. Then, the pixel information calculation unit 701 sets, from the intersection of the calculated motion vector and the interpolation frame, a pixel (hereinafter referred to as an allocation target pixel) to which the motion vector is assigned on the interpolation frame. The motion vector and the position information of the allocation target pixel are output to the vector selection unit 705. The image information calculation unit 701 calculates the position P of the frame t and the position Q on the frame t + 1 that are associated with the allocation target pixel and the motion vector, and the calculated position information on the frame t and the frame t + 1 The result is output to the evaluation value calculation unit 702 and the target pixel difference calculation unit 703.

  When the evaluation value calculation unit 702 receives from the pixel information calculation unit 701 position information on the frame t and the frame t + 1 associated with the allocation target pixel and the motion vector, the evaluation value calculation unit 702 receives the position P of the frame t and the position Q of the frame t + 1. In order to calculate the evaluation value DFD, a certain range of DFD calculation ranges (m × n) centering on the position P and the position Q are respectively set, and it is determined whether or not these DFD calculation ranges are within the image frame. . When the evaluation value calculation unit 702 determines that the DFD calculation range is within the image frame, the evaluation value calculation unit 702 calculates the evaluation value DFD of the allocation target pixel for the motion vector by calculating using the DFD calculation range, and the calculated evaluation value The DFD is output to the vector evaluation unit 704.

  When the pixel-of-interest difference calculation unit 703 receives from the pixel information calculation unit 701 position information on the frame t and the frame t + 1 associated with the allocation target pixel and the motion vector, the position P of the frame t and the position Q of the frame t + 1 Is used to obtain the absolute value of the luminance difference for the allocation target pixel, and the obtained absolute value of the luminance difference is output to the vector evaluation unit 704.

  The vector evaluation unit 704 includes a pixel difference determination unit 711 and an evaluation value determination unit 712. The pixel difference determination unit 711 determines whether the luminance difference absolute value for the allocation target pixel input from the target pixel difference calculation unit 703 is smaller than a predetermined threshold value. The evaluation value determination unit 712 calculates the evaluation value when the pixel difference determination unit 711 determines that the luminance difference absolute value for the allocation target pixel input from the target pixel difference calculation unit 703 is smaller than a predetermined threshold value. It is determined whether the evaluation value DFD of the allocation target pixel input from the unit 702 is smaller than the minimum evaluation value of the DFD table included in the vector selection unit 705. When the evaluation value determination unit 712 determines that the evaluation value DFD of the allocation target pixel is smaller than the minimum evaluation value of the DFD table, the evaluation value determination unit 712 determines that the reliability of the motion vector corresponding to the allocation target pixel is high, The evaluation value DFD of the allocation target pixel is output to the selection unit 705.

  The vector selection unit 705 has a DFD table that holds a minimum evaluation value at each pixel on the interpolation frame, and an evaluation value DFD0 when a 0 vector is assigned to each pixel on the interpolation frame. The DFD table holds the minimum evaluation value for each pixel on the interpolation frame in advance. When the vector selection unit 705 receives the evaluation value DFD of the allocation target pixel from the vector evaluation unit 704, the vector selection unit 705 sets the flag of the allocation flag memory 56 to 1 (based on the position information of the allocation target pixel from the pixel information calculation unit 701). true), and the minimum evaluation value in the DFD table of the allocation target pixel is rewritten to the evaluation value DFD of the allocation target pixel. Further, the vector selection unit 705 allocates the motion vector from the pixel information calculation unit 701 to the allocation target pixel in the allocation vector memory 55 based on the information on the position of the allocation target pixel from the pixel information calculation unit 701.

  Next, the accuracy below the pixel of the motion vector will be described. In the calculation of the DFD evaluation expressed by the above formula (1), the phase p + v on the previous frame t + 1 obtained by shifting the pixel position p of the frame t by the amount of the vector v is actually on the frame t + 1 of the 24p signal. In many cases, the luminance value is not defined. Therefore, in order to perform the calculation of the evaluation value DFD for the motion vector v having subpixel accuracy, it is necessary to generate a luminance value in a phase below the pixel by some method.

  Corresponding to this, there is a method of using the luminance value of the pixel closest to the phase p + v on the previous frame t + 1 obtained by shifting the pixel position p of the frame t by the vector v amount. However, in this method, since the sub-pixel components of the motion vector to be evaluated are rounded, the sub-pixel components of the motion vector are discarded, and the reliability of the evaluation value DFD thus obtained is lowered. End up.

  Therefore, in the present invention, a four-point interpolation process using the luminance values of the surrounding four pixels is used. FIG. 39 is a diagram showing the concept of the four-point interpolation process of the present invention. In FIG. 39, the arrow X indicates the horizontal direction in the frame t + 1, and the arrow Y indicates the vertical direction in the frame t + 1. In this frame t + 1, the white circle represents the pixel position on the frame t + 1, and the black dot represents the position below the pixel. Further, the upper left black point p + v and its surrounding four pixels on the frame t + 1 are shown enlarged in the window E. In window E, the alphabet in the white circle indicates the luminance value of the surrounding four pixels.

If the black point p + v at the upper left in the frame t + 1 is the phase p + v after the pixel position p of the frame t is shifted by the vector v amount, the luminance value F _{t + 1} (p + v) of the phase p + v is the horizontal direction of the phase p + v. The pixel sub-component α and the pixel sub-component β in the vertical direction and the luminance values L0 to L4 of the four surrounding pixels of the phase p + v are used to obtain the sum of the inverse ratios of the distances of the four surrounding pixels. That is, the luminance value F _{t + 1} (p + v) is expressed by the following equation (12).

F _{t + 1} (p + v) = (1-α) (1-β) L0 + α (1-β) L1
+ (1-α) βL2 + αβL3 (12)

As described above, the calculation of the DFD evaluation is performed using the luminance value F _{t + 1} (p + v) obtained by the four-point interpolation process, so that the reliability of the evaluation value DFD can be increased without increasing the hardware implementation cost. A decrease in the degree can be suppressed. In the following, an example in which the four-point interpolation is applied in the calculation of the evaluation value DFD and the luminance difference absolute value at the time of vector allocation will be described. Of course, the above-described initial vector selection processing, vector detection processing, Alternatively, this four-point interpolation is also applied to the calculation of the evaluation value DFD in the case of evaluating a vector such as allocation compensation processing described later.

  Next, the concept of vector allocation processing will be described. For example, when it is assumed that an object is moving at a velocity v on a 24P signal and a constant velocity assumption is made for the motion of the object between any two frames, a new frame is interpolated between the frames of the 24P signal. Think about what to do. In this case, when the motion vector v is extended from the 24P signal object, the intersection of the motion vector v and the interpolation frame is the same object and has the same velocity v.

  Therefore, the motion vector of the frame of the 24P signal detected by the vector detection unit 53 (hereinafter also referred to as an original frame with respect to the interpolation frame) is represented by the motion vector and the interpolation frame of the 60P signal to be interpolated. By assigning them to the intersections, the movement of each pixel on the interpolation frame can be obtained. Conversely, from the allocated motion vector, it can be determined from which position the pixel on the interpolation frame has moved on the 24P signal frame.

  FIG. 40 shows one-dimensional examples of motion vectors detected in the original frame of the 24P signal and pixels on the interpolation frame of the 60P signal. In the example of FIG. 40, between the frame t at the time t and the frame t + 1 at the time t + 1 of the two 24P signals, for example, the interpolation frame F1 at the time t + 0.4 and the interpolation frame F2 at the time t + 0.8 are Two are inserted. As described above, the position of this interpolation frame is the position where the time phase on the 24P signal is 0.0, 0.4, 0.8, 1.2, and 1.6, and the 60P signal. Are set in advance in the signal processing apparatus 1.

  A circle on each frame indicates each pixel. Motion vectors v1, v2, and v3 detected in the frame t by the preceding vector detection unit 52 are extended in the frame t + 1 direction. When these motion vectors are allocated to each pixel of the interpolation frames F1 and F2, the motion vector passing through the vicinity of each pixel on the interpolation frame is a candidate vector (hereinafter also referred to as allocation candidate vector) allocated to that pixel. Is done.

  Accordingly, the motion vector v1 from the pixel on the left side of the frame t to the vicinity of the fourth and fifth pixels from the left of the frame t + 1 is between the second and third pixels from the left on the interpolation frame F1. It passes between the third and fourth pixels from the left on F2. Therefore, the motion vector v1 is a pixel included in the vicinity N1 of the point where the motion vector v1 intersects the interpolation frames F1 and F2 (second and third pixels from the left of the interpolation frame F1 and the left of the interpolation frame F2). To the third and fourth pixels).

  The motion vector v2 from the third pixel from the left of the frame t to the vicinity of the second and third pixels from the left of the frame t + 1 is between the second and third pixels from the left on the interpolation frame F1, It passes between the second and third pixels from the left on the interpolation frame F2. Therefore, the motion vector v2 is a pixel included in the vicinity region N2 of the point where the motion vector v2 intersects the interpolation frames F1 and F2 (second and third pixels from the left of the interpolation frame F1 and the interpolation frame F2). Allocation candidate vectors to be allocated to the second and third pixels from the left).

  Furthermore, the motion vector v3 from the fifth pixel from the left of the frame t to the vicinity of the fourth and fifth pixels from the left of the frame t + 1 is between the fourth and fifth pixels from the left on the interpolation frame F1, It passes between the fourth and fifth pixels from the left on the interpolation frame F2. Therefore, the motion vector v3 includes the pixels included in the vicinity region N3 of the point where the motion vector v3 intersects the interpolation frames F1 and F2 (the fourth and fifth pixels from the left of the interpolation frame F1 and the interpolation frame F2). 4th and 5th pixels from the left).

  That is, the allocation candidate vector of the second pixel from the left of the interpolation frame F2 is the motion vector v2, and the second and third pixels from the left on the interpolation frame F1, and 3 from the left of the interpolation frame F2. The allocation candidate vectors of the th pixel are motion vectors v1 and v2. Further, the allocation candidate vectors of the fourth pixel from the left of the interpolation frame F2 are motion vectors v1 and v3, the fourth and fifth pixels from the left on the interpolation frame F1, and the left of the interpolation frame F2. The fifth pixel allocation candidate vector from is the motion vector v3.

  As described above, an allocation candidate vector to be allocated to each pixel on the interpolation frame is obtained from the motion vectors detected in the original frame. In the leftmost pixel and the rightmost pixel (black circles in the figure) of the interpolation frames F1 and F2, motion vectors passing through the vicinity are not shown. That is, there is no allocation candidate vector to be allocated to the leftmost pixel and the rightmost pixel of the interpolation frames F1 and F2. Therefore, an allocation compensation process is performed on these pixels in an allocation compensation unit 57 in a later stage described later.

Furthermore, with reference to FIG. 41, the motion vector detected in the original frame and the pixels on the interpolation frame of the 60P signal will be described in detail. In the example of FIG. 41, the arrow T indicates the direction of time passage from the frame t at the time t at the left front in the figure to the frame t + 1 at the time t + 1 at the back right. Further, interpolation frame F1 is placed at time t + pos _t between times t and time t + 1.

In the example of FIG. 41, the motion vector v _a (x _va , y _va ) detected by the pixel (x _a , y _a ) on the frame t is extended in the frame t + 1 direction, and the intersection (x _ia , y _ia ) is calculated. Intersection, it is considered that pixels corresponding to end points of the motion vector v _a of the frame t of the 24P signal to a point moved, specifically, is expressed by the equation (13) and (14).

_{x ia = x a + pos t} x va ··· (13)
_{y ia = y a + pos t} y va ··· (14)

Here, as described above, when the motion vector v _a has a less precision pixels, the intersection of the motion vector v _a, pixel positions on the interpolation frame F1 does not always match. Otherwise, as shown in FIG. 41, the motion vector v _a is allocated with respect to the four neighboring pixels G1 to G4 of the intersection on the interpolated frame F1. That is, the motion vector v _a is shifted (parallel movement) on each of the pixels G1 to G4 in the neighborhood, is the allocation candidate vector to be allocated to each of the pixels, the allocation compensating process is executed.

  As described above, since one motion vector may be a candidate assigned to four neighboring pixels, a plurality of motion vectors may be assigned candidate vectors depending on the pixel. In this case, the vector allocating unit 54 calculates, for each motion vector, a pixel on the interpolation frame and an intersection point on the original frame associated with the motion vector, and evaluates each motion vector using the intersection point. Thus, finally, a motion vector to be assigned to the pixel on the interpolation frame is determined.

With reference to FIG. 42, the motion vector evaluation in the vector allocation unit 54 will be described. Figure 42 is a bottom, frame t at time t in FIG. 41, the time t + pos _t interpolated frame F1, and the frame t + 1 at time t + 1, is one-dimensionally illustrated.

In the example of FIG. 42, the motion vector sv _a is the pixel (x _a, y _a) of the frame t detected motion vector v _a at is as allocation candidate vectors of the pixels in the neighborhood G4, shifts on the pixel G4 ( Translated). Here, the intersections of the motion vector sva, shifted onto the pixel G4, and the frame t and the frame t + 1 are defined as _a point P and a point Q, respectively.

The vector allocating unit 54, a first evaluation of motion vector sv _a, first, obtains a DFD operation range around the point P and point Q respectively, DFD operation range determined whether jutting out the image frame Judging. Therefore, when the DFD operation range around the point P and point Q had protruding a picture frame, the motion vector sv _a is excluded from the candidates.

Further, when the point P and the point Q belong to different objects, for example, the difference between the luminance value F _t (P) of the point P and the luminance value F _{t + 1} (Q) of the point Q becomes large. . Accordingly, the vector allocating unit 54, a second evaluation of motion vector sv _a, using the points P and Q, determine the luminance difference absolute value dp of the pixel G4, the luminance difference absolute value dp is greater than a predetermined value Determine whether or not. If the luminance difference absolute value dp is determined to be larger than the predetermined value is a low reliability of the motion vector sv _a in the pixel G4, a motion vector sv _a is excluded from the candidates. The luminance difference absolute value dp is expressed by the following equation (15).

dp = | F _t (P) − F _{t + 1} (Q) | (15)

Then, as the third evaluation of the motion vector sv _a, the vector allocating unit 54 performs evaluation determined by the difference absolute value that represents the correlation value DFD operation range around the point P and the point Q. That is, the vector allocating unit 54, using the DFD operation range around the point P and the point Q, obtains the evaluated value DFD of the motion vector sv _a (absolute differences) in the pixel G4, obtained evaluation value DFD is It is determined whether or not it is smaller than the minimum evaluation value of the DFD table. As a result of the above evaluation, the vector assigning unit 54 assigns a motion vector having the smallest evaluation value DFD among the obtained evaluation values DFD to the pixel G4.

  As described above, since the motion vector of the allocation candidate in the pixel of the interpolation frame is evaluated using not only the evaluation value DFD of the allocation target pixel but also the luminance difference absolute value in the allocation target pixel, the conventional evaluation Rather than using only the value DFD, a reliable motion vector can be allocated to the allocation target pixel. As a result, the accuracy of vector allocation is improved.

  As described above, in order to evaluate the motion vector of the allocation candidate in the pixel of the interpolation frame, the position on the original frame associated with the motion vector is used with reference to the pixel of the interpolation frame. When obtaining the luminance difference absolute value dp and the evaluation value DFD, the motion vector of the allocation candidate is extended with reference to the pixel position of the interpolation frame, so that the intersection of the motion vector and the original frame is the pixel position of the original frame. The pixel values may not match, and the pixel value cannot be obtained as it is. In such a case, the four-point interpolation process described above with reference to FIG. 39 is executed.

  FIG. 43 shows an example of four-point interpolation in the vector allocation process. In FIG. 43, portions corresponding to those in FIGS. 41 and 42 are denoted by the corresponding reference numerals, and the description thereof will be omitted because it will be repeated.

In the example of FIG. 43, the motion vector sv _a assignment candidate, because it is extended pixel position G4 in interpolated frame F1 as a reference, the point of intersection P between the motion vector sv _a frame t, in frame t does not coincide with the pixel position (white circle on the frame t), also intersecting point Q between the motion vector sv _a frame t + 1 also does not match the pixel positions on the frame t + 1 (open circles in frame t). Therefore, in the frame t and the frame t + 1, the above-described four-point interpolation is performed using the four pixels in the vicinity E centered at the intersection point P and the intersection point Q (white circles on the frame t and the frame t + 1), respectively. Pixel values of P and intersection point Q are obtained.

  As described above, in the vector allocation processing of the present invention, the luminance difference absolute value dp and the evaluation value DFD are calculated using the pixel values of the intersection P and the intersection Q obtained by the four-point interpolation processing. The luminance difference absolute value dp and the evaluation value DFD can be obtained with higher accuracy than the method of rounding the sub-pixel components.

  Next, details of the vector allocation processing will be described with reference to the flowchart of FIG. The frame t of the image at time t and the frame t + 1 of the image at time t + 1, which are the original frames of the 24P signal, are input to the pixel information calculation unit 701, the evaluation value calculation unit 702, and the target pixel difference calculation unit 703.

  When a new original frame is input, the pixel information calculation unit 701 controls the vector selection unit 705, initializes the allocation flag of the allocation flag memory 56 to 0 (False) in step S701, and proceeds to step S702. The allocation vector memory 55 is initialized with the 0 vector, and the process proceeds to step S703. As a result, a zero vector is assigned to a pixel to which no motion vector is assigned.

  In step S703, the pixel information calculation unit 701 controls the evaluation value calculation unit 702 to calculate the evaluation value DFD0 using the 0 vector for all the pixels on the interpolation frame, and the vector selection unit 705. And the evaluation value DFD0 of the 0 vector calculated by the evaluation value calculation unit 702 is stored in the DFD table as the minimum evaluation value for each pixel of the interpolation frame, and the process proceeds to step S704. That is, in step S703, the evaluation value calculation unit 702 calculates the evaluation value DFD0 using the 0 vector for all the pixels in the interpolation frame, and the calculated evaluation value DFD0 is passed through the vector evaluation unit 704. The data is output to the vector selection unit 705. Then, the vector selection unit 705 stores the evaluation value DFD0 input via the vector evaluation unit 704 as the minimum evaluation value of the corresponding pixel in the DFD table.

  In step S704, the pixel information calculation unit 701 selects a pixel from the original frame on the detection vector memory 53, and proceeds to step S705. In this case, pixels are selected in the raster scan order from the upper left of the frame.

  In step S705, the pixel information calculation unit 701 performs pixel position calculation processing, and proceeds to step S706. The details of this pixel position calculation process will be described later with reference to FIG. 45, but this pixel position calculation process allows the motion vector detected in the pixel selected in step S704 to be assigned to the interpolation frame. An allocation target pixel is calculated, and a position on the original frame associated with the motion vector is calculated on the basis of the calculated allocation target pixel.

  In step S706, the pixel information calculation unit 701 selects the calculated allocation target pixel, outputs the selected allocation target pixel and its motion vector to the vector selection unit 705, and proceeds to step S707. At the same time, the pixel information calculation unit 701 outputs, to the evaluation value calculation unit 702 and the pixel-of-interest calculation unit 703, information on the position on the original frame associated with the motion vector based on the selected allocation target pixel. To do. In step S706, the pixel information calculation unit 701 selects an upper left pixel when there are a plurality of allocation target pixels.

  In step S707, the pixel information calculation unit 701 executes an allocation vector evaluation process for the selected allocation target pixel, and the process proceeds to step S708. Details of the allocation vector evaluation process will be described later with reference to FIG. 46. By this allocation vector evaluation process, the motion vector evaluation value DFD and the luminance difference absolute value in the allocation target pixel are obtained, and the motion in the allocation target pixel is determined. The reliability of the vector is determined, and the motion vector in the allocation vector memory 55 is rewritten with the motion vector determined to have high reliability as a result of these determinations.

  In step S708, the pixel information calculation unit 701 determines whether or not the processing of all allocation target pixels has been completed. If it is determined that the processing of all allocation target pixels has not yet been completed, Returning to S706, the next allocation target pixel is selected, and the subsequent processing is repeated.

  If it is determined in step S708 that the processing of all the allocation target pixels has been completed, the pixel information calculation unit 701 determines whether or not the processing of all the pixels of the original frame on the detection vector memory 53 has been completed in step S709. Determine whether. If the pixel information calculation unit 701 determines in step S709 that the processing of all the pixels of the original frame on the detection vector memory 53 has not been completed, the process returns to step S704 and the next of the original frame on the detection vector memory 53. This pixel is selected, and the subsequent processing is repeated. If the pixel information calculation unit 701 determines in step S709 that the process for all the pixels in the detection vector memory 53 has been completed, the pixel information calculation unit 701 ends the vector allocation process.

  Next, details of the pixel position calculation process will be described with reference to the flowchart of FIG. FIG. 45 shows an example of the pixel position calculation process in step S705 of FIG.

  In step S721, the pixel information calculation unit 701 acquires the motion vector detected by the pixel selected by the process in step S704 from the detection memory vector 53, and the process proceeds to step S722. If the motion vector of the selected pixel is a 0 vector, the allocation vector memory 55 stores a 0 vector as an initial value in advance, so that the subsequent steps S722 to S724 and step S706 in FIG. 44 are performed. The processes from S708 to S708 are skipped, and the process proceeds to step S709.

  In step S722, the pixel information calculation unit 701 calculates the intersection of the acquired motion vector and the interpolation frame. That is, the pixel information calculation unit 701 extends the acquired motion vector in the direction of the next frame t + 1, calculates the intersection point of the extended motion vector and the interpolation frame, and proceeds to step S723.

  In step S723, the pixel information calculation unit 701 sets an allocation target pixel from the intersection calculated from the motion vector and the interpolation frame, and proceeds to step S724. At this time, if the intersection point coincides with the pixel position on the interpolation frame, the pixel information calculation unit 701 sets the intersection point as the allocation target pixel. On the other hand, when the intersection does not coincide with the pixel position on the interpolation frame, the pixel information calculation unit 701 sets four pixels near the intersection on the interpolation frame as allocation target pixels as described above.

  In step S724, the pixel information calculation unit 701 is acquired with reference to each allocation target pixel, which is necessary for the evaluation value calculation unit 702 and the target pixel difference calculation unit 703 to obtain the evaluation value DFD and the luminance difference absolute value. The position on the original frame associated with the motion vector is calculated. Specifically, in step S724, the pixel information calculation unit 701 shifts (translates) the acquired motion vector to the set allocation target pixel, and the shifted motion vector and the intersection of the original frame The position is obtained, the pixel position calculation process is terminated, and the process returns to step S706 in FIG.

  Next, the details of the allocation vector evaluation process will be described with reference to the flowchart of FIG. 46 shows an example of the allocation vector evaluation process in step S707 of FIG.

  In step S706 of FIG. 44, the pixel information calculation unit 701 obtains the position on the original frame associated with the motion vector with reference to the selected allocation target pixel, and information on the obtained position on the original frame is obtained. The evaluation value calculation unit 702 and the target pixel difference calculation unit 703 are input.

  When the position information on the original frame is input from the pixel information calculation unit 701 to the evaluation value calculation unit 702, in step S741, the evaluation value calculation unit 702 calculates the evaluation value DFD of the motion vector in the allocation target pixel by using the frame t and the frame Each DFD calculation range (m × n) centered on the position on t + 1 is obtained, and the process proceeds to step S742, where it is determined whether or not the obtained DFD calculation range is within the image frame. If the evaluation value calculation unit 702 determines in step S742 that the DFD calculation range is outside the image frame, the evaluation value calculation unit 702 determines that the motion vector is not an allocation candidate vector to be allocated to the allocation target pixel, and steps S743 to S749. Is skipped, the assigned vector evaluation process is terminated, and the process returns to step S708 in FIG.

  If the evaluation value calculation unit 702 determines in step S742 that the obtained DFD calculation range is within the image frame, the evaluation value calculation unit 702 proceeds to step S743 and uses the DFD calculation range determined to be within the image frame, The evaluation value DFD of the target pixel is calculated, and the obtained evaluation value DFD is output to the evaluation value determination unit 712, and the process proceeds to step S744. At this time, if the position on the original frame is equal to or less than the pixel, the evaluation value DFD of the allocation target pixel is obtained by obtaining the luminance value of the intersection on the original frame using the above-described four-point interpolation. Calculated.

  On the other hand, when the position information on the original frame is input from the pixel information calculation unit 701, the target pixel difference calculation unit 703 obtains the luminance difference absolute value dp in the allocation target pixel in step S744, and the obtained luminance The difference absolute value dp is output to the pixel difference determining unit 711, and the process proceeds to step S745. At this time, if the position on the original frame is equal to or less than the pixel, the pixel-of-interest calculation unit 703 obtains the luminance value of the intersection on the original frame using the above-described four-point interpolation. The luminance difference absolute value dp in the allocation target pixel is calculated.

  In step S745, the pixel difference determination unit 711 determines whether the luminance difference absolute value dp of the allocation target pixel from the target pixel difference calculation unit 703 is equal to or less than a predetermined threshold value, and determines the luminance of the allocation target pixel. When it is determined that the difference absolute value dp is larger than the predetermined threshold value, it is determined that there is a high possibility that the intersections of the frame t and the frame t + 1 belong to different objects, that is, the motion vector is determined in the allocation target pixel. It is determined that the reliability is low and the assignment candidate vector to be assigned to the assignment target pixel is not determined, the processes of steps S746 to S749 are skipped, the assignment vector evaluation process is terminated, and the process returns to step S708 of FIG.

  If the pixel difference determination unit 711 determines in step S745 that the luminance difference absolute value dp of the allocation target pixel is equal to or less than the predetermined threshold value, the process proceeds to step S746. In step S746, the evaluation value determination unit 712 refers to the DFD table of the vector selection unit 705, and the evaluation value DFD of the allocation target pixel from the evaluation value calculation unit 702 is the minimum allocation target pixel stored in the DFD table. It is determined whether or not the evaluation value (in this case, the evaluation value DFD0 of the zero vector) is smaller, and the evaluation value DFD of the allocation target pixel from the evaluation value calculation unit 702 is the allocation target pixel stored in the DFD table. If it is determined that the motion vector is equal to or greater than the minimum evaluation value, it is determined that the reliability of the motion vector is not high in the allocation target pixel, the processing in steps S747 to S749 is skipped, and the allocation vector evaluation processing is terminated. The process returns to step S708 of 44.

  On the other hand, when the evaluation value determination unit 712 determines in step S746 that the evaluation value DFD of the allocation target pixel from the evaluation value calculation unit 702 is smaller than the minimum evaluation value of the allocation target pixel stored in the DFD table. The motion vector is determined to have the highest reliability based on the evaluation value DFD among the motion vectors compared so far in the allocation target pixel, and the evaluation value DFD of the allocation target pixel determined to have high reliability. Is output to the vector selection unit 705, and the process proceeds to step S747.

  Upon receiving the evaluation value DFD of the allocation target pixel from the evaluation value determination unit 712, the vector selection unit 705 rewrites the allocation flag of the allocation target pixel in the allocation flag memory 56 to 1 (True) in step S747, and in step S748. The minimum evaluation value corresponding to the pixel to be allocated in the DFD table is rewritten to the evaluation value DFD determined to have high reliability by the evaluation value determination unit 712, and the process proceeds to step S749.

  The vector selection unit 705 receives the allocation target pixel selected from the pixel information calculation unit 701 and its motion vector in step S706. Therefore, in step S749, the vector selection unit 705 rewrites the motion vector allocated to the allocation target pixel in the allocation vector memory 55 with the motion vector corresponding to the evaluation value DFD determined to have high reliability, and allocates the allocation vector. The evaluation process ends, and the process returns to step S708 in FIG.

  As described above, when selecting a motion vector to be allocated to the allocation target pixel of the interpolation frame, not only the evaluation value DFD but also a position on the original frame associated with the motion vector based on the allocation target pixel is obtained. Since the absolute value of the luminance difference of the allocation target pixel is treated separately and evaluated, the most probable motion vector is selected from the allocation candidate vectors rather than using the conventional evaluation value DFD. , It can be assigned to the pixel to be assigned. Thereby, the accuracy of vector allocation is improved, discontinuity of the image generated in the subsequent image interpolation processing, etc. can be suppressed, and the image quality can be improved.

  Furthermore, when obtaining the evaluation value DFD and the luminance difference absolute value, etc., when a pixel value at a position below the pixel is necessary, the value is obtained by linear interpolation based on the distance to the four neighboring pixels at the position below the pixel. As a result, it is possible to perform processing with sub-pixel position accuracy, and more accurately obtain the luminance difference absolute value dp and the evaluation value DFD than the conventional method of rounding off the sub-pixel components. From the allocation candidate vectors, it is possible to allocate a more probable motion vector by the pixel of interest. That is, the accuracy of vector allocation processing is improved.

  In addition, the evaluation value DFD based on the 0 vector is stored as an initial value in the DFD table in advance, and while the motion vector is sequentially processed, the evaluation value DFD based on a certain motion vector becomes the minimum evaluation value at that time. In this case, since the minimum evaluation value of the DFD table and the motion vector allocated to the allocation vector memory 55 are updated as needed, time and resources can be used efficiently.

  Next, details of the configuration of the allocation compensation unit 57 will be described.

  FIG. 47 is a block diagram showing a configuration of the allocation compensation unit 57. 47 includes a vector compensation unit 1001, an evaluation value calculation unit 1002, a compensation candidate vector memory 1003, and the same vector determination unit 1004. A motion vector is not allocated by the vector allocation unit 54. In addition, a process is performed in which the pixels on the interpolation frame of the 60P signal are allocated with the motion vectors of the surrounding pixels supplemented.

  The motion vector is assigned to the pixel of the interpolation frame on the assigned vector memory 55 by the preceding vector assigning unit 54. Further, 1 (True) is written in the allocation flag of the pixel allocation flag memory 56 to which the motion vector is allocated by the vector allocation unit 54, and the allocation of the allocation flag memory 56 of the pixel to which no motion vector is allocated. 0 (False) is written in the flag.

  The vector compensation unit 1001 is a compensation processing target block serving as a unit of processing for allocating a motion vector from a frame to be processed (interpolated frame) based on coordinates (x, y) on the frame (screen). Select. The compensation processing target block is an example of a target block and includes a plurality of pixels that are target pixels. Here, x indicates a horizontal position on the frame (screen), and y indicates a vertical position on the frame (screen).

  The vector compensation unit 1001 selects a compensation processing target block in order from a frame to be processed, thereby allocating a motion vector for all the pixels to which no motion vector is assigned among the pixels of the frame.

  For example, the vector compensation unit 1001 selects a compensation processing target block from the frame to be processed in the raster scan order.

  Further, the vector compensation unit 1001 determines whether or not there is a target pixel to which no motion vector is assigned in the selected compensation processing target block, based on the assignment flag in the assignment flag memory 56. When it is determined that there is no target pixel to which no motion vector is assigned in the selected compensation processing target block, it is not necessary to assign a motion vector to the target pixel belonging to the compensation processing target block, and thus the vector compensation unit 1001 Does not execute the process of assigning a motion vector for that block to be compensated (skips the process), and selects the next block to be compensated.

  When it is determined that there is a target pixel to which no motion vector is assigned in the selected compensation processing target block, it is necessary to assign a motion vector to the target pixel belonging to the compensation processing target block, and thus the vector compensation unit 1001 Executes a process of assigning a motion vector to the compensation process target block.

  When executing a motion vector allocation process for the compensation processing target block, the vector compensation unit 1001 further includes adjacent pixels that are pixels around the compensation processing target block and are adjacent to the compensation processing target block. A candidate vector selection region consisting of is determined.

  The vector compensation unit 1001 assigns priorities to each pixel of interest and adjacent pixels included in the compensation processing target block. This priority is assigned to each of the target pixel and the adjacent pixel based on the position of the target pixel in the compensation processing target block, the position of the adjacent pixel with respect to the compensation processing target block, and the mutual positional relationship between the target pixel and the adjacent pixel. The

  For example, the vector compensation unit 1001 assigns priorities to each pixel of interest and adjacent pixels based on the raster scan order. More specifically, for example, the vector compensation unit 1001 assigns a priority order that gives priority to the target pixel relative to the adjacent pixel and that corresponds to the raster scan order of the target pixel. Priorities corresponding to the raster scan order of pixels are assigned to adjacent pixels.

  The vector compensation unit 1001 instructs the evaluation value calculation unit 1002 to calculate the evaluation value. In this case, first, the vector compensation unit 1001 evaluates the evaluation value calculation unit 1002 for the 0 vector (the vector (static vector) in which the x-direction component vx is 0 and the y-direction component vy is 0). Instructs the calculation of the value.

  Next, the vector compensation unit 1001 instructs the evaluation value calculation unit 1002 to calculate evaluation values for the target pixel and adjacent pixels to which the motion vector is assigned, in order of priority. In this case, the vector compensation unit 1001 determines whether or not a motion vector is allocated to the target pixel and the adjacent pixel based on the allocation flag of the allocation flag memory 56, and it is determined that the motion vector is allocated. In this case, the calculation of the evaluation value is instructed for the target pixel and the adjacent pixel.

  Although details will be described later, the evaluation value calculated by the evaluation value calculation unit 1002 is the pixel value of a pixel belonging to a predetermined block (region) centered on the start point of the motion vector in the frame t of the input image, and the input image A correlation with a pixel value of a pixel belonging to a predetermined block (area) around the end point of the motion vector in the frame t + 1 is shown.

  When the evaluation value calculation unit 1002 is instructed to calculate an evaluation value for a predetermined motion vector, the evaluation value and the motion vector as a result of the calculation are supplied from the evaluation value calculation unit 1002, so that the vector compensation unit 1001 Based on the evaluation value, a motion vector is assigned to a pixel of interest to which no motion vector is assigned. That is, for example, the vector compensation unit 1001 sets the evaluation value indicating the strongest correlation among the evaluation values supplied from the evaluation value calculation unit 1002 among the zero vectors or the motion vectors assigned to the target pixel or adjacent pixels. The corresponding motion vector is selected as the motion vector to be compensated. Then, the vector compensation unit 1001 assigns a motion vector to be compensated to a target pixel to which no motion vector is assigned.

  In other words, the vector compensation unit 1001 includes the pixel value of the start pixel in the frame t of the input image and the pixel of the end pixel in the frame t + 1 of the input image among the motion vectors assigned to the target pixel or adjacent pixels. A motion vector having the strongest correlation with a value is assigned to a pixel of interest to which no motion vector is assigned.

  Note that the vector compensation unit 1001 has the strongest of the calculated evaluation values when the evaluation values are calculated for a predetermined number of motion vectors, or when the evaluation values are calculated for all the target pixels or adjacent pixels. A motion vector corresponding to the evaluation value indicating the correlation may be selected as a motion vector to be compensated.

  In response to the calculation instruction of the vector compensation unit 1001, the evaluation value calculation unit 1002 acquires the target motion vector from the same vector determination unit 1004 as described later, and based on the frame t + 1 and the frame t of the input image. Then, an evaluation value for the motion vector is calculated. For example, the evaluation value calculation unit 1002 controls the pixel values of pixels belonging to a predetermined block centered on the start point of the motion vector in the frame t of the input image and the frame t + 1 of the input image under the control of the vector compensation unit 1001. An evaluation value that is the sum of absolute differences from the pixel values of pixels belonging to a predetermined block with the end point of the motion vector as the center is calculated.

  Note that the evaluation value is sufficient if it indicates a correlation, and may be, for example, a variance or a sum of squared differences.

  More specifically, for example, the evaluation value calculation unit 1002 calculates the start point of the motion vector on the frame t, that is, the pixel value of the pixel belonging to the block centered on the target pixel or the adjacent pixel for calculating the evaluation value, and the motion An evaluation value indicating a correlation with a pixel value of a pixel belonging to a block centering on the end point of the vector frame t + 1 is calculated.

  Further, for example, the evaluation value calculation unit 1002 has a point where the motion vector and the frame to be processed (interpolated frame) intersect when the start point of the motion vector is the frame t and the end point of the motion vector is the frame t + 1. The motion vector is moved so that becomes a point in the compensation processing target block, the pixel value of the pixel belonging to the block centered on the start point of the motion vector frame t at that position, and the motion vector frame t + 1 at that position An evaluation value indicating a correlation with a pixel value of a pixel belonging to a block centering on the end point is calculated. Further, when the motion vector is moved so that the point where the motion vector and the interpolation frame intersect is a point in the compensation processing target block, for example, the intersected point after the movement is the target pixel with the highest priority. Or the center of the compensation processing target block.

  The evaluation value calculation unit 1002 supplies the calculated evaluation value to the vector compensation unit 1001 together with the motion vector. Further, the evaluation value calculation unit 1002 supplies the motion vector obtained by calculating the evaluation value to the compensation candidate vector memory 1003 as a compensation candidate vector.

  The compensation candidate vector memory 1003 stores the compensation candidate vector supplied from the evaluation value calculation unit 1002. For example, the compensation candidate vector memory 1003 stores the size and direction of the compensation candidate vector, which is a motion vector for which the evaluation value is calculated, in association with the order in which the compensation candidate vectors are supplied from the evaluation value calculation unit 1002. The compensation candidate vector memory 1003 supplies the stored compensation candidate vector to the same vector determination unit 1004.

  Based on the control of the vector compensation unit 1001, the same vector determination unit 1004 determines whether the direction and magnitude of the motion vector assigned to the adjacent pixel or the target pixel for which an evaluation value is to be calculated next is the compensation candidate vector memory It is determined whether or not the direction and size of the compensation candidate vector stored in 1003 is the same. For example, the same vector determination unit 1004 acquires a motion vector allocated to the adjacent pixel or the target pixel from the allocation vector memory 55 based on the control of the vector compensation unit 1001, and the compensation candidate vector from the compensation candidate vector memory 1003. Is determined, and it is determined whether the x-direction component and the y-direction component of the allocated motion vector are the same as the x-direction component and the y-direction component of the compensation candidate vector, respectively.

  When it is determined that the direction and magnitude of the motion vector are not the same as any direction and magnitude of the compensation candidate vector, the same vector determination unit 1004 may calculate a different evaluation value for the motion vector. Since it is expected, the motion vector is supplied to the evaluation value calculation unit 1002 in order to cause the evaluation value calculation unit 1002 to calculate the evaluation value.

  When it is determined that the direction and magnitude of the motion vector are the same as any direction and magnitude of the compensation candidate vector, the same vector determination unit 1004 does not need to calculate an evaluation value for the same motion vector. The motion vector is not supplied to the evaluation value calculation unit 1002 in order to suppress the calculation of the evaluation value in the value calculation unit 1002.

  That is, if the direction and magnitude of a motion vector for which an evaluation value is to be calculated is the same as any one of the compensation candidate vectors, the evaluation value calculation for that motion vector is skipped. The amount of calculation for calculating the value can be further reduced.

  FIG. 48 is a diagram illustrating an example of priorities assigned to each of the target pixel and adjacent pixels. In the example shown in FIG. 48, the compensation processing target block 1021 includes 2 × 2 four target pixels. In the example shown in FIG. 48, the adjacent pixel area 1022 includes 12 adjacent pixels that are pixels around the compensation processing target block 1021 and are arranged adjacent to the compensation processing target block 1021. . Evaluation values are calculated for the motion vectors assigned to the four target pixels and the twelve adjacent pixels included in the compensation processing target block 1021 or the adjacent pixel region 1022.

  A priority order that is prioritized over adjacent pixels is assigned to the pixel of interest. That is, in the example shown in FIG. 48, any one of the priorities “1” to “4” is assigned to each pixel of interest, and any one of the priorities “5” to “16” is assigned to each of the adjacent pixels. It is attached to.

  In the example shown in FIG. 48, the priority order “1” is assigned to the pixel of interest on the upper left of the compensation processing target block 1021, and the priority order of “2” is assigned to the attention pixel on the upper right of the compensation processing target block 1021. Then, the priority order “3” is assigned to the lower left target pixel of the compensation processing target block 1021, and the priority order “4” is assigned to the lower right attention pixel of the compensation processing target block 1021. That is, the priority order corresponding to the raster scan order of the target pixel in the compensation processing target block 1021 is assigned to the target pixel.

  In the example shown in FIG. 48, the priority order “5” is assigned to the upper left adjacent pixel in the adjacent pixel area 1022, and the second adjacent pixel from the left in the first row from the top of the adjacent pixel area 1022 is “ 6 ”, and the third pixel from the left in the first row from the top of the adjacent pixel region 1022 is given the priority“ 7 ”, and the upper right adjacent pixel in the adjacent pixel region 1022 Is assigned a priority of “8”.

  In the example shown in FIG. 48, the priority order “9” is assigned to the left side adjacent pixel in the second row from the top of the adjacent pixel region 1022, and the right side of the second row from the top in the adjacent pixel region 1022 is assigned. Adjacent pixels are given a priority of “10”. In the example shown in FIG. 48, the priority order “11” is assigned to the left side adjacent pixel in the third row from the top of the adjacent pixel region 1022, and the right side of the third row from the top of the adjacent pixel region 1022 is assigned. Adjacent pixels are given a priority of “12”.

  Further, in the example shown in FIG. 48, the priority order “13” is given to the lower left adjacent pixel in the adjacent pixel area 1022, and the second adjacent pixel from the left in the first row from the bottom of the adjacent pixel area 1022. Is assigned a priority of “14”, and the third adjacent pixel from the left in the first row from the bottom of the adjacent pixel region 1022 is assigned a priority of “15”, and the lower right of the adjacent pixel region 1022 A priority order of “16” is assigned to the adjacent pixels.

  That is, the priority order corresponding to the raster scan order of the adjacent pixels in the adjacent pixel area 1022 is given to the adjacent pixels.

  In other words, for each compensation processing target block 1021 and adjacent pixel region 1022, the pixel of interest in the compensation processing target block 1021 in an order (order) corresponding to the order (order) in which the compensation processing target block 1021 on the frame is selected. The priorities of the adjacent pixels in the adjacent pixel region 1022 are assigned.

  In the calculation area 1023 of FIG. 48, the evaluation value calculation unit 1002 sets the pixel value of the pixel belonging to the start point of the motion vector on the frame t, that is, the target pixel for calculating the evaluation value or the block centered on the adjacent pixel, A block centered on a target pixel or an adjacent pixel that is a target of evaluation value calculation when calculating an evaluation value indicating a correlation with a pixel value of a pixel belonging to the block centering on the end point of the motion vector frame t + 1 Indicates a range of pixels belonging to.

  For example, the evaluation value calculation unit 1002 calculates the pixel value of a pixel belonging to a block of 5 × 5 pixels centered on the pixel of interest or the neighboring pixel that is the starting point of the motion vector on the frame t, and the frame t + 1 of the motion vector. An evaluation value indicating a correlation with a pixel value of a pixel belonging to a 5 × 5 pixel block centering on the end point is calculated.

  In this case, the pixel located two rows above the upper left adjacent pixel in the adjacent pixel region 1022 and the pixel on the left side of the second column is the target of the evaluation value calculation, and two rows above the upper right adjacent pixel in the adjacent pixel region 1022 The pixel on the right side of the second column is the target of the evaluation value calculation, the pixel on the lower side of the second row and the pixel on the left side of the second column is the target of the calculation of the evaluation value, As shown in FIG. 48, the calculation region 1023 is compensated because the pixel on the lower side of the second row and the pixel on the right side of the second column with respect to the lower right adjacent pixel in the adjacent pixel region 1022 is the target of the evaluation value calculation. This is an 8 × 8 pixel area centered on the processing target block 1021.

  When the compensation processing target block 1021 is selected in the raster scan order, the pixel in the upper row or the left pixel of the selected compensation processing target block 1021 is subjected to either the vector allocation unit 54 or the allocation compensation unit 57. A motion vector is always assigned.

  That is, when the compensation processing target block 1021 is selected in the raster scan order, the allocation compensation unit 57 applies the motion vector to the pixel in the order from the left side to the right side on the frame and from the upper side to the lower side on the frame. Therefore, the pixels in the upper row (the so-called line on the screen) of the currently selected compensation processing target block 1021 are selected as the compensation processing target block 1021 before that, and the assignment compensation unit 57 A motion vector is allocated. Similarly, the pixel on the left side of the currently selected compensation processing target block 1021 is selected as the previous compensation processing target block 1021, and a motion vector is assigned by the assignment compensation unit 57.

  For example, as shown in FIG. 49, when the compensation processing target block 1021 is a 2 × 2 pixel block, four pixels (adjacent to the upper row of the selected compensation processing target block 1021 belonging to the region 1041). Pixel) and two pixels (adjacent pixels) on the left side of the selected compensation processing target block 1021 are always assigned motion vectors.

  On the other hand, as shown in FIG. 49, when the compensation processing target block 1021 is a 2 × 2 pixel block, the four pixels in the lower row of the selected compensation processing target block 1021 belonging to the region 1042 Is not necessarily assigned a motion vector. Similarly, motion vectors are not always assigned to the two pixels on the right side of the selected compensation processing target block 1021.

  Note that when a pixel of interest to which no motion vector has been assigned by the vector assigning unit 54 is included in the compensation processing target block 1021, the spatial correlation of the image is high, so that the motion vector of another pixel of interest of the compensation processing target block 1021 is Are often assigned to the pixel of interest.

  Further, although the compensation processing target block 1021 is described as a 2 × 2 pixel block, the compensation processing target block 1021 is not limited to a 2 × 2 pixel block, but a block including a desired number of pixels. Can do.

  FIG. 50 is a diagram for explaining the skip of evaluation value calculation. In FIG. 50, a circle indicates one pixel that is a target pixel or an adjacent pixel.

  An arrow attached to a circle in FIG. 50 indicates a motion vector assigned to the target pixel or an adjacent pixel. For example, two motion vectors indicated by a in the figure have the same size and orientation. The two motion vectors indicated by b in the figure have the same size and direction. Further, the two motion vectors indicated by c in the figure have the same size and orientation.

  Therefore, in the example shown in FIG. 50, there are 11 motion vectors assigned to the target pixel or adjacent pixels, but there are 6 motion vectors having the same size and orientation as other motion vectors. Therefore, among the six motion vectors having the same size and orientation as other motion vectors, the calculation of the evaluation value is skipped for three motion vectors. Therefore, in such a case, evaluation values are calculated for the eight motion vectors.

  For example, when the same priority order as the priority order shown in FIG. 48 is given to the target pixel or the adjacent pixel shown in FIG. 50, the motion assigned to the upper right adjacent pixel (adjacent pixel having a priority order of 8). Since the magnitude and direction of the vector are the same as the magnitude and direction of the motion vector assigned to the second adjacent pixel (adjacent pixel with priority 6) from the left side of the top row, the upper right adjacent For the motion vector assigned to the pixel, the calculation of the evaluation value is skipped.

  Similarly, when the same priority order as the priority order shown in FIG. 48 is given to the target pixel or the adjacent pixel shown in FIG. 50, the left side adjacent pixel (priority order is 9) from the top row Since the magnitude and direction of the motion vector assigned to the (adjacent pixel) are the same as the magnitude and orientation of the motion vector assigned to the upper left adjacent pixel (adjacent pixel with priority 5), the second from the top The calculation of the evaluation value is skipped for the motion vector assigned to the adjacent pixel on the left side of the row.

  Furthermore, when the same priority order as the priority order shown in FIG. 48 is given to the target pixel or the adjacent pixel shown in FIG. 50, it is assigned to the lower right adjacent pixel (adjacent pixel having a priority order of 16). Since the magnitude and direction of the motion vector is the same as the magnitude and direction of the motion vector assigned to the right adjacent pixel (adjacent pixel having a priority of 12) in the third row from the top, the lower right adjacent For the motion vector assigned to the pixel, the calculation of the evaluation value is skipped.

  As described above, since the calculation of the evaluation value is skipped for the motion vector having the same size and orientation with respect to the motion vector for which the evaluation value has already been calculated, the amount of calculation can be further reduced.

  Note that the motion vectors having the same magnitude and orientation, that is, the motion vectors having the same magnitude and the same orientation are also referred to as the same motion vectors.

  Next, the results of experiments conducted by the applicant will be described.

  51 to 53 are diagrams showing results of experiments in the case where the compensation processing target block 1021 shown in FIG. 48 and the priority order are adopted.

  FIG. 51 is a diagram illustrating an example of the frequency of appearance of a target pixel or an adjacent pixel to which a motion vector is assigned for each priority. As described with reference to FIG. 49, the four pixels (adjacent pixels having a priority level of 5 to 8) in the upper row of the selected block to be compensated 1021 and the selected compensation Since motion vectors are assigned to the two pixels on the left side of the processing target block 1021 (adjacent pixels having a priority of 9 or 11), the adjacent ones having a priority of 5 to 8, or 9 or 11 A pixel appears as a pixel to which a motion vector is assigned.

  FIG. 52 is a diagram illustrating an example of the frequency of appearance of a target pixel or an adjacent pixel to which a motion vector having a magnitude or direction different from that of another motion vector is assigned for each priority. It can be said that the pixel of interest having a priority of 2 and the adjacent pixel having a priority of 5 are frequently assigned motion vectors having a magnitude or direction different from those of other motion vectors.

  FIG. 53 is a diagram illustrating an example of a frequency at which a motion vector to be assigned appears in a target pixel to which no motion vector is assigned for each priority. That is, FIG. 53 is a diagram illustrating the frequency of occurrence of a pixel of interest or an adjacent pixel to which a motion vector used for compensation is assigned for each priority. Due to the high spatial correlation of the image, the four target pixels belonging to the compensation processing target block 1021 are often assigned motion vectors having the same size and orientation. The motion vector assigned to the pixel is frequently used for compensation. In addition, the frequency with which motion vectors assigned to adjacent pixels with a priority of 5 are used for compensation is high.

  Next, the allocation compensation processing in the allocation compensation unit 57 corresponding to the processing in step S4 in FIG. 6 will be described with reference to the flowcharts in FIGS.

  In the following, the allocation compensation processing will be described by taking as an example the case where the compensation processing target block 1021 is composed of 2 × 2 pixels.

  In step S1001, the vector compensation unit 1001 sets (0, 0) as an initial value to the coordinates (x, y) for selecting the compensation processing target block 1021. In step S1002, the vector compensation unit 1001 determines a compensation processing target block 1021 of a frame to be processed based on the set coordinates (x, y). In step S1003, the vector compensation unit 1001 sets initial values for the compensation candidate vector memory 1003, the number of compensation candidate vectors, the priority, and the minimum evaluation value.

  Here, the number of compensation candidate vectors indicates the number of motion vectors for which evaluation values are calculated. The priority is a value for specifying the target pixel and the adjacent pixel whose evaluation values are to be calculated from the priority order assigned to the target pixel and the adjacent pixel. The minimum evaluation value is an evaluation value that shows the strongest correlation among the calculated evaluation values. When the evaluation value DFD that is the sum of absolute differences is adopted as the evaluation value, the minimum evaluation value is the smallest of the calculated evaluation values DFD. This is the evaluation value DFD.

  For example, in step S1003, the vector compensation unit 1001 clears the compensation candidate vector memory 1003, sets the number of compensation candidate vectors to 0, sets the priority to 1, and sets an upper limit value that can be set to the minimum evaluation value. By doing so, initial values are set to the compensation candidate vector memory 1003, the number of compensation candidate vectors, the priority, and the minimum evaluation value.

  In step S1004, the vector compensation unit 1001 determines whether or not there is a pixel of interest in which no motion vector is allocated in the compensation processing target block 1021, based on the allocation flag in the allocation flag memory 56. If it is determined that there is a pixel of interest that has not been assigned, the process proceeds to step S1005 to assign a motion vector to the pixel of interest, and the evaluation value calculation unit 1002 receives the evaluation value DFD0 based on the zero vector (still vector). Is calculated.

  For example, in step S1005, the vector compensation unit 1001 evaluates the evaluation value calculation unit 1002 based on the 0 vector (still vector) that intersects the target pixel with the priority “1” shown in FIG. The value DFD0 is calculated. That is, in this case, the evaluation value calculation unit 1002 includes the pixel value of the pixel belonging to the block centered on the pixel at the position corresponding to the position of the target pixel with the priority of “1” in the frame t, In the frame t + 1, an evaluation value that is a sum of absolute differences from a pixel value of a pixel that belongs to a block centered on a pixel at a position corresponding to the position of the target pixel with a priority of “1” is calculated.

  The evaluation value calculation unit 1002 supplies the evaluation value DFD0 calculated together with the 0 vector to the vector compensation unit 1001.

  In step S1006, the vector compensation unit 1001 sets the evaluation value DFD0 supplied from the evaluation value calculation unit 1002 as the minimum evaluation value. In step S1007, the vector compensation unit 1001 sets a zero vector as a compensation vector that is an internal variable.

  In step S <b> 1008, the vector compensation unit 1001 determines whether or not the allocation flag of the allocation flag memory 56 corresponding to the target pixel or the adjacent pixel having the priority indicated by the priority is “True”. If it is determined in step S1008 that the allocation flag of the allocation flag memory 56 corresponding to the target pixel of the priority indicated by the priority or the adjacent pixel is “True”, the target pixel of the priority indicated by the priority Alternatively, since a motion vector is assigned to an adjacent pixel, the process advances to step S1009 to calculate an evaluation value for the motion vector, and the vector compensation unit 1001 sets 0 as the initial value for the variable p. The variable p is a variable indicating the order of compensation candidate vectors that are motion vectors whose evaluation values have already been calculated.

  In step S1010, the vector compensation unit 1001 determines whether the variable p is larger than the number of compensation candidate vectors. More precisely, in step S1010, the vector compensation unit 1001 determines whether or not the number of compensation candidate vectors is not 0 and the variable p is larger than the number of compensation candidate vectors.

  If it is determined in step S1010 that the variable p is not larger than the number of compensation candidate vectors, it is not compared whether the same motion vector is assigned to the target pixel or the adjacent pixel having the priority indicated by the priority. Since there is a compensation candidate vector, the process proceeds to step S1011. The vector compensation unit 1001 determines that the motion vector assigned to the target pixel or the adjacent pixel of the priority order indicated by the priority is the compensation candidate vector. It is determined whether or not the compensation candidate vector stored in the memory 1003 is the same as the p-th compensation candidate vector.

  In step S1011, the motion vector assigned to the target pixel or adjacent pixel having the priority indicated by the priority is the same as the p-th compensation candidate vector among the compensation candidate vectors stored in the compensation candidate vector memory 1003. When it is determined that it is not, the process proceeds to step S1012, and the vector compensation unit 1001 increments the variable p. Then, the procedure returns to step S1010, and it is determined whether or not all the compensation candidate vectors have been compared with the motion vector assigned to the target pixel or the adjacent pixel having the priority indicated by the priority. When it is determined that the compensation candidate vector and the motion vector assigned to the target pixel or the adjacent pixel of the priority indicated by the priority are not compared, the next compensation candidate vector and the priority indicated by the priority The motion vector assigned to the target pixel or the adjacent pixel is compared.

  In this way, in Steps S1009 to S1012, it is determined whether or not the motion vector assigned to the target pixel or the adjacent pixel having the priority indicated by the priority is the same as one of the compensation candidate vectors.

  If it is determined in step S1010 that the variable p is larger than the number of compensation candidate vectors, the motion vector assigned to the target pixel or the adjacent pixel having the priority indicated by the priority is not the same as any of the compensation candidate vectors. In step S1013, the same vector determination unit 1004 supplies the motion vector assigned to the pixel of interest or the adjacent pixel having the priority indicated by the priority to the evaluation value calculation unit 1002, and the vector compensation unit 1001 The evaluation value calculation unit 1002 is caused to calculate an evaluation value DFD, which is an example of an evaluation value, based on the motion vector assigned to the target pixel or the adjacent pixel having the priority indicated by the priority.

  For example, in step S1013, the start point of the motion vector on the frame t, that is, the pixel value of the pixel belonging to the block centered on the target pixel or the adjacent pixel for calculating the evaluation value, and the end point of the frame t + 1 of the motion vector are centered. An evaluation value indicating a correlation with the pixel values of the pixels belonging to the block is calculated. Also, for example, the motion vector is moved so that the point where the motion vector and the frame to be processed (interpolated frame) intersect becomes a point in the compensation processing target block, and the motion vector frame t at that position is moved. An evaluation value indicating a correlation between the pixel value of the pixel belonging to the block centered on the start point and the pixel value of the pixel belonging to the block centered on the end point of the motion vector frame t + 1 at the position is calculated.

  In the calculation of the evaluation value, when the start point or the end point of the motion vector does not coincide with the pixel position, a four-point interpolation process using the luminance values of the surrounding four pixels may be used.

  In step S1013, the evaluation value calculation unit 1002 supplies a motion vector together with the calculated evaluation value DFD to the vector compensation unit 1001. Then, the evaluation value calculation unit 1002 supplies a compensation candidate vector, which is a motion vector obtained by calculating the evaluation value, to the compensation candidate vector memory 1003.

  In step S1014, the compensation candidate vector memory 1003 stores the compensation candidate vector supplied from the evaluation value calculation unit 1002. That is, in step S1014, the compensation candidate vector memory 1003 stores, as a compensation candidate vector, the motion vector assigned to the target pixel or the adjacent pixel having the priority indicated by the priority.

  In step S1015, the vector compensation unit 1001 increments the number of compensation candidate vectors.

  In step S1016, the vector compensation unit 1001 determines whether or not the evaluation value DFD calculated in the process of step S1013 is smaller than the minimum evaluation value, that is, the correlation indicated by the evaluation value calculated in the process of step S1013 is the minimum evaluation value. When the evaluation value DFD is determined to be smaller than the minimum evaluation value, that is, the correlation indicated by the calculated evaluation value is determined to be stronger than the correlation indicated by the minimum evaluation value. If YES in step S1017, the calculated evaluation value DFD is set as the minimum evaluation value.

  In step S1017, the evaluation value DFD showing the strongest correlation among the evaluation values calculated so far is set to the minimum evaluation value.

  In step S1018, the vector compensation unit 1001 sets the motion vector assigned to the target pixel or the adjacent pixel having the priority indicated by the priority as the compensation vector. Thereby, the motion vector in which the evaluation value DFD indicating the strongest correlation among the evaluation values DFD so far is calculated is set as the compensation vector.

  In step S1019, the vector compensation unit 1001 increments the priority. In step S1020, the vector compensation unit 1001 determines whether the priority has exceeded 16 or the number of compensation candidate vectors has exceeded a predetermined threshold.

  In step S1020, when it is determined that the priority is 16 or less and the number of compensation candidate vectors is less than or equal to a predetermined threshold, the procedure returns to step S1008, and the next priority pixel of interest or adjacent pixel is selected. The above-described processing is repeated for the assigned motion vector.

  If it is determined in step S1020 that the priority has exceeded 16 or the number of compensation candidate vectors has exceeded a predetermined threshold, the processing of steps S1008 to S1018 is executed for all the target pixels and adjacent pixels. Since the motion vector from which the evaluation value indicating the strongest correlation is obtained is set as the compensation vector among the predetermined number of motion vectors determined by the threshold value, the process proceeds to step S1021, and the vector compensation unit 1001 performs compensation processing. A compensation vector is assigned to all target pixels to which no motion vector is assigned in the target block 1021. For example, in step S1021, the vector compensation unit 1001 causes the allocation vector memory 55 to store a motion vector that is a compensation vector in association with a target pixel to which no motion vector of the interpolation frame is allocated. More precisely, the allocation vector memory 55 stores a motion vector having the same size and direction as the compensation vector in correspondence with the target pixel to which no motion vector of the interpolation frame is allocated.

  The threshold to be compared with the number of compensation candidate vectors can be an arbitrary value, for example, 5.

  In step S1022, the vector compensation unit 1001 rewrites the allocation flag of the allocation flag memory 56 corresponding to the pixel to which the motion vector is allocated to “True” (for example, “1”).

  In step S1023, the vector compensation unit 1001 adds 2 to x of coordinates (x, y) for selecting the compensation processing target block 1021. As a result, the compensation processing target block 1021 to be selected next moves two pixels in the right direction of the frame.

  In step S1024, the vector compensation unit 1001 determines whether x exceeds the image frame in the x direction of the frame, and if it is determined that x does not exceed the image frame in the x direction of the frame, the procedure Return to step S1002 and repeat the above-described processing for the right compensation processing target block 1021 in the frame.

  If it is determined in step S1024 that x exceeds the image frame in the x direction of the frame, since there is no pixel at that position, the pixel on the left side of the frame and further below is set as the compensation processing target block 1021. Next, in order to make a selection, the process proceeds to step S1025, and the vector compensation unit 1001 sets 0 to x of coordinates (x, y) for selecting the compensation processing target block 1021, and adds 2 to y.

  In step S1026, the vector compensation unit 1001 determines whether y exceeds the image frame in the y direction of the frame, and if it is determined that y does not exceed the image frame in the y direction of the frame, the procedure Returns to step S1002 and repeats the above-described processing for the lower compensation processing target block 1021 on the left side of the frame.

  In other words, the compensation processing target block 1021 is selected in the raster scan order by the processing in steps S1023 to S1026.

  If it is determined in step S1026 that y has exceeded the image frame in the y direction of the frame, since the allocation compensation processing has been executed for all the pixels of the frame, the processing ends.

  In step S1004, when it is determined that there is no pixel of interest in which no motion vector is assigned in the compensation processing target block 1021, there is no need to execute processing for assigning a motion vector to the compensation processing target block 1021. Proceeding to step S1023, the above-described processing is repeated for the next compensation processing target block 1021.

  If it is determined in step S1008 that the allocation flag of the allocation flag memory 56 corresponding to the target pixel or adjacent pixel having the priority indicated by the priority is not “True”, the target pixel or adjacent pixel includes a motion vector. Is not allocated, the process proceeds to step S1019, and the process of calculating the evaluation value of the allocated motion vector is executed for the next pixel of interest or adjacent pixel.

  In step S1011, the motion vector assigned to the target pixel or adjacent pixel having the priority indicated by the priority is the same as the p-th compensation candidate vector among the compensation candidate vectors stored in the compensation candidate vector memory 1003. If it is determined that the evaluation value is determined, it is not necessary to calculate the evaluation value for the same motion vector as the motion vector for which the evaluation value has already been calculated. Therefore, the process proceeds to step S1019, and the next target pixel or adjacent pixel is allocated. A process for calculating the evaluation value of the motion vector being executed is executed.

  If it is determined in step S1016 that the calculated evaluation value DFD is not smaller than the minimum evaluation value, there are other motion vectors for which an evaluation value indicating a stronger correlation than the motion vector is calculated, so steps S1017 and S1018 are used. This process is skipped, and the process proceeds to step S1019 to execute the process of calculating the evaluation value of the allocated motion vector for the next target pixel or adjacent pixel.

  In this way, the allocation compensator 57 can allocate the motion vector predicted to be most accurate with a smaller amount of computation to the pixels for which the motion vector has not been allocated by the vector allocation unit 54. That is, the allocation compensator 57 generates a motion vector with fewer failures in generating a pixel value when generating a pixel value of an interpolation frame for a pixel for which a motion vector has not been allocated by the vector allocation unit 54. Can be assigned.

  According to the present invention, it is possible to allocate the motion vector predicted to be the most accurate with a small amount of calculation, and as a result, the processing is performed by hardware using a smaller circuit scale and a slower circuit. It becomes possible.

  As described above, the evaluation value indicating the correlation of the pixel at the position indicated by the motion vector is obtained, the motion vector is selected based on the evaluation value, and the selected motion vector is assigned to the pixel to which no motion vector is assigned. When assigned, a motion vector can be assigned to a pixel to which no motion vector has been assigned in the previous processing.

  In addition, for each target block that is a target block on the third frame, which is composed of a plurality of target pixels that are adjacent to each other on the third frame, Motion vectors assigned to the target pixel or the adjacent pixels in the order of priority assigned to the adjacent pixel and the target pixel, which are pixels adjacent to the target block. The pixel value of the pixel of the first frame belonging to the first area centered on the start point in the first frame and the second value belonging to the second area centered on the end point of the motion vector in the second frame When the evaluation value indicating the correlation with the pixel value of the pixel in the frame is calculated and the evaluation value is calculated for a predetermined number of motion vectors, the motion value of the target pixel in the target block is calculated. When a motion vector with an evaluation value indicating the strongest correlation is assigned to a target pixel to which no vector is assigned, the number of pixels to which no motion vector is assigned in the previous processing is smaller. A motion vector can be assigned with a calculation amount.

  In addition, the allocation flag of the pixel to which the motion vector is allocated by the allocation compensation process described above is rewritten to 1 (True), and the motion vector allocated by the allocation compensation process is also used as a compensation candidate vector for the next pixel. Therefore, almost the same motion vector is selected for pixels that perform substantially the same motion in the object, and a stable motion vector with few errors can be obtained. As a result, block noise or the like of an image generated in the subsequent stage can be suppressed and the quality can be improved.

  In addition, for a pixel of interest for which a motion vector is to be calculated, an evaluation value DFD of 0 vector is calculated in advance and stored in a memory as a minimum evaluation value, so that all compensation candidate vector evaluation values DFD are calculated at once. Thus, time and resources can be used more efficiently than when the smallest evaluation value DFD is selected.

  In the above description, the vector compensation processing is performed on the pixels that are not assigned by the vector assigning unit 54, but there are some kinds of pixels such as pixels that are not detected by the vector detecting unit 52 (the zero vector is detected). You may make it perform a vector compensation process with respect to the pixel in which the motion vector was not calculated | required in the process. Furthermore, vector compensation processing may be performed on pixels for which the detected motion vector or the assigned motion vector is uncertain (low reliability).

  In this embodiment, the evaluation value DFD that is the sum of absolute differences is used as the evaluation value when selecting a motion vector. However, the evaluation value DFD is not limited to the evaluation value DFD, and the reliability of the motion vector is evaluated. If it is a thing, you may make it use another thing.

  In the present embodiment, the block for performing each process has been described as being configured by, for example, 8 pixels × 8 pixels, 9 pixels × 9 pixels, and the like. However, these are examples, and each process is performed. The number of pixels constituting the block to be performed is not limited to the number of pixels.

  Furthermore, in the present embodiment, the signal conversion from the 24P signal to the 60P signal has been described as an example. However, the present invention can be applied to, for example, interlaced signals and other frame rates as frame frequency conversion of moving images. It can also be applied to conversion.

  The series of processes described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program storage medium in a general-purpose personal computer or the like.

  As shown in FIG. 1, a program storage medium that stores a program that is installed in a computer and can be executed by the computer includes a magnetic disk 31 (including a flexible disk), an optical disk 32 (CD-ROM (Compact Disc -Removable recording media (package media) comprising a read only memory (including DVD (Digital Versatile Disc)), magneto-optical disc 33 (including MD (Mini-Disc) (trademark)), or semiconductor memory 34, or The ROM 12 stores the program temporarily or permanently.

  In the present specification, the steps shown in the flowcharts include not only processes performed in time series according to the order described, but also processes executed in parallel or individually even if not necessarily performed in time series. Is included.

It is a block diagram which shows the structural example of the signal processing apparatus of this invention. It is a block diagram which shows the structure of a signal processing apparatus. It is a figure explaining the principle of the process of this invention. It is a figure explaining the process of this invention concretely. It is a figure explaining the evaluation value of the motion vector used in a signal processor. It is a flowchart explaining the frame frequency conversion process of a signal processing apparatus. It is a block diagram which shows the structure of an image interpolation part. It is a flowchart explaining an image interpolation process. It is a block diagram which shows the structure of a vector detection part. It is a figure explaining the gradient method used in a vector detection part. It is a figure explaining the iterative gradient method using an initial vector. It is a flowchart explaining a motion vector detection process. It is a block diagram which shows the structure of a shift initial vector allocation part. It is a figure explaining the concept of a shift initial vector. It is a figure explaining a shift initial vector concretely. It is a figure explaining a shift initial vector concretely. It is a figure explaining the determination method of a shift initial vector. It is a figure explaining the example of allocation of a shift initial vector. It is a figure explaining the example of a shift initial vector in case a motion vector competes. It is a flowchart explaining a shift initial vector allocation process. It is a block diagram which shows the structure of an initial vector selection part. It is a figure which shows the peripheral area | region of an initial vector. It is a figure which shows the candidate block of an initial vector. It is a flowchart explaining an initial vector selection process. It is a block diagram which shows the structure of an iterative gradient method calculating part. It is a block diagram which shows the structure of an effective pixel determination part. It is a block diagram which shows the structure of a gradient method calculating part. It is a figure explaining the detection target block and calculation block of a motion vector. It is a figure explaining the motion of the object in a detection target block. It is a figure explaining the state of the brightness | luminance in the detection target block which has the object of the same motion. It is a figure explaining the state of the brightness | luminance in the detection target block which has an object of a different motion. It is a figure explaining the example of the calculation block of a pixel unit process. It is a figure explaining the pixel unit process in the detection object block of FIG. It is a flowchart explaining an iterative gradient method calculation process. It is a flowchart explaining an effective pixel determination process. It is a flowchart explaining a gradient method calculation process. It is a flowchart explaining the iterative gradient method calculation process of a pixel unit. It is a block diagram which shows the structure of a vector allocation part. It is a figure explaining the concept of the 4-point interpolation process of this invention. It is a figure explaining the outline | summary of a vector allocation process. It is a figure explaining the example of the neighborhood pixel of the intersection of a motion vector and an interpolation frame. It is a figure explaining the evaluation method of the motion vector of an allocation candidate. It is a figure explaining the example of 4-point interpolation in vector allocation. It is a flowchart explaining a vector allocation process. It is a flowchart explaining a pixel position calculation process. It is a flowchart explaining the allocation vector evaluation process. It is a block diagram which shows the structure of an allocation compensation part. It is a figure which shows the example of a priority. It is a figure explaining allocation of a motion vector. It is a figure explaining the skip of the calculation of an evaluation value. It is a figure which shows the example of the frequency that the attention pixel to which a motion vector is allocated, or an adjacent pixel appears. It is a figure which shows the example of the frequency which the attention pixel or adjacent pixel to which the motion vector of the magnitude | size or direction different from another motion vector is allocated appears. It is a figure which shows the example of the frequency where the motion vector allocated appears to the attention pixel to which the motion vector is not allocated. It is a flowchart explaining an allocation compensation process. It is a flowchart explaining an allocation compensation process.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Signal processing apparatus, 11 CPU, 12 ROM, 13 RAM, 31 Magnetic disk, 32 Optical disk, 33 Magneto-optical disk, 34 Semiconductor memory, 51 Frame memory, 52 Vector detection part, 53 Detection vector memory, 54 Vector allocation part, 55 Allocation vector memory, 56 Allocation flag memory, 57 Allocation compensation unit, 58 Image interpolation unit, 101 Initial vector selection unit, 103 Iterative gradient method calculation unit, 104 Vector evaluation unit, 105 Shift initial vector allocation unit, 106 Evaluation value memory, 107 Shift initial vector memory, 201 allocation target position calculation unit, 202 evaluation value comparison unit, 203 shift initial vector replacement unit, 204 evaluation value replacement unit, 251 candidate vector position calculation unit, 252 detection vector acquisition unit, 253 shift initial vector acquisition unit , 254 Offset position calculation unit, 255 evaluation value calculation unit, 256 evaluation value comparison unit, 257 optimum candidate storage register, 401 mode selection unit, 402 selector, 403 effective pixel determination unit, 404 gradient method calculation unit, 405 delay unit, 411 Time pixel difference calculation unit, 412 pixel difference value determination unit, 413 effective pixel number counter, 414 gradient method continuation determination unit, 421 time pixel difference calculation unit, 422 pixel difference value determination unit, 423 horizontal / vertical pixel difference calculation unit, 424 gradient Integration unit, 425 vector calculation unit, 701 pixel information calculation unit, 702 evaluation value calculation unit, 703 focused pixel difference calculation unit, 704 vector evaluation unit, 705 vector selection unit, 711 pixel difference determination unit, 712 evaluation value determination unit, 1001 Vector compensation unit, 1002 Evaluation value computation unit, 1003 Compensation candidate Kutorumemori, 1004 same vector determination unit

Claims (8)

A motion vector detected based on the first frame and the second frame, the motion vector starting from the first frame and ending at the second frame; the first frame and the second frame; Assigning the motion vector to a pixel on the third frame in the vicinity of a point where a third frame arranged between two frames intersects, and based on the assigned motion vector, In the image processing apparatus for generating the pixel value of the pixel of the third frame,
For each target block that is a target block on the third frame, the target block being a target pixel that is a plurality of target pixels that are adjacent to each other on the third frame. Neighboring pixels, which are assigned to the pixel of interest or the adjacent pixel in the order of priority assigned to the pixel of interest and adjacent pixels that are arranged adjacent to the block of interest For a motion vector, a pixel value of a pixel of the first frame belonging to a first region centered on a start point of the motion vector in the first frame, and an end point of the motion vector in the second frame An evaluation value calculating means for calculating an evaluation value indicating a correlation with a pixel value of a pixel of the second frame belonging to the second region as a center;
When the evaluation value is calculated for a predetermined number of the motion vectors, the evaluation indicating that the correlation among the target pixels of the target block to which the motion vector is not allocated is the strongest among the target pixels of the target block. Allocating means for allocating the motion vector whose value is calculated ;
Determining means for determining whether the magnitude and direction of the motion vector for which the evaluation value is to be calculated are the same as the magnitude and direction of the motion vector for which the evaluation value has already been calculated;
Including
When the evaluation value calculation means determines that the magnitude and direction of the motion vector for which the evaluation value is to be calculated are the same as the magnitude and direction of the motion vector for which the evaluation value has already been calculated , Skip the calculation of the evaluation value for the motion vector
The image processing apparatus characterized by. The evaluation value calculation means calculates the evaluation value for the motion vector assigned by the assignment means in the processing for another block of interest.
The image processing apparatus according to claim 1. The evaluation value calculation means moves, in the order of priority, a point where the motion vector assigned to the target pixel or the adjacent pixel intersects the third frame to a point in the target block. A pixel value of a pixel belonging to the first region centered on the start point of the first frame of the motion vector at the position, and the end point of the second frame of the motion vector at the position. The image processing apparatus according to claim 2 , wherein the evaluation value indicating a correlation with a pixel value of a pixel belonging to the second region at the center is calculated. The evaluation value calculation means is the priority that the pixel of interest has priority over the adjacent pixel, the priority of the pixel of interest corresponding to the raster scan order of the pixel of interest, and the raster of the adjacent pixel The image processing according to claim 2 , wherein the evaluation value is calculated for the motion vector assigned to the target pixel or the adjacent pixel in the order of the priority of the adjacent pixels corresponding to a scan order. apparatus. The evaluation value calculating means, wherein, characterized in that computing the evaluation value difference is an absolute value sum of the pixel values of the pixels belonging to the first of said pixel values of the pixels belonging to the region second region Item 3. The image processing apparatus according to Item 2 . A motion vector detected based on the first frame and the second frame, the motion vector starting from the first frame and ending at the second frame; the first frame and the second frame; Assigning the motion vector to a pixel on the third frame in the vicinity of a point where a third frame arranged between two frames intersects, and based on the assigned motion vector, In an image processing method for generating a pixel value of a pixel of a third frame,
For each target block that is a target block on the third frame, the target block being a target pixel that is a plurality of target pixels that are adjacent to each other on the third frame. Neighboring pixels, which are assigned to the pixel of interest or the adjacent pixel in the order of priority assigned to the pixel of interest and adjacent pixels that are arranged adjacent to the block of interest For a motion vector, a pixel value of a pixel of the first frame belonging to a first region centered on a start point of the motion vector in the first frame, and an end point of the motion vector in the second frame An evaluation value calculating step of calculating an evaluation value indicating a correlation with a pixel value of a pixel of the second frame belonging to the second region as a center;
When the evaluation value is calculated for a predetermined number of the motion vectors, the evaluation indicating that the correlation among the target pixels of the target block to which the motion vector is not allocated is the strongest among the target pixels of the target block. An assigning step for assigning the motion vector whose value has been calculated ;
A determination step for determining whether the magnitude and direction of the motion vector for which the evaluation value is to be calculated are the same as the magnitude and direction of the motion vector for which the evaluation value has already been calculated;
Including
When it is determined in the evaluation value calculation step that the magnitude and direction of the motion vector for which the evaluation value is to be calculated are the same as the magnitude and direction of the motion vector for which the evaluation value has already been calculated , Skip the calculation of the evaluation value for the motion vector
Image processing method, characterized in that. A motion vector detected based on the first frame and the second frame, the motion vector starting from the first frame and ending at the second frame; the first frame and the second frame; Assigning the motion vector to a pixel on the third frame in the vicinity of a point where a third frame arranged between two frames intersects, and based on the assigned motion vector, An image processing program for generating a pixel value of a pixel of a third frame,
For each target block that is a target block on the third frame, the target block being a target pixel that is a plurality of target pixels that are adjacent to each other on the third frame. Neighboring pixels, which are assigned to the pixel of interest or the adjacent pixel in the order of priority assigned to the pixel of interest and adjacent pixels that are arranged adjacent to the block of interest For a motion vector, a pixel value of a pixel of the first frame belonging to a first region centered on a start point of the motion vector in the first frame, and an end point of the motion vector in the second frame An evaluation value calculating step of calculating an evaluation value indicating a correlation with a pixel value of a pixel of the second frame belonging to the second region as a center;
When the evaluation value is calculated for a predetermined number of the motion vectors, the evaluation indicating that the correlation among the target pixels of the target block to which the motion vector is not allocated is the strongest among the target pixels of the target block. An assigning step for assigning the motion vector whose value has been calculated ;
A determination step for determining whether the magnitude and direction of the motion vector for which the evaluation value is to be calculated are the same as the magnitude and direction of the motion vector for which the evaluation value has already been calculated;
Including
When it is determined in the evaluation value calculation step that the magnitude and direction of the motion vector for which the evaluation value is to be calculated are the same as the magnitude and direction of the motion vector for which the evaluation value has already been calculated , Skip the calculation of the evaluation value for the motion vector
Recording medium having a computer is recorded readable program characterized by. A motion vector detected based on the first frame and the second frame, the motion vector starting from the first frame and ending at the second frame; the first frame and the second frame; Assigning the motion vector to a pixel on the third frame in the vicinity of a point where a third frame arranged between two frames intersects, and based on the assigned motion vector, In a program for causing a computer to perform image processing for generating a pixel value of a pixel of a third frame,
For each target block that is a target block on the third frame, the target block being a target pixel that is a plurality of target pixels that are adjacent to each other on the third frame. Neighboring pixels, which are assigned to the pixel of interest or the adjacent pixel in the order of priority assigned to the pixel of interest and adjacent pixels that are arranged adjacent to the block of interest For a motion vector, a pixel value of a pixel of the first frame belonging to a first region centered on a start point of the motion vector in the first frame, and an end point of the motion vector in the second frame An evaluation value calculating step of calculating an evaluation value indicating a correlation with a pixel value of a pixel of the second frame belonging to the second region as a center;
When the evaluation value is calculated for a predetermined number of the motion vectors, the evaluation indicating that the correlation among the target pixels of the target block to which the motion vector is not allocated is the strongest among the target pixels of the target block. An assigning step for assigning the motion vector whose value has been calculated ;
A determination step for determining whether the magnitude and direction of the motion vector for which the evaluation value is to be calculated are the same as the magnitude and direction of the motion vector for which the evaluation value has already been calculated;
Including
When it is determined in the evaluation value calculation step that the magnitude and direction of the motion vector for which the evaluation value is to be calculated are the same as the magnitude and direction of the motion vector for which the evaluation value has already been calculated , Skip the calculation of the evaluation value for the motion vector
Program, characterized in that.

Images