JP4655215B2 - Image processing apparatus and method, and program - Google Patents

Description

  The present invention relates to an image processing apparatus and method, and a program, and more particularly, to an image processing apparatus and method, and a program capable of selecting a motion vector with higher accuracy.

  In an apparatus for up-converting and outputting the frame frequency of an input image (frame), an interpolated frame to be inserted between those frames is generated from the input frame. The input frame is output, and the generated interpolation frame is output at a predetermined timing between the input frames, thereby realizing frame frequency up-conversion.

  In this interpolation frame, for example, a motion vector is allocated to each pixel constituting the interpolation frame, and each pixel constituting the interpolation frame is composed of pixels on two input frames associated with the allocated motion vector. Generated by interpolating pixels.

  Therefore, the accuracy of the motion vector assigned to each pixel constituting the interpolated frame greatly affects the quality of the output image. For example, when a moving image whose content is such that an object of a certain shape moves at a constant speed from one side of the image frame to the other is input and the frame frequency is up-converted and output, it is based on a motion vector with low accuracy. When the interpolation frame generated in this way is output, the movement of the object does not appear to be uniform, or the object appears to move while deforming.

As a method for detecting a motion vector assigned to each pixel constituting the interpolation frame, for example, there are a gradient method and a block matching method as described in Patent Document 1.
JP 60-158786 A

  As described above, since the quality of the output image is affected, it is preferable that the motion vector assigned to each pixel constituting the interpolation frame has higher accuracy.

  For example, when a motion vector detected by repeated gradient method calculation is assigned to each pixel constituting the interpolation frame, the motion vector obtained as a detection result by repeated gradient method calculation is As the processing proceeds, the accuracy becomes higher. Therefore, it is preferable that the motion vector obtained after the processing proceeds is assigned to each pixel constituting the interpolation frame.

  The present invention has been made in view of such a situation, and makes it possible to select a motion vector with higher accuracy.

  An image processing apparatus according to one aspect of the present invention focuses on each region included in a first frame in raster order or reverse raster order, detects a motion vector by a gradient method, and generates a second target to be generated. The detection means pays attention to each area included in the frame in the order opposite to the order in which the detection means pays attention to the area, and one motion vector among the motion vectors assigned to the surrounding areas is assigned to the attention area. Allocating means for allocating.

  The first calculation target range of the first frame and the first calculation unit associated with the first calculation target range by a motion vector assigned to a region around the region of interest in the second frame. There can be further provided a calculation means for obtaining a correlation with the second calculation target range of the third frame temporally after this frame. In this case, the allocating unit can select one motion vector from among the motion vectors allocated to the peripheral region based on the correlation obtained by the calculating unit.

  Within the first calculation target range obtained by subtracting the average value of the pixel values of the pixels within the first calculation target range from the pixel value of each pixel within the first calculation target range of the first frame. After the first frame associated with the first calculation target range by the pixel value of each pixel and the motion vector assigned to the area around the area of interest in the second frame In the second calculation target range obtained by subtracting the average value of the pixel values of the pixels in the second calculation target range from the pixel values of the pixels in the second calculation target range of the third frame A calculation means for obtaining a correlation between the first and second calculation target ranges from the pixel value of each pixel can be further provided. In this case, the allocating unit can select one motion vector from among the motion vectors allocated to the peripheral region based on the correlation obtained by the calculating unit.

  An information processing method or program according to one aspect of the present invention focuses on each region included in a first frame in raster order or reverse raster order, detects a motion vector by a gradient method, and generates a second target to be generated. Pay attention to each area included in the frame in the opposite order to the order in which the area is focused when detecting the motion vector, and one of the motion vectors assigned to the surrounding area in the focused area Including assigning a vector.

  In one aspect of the present invention, attention is paid to each region included in the first frame in raster order or reverse raster order, and a motion vector is detected by a gradient method. In addition, attention is paid to each area included in the second frame to be generated in an order opposite to the order in which the areas are noticed when the motion vector is detected, and the attention areas are assigned to the surrounding areas. One of the motion vectors being assigned is assigned.

  According to one aspect of the present invention, a motion vector with higher accuracy can be selected.

  Embodiments of the present invention will be described below. Correspondences between the configuration requirements of the present invention and the embodiments described in the detailed description of the present invention are exemplified as follows. This description is to confirm that the embodiments supporting the present invention are described in the detailed description of the invention. Accordingly, although there are embodiments that are described in the detailed description of the invention but are not described here as embodiments corresponding to the constituent elements of the present invention, It does not mean that the embodiment does not correspond to the configuration requirements. On the contrary, even if an embodiment is described herein as corresponding to the invention, this does not mean that the embodiment does not correspond to other than the configuration requirements. .

  An image processing apparatus according to an aspect of the present invention focuses on each region (block, pixel) included in a first frame (for example, a frame at time t) in raster order or reverse raster order, and uses a gradient method to calculate a motion vector. The detection means (for example, the vector detection unit 12 in FIG. 1) and the order in which the detection means pays attention to the respective areas included in the second frame (for example, the interpolation frame) to be generated And allocating means (for example, the allocation compensator 17 in FIG. 1) that allocates one motion vector among the motion vectors allocated to the peripheral region to the focused region.

  The first calculation target range of the first frame and the first calculation unit associated with the first calculation target range by a motion vector assigned to a region around the region of interest in the second frame. 36 (for example, the evaluation value of FIG. 36) for obtaining a correlation (for example, the sum of absolute differences represented by the expression (1)) with the second calculation target range of the third frame temporally after the frame of A calculation unit 142) can be further provided. In this case, the allocating unit can select one motion vector from among the motion vectors allocated to the peripheral region based on the correlation obtained by the calculating unit.

  Obtained by subtracting the average value of the pixel values of the pixels within the first calculation target range from the pixel values of the pixels within the first calculation target range set with the predetermined position of the first frame as a reference. Focused on the second frame from the position corresponding to the predetermined position of the third frame temporally after the first frame and the pixel value of each pixel in the first calculation target range Obtained by subtracting the average value of the pixel values of the pixels in the second calculation target range from the pixel values of the pixels in the second calculation target range obtained according to the motion vector assigned to the area around the area. Calculating means (for example, a deviation DFD represented by the equation (13)) between the first calculation target range and the pixel value of each pixel in the second calculation target range. 36, the evaluation value calculation unit 142) of FIG. It can be provided to. In this case, the allocating unit can select one motion vector from among the motion vectors allocated to the peripheral region based on the correlation obtained by the calculating unit.

  An information processing apparatus or program according to one aspect of the present invention focuses on each region (block, pixel) included in a first frame (for example, a frame at time t) in raster order or reverse raster order, and uses a gradient method. Attention is paid to each region included in the step of detecting a motion vector (for example, step S2 in FIG. 5) and each region included in the second frame (for example, interpolation frame) to be generated. And a step of assigning one of the motion vectors assigned to the surrounding region to the region of interest (for example, step S4 in FIG. 5).

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 is a block diagram illustrating a configuration example of a signal processing device 1 according to an embodiment of the present invention. Each configuration shown in FIG. 1 may be realized by hardware or may be realized by software.

  In the signal processing device 1, for example, a progressive image signal (24P signal) with a frame frequency of 24 Hz is input, and the input signal is converted into a progressive image signal (60P signal) with a frame frequency of 60 Hz and output. The input image of the 24P signal input to the signal processing device 1 is supplied to the frame memory 11, the vector detection unit 12, the vector allocation unit 14, the allocation compensation unit 17, and the image interpolation unit 18.

  The frame memory 11 stores a frame at time t, which is a frame at the time immediately before the frame (image) at time t + 1. The frame at time t stored in the frame memory 11 is output to the vector detection unit 12, the vector allocation unit 14, the allocation compensation unit 17, and the image interpolation unit 18. Hereinafter, the frame at time t is referred to as frame t, and the frame at time t + 1 is referred to as frame t + 1.

  The vector detection unit 12 detects a motion vector between the target block of the frame t stored in the frame memory 11 and the target block of the frame t + 1, and stores the detected motion vector in the detection vector memory 13. A gradient method or a block matching method is used to detect the motion vector between the two frames. Details of the configuration of the vector detection unit 12 will be described later.

  The detection vector memory 13 stores the motion vector detected by the vector detection unit 12 with respect to the frame t.

  The vector allocation unit 14 allocates the motion vector obtained on the frame t of the 24P signal to the pixel on the frame of the 60P signal to be interpolated, which is stored in the allocation vector memory 15. Hereinafter, the frame of the 60P signal is also referred to as an interpolation frame as appropriate.

  Further, when the motion vector is allocated to the interpolation frame, the vector allocation unit 14 rewrites the allocation flag of the pixel allocated with the motion vector stored in the allocation flag memory 16 to 1 (True). The allocation flag indicates whether or not the motion vector of each pixel is allocated. For example, an allocation flag of True (1) indicates that a motion vector is allocated to the corresponding pixel, and an allocation flag of False (0) indicates that a motion vector is not allocated to the corresponding pixel. Details of the configuration of the vector allocation unit 14 will be described later.

  The allocation vector memory 15 stores a motion vector (hereinafter, appropriately referred to as an allocation vector) allocated to the pixels of the interpolation frame by the vector allocation unit 14.

  The allocation flag memory 16 stores an allocation flag for each pixel of the interpolation frame.

  The allocation compensator 17 refers to the allocation flag stored in the allocation flag memory 16, and when a pixel to which no motion vector is allocated by the vector allocation unit 14 is in the interpolation frame, the motion of the surrounding pixels is included in that pixel. Allocate (complement) a vector. When a motion vector is newly allocated, the allocation compensator 17 rewrites the allocation flag of the pixel allocated with the motion vector stored in the allocation flag memory 16 to 1 (True). Details of the configuration of the allocation compensation unit 17 will be described later.

  The image interpolation unit 18 uses the motion vector stored in the allocation vector memory 15 and allocated to each pixel of the interpolation frame and the pixel values (for example, luminance values) of the frames t and t + 1. The pixel value of the interpolation frame is generated by interpolation. The image interpolation unit 18 outputs the generated interpolation frame, and then outputs the frame t + 1 as necessary, thereby outputting the 60P signal to the subsequent stage.

  FIG. 2 is a diagram illustrating an example of frame interpolation.

  In FIG. 2, a dotted line represents a frame of a 24P signal at time t, t + 1, t + 2 input to the signal processing device 1, and a solid line is generated and output in the signal processing device 1. It represents a frame of 60P signal at time t, t + 0.4, t + 0.8, t + 1.2, t + 1.6, t + 2.

  Generally, in order to convert a 24P signal to a 60P signal, it is necessary to generate 60/24 (5/2) times frames, that is, five frames from two input frames. At this time, the interpolated frames of the 60P signal are arranged at positions where the time phases on the 24P signal are 0.0, 0.4, 0.8, 1.2, and 1.6 in order to equalize the respective frame intervals. Among these, four frames (frames of t + 0.4, t + 0.8, t + 1.2, and t + 1.6) except for one frame at time t with a time phase of 0.0 are images that do not exist on the 24P signal. .

  Therefore, in the signal processing device 1, when an image of the 24P signal is input, four interpolation frames are generated from the two frames at the time t and the time t + 1 of the 24P signal, and the time t, t + 0.4, A 60P signal composed of five frames t + 0.8, t + 1.2, and t + 1.6 is output.

  Thereby, the frame frequency conversion from the 24P signal to the 60P signal is realized.

  In principle, there are five 60P signal frames at times t, t + 0.4, t + 0.8, t + 1.2, and t + 1.6 from two frames at time t and time t + 1 of the 24P signal. In the case of the example of FIG. 2, a 60P signal frame of t, t + 0.4, t + 0.8 is actually generated based on two frames of time t and time t + 1 of the 24P signal. Based on the two frames at the times t + 1 and t + 2 of the 24P signal, frames of the 60P signal of t + 1.2, t + 1.6, and t + 2 are generated.

  FIG. 3 is a diagram for explaining a series of processes performed in the signal processing apparatus 1.

  In FIG. 3, a thick arrow represents a state transition, and an arrow T represents a time passage direction in each state. In addition, states S1 to S5 indicate that the 24P signal is in frame t, frame t + 1, or between frame t and frame t + 1 at the time of input to each part of signal processing apparatus 1 and at the time of output from each part. The state of the interpolation frame F of the 60P signal produced | generated is shown. That is, in practice, for example, a frame in which a motion vector is detected as shown in the state S2 is not input, and the frame and the detected motion vector are input separately.

  The state S1 represents the state of the frames t and t + 1 of the 24P signal input to the vector detection unit 12. Black dots on the frame t represent pixels.

  The vector detection unit 12 detects to which position the pixel on the frame t moves in the frame t + 1 at the next time, and outputs the detected motion as a motion vector corresponding to each pixel. When a plurality of motion vectors are detected as corresponding to one pixel, the vector detection unit 12 obtains an evaluation value for each motion vector, and selects one motion vector based on the obtained evaluation value.

  The state S2 represents the state of the frames t and t + 1 that are input to the vector allocation unit 14. In the state S2, an arrow extending from each pixel of the frame t represents a motion vector detected by the vector detection unit 12.

  The vector allocating unit 14 extends the motion vector detected at each pixel of the frame t to the next frame t + 1, and performs interpolation at a preset time phase (for example, t + 0.4 in FIG. 2). Which position on the frame F is passed through is determined. This is because an interpolation frame in which a point at which a motion vector passes through the interpolation frame F corresponds to the certain pixel when a certain pixel takes a certain motion between the frame t and the frame t + 1. This is because the pixel position is F.

  For example, the vector assigning unit 14 assigns the passing motion vector to four pixels in the vicinity of the passing point on the interpolation frame F. At this time, depending on the pixels of the interpolated frame, there may be no motion vector or a plurality of motion vectors may be allocation candidates. In the latter case, the vector allocation unit 14 An evaluation value for the vector is obtained, and one motion vector to be assigned is selected based on the obtained evaluation value.

  The state S3 represents the states of the frames t and t + 1 and the interpolation frame F that are input to the allocation compensator 17. In the interpolation frame F in the state S3, there are pixels to which a motion vector is assigned and pixels to which no motion vector is assigned.

  The allocation compensation unit 17 compensates for a pixel to which no motion vector is allocated, using the motion vector allocated to the peripheral pixels of the pixel. This is because the motion vector of the pixel of interest is considered to be similar to the motion vector of the surrounding pixels if the assumption that a region near the pixel of interest has the same motion as that of the pixel of interest holds.

  In this way, by assigning motion vectors of peripheral pixels, a motion vector that is accurate to some extent is also given to pixels that have not been assigned a motion vector, and motion vectors are assigned to all the pixels of the interpolation frame F. When a plurality of motion vectors exist as motion vector candidates for the pixel of interest, the allocation compensator 17 obtains an evaluation value for each motion vector and selects one motion vector to be assigned based on the obtained evaluation value. .

  The state S4 represents the state of the frames t and t + 1 and the interpolation frame F that are input to the image interpolation unit 18. In the interpolation frame F in the state S4, motion vectors are assigned to all the pixels.

  As described above, since the motion vector is assigned to all the pixels, the image interpolation unit 18 can determine the positional relationship between the pixels on the interpolation frame F and the pixels of the frame t and the frame t + 1. it can.

  The image interpolation unit 18 generates a pixel value of each pixel of the interpolation frame F by using the motion vector allocated on the interpolation frame F and the pixel values of the frames t and t + 1, and generates the frame t The generated interpolation frame F and frame t + 1 are each output at a predetermined timing. As a result, the 60P signal is output from the signal processing device 1.

  Here, with reference to FIG. 4, an evaluation value used in selecting a motion vector will be described.

  As described above, in each unit of the signal processing device 1 (the vector detection unit 12, the vector allocation unit 14, and the allocation compensation unit 17), when selecting a motion vector used in subsequent processing, a motion vector serving as an index thereof As the evaluation value, for example, a sum of absolute differences (DFD (Displaced Frame Difference)) representing a correlation value between blocks shifted by the vector amount of interest of two frames is used.

In the example of FIG. 4, a motion vector v (candidate motion vector) of interest from an m × n block centered on the pixel position p on the frame t and the pixel position p on the frame t + 1. An m × n block centered on a pixel position p + v shifted by the vector amount is shown, and the sum of absolute differences DFD _t (p) obtained between these blocks is expressed by the following equation (1). Is done.

Here, F _t (p) represents a luminance value at the pixel position p at time t, and m × n represents a DFD calculation range (block) for obtaining a sum of absolute differences. Since the difference absolute value sum represents the correlation value between the blocks in two frames, it can be said that the smaller the difference absolute value sum, the more the waveforms of the blocks between the frames match and the higher the reliability of the motion vector v. . Therefore, in each part of the signal processing device 1, the motion vector v having such a high reliability is selected as a motion vector optimal for the subsequent processing.

  Next, the frame frequency conversion process of the signal processing apparatus 1 will be described with reference to the flowchart of FIG.

  In step S <b> 1, the vector detection unit 12 acquires the pixel value of the frame t + 1 and the frame t stored in the frame memory 11. Pixel values of the frame t + 1 and the frame t stored in the frame memory 11 are also supplied to the vector allocation unit 14, the allocation compensation unit 17, and the image interpolation unit 18.

  In step S2, the vector detection unit 12 performs a motion vector detection process. With this process, a motion vector is detected between the target block of frame t and the target block of frame t + 1, and the detected motion vector is stored in the detection vector memory 13.

  In step S3, the vector allocation unit 14 performs vector allocation processing. By this processing, the motion vector obtained on the frame t is assigned to the pixel of the interpolation frame. The assigned motion vector is stored in the assigned vector memory 15 in association with the pixel.

  In step S4, the allocation compensator 17 performs allocation compensation processing. With this process, the motion vectors of the peripheral pixels of the pixel are assigned to the pixels to which no motion vector is assigned by the vector assigning unit 14.

  In step S5, the image interpolation unit 18 performs an image interpolation process. By this processing, the pixel value of the interpolation frame is generated by interpolation using the motion vector assigned to the interpolation frame and the pixel values of the frames t and t + 1.

  The motion vector detection processing, vector allocation processing, allocation compensation processing, and image interpolation processing performed in steps S2 to S5 will be described in detail later.

  In step S6, the image interpolation unit 18 outputs the generated interpolation frame, and then outputs a frame t + 1 as necessary, thereby outputting an image of the 60P signal to the subsequent stage.

  In step S7, the vector detection unit 12 determines whether or not the processing of all the frames has been completed. If it is determined that the processing of all the frames has not been completed, the vector detection unit 12 returns to step S1 and performs the subsequent processing. repeat. If the vector detection unit 12 determines in step S7 that all the frames have been processed, the vector detection unit 12 ends the processing.

  As described above, in the signal processing device 1, basically, a process for detecting a motion vector, a process for assigning the detected motion vector to a pixel on the interpolation frame, and a pixel in an interpolation frame to which no motion vector is assigned. The following four processes are performed: a process for compensating the motion vector, and a process for interpolating and generating an interpolation frame from the allocated motion vector and the pixel values of the two frames. Hereinafter, each process and the detail of the structure which implement | achieves it are demonstrated in order.

  First, motion vector detection will be described.

  FIG. 6 is a block diagram illustrating a configuration example of the vector detection unit 12 of FIG.

  As described above, the vector detection unit 12 detects the motion vector of each pixel on the frame t using the input frame t and frame t + 1. The process of detecting the motion vector is executed for each block including a plurality of pixels, for example.

  The initial vector selection unit 31 uses the motion vectors of peripheral blocks (peripheral areas) obtained in the past, stored in the detection vector memory 13, and the shifted initial vectors stored in the shifted initial vector memory 37 as initial vectors. As candidates (hereinafter also referred to as initial candidate vectors as appropriate).

  Further, the initial vector selection unit 31 obtains the evaluation value DFD of each initial candidate vector, and selects the initial candidate vector with the highest reliability selected based on the evaluation value DFD as the initial vector V0 to the iterative gradient method calculation unit 33. Output.

  The pre-filters 32-1 and 32-2 are composed of a low-pass filter and a Gaussian filter, and remove noise components from the input frames t and t + 1, respectively, and obtain frames t and t obtained by removing the noise components. +1 is output to the iterative gradient method computing unit 33.

  The iterative gradient method computing unit 33 uses the initial vector V0 supplied from the initial vector selection unit 31 and the frames t and t + 1 supplied from the prefilters 32-1 and 32-2 to generate a predetermined block. Each time, the motion vector Vn is calculated by the gradient method, and the initial vector V0 and the calculated motion vector Vn are output to the vector evaluation unit 34.

  The vector evaluation unit 34 obtains the motion vector Vn-1 (or initial vector V0) obtained by the iterative gradient method computing unit 33 and the evaluation value DFD of the motion vector Vn, and if the reliability is high from the obtained evaluation value DFD, Until the possible motion vector Vn is obtained, the iterative gradient method computing unit 33 is repeatedly executed with the gradient method.

  Further, the vector evaluation unit 34, when a vector considered to have high reliability is obtained by the iterative gradient method computing unit 33, stores it in the detection vector memory 13 as a motion vector V, and the motion vector V, The evaluation value DFD of the motion vector V is output to the shifted initial vector allocation unit 35.

  When the motion vector V and the evaluation value DFD are supplied from the vector evaluation unit 34, the shift initial vector allocation unit 35 shifts the motion vector passing through the block of interest on the next frame to the block of interest. This is set as a vector and stored in the shifted initial vector memory 37. For example, the shift initial vector allocating unit 35 uses a motion vector having the same size and the same direction as the motion vector V, starting from the target block on the next frame at the same position as the end block of the motion vector V. Set as.

  The evaluation value memory 36 stores the evaluation value DFD of the shifted initial vector candidate for each block.

  The shifted initial vector memory 37 stores a motion vector having the smallest evaluation value DFD in association with each block as a shifted initial vector.

  Here, the details of the initial vector selection unit 31, the iterative gradient method calculation unit 33, and the shifted initial vector allocation unit 35 constituting the vector detection unit 12 will be described.

  FIG. 7 is a block diagram illustrating a configuration example of the initial vector selection unit 31.

  The candidate vector position calculation unit 51 selects a target block to be processed on the frame t, acquires the initial candidate vector of the target block from the peripheral area of the target block, and the motion as the initial candidate vector Vector types and priorities are obtained, and candidate block position information and initial candidate vector type information are output to the detected vector obtaining unit 52 and the shifted initial vector obtaining unit 53 in the order of the obtained priorities. The number of initial candidate vectors, the positions of candidate blocks, the types of initial candidate vectors, and priorities are set in advance.

  The candidate vector position calculation unit 51 also supplies the position information of the candidate block to the offset position calculation unit 54.

  For example, if the preset type of the initial candidate vector is a past vector or a current vector (the vector type will be described later), the candidate vector position calculation unit 51 determines the position information of the candidate block and the type information of the initial candidate vector. When the initial candidate vector type is a shifted initial vector, the position information of the candidate block and the initial candidate vector type information are output to the shifted initial vector acquiring unit 53. If neither of them is present (for example, when the type of the initial candidate vector is 0 vector), the candidate vector position calculation unit 51 outputs the position information of the candidate block together with the 0 vector to the offset position calculation unit 54. .

  FIG. 8 is a diagram illustrating an example of a peripheral region that is a candidate block of an initial vector.

  In FIG. 8, an arrow T indicates the direction of time passage from the past frame t-1 at time t-1 on the left front side to the current frame t at time t on the right back. In the example of FIG. 8, the peripheral region that can be a candidate for the initial vector is configured by 7 × 7 blocks centered on the block Bt of interest. Each block is composed of 4 × 4 pixels, for example.

  The motion vector detection process is performed, for example, by switching the block to be processed in raster order with the upper left block of the frame as the start position. Therefore, in the initial vector selection unit 31, in the motion vector detection processing of the target block Bt in the current frame t, the motion vector detection result up to the block immediately before can be used as the initial vector candidate. Become.

  That is, the peripheral area of the block of interest Bt is composed of a block CVB processed before the block of interest Bt and a block PVB processed after the block of interest Bt. Therefore, when obtaining the initial vector of the block of interest Bt, the initial candidate vector is detected on the motion vector (current vector CV) detected on the current frame t of the block CVB and on the past frame t-1 of the block PVB. Motion vector (past vector PV) can be selected. At this time, the shifted initial vector SV allocated to the block in the same peripheral region can also be a candidate for the initial candidate vector.

  FIG. 9 is a diagram illustrating an example of initial vector candidate blocks set by the candidate vector position calculation unit 51.

  In the example of FIG. 9, eight blocks each having an “alphabet / number” symbol in the peripheral area of the block of interest Bt are set in advance as candidate blocks from which initial candidate vectors are acquired. . Symbols before and after the hatched lines shown in these blocks represent “initial candidate vector type” and “priority”, respectively. Of the types of initial candidate vectors, P represents the past vector PV detected in the past frame t-1, and C represents the current vector detected in the current frame t. S represents a shifted initial vector SV.

  “S / 1” of the block of interest Bt represents that the shifted initial vector SV assigned to the block of interest Bt is used first as an initial candidate vector. Further, “C / 2” of the block adjacent to the left of the block of interest Bt indicates that the current vector CV of the block detected in the current frame t is used second as the initial candidate vector. Similarly, for other symbols, the alphabetic characters indicate the types of motion vectors, and the numbers indicate priority.

  As described above, in the signal processing device 1, the number of candidate blocks, the position of the candidate block, the type of the initial candidate vector, and the priority order are set in advance, and the initial candidate vector is acquired based on the setting, and the initial Information on candidate vectors is output to each section.

  Returning to the description of FIG. 7, the detection vector acquisition unit 52 acquires, from the detection vector memory 13, the motion vector corresponding to the position information of the candidate block and the type information of the initial candidate vector supplied from the candidate vector position calculation unit 51. The obtained motion vector is output to the offset position calculation unit 54 as an initial candidate vector.

  The shifted initial vector acquisition unit 53 receives the shifted initial vector corresponding to the position of the candidate block according to the position information of the candidate block and the type information of the initial candidate vector supplied from the candidate vector position calculating unit 51. 37 and outputs the acquired shifted initial vector to the offset position calculation unit 54 as an initial candidate vector.

  In addition, the shifted initial vector acquisition unit 53 outputs a zero vector to the offset position calculating unit 54 when the shifted initial vector is not allocated to the block instructed by the candidate vector position calculating unit 51.

  When the initial candidate vector (or 0 vector from the candidate vector position calculation unit 51) is supplied from the detection vector acquisition unit 52 or the shifted initial vector acquisition unit 53, the offset position calculation unit 54 receives from the candidate vector position calculation unit 51. Based on the supplied position information of the candidate block, an offset destination block position obtained by offsetting (motion compensation) the target block of the frame t to the frame t + 1 is obtained for each initial candidate vector.

  Further, the offset position calculation unit 54 outputs information on the position of the candidate block and the block position of the offset destination to the evaluation value calculation unit 55 together with the initial candidate vector.

  The evaluation value calculation unit 55 obtains the evaluation value DFD of the supplied initial candidate vector when information on the position of the candidate block and the offset destination block position is supplied from the offset position calculation unit 54 together with the initial candidate vector. The obtained evaluation value DFD is output to the evaluation value comparison unit 56 together with the initial candidate vector.

  The evaluation value comparison unit 56 compares the evaluation value DFD supplied from the evaluation value calculation unit 55 with the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 57, and from the evaluation value calculation unit 55 If the evaluation value DFD of the supplied initial candidate vector is smaller than the evaluation value DFD of the optimal candidate vector, that is, if it is determined that the initial candidate vector is more reliable than the optimal candidate vector, The optimal candidate vector and its evaluation value DFD stored in the register 57 are replaced with the initial candidate vector determined to have high reliability and its evaluation value DFD. Further, the evaluation value comparison unit 56 outputs the optimum candidate vector determined to have the highest reliability among all candidate vectors as the initial vector V0 from the optimum candidate storage register 57 to the iterative gradient method computing unit 33. .

  Next, the iterative gradient method computing unit 33 in FIG. 6 will be described.

  FIG. 10 is a block diagram illustrating a configuration example of the iterative gradient method computing unit 33.

  The mode selection unit 71 selects a gradient method calculation mode for each block, for example, according to control by the effective pixel determination unit 73, and sends the initial vector V 0 supplied from the initial vector selection unit 31 to the selector 72 and the vector evaluation unit 34. Output. The gradient method calculation mode includes a block unit processing mode in which a motion vector detection target is a block and a pixel unit processing mode in which a motion vector detection target is a pixel. For example, the block unit processing mode is selected as the initial value.

  Under the control of the vector evaluation unit 34, the selector 72 selects one of the initial vector V0 supplied from the mode selection unit 71 and the motion vector Vn supplied from the gradient method calculation unit 74 for gradient method calculation. The motion vector used as an initial value (hereinafter referred to as an offset vector) is output to the effective pixel determination unit 73 and the gradient method calculation unit 74.

  The effective pixel determination unit 73 uses the blocks of frames t and t + 1 supplied from the pre-filters 32-1 and 32-2 to calculate the position calculated using the offset vector supplied from the selector 72 as an offset. It is determined whether or not the number of pixels effective in the gradient method calculation included in the block is greater than a threshold value, and the mode selection unit 71 is made to select the gradient calculation mode according to the result.

  For example, when the effective pixel determination unit 73 determines that the number of pixels effective for the gradient method calculation included in the calculation block to be processed is larger than the threshold value, the gradient method calculation unit 74 performs the gradient method calculation process. Is executed. If it is determined that there are few effective pixels, the effective pixel determination unit 73 changes the processing mode of the block currently focused on from the block unit processing mode to the pixel unit processing mode, or performs processing by the gradient method calculation unit 74. Or stop it.

  The gradient method computing unit 74 performs the gradient method computation according to the selected processing mode on the block at the position calculated using the offset vector supplied from the selector 72 as an offset, and obtains the calculated motion vector Vn. The result is output to the vector evaluation unit 34 and the delay unit 75.

  Here, the gradient method calculation performed by the gradient method calculation unit 74 will be described.

Let g (x, y, t) be the luminance value of a pixel constituting a moving image represented by coordinates (x, y, t) using the horizontal axis, vertical axis, and time axis. In this case, when the pixel of interest (x ₀ , y ₀ , t ₀ ) is displaced by (dx, dy, dt) during a minute time, the gradients (difference differences) of the horizontal axis, the vertical axis, and the time axis are gx ( When expressed as x ₀ , y ₀ , t ₀ ), gy (x ₀ , y ₀ , t ₀ ), and gt (x ₀ , y ₀ , t ₀ ), the luminance value of the pixel after displacement uses Taylor expansion approximation. Is represented by the following equation (2).

  When a pixel of interest in a moving image moves by horizontal vx and vertical vy after one frame (hereinafter referred to as (vx, vy)), the luminance value of that pixel is expressed by the following equation (3).

  Substituting equation (2) into equation (3) yields equation (4) below.

  Since the equation (4) is a two-variable equation of vx and vy, the solution cannot be obtained by a single equation for one pixel of interest. Therefore, a block that is a peripheral region of the pixel of interest is considered as one processing unit, and it is assumed that all the pixels in the block (peripheral region) have the same movement (vx, vy), and a similar expression is established for each pixel. As a result, equations for the number of peripheral pixels are obtained, and by combining these equations, (vx, vy) is obtained such that the sum of squares of the motion compensation frame differences of all the pixels in the block is minimized.

  When the pixel (x, y, t) moves by (vx, vy) between one frame, the motion compensation inter-frame difference d is expressed by the following equation (5).

  In equation (5), Δx = gx (x, y, t), which represents a horizontal gradient. Also, Δy = gy (x, y, t), which represents a vertical gradient, and Δt = gt (x, y, t), which represents a gradient in the time direction. Using these, the square sum E of the difference between the motion compensation frames is expressed by Expression (6).

  Here, (vx, vy) at which the sum of squares E is minimum is when the partial differential value in each variable becomes 0, that is, when the condition of δE / δvx = δE / δvy = 0 holds. Expressions (7) and (8) are obtained from Expression (6).

  From these equations (7) and (8), (vx, vy) which is the desired motion can be obtained by the following equation (9).

  This will be specifically described with reference to FIG.

  In FIG. 11, the arrow X indicates the horizontal direction, and the arrow Y indicates the vertical direction. An arrow T indicates the direction of passage of time from the frame t at the far right in the figure to the frame t + 1 at the left front. In FIG. 11, only an 8 × 8 pixel region that is a peripheral region of the pixel of interest p and is used for the gradient method calculation is shown as each frame.

  In the frame t in FIG. 11, when the motion vector V (vx, vy) of the pixel of interest p, which is the fifth pixel down from the upper left pixel and the fifth pixel on the right, is obtained by the gradient method, the motion vector V (vx, vy ) Is the difference in luminance difference between adjacent pixels px and py obtained in the x and y directions of the pixel of interest p (ie, gradients) Δx and Δy, and the same phase of the pixel of interest p obtained in the frame t + 1. A difference in luminance Δt in the time direction with respect to the pixel q located is obtained for all pixels in the peripheral region (8 × 8 pixels) of the pixel of interest p, and the difference between them is calculated using equation (9). It can ask for.

  That is, the gradient method is to obtain gradients Δx, Δy, and Δt between two frames, and statistically calculate a motion vector V (vx, vy) using the sum of squares of differences from the obtained Δx, Δy, and Δt. ) Is calculated.

  In such a motion vector detection method using the gradient method, a high-accuracy result can be obtained for a minute motion. However, when calculating a motion in an actual moving image, the calculation of the gradient method requires a motion amount. It is not practical because it is too large, and it is conceivable to repeat the calculation of the gradient method in order to cope with a large amount of motion. As a result, the amount of motion obtained by each calculation converges, and a correct motion can be obtained gradually.

  However, simply repeating the gradient method is not practical in terms of computation time. Therefore, the vector detection unit 12 reduces the number of times the gradient method is repeated by using, as an initial value, an initial vector that is obtained based on the motion of surrounding pixels of the past frame and the current frame. In other words, if a rough motion is calculated by adding an offset in advance from the pixel of interest that is the starting point of the motion to the destination indicated by the initial vector, an operation using the gradient method is performed from the position where the offset is added. It is possible to detect an accurate motion vector without increasing the calculation time.

  FIG. 12 is a diagram illustrating the iterative gradient method performed using the initial vector.

  In FIG. 12, the arrow T indicates the passage of time from the left front frame t to the right back frame t + 1 in the figure. A block centered on the pixels p, q0, q1, q2, and q3 represents a peripheral region (block) used for the gradient method calculation of each pixel.

  In the case of the example in FIG. 12, the position (pixel) calculated using the initial vector v0 as an offset in the frame t + 1, not the pixel q0 located in the same phase of the pixel of interest p, as compared to the pixel of interest p in the frame t. The first gradient method calculation is performed with q1 as a starting point, and as a result, a motion vector v1 is obtained.

  Next, the gradient method calculation is performed a second time with the position (pixel) q2 calculated from the pixels q0 to v0 + v1 as an offset, and as a result, a motion vector v2 is obtained. Thereby, the motion vector V is finally obtained as the following equation (10).

  As described above, by performing the gradient method calculation using the initial vector, it is possible to obtain a highly accurate motion vector while reducing the calculation time.

  Returning to the description of FIG. 10, the delay unit 75 holds the motion vector Vn supplied from the gradient method calculation unit 74 until the next processing cycle of the effective pixel determination unit 73 and the gradient method calculation unit 74. In accordance with the control, the held motion vector Vn is output to the effective pixel determination unit 73.

  Here, the details of the effective pixel determination unit 73 and the gradient method calculation unit 74 in FIG. 10 will be described in order.

  FIG. 13 is a block diagram illustrating a configuration example of the effective pixel determination unit 73.

  The time pixel difference calculation unit 81 selects a pixel in a calculation unit in units of blocks, the offset vector supplied from the selector 72, and the frames t and t + 1 supplied from the prefilters 32-1 and 32-2. Is used to calculate the pixel difference Δt in the time direction of the selected pixel, and the calculated pixel difference Δt in the time direction is output to the pixel difference value determination unit 82. The pixel difference Δt is expressed as, for example, a difference in luminance value of corresponding pixels between the frame t and the frame t + 1.

  The pixel difference value determination unit 82 determines whether or not the pixel difference Δt in the time direction calculated by the time pixel difference calculation unit 81 is smaller than a predetermined threshold (hereinafter referred to as a pixel difference value). When it is determined that the value is smaller than the value, the number of effective pixels counted by the effective pixel counter 83 is incremented by one.

  The effective pixel number counter 83 counts the number of pixels determined to be effective by the pixel difference value determination unit 82 for each calculation block.

  The gradient method continuation determination unit 84 determines whether or not the number of pixels effective for gradient method calculation included in the calculation block is greater than a threshold value. If it is determined that the number is greater than the threshold value, the gradient method calculation is performed. A flag (flg = 1) indicating execution is output to the gradient method computing unit 74.

  In addition, when the gradient method continuation determination unit 84 determines that the number of pixels effective for the calculation of the gradient method included in the calculation block is less than the threshold value, the gradient selection continuation determination unit 84 causes the mode selection unit 71 to select another processing mode, A flag (flg = 0) indicating that the gradient method calculation is terminated is output to the gradient method calculation unit 74.

  Here, pixels effective for the calculation of the gradient method will be described.

  When pixels included in one block display the same object, it is considered that the motion of each pixel is almost the same, and therefore, detection of a motion vector is performed by, for example, FIG. Is performed in units of blocks as shown in FIG.

  In FIG. 14, the arrow X indicates the horizontal direction, and the arrow Y indicates the vertical direction. An arrow T indicates the direction of passage of time from the frame t at the far right in the figure to the frame t + 1 at the left front.

  In the example of FIG. 14, an 8 × 8 pixel block is shown as the detection target block K (calculation block) of each frame. The motion vector obtained by the gradient method calculation using the detection target block K is a motion vector corresponding to all the pixels constituting the detection target block K.

  By the way, in the process of handling the pixels used for the calculation of the gradient method in units of calculation blocks, as shown in FIG. 15, there is a problem when pixels of objects having different motions are mixed in the detection target block.

  In the detection target block K in FIG. 15, the upper left pixel (thick line circle) has a diagonally upper left movement as indicated by the arrow A, and the lower right pixel (thin line circle) is indicated by the arrow B. It has a right-to-right movement.

  Therefore, in the detection target block K in FIG. 15, the assumption that the pixels in one block move in the same manner is broken, and the gradient method is calculated using an operation block that includes pixels with different motions. In such a case, the motion vector detection accuracy is significantly reduced. That is, this is a problem of a decrease in detection accuracy regarding the disappearance region (covered background region) and the generation region (uncovered background region) of the object that occurs at the boundary between objects having different movements.

  The disappearance region is a mixed region at a position corresponding to the front end portion of the foreground object in the traveling direction with respect to the foreground region, and is a region where the background component is covered with the foreground as time passes. The generation area is a mixed area at a position corresponding to the rear end of the foreground object in the advancing direction with respect to the foreground area, and refers to an area where a background component appears as time passes.

  As described above, for example, when pixels of an object having different movements are mixed in one detection block due to disappearance, generation, or deformation of the object, the luminance value between corresponding pixels of the frame t and the frame t + 1. Disturbance of movement occurs due to the fact that there are many changes in Therefore, the accuracy of motion vector detection can be improved by considering only pixels with small luminance changes as effective pixels.

  In addition, since the gradient method operation is an operation based on the least square method which is a statistical solution, if the operation is performed on a small number of pixels (effective pixels), the reliability of the operation result decreases, There is a possibility that the accuracy of the detected motion vector is deteriorated.

  Therefore, in the effective pixel determination unit 73, when the number of effective pixels for the gradient method calculation is small, the calculation in units of blocks is assumed to be unstable, and the processing mode is switched to the processing mode in which the calculation is performed in units of pixels.

  FIG. 16 is a diagram illustrating an example of a calculation block in the processing mode in units of pixels.

  As shown in FIG. 16, when the pixel p is focused and the processing mode is switched from the block unit mode to the pixel unit mode when obtaining the motion vector, the detection target block surrounded by a dotted line is shown. For example, a 9 × 9 pixel calculation block E centered on the pixel p is used as the calculation block instead of K.

  Furthermore, if the number of pixels that are effective for gradient method calculation is extremely small and it is not possible to detect a motion vector that is reliable to some extent even by pixel-by-pixel processing, the motion vector calculation is aborted, and the motion of the detection result For example, the gradient method computing unit 74 is controlled so that a 0 vector (stationary state) is output as a vector.

  FIG. 17 is a block diagram illustrating a configuration example of the gradient method computing unit 74 of FIG.

  When the flag supplied from the effective pixel determination unit 73 indicates 1, the time pixel difference calculation unit 91 uses the frames t and t + 1 supplied from the pre-filters 32-1 and 32-2 to select the time pixel difference calculation unit 91. The pixel in the calculation block centering on the pixel calculated using the offset vector from 72 as an offset is selected, and the pixel difference Δt in the time direction of the selected pixel is calculated. The temporal pixel difference calculation unit 91 outputs the calculated pixel difference Δt to the pixel difference value determination unit 92 together with the offset vector.

  Further, the temporal pixel difference calculation unit 91 outputs an offset vector to the vector calculation unit 95 when the processing of the pixels in the calculation block is completed.

  On the other hand, when the flag indicates 0, the time pixel difference calculation unit 91 aborts the gradient method calculation process. At this time, the temporal pixel difference calculation unit 91 sets the motion vector V output from the vector calculation unit 95 to the 0 vector.

  The pixel difference value determination unit 92 determines whether or not the pixel difference Δt in the time direction calculated by the time pixel difference calculation unit 91 is smaller than a predetermined threshold (hereinafter referred to as a pixel difference value). If it is determined that the pixel value is smaller than the value, the pixel is set as a calculation target of the gradient method, and the pixel difference Δt and the offset vector are output to the horizontal / vertical pixel difference calculation unit 93.

  If the pixel difference value determination unit 92 determines that the pixel difference Δt is equal to or greater than the pixel difference value, the pixel difference value determination unit 92 prohibits the horizontal / vertical pixel difference calculation unit 93 and the gradient integration unit 94 from processing the pixel.

  The horizontal / vertical pixel difference calculation unit 93 calculates the offset vector supplied from the pixel difference value determination unit 92 as an offset using the frames t and t + 1 supplied from the pre-filters 32-1 and 32-2. In the calculation block centered on the pixel, the pixel difference value determination unit 92 calculates the pixel difference Δx in the horizontal direction and the pixel difference Δy in the vertical direction of the pixel to be calculated. The horizontal / vertical pixel difference calculation unit 93 outputs the calculated pixel differences Δx and Δy to the gradient integration unit 94 together with the pixel difference Δt.

  The gradient integrating unit 94 integrates the gradients (pixel differences Δx, Δy, Δt) supplied from the horizontal / vertical pixel difference calculating unit 93 and outputs the integrated gradient values to the vector calculating unit 95.

  When the offset vector is supplied from the time pixel difference calculation unit 91, the vector calculation unit 95 calculates the motion vector vn from the gradient value accumulated by the gradient accumulation unit 94 and the least square sum of the above equation (9). To do.

  Further, the vector calculation unit 95 adds the offset vector from the time pixel difference calculation unit 91 to the calculated motion vector vn to obtain the motion vector Vn, and outputs the obtained motion vector Vn to the vector evaluation unit 34 and the delay unit 75. To do.

  Next, the shifted initial vector allocation unit 35 in FIG. 6 will be described.

  FIG. 18 is a block diagram illustrating a configuration example of the shifted initial vector allocation unit 35.

  The allocation target position calculation unit 101 is the position of the block through which the motion vector V supplied from the vector evaluation unit 34 passes on the frame at the next time (the block at the end point of the motion vector V detected on the current frame). And the position of the obtained block is output to the evaluation value memory 36 and the shifted initial vector replacement unit 103.

  When the motion vector V and the evaluation value DFD of the motion vector V are supplied from the vector evaluation unit 34, the evaluation value comparison unit 102 evaluates the evaluation value DFD of the block position obtained by the allocation target position calculation unit 101. The evaluation value DFD read out from the value memory 36 is compared with the evaluation value DFD of the motion vector V supplied from the vector evaluation unit 34. When the evaluation value comparison unit 102 determines that the evaluation value DFD of the motion vector V supplied from the vector evaluation unit 34 is smaller (higher reliability), the evaluation value comparison unit 102 controls the shifted initial vector replacement unit 103 to perform the initial shifting. The shifted initial vector stored in the vector memory 37 is rewritten with the motion vector V determined to have high reliability.

  Further, the evaluation value comparison unit 102 controls the evaluation value replacement unit 104, and uses the evaluation value DFD of the position of the block selected by the allocation target position calculation unit 101 stored in the evaluation value memory 36 as the motion vector V. Rewrite with the evaluation value DFD.

  The shifted initial vector replacement unit 103 rewrites the shifted initial vector stored in the shifted initial vector memory 37 according to the control by the evaluation value comparing unit 102.

  The evaluation value replacement unit 104 rewrites the evaluation value DFD stored in the evaluation value memory 36 according to the control by the evaluation value comparison unit 102.

  FIG. 19 is a diagram showing the shifted initial vector one-dimensionally.

  In the example of FIG. 19, frames t-1, t, t + 1 are shown in order from the top. Partitions on each frame represent block boundaries.

  In the case of the example in FIG. 19, the motion vector detected in the block B on the frame t−1 is the motion vector V indicated by the solid line arrow, and the motion compensation destination (offset destination) on the frame t by this motion vector V is Block Bt. In addition, a motion vector obtained by shifting the motion vector V on the frame t-1 to the block Bt on the frame t is set as a shifted initial vector SV indicated by a dotted arrow.

  The shift means that the start point of the motion vector having the same magnitude and the same direction as the motion vector V detected on the frame t-1 is at the same position as the end block of the motion vector V on the frame t-1. The block Bt on the frame t.

  In general, there is a certain degree of continuity in the amount of movement of moving objects between consecutive frames, and the change in the amount of movement is small. Therefore, in the example of FIG. In the case of close motion, the motion vector in the block Bt is often the motion vector V or a motion vector close thereto.

  Therefore, when the shifted initial vector SV is used as a candidate for an initial vector given to the gradient method calculation when detecting a motion vector of the block of interest Bt on the frame t, only the motion vector of the peripheral block is used as a candidate for the initial vector. A more appropriate initial vector can be obtained than in the case.

  By the way, when an initial vector is allocated to each block in this way, depending on the distribution of motion vectors detected on the frame t-1, a plurality of shifted initial vectors compete as allocation target vectors on the frame t. Blocks that do not, and conversely, blocks that are not allocated.

  FIG. 20 is a diagram illustrating an example of allocation of the shifted initial vector.

  In FIG. 20, blocks B1 to B5 are shown from the left on the frame t-1, and blocks Bt1 to Bt5 as areas corresponding to the blocks B1 to B5 are shown from the left on the frame t. Has been.

  In addition, the motion vector V1 is detected in the block B1 of FIG. 20, and the offset destination A1 of the block B1 overlaps with the blocks Bt1 and Bt2 on the frame t. In addition, the motion vector V2 is detected in the block B2, and the offset destination A2 of the block B2 overlaps the blocks Bt1 and Bt2 on the frame t.

  Similarly, in the block B3, the motion vector V3 is detected, and the offset destination A3 of the block B3 overlaps with the blocks Bt4 and Bt5 on the frame t. In block B4, motion vector V4 is detected, and offset destination A4 of block B4 overlaps with blocks Bt4 and Bt5 on frame t. In the block B5, the motion vector V5 is detected, and the offset destination A5 of the block B5 overlaps with the block Bt5 on the frame t and an adjacent block (not shown).

  That is, in the example of FIG. 20, one of the motion vectors V1 and V2 is assigned to the blocks Bt1 and Bt2 on the frame t as a shifted initial vector. In addition, one of the motion vectors V3 and V4 is assigned to the block Bt4 on the frame t as a shift initial vector, and any one of the motion vectors V3, V4, and V5 is assigned to the block Bt5 on the frame t. Are assigned as shifted initial vectors.

  However, in the block Bt3 on the frame t, there is no motion vector that is a candidate for the shifted initial vector, and the shifted initial vector is not allocated.

  Therefore, the shifted initial vector allocating unit 35 allocates the 0 vector as the shifted initial vector to a block in which allocation of the shifted initial vector does not occur, such as the block Bt3.

  On the other hand, the shifted initial vector allocating unit 35 selects a motion vector with high reliability based on the above-described evaluation value DFD for blocks such as blocks Bt1, Bt2, Bt4, and Bt5 that compete with a plurality of motion vectors. Then, the selected motion vector is allocated to each block as a shifted initial vector.

  Next, processing of each unit of the vector detection unit 12 having the above configuration will be described.

  First, the entire motion vector detection process performed by the vector detection unit 12 will be described with reference to the flowchart of FIG. This process is a process performed in step S2 of FIG.

  In step S21, the initial vector selection unit 31 selects a block to be processed from among the blocks on the frame t as a target block. For example, the target block is selected in raster order from the upper left block.

  In step S22, the initial vector selection unit 31 executes initial vector selection processing. The initial vector selected by the initial vector selection process is output to the iterative gradient method computing unit 33. Details of the initial vector selection processing will be described later with reference to the flowchart of FIG.

  In step S23, the iterative gradient method computing unit 33 and the vector evaluating unit 34 execute an iterative gradient method computing and evaluating process. That is, here, a motion vector is obtained by the iterative gradient method computing unit 33, and the obtained motion vector is evaluated by the vector evaluation unit 34. The obtained motion vector V is output to and stored in the detection vector memory 13. Details of the iterative gradient method computation evaluation process will be described later with reference to the flowchart of FIG.

  In step S24, the shifted initial vector allocation unit 35 performs a shifted initial vector allocation process. The shifted initial vector set by the shifted initial vector allocation process is allocated to the shifted initial vector memory 37 in a form corresponding to the block of interest. Details of the shifted initial vector allocation processing will be described later with reference to the flowchart of FIG.

  In step S25, the initial vector selection unit 31 determines whether or not the processing of all the blocks in the frame t has been completed. If it is determined that the processing of all the blocks has not been completed, the initial vector selection unit 31 returns to step S21. The subsequent processing is repeated.

  If the initial vector selection unit 31 determines in step S25 that all the blocks have been processed, that is, if it is determined that the motion vectors V have been detected in all the blocks on the frame t, the initial vector selection unit 31 performs the motion vector detection process. Terminate. Thereafter, the processing returns to step S2 in FIG. 5 and the subsequent processing is performed.

  Next, the initial vector selection process performed in step S22 of FIG. 21 will be described with reference to the flowchart of FIG.

  In step S31, the candidate vector position calculation unit 51 (FIG. 7) obtains the position of the candidate block for obtaining the preset initial candidate vector of the target block from the peripheral region of the selected target block, and the initial candidate vector Find the type and priority.

  In step S32, the candidate vector position calculation unit 51 determines whether or not the type of the initial candidate vector of the candidate block is a past vector or a current vector in the order of priority obtained in step S31.

  In step S32, when the candidate vector position calculation unit 51 determines that the type of the initial candidate vector of the candidate block is a past vector or the current vector, the candidate vector position calculation unit 51 detects the position information of the candidate block and the type information of the initial candidate vector. The process proceeds to step S33.

  In step S33, the detection vector acquisition unit 52 acquires a motion vector (past vector PV or current vector CV) corresponding to the position of the candidate block and the type of the initial candidate vector from the detection vector memory 13, and offsets the acquired motion vector. It outputs to the position calculation part 54.

  On the other hand, if the candidate vector position calculation unit 51 determines in step S32 that the type of the initial candidate vector of the candidate block is not the past vector or the current vector, the process proceeds to step S34, and then the type of the initial candidate vector of the candidate block. Is a shifted initial vector.

  If the candidate vector position calculation unit 51 determines in step S34 that the type of the initial candidate vector of the candidate block is a shifted initial vector, the candidate vector position calculation unit 51 obtains the position information of the candidate block and the type information of the initial candidate vector as a shifted initial vector acquisition unit 53. And proceed to step S35.

  In step S <b> 35, the shifted initial vector acquisition unit 53 acquires the shifted initial vector corresponding to the position of the candidate block from the shifted initial vector memory 37 and outputs the acquired shifted initial vector to the offset position calculation unit 54.

  On the other hand, in step S34, the candidate vector position calculation unit 51 determines that the type of the initial candidate vector of the candidate block is not the shifted initial vector, that is, determines that the type of the initial candidate vector of the candidate block is the 0 vector. If so, the process proceeds to step S36.

  In step S36, the candidate vector position calculation unit 51 sets a zero vector as the initial candidate vector, and outputs the position information of the candidate block together with the zero vector to the offset position calculation unit 54. Note that the position information of the candidate block is also output to the offset position calculation unit 54 in steps S33 and S35.

  In step S37, the offset position calculation unit 54 sets the target block of the frame t to the frame t + 1 for each initial candidate vector based on the position information of the candidate blocks supplied from the candidate vector position calculation unit 51. The offset destination block position that has been offset to is obtained, and information on the obtained offset destination block position is output to the evaluation value calculation unit 55 together with the initial candidate vector and the candidate block position information.

  In step S38, the evaluation value calculation unit 55 obtains the evaluation value DFD of the initial candidate vector from the information supplied from the offset position calculation unit 54 and the frames t and t + 1, and uses the obtained evaluation value DFD as the initial candidate. The result is output to the evaluation value comparison unit 56 together with the vector.

  In step S39, the evaluation value comparison unit 56 determines whether or not the evaluation value DFD obtained by the evaluation value calculation unit 55 is smaller than the evaluation value DFD of the optimum candidate vector stored in the optimum candidate storage register 57. To do.

  In step S39, the evaluation value comparison unit 56 determines that the evaluation value DFD obtained by the evaluation value calculation unit 55 is smaller than the evaluation value DFD of the optimum candidate vector stored in the optimum candidate storage register 57. In step S40, the optimum candidate vector and its evaluation value DFD stored in the optimum candidate storage register 57 are rewritten with the initial candidate vector and its evaluation value DFD that are determined to have high reliability.

  If it is determined in step S39 that the evaluation value DFD obtained by the evaluation value calculator 55 is not smaller than the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 57, the process of step S40 is performed. Skipped.

  In step S41, the candidate vector position calculation unit 51 determines whether or not all the initial candidate vectors have been processed. If it is determined that all the initial candidate vectors have not been processed, the process returns to step S32. Repeat the subsequent processing.

  On the other hand, if it is determined in step S41 that all the initial candidate vectors have been processed, the process proceeds to step S42, where the evaluation value comparison unit 56 has the highest reliability stored in the optimum candidate storage register 57. The optimal candidate vector determined to be high is output to the iterative gradient method computing unit 33 as the initial vector V0. Thereafter, the process returns to step S22 in FIG. 21, and the subsequent processing is performed.

  As described above, the evaluation value DFD of a plurality of initial candidate vectors is obtained, and the initial candidate vector having the smallest evaluation value DFD, that is, the highest reliability is selected as the initial vector. An optimal initial vector can be given. In addition, since the motion vector of a predetermined block is used as the initial candidate vector, the number of initial candidate vectors is increased, and the amount of calculation is suppressed from becoming enormous.

  Next, the iterative gradient method computation evaluation process performed in step S23 of FIG. 21 will be described with reference to the flowchart of FIG.

  In step S51, the mode selection unit 71 (FIG. 10) selects a block unit processing mode. As a result, the block on the frame t is set as a detection target block (calculation block), and the initial vector V0 of the detection target block is output to the vector evaluation unit 34 and the selector 72.

  In step S52, the selector 72 uses the initial vector V0 supplied from the mode selection unit 71 as an offset vector, and the effective pixel determination unit 73 (time pixel difference calculation unit 81 in FIG. 13) and the gradient method calculation unit 74 (in FIG. 17). It outputs to the time pixel difference calculation part 91).

  In step S <b> 53, the effective pixel determination unit 73 executes a block-based effective pixel determination process. By this effective pixel determination processing, the number of effective pixels included in the calculation block in block units is counted. Details of the effective pixel determination processing will be described later with reference to the flowchart of FIG.

  In step S54, the effective pixel determination unit 73 (gradient method continuation determination unit 84 in FIG. 13) determines whether or not the number of effective pixels is greater than the threshold value α, and if the number of effective pixels is greater than the threshold value α. If it is determined, a flag (flg = 1) for executing the gradient method calculation in units of blocks is output to the gradient method calculation unit 74.

  In step S55, the gradient method computing unit 74 executes gradient method computation processing in units of blocks. A motion vector Vn is obtained by this gradient method arithmetic processing, and the obtained motion vector Vn is output to the vector evaluation unit 34 and the delay unit 75. Details of the gradient method calculation processing in units of blocks will be described later with reference to the flowchart of FIG.

  In step S56, the vector evaluation unit 34 evaluates the motion vector Vn obtained by the iterative gradient method computing unit 33 and the evaluation value DFD (n−) of the motion vector Vn−1 used as the offset vector. 1) is obtained, and it is determined whether or not the evaluation value DFD (n) is smaller than the evaluation value DFD (n−1). For example, in the first process, the obtained evaluation value DFD (1) of the motion vector V1 is compared with the evaluation value DFD (0) of the initial vector V0 used as the offset vector. In the second process, The obtained evaluation value DFD (2) of the motion vector V2 is compared with the evaluation value DFD (1) of the motion vector V1 used as the offset vector.

  In step S56, the vector evaluation unit 34 determines that the evaluation value DFD (n) is smaller than the evaluation value DFD (n-1), that is, the reliability of the motion vector Vn than the motion vector Vn-1. When it is determined that the motion vector is high, the process proceeds to step S57, and the calculated motion vector Vn is set as the motion vector V of the detection target block.

  In step S58, the vector evaluation unit 34 counts the number of iterations n by 1, and proceeds to step S59 to determine whether or not the number of iterations n has reached a set maximum number of iterations (for example, 2 times).

  If it is determined in step S59 that the number of iterations n has not reached the maximum number of iterations, the vector evaluation unit 34 returns to step S52 and repeats the subsequent processing. That is, the motion vector V1 held in the delay unit 75 is selected as an offset vector, and the subsequent processing is performed.

  In step S56, the vector evaluation unit 34 determines that the evaluation value DFD (n) is not smaller than the evaluation value DFD (n-1), that is, the motion vector Vn-1 has a motion. When it is determined that the reliability is higher than that of the vector Vn, the process proceeds to step S60, and the vector Vn-1 as an offset of the gradient method calculation is set as the motion vector V of the detection target block.

  If it is determined in step S59 that the number of iterations n has reached the maximum number of iterations, or if the vector Vn-1 is set as the motion vector V of the detection target block in step S60, the process proceeds to step S61. The evaluation unit 34 determines whether or not the motion vector V is within a search area set in advance as a range for detecting a motion vector.

  If the vector evaluation unit 34 determines in step S61 that the motion vector V is within the search area, the vector evaluation unit 34 proceeds to step S62 and stores the motion vector V in the detection vector memory 13 in association with the detection target block.

  If the vector evaluation unit 34 determines in step S61 that the motion vector V is not within the search area, the vector evaluation unit 34 proceeds to step S63, sets the 0 vector as the motion vector V, and sets it as the detection target block in step S62. And stored in the detection vector memory 13.

  On the other hand, if the effective pixel determination unit 73 determines in step S54 that the number of effective pixels is equal to or less than the threshold value α, the process proceeds to step S64, where the mode selection unit 71 is controlled to select the pixel unit processing mode.

  In step S65, the iterative gradient method computing unit 33 executes a gradient method computation process in units of pixels. By this iterative gradient method computing process, motion vectors V of all the pixels in the detection target block are obtained. The obtained motion vector V is stored in the detection vector memory 13 in step S62. Details of the gradient method calculation processing in units of pixels will be described later with reference to the flowchart of FIG.

  After the motion vector V is stored in the detection vector memory 13 in step S62, the process returns to step S23 in FIG. 21, and the subsequent processing is performed.

  Next, the effective pixel determination process performed in step S53 in FIG. 23 will be described with reference to the flowchart in FIG.

  In step S71, the temporal pixel difference calculation unit 81 (FIG. 13) of the effective pixel determination unit 73 controls the pixel difference value determination unit 82 to reset the effective pixel number counter 83, and the process proceeds to step S72 to configure a calculation block. One of the pixels to be selected is selected. Here, for example, the pixels are selected in raster order from the upper left pixel of the calculation block.

  In step S73, the time pixel difference calculation unit 81 uses the offset vector supplied from the selector 72 and the frames t and t + 1 supplied from the pre-filters 32-1 and 32-2 in step S73. The pixel difference Δt in the time direction of the selected pixel is calculated, and the calculated pixel difference Δt is output to the pixel difference value determination unit 82.

  In step S74, the pixel difference value determination unit 82 determines whether or not the pixel difference Δt in the time direction is smaller than a predetermined pixel difference value, that is, the currently selected pixel is effective for the gradient method calculation in the subsequent stage. It is determined whether or not.

  If the pixel difference value determination unit 82 determines in step S74 that the pixel difference Δt is smaller than the predetermined pixel difference value, the pixel difference value determination unit 82 proceeds to step S75 and increments the number of effective pixels counted by the effective pixel number counter 83 by one. Increment.

  If the pixel difference value determination unit 82 determines in step S74 that the pixel difference Δt is greater than or equal to the predetermined pixel difference value, that is, the currently selected pixel is not a valid pixel for the gradient method calculation. When it determines, the process of step S75 is skipped and it progresses to step S76.

  In step S76, the time pixel difference calculation unit 81 determines whether or not the processing of all the pixels in the calculation block has been completed, and determines that the processing of all the pixels in the calculation block has not ended. Returning to step S72, the next pixel is selected, and the subsequent processing is repeated.

  If the time pixel difference calculation unit 81 determines in step S76 that the processing of all the pixels in the calculation block has been completed, the time pixel difference calculation unit 81 returns to step S53 in FIG. 23 to perform the subsequent processing.

  Next, with reference to the flowchart of FIG. 25, the gradient method calculation processing in units of blocks performed in step S55 of FIG. 23 will be described.

  In step S91, the temporal pixel difference calculation unit 91 (FIG. 17) of the gradient method calculation unit 74 selects one pixel from among the pixels constituting the calculation block. For example, the pixels are selected in raster order from the upper left pixel of the calculation block.

  In step S92, the time pixel difference calculation unit 91 uses the offset vector supplied from the selector 72 and the frames t and t + 1 supplied from the pre-filters 32-1 and 32-2 in step S91. The pixel difference Δt in the time direction of the selected pixel is calculated, and the calculated pixel difference Δt and the offset vector are output to the pixel difference value determination unit 92.

  In step S93, the pixel difference value determination unit 92 determines whether or not the calculated pixel difference Δt is smaller than a predetermined pixel difference value, that is, whether the currently selected pixel is a valid pixel for the gradient method calculation. Determine whether or not.

  When determining that the pixel difference Δt is smaller than the predetermined pixel difference value in Step S93, the pixel difference value determining unit 92 outputs the pixel difference Δt in the time direction and the offset vector to the vertical horizontal pixel difference calculating unit 93.

  In step S94, the vertical / horizontal pixel difference calculation unit 93 uses the offset vector supplied from the pixel difference value determination unit 92 and the frames t and t + 1 supplied from the pre-filters 32-1 and 32-2. Then, the pixel difference Δx in the horizontal direction of the pixel determined to be effective is calculated.

  Further, the vertical horizontal pixel difference calculation unit 93 calculates the vertical pixel difference Δy of the pixel determined to be valid in step S95, and calculates the calculated horizontal pixel difference Δx and the vertical pixel difference Δy. , The pixel difference Δt in the time direction is output to the gradient integration unit 94.

  In step S <b> 96, the gradient integration unit 94 integrates the pixel differences Δt, Δx, Δy supplied from the vertical / horizontal pixel difference calculation unit 93 and outputs the integration result to the vector calculation unit 95.

  On the other hand, if the pixel difference value determination unit 92 determines in step S93 that the pixel difference Δt in the time direction is greater than or equal to the predetermined pixel difference value, the process of steps S94 to S96 is skipped and the process proceeds to step S97. That is, in this case, since the currently selected pixel is not a pixel effective for the gradient method calculation, the pixel difference value is not used for the calculation.

  In step S97, the time pixel difference calculation unit 91 determines whether or not the processing of all the pixels in the calculation block has been completed, and determines that the processing of all the pixels in the calculation block has not ended. Returning to step S91, the next pixel is selected, and the subsequent processing is repeated.

  On the other hand, the time pixel difference calculation unit 91 outputs an offset vector to the vector calculation unit 95 when it is determined in step S97 that all the pixels in the calculation block have been processed.

  In step S98, the vector calculation unit 95 calculates the motion vector vn using the integration result of the gradient supplied from the gradient integration unit 94 and the least square sum of the above equation (9).

  In step S99, the vector calculation unit 95 obtains the motion vector Vn by adding the motion vector vn calculated in step S98 to the offset vector from the time pixel difference calculation unit 91, and calculates the obtained motion vector Vn as a vector evaluation unit. 34. Thereafter, the process returns to step S55 in FIG. 23, and the subsequent processing is performed.

  As described above, when the pixel difference Δt in the time direction is a value greater than or equal to a predetermined pixel difference value, the pixel selected at that time is detected as a motion vector, and a motion vector is detected. Since the gradient method is excluded from the calculation of the gradient method, more stable gradient method calculation can be performed.

  Next, with reference to the flowchart of FIG. 26, the iterative gradient method computing process in units of pixels performed in step S65 of FIG. 23 will be described. This process is performed for each pixel in the detection target block.

  In step S111, the mode selection unit 71 selects one of the pixels constituting the detection target block as a detection target pixel, and an initial vector V0 of a calculation block (for example, a 9 × 9 pixel block) of the pixel. Is output to the vector evaluation unit 34 and the selector 72. For example, pixels are selected in raster order from the upper left pixel of the detection target block.

  In step S112, the selector 72 uses the initial vector V0 supplied from the mode selection unit 71 as an offset vector, and the effective pixel determination unit 73 (time pixel difference calculation unit 81 in FIG. 13) and the gradient method calculation unit 74 (in FIG. 17). It outputs to the time pixel difference calculation part 91).

  In step S113, the effective pixel determination unit 73 performs an effective pixel determination process in units of pixels using the selected offset vector. Here, basically the same processing as the effective pixel determination processing in units of blocks described with reference to FIG. 24 is performed except that the target arithmetic block is different.

  In step S114, the effective pixel determination unit 73 (gradient method continuation determination unit 84 in FIG. 13) determines whether or not the number of effective pixels is greater than the threshold value β, and if the number of effective pixels is greater than the threshold value β. If it is determined, a flag (flg = 1) for executing the gradient method calculation is output to the gradient method calculation unit 74.

  In step S115, the gradient method computing unit 74 executes a gradient method computation process in units of pixels. Here, the same processing as the gradient method arithmetic processing in units of blocks in FIG. 25 is performed except that the target arithmetic blocks are different. That is, the motion vector Vn of the detection target pixel is obtained, and the obtained motion vector Vn is output to the vector evaluation unit 34 and the delay unit 75 (FIG. 10).

  In step S116, the vector evaluating unit 34 evaluates the motion vector Vn obtained by the iterative gradient method computing unit 33, and the evaluated value DFD (n−) of the motion vector Vn−1 used as the offset vector. 1) is obtained, and it is determined whether or not the evaluation value DFD (n) is smaller than the evaluation value DFD (n−1).

  In step S116, the vector evaluation unit 34 determines that the evaluation value DFD (n) is smaller than the evaluation value DFD (n-1), that is, the reliability of the motion vector Vn than the motion vector Vn-1. When it is determined that is high, the process proceeds to step S117, and the calculated motion vector Vn is set as the motion vector V of the detection target pixel.

  In step S118, the vector evaluation unit 34 counts the number of iterations n by 1, and proceeds to step S119 to determine whether or not the number of iterations n has reached a set maximum number of iterations (for example, 2 times). .

  If the vector evaluation unit 34 determines in step S119 that the number of iterations n has not reached the maximum number of iterations, the vector evaluation unit 34 returns to step S112 and repeats the subsequent processing. That is, the motion vector V1 held in the delay unit 75 is selected as an offset vector, and the subsequent processing is performed.

  If it is determined in step S119 that the number of iterations n has reached the maximum number of iterations, the process proceeds to step S121.

  In step S116, the vector evaluation unit 34 determines that the evaluation value DFD (n) is not smaller than the evaluation value DFD (n-1), that is, the motion vector Vn-1 has a motion. When it is determined that the reliability is higher than that of the vector Vn, the process proceeds to step S120, and the vector Vn-1 as an offset of the gradient method calculation is set as the motion vector V of the detection target block.

  If it is determined in step S119 that the number of iterations n has reached the maximum number of iterations, or if the vector Vn-1 is set as the motion vector V of the detection target block in step S120, the process proceeds to step S121. The evaluation unit 34 determines whether or not the motion vector V is within a search area set in advance as a range for detecting a motion vector.

  In step S121, when the vector evaluation unit 34 determines that the motion vector V is not within the search area, the process proceeds to step S122.

  In step S114, if the effective pixel determination unit 73 determines that the number of effective pixels is smaller than the threshold value β, the effective pixel determination unit 73 outputs a flag (flg = 0) to stop the gradient method calculation to the gradient method calculation unit 74, and the step The process proceeds to S122. In step S122, the vector evaluation unit 34 sets the motion vector V to 0 vector.

  If it is determined in step S121 that the motion vector V is within the search area, or if the motion vector V is set to 0 vector in step S122, the process proceeds to step S123.

  In step S123, the mode selection unit 71 determines whether or not the processing of all the pixels in the detection target block has been completed, and determines that the processing of all the pixels in the detection target block has not been completed. Returning to step S111, the next pixel of the detection target block is selected as the detection target pixel, and the subsequent processing is repeated.

  If the mode selection unit 71 determines in step S123 that all the pixels in the detection target block have been processed, the mode selection unit 71 returns to step S65 in FIG. 23 and performs the subsequent processing.

  That is, the motion vector V is obtained for all the pixels in the detection target block by the iterative gradient method computing process in units of pixels, and the detected vector memory is configured so that the obtained motion vector V corresponds to each pixel of the detection target block. 13 is stored.

  Next, the shifted initial vector allocation process performed in step S24 of FIG. 21 will be described with reference to the flowchart of FIG.

  In step S131, the evaluation value comparison unit 102 (FIG. 18) of the shifted initial vector allocation unit 35 acquires the evaluation value DFD of the motion vector V from the vector evaluation unit 34 together with the motion vector V. At this time, the motion vector V is also acquired in the allocation target position calculation unit 101.

  In step S132, the allocation target position calculation unit 101 obtains the position of the allocation target block that is the offset (motion compensation) destination of the motion vector V in the frame t.

  In step S133, the allocation target position calculation unit 101 selects one allocation target block among the allocation target blocks obtained in step S132, and uses the evaluation value memory 36 and the shifted initial vector as information on the position of the selected allocation target block. The data is output to the replacement unit 103. For example, among the allocation target blocks, the allocation target blocks are selected in order from the upper left block on the frame t.

  In step S134, the evaluation value comparison unit 102 acquires the evaluation value DFD of the allocation target block selected by the allocation target position calculation unit 101 from the evaluation value memory 36, proceeds to step S135, and the motion vector acquired in step S131. It is determined whether or not the evaluation value DFD of V is smaller than the evaluation value DFD stored in the evaluation value memory 36.

  If the evaluation value comparison unit 102 determines in step S135 that the evaluation value DFD of the motion vector V is smaller than the evaluation value DFD of the evaluation value memory 36, the evaluation value comparison unit 102 proceeds to step S136, and sets the shifted initial vector replacement unit 103. Then, the shift initial vector stored in the shift initial vector memory 37 of the allocation target block selected by the allocation target position calculation unit 101 is rewritten to the motion vector V.

  Further, in step S137, the evaluation value comparison unit 102 controls the evaluation value replacement unit 104 so that the evaluation value DFD of the allocation target block selected by the allocation target position calculation unit 101 is the evaluation value DFD of the motion vector V. Let them be rewritten.

  When the evaluation value is rewritten in step S137, the process proceeds to step S138. If it is determined in step S135 that the evaluation value DFD of the motion vector V acquired in step S131 is not smaller than the evaluation value DFD stored in the evaluation value memory 36, the processes in steps S136 and S137 are performed. Skipped and processing proceeds to step S138.

  In step S138, the allocation target position calculation unit 101 determines whether all the processes for the allocation target block of the motion vector V have been completed. If it is determined that all the processes have not been completed, the process returns to step S133. Repeat the subsequent processing.

  If the allocation target position calculation unit 101 determines in step S138 that all the processes for the allocation target block of the motion vector V have been completed, the allocation target position calculation unit 101 returns to step S24 in FIG. 21 and performs the subsequent processing.

  As described above, based on the fact that there is a certain degree of continuity in the amount of movement of the moving object between successive frames, a shift that is a motion vector that passes through the target block of the frame at the next time from the frame at the previous time By using the initial vector as the initial vector candidate vector, it is possible to give an appropriate motion vector as the initial value of the gradient method calculation, compared to the case where only the motion vector obtained in the past in the peripheral block is used as the initial vector candidate. Can do.

  Next, motion vector allocation will be described.

  FIG. 28 is a block diagram illustrating a configuration example of the vector allocation unit 14 of FIG.

  As described above, the vector allocating unit 14 performs a process of allocating the motion vector detected in the pixel of the frame t of the 24P signal to the pixel of the interpolation frame.

  The pixel information calculation unit 111 acquires the motion vector detected in the pixel of the frame t stored in the detection vector memory 13 in raster order from the upper left pixel, and the acquired motion vector is obtained at the frame at the next time. Extending in the t + 1 direction, the intersection of the motion vector and the interpolation frame is calculated. The pixel information calculation unit 111 sets a pixel (hereinafter referred to as an allocation target pixel) on an interpolation frame to which a motion vector is allocated from the calculated intersection, and selects a vector of information on the position of the motion vector and the allocation target pixel. Output to the unit 115.

  In addition, the image information calculation unit 111 calculates the allocation target pixel, the position P of the frame t associated with the motion vector, and the position Q of the frame t + 1, and uses the calculated position information as the evaluation value calculation unit 112. The result is output to the target pixel difference calculation unit 113.

  When the allocation target pixel and the position information are supplied from the pixel information calculation unit 111, the evaluation value calculation unit 112 sets a DFD calculation range (m × n) centered on the position P and the position Q on each frame. If the DFD calculation range is within the image frame, the evaluation value DFD of the allocation target pixel for the motion vector is obtained using the set DFD calculation range. The obtained evaluation value DFD is output to the vector evaluation unit 114.

  The pixel-of-interest difference calculation unit 113, when the allocation target pixel and the position information are supplied from the pixel information calculation unit 111, from the position P of the frame t and the position Q of the frame t + 1, the luminance difference absolute value with respect to the allocation target pixel And the calculated luminance difference absolute value is output to the vector evaluation unit 114.

  The vector evaluation unit 114 includes a pixel difference determination unit 121 and an evaluation value determination unit 122. Among them, the pixel difference determination unit 121 determines whether or not the luminance difference absolute value for the allocation target pixel supplied from the target pixel difference calculation unit 113 is smaller than a predetermined threshold value.

  When the pixel difference determination unit 121 determines that the luminance difference absolute value for the allocation target pixel is smaller than a predetermined threshold, the evaluation value determination unit 122 determines the allocation target pixel supplied from the evaluation value calculation unit 112. It is determined whether or not the evaluation value DFD is smaller than the minimum evaluation value of the DFD table included in the vector selection unit 115. When the evaluation value determination unit 122 determines that the evaluation value DFD of the allocation target pixel is smaller than the minimum evaluation value of the DFD table, the evaluation value determination unit 122 determines that the reliability of the motion vector corresponding to the allocation target pixel is high, and the allocation target The pixel evaluation value DFD is output to the vector selection unit 115.

  The vector selection unit 115 has a DFD table. In this DFD table, an evaluation value DFD0 when a 0 vector is assigned to each pixel on the interpolation frame is held as a minimum evaluation value for each pixel on the interpolation frame.

  Further, the vector selection unit 115, when the evaluation value DFD of the allocation target pixel is supplied from the vector evaluation unit 114, the allocation flag memory 16 based on the position information of the allocation target pixel calculated by the pixel information calculation unit 111. Is rewritten to 1 (true), and the minimum evaluation value in the DFD table of the allocation target pixel is rewritten to the evaluation value DFD of the allocation target pixel. Further, the vector selection unit 115 allocates the motion vector from the pixel information calculation unit 111 to the allocation target pixel in the allocation vector memory 15.

  Here, the concept of vector allocation processing will be described.

  For example, when an object is moving at a speed v on a 24P signal and a constant velocity assumption is made about the movement of an object between any two frames, a new interpolation of frames between frames of the 24P signal is considered. In this case, when the motion vector v is extended from the 24P signal object, the intersection of the motion vector v and the interpolation frame is the same object position and has the same velocity v.

  Therefore, by assigning the motion vector of the frame of the 24P signal (hereinafter also referred to as the original frame) detected by the vector detection unit 12 to the intersection of the motion vector and the interpolation frame of the 60P signal to be interpolated, The movement of each pixel on the interpolation frame can be obtained.

  FIG. 29 is a one-dimensional diagram illustrating an example of a motion vector detected in an original frame of a 24P signal and pixels of an interpolation frame of a 60P signal. A circle on each frame represents a pixel.

  In the example of FIG. 29, an interpolation frame F1 at time t + 0.4 and an interpolation frame F2 at time t + 0.8 are inserted between the frames t and t + 1 of the two 24P signals.

  In the example of FIG. 29, the motion vectors v1, v2, and v3 detected in the frame t by the vector detection unit 12 are extended in the frame t + 1 direction. When assigning these motion vectors to each pixel of the interpolation frames F1 and F2, a motion vector passing through the vicinity of each pixel on the interpolation frame is a candidate vector (hereinafter also referred to as an allocation candidate vector) assigned to that pixel. Is done.

  For example, the motion vector v1 from the pixel on the left side of the frame t to the vicinity of the fourth and fifth pixels from the left of the frame t + 1 is interpolated between the second and third pixels from the left of the interpolation frame F1. Since it passes between the third and fourth pixels from the left of the frame F2, this motion vector v1 is assigned to the pixels included in the region N1 near the point where the motion vector v1 intersects the interpolation frames F1 and F2. Allocation candidate vector to be assigned.

  The motion vector v2 from the third pixel from the left of the frame t to the vicinity of the second and third pixels from the left of the frame t + 1 is between the second and third pixels from the left of the interpolation frame F1. Since this passes between the second and third pixels from the left of the interpolation frame F2, the motion vector v2 is included in the region N2 near the point where the motion vector v2 intersects the interpolation frames F1 and F2. This is an assignment candidate vector assigned to the pixel.

  Similarly, the motion vector v3 from the fifth pixel from the left of the frame t to the vicinity of the fourth and fifth pixels from the left of the frame t + 1 is the fourth and fifth pixels from the left of the interpolation frame F1. Since this passes between the fourth and fifth pixels from the left of the interpolation frame F2, the motion vector v3 is included in the region N3 near the point where the motion vector v3 intersects the interpolation frames F1 and F2. It becomes an allocation candidate vector allocated to the pixels to be allocated.

  As described above, an allocation candidate vector to be allocated to each pixel of the interpolation frame is obtained from the motion vectors detected in the original frame, and interpolation is performed based on the evaluation value DFD or the like obtained for each allocation candidate vector. One motion vector is assigned to each pixel of the frame.

  In the example of FIG. 29, since no allocation candidate vector exists in the leftmost pixel and rightmost pixel (black circle in the figure) of the interpolation frame, a motion vector passing through the vicinity thereof is not shown. For such pixels for which there is no motion vector as an allocation candidate, a motion vector is allocated by an allocation compensation process described later.

  Next, motion vector evaluation will be described.

  FIG. 30 is a diagram for explaining motion vector evaluation.

  In the example of FIG. 30, the frame t, the interpolation frame F1, and the frame t + 1 are shown one-dimensionally from the bottom.

Motion vector sv _a is the pixel (x _a, y _a) of the frame t motion vector v _a detected at is one that is shifted (parallel movement) on the pixel G4 as allocated candidate vectors neighboring pixels G4 is there. A motion vector sv _a shifted on the pixel G4, the intersection of the frame t and frame t + 1, respectively point P, and the point Q.

Vector allocating unit 14, a first evaluation of motion vector sv _a, set the DFD operation range around the point P and point Q respectively, DFD operation range determined as to whether or not jutting out of the picture frame judge. Vector allocating unit 14, if the DFD operation range around the point P and the point Q is determined to protrude a picture frame, excluding a motion vector sv _a from the candidate.

Further, when the point P and the point Q belong to different objects, for example, the difference between the luminance value F _t (P) of the point P and the luminance value F _{t + 1} (Q) of the point Q becomes large. Accordingly, the vector allocating unit 14, a second evaluation of motion vector sv _a, using the points P and Q, determine the luminance difference absolute value dp of the pixel G4, the luminance difference absolute value dp is greater than a predetermined value It is determined whether or not. If the luminance difference absolute value dp is determined to be greater than the predetermined value, the vector allocating unit 14 excludes the motion vector sv _a from the candidate. The luminance difference absolute value dp is expressed by the following equation (11).

Furthermore, the vector allocating unit 14, a third evaluation of the motion vector sv _a, the evaluation determination by the difference absolute value that represents the correlation value DFD operation range around the point P and the point Q. That is, the vector allocating unit 14 uses the DFD operation range around the points P and Q, determine the evaluation value DFD of the motion vector sv _a in the pixel G4, obtained evaluation value DFD is the minimum evaluation of DFD table It is determined whether it is smaller than the value. The vector assigning unit 14 assigns a motion vector having the smallest evaluation value DFD among the obtained evaluation values DFD to the pixel G4.

  As described above, only the evaluation value DFD is obtained by evaluating the motion vector of the allocation candidate at the pixel of the interpolation frame using not only the evaluation value DFD of the allocation target pixel but also the luminance difference absolute value at the allocation target pixel. It is possible to assign a motion vector that is more reliable than when using.

  Next, the accuracy below the pixel of the motion vector will be described.

  The phase p + v on the previous frame t + 1 obtained by shifting the pixel position p of the frame t by the amount of the vector v may not coincide with the pixel position on the frame t + 1 of the 24p signal. The luminance value at + v is not defined. Therefore, in order to calculate the luminance difference absolute value dp and the evaluation value DFD with respect to the motion vector v having the accuracy below the pixel, the luminance value in the phase below the pixel must be generated. In order to obtain the luminance value For example, an interpolation process using the luminance values of the surrounding four pixels is used.

  FIG. 31 is a diagram illustrating the concept of four-point interpolation processing.

  In FIG. 31, arrow X indicates the horizontal direction in frame t + 1, and arrow Y indicates the vertical direction in frame t + 1. In this frame t + 1, a white circle represents a pixel position on the frame t + 1, and a black dot represents a position (granularity) below the pixel.

  Further, the pixels shown in the window E are obtained by enlarging the upper left black point p + v on the frame t + 1 and the surrounding four pixels. In window E, the alphabet in the white circle indicates the luminance value of the surrounding four pixels.

Assuming that the upper left black point p + v in this frame t + 1 is the phase p + v after shifting the pixel position p of the frame t (not shown) by the vector v amount, the luminance value F _{t + 1 of the} phase p + v (P + v) is obtained by using the horizontal pixel sub-component α and the vertical pixel sub-component β of the phase p + v and the luminance values L0 to L4 of the four peripheral pixels of the phase p + v. It is obtained as the sum of the inverse ratio of the distances of 4 pixels. That is, the luminance value F _{t + 1} (p + v) is expressed by the following equation (12), and the luminance value F _{t + 1} (p + v) is used for calculating the luminance difference absolute value dp and the evaluation value DFD. It is done.

  Note that this four-point interpolation is also applied to the calculation of the evaluation value DFD during the initial vector selection process, the vector detection process, and the allocation compensation process described later.

  Next, processing of each unit of the vector allocation unit 14 having the above configuration will be described.

  First, the entire vector allocation process performed by the vector allocation unit 14 will be described with reference to the flowchart of FIG. This process is a process performed in step S3 of FIG.

  In step S201, when a new original frame is input, the pixel information calculation unit 111 controls the vector selection unit 115 to initialize the allocation flag in the allocation flag memory 16 with 0 (False).

  In addition, in step S202, the pixel information calculation unit 111 initializes the allocation vector memory 15 with a zero vector.

  In step S203, the pixel information calculation unit 111 controls the evaluation value calculation unit 112 to calculate the evaluation value DFD0 using the 0 vector for all the pixels of the interpolation frame. The pixel information calculation unit 111 causes the vector selection unit 115 to hold the evaluation value DFD0 of the 0 vector calculated by the evaluation value calculation unit 112 as the minimum evaluation value for each pixel of the interpolation frame.

  In step S <b> 204, the pixel information calculation unit 111 selects a pixel from the original frame on the detection vector memory 13. For example, the pixels are selected in raster order from the upper left pixel of the frame.

  In step S205, the pixel information calculation unit 111 executes a pixel position calculation process. By this process, the allocation target pixel on the interpolation frame to which the motion vector detected in the pixel selected in step S204 is allocated is calculated, and the element that is associated with the motion vector on the basis of the calculated allocation target pixel A position on the frame is calculated. Details of the pixel position calculation process will be described later with reference to the flowchart of FIG.

  In step S <b> 206, the pixel information calculation unit 111 selects the allocation target pixel, and outputs the position information of the selected allocation target pixel and its motion vector to the vector selection unit 115. Further, the pixel information calculation unit 111 outputs, to the evaluation value calculation unit 112 and the pixel-of-interest calculation unit 113, information on the position on the original frame associated with the motion vector with reference to the selected allocation target pixel.

  In step S207, an allocation vector evaluation process is executed for the allocation target pixel. By this allocation vector evaluation process, the reliability of the motion vector in the allocation target pixel is determined, and the motion vector stored in the allocation vector memory 15 is rewritten by the motion vector determined to have high reliability. Details of the allocation vector evaluation process will be described later with reference to the flowchart of FIG.

  In step S208, the pixel information calculation unit 111 determines whether or not the processing of all allocation target pixels has been completed. If it is determined that the processing of all allocation target pixels has not been completed, the process returns to step S206. Then, the next allocation target pixel is selected, and the subsequent processing is repeated.

  On the other hand, if it is determined in step S208 that the processing for all the allocation target pixels has been completed, the process proceeds to step S209, and the pixel information calculation unit 111 has completed the processing for all the pixels in the original frame on the detection vector memory 13. It is determined whether or not.

  If the pixel information calculation unit 111 determines in step S209 that all pixels have not been processed, the pixel information calculation unit 111 returns to step S204, selects the next pixel of the original frame on the detection vector memory 13, and thereafter If the process is repeated and it is determined that the process has been completed for all pixels, the process is terminated. Thereafter, the process returns to step S3 in FIG. 5, and the subsequent processes are performed.

  Next, the pixel position calculation process performed in step S205 in FIG. 32 will be described with reference to the flowchart in FIG.

  In step S221, the pixel information calculation unit 111 acquires, from the detection vector memory 13, the motion vector detected by the pixel selected in step S204 in FIG. If the motion vector of the selected pixel is a 0 vector, the assigned vector memory 15 stores a 0 vector as an initial value in advance, so that subsequent steps S222 to S224 and steps S206 to S206 of FIG. The process of S208 is skipped.

  In step S222, the pixel information calculation unit 111 calculates the intersection of the motion vector acquired in step S221 and the interpolation frame.

  In step S223, the pixel information calculation unit 111 sets an allocation target pixel from the intersection of the motion vector and the interpolation frame. For example, when the intersection point coincides with the pixel position on the interpolation frame, the pixel at the intersection point is set as the allocation target pixel. Further, when the intersection does not coincide with the pixel position on the interpolation frame, as described above, four pixels near the intersection on the interpolation frame are set as the allocation target pixels.

  In step S224, the pixel information calculation unit 111 calculates a position on the original frame associated with the acquired motion vector with reference to each allocation target pixel. For example, the motion vector shifted to the allocation target pixel and the position of the intersection point on the original frame are calculated. The calculated position on the original frame is used for the evaluation value calculation unit 112 to obtain the evaluation value DFD, and for the pixel-of-interest difference calculation unit 113 to obtain the luminance difference absolute value. Thereafter, the process returns to step S205 in FIG. 32, and the subsequent processing is performed.

  Next, the allocation vector evaluation process performed in step S207 of FIG. 32 will be described with reference to the flowchart of FIG.

  In step S241, the evaluation value calculation unit 112 obtains DFD calculation ranges centered on the positions on the frames t and t + 1 when the position information on the original frame is supplied from the pixel information calculation unit 111. In step S242, it is determined whether or not the obtained DFD calculation range is within the image frame.

  If the evaluation value calculation unit 112 determines in step S242 that the DFD calculation range extends beyond the image frame, the evaluation value calculation unit 112 determines that the motion vector is not an allocation candidate vector to be allocated to the allocation target pixel, and ends the process.

  On the other hand, in step S242, when the evaluation value calculation unit 112 determines that the DFD calculation range is within the image frame, the evaluation value calculation unit 112 proceeds to step S243 and obtains the evaluation value DFD of the allocation target pixel using the DFD calculation range. The evaluation value DFD is output to the evaluation value determination unit 122 of the vector evaluation unit 114.

  In step S244, when the information on the position on the original frame is supplied from the pixel information calculation unit 111, the pixel-of-interest difference calculation unit 113 calculates the luminance difference absolute value dp in the allocation target pixel, and the calculated luminance difference absolute value dp is output to the pixel difference determination unit 121 of the vector evaluation unit 114.

  In step S245, the pixel difference determination unit 121 determines whether or not the luminance difference absolute value dp obtained by the target pixel difference calculation unit 113 is equal to or less than a predetermined threshold value, and the luminance difference absolute value dp is predetermined. If it is determined that the intersection of frames t and t + 1 belongs to different objects, the reliability of the motion vector is low and the allocation candidate vector to be allocated to the allocation target pixel is determined. It is determined that it should not occur, and the process is terminated.

  On the other hand, if the pixel difference determination unit 121 determines in step S245 that the luminance difference absolute value dp of the allocation target pixel is equal to or less than a predetermined threshold value, the process proceeds to step S246.

  In step S246, the evaluation value determination unit 122 refers to the DFD table held by the vector selection unit 115, and the evaluation value DFD of the allocation target pixel obtained by the evaluation value calculation unit 112 is stored in the DFD table. It is determined whether or not it is smaller than the minimum evaluation value of the allocation target pixel (in this case, the evaluation value DFD0 of the 0 vector).

  If the evaluation value determination unit 122 determines in step S246 that the evaluation value DFD of the allocation target pixel is equal to or greater than the minimum evaluation value, the evaluation value determination unit 122 determines that the motion vector is not highly reliable and ends the process.

  On the other hand, when it is determined in step S246 that the evaluation value DFD of the allocation target pixel is smaller than the minimum evaluation value, the evaluation value determination unit 122 has the highest reliability among the motion vectors compared so far. The evaluation value DFD of the allocation target pixel that is determined to be high and determined to have high reliability is output to the vector selection unit 115.

  In step S247, the vector selection unit 115 rewrites the allocation flag of the allocation target pixel to which the evaluation value DFD is supplied from the evaluation value determination unit 122, which is stored in the allocation flag memory 16, to 1 (True).

  In step S248, the vector selection unit 115 rewrites the minimum evaluation value corresponding to the allocation target pixel held in the DFD table with the evaluation value DFD determined to have high reliability by the evaluation value determination unit 122. .

  In step S249, the vector selection unit 115 rewrites the motion vector allocated to the allocation target pixel stored in the allocation vector memory 15 with the motion vector corresponding to the evaluation value DFD determined to have high reliability. . The vector selection unit 115 is supplied with the pixel to be allocated and its motion vector from the pixel information calculation unit 111 in step S206 of FIG. Thereafter, the process returns to step S207 in FIG. 33, and the subsequent processing is performed.

  Next, motion vector compensation will be described.

  FIG. 35 is a block diagram illustrating a configuration example of the allocation compensator 17 of FIG.

  As described above, the allocation compensator 17 performs a process of allocating the pixels of the interpolation frame to which the motion vector is not allocated by the vector allocator 14 with the motion vectors of the surrounding pixels being supplemented.

  The allocation vector determination unit 131 determines whether a motion vector is allocated to each pixel of the interpolation frame based on the allocation flag stored in the allocation flag memory 16. Also, the allocation vector determination unit 131 controls the vector compensation unit 132 to allocate a motion vector to a pixel to which no motion vector is allocated.

  For example, each pixel is focused on in reverse raster order with the lower right pixel of the interpolation frame as a reference, and a motion vector is assigned to a pixel to which no motion vector is assigned.

  The vector compensation unit 132 selects one motion vector having the highest reliability among the motion vectors assigned to the peripheral pixels of the pixel of interest based on the evaluation value DFD, and assigns the selected motion vector to the pixel of interest. The motion vector assigned to the pixel by the vector compensation unit 132 is stored in the assigned vector memory 15 in association with the pixel.

  FIG. 36 is a block diagram illustrating a detailed configuration example of the vector compensation unit 132.

  The compensation processing unit 141 includes a memory 151 that stores the minimum evaluation value DFD and the motion vector of the minimum evaluation value DFD as candidate vectors (hereinafter also referred to as compensation candidate vectors). The compensation processing unit 141 stores the zero vector evaluation value DFD as the minimum evaluation value and the zero vector as the compensation candidate vector in the memory 151 at the start of the processing.

  Further, the compensation processing unit 141 refers to the allocation flag memory 16 to acquire the motion vector allocated to the peripheral pixel of the target pixel from the allocation vector memory 15, and controls the evaluation value calculation unit 142 to perform the motion. The vector evaluation value DFD is calculated.

  The compensation processing unit 141 determines whether or not the evaluation value DFD obtained by the evaluation value calculation unit 142 is smaller than the minimum evaluation value stored in the memory 151, and the calculated evaluation value DFD is the minimum evaluation value. Is smaller than the compensation candidate vector and the minimum evaluation value in the memory 151, the calculated evaluation value DFD and its motion vector are rewritten, and finally the motion vector (compensation of the peripheral pixel having the smallest evaluation value DFD is obtained. (Candidate vector) is assigned to the pixel of interest as the motion vector of the pixel of interest. At this time, the compensation processing unit 141 rewrites the allocation flag of the allocation flag memory 16 of the target pixel to which the motion vector is allocated to 1 (True).

  The evaluation value calculation unit 142 calculates the evaluation value DFD of each motion vector of the peripheral pixels acquired from the allocation vector memory 15 using the input frames t and t + 1, and calculates the calculated evaluation value DFD. The data is output to the compensation processing unit 141.

  Here, the concept of compensation processing will be described.

  FIG. 37 is a diagram illustrating motion vector allocation compensation.

  In FIG. 37, a circle represents one pixel on the interpolation frame, and an arrow extending from the pixel represents a motion vector assigned to each pixel. A pixel without an arrow indicates that no motion vector is assigned to that pixel.

  In the example of FIG. 37, the central pixel P, to which no motion vector is assigned by the vector assigning unit 14, is the target pixel. Also, motion vectors V1 to V5 assigned to the upper left, upper, upper right, lower left, and lower right pixels, which are peripheral pixels of the pixel P, are selected as compensation candidate vectors. The motion vector V2 of the pixel located on the pixel P, which has been determined to have high reliability based on this, is assigned.

  As described above, the compensation is performed using the motion vectors assigned to the peripheral pixels. When attention is paid to a certain pixel, the motion of the pixel is the same as the motion of the peripheral pixel in the spatial direction and the temporal direction. This is based on the fact that the correlation is considered high.

  Next, the order of motion vector allocation compensation will be described.

  FIG. 38 is a diagram illustrating an example of allocated motion vectors.

  In the example of FIG. 38, 5 × 5 pixels on the interpolated frame are shown, and black circles thereof indicate pixels to which motion vectors are assigned by the vector assigning unit 14. That is, a white circle is a pixel that is a target in the allocation compensation process.

  For example, as shown in FIG. 39, the motion vector allocation compensation is performed by paying attention to the pixels in reverse raster order with the pixel at the lower right of the interpolation frame as a reference.

  Therefore, when a motion vector is assigned to each pixel as shown in FIG. 38, attention is first paid to the pixel P1 located at the second row from the bottom and the second column from the right as the assignment compensation target. Of the motion vectors of the surrounding pixels, the motion vector having the highest reliability is assigned to the pixel P1. In the example of FIG. 38, motion vectors assigned to the five pixels at the lower right, lower, lower left, right, upper, and upper left of the pixel P1 are selected as compensation candidate vectors, and one of the motion vectors is the pixel P1. Assigned.

  FIG. 40 is a diagram illustrating a state in which a motion vector is assigned to the pixel P1.

  In the example of FIG. 40, the motion vector of the pixel on the right side is selected and assigned to the pixel P1. In this way, when a motion vector is assigned to the pixel of interest, the pixel of interest that is the object of assignment compensation is switched.

  That is, since the pixel of interest is selected in reverse raster order, next to the pixel P1, as shown in FIG. 41, a pixel P2 positioned to the left of the pixel P1 as a pixel to which no motion vector is assigned is the pixel of interest. The motion vector having the highest reliability among the motion vectors of the surrounding pixels is assigned to the pixel P2. In the example of FIG. 41, motion vectors assigned to the seven pixels at the bottom right, bottom, bottom left, right, left, top right, and top of the pixel P2 are selected as compensation candidate vectors, and one of the motion vectors is Assigned to pixel P2.

  As described above, in the allocation compensation processing, when the target pixel is switched, the motion vector allocated to the pixel focused on immediately before is also selected as a candidate motion vector allocated to the pixel currently focused on. Become. In the example of FIG. 41, a motion vector newly assigned to the pixel P1 that has been noticed immediately before is also selected as a candidate motion vector to be assigned to the pixel P2.

  As described above, the allocation compensation unit 17 selects the pixel of interest in reverse raster order, and performs motion vector allocation compensation. The selection of the pixel of interest in reverse raster order is based on the following reason.

  FIG. 42 is a diagram illustrating motion vector detection performed by the vector detection unit 12 and an example of a motion vector detected thereby.

  As described above, in the vector detection unit 12, the pixels are focused on in the raster order, and the motion vector is detected using the gradient method. As described with reference to FIG. 12 and the like, the calculation by the gradient method is a candidate for giving an offset when the motion vector previously detected in the peripheral pixels of the pixel of interest obtains the motion vector of another pixel. As the process proceeds with the pixel of interest switched in raster order, that is, the pixel located at the lower right of the frame, the detected motion vector represents the actual motion of the object. In other words, it will converge.

  In the example of FIG. 42, an object O is shown, and an arrow extending from the object O represents a motion vector detected by pixels belonging to the object O. As shown in FIG. 42, the ideal motion of the object O, which is indicated by the thick arrow A as the motion vector detected in the pixel belonging to the object O is detected at the lower right side where the processing has progressed. It is close to (actual movement).

  Therefore, in the allocation compensation, a motion vector that has converged once, which is considered to represent the actual motion most accurately, is selected as the motion vector of the allocation candidate, and motion vectors allocated to other peripheral pixels Depending on the evaluation result, the target pixel is selected and processed in reverse raster order so that it is assigned to the target pixel. If the allocation compensation is also processed in raster order, the motion vector assigned to the lower right pixel, which is considered to represent the actual motion of the object most accurately, is the compensation candidate assigned to the other pixels. It is never selected as a vector. That is, it is not compensated by the most accurate motion vector.

  FIG. 43 is a diagram illustrating an example of the compensated motion vector. In FIG. 43, a dotted arrow extending from a pixel belonging to the object O represents a compensated motion vector.

  In the motion vector allocation compensation, as described with reference to FIG. 41, the motion vector allocated immediately before is also set as the motion vector of the next candidate pixel allocation candidate, and is allocated depending on the evaluation result. As the processing proceeds in sequence, the converged motion vector at the lower right is assigned to the upper left pixel one after another as shown in FIG.

  In the example of FIG. 43, the motion vectors newly compensated for the pixels belonging to the object O are assigned in order from the lower right, and these are the converged motion vectors (the lower right pixels are detected at the time of detection). The detected motion vector is close to or the same.

  As described above, since the motion vector detection proceeds in the raster order, the motion vector converges as the processing target moves to the lower right of the frame, and conversely, the allocation compensation proceeds in the reverse raster order. The converged motion vector is spatially propagated to the upper left. FIG. 44 illustrates this.

  In this way, the allocation candidates are advanced in reverse raster order, so that it is possible to allocate a motion vector that is considered to represent the actual motion most accurately to a pixel that has not yet been allocated a motion vector. Is assigned to different pixels in the same object, it is possible to prevent image quality from deteriorating.

  In the above description, the allocation compensation process is performed in reverse raster order. However, when the motion vector detection process is performed in reverse raster order, the allocation compensation process is performed in raster order opposite to the detection process. . That is, the motion vector detection process and the allocation compensation process are performed in the reverse processing order.

  Next, motion vector evaluation will be described.

  The evaluation value DFD serving as a reference for selecting a motion vector to be actually allocated from the motion vectors of the compensation candidates is obtained as shown in FIGS. 45 to 47, for example.

  FIG. 45 is a diagram illustrating an example of obtaining the evaluation value DFD of the 0 vector S0 selected from the compensation candidate vectors of the target pixel P shown on the left side.

  The evaluation value DFD for the zero vector S0 is based on the pixel of interest P on the interpolation frame as a reference, and the DFD calculation range D1-1 centering on the respective intersections on the frame t and the frame t + 1 associated with the zero vector S0. , D1-2 is used to calculate the above equation (1).

  FIG. 46 is a diagram illustrating an example of obtaining the evaluation value DFD of the motion vector VK1 selected from the compensation candidate vectors of the target pixel P shown on the left side.

  The evaluation value DFD for the motion vector VK1 is based on the pixel of interest P on the interpolated frame as a reference, and the DFD calculation range D2-1 centering on each intersection on the frame t and the frame t + 1 associated with the motion vector VK1. , D2-2, and calculating the above equation (1).

  FIG. 47 is a diagram illustrating an example of obtaining the evaluation value DFD of the motion vector VK2 selected from the compensation candidate vectors of the target pixel P shown on the left side.

  The evaluation value DFD for the motion vector VK2 is based on the pixel of interest P on the interpolation frame as a reference, and the DFD calculation range D3-1 centering on each intersection on the frame t and the frame t + 1 associated with the motion vector VK2. , D3-2 and calculating the above equation (1).

  The evaluation value DFD is obtained by the same process for other compensation candidate vectors assigned to the peripheral pixels of the pixel of interest P, and among them, the compensation candidate vector having the smallest evaluation value DFD is assigned to the pixel of interest P. Selected as a motion vector.

  Next, processing of each unit of the allocation compensation unit 17 having the above configuration will be described.

  First, the entire allocation compensation processing performed by the allocation compensation unit 17 will be described with reference to the flowchart of FIG. This process is a process performed in step S4 of FIG.

  In step S301, the allocation vector determination unit 131 sets one pixel selected from the pixels in the interpolation frame in reverse raster order based on the pixel located at the lower rightmost position as a target pixel.

  In step S302, the allocation vector determination unit 131 determines whether the allocation flag of the pixel of interest stored in the allocation flag memory 16 is 0 (False), and if it is determined that it is 0 (False), Recognizing that no motion vector is assigned to the pixel of interest currently selected, the process proceeds to step S303.

  In step S303, the vector compensation unit 132 executes vector compensation processing. By this vector compensation processing, a motion vector that minimizes the evaluation value DFD is stored in the memory 151 as a compensation candidate vector from among the motion vectors assigned to the peripheral pixels. Details of the vector compensation processing will be described later with reference to the flowchart of FIG.

  In step S304, the compensation processing unit 141 (FIG. 36) of the vector compensation unit 132 allocates the compensation candidate vector stored in the memory 151 to the allocation vector memory 15 as the motion vector of the pixel of interest.

  In step S305, the compensation processing unit 141 rewrites the allocation flag of the target pixel stored in the allocation flag memory 16 to 1 (True), and proceeds to step S306.

  On the other hand, if the allocation vector determination unit 131 determines in step S302 that the allocation flag of the pixel of interest is not 0 (False), the allocation vector determination unit 131 recognizes that the motion vector has already been allocated to the pixel of interest, and performs steps S303 to S303. The process of S305 is skipped.

  In step S306, the allocation vector determination unit 131 determines whether or not the processing for all the pixels in the interpolation frame has been completed. If it is determined that the processing for all the pixels has not been completed, the process returns to step S301. The subsequent processing is repeatedly executed. That is, one pixel is selected in reverse raster order, and the process using the selected pixel as the target pixel is repeated.

  If the allocation vector determination unit 131 determines in step S306 that the processing for all the pixels in the interpolation frame has been completed, the allocation vector determination unit 131 ends the allocation compensation processing. Thereafter, the process returns to step S4 in FIG. 5 and the subsequent processing is performed.

  Next, the vector compensation processing performed in step S303 in FIG. 48 will be described with reference to the flowchart in FIG.

  In step S321, the evaluation value calculation unit 142 of the vector compensation unit 132 calculates an evaluation value DFD0 using the 0 vector for the pixel of interest. The calculation of the evaluation value DFD0 using the 0 vector is performed, for example, as described with reference to FIG. The obtained evaluation value DFD0 is output to the compensation processing unit 141.

  In step S322, the compensation processing unit 141 stores the evaluation value DFD0 obtained in step S321 in the memory 151 as the minimum evaluation value. In step S323, the compensation processing unit 141 stores the 0 vector in the memory 151 as a compensation candidate vector.

  In step S324, the compensation processing unit 141 selects one of the eight pixels around the pixel of interest. For example, among the eight peripheral pixels, the peripheral pixels are selected in reverse raster order from the lower right pixel.

  In step S325, the compensation processing unit 141 refers to the allocation flag memory 16 to determine whether or not there is a motion vector in the peripheral pixel selected in step S324, that is, the allocation flag of the peripheral pixel is 1 (True). It is determined whether or not there is.

  In step S325, if the compensation processing unit 141 determines that a motion vector exists in the surrounding pixels, the compensation processing unit 141 proceeds to step S326, and acquires the motion vector of the surrounding pixels from the allocated vector memory 15. At this time, the motion vectors of the peripheral pixels are also supplied from the assigned vector memory 15 to the evaluation value calculation unit 142.

  In step S327, the evaluation value calculation unit 142 calculates the evaluation value DFD of the motion vector of the peripheral pixels acquired in step S326, and outputs the calculated evaluation value DFD to the compensation processing unit 141.

  In step S328, the compensation processing unit 141 determines whether or not the evaluation value DFD supplied from the evaluation value calculation unit 142 is smaller than the minimum evaluation value of the target pixel stored in the memory 151.

  When the compensation processing unit 141 determines in step S328 that the newly supplied evaluation value DFD is smaller than the minimum evaluation value of the pixel of interest, the compensation processing unit 141 proceeds to step S329 and stores the minimum value stored in the memory 151. The evaluation value is rewritten to the newly supplied evaluation value DFD.

  In step S330, the compensation processing unit 141 rewrites the compensation candidate vector stored in the memory 151 with the motion vector of the peripheral pixels acquired in step S326, and the process proceeds to step S331.

  On the other hand, when it is determined in step S325 that the allocation flag of the peripheral pixel stored in the allocation flag memory 16 is 0 (False), and it is determined that there is no motion vector allocated to the peripheral pixel, the compensation processing unit In step 141, the process of steps S326 to S330 is skipped, and the process proceeds to step S331.

  In step S328, when the compensation processing unit 141 determines that the newly obtained evaluation value DFD is greater than or equal to the minimum evaluation value of the target pixel stored in the memory 151, the compensation processing unit 141 performs the processing in steps S329 and S330. Skip to step S331.

  In step S331, the compensation processing unit 141 determines whether or not all the processes for the peripheral pixels of the pixel of interest have been completed. If it is determined that all the processes have not been completed, the process returns to step S324, and the next peripheral A pixel is selected and the subsequent processing is repeated.

  If the compensation processing unit 141 determines in step S331 that all the processing for the peripheral pixels of the pixel of interest has been completed, the compensation processing unit 141 ends the vector compensation processing. Thereafter, the process returns to step S303 in FIG. 48, and the subsequent processing is performed.

  In the above, when each pixel is selected as the target pixel and no motion vector is assigned to the selected target pixel, one motion vector selected from the evaluation result of the motion vectors of the surrounding pixels is the target. Although the case where the allocation compensation process is performed for each pixel so as to be allocated to the pixel has been described, this process may be performed not on a pixel basis but on a block basis. When the allocation compensation processing is performed in units of blocks, the same motion vector allocated to the block is allocated to all the pixels included in one block.

  Here, the entire allocation compensation process performed in units of blocks will be described with reference to the flowchart of FIG. That is, this process is a process corresponding to the process of FIG. 48 performed in units of pixels.

  In step S341, the allocation vector determination unit 131 sets, as a block of interest, one block selected in reverse raster order based on the block at the lowermost right among the blocks obtained by dividing the interpolation frame for each m × n pixels.

  In step S342, the allocation vector determination unit 131 determines whether there is a pixel whose allocation flag stored in the allocation flag memory 16 is 0 (False) among the pixels constituting the target block. When it is determined that there is such a pixel, it is recognized that there is a pixel having no motion vector among the pixels constituting the target block.

  In step S343, the vector compensation unit 132 executes vector compensation processing. By this vector compensation processing, a motion vector having the smallest evaluation value DFD is stored in the memory 151 as a compensation candidate vector from among the motion vectors assigned to the peripheral block (pixels of the peripheral block) of the target block. That is, here, the same processing as that described with reference to FIG. 49 is performed except that the target of interest is not a pixel but a block.

  In step S344, the compensation processing unit 141 of the vector compensation unit 132 selects one pixel from among the pixels constituting the target block as the target pixel, and the allocation flag of the target pixel stored in the allocation flag memory 16 is stored. Is determined to be 0 (False).

  If the compensation processing unit 141 determines in step S344 that the allocation flag of the pixel of interest is 0 (False), the compensation processing unit 141 proceeds to step S345 and uses the compensation candidate vector stored in the memory 151 as the motion vector of the pixel of interest. Allocates to the allocation vector memory 15.

  In step S346, the compensation processing unit 141 rewrites the allocation flag of the pixel of interest stored in the allocation flag memory 16 to 1 (True), and proceeds to step S347.

  On the other hand, if the allocation vector determination unit 131 determines in step S344 that the allocation flag of the target pixel is not 0 (False), the allocation vector determination unit 131 recognizes that a motion vector has already been allocated to the target pixel, and step S345, The process of S346 is skipped.

  In step S347, the compensation processing unit 141 determines whether or not the processing for all the pixels in the target block has been completed. If it is determined that the processing for all the pixels has not been completed, the compensation processing unit 141 proceeds to step S348.

  In step S348, the compensation processing unit 141 selects a pixel that has not been focused yet among the pixels constituting the focused block as the focused pixel, and repeats the processing from step S344 onward.

  On the other hand, when it is determined in step S342 that there is no pixel whose allocation flag stored in the allocation flag memory 16 is 0 (False) among the pixels constituting the target block, or in step S347, the target If it is determined that all the pixels in the block have been processed, the process proceeds to step S349, and the compensation processing unit 141 determines whether the processing for all the blocks in the interpolation frame has been completed.

  If the compensation processing unit 141 determines in step S349 that all blocks have not been processed, the compensation processing unit 141 returns to step S341, selects the next block as the target block in reverse raster order, and performs the subsequent processing. Execute.

  In step S349, when the compensation processing unit 141 determines that the processing for all the blocks of the interpolation frame has been completed, the compensation processing unit 141 ends the processing.

  In this way, it is also possible to perform allocation compensation processing in units of blocks.

  Next, the interpolation frame interpolation will be described.

  FIG. 51 is a block diagram illustrating a configuration example of the image interpolation unit 18 of FIG.

  As described above, the image interpolation unit 18 performs a process of interpolating and generating the pixel value of the interpolation frame using the motion vector assigned to the interpolation frame and the pixel values of the frames t and t + 1.

  The interpolation control unit 161 acquires a motion vector allocated to one pixel selected from the pixels of the interpolation frame from the allocation vector memory 15, and associates the motion vector with the acquired motion vector using the pixel of the interpolation frame as a reference. The position on the frame t is obtained. In addition, the interpolation control unit 161 obtains the spatial shift amount from the obtained position on the frame t and the pixel position of the interpolation frame, and outputs the spatial shift amount to the spatial filter 162-1.

  Similarly, the interpolation control unit 161 obtains a position on the frame t + 1 associated with the acquired motion vector with reference to the pixel of the interpolation frame. In addition, the interpolation control unit 161 obtains the spatial shift amount from the obtained pixel position on the frame t + 1 and the interpolated frame pixel position, and outputs the spatial shift amount to the spatial filter 162-2.

  Furthermore, the interpolation control unit 161 obtains an interpolation weight between the frame t and the frame t + 1 based on the time phase (time) of the interpolated frame, and sends the obtained interpolation weight to the multipliers 163-1 and 163-2. Set.

  For example, when the time of the interpolated frame is a time separated by “k” from the time t + 1 and is a time separated by “1-k” from the time t (the interpolated frame is time t and time The interpolation control unit 161 sets the interpolation weight of “1-k” to the multiplier 163-1 and the multiplier 163-2. Is set to “k”.

  Spatial filters 162-1 and 162-2 are constituted by cubic filters, for example. The spatial filter 162-1 calculates the pixel value of the pixel on the frame t corresponding to the pixel of the interpolation frame based on the pixel value of the pixel on the frame t and the spatial shift amount supplied from the interpolation control unit 161. The obtained pixel value is output to the multiplier 163-1.

  The spatial filter 162-2 is a pixel on the frame t + 1 corresponding to the pixel of the interpolation frame based on the pixel value of the pixel on the frame t + 1 and the spatial shift amount supplied from the interpolation control unit 161. And the obtained pixel value is output to the multiplier 163-2.

  In addition, when the position of the pixel of the interpolation frame is a position equal to or smaller than the pixel in the frame t or the frame t + 1, the pixel value at the position equal to or smaller than the pixel is, for example, the above-described linearity based on the distance from the surrounding four pixels. It is obtained by interpolation.

  The multiplier 163-1 multiplies the pixel value of the pixel on the frame t supplied from the spatial filter 162-1 by the interpolation weight “1-k” set by the interpolation control unit 161, and weighted pixels. The value is output to the adder 164.

  The multiplier 163-2 multiplies the pixel value of the pixel on the frame t + 1 supplied from the spatial filter 162-2 by the interpolation weight “k” set by the interpolation control unit 161, and obtains a weighted pixel. The value is output to the adder 164.

  The adder 164 generates the pixel value of the pixel of the interpolation frame by adding the pixel value supplied from the multiplier 163-1 and the pixel value supplied from the multiplier 163-2, The pixel value of the pixel in the interpolation frame is output to the buffer 165.

  The buffer 165 generates an interpolated frame based on the pixel value supplied from the adder 164, and sets the generated interpolated frame and the stored frame t + 1 to a preset 60P signal time. Output at the timing according to the phase. The buffer 165 is supplied with the frame t + 1 from the outside and stores it.

  Next, the interpolation processing of the image interpolation unit 18 having the above configuration will be described with reference to the flowchart of FIG. This process is a process performed in step S5 of FIG.

  In step S401, the interpolation control unit 161 obtains the interpolation weight of the interpolation frame between the frame t and the frame t + 1 based on the time phase of the interpolation frame, and uses the obtained interpolation weight as the multiplier 163-1. And 163-2, respectively.

  In step S402, the interpolation control unit 161 selects a pixel of the interpolation frame. For example, pixels are selected in raster order using the upper left pixel of the interpolation frame as a reference.

  In step S403, the interpolation control unit 161 acquires the motion vector allocated to the selected pixel from the allocation vector memory 15, and uses the pixel of the interpolation frame as a reference on the frame t associated with the acquired motion vector. Find the position. In addition, the interpolation control unit 161 obtains the spatial shift amount from the obtained position on the frame t and the pixel position of the interpolation frame, and outputs the spatial shift amount to the spatial filter 162-1.

  Similarly, the interpolation control unit 161 obtains a position on the frame t + 1 associated with the acquired motion vector with reference to the pixel of the interpolation frame. In addition, the interpolation control unit 161 obtains the spatial shift amount from the obtained pixel position on the frame t + 1 and the interpolated frame pixel position, and outputs the spatial shift amount to the spatial filter 162-2.

  In step S <b> 404, the spatial filters 162-1 and 162-2 determine the pixels of the interpolation frame based on the pixel values of the pixels on the frames t and t + 1 and the spatial shift amount supplied from the interpolation control unit 161. Are obtained, and the obtained pixel values are output to the multipliers 163-1 and 163-2, respectively.

  In step S405, the multiplier 163-1 multiplies the pixel value of the pixel on the frame t supplied from the spatial filter 162-1 by the interpolation weight “1-k” set by the interpolation control unit 161. The weighted pixel value is output to the adder 164. Further, the multiplier 163-2 multiplies the pixel value of the pixel on the frame t + 1 supplied from the spatial filter 162-2 by the interpolation weight “k” set by the interpolation control unit 161, and performs weighting. The obtained pixel value is output to the adder 164.

  In step S406, the adder 164 generates the pixel value of the pixel of the interpolation frame by adding the pixel value supplied from the multiplier 163-1 and the pixel value supplied from the multiplier 163-2. The generated pixel value is output to the buffer 165.

  In step S407, the interpolation control unit 161 determines whether or not all the pixels in the interpolation frame have been processed. If the interpolation control unit 161 determines that all the pixels have not been processed, the process returns to step S402. The subsequent processing is repeated.

  If the interpolation control unit 161 determines in step S407 that all the pixels in the interpolation frame have been processed, the interpolation control unit 161 ends the image interpolation process. Thereafter, the process returns to step S5 in FIG. 5 and the subsequent processing is performed.

  24P-60P frame frequency conversion is realized in the signal processing apparatus 1 by the series of processes including the motion vector detection process, the allocation process, the allocation compensation process, and the interpolation process.

  In the above, in each case of selecting a motion vector as a detection result, selecting a motion vector to be assigned to a pixel of an interpolation frame, selecting a motion vector to be compensated, etc., the expression (1) The evaluation value DFD obtained in step (3) is used to evaluate the motion vector. This evaluation value DFD is the sum of absolute values of differences in luminance values in a certain range of different frames as can be seen from equation (1). Therefore, when the range for which the evaluation value DFD is calculated is set, for example, in a flat part where the change in luminance value is small, the absolute value sum of the difference cannot be obtained accurately, and so Therefore, there is a possibility that a more suitable motion vector cannot be selected from the evaluation value DFD lacking accuracy. This failure to select a suitable motion vector results in a failure of the generated image (frame).

  Therefore, the deviation DFD (vDFD), which is an evaluation value that can accurately represent the difference even when the range where the luminance value changes little, such as a flat portion, is used, is used when evaluating the motion vector. You may do it. This deviation DFD is, for example, an absolute value sum of differences obtained by subtracting the DC component from the luminance value of each pixel in the target range using the average luminance value of the block of interest consisting of 2 × 2 pixels as an offset. 13).

Here, m × n is a target range for obtaining the deviation DFD. Pave _{t + 1} and Pave _t are average luminance values of pixels in the target range, P _{t + 1} (x, y) and P _t (x, y) are luminance values of pixels in the target region, and Pmin _{t + 1,} Pmin _t is the minimum luminance value in the region of interest.

  When the DC component is cut, the difference in luminance value can be detected with high accuracy. Therefore, when the calculation range is set in a flat portion or the like by using the deviation DFD as calculated by the equation (13) Even so, it is possible to correctly evaluate the motion vector.

  In addition, the evaluation of the motion vector using the deviation DFD is performed in addition to the evaluation using the evaluation value DFD obtained by Expression (1), so that the motion vector is selected only when it is determined to be correct in both evaluations. Thus, a more accurate motion vector can be selected.

  Instead of the deviation DFD, a DFD with an offset (oDFD) obtained by the following equation (14) may be used, and the motion vector may be evaluated accordingly.

  Here, with reference to the flowchart of FIG. 53, a process for evaluating a motion vector using the above deviation DFD and compensating the motion vector will be described. This process corresponds to the process of FIG.

  In step S501, the evaluation value calculation unit 142 of the vector compensation unit 132 calculates an evaluation value DFD_zr from Expression (1) using the 0 vector for the pixel of interest. The obtained evaluation value DFD_zr is output to the compensation processing unit 141.

  In step S502, the compensation processing unit 141 stores the evaluation value DFD_zr obtained in step S501 in the memory 151 as the minimum evaluation value minDFD.

  In step S503, the evaluation value calculation unit 142 of the vector compensation unit 132 calculates a deviation DFD (vDFD_zr) from the equation (13) using the 0 vector for the pixel of interest. The obtained deviation DFD (vDFD_zr) is also output to the compensation processing unit 141.

  In step S504, the compensation processing unit 141 stores the 0 vector in the memory 151 as a compensation candidate vector.

  In step S <b> 505, the compensation processing unit 141 selects one pixel among the eight pixels around the pixel of interest. For example, among the eight peripheral pixels, the peripheral pixels are selected in reverse raster order from the lower right pixel.

  In step S506, the compensation processing unit 141 refers to the allocation flag memory 16 to determine whether or not a motion vector exists in the peripheral pixel selected in step S505, that is, the allocation flag of the peripheral pixel is 1 (True). It is determined whether or not there is.

  In step S506, if the compensation processing unit 141 determines that a motion vector exists in the surrounding pixels, the compensation processing unit 141 proceeds to step S507, and acquires the motion vector of the surrounding pixels from the allocation vector memory 15. At this time, the motion vectors of the peripheral pixels are also supplied from the assigned vector memory 15 to the evaluation value calculation unit 142.

  In step S508, the evaluation value calculation unit 142 calculates a deviation DFD (vDFD_tmp) from equation (13) using the motion vector acquired in step S507. The obtained deviation DFD (vDFD_tmp) is output to the compensation processing unit 141.

  In step S509, the compensation processing unit 141 stores the deviation DFD (vDFD_tmp) supplied from the evaluation value calculation unit 142 in the memory 151. The deviation DFD (vDFD_zr) obtained in step S503 is a predetermined parameter. It is determined whether or not it is smaller than the sum of the value TH_cmp.

  In step S509, when the compensation processing unit 141 determines that the deviation DFD (vDFD_tmp) is smaller than the deviation DFD (vDFD_zr) added with the value TH_cmp, the process proceeds to step S510, and the motion vector acquired in step S507 is obtained. The evaluation value DFD_tmp is calculated from the equation (1). The obtained evaluation value DFD_tmp is output to the compensation processing unit 141.

  In step S511, the compensation processing unit 141 determines whether or not the evaluation value DFD_tmp calculated in step S510 is smaller than the minimum evaluation value minDFD of the target pixel stored in the memory 151.

  If the compensation processing unit 141 determines in step S511 that the evaluation value DFD_tmp is smaller than the minimum evaluation value minDFD, the compensation processing unit 141 proceeds to step S512, and the minimum evaluation value minDFD stored in the memory 151 is obtained using the motion vector. The evaluation value is rewritten to DFD_tmp.

  In step S513, the compensation processing unit 141 rewrites the compensation candidate vector stored in the memory 151 with the motion vector acquired in step S507, and proceeds to step S514.

  On the other hand, if the allocation flag of the surrounding pixels is 0 (False) in step S506, if it is determined that there is no motion vector allocated to the surrounding pixels, the compensation processing unit 141 performs the processing of steps S507 to S513. Skip to step S514.

  In step S511, when the compensation processing unit 141 determines that the evaluation value DFD_tmp is equal to or greater than the minimum evaluation value minDFD stored in the memory 151, the processing of steps S512 and S513 is skipped, and the process proceeds to step S514. .

  In step S514, the compensation processing unit 141 determines whether or not all the processes for the peripheral pixels of the target pixel have been completed, and if it is determined that all the processes have not been completed, the process returns to step S505, and the next peripheral A pixel is selected and the subsequent processing is repeated.

  Also, in step S509, when the compensation processing unit 141 determines that the deviation DFD (vDFD_tmp) is equal to or greater than the value obtained by adding the value TH_cmp to the deviation DFD (vDFD_zr), the process returns to step S505 in the same manner. Repeat the process.

  If the compensation processing unit 141 determines in step S514 that all the processing for the peripheral pixels of the target pixel has been completed, the compensation processing unit 141 ends the vector compensation processing. Thereafter, the process returns to step S303 in FIG. 48, and the subsequent processing is performed.

  As described above, the motion vector is evaluated based on the two criteria of the deviation DFD and the evaluation value DFD obtained from Expression (1), so that the motion vector is compensated with a more accurate motion vector satisfying the two criteria. It becomes possible.

  Note that, as described above, the evaluation of the motion vector using the deviation DFD can be used at times other than the time of motion vector compensation.

  Next, with reference to the flowchart of FIG. 54, a process for evaluating a motion vector using the deviation DFD and selecting an initial vector will be described. This process corresponds to the process of FIG. Here, it is assumed that a zero vector is set as an initial candidate vector at the start of processing (corresponding to the processing after step S36 in FIG. 22).

  In step S521, the evaluation value calculation unit 55 (FIG. 7) of the initial vector selection unit 31 calculates the evaluation value DFD_zr from Expression (1) using the 0 vector for the block of interest. The obtained evaluation value DFD_zr is output to the evaluation value comparison unit 56.

  In step S522, the evaluation value comparison unit 56 stores the evaluation value DFD_zr obtained in step S521 in the optimum candidate storage register 57 as the minimum evaluation value minDFD.

  In step S523, the evaluation value calculation unit 55 calculates the deviation DFD (vDFD_zr) from Expression (13) using the 0 vector for the block of interest. The obtained deviation DFD (vDFD_zr) is also output to the evaluation value comparison unit 56.

  In step S524, the evaluation value comparison unit 56 stores the zero vector in the optimum candidate storage register 57 as an initial candidate vector.

  In step S525, the evaluation value calculator 55 selects one of the peripheral pixels.

  In step S526, the evaluation value calculator 55 determines whether or not a motion vector exists in the peripheral pixel selected in step S525.

  In step S526, when the evaluation value calculation unit 55 determines that there is a motion vector in the peripheral pixel, the process proceeds to step S527, and acquires the motion vector of the peripheral pixel. For example, the motion vectors of the peripheral pixels read from the detection vector memory 13 by the detection vector acquisition unit 52 are output to the evaluation value calculation unit 55 via the offset position calculation unit 54.

  In step S528, the evaluation value calculator 55 calculates a deviation DFD (vDFD_tmp) from the equation (13) using the motion vector acquired in step S527. The obtained deviation DFD (vDFD_tmp) is output to the evaluation value comparison unit 56.

  In step S529, the evaluation value comparison unit 56 stores the deviation DFD (vDFD_tmp) supplied from the evaluation value calculation unit 55 in the optimum candidate storage register 57. The deviation DFD (vDFD_zr) obtained in step S523. ) Is added to a value TH_cmp which is a predetermined parameter.

  In step S529, when the evaluation value calculation unit 55 determines that the deviation DFD (vDFD_tmp) is smaller than the minimum evaluation value minDFD of the target pixel plus the value TH_cmp, the process proceeds to step S530 and acquired in step S527. The evaluation value DFD_tmp is calculated from the equation (1) using the motion vector. The obtained evaluation value DFD_tmp is output to the evaluation value comparison unit 56.

  In step S531, the evaluation value comparison unit 56 determines whether or not the evaluation value DFD_tmp calculated in step S530 is smaller than the minimum evaluation value minDFD stored in the optimum candidate storage register 57.

  If the evaluation value comparison unit 56 determines in step S531 that the evaluation value DFD_tmp is smaller than the minimum evaluation value minDFD, the evaluation value comparison unit 56 proceeds to step S532, and uses the minimum evaluation value minDFD stored in the optimum candidate storage register 57 as the motion vector. To the evaluation value DFD_tmp obtained using.

  In step S533, the evaluation value comparison unit 56 rewrites the initial candidate vector stored in the optimum candidate storage register 57 with the motion vector acquired in step S527, and proceeds to step S534.

  On the other hand, if it is determined in step S526 that there is no motion vector in the surrounding pixels, the evaluation value calculation unit 55 skips the processes in steps S527 to S533 and proceeds to step S534.

  If the evaluation value comparison unit 56 determines in step S531 that the evaluation value DFD_tmp is greater than or equal to the minimum evaluation value minDFD stored in the optimum candidate storage register 57, the process of steps S532 and S533 is skipped. The process proceeds to step S534.

  In step S534, the evaluation value calculation unit 55 determines whether or not all the candidate vectors have been processed. If it is determined that all the processes have not ended, the evaluation value calculation unit 55 returns to step S525 and performs the subsequent processing. Run repeatedly.

  Also, in step S529, when the evaluation value calculation unit 55 determines that the deviation DFD (vDFD_tmp) is greater than or equal to the value obtained by adding the value TH_cmp to the deviation DFD (vDFD_zr), the process returns to step S525. Repeat the process.

  If the evaluation value calculation unit 55 determines in step S534 that all candidate vectors have been processed, the evaluation value calculation unit 55 proceeds to step S535 and determines the initial candidate vector stored in the optimum candidate storage register 57 as the initial vector V0. And output. Thereafter, the process returns to step S22 in FIG. 21, and the subsequent processing is performed.

  As described above, the evaluation of the motion vector using the deviation DFD can be employed other than in the case of motion vector allocation compensation.

  The series of processes described above can be executed by hardware or can be executed by software. When a series of processing is executed by software, a program constituting the software executes various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program recording medium in a general-purpose personal computer or the like.

  FIG. 55 is a block diagram showing an example of the configuration of a personal computer that executes the series of processes described above by a program.

  A CPU (Central Processing Unit) 201 executes various processes according to a program stored in a ROM (Read Only Memory) 202 or a storage unit 208. A RAM (Random Access Memory) 203 appropriately stores programs executed by the CPU 201 and data. The CPU 201, ROM 202, and RAM 203 are connected to each other by a bus 204.

  An input / output interface 205 is also connected to the CPU 201 via the bus 204. Connected to the input / output interface 205 are an input unit 206 made up of a keyboard, mouse, microphone, and the like, and an output unit 207 made up of a display, a speaker, and the like. The CPU 201 executes various processes in response to commands input from the input unit 206. Then, the CPU 201 outputs the processing result to the output unit 207.

  A storage unit 208 connected to the input / output interface 205 includes, for example, a hard disk, and stores programs executed by the CPU 201 and various data. The communication unit 209 communicates with an external device via a network such as the Internet or a local area network.

  Further, a program may be acquired via the communication unit 209 and stored in the storage unit 208.

  The drive 210 connected to the input / output interface 205 drives a removable medium 211 such as a magnetic disk, an optical disk, a magneto-optical disk, or a semiconductor memory, and drives the programs and data recorded therein. Get etc. The acquired program and data are transferred to and stored in the storage unit 208 as necessary.

  As shown in FIG. 55, a program recording medium that stores a program that is installed in a computer and is ready to be executed by the computer includes a magnetic disk (including a flexible disk), an optical disk (CD-ROM (Compact Disc-Read Only). Memory, DVD (Digital Versatile Disc), a magneto-optical disk, a removable medium 211 that is a package medium made of a semiconductor memory, a ROM 202 in which a program is temporarily or permanently stored, or a storage unit 208 It is comprised by the hard disk etc. which comprise. The program is stored in the program recording medium using a wired or wireless communication medium such as a local area network, the Internet, or digital satellite broadcasting via a communication unit 209 that is an interface such as a router or a modem as necessary. Done.

  In the present specification, the step of describing the program stored in the program recording medium is not limited to the processing performed in time series in the described order, but is not necessarily performed in time series. Or the process performed separately is also included.

It is a block diagram which shows the structural example of a signal processing apparatus. It is a figure which shows the example of the interpolation of a flame | frame. It is a figure explaining a series of processes performed in a signal processor. It is a figure explaining the evaluation value of a motion vector. It is a flowchart explaining the frame frequency conversion process of a signal processing apparatus. It is a block diagram which shows the structural example of the vector detection part of FIG. It is a block diagram which shows the structural example of the initial vector selection part of FIG. It is a figure which shows the example of a peripheral region. It is a figure which shows the example of the candidate block of an initial vector. It is a block diagram which shows the structural example of the iterative gradient method calculating part of FIG. It is a figure explaining a gradient. It is a figure explaining the iterative gradient method calculation performed using an initial vector. It is a block diagram which shows the structural example which is an effective pixel judgment part of FIG. It is a figure which shows the example of the detection block of a motion vector. It is a figure which shows the example of the motion of the pixel contained in a detection block. It is a figure which shows the example of a calculation block. It is a block diagram which shows the structural example of the gradient method calculating part of FIG. It is a block diagram which shows the structural example of the shift initial vector allocation part of FIG. It is a figure explaining a shift initial vector. It is a figure which shows the example of allocation of a shift initial vector. It is a flowchart explaining the motion vector detection process of a vector detection part. It is a flowchart explaining the initial vector selection process performed in step S22 of FIG. It is a flowchart explaining the iterative gradient method calculation evaluation process performed in step S23 of FIG. It is a flowchart explaining the effective pixel determination process performed in step S53 of FIG. It is a flowchart explaining the gradient method calculation process of the block unit performed in step S55 of FIG. It is a flowchart explaining the iterative gradient method calculation process of the pixel unit performed in step S65 of FIG. It is a flowchart explaining the shift initial vector allocation process performed in step S24 of FIG. It is a block diagram which shows the structural example of the vector allocation part of FIG. It is a figure which shows the example of the pixel of an interpolation frame. It is a figure explaining evaluation of a motion vector. It is a figure which shows the concept of a 4-point interpolation process. It is a flowchart explaining the vector allocation process performed by a vector allocation part. FIG. 33 is a flowchart for describing pixel position calculation processing performed in step S205 of FIG. 32. FIG. It is a flowchart explaining the allocation vector evaluation process performed in step S207 of FIG. It is a block diagram which shows the structural example of the allocation compensation part of FIG. FIG. 36 is a block diagram illustrating a configuration example of a vector compensation unit in FIG. 35. It is a figure explaining the allocation compensation of a motion vector. It is a figure which shows the example of the allocated motion vector. It is a figure which shows the example of the order that an allocation compensation process is advanced. It is a figure which shows the example of allocation of a motion vector. It is another figure which shows the example of allocation of a motion vector. It is a figure which shows the example of the detection of a motion vector, and the motion vector detected by it. It is a figure which shows the example of the compensation of a motion vector, and the motion vector compensated by it. It is a figure shown about convergence and propagation of a motion vector. It is a figure which shows the example which calculates | requires evaluation value DFD of 0 vector. It is a figure which shows the example which calculates | requires evaluation value DFD of motion vector VK1. It is a figure which shows the example which calculates | requires evaluation value DFD of motion vector VK2. It is a flowchart explaining the allocation compensation process of an allocation compensation part. It is a flowchart explaining the vector compensation process performed in step S303 of FIG. It is a flowchart explaining the allocation compensation process of a block unit. It is a block diagram which shows the structural example of the image interpolation part of FIG. It is a flowchart explaining the interpolation process of an image interpolation part. It is a flowchart explaining the allocation compensation process of the motion vector performed using deviation DFD. It is a flowchart explaining the initial vector selection process performed using deviation DFD. And FIG. 16 is a block diagram illustrating a configuration example of a personal computer.

Explanation of symbols

  1 signal processing device, 11 frame memory, 12 vector detection unit, 13 detection vector memory, 14 vector allocation unit, 15 allocation vector memory, 16 allocation flag memory, 17 allocation compensation unit, 18 image interpolation unit

Claims (5)

Focusing on the respective regions included in the first frame in raster order or reverse raster order, detection means for detecting a motion vector by a gradient method;
The detection means pays attention to each area included in the second frame to be generated in the order opposite to the order in which the detection means pays attention to the area, and among the motion vectors assigned to the surrounding areas, An image processing apparatus comprising: an assigning unit that assigns one motion vector. The first calculation target range of the first frame and the first calculation unit associated with the first calculation target range by a motion vector assigned to a region around the region of interest in the second frame. A calculation means for obtaining a correlation with the second calculation target range of the third frame temporally after the frame of
The image processing apparatus according to claim 1, wherein the allocating unit selects one motion vector among motion vectors allocated to a peripheral region based on the correlation obtained by the computing unit. Within the first calculation target range obtained by subtracting the average value of the pixel values of the pixels within the first calculation target range from the pixel value of each pixel within the first calculation target range of the first frame. After the first frame associated with the first calculation target range by the pixel value of each pixel and the motion vector assigned to the area around the area of interest in the second frame In the second calculation target range obtained by subtracting the average value of the pixel values of the pixels in the second calculation target range from the pixel values of the pixels in the second calculation target range of the third frame A calculation means for obtaining a correlation between the first and second calculation target ranges from a pixel value of each pixel;
The image processing apparatus according to claim 1, wherein the allocating unit selects one motion vector among motion vectors allocated to a peripheral region based on the correlation obtained by the computing unit. Focusing on each region included in the first frame in raster order or reverse raster order, detecting a motion vector by the gradient method,
Focus on each area included in the second frame to be generated in the opposite order to the order of focusing on the areas when detecting the motion vector, and the motion assigned to the surrounding areas in the focused area An image processing method including a step of assigning a motion vector of one of vectors. Focusing on each region included in the first frame in raster order or reverse raster order, detecting a motion vector by the gradient method,
Focus on each area included in the second frame to be generated in the opposite order to the order of focusing on the areas when detecting the motion vector, and the motion assigned to the surrounding areas in the focused area A program that causes a computer to execute a process including a step of allocating one motion vector of vectors.

Images