JP4432714B2 - Image processing apparatus and method, recording medium, and program - Google Patents

Description

  The present invention relates to an image processing apparatus and method, a recording medium, and a program. In particular, the accuracy of motion vector allocation to an interpolation frame can be improved by using motion vector variation as an evaluation value. The present invention relates to an image processing apparatus and method, a recording medium, and a program.

  When converting the frame frequency of an image, the image processing device detects a motion vector on the input original frame using a gradient method as shown in Patent Document 1, for example, The pixel value of the interpolation frame is generated by interpolation by assigning the pixel of interest on the interpolation frame to be newly inserted.

  Here, when the detected motion vector is assigned to the target pixel on the interpolation frame on the original frame, the two points on the original frame connected by the motion vector candidates passing through the target pixel on the interpolation frame are centered. In the predetermined area, the sum of absolute differences is obtained, and the obtained sum of absolute differences is compared and evaluated, so that an appropriate motion vector is assigned to the pixel of interest from among motion vector candidates.

JP 60-158786 A

  However, there is a problem that sufficient accuracy cannot be expected only by comparing and evaluating the sum of absolute differences as described above. As a result, there is a problem that a motion vector that is not certain is assigned to the interpolation frame. The reason for this is that each pixel is individually handled, such as the positional relationship in the spatial direction, in order to uniformly handle a plurality of absolute differences obtained within a certain area, and to evaluate a value obtained by integrating the multiple absolute differences. It is conceivable that the information held in is lost.

  In addition, when the spatial position of the pixel value used for the comparative evaluation does not match the pixel position on the frame, conventionally, the pixel value of the nearest pixel at the target spatial position or rounding processing such as rounding off is performed. Since the sum of absolute differences is obtained using the pixel value at the calculated pixel position, the accuracy of the motion vector to be obtained is in units of pixels, and there is a problem that the accuracy of assigning the motion vector is reduced.

  Furthermore, in order to compare and evaluate the difference absolute value sum, a plurality of difference absolute values must be integrated, and there is a problem that it takes a processing time to obtain the difference absolute sum.

  The present invention has been made in view of such a situation, and makes it possible to assign a more probable motion vector to an interpolation frame among the motion vectors detected in the original frame.

The image processing apparatus according to the present invention includes a difference calculation means for calculating a difference between a motion vector detected on the first frame and a surrounding motion vector on the first frame, a motion vector, When the intersection with the second frame matches a pixel on the second frame, the pixel that matches the intersection is selected as a target pixel to which a motion vector is assigned, and the motion vector and the second frame When the intersection point does not coincide with the pixel on the second frame, the target pixel selection unit that selects the four pixels near the intersection as the target pixel, and the difference calculation unit for each target pixel selected by the target pixel selection unit the difference of which is calculated, determines differences of whether less than the calculated difference of the other motion vectors that may be allocated to the target pixel determination And stage, the difference determination unit differences of calculated by is small and the determined target pixel than the difference of which is calculated for the other motion vector, allocating a motion vector, which is calculated by the difference determining means differences And a vector allocating unit that allocates another motion vector to a pixel of interest that has not been determined to be smaller than the difference calculated for the other motion vector .

  Motion vector detecting means for detecting a motion vector on the first frame, and pixel interpolation generated by interpolating a pixel value on the second frame based on the motion vector assigned to the pixel of interest by the vector assigning means And a generating means.

According to an image processing method of the present invention, an image processing method of an image processing apparatus that detects a motion vector on a first frame and generates a pixel value on a second frame based on the detected motion vector. A difference calculating step for calculating a difference between a motion vector detected on the first frame and a surrounding motion vector on the first frame; and the motion vector and the second frame. When the intersection point with the second frame matches the pixel on the second frame, the pixel that matches the intersection point is selected as the target pixel to which the motion vector is assigned, and the intersection point between the motion vector and the second frame is It does not match the pixel on the second frame, the four neighboring pixels of the intersection, and the target pixel selection step of selecting a pixel of interest, by the processing of the target pixel selection step Each pixel of interest is-option, differences of calculated by the processing of the difference of calculation step determines whether less than the calculated difference of the other motion vectors that may be assigned to the target pixel A difference determination step and a difference determination step are performed by assigning a motion vector to a pixel of interest determined to have a difference calculated by the difference determination processing smaller than that calculated for another motion vector. A vector allocation step of allocating another motion vector to the pixel of interest that has not been determined to be less than the difference calculated for the other motion vector .

A program recorded on the recording medium of the present invention detects a motion vector on the first frame, and generates a pixel value on the second frame based on the detected motion vector. A difference calculating step for calculating a difference between a motion vector detected on the first frame and a surrounding motion vector on the first frame; and an intersection of the motion vector and the second frame If the pixel matches the pixel on the second frame, the pixel that matches the intersection is selected as the pixel of interest to which the motion vector is assigned, and the intersection of the motion vector and the second frame is the second frame. It does not match the pixel above selection, the four neighboring pixels of the intersection, and the target pixel selection step of selecting a pixel of interest, by the processing of the target pixel selection step Each pixel of interest that is, differences of calculated by the processing of the difference of calculation step, the difference determines whether less than the difference of which is calculated for the other motion vector is likely to be assigned to the pixel of interest The motion vector is allocated to the pixel of interest determined that the difference calculated by the processing of the sex determination step and the difference determination step is smaller than the difference calculated for the other motion vectors, and the difference determination step A pixel allocation step of assigning another motion vector to a pixel of interest that has not been determined that the difference calculated by the processing is smaller than the difference calculated for another motion vector is performed.

The program of the present invention detects a motion vector on the first frame and generates a pixel value on the second frame on the basis of the detected motion vector. A difference calculating step for calculating a difference between the motion vector detected in step 1 and a surrounding motion vector on the first frame of the motion vector, and an intersection of the motion vector and the second frame is a second frame. If the pixel matches the upper pixel, the pixel that matches the intersection is selected as the target pixel to which the motion vector is assigned, and the intersection of the motion vector and the second frame does not match the pixel on the second frame when the four neighboring pixels of the intersection, and the target pixel selection step of selecting a pixel of interest, each pixel of interest selected by the process of the pixel of interest selection step, Differences of calculated by the processing of the opposite sex calculating step, the difference determination step of determining whether the smaller or not than the calculated difference of the other motion vectors that may be assigned to the target pixel, the difference determination The motion vector is assigned to the pixel of interest determined that the difference calculated by the step processing is smaller than the difference calculated for other motion vectors, and the difference calculated by the processing of the difference determination step is The target pixel that has not been determined to be smaller than the difference calculated for the other motion vector is subjected to processing including a vector allocation step for assigning the other motion vector .

In the present invention, the difference between the motion vector detected on the first frame and the surrounding motion vector on the first frame of the motion vector is calculated, and the intersection of the motion vector and the second frame is If the pixel matches the pixel on the second frame, the pixel matching the intersection is selected as the target pixel to which the motion vector is assigned, and the intersection of the motion vector and the second frame is If it does not coincide with the pixel, the four pixels near the intersection are selected as the target pixel. Then, for each selected pixel of interest, it is determined whether or not the calculated difference is smaller than the difference calculated for other motion vectors that may be assigned to the pixel of interest. When the motion vector is assigned to the target pixel determined to be smaller than the difference calculated for the other motion vectors, and the calculated difference is smaller than the difference calculated for the other motion vectors. Another motion vector is assigned to the pixel of interest that has not been determined .

  According to the present invention, a more probable motion vector can be assigned to a generated frame. That is, it is possible to further improve the accuracy of motion vector assignment to the generated frame. Further, according to the present invention, the processing time is shortened and the circuit scale is also reduced.

  Embodiments of the present invention will be described below. Correspondences between constituent elements described in the claims and specific examples in the embodiments of the present invention are exemplified as follows. This description is to confirm that specific examples supporting the invention described in the claims are described in the embodiments of the invention. Accordingly, although there are specific examples that are described in the embodiment of the invention but are not described here as corresponding to the configuration requirements, the specific examples are not included in the configuration. It does not mean that it does not correspond to a requirement. On the contrary, even if a specific example is described here as corresponding to a configuration requirement, this means that the specific example does not correspond to a configuration requirement other than the configuration requirement. not.

  Further, this description does not mean that all the inventions corresponding to the specific examples described in the embodiments of the invention are described in the claims. In other words, this description is an invention corresponding to the specific example described in the embodiment of the invention, and the existence of an invention not described in the claims of this application, that is, in the future, a divisional application will be made. Nor does it deny the existence of an invention added by amendment.

The image processing apparatus according to claim 1 is a difference calculation unit that calculates a difference between a motion vector detected on the first frame and a surrounding motion vector on the first frame of the motion vector (for example, 47, when the intersection of the motion vector and the second frame matches the pixel on the second frame, the pixel that matches the intersection is a target to which the motion vector is assigned. When the pixel of interest is selected, and the intersection of the motion vector and the second frame does not coincide with the pixel on the second frame, pixel-of-interest selection means for selecting four pixels near the intersection as the pixel of interest (for example, FIG. 47 pixel information operation unit 751) of possible to have been focused each pixel selected by the pixel of interest selection means, the difference of which is calculated by the difference of calculation means, assigned to the target pixel The difference calculated by the difference determination means (for example, the vector evaluation unit 753 in FIG. 47) for determining whether or not the difference is smaller than the difference calculated for another motion vector, A motion vector is allocated to the target pixel determined to be smaller than the difference calculated for the other motion vector, and the difference calculated by the difference determination unit is smaller than the difference calculated for the other motion vector. A vector allocating unit (for example, the vector selection unit 754 in FIG. 47) that allocates another motion vector to the pixel of interest that has not been determined .

  The image processing apparatus according to claim 2, wherein a motion vector detection unit (for example, the vector detection unit 52 in FIG. 2) that detects a motion vector on the first frame and a motion allocated to the pixel of interest by the vector allocation unit. A pixel interpolation generation unit (for example, the image interpolation unit 58 in FIG. 2) that generates by interpolating pixel values on the second frame based on the vector can be further provided.

The image processing method according to claim 3 is a difference calculation step (for example, calculating a difference between a motion vector detected on the first frame and a surrounding motion vector on the first frame of the motion vector (for example, 52, and when the intersection of the motion vector and the second frame matches the pixel on the second frame, the pixel that matches the intersection is set as the target pixel to which the motion vector is assigned. If the intersection point between the motion vector and the second frame is not coincident with the pixel on the second frame, a pixel-of-interest selection step (for example, the step of FIG. 52) selects four pixels near the intersection as the pixel of interest. and S782), for each pixel of interest selected by the process of the pixel of interest selection step, the difference of which is calculated by the process of the difference of calculation step, focused image Another motion vector difference determination step of determining whether the smaller or not than the differences of calculated for that may be assigned to the (e.g., step S785 in FIG. 52) and is calculated by the processing of the difference determination step The motion vector is assigned to the pixel of interest determined to be smaller than the difference calculated for the other motion vectors, and the difference calculated by the processing of the difference determination step is determined for the other motion vectors. the calculated target pixel is not determined to be smaller than the difference of, and a vector allocation step of allocating another motion vector (e.g., step S787 in FIG. 52).

  Since the recording medium according to claim 4 and the program according to claim 5 are basically the same processing as the image processing method according to claim 3 described above, the description thereof is omitted because it is repeated. To do.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a configuration example of a signal processing apparatus 1 to which the present invention is applied. The signal processing device 1 is composed of, for example, a personal computer. In FIG. 1, a CPU (Central Processing Unit) 11 executes various processes according to a program stored in a ROM (Read Only Memory) 12 or a storage unit 18. A RAM (Random Access Memory) 13 appropriately stores programs executed by the CPU 11 and data. The CPU 11, ROM 12, and RAM 13 are connected to each other by a bus 14.

  An input / output interface 15 is also connected to the CPU 11 via the bus 14. The input / output interface 15 is connected to an input unit 16 including a keyboard, a mouse, a microphone, and the like, and an output unit 17 including a display and a speaker. The CPU 11 executes various processes in response to commands input from the input unit 16. Then, the CPU 11 outputs an image, sound, or the like obtained as a result of the processing to the output unit 17.

  The storage unit 18 connected to the input / output interface 15 is composed of, for example, a hard disk, and stores programs executed by the CPU 11 and various data. The communication unit 19 communicates with an external device via the Internet or other networks. A program may be acquired via the communication unit 19 and stored in the storage unit 18.

  The drive 20 connected to the input / output interface 15 drives the magnetic disk 31, the optical disk 32, the magneto-optical disk 33, or the semiconductor memory 34 when they are mounted, and programs and data recorded there. Get etc. The acquired program and data are transferred to and stored in the storage unit 18 as necessary.

  Note that the signal processing device 1 can be, for example, a television receiver, an optical disc player, or a signal processing unit thereof.

  FIG. 2 is a block diagram showing the signal processing apparatus 1.

  It does not matter whether each function of the signal processing device 1 is realized by hardware or software. That is, each block diagram in this specification may be considered as a hardware block diagram or a software functional block diagram.

  In the signal processing device 1 having the configuration shown in FIG. 2, for example, an image of a progressive image signal (hereinafter referred to as a 24P signal) with a frame frequency of 24 Hz is input, and the input image (input image) has a frame frequency of 60 Hz. It is converted into an image of a progressive image signal (hereinafter referred to as 60P signal) and output. That is, FIG. 2 is a diagram illustrating a configuration of a signal processing device that is an image processing device.

  The input image of the 24P signal input to the signal processing device 1 is supplied to the frame memory 51, the vector detection unit 52, the vector allocation unit 54, the allocation compensation unit 57, and the image interpolation unit 58. The frame memory 51 stores input images in units of frames. The frame memory 51 stores a frame at time t one before the input image at time t + 1. The frame at time t stored in the frame memory 51 is supplied to the vector detection unit 52, the vector allocation unit 54, the allocation compensation unit 57, and the image interpolation unit 58. Hereinafter, the frame at time t on the frame memory 51 is referred to as frame t, and the frame of the input image at time t + 1 is referred to as frame t + 1.

  The vector detection unit 52 detects a motion vector between the target block of the frame t on the frame memory 51 and the target block of the frame t + 1 of the input image, and stores the detected motion vector in the detection vector memory 53. As a method for detecting the motion vector between the two frames, a gradient method or a block matching method is used. Details of the configuration of the vector detection unit 52 will be described later. The detection vector memory 53 stores the motion vector detected by the vector detection unit 52 in the frame t.

  The vector allocation unit 54 distinguishes the motion vector obtained on the frame t of the 24P signal from the frame of the 60P signal to be interpolated in the allocation vector memory 55 (hereinafter, the frame of the 60P signal is distinguished from the frame of the 24P signal). , Which is also referred to as an interpolation frame), and the allocation flag of the allocation flag memory 56 of the pixel to which the motion vector is allocated is rewritten to 1 (True). Details of the configuration of the vector allocation unit 54 will be described later.

  The allocation vector memory 55 stores the motion vector allocated by the vector allocation unit 54 in association with each pixel of the interpolation frame. The allocation flag memory 56 stores an allocation flag indicating the presence / absence of a motion vector to be allocated for each pixel of the interpolation frame. For example, an assignment flag of True (1) indicates that a motion vector is assigned to the corresponding pixel, and an assignment flag of False (0) indicates that a motion vector is not assigned to the corresponding pixel. Show.

  The allocation compensation unit 57 refers to the allocation flag of the allocation flag memory 56, and supplements the motion vector of the peripheral pixel of the target pixel with respect to the target pixel for which the motion vector has not been allocated by the vector allocation unit 54. Allocate on 55 interpolation frames. At this time, the allocation compensator 57 rewrites the allocation flag of the target pixel to which the motion vector is allocated to 1 (True). Details of the configuration of the allocation compensator 57 will be described later.

  The image interpolation unit 58 interpolates and generates the pixel value of the interpolation frame using the motion vector allocated to the interpolation frame in the allocation vector memory 55 and the pixel values of the frame t and the next frame t + 1. Then, the image interpolation unit 58 outputs the generated interpolated frame, and then outputs the frame t + 1 as necessary, thereby outputting the 60P signal image to a subsequent stage (not shown). In the following, the pixel value is also referred to as a luminance value as appropriate.

  FIG. 3 is a diagram for explaining the principle of processing in the signal processing apparatus 1 according to the present invention. In the example of FIG. 3, a dotted line represents a frame of a 24P signal that is input to the signal processing device 1 at times t, t + 1, and t + 2, and a solid line is represented by the signal processing device 1 from the input 24P signal. , The 60P signal interpolation frames at the generated times t, t + 0.4, t + 0.8, t + 1.2, t + 1.6, and t + 2.

  Generally, in order to convert a 24P signal into a 60P signal, 5/2 times as many frames are required. That is, five 60P signal images must be generated from two 24P signal images. At this time, the generated interpolated frame of the 60P signal has a time phase of 0.0, 0.4, 0.8, 1.2, and 1 on the 24P signal in order to equalize the frame interval. 6 is arranged at a position. Among these, 4 frames (frames of t + 0.4, t + 0.8, t + 1.2, and t + 1.6) except for one frame at time t with a time phase of 0.0 do not exist on the 24P signal. It is an image. Therefore, when the image of the 24P signal is input, the signal processing device 1 generates four interpolation frames from the two frames at the time t and the time t + 1 of the 24P signal. Therefore, the signal processing apparatus 1 outputs a 60P signal image composed of five frames at times t, t + 0.4, t + 0.8, t + 1.2, and t + 1.6.

  As described above, the signal processing apparatus 1 executes the process of converting the frame frequency from the 24P signal image to the 60P signal image.

  In principle, as described above, from the two frames at time t and time t + 1 of the 24P signal, five frames at times t, t + 0.4, t + 0.8, t + 1.2, and t + 1.6 are obtained. A frame of 60P signal is newly generated. Actually, in the example of FIG. 3, 60P of t, t + 0.4, and t + 0.8 is based on two frames at time t and time t + 1 of the 24P signal. A frame of the signal is generated, and a frame of the 60P signal of t + 1.2, t + 1.6, and t + 2 is generated based on the two frames at the times t + 1 and t + 2 of the 24P signal.

  FIG. 4 is a diagram for more specifically explaining the processing of the present invention. In the example of FIG. 4, a thick arrow represents a transition to each state, and an arrow T represents a time passage direction in the states 81 to 85. The states 81 to 85 are the frame t at the time t of the 24P signal, the frame t + 1 at the time t + 1 next to the time t, or the frame t and the frame t + 1 at the time of input / output to each unit constituting the signal processing device 1. The state of the interpolation frame F of the 60P signal produced | generated during is shown notionally. That is, actually, for example, a frame in which a motion vector is detected as shown in the state 82 is not input, and the frame and the motion vector are input separately.

  The state 81 represents the state of the frame t and the frame t + 1 of the 24P signal input to the vector detection unit 52. A black dot on the frame t in the state 81 represents a pixel on the frame t. The vector detection unit 52 detects the position where the pixel on the frame t in the state 81 moves in the frame t + 1 of the next time, and the movement is indicated as shown on the frame t in the state 82. It outputs as a motion vector corresponding to each pixel. As a method for detecting the motion vector between the two frames, a block matching method or a gradient method is used. At this time, when a plurality of motion vectors are detected in the pixel, the vector detection unit 52 obtains an evaluation value to be described later for each motion vector, and selects a motion vector based on the evaluation value.

  A state 82 represents the state of the frame t and the frame t + 1 input to the vector allocation unit 54. In the state 82, the arrow of each pixel of the frame t represents the motion vector detected by the vector detection unit 52.

  The vector allocating unit 54 extends the motion vector detected for each pixel of the frame t in the state 82 to the next frame t + 1 and sets it to a preset time phase (for example, t + 0.4 in FIG. 3). A position on a certain interpolation frame F is obtained. This is because, assuming that there is a constant motion between the frame t and the frame t + 1, the point where the motion vector passes through the interpolation frame F is the pixel position in that frame. Therefore, the vector allocating unit 54 allocates the passing motion vector to four neighboring pixels on the interpolation frame F in the state 83. At this time, depending on the pixel of the interpolation frame, there may be a case where no motion vector exists, or a plurality of motion vectors may be candidates for allocation. In the latter case, like the vector detection unit 52, the vector allocation unit 54 obtains an evaluation value for each motion vector and selects a motion vector to be allocated based on the evaluation value.

  A state 83 represents the state of the frame t and the frame t + 1 and the interpolation frame F to which the motion vector is assigned, which is input to the assignment compensation unit 57. In the interpolation frame F in the state 83, the pixels to which the motion vector is assigned by the vector assigning unit 54 and the pixels to which the motion vector is not assigned are shown.

  The allocation compensation unit 57 compensates for a pixel to which a motion vector in the state 83 is not allocated by using a motion vector allocated to a peripheral pixel of the pixel. This is because the motion vector of the peripheral pixel of the pixel of interest and the motion vector of the pixel of interest are similar if the assumption that the neighboring region of the pixel of interest has the same motion holds. As a result, a motion vector that is accurate to some extent is also given to pixels for which no motion vector has been assigned, and a motion vector is assigned to all pixels on the interpolation frame F in the state 84. In this case as well, since motion vectors of a plurality of surrounding pixels exist as candidates, the allocation compensator 57 obtains an evaluation value for each motion vector in the same manner as the vector allocation unit 54, and based on the evaluation values Select the motion vector to be assigned.

  A state 84 represents the state of the frame t and the frame t + 1 input to the image interpolation unit 58 and the state of the interpolation frame F in which motion vectors are assigned to all the pixels. Based on the motion vectors assigned to all these pixels, the image interpolation unit 58 can determine the positional relationship between the pixels on the interpolation frame F and the two frames t and t + 1. Therefore, the image interpolation unit 58 uses the motion vector allocated on the interpolation frame F and the pixel values of the frame t and the frame t + 1 to interpolate as indicated by the black dot of the interpolation frame F in the state 85. Pixel values on the frame F are generated by interpolation. Then, the image interpolation unit 58 outputs the generated interpolated frame, and then outputs the frame t + 1 as necessary, thereby outputting the 60P signal image to a subsequent stage (not shown).

  Next, with reference to FIG. 5, the evaluation value of the motion vector used in the signal processing apparatus 1 according to the present invention will be described. As described above with reference to FIG. 4, in each unit (the vector detection unit 52, the vector allocation unit 54, and the allocation compensation unit 57) of the signal processing device 1, an optimal motion vector is selected for subsequent processing. At this time, in each part of the signal processing device 1, as an evaluation value for the motion vector, a sum of absolute differences (DFD (Displaced Frame Difference)) representing a correlation value between the blocks shifted by the vector amount of interest of the two frames is used. Used.

  In the example of FIG. 5, the m × n block centered on the pixel position p on the frame t at the time t and the vector amount of the motion vector v of interest are shifted from the pixel position p on the frame t + 1 at the time t + 1. Two blocks of m × n blocks centered on the pixel position p + v are shown. The sum of absolute differences DFDt (p) obtained between these two blocks is expressed by the following equation (1).

・・・（１）

... (1)

  Here, Ft (p) represents the luminance value at the pixel position p at time t, and m × n represents the DFD calculation range (block) for obtaining the sum of absolute differences. Since this difference absolute value sum represents a correlation value between DFD calculation ranges (blocks) in two frames, generally, the smaller the difference absolute value sum, the more the waveform of the block between the frames matches. Thus, it is determined that the reliability of the motion vector v is higher as the difference absolute value sum is smaller. As a result, this sum of absolute differences is used when the most probable motion vector is selected from a plurality of candidates.

  Therefore, hereinafter, in each unit (vector detection unit 52, vector allocation unit 54, and allocation compensation unit 57) of signal processing device 1, as an evaluation value when a motion vector is selected, a difference is obtained unless otherwise specified. An absolute value sum (hereinafter referred to as an evaluation value DFD) is used.

  Next, the process of converting the frame frequency of the signal processing apparatus 1 will be described with reference to the flowchart of FIG.

  In step S1, the vector detection unit 52 inputs the pixel value of the frame t + 1 of the input image at the time t + 1 and the pixel t of the frame t at the time t immediately before the input image of the frame memory 51, and proceeds to step S2. At this time, the vector allocating unit 54, the allocation compensating unit 57, and the image interpolating unit 58 determine the pixel value of the frame t + 1 of the input image at the time t + 1 and the frame t at the time t immediately before the input image of the frame memory 51. Enter.

  In step S2, the vector detection unit 52 executes a motion vector detection process and proceeds to step S3. That is, the vector detection unit 52 detects a motion vector between the target block of the frame t on the frame memory 51 and the target block of the next frame t + 1 that is the input image, and the detected motion vector is detected by the detection vector memory 53. And proceed to step S3. As a method for detecting the motion vector between the two frames, a gradient method or a block matching method is used. When there are a plurality of motion vector candidates, an evaluation value DFD is obtained for each motion vector, and a highly reliable motion vector based on the obtained evaluation value DFD is detected. That is, in this case, the most probable motion vector is selected and detected in the target block for detecting the motion vector. Details of the motion vector detection process in step S2 will be described later.

  In step S3, the vector allocation unit 54 executes vector allocation processing, and proceeds to step S4. That is, in step S3, the vector allocation unit 54 allocates the motion vector obtained on the frame t to the pixel of interest on the interpolation frame to be interpolated in the allocation vector memory 55, and the pixel to which the motion vector is allocated. The allocation flag in the allocation flag memory 56 is rewritten to 1 (True). For example, an allocation flag that is True indicates that a motion vector is allocated to the corresponding pixel, and an allocation flag that is False indicates that a motion vector is not allocated to the corresponding pixel. When there are a plurality of motion vector candidates in each pixel, an evaluation value DFD is obtained for each motion vector, and a highly reliable motion vector based on the obtained evaluation value DFD is assigned. That is, in this case, the most probable motion vector is selected and assigned in the target pixel to which the motion vector is assigned. Details of the vector allocation processing in step S3 will be described later.

  In step S4, the allocation compensator 57 performs an allocation compensation process, and proceeds to step S5. That is, the allocation compensation unit 57 refers to the allocation flag in the allocation flag memory 56 in step S4, and for the pixel of interest for which no motion vector has been allocated by the vector allocation unit 54, the motion vector of the surrounding pixels of the pixel of interest. Is allocated on the interpolation frame of the allocation vector memory 55. At this time, the allocation compensator 57 compensates the motion vector and rewrites the allocation flag of the allocated pixel of interest to 1 (True). When there are a plurality of motion vectors of peripheral pixels, an evaluation value DFD is obtained for each motion vector, and a highly reliable motion vector is assigned based on the obtained evaluation value DFD. That is, in this case, the most probable motion vector is selected and assigned in the target pixel to which the motion vector is assigned. Details of the allocation compensation processing in step S4 will be described later.

  In step S5, the image interpolation unit 58 executes an image interpolation process. That is, in step S5, the image interpolation unit 58 uses the motion vector allocated to the interpolation frame of the allocation vector memory 55 and the pixel values of the frame t and the frame t + 1 to generate an interpolation frame pixel value. The process proceeds to step S6. Details of the image interpolation processing in step S5 will be described later. In step S6, the image interpolation unit 58 outputs the generated interpolation frame, and then outputs the frame t + 1 as necessary, thereby outputting the 60P signal image to a subsequent stage (not shown). The process proceeds to step S7.

  In step S7, the vector detection unit 52 determines whether or not all the frames have been processed. If the vector detection unit 52 determines that all the frames have not been processed, the process returns to step S1 and the subsequent processing is performed. repeat. On the other hand, when the vector detection unit 52 determines in step S7 that all the frames have been processed, the vector detection unit 52 ends the process of converting the frame frequency.

  As described above, the signal processing apparatus 1 according to the present invention detects a motion vector from a frame of a 24P signal input image, assigns the detected motion vector to a pixel on a frame of a 60P signal, and assigns the assigned motion vector. Based on the above, the pixel value on the frame of the 60P signal is generated. At this time, the signal processing apparatus 1 selects a motion vector with higher reliability based on the evaluation value DFD (sum of absolute differences) in each process, and outputs it to the subsequent stage. Therefore, in the signal processing device 1, it is possible to suppress a failure of the movement and the like and generate a more accurate image.

  Next, details of the configuration of the image interpolation unit 58 will be described.

  FIG. 7 is a block diagram illustrating a configuration of the image interpolation unit 58. The image interpolation unit 58 having the configuration shown in FIG. 7 interpolates and generates the pixel value of the interpolation frame using the motion vector allocated to the interpolation frame in the allocation vector memory 55 and the pixel values of the frame t and the frame t + 1. , 60P signal image processing is performed.

  In the example of FIG. 7, the frame t of the image at time t is input to the spatial filter 92-1, and the frame t + 1 of the image at time t + 1 is input to the spatial filter 92-2 and the buffer 95.

  The interpolation control unit 91 selects the pixel of the interpolation frame in the allocation vector memory 55, and based on the motion vector assigned to the selected pixel, the pixel on the interpolation frame, the two frames t and t + 1 The positional relationship (spatial shift amount) with each pixel is obtained. That is, the interpolation control unit 91 uses, based on the pixels of the interpolated frame, the position on the frame t associated with the motion vector and the position of the pixel on the frame t corresponding to the pixel of the interpolated frame. The spatial shift amount is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-1. Similarly, the interpolation control unit 91 uses, based on the pixels of the interpolation frame, the position on the frame t + 1 associated with the motion vector and the position of the pixel on the frame t + 1 corresponding to the pixel of the interpolation frame. Is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-2.

  Further, the interpolation control unit 91 obtains an interpolation weight between the frame t and the frame t + 1 based on a preset time phase (time) of the interpolation frame, and uses the obtained interpolation weight as a multiplier 93-1. And 93-2. For example, when the time of the interpolation frame is a time that is “k” away from the time t + 1 of the frame t + 1 and a time that is “1−k” away from the time t of the frame t (ie, the time of the interpolation frame is the time t and time t + 1 is “1−k”: generated at a time that is internally divided into “k”), the interpolation control unit 91 sets an interpolation weight of “1-k” in the multiplier 93-1. An interpolation weight of “k” is set in the multiplier 93-2.

  Spatial filters 92-1 and 92-2 are constituted by cubic filters, for example. The spatial filter 92-1 is a pixel value on the frame t corresponding to the pixel of the interpolation frame based on the input pixel value of the pixel on the frame t and the spatial shift amount supplied from the interpolation control unit 91. And the obtained pixel value is output to the multiplier 93-1. The spatial filter 92-2, based on the input pixel value of the pixel on the frame t + 1 and the spatial shift amount supplied from the interpolation control unit 91, the pixel value on the frame t + 1 corresponding to the pixel of the interpolation frame And the obtained pixel value is output to the multiplier 93-2.

  When the position of the pixel in the interpolation frame does not match the position of the pixel on frame t or frame t + 1 (that is, when the position of the pixel in the interpolation frame is a component equal to or less than the pixel in frame t or frame t + 1). The spatial filters 92-1 and 92-2 obtain the sum of the inverse ratios of the distances of the four surrounding pixels by using the pixel values of the four surrounding pixels at the position of the pixel of the interpolation frame in the frame t or the frame t + 1. Then, the pixel value on the frame corresponding to the pixel of the interpolation frame is obtained. That is, the pixel value at the position below the pixel is obtained by linear interpolation based on the distance to the surrounding four pixels (details will be described later).

  The multiplier 93-1 multiplies the pixel value on the frame t input from the spatial filter 92-1 by the interpolation weight “1-k” set by the interpolation control unit 91, and the weighted pixel value is The result is output to the adder 94. The multiplier 93-2 multiplies the pixel value on the frame t + 1 input from the spatial filter 92-2 by the interpolation weight “k” set by the interpolation control unit 91, and adds the weighted pixel value to the adder. Output to 94.

  The adder 94 generates the pixel value of the pixel of the interpolation frame by adding the pixel value input from the multiplier 93-1 and the pixel value input from the multiplier 93-2. The pixel value of the interpolation frame is output to the buffer 95. The buffer 95 buffers the input frame t + 1. The buffer 95 outputs the generated interpolation frame, and then outputs a buffered frame t + 1 as necessary based on a preset time phase (time) of the 60P frame. Thus, an image of the 60P signal is output to a subsequent stage (not shown).

  Details of the image interpolation processing of the image interpolation unit 58 configured as described above will be described with reference to the flowchart of FIG.

  In step S51, the interpolation control unit 91 calculates interpolation weights (for example, “k” and “1-k”) of the interpolation frame between the frame t and the frame t + 1 based on the time phase of the interpolation frame to be processed. The obtained interpolation weights are set in the multipliers 93-1 and 93-2, respectively, and the process proceeds to step S52. In step S52, the interpolation control unit 91 selects a pixel of the interpolation frame in the allocation vector memory 55, and proceeds to step S53. Note that the pixels on the interpolation frame are selected in the raster scan order from the upper left pixel of the frame.

  In step S53, based on the motion vector assigned to the selected pixel, the interpolation control unit 91 determines the positional relationship between the pixels on the interpolation frame and the two frames t and t + 1 (space shift amount). ) And the obtained spatial shift amounts are supplied to the spatial filters 92-1 and 92-2, respectively, and the process proceeds to step S54. Specifically, in step S53, the interpolation control unit 91 uses the pixel of the interpolation frame as a reference, the position on the frame t associated with the motion vector, and the frame t corresponding to the pixel of the interpolation frame. The spatial shift amounts are obtained from the pixel positions, and the obtained spatial shift amounts are supplied to the spatial filter 92-1. Similarly, the interpolation control unit 91 uses, based on the pixels of the interpolation frame, the position on the frame t + 1 associated with the motion vector and the position of the pixel on the frame t + 1 corresponding to the pixel of the interpolation frame. Is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-2.

  The pixel value of the frame t of the image at time t is input to the spatial filter 92-1, and the pixel value of the frame t + 1 of the image at time t + 1 is input to the spatial filter 92-2. In step S54, the spatial filters 92-1 and 92-2 determine the pixels of the interpolation frame based on the pixel values of the pixels on the input frames t and t + 1 and the spatial shift amount supplied from the interpolation control unit 91. The pixel values on each frame corresponding to are obtained, the obtained pixel values are output to the multipliers 93-1 and 93-2, and the process proceeds to step S55.

  In Step S55, the multipliers 93-1 and 93-2 weight the interpolation weight set by the interpolation control unit 91 to the pixel value on each frame input from the spatial filter 92-1 or 92-2, The weighted pixel value is output to the adder 94, and the process proceeds to step S56. That is, the multiplier 93-1 multiplies the pixel value on the frame t input from the spatial filter 92-1 by the interpolation weight “1-k” set by the interpolation control unit 91, and weighted pixel values. Is output to the adder 94. The multiplier 93-2 multiplies the pixel value on the frame t + 1 input from the spatial filter 92-2 by the interpolation weight “k” set by the interpolation control unit 91, and adds the weighted pixel value to the adder. Output to 94.

  In step S56, the adder 94 adds the pixel value weighted by the multiplier 93-1 and the pixel value weighted by the multiplier 93-2 to generate the pixel value of the pixel of the interpolation frame. The generated pixel value is output to the buffer 95, and the process proceeds to step S57. In step S57, the interpolation control unit 91 determines whether or not the processing for all the pixels on the interpolation frame is completed, and determines that the processing for all the pixels on the interpolation frame is not completed. If so, the process returns to step S52, and the subsequent processing is repeated. If the interpolation control unit 91 determines in step S57 that the processing for all the pixels on the interpolation frame has been completed, the interpolation control unit 91 ends the image interpolation processing.

  As described above, the pixel value of the interpolation frame is generated based on the motion vector assigned to the interpolation frame, and the interpolation frame is output by the buffer 95 in step S6 of FIG. If necessary, the frame t + 1 is output, so that an image of the 60P signal is output to the subsequent stage. Therefore, since the most probable motion vector is assigned to the pixel of the interpolation frame, an accurate interpolation frame can be generated.

  Next, details of the configuration of the vector detection unit 52 will be described.

  FIG. 9 is a block diagram showing the configuration of the vector detection unit 52. The vector detection unit 52 having the configuration shown in FIG. 9 detects a motion vector on the frame t by using the input frame t of the image at time t and the frame t + 1 of the image at time t + 1. The process of detecting the motion vector is executed for each predetermined block composed of a plurality of pixels.

  The initial vector selection unit 101 uses, as an initial vector V0 that is an initial value used in the gradient method, a high-reliability motion vector obtained from the detection result of the past motion vector for each predetermined block, as an iterative gradient method calculation unit. To 103. Specifically, the initial vector selection unit 101 uses a motion vector of a peripheral block obtained in the past stored in the detection vector memory 53 or a shift initial vector stored in the shift initial vector memory 107 as an initial vector. Select as a candidate vector. Then, the initial vector selection unit 101 obtains the evaluation value DFD of the candidate vector using the frame t and the frame t + 1, and selects the candidate vector having the highest reliability based on the obtained evaluation value DFD. And output as an initial vector V0. Details of the configuration of the initial vector selection unit 101 will be described later.

  The pre-filters 102-1 and 102-2 are configured by a low-pass filter and a Gaussian filter, respectively, and remove noise components of the frame t and the frame t + 1 of the input image, and output them to the iterative gradient method computing unit 103.

  The iterative gradient method computing unit 103 uses the initial vector V0 input from the initial vector selection unit 101 and the frame t and the frame t + 1 input via the prefilters 102-1 and 102-2, for each predetermined block. Then, the motion vector Vn is calculated by the gradient method. The iterative gradient method computing unit 103 outputs the initial vector V0 and the calculated motion vector Vn to the vector evaluation unit 104. In addition, the iterative gradient method computing unit 103 repeatedly performs the gradient method computation based on the motion vector evaluation result by the vector evaluation unit 104 to calculate the motion vector Vn. Details of the configuration of the iterative gradient method computing unit 103 will be described later.

  The vector evaluation unit 104 obtains the motion vector Vn−1 (or initial vector V0) from the iterative gradient method computing unit 103 and the evaluation value DFD of the motion vector Vn, and based on the obtained evaluation value DFD, the iterative gradient method The calculation unit 103 is controlled to repeatedly execute the calculation of the gradient method. Finally, a highly reliable one based on the evaluation value DFD is selected and stored in the detection vector memory 53 as a motion vector V. At this time, the vector evaluation unit 104 supplies the evaluation value DFD obtained for the motion vector V together with the motion vector V to the shifted initial vector allocation unit 105.

  When the motion vector V and the evaluation value DFD are supplied from the vector evaluation unit 104, the shifted initial vector allocation unit 105 shifts the motion vector passing through the target block on the next frame to the target block. Set as initial vector. In other words, the shifted initial vector allocation unit 105 shifts a motion vector of the same size and the same direction as the motion vector V, starting from the target block on the next frame at the same position as the end block of the motion vector V. Set as initial vector. Then, the shifted initial vector allocation unit 105 allocates the set shifted initial vector to the shifted initial vector memory 107 in association with the block of interest.

  Specifically, the shifted initial vector allocation unit 105 stores the evaluation value DFD of the motion vector V allocated as the shifted initial vector in the evaluation value memory 106 in association with the target block, and stores the same target block. It is compared with the evaluation value DFD of another motion vector V that passes (that is, the block of the past frame at the same position as the target block). Then, the shifted initial vector allocation unit 105 shifts the motion vector V determined to have high reliability based on the evaluation value DFD to the target block, and allocates it to the shifted initial vector memory 107 as the shifted initial vector of the target block. Details of the configuration of the shifted initial vector allocation unit 105 will be described later.

Next, the principle of the gradient method used in the vector detection unit 52 will be described. First, a luminance value of a pixel represented by coordinates (x, y, t) using a horizontal, vertical, and time axis in a moving image is assumed to be g (x, y, t). Here, when the pixel of interest (x ₀ , y ₀ , t ₀ ) is displaced by (dx, dy, dt) during a minute time, the horizontal (vertical) and time axis gradients (difference differences) are respectively expressed as gx. When expressed as (x ₀ , y ₀ , t ₀ ), gy (x ₀ , y ₀ , t ₀ ), and gt (x ₀ , y ₀ , t ₀ ), the luminance value of the pixel after displacement is approximated by Taylor expansion. Is expressed by the following equation (2).

g (x ₀ + dx, y ₀ + dy, t ₀ + dt)
≒ g (x ₀ , y ₀ , t ₀ ) + gx (x ₀ , y ₀ , t ₀ ) dx
+ Gy (x ₀ , y ₀ , t ₀ ) dy + gt (x ₀ , y ₀ , t ₀ ) dt
... (2)

  Here, when a pixel of interest in a moving image moves by horizontal vx and vertical vy after one frame (hereinafter, expressed as (vx, vy)), the luminance value of the pixel is expressed by the following equation (3). Is done.

_{g (x 0 + vx, y} 0 + vy, t 0 +1) = g (x 0, y 0, t 0)
... (3)

  When Expression (2) is substituted into Expression (3), it is expressed by the following Expression (4).

gx (x ₀ , y ₀ , t ₀ ) vx + gy (x ₀ , y ₀ , t ₀ ) vy
+ Gt (x ₀ , y ₀ , t ₀ ) = 0
... (4)

  Since the equation (4) is a two-variable equation of vx and vy, the solution cannot be obtained with a single equation for the target pixel. Therefore, as described below, a block that is a peripheral region of the target pixel is considered as one processing unit, and it is assumed that all pixels in the block (peripheral region) have the same movement (vx, vy). Build a similar formula. Assuming an assumption, an expression for the number of neighboring pixels is obtained for two variables. Therefore, these equations are combined to obtain (vx, vy) that minimizes the sum of squares of motion compensation frame differences of all pixels in the block.

  When the pixel (x, y, t) moves by (vx, vy) by one frame, the motion compensation inter-frame difference d is expressed by the following equation (5).

d = g (x + vx, y + vy, t + 1) −g (x, y, t)
= Δxvx + Δyvy + Δt
... (5)

  In equation (5), Δx = gx (x, y, t), which represents a horizontal gradient, Δy = gy (x, y, t), which represents a vertical gradient, Δt = gt ( x, y, t), which represents the gradient in the time direction. Using these, assuming that the square sum of the difference between motion compensation frames is E, it is expressed by Equation (6).

E = Σd ²
= Σ (Δx ² vx ² + Δy ² vy ² + 2ΔxΔyvxvy
+ 2ΔxΔtvx + 2ΔyΔtvy + Δt ² )
= Vx ² ΣΔx ² + vy ² ΣΔy ² +2 vxvy ΣΔxΔy
+ 2vxΣΔxΔt + 2vyΣΔyΔt + ΣΔt ²
... (6)

  Here, E becomes the minimum (vx, vy) when the partial differential value in each variable becomes 0, that is, when the condition of ΔE / Δvx = ΔE / Δvy = 0 is satisfied. Thus, the following equations (7) and (8) are obtained.

vxΣΔx ² + vyΣΔxΔy + ΣΔxΔt = 0
... (7)
vyΣΔy ² + vxΣΔxΔy + ΣΔyΔt = 0
... (8)

  From these equations (7) and (8), (vx, vy) which is the desired motion can be obtained by calculating the following equation (9).

・・・（９）

... (9)

  Here, it demonstrates concretely with reference to FIG. In the example of FIG. 10, an arrow X indicates the horizontal direction, and an arrow Y indicates the vertical direction. In addition, an arrow T indicates the direction of time passage from the frame t at the time t in the back right to the frame t + 1 at the time t + 1 in the left front. In the example of FIG. 10, each frame shows only an area of 8 pixels × 8 pixels used for the gradient method calculation as a peripheral area (block) of the pixel of interest p.

  In the frame t, when the motion vector V (vx, vy) of the pixel of interest p, which is the fifth pixel down from the upper left pixel and the fifth pixel on the right, is obtained using the gradient method described above, the motion vector V (vx , Vy) is the difference in luminance (ie, gradient) Δx and Δy between adjacent pixels px and py obtained for the x and y directions of the pixel of interest p, and the same phase of the pixel of interest p obtained in frame t + 1. A difference in luminance (gradient) Δt in the time direction with respect to the pixel q located is obtained for all pixels in the peripheral region (8 pixels × 8 pixels) of the pixel of interest p, and the difference between them is expressed by Equation (9). It can obtain | require by calculating using.

  That is, the gradient method calculates gradients Δx, Δy, and Δt between two frames, and statistically calculates the motion vector V (vx, Δt from the obtained Δx, Δy, and Δt using the sum of squares of differences. vy).

  Generally, in a motion vector detection method using such a gradient method, a highly accurate result can be obtained for a minute motion. However, when trying to obtain a motion in an actual moving image, this gradient method is not practical because the amount of motion is too large. Correspondingly, a method of repeating this gradient method a plurality of times is conceivable. By repeatedly executing the gradient method, the amount of motion obtained by each calculation converges, so that a correct motion is gradually obtained.

  However, it is not practical from the viewpoint of calculation time when trying to perform real-time processing only by repeating the gradient method. Therefore, the vector detection unit 52 reduces the number of iterations of the gradient method by using, as an initial value, an initial vector obtained based on the motion of surrounding pixels in the past frame and the current frame. In other words, if a rough motion is calculated by adding an offset in advance from the pixel of interest that is the starting point of the motion to the destination indicated by the initial vector, an operation using the gradient method is performed from the position where the offset is added. Fine adjustment including movement below the pixel can be performed. Thereby, an accurate motion vector can be detected without increasing the calculation time.

  FIG. 11 is a diagram specifically explaining the iterative gradient method executed using the initial vector. In the example of FIG. 11, the arrow T indicates the passage of time from the frame t at the time t on the left front side to the frame t + 1 at the time t + 1 on the right back in the drawing. A block centered on each pixel p, q0, q1, q2, and q3 represents a peripheral region (block) used for the gradient method calculation of that pixel.

  In the case of the example in FIG. 11, the initial vector v0 obtained in advance is offset (moved) instead of the pixel q0 located in the same phase of the target pixel p in the frame t + 1 with respect to the target pixel p in the frame t. The first gradient method calculation is performed with the position (pixel) q1 calculated in this way as the starting point, and as a result, a motion vector v1 is obtained.

  Next, the second gradient method calculation is performed with the position (pixel) q2 calculated by offsetting v0 + v1 from the pixel q0 as a starting point, and as a result, a motion vector v2 is obtained. Thereby, the motion vector V is finally obtained as Expression (10).

V = v0 + v1 + v2
... (10)

  As described above, by performing the iterative gradient method using the initial vector, it is possible to obtain a highly accurate motion vector while reducing the calculation time.

  Next, details of the motion vector detection process will be described with reference to the flowchart of FIG. The vector detection unit 52 receives the input frame t of the image at time t and input frame t + 1 of the image at time t + 1.

  In step S101, the initial vector selection unit 101 selects a block to be processed on the frame t as a target block, and the process proceeds to step S102. On the frame, the processing is executed in the raster scan order from the upper left block.

  In step S102, the initial vector selection unit 101 performs an initial vector selection process. In step S101, the initial vector selection unit 101 selects a motion vector with high reliability from the detection result of the past motion vector for each predetermined block, and uses the selected motion vector as an initial value used in the gradient method. The initial vector V0 is output to the iterative gradient method computing unit 103, and the process proceeds to step S103.

  That is, the initial vector selection unit 101 obtains the motion vector of the peripheral blocks obtained in the past gradient method calculation evaluation process (step S103 described later) and stored in the detection vector memory 53, and the past shifted initial vector allocation process (described later). In step S104), the shifted initial vector stored in the shifted initial vector memory 107 is selected as an initial vector candidate vector. Then, the initial vector selection unit 101 obtains the evaluation value DFD of the candidate vector using the frame t and the frame t + 1, and selects a candidate vector having a high reliability based on the obtained evaluation value DFD. The selected candidate vector is output as the initial vector V0. Details of the initial vector selection process in step S102 will be described later.

  In step S103, the iterative gradient method calculation unit 103 and the vector evaluation unit 104 execute an iterative gradient method calculation evaluation process (also referred to as an iterative gradient method calculation process), and the process proceeds to step S104. Specifically, in step S103, the iterative gradient method computing unit 103 receives the initial vector V0 input from the initial vector selecting unit 101, the frame t and the frame input via the prefilters 102-1 and 102-2. Based on the motion vector evaluation result by the vector evaluation unit 104 using t + 1, the gradient method is repeatedly calculated to calculate the motion vector Vn. Further, the vector evaluation unit 104 obtains the motion vector Vn−1 from the iterative gradient method computing unit 103 and the evaluation value DFD of the motion vector Vn, and selects the most reliable one based on the obtained evaluation value DFD. And stored in the detected vector memory 53 as a motion vector V. At this time, the vector evaluation unit 104 supplies the evaluation value DFD obtained for the motion vector V together with the motion vector V to the shifted initial vector allocation unit 105. Details of the iterative gradient method calculation processing in step S103 will be described later.

  In step S104, the shifted initial vector allocation unit 105 executes a shifted initial vector allocation process, and proceeds to step S105. When the motion vector V and its evaluation value DFD are supplied from the vector evaluation unit 104, the shifted initial vector allocation unit 105 shifts the motion vector passing through the target block on the next frame to the target block in step S104. The shifted initial vector is set. That is, in other words, a motion vector having the same size and the same direction as the motion vector V is set as the initial shift vector, starting from the block of interest on the next frame at the same position as the end block of the motion vector V. . Then, the shifted initial vector allocation unit 105 allocates the set shifted initial vector to the shifted initial vector memory 107 in association with the block of interest.

  Specifically, the shifted initial vector allocation unit 105 stores the evaluation value DFD of the motion vector V allocated as the shifted initial vector in the evaluation value memory 106 in association with the target block, so that the same target Compared with the evaluation value DFD of another motion vector V that passes through the block (that is, the block of the past frame at the same position as the target block is the end point), the motion determined to have high reliability based on the evaluation value DFD The vector V is shifted to that block and set as a shifted initial vector, and is allocated to the shifted initial vector memory 107 in correspondence with the shifted block. Details of the configuration of the shifted initial vector allocation unit 105 will be described later.

  In step S105, the initial vector selection unit 101 determines whether or not all blocks have been processed in the frame t. If it is determined that all blocks have not been processed, the initial vector selection unit 101 returns to step S101. The subsequent processing is repeated. In step S105, the initial vector selection unit 101 determines that the motion vectors V have been detected in all the blocks on the frame t when it is determined that all the blocks have been processed in the frame t. Then, the motion vector detection process ends.

  As described above, an initial vector is selected from motion vectors detected in the past, an iterative gradient method is calculated based on the selected initial vector, and a repeated motion vector is calculated. Among them, a motion vector having a high reliability (that is, most likely) based on the evaluation value DFD is detected. As a result, the motion vector V corresponding to all the blocks on the frame t is stored in the detection vector memory 53.

  Next, the details of the configuration of the shifted initial vector allocation unit 105 will be described.

  FIG. 13 is a block diagram showing a configuration of the shifted initial vector allocating unit 105. The shifted initial vector allocation unit 105 having the configuration shown in FIG. 13 sets a shifted initial vector that is a candidate vector of the initial vector based on the motion vector V detected by the vector evaluation unit 104 in the previous (past) frame, A process of assigning to the shifted initial vector memory 107 is performed. The shifted initial vector allocation unit 105 receives the motion vector V detected by the vector evaluation unit 104 and the evaluation value DFD of the motion vector V.

  The allocation target position calculation unit 201 detects the position of the block through which the motion vector V detected by the vector evaluation unit 104 passes on the frame at the next time (that is, the end point of the motion vector V detected on the current frame). The position of the block on the next frame at the same position as the block) is calculated, and the calculated position of the block is supplied to the evaluation value memory 106 and the shifted initial vector replacement unit 203.

  When the evaluation value comparison unit 202 receives the motion vector V and the evaluation value DFD of the motion vector V, the evaluation value comparison unit 202 reads the evaluation value DFD of the block position from the allocation target position calculation unit 201 from the evaluation value memory 106. . Then, the evaluation value comparison unit 202 compares the evaluation value DFD read from the evaluation value memory 106 with the evaluation value DFD of the motion vector V detected by the vector evaluation unit 104, and detects the detected motion vector V When the evaluation value DFD is smaller (that is, the reliability is higher), the shifted initial vector replacement unit 203 is controlled to determine the block of the block supplied by the shifted initial vector allocation unit 105 of the shifted initial vector memory 107. The position initial shift vector is rewritten with the motion vector V determined to have high reliability based on the evaluation value DFD. At the same time, the evaluation value comparison unit 202 controls the evaluation value replacement unit 204, and the evaluation value memory 106 selects the evaluation value DFD of the position of the block selected by the allocation target position calculation unit 201 in the evaluation value memory 106. Rewrite with evaluation value DFD.

  The shift initial vector replacement unit 203 uses the motion vector V (that is, the motion) supplied from the evaluation value comparison unit 202 as the shift initial vector of the block position supplied from the allocation target position calculation unit 201 in the shift initial vector memory 107. The motion vector having the same magnitude as the vector V and the same direction) is rewritten. The evaluation value replacement unit 204 converts the evaluation value DFD of the position of the block selected by the allocation target position calculation unit 201 into the evaluation value DFD of the motion vector V in the evaluation value memory 106 under the control of the evaluation value comparison unit 202. Rewrite with.

  The evaluation value memory 106 stores, for each block, an evaluation value DFD of a shifted initial vector candidate assigned to each block on the next frame. The shifted initial vector memory 107 stores the motion vector having the smallest evaluation value DFD (that is, the most reliable) in each block in the next frame as a shifted initial vector in association with the block.

  FIG. 14 is a diagram for describing one-dimensionally the shifted initial vectors allocated by the shifted initial vector allocating unit 105. In the example of FIG. 14, a frame t−1 at time T = t−1, a frame t at time T = t, and a frame t + 1 at time T = t + 1 are shown in order from the top. The partition indicates a block boundary in each frame.

  In the case of the example in FIG. 14, the motion vector detected in the block B on the frame t−1 is the motion vector V (solid arrow in the figure), and the motion on the frame t by the motion vector V starting from the block B. A compensation destination (hereinafter also referred to as an offset destination) is a block Bt. A motion vector obtained by shifting the motion vector V on the frame t-1 to the block Bt on the frame t is defined as a shifted initial vector SV (dotted line arrow in the figure). In this case, the offset destination represents the block Bt on the frame t at the same position as the end block of the motion vector V detected on the frame t−1, and the shift means the frame t The start point of a motion vector having the same magnitude and the same direction as the motion vector V detected on -1 is a block Bt on the frame t at the same position as the end point block of the motion vector V on the frame t-1. That means. That is, the shifted initial vector SV is the same size and the same direction as the motion vector V, starting from the block Bt on the frame t at the same position as the end block of the motion vector V detected on the frame t−1. It is a motion vector.

  In general, there is a certain degree of continuity in the amount of movement of the moving object between consecutive frames, and the change in the amount of movement is considered to be small. Therefore, in the example of FIG. When the motion is at or close to it, the motion vector in the block Bt is often the motion vector V or a motion vector close thereto.

  Therefore, when this shifted initial vector SV is used as a candidate for an initial vector given to the gradient method calculation when detecting a motion vector of the block of interest Bt on the frame t, only the motion vectors of the peripheral blocks are used as initial vector candidates. It is possible to obtain a more appropriate initial vector for motion vector detection by the gradient method calculation than in the case of using as.

  The above will be specifically described with reference to FIGS. 15 and 16. In FIGS. 15 and 16, nine blocks are shown on each frame.

  FIG. 15 shows an example of a motion vector of a block through which a large motion object passes and its neighboring blocks. In the case of the example in FIG. 15, the image object of a certain object O1 consisting of one block is a motion vector V1 of a large motion, and is on the frame t from the rightmost block (the ninth from the left) on the frame t-1. It moves to the leftmost block on frame t + 1 via the fifth block from the left. On the other hand, the image object of the object O2 composed of four blocks is a motion vector V2 having a motion smaller than the motion vector V1, and the fourth to fourth blocks from the third to sixth from the left on the frame t−1 and 4 from the left on the frame t. Through the 4th to 7th blocks, the frame moves to the 5th to 8th 4 blocks from the left on the frame t + 1. That is, in the fifth block from the left on the frame t, the image object of the object O1 and the image object of the object O2 intersect each other with the image object of the object O1 in front.

  At this time, in order to detect the motion vector of the fifth block from the left on which the image object of the object O1 passes by the motion vector V1 on the frame t, the initial vector is extracted from the motion of the peripheral blocks. Since the image object of the object O2 passes through the block (the fourth or sixth block from the left) with a motion vector V2 different from that of the image object of the object O1, there is no appropriate motion vector. On the other hand, the shift initial vector SV1 (dotted arrow in the figure) is the fifth motion vector from the left on the frame t, even in such a case, with the motion vector V1 detected on the past frame T-1. Motion vector shifted to a block (that is, the same size and the same direction as the motion vector V1, starting from a block on the frame t at the same position as the end block of the motion vector V1 detected on the frame t-1) Therefore, the motion vector can be reliably acquired.

  FIG. 16 shows an example of a motion vector of a block at the boundary of a moving object and its surrounding blocks. In the case of the example of FIG. 16, the image object of a certain object O3 consisting of one block is a motion vector V3, and the sixth block from the left on the frame t-1 to the fifth block from the left on the frame t. Via, it moves to the fourth block from the left on the frame t + 1. On the other hand, the image object of the object O4 composed of 9 blocks or more is a motion vector V4 different from the motion vector V1, and from the 9 blocks on the frame t-1 to the 2nd to 9th 8 blocks from the left on the frame t. Via, it moves to the third to ninth seventh blocks from the left on the frame t + 1. That is, the image object of the object O3 is in front of the image object of the object O4 in the sixth block from the left of the frame t-1, the fifth block from the left on the frame t, and the fourth block from the left of the frame t + 1. Is going through.

  At this time, in order to detect the motion vector of the fifth block from the left on which the image object of the object O3 passes with the motion vector V3 on the frame t, the initial vector is extracted from the motion of the peripheral blocks. Since the image object of the object O4 passes through the block (the fourth or sixth block from the left) with a motion vector V4 different from that of the image object of the object O3, there is no appropriate motion vector. However, in this case as well, as in the case of FIG. 15, the shifted initial vector SV3 (indicated by a dotted arrow in the figure) represents the motion vector V3 detected on the past frame T-1 as the right on the frame t. To the fifth block (ie, the same size as the motion vector V3, starting from the block on the frame t at the same position as the end block of the motion vector V3 detected on the frame t-1) The motion vectors in the same direction in FIG.

  As described above, at the boundary of an object having a large movement as shown in FIG. 15 or an object moving as shown in FIG. There are many cases where the optimal motion vector does not exist as the initial vector of the block of interest in the peripheral block. On the other hand, the initial shift vector has the same size and the same size as the motion vector starting from the block on the frame t at the same position as the end block of the motion vector detected on the past frame t-1. Since it is a motion vector in the direction, it does not depend on the spatial distance between the blocks, so even if there is no appropriate motion vector as the initial vector of the block of interest in the surrounding blocks, it can be obtained reliably. It can be used as an optimal candidate.

  Here, returning to FIG. 14, a method for determining the shifted initial vector will be described. In the example of FIG. 14, with reference to the target block Bt on the frame t on which the motion vector detection process is performed, the motion vector passing from the frame t-1 through the target block Bt (that is, at the same position as the block Bt on the frame t). If an attempt is made to search for a motion vector whose end point is a block of a certain frame t-1, the motion vectors of all the blocks on the frame t-1 must be searched, and the processing becomes enormous. Therefore, in this case, when detecting a motion vector, it is necessary to deal with a huge amount of processing, and it is difficult to implement (hardware).

  Therefore, the shifted initial vector allocating unit 105 detects the position of the block through which the motion vector V detected in the block of interest Bt passes on the frame t at the next time (that is, when detecting the motion vector in the frame t−1). A block on the frame t at the same position as the end point block of the motion vector V) is obtained, and the motion vector V is assigned in advance as a shift initial vector SV for the block Bt. As a result, it is possible to reduce the amount of calculation for searching for the initial shift vector necessary for detecting the motion vector on the frame t, and to realize hardware implementation. It can be used as a candidate for an initial vector.

  With reference to FIG. 17, a method for determining the shifted initial vector by shifted initial vector allocating unit 105 will be specifically described. In the example of FIG. 17, the arrow T indicates the passage of time from the frame t-1 at time t-1 on the left front side to the frame t at time t on the right back in the drawing. A circle on each frame represents a pixel.

  On the frame t−1, the block B0 is composed of 4 × 4 pixels. Assuming that the motion vector Vb detected in the block B0, an offset destination (a block on the frame t at the same position as the end block of the motion vector Vb of the block B0) A0 obtained by offsetting (motion compensation) the block B0 on the frame t is In general, there is an overlap with four blocks Bt1 to Bt4 configured by 4 × 4 pixels on the frame t. Therefore, in the example of FIG. 17, all of the four blocks Bt1 to Bt4 that overlap with the offset destination A0 are set as target blocks to which the motion vector Vb is allocated as the shifted initial vector.

  The motion vector detected in the previous frame as described above is assigned as the shifted initial vector. In this case, the position of the block to be allocated depends on the magnitude and direction of the motion vector. Therefore, depending on the distribution of motion vectors detected on the frame t−1, as shown in FIG. 18, on the frame t, a block that is a target for allocation of a plurality of shift initial vectors, or conversely, a shift Blocks that do not have an initial vector assignment will occur.

  FIG. 18 shows an example of allocation of shifted initial vectors. In the example of FIG. 18, the partition of each frame indicates the boundary of the block, and is composed of blocks B1 to B5 on the frame t−1 from the left in the figure, and on the frame t on the left in the figure. Thus, blocks Bt1 to Bt5 are shown.

  In the block B1, the motion vector V1 is detected, and the offset destination A1 of the block B1 overlaps with the blocks Bt1 and Bt2 on the frame t. In block B2, motion vector V2 is detected, and offset destination A2 of block B2 overlaps with blocks Bt1 and Bt2 on frame t. In block B3, motion vector V3 is detected, and offset destination A3 of block B3 overlaps with blocks Bt4 and Bt5 on frame t. In block B4, motion vector V4 is detected, and offset destination A4 of block B4 overlaps with blocks Bt4 and Bt5 on frame t. In the block B5, the motion vector V5 is detected, and the offset destination A5 of the block B5 overlaps with the block Bt5 on the frame t and an adjacent block (not shown).

  That is, in the example of FIG. 18, one of the motion vectors V1 and V2 is allocated as the shifted initial vector to the blocks Bt1 and Bt2 on the frame t. Also, one of the motion vectors V3 and V4 is assigned as the initial shift vector to the block Bt4 on the frame t, and any one of the motion vectors V3, V4, and V5 is assigned to the block Bt5 on the frame t. Are assigned as shifted initial vectors. However, in the block Bt3 on the frame t, there is no motion vector that is a candidate for the shifted initial vector, and the shifted initial vector is not allocated.

  Therefore, the shifted initial vector allocating unit 105 allocates the zero vector as the shifted initial vector to a block such as the block Bt3 where allocation of the shifted initial vector does not occur. On the other hand, the shifted initial vector allocating unit 105 assigns a motion vector having high reliability based on the above-described evaluation value DFD to a block in which a plurality of motion vectors compete, such as the blocks Bt1, Bt2, Bt4, and Bt5. The selected motion vector is assigned to each block as a shifted initial vector.

  In the example of FIG. 18, for convenience of explanation, the offset destination A is shown slightly shifted from the frame t.

  FIG. 19 is a diagram illustrating an example of competing motion vectors. In the example of FIG. 19, the motion vectors V1 and V2 detected in the blocks B1 and B2 on the frame t-1 respectively pass the block Bt on the frame t. That is, the block Bt is a block on the frame t at the same position as the end point on the frame t-1 of the motion vector V1, and is a block on the frame t at the same position as the end point on the frame t-1 of the motion vector V2. is there. Thus, when a plurality of motion vectors compete for the same block, the evaluation value DFD1 for the motion vector V1 is compared with the evaluation value DFD2 for the motion vector V2, and the evaluation value DFD is smaller (that is, , With higher reliability) is selected as the shifted initial vector SV2 corresponding to the block Bt. As a result, a highly reliable shifted initial vector based on the evaluation value DFD can be obtained.

  Next, the details of the shifted initial vector allocation processing will be described with reference to the flowchart of FIG. In the preceding stage, when the vector evaluation unit 104 detects the motion vector V of the block of interest on the frame t−1, the vector evaluation unit 104 uses the detected motion vector V and the evaluation value DFD obtained for the motion vector V as the shifted initial vector. This is supplied to the allocation unit 105.

  In step S <b> 201, the evaluation value comparison unit 202 inputs the evaluation value DFD of the motion vector V together with the motion vector V from the vector evaluation unit 104. At this time, the allocation target position calculation unit 201 also receives the motion vector V and proceeds to step S202. The allocation target position calculation unit 201 obtains the position of the allocation target block that is the offset (motion compensation) destination of the motion vector V in the frame t. That is, the allocation target position calculation unit 201 obtains the position of the block on the frame t at the same position as the end point block of the motion vector V detected on the frame t−1, and proceeds to step S203. In step S203, the allocation target position calculation unit 201 selects one allocation target block from the obtained allocation target blocks, and sets the position of the selected allocation target block to the evaluation value memory 106 and the shifted initial vector replacement unit 203. To proceed to step S204. In step S203, the blocks to be allocated are selected in order from the upper left block on the frame t.

  In step S204, the evaluation value comparison unit 202 acquires the evaluation value DFD of the allocation target block selected by the allocation target position calculation unit 201 from the evaluation value memory 106, proceeds to step S205, and the motion input in step S201. Whether the evaluation value DFD of the vector V is smaller than the evaluation value DFD of the evaluation value memory 106 (that is, whether the evaluation value DFD of the motion vector V is more reliable than the evaluation value DFD of the evaluation value memory 106). If the evaluation value DFD of the motion vector V is determined to be smaller than the evaluation value DFD of the evaluation value memory 106, the process proceeds to step S206.

  In step S206, the evaluation value comparison unit 202 controls the shift initial vector replacement unit 203 to determine the shift initial vector of the allocation target block in the shift initial vector memory 107 selected by the allocation target position calculation unit 201 as the motion vector V ( That is, the motion vector V is rewritten to a motion vector having the same magnitude and the same direction as the motion vector V, and the process proceeds to step S207 where the evaluation value replacement unit 204 is controlled and the evaluation value of the allocation target block selected by the allocation target position calculation unit 201 The DFD is rewritten with the evaluation value DFD of the motion vector V, and the process proceeds to step S208.

  If the evaluation value comparison unit 202 determines in step S205 that the evaluation value DFD of the motion vector V input in step S201 is not smaller than the evaluation value DFD stored in the evaluation value memory 106, step S206 is performed. And the process of S207 is skipped and it progresses to step S208. That is, in this case, since the evaluation value DFD of the evaluation value memory 106 is determined to have higher reliability than the evaluation value DFD of the motion vector V, the values of the evaluation value memory 106 and the shifted initial vector memory 107 are rewritten. There is nothing.

  In step S208, the allocation target position calculation unit 201 determines whether all the allocation target blocks of the motion vector V have been processed, and if it determines that all the allocation target blocks have not been processed, step S203. Return to and repeat the subsequent processing. If the allocation target position calculation unit 201 determines in step S208 that all the allocation target blocks of the motion vector V have been processed, the shifted initial vector allocation process ends.

  In the first process, the shifted initial vector corresponding to the selected allocation target block is not yet stored in the shifted initial vector memory 107. Therefore, if the evaluation value DFD of the shifted initial vector is not yet stored in the corresponding allocation target block of the evaluation value memory 106, the evaluation value DFD is obtained from the selected allocation target block in step S204. Since it is not carried out, it will be judged as Yes in Step S205 and processing of Steps S206 and S207 will be performed.

  As described above, when the motion vector detection of the block of interest is processed, the motion amount of the moving object has a certain degree of continuity between successive frames, and the change in the motion amount is small. A motion vector that passes through the target block of the frame at the next time from the frame at the time (past) (that is, a block on the next frame at the same position as the end block of the motion vector V detected on the past frame) As a conventional method, a shift initial vector that is a motion vector having the same magnitude as the motion vector V and having the same direction as a starting point is used as a candidate vector of the initial vector, and thus is obtained in the past in the peripheral block as in the past. Compared to the case where only the motion vector is the candidate vector of the initial vector, especially at the boundary of the moving object, the gradient method It can give an appropriate motion vector as an initial value for calculation, which makes it possible to improve the accuracy of motion vector detection by computation of the gradient method.

  Further, when the initial shift vector is obtained, the motion vector detected in the frame at the previous time passes in the frame at the next time (that is, the block at the end of the motion vector V detected on the frame t−1). And a motion vector passing through the block of interest is assigned to all the frames of the previous time. Compared to the search from the motion vector of the block, the calculation amount of processing is reduced, so that it is possible to realize hardware that is difficult to realize due to the huge amount of calculation.

  Next, details of the configuration of the initial vector selection unit 101 will be described.

  FIG. 21 is a block diagram illustrating a configuration of the initial vector selection unit 101. The initial vector selection unit 101 having the configuration shown in FIG. 21 uses a motion vector with high reliability from a motion vector detected in the previous (past) frame and a candidate vector such as a shifted initial vector (hereinafter also referred to as an initial candidate vector). Is selected as the initial vector. The initial vector selection unit 101 receives the frame t of the image at time t and the frame t + 1 of the image at time t + 1.

  When the frame t is input, the candidate vector position calculation unit 251 selects a target block to be processed on the frame t, and obtains an initial candidate vector of the target block from the peripheral region of the target block. The position, the type of motion vector to be the initial candidate vector and the priority order are obtained, and the position information of the candidate block and the type information of the initial candidate vector are obtained in the order of the obtained priority order, and the detected vector acquisition unit 252 and the shifted initial vector acquisition unit 253. The candidate vector position calculation unit 251 also supplies the position information of the candidate block to the offset position calculation unit 254.

  In the signal processing device 1, the number of initial candidate vectors is set to a predetermined number based on the balance between the accuracy of the initial vector and the hardware capability, and further, the position of the candidate block and the initial candidate vector The type and priority are also set in advance as will be described later with reference to FIG. In addition, as a type of the initial candidate vector, a motion vector obtained by shifting a motion vector passing through a predetermined block in the past frame to the predetermined block (that is, a block at the end point of the motion vector detected on the past frame) A shift initial vector SV, which is a motion vector having the same magnitude and the same direction as the motion vector V, starting from a block on the next frame at the same position, and a motion vector detected in the past frame (hereinafter, past vector) PV), a motion vector detected in a block before the target block in the current frame (also referred to as current vector CV), and a zero vector.

  Accordingly, when the preset initial candidate vector type information is a past vector or a current vector, the candidate vector position calculation unit 251 displays the candidate block position information and the initial candidate vector type information as the detected vector acquisition unit 252. If the type information of the obtained initial candidate vector is a shifted initial vector, the position information of the candidate block and the type information of the initial candidate vector are supplied to the shifted initial vector acquisition unit 253, which is neither of them. In the case (for example, when the type information of the initial candidate vector is 0 vector), the 0 vector is set, and the position information of the candidate block is supplied to the offset position calculation unit 254 together with the 0 vector.

  The detection vector acquisition unit 252 acquires the motion vector corresponding to the position information of the candidate block and the type information of the initial candidate vector supplied from the candidate vector position calculation unit 251 from the detection vector memory 53, and the acquired motion vector is The initial candidate vector is output to the offset position calculation unit 254.

  The shifted initial vector acquisition unit 253 generates the shifted initial vector corresponding to the position information of the candidate block according to the position information of the candidate block and the type information of the initial candidate vector supplied by the candidate vector position calculating unit 251. Obtained from the memory 107 and output to the offset position calculation unit 254 as an initial candidate vector. In addition, the shifted initial vector acquisition unit 253 outputs a zero vector to the offset position calculating unit 254 when the shifted initial vector is not allocated to the position of the block designated by the candidate vector position calculating unit 251. If no shifted initial vector is assigned, the zero vector may be stored in advance in the shifted initial vector memory 107.

  When the initial position candidate vector (or 0 vector from the candidate vector position calculation section 251) is input from the detection vector acquisition section 252 or the shifted initial vector acquisition section 253, the offset position calculation section 254 is supplied by the candidate vector position calculation section 251. Based on the position information of the candidate block, an offset destination block position obtained by offsetting (motion compensation) the target block of frame t to frame t + 1 is calculated for each initial candidate vector. Then, the offset position calculation unit 254 outputs the candidate block position information and the offset destination block position information to the evaluation value calculation unit 255 together with the initial candidate vector.

  When the evaluation value calculation unit 255 inputs the position information of the candidate block and the information of the offset destination block position together with the initial candidate vector from the offset position calculation unit 254, the evaluation value of the initial candidate vector is used using the frame t and the frame t + 1. Find the DFD. Then, the evaluation value calculation unit 255 outputs the obtained evaluation value DFD together with the initial candidate vector to the evaluation value comparison unit 256.

  The evaluation value comparison unit 256 compares the evaluation value DFD input by the evaluation value calculation unit 255 with the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 257 and inputs the evaluation value DFD by the evaluation value calculation unit 255. When the evaluation value DFD of the initial candidate vector is smaller than the evaluation value DFD of the optimal candidate vector, that is, when it is determined that the initial candidate vector has higher reliability than the optimal candidate vector, the optimal candidate storage register 257 The optimal candidate vector and its evaluation value DFD are replaced with the initial candidate vector and its evaluation value DFD that are determined to have high reliability. Finally, the evaluation value comparison unit 256 controls the optimum candidate storage register 257 to select the optimum candidate vector determined to have the highest reliability based on the evaluation value DFD from among all candidate vectors. The iterative gradient method computing unit 103 outputs the initial vector V0.

  In the optimum candidate storage register 257, an initial candidate vector whose evaluation value DFD is made small (high reliability) by the evaluation value comparison unit 256 is stored as an optimum candidate vector together with the evaluation value DFD. Then, under the control of the evaluation value comparison unit 256, the optimal candidate storage register 257 outputs the finally stored optimal candidate vector to the iterative gradient method computing unit 103 as the initial vector V0.

  FIG. 22 shows a peripheral region that can be a candidate block of the initial vector. In the example of FIG. 22, the arrow T indicates the direction of time passage from the past (previous) frame t−1 at the time t−1 on the left front side to the current frame t at the time t on the right rear side. . In the example of FIG. 22, the peripheral region that can be a candidate for the initial vector is configured by 7 blocks × 7 blocks centered on the block Bt of interest, and each block is configured by 4 pixels × 4 pixels. Has been.

  As described above, the motion vector detection process is executed in the raster scan order from the upper left block of the frame. Therefore, the initial vector selection unit 101 can use the result of motion vector detection up to the immediately preceding block as an initial vector candidate in the motion vector detection processing of the block Bt of interest in the current frame t.

  That is, the peripheral area of the block of interest Bt is composed of a block CVB processed before the block of interest Bt and a block PVB processed after the block of interest Bt. Therefore, when obtaining the initial vector of the block of interest Bt, the initial candidate vector is detected on the motion vector (current vector CV) detected on the current frame t of the block CVB and on the past frame t-1 of the block PVB. Motion vectors (past vectors PV) can be selected. At this time, a shifted initial vector assigned to a block in the same peripheral area can also be a candidate for an initial candidate vector.

  FIG. 23 illustrates an example of initial vector candidate blocks set in the candidate vector position calculation unit 251. In the example of FIG. 23, eight predetermined blocks each having an “alphabet / number” symbol are set as candidate blocks from which an initial candidate vector is acquired in the peripheral area of the block of interest Bt. Yes. The symbols before and after the hatched lines shown in these eight predetermined blocks represent “initial candidate vector type” and “priority”, respectively. Of the types of initial candidate vectors, P represents the past vector PV detected in the past frame t−1, C represents the current vector detected in the current frame t, and S is The shifted initial vector SV is represented.

  “S / 1” of the target block Bt indicates that the shifted initial vector SV assigned to the target block Bt is used as the initial candidate vector in the priority “1”. “C / 2” of the block adjacent to the left of the block of interest Bt indicates that the current vector CV of the block detected in the current frame t is used as the initial candidate vector in the “2” priority order. “C / 3” in the upper right block of the block of interest Bt indicates that the current vector CV of the block detected in the current frame t is used as the initial candidate vector with the priority “3”. “P / 4” in the lower right block of the target block Bt indicates that the past vector PV of the block detected in the past frame t−1 is used as the initial candidate vector with the priority “4”. Yes.

  As for “C / 6” of the lower right block at the upper left corner of the peripheral area 7 × 7, the current vector CV of the block detected in the current frame t is used as the initial candidate vector with the priority “6”. Represents. “C / 5” of the lower left block in the upper right corner of the peripheral area 7 × 7 indicates that the current vector CV of the block detected in the current frame t is used as the initial candidate vector with the priority “5”. Represents. For the upper right block “P / 7” in the lower left corner of the peripheral area 7 × 7, the past vector PV of the block detected in the past frame t−1 is used as the initial candidate vector with the priority “7”. Represents that. In the upper left block “P / 8” in the lower right corner of the peripheral area 7 × 7, the past vector PV of the block detected in the past frame t−1 is used as the initial candidate vector with the priority “8”. Represents that.

  As described above, in the example of FIG. 23, the initial candidate vectors include the block of interest Bt, the block adjacent to the block of interest Bt, the block next to the block of interest Bt, the block adjacent to the left of the block of interest Bt, the upper right, and the lower right. Among the blocks that are not adjacent to the block of interest Bt, the block is selected from the blocks at positions one inside each from the four corners of the peripheral area 7 × 7.

  That is, in the case of the example in FIG. 23, a block adjacent to the block of interest Bt (a block that is spatially and temporally close) has a higher priority than a block slightly apart, and the current vector detected in the current frame t. The priority is set higher than the past vector detected in the past frame t-1. Note that the positions, types, and priorities of these initial vector candidates are not only in the vicinity of the block of interest Bt but also slightly apart so that a reliable initial candidate vector can be obtained with as few candidates as possible. These blocks are set on the basis of spatial correlation and temporal correlation so that there is no bias. Therefore, using the motion vector of a predetermined block set in advance as an initial candidate vector in this way prevents the number of initial candidate vectors from increasing and the amount of computation of processing from becoming enormous. .

  Note that the types of candidate blocks and the positions of the candidate blocks are not limited to the example of FIG. 23, and other types of vectors and block positions may be used. In consideration of the case where the optimum vector does not exist in the peripheral blocks, a zero vector that suppresses deterioration in such a case may be used. Also, the number of candidate blocks may be more or less than eight.

  Next, the details of the initial vector selection process will be described with reference to the flowchart of FIG.

  In step S251, the candidate vector position calculation unit 251 determines the position of the candidate block, the type of the initial candidate vector, and the priority order from which the initial candidate vector of the target block set in advance is acquired from the peripheral area of the selected target block. In step S252, it is determined whether or not the type of the initial candidate vector of the candidate block is a past vector or a current vector in the order of the obtained priority. In step S252, when the candidate vector position calculation unit 251 determines that the type of the initial candidate vector of the candidate block is the past vector or the current vector, the process proceeds to step S253, where the candidate block position information and the type of initial candidate vector are determined. Information is supplied to the detection vector acquisition unit 252, and the motion vector (past vector PV or current vector CV) according to the position information of the candidate block and the type information of the initial candidate vector is detected in the detection vector memory. 53, the acquired motion vector is output to the offset position calculation unit 254, and the process proceeds to step S257.

  In step S252, when the candidate vector position calculation unit 251 determines that the type of the initial candidate vector of the candidate block is not a past vector or a current vector, the process proceeds to step S254, where the candidate vector position calculation unit 251 It is determined whether or not the type of candidate vector is a shifted initial vector. In step S254, if the candidate vector position calculation unit 251 determines that the type of the initial candidate vector of the candidate block is a shifted initial vector, the process proceeds to step S255, where the candidate block position information and the initial candidate vector type information are obtained. The shifted initial vector acquisition unit 253 supplies the shifted initial vector acquisition unit 253 with the shifted initial vector corresponding to the position information of the candidate block from the shifted initial vector memory 107, and the acquired shifted initial vector is offset. The position calculation unit 254 is output, and the process proceeds to step S257.

In step S254, when the candidate vector position calculation unit 251 determines that the type of the initial candidate vector of the candidate block is not the shifted initial vector (that is, determines that the type of the initial candidate vector of the candidate block is the 0 vector). In step S256, a zero vector is set as the initial candidate vector, and the position information of the candidate block is supplied to the offset position calculation unit 254 together with the zero vector, and the process proceeds to step S257. Also in steps S253 and S255, the candidate vector position calculation unit 251 supplies the position information of the candidate block to the offset position calculation unit 254.

  In step S257, the offset position calculation unit 254 receives the initial candidate vector from the detection vector acquisition unit 252 or the shifted initial vector acquisition unit 253, and based on the position information of the candidate block supplied by the candidate vector position calculation unit 251, For each initial candidate vector, an offset destination block position obtained by offsetting the target block of frame t to frame t + 1 is calculated. Then, the offset position calculation unit 254 outputs the position information of the candidate block and the information of the offset destination block position together with the initial candidate vector to the evaluation value calculation unit 255, and proceeds to step S258.

  When the evaluation value calculation unit 255 inputs the position information of the candidate block and the information of the offset destination block position together with the initial candidate vector from the offset position calculation unit 254, in step S258, using the frame t and the frame t + 1, the initial candidate An evaluation value DFD of the vector is obtained, and the obtained evaluation value DFD is output to the evaluation value comparison unit 256 together with the initial candidate vector, and the process proceeds to S259.

  In step S259, the evaluation value comparison unit 256 determines whether or not the evaluation value DFD obtained by the evaluation value calculation unit 255 is smaller than the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 257. The evaluation value DFD obtained by the evaluation value calculator 255 is smaller than the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 257, that is, the initial candidate vector is more reliable than the optimal candidate vector. Is determined to be high, in step S260, the optimal candidate vector of the optimal candidate storage register 257 and its evaluation value DFD are rewritten with the initial candidate vector and its evaluation value DFD that are determined to have high reliability, and the process returns to step S261. move on.

  In step S259, the evaluation value comparison unit 256 determines that the evaluation value DFD obtained by the evaluation value calculation unit 255 is not smaller than the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 257. If so, the process of step S260 is skipped, and the process proceeds to step S261.

  In step S261, the candidate vector position calculation unit 251 determines whether all the initial candidate vectors (eight vectors in the example of FIG. 23) have been processed, and all the initial candidate vectors have been processed. If it is determined that it is not, the process returns to step S252 and the subsequent processing is repeated.

  When it is determined in step S261 that all the initial candidate vectors have been processed, the process proceeds to step S262, and the evaluation value comparison unit 256 controls the optimal candidate storage register 257 to control all the initial candidate vectors. The optimal candidate vector having the highest reliability based on the evaluation value DFD is output to the iterative gradient method computing unit 103 as the initial vector V0, and the initial vector selection process is terminated.

  As described above, in the target block, the evaluation value DFD of a plurality of initial candidate vectors is obtained, and the initial candidate vector having the smallest evaluation value DFD, that is, the highest reliability is selected as the initial vector. Therefore, it is possible to provide an optimal initial vector for the subsequent motion vector detection, and as a result, it is possible to improve the accuracy of the subsequent motion vector detection. In addition, since a predetermined motion vector of a predetermined block is used as an initial candidate vector, the number of initial candidate vectors is increased, and the amount of calculation of processing is suppressed from becoming enormous.

  Furthermore, based on the fact that there is a certain degree of continuity in the amount of movement of moving objects between consecutive frames and the change in the amount of movement is small, an initial shift vector that is a motion vector that passes through the block of interest from the previous frame Since the vector candidates are used, it is possible to perform motion detection with higher accuracy than in the case where only the motion vectors obtained in the past in the peripheral blocks are used as the initial vector candidates. This is particularly effective at the boundaries of moving objects.

  Next, details of the configuration of the iterative gradient method computing unit 103 will be described.

  FIG. 25 is a block diagram showing a configuration of the iterative gradient method computing unit 103. The iterative gradient method computing unit 103 having the configuration shown in FIG. 25 performs processing for detecting an optimal motion vector using the frame t of the image at time t and the frame t + 1 of the image at time t + 1. The process of detecting the motion vector is a process executed for each predetermined block composed of a plurality of pixels, and the iterative gradient method computing unit 103 performs gradients for each block or for each pixel in each block. By repeatedly executing the calculation using the method, an optimal motion vector with high reliability based on the evaluation value DFD is output.

  The iterative gradient method computing unit 103 receives the frame t of the image at time t and the frame t + 1 of the image at time t + 1 via the pre-filters 102-1 and 102-2. The initial vector V0 is input.

  The mode selection unit 401 selects a gradient method calculation processing mode for each predetermined block under the control of the effective pixel determination unit 403, and selects the initial vector V0 from the initial vector selection unit 102 with the selector 402 and the vector evaluation. Output to the unit 104. The gradient method processing mode consists of a block unit processing mode in which the motion vector detection target is a block and a pixel unit processing mode in which the motion vector detection target is a pixel, and the block unit processing mode is selected as an initial value. The Under the control from the vector evaluation unit 104, the selector 402 uses either the initial vector V0 from the mode selection unit 401 or the motion vector Vn output from the gradient method calculation unit 404 as an initial value of the gradient method calculation. The motion vector to be used (hereinafter referred to as an offset vector) is output to the effective pixel determination unit 403 and the gradient method calculation unit 404.

  When the effective pixel determination unit 403 receives the offset vector selected by the selector 402, the frame t of the image at time t and the frame t + 1 of the image at time t + 1 input via the pre-filters 102-1 and 102-2. , Using the position calculated by offsetting the offset vector as a starting point, it is determined whether or not the number of pixels effective for the calculation of the gradient method is greater than the threshold in the calculation block of each unit processing mode, The processing method is switched according to the result. That is, when the effective pixel determination unit 403 determines that the number of pixels effective for the calculation of the gradient method is equal to or less than the threshold value, the effective pixel determination unit 403 controls the mode selection unit 401 and sets the processing mode for the predetermined block. Then, change to the pixel unit processing mode or stop the calculation. When the effective pixel determination unit 403 determines that the number of pixels effective for gradient method calculation is greater than the threshold value in the calculation block of each unit processing mode, the effective pixel determination unit 403 controls the gradient method calculation unit 404 to The gradient method calculation process is executed.

  When the gradient vector calculation unit 404 receives the offset vector input from the selector 402, the frame t of the image at time t and the frame t + 1 of the image at time t + 1 input via the prefilters 102-1 and 102-2. Using the offset vector from the selector 402 as a starting point, the gradient method calculation of each unit processing mode is executed, and the calculated motion vector Vn is used as the delay unit 405 and the vector evaluation unit 104. Output to.

  The delay unit 405 holds the motion vector Vn output from the gradient method calculation unit 404 until the next processing cycle of the effective pixel determination unit 403 and the gradient method calculation unit 404, and according to the control of the selector 402, The held motion vector Vn is output to the effective pixel determination unit 403.

  FIG. 26 is a block diagram illustrating a detailed configuration of the effective pixel determination unit 403. In the example of FIG. 26, the effective pixel determination unit 403 includes a temporal pixel difference calculation unit 411, a pixel difference value determination unit 412, an effective pixel counter 413, and a gradient method continuation determination unit 414.

  When the offset vector selected by the selector 402 is input, the temporal pixel difference calculation unit 411 selects a pixel in an arithmetic block in units of blocks, and inputs the offset vector and the pre-filters 102-1 and 102-2. The pixel difference Δt in the time direction of the selected pixel is calculated using the frame t of the image at time t and the frame t + 1 of the image at time t + 1, and the calculated pixel difference Δt in the time direction is calculated as the pixel difference. The data is output to the value determination unit 412.

  The pixel difference value determination unit 412 determines whether or not the pixel difference Δt in the time direction calculated by the time pixel difference calculation unit 411 is smaller than a predetermined threshold (hereinafter referred to as a pixel difference value). When it is determined that the pixel difference Δt in the time direction is smaller than the predetermined pixel difference value, the number of effective pixels of the effective pixel number counter 413 is counted by one. The effective pixel number counter 413 counts the value of pixels determined to be effective by the pixel difference value determination unit 412 for each calculation block. The value of the effective pixel number counter 413 is reset for each calculation block.

  The gradient method continuation determination unit 414 determines whether or not the number of effective pixels for the gradient method calculation is greater than a threshold value in each unit processing mode calculation block, and the unit processing mode calculation block includes a gradient. If it is determined that the number of pixels effective for the arithmetic operation is greater than the threshold value, a flag (flg = 1) that causes the gradient operation unit 404 to execute the gradient operation for each unit processing mode is output. If the gradient method continuation determination unit 414 determines that the number of pixels effective for gradient method calculation is less than the threshold in the calculation block of each unit processing mode, the gradient method continuation determination unit 414 controls the mode selection unit 401 to perform other unit processing. A mode is selected, or a flag (flg = 0) for stopping the gradient method calculation is output to the gradient method calculation unit 404.

  FIG. 27 is a block diagram showing a detailed configuration of the gradient method computing unit 404. In the example of FIG. 27, the gradient method calculation unit 404 includes a time pixel difference calculation unit 421, a pixel difference value determination unit 422, a horizontal / vertical pixel difference calculation unit 423, a gradient integration unit 424, and a vector calculation unit 425.

  The temporal pixel difference calculation unit 421 controls each unit of the gradient method calculation unit 404 based on the flag input from the effective pixel determination unit 403. In other words, when the flag indicates 1 (flg = 1), the temporal pixel difference calculation unit 421 causes each unit of the gradient method calculation unit 404 to execute gradient method calculation processing. Specifically, when the time pixel difference calculation unit 421 receives a flag from the effective pixel determination unit 403, the frame t of the image at time t input via the prefilters 102-1 and 102-2, and the time t + 1 The pixel in the calculation block of each unit processing mode is selected around the pixel calculated by offsetting the offset vector from the selector 402 using the frame t + 1 of the image of the selected pixel, and the pixel in the time direction of the selected pixel is selected. The difference Δt is calculated, and the calculated pixel difference Δt and offset vector in the time direction are output to the pixel difference value determination unit 422. Also, the time pixel difference calculation unit 421 outputs the offset vector to the vector calculation unit 425 when the processing of the pixels in the calculation block is completed, and controls the vector calculation unit 425 to obtain the motion vector of the detection target block. Let it be calculated.

  On the other hand, when the flag indicates 0 (flg = 0), the time pixel difference calculation unit 421 does not calculate the pixel difference Δt in the time direction of the pixels in the calculation block. The gradient method calculation process is not executed, and the process is terminated. At this time, the temporal pixel difference calculation unit 421 controls the vector calculation unit 425 to set the motion vector V to 0 vector.

  The pixel difference value determination unit 422 determines whether or not the pixel difference Δt in the time direction calculated by the time pixel difference calculation unit 421 is smaller than a predetermined threshold (hereinafter referred to as a pixel difference value). When it is determined that the pixel difference Δt in the time direction is smaller than the predetermined pixel difference value, the pixel is subjected to the gradient method calculation, and the pixel difference Δt in the time direction and the offset vector are input to the horizontal / vertical pixel difference calculation unit 423. Output. That is, the pixel of the pixel difference Δt determined to be smaller than the predetermined pixel difference value is set as a calculation target in the horizontal / vertical pixel difference calculation unit 423 and the gradient integration unit 424. In addition, when the pixel difference value determination unit 422 determines that the pixel difference Δt in the time direction is equal to or greater than a predetermined pixel difference value, the pixel difference value determination unit 422 causes the horizontal / vertical pixel difference calculation unit 423 and the gradient integration unit 424 to process the pixel. Prohibit.

  When the horizontal / vertical pixel difference calculation unit 423 receives the offset vector input from the pixel difference value determination unit 422, the frame t of the image at time t input through the prefilters 102-1 and 102-2, and the time The pixel difference Δt in the time direction is smaller than a predetermined pixel difference value in the calculation block of each unit processing mode centered on the pixel calculated by offsetting the offset vector using the frame t + 1 of the t + 1 image. The pixel difference Δx in the horizontal direction and the pixel difference Δy in the vertical direction of the pixel determined to be calculated are calculated. The horizontal / vertical pixel difference calculation unit 423 outputs the pixel difference Δt in the time direction, the pixel difference Δx in the horizontal direction, and the pixel difference Δy in the vertical direction to the gradient integration unit 424. The gradient integration unit 424 integrates the gradients of pixels that are determined to have a pixel difference Δt in the time direction smaller than a predetermined pixel difference value. That is, the gradient integrating unit 424 is configured to calculate the temporal pixel difference Δt calculated by the temporal pixel difference calculating unit 421, the horizontal pixel difference Δx calculated by the horizontal / vertical pixel difference calculating unit 423, and the vertical pixel difference. Δy is integrated, and the integrated gradient value is output to the vector calculation unit 425.

  When the offset vector is input from the time pixel difference calculation unit 421, the vector calculation unit 425 uses the value of the gradient accumulated by the gradient accumulation unit 424 and the least square sum of Equation (9) described above to use the motion vector. vn is calculated. Further, the vector calculation unit 425 adds the offset vector from the temporal pixel difference calculation unit 421 to the calculated motion vector vn to obtain a motion vector Vn, and outputs it to the vector evaluation unit 104 and the delay unit 405.

  Next, the motion vector detection target block and the calculation block will be described. When considering the movement of an object in an image, the same object often moves the same. Therefore, the unit of detection processing for detecting a motion vector is usually performed in units of blocks as shown in FIG.

  In the example of FIG. 28, the arrow X indicates the horizontal direction, and the arrow Y indicates the vertical direction. In addition, an arrow T indicates the direction of time passage from the frame t at the time t in the back right to the frame t + 1 at the time t + 1 in the left front.

  In the case of the example of FIG. 28, an 8 × 8 pixel block (hereinafter referred to as an 8 × 8 block) is shown as the detection target block K of each frame. The detection target block K is a block in which the same motion is obtained for the 8 × 8 block when the motion vector is detected for the 8 × 8 block.

  On the other hand, motion vector detection (that is, calculation) by the gradient method has a feature that a motion vector can be detected for each pixel. However, when the motion vector is calculated in units of pixels, the gradient method using the least square method, which is a statistical solution, has a drawback that the detection accuracy is significantly reduced. Therefore, in general, when calculating a motion vector not only for each block but also for each pixel, the pixels used for the calculation of the gradient method are configured in units of blocks (calculation blocks). There are many.

  As described above, the pixels used for the calculation of the gradient method are the pixels of the calculation block of 8 × 8 blocks. That is, in the example of FIG. 28, the motion vector obtained as a result of the gradient method computation using the computation block (8 × 8) is the motion corresponding to all the pixels of the detection target block K (8 × 8). It will be a vector.

  However, as described above, as a problem that occurs when the pixels used for the calculation of the gradient method are handled in units of calculation blocks as described above, different motions are generated in the pixels constituting the detection target block as shown in FIG. An example is a case where pixels of objects that are held are mixed.

  In the example of FIG. 29, in the detection target block (= calculation block) K composed of 8 × 8 pixels (64 pixels), 28 pixels (thick line circles) from the upper left pixel to the diagonally upper left indicated by the arrow A. In the 8 × 8 detection target block K, 36 pixels (thin line circles) from the lower right pixel have a rightward movement indicated by an arrow B.

  Therefore, in the detection target block K, the assumption that the movements are almost the same as described above is lost, and if the calculation of the gradient method is performed using a calculation block that includes pixels with different movements, the movement is remarkably increased. The vector detection accuracy is reduced. That is, this is a problem of a decrease in detection accuracy regarding the disappearance region (covered background region) and the generation region (uncovered background region) of the object that occurs at the boundary between objects having different movements. The disappearance region is a mixed region at a position corresponding to all end portions in the advancing direction of the foreground object with respect to the foreground region, and is a region where the background component is covered with the foreground as time passes. On the other hand, the generation region is a mixed region at a position corresponding to the rear end of the foreground object in the traveling direction with respect to the foreground region, and refers to a region where a background component appears as time passes.

  FIG. 30 is a diagram for explaining the luminance state of the detection target block in the case of the same object. On the left side in the figure, the arrow X indicates the horizontal direction, and the arrow Y indicates the vertical direction. In addition, an arrow T indicates the direction of time passage from the frame t at the time t in the back right to the frame t + 1 at the time t + 1 in the left front. On the right side in the figure, the arrow X indicates the horizontal direction, and the arrow L indicates the luminance value. That is, the luminance values Lp1, Lp2, Lp3,..., Lpi (3 <i <7),..., Lp7 on the right side in the figure are, for example, the upper left in the 8 × 8 detection target block of the frame t on the left side in the figure. .., Pi (3 <i <7),..., P7,..., And similarly, the luminance values Lq1, Lq2, and Lq3. ,..., Lqi (3 <i <7),..., Lq7 are pixels p1, p2, p3,..., Pi (3 <i <7) in the 8 × 8 detection target block in the left frame t + 1 in the drawing. ,..., P7,... Are pixels located in the same phase as the pixels q1, q2, q3,..., Qi (3 <i <7),. Represents the luminance value. The arrows between the luminance values Lp1 to Lp7 and Lq1 to Lq7 indicate the time direction difference Δt between the pixel on the frame t and the pixel on the frame t + 1.

  Therefore, in the example of FIG. 30, the time direction difference Δt between the pixel p1 and the pixel q1 is represented by the difference between the luminance value Lp1 and the luminance value Lq1, and the time direction difference Δt between the pixel p2 and the pixel q2 is the luminance value Lp2. The time direction difference Δt between the pixel p3 and the pixel q3 is represented by the difference between the luminance value Lp3 and the luminance value Lq3, and the time direction difference Δt between the pixel pi and the pixel qi is represented by the luminance value Lpi. The time direction difference Δt between the pixel p7 and the pixel q7 is represented by the difference between the luminance value Lp7 and the luminance value Lq7, and each luminance value difference is a curve connecting the luminance values Lp1 to Lp7. And, the change in luminance is small so that the shape of the curve connecting the luminance values Lq1 to Lq7 is shown as substantially the same shape.

  As described above, when the pixels in the detection target block are the same object (that is, the same motion), the luminance change is not so large for the corresponding pixel in the time direction.

  FIG. 31 is a diagram illustrating the luminance state of a detection target block when pixels of objects having different movements are mixed. In FIG. 30, portions corresponding to those in FIG. 31 are denoted by the corresponding reference numerals, and the description thereof will be omitted to avoid repetition. However, in the example of FIG. 31, the pixel p7 on the frame t is , Has a different motion from other pixels.

  For example, when the pixel p7 on the frame t has the same movement as other pixels, the luminance value of the pixel q7 located in the same phase as the pixel p7 on the frame t + 1 is the luminance value Lq7-1. In the example of FIG. 31, the pixel p7 has a different motion from the other pixels. Therefore, the luminance value of the pixel q7 is represented by the luminance value Lq7-2. The luminance value Lq7-1 is greatly changed to the lower side in the figure. Therefore, the time direction difference Δt (the difference between the luminance value Lp7 and the luminance value Lq7-2) between the pixel p7 and the pixel q7 becomes large.

  Thus, for example, when pixels of an object having different movements are mixed in the detection target block due to the disappearance, generation, or deformation of the above-described object, the luminance value of the corresponding pixel greatly changes. This often leads to disturbances in movement. Therefore, in the effective pixel determination unit 403, a pixel with a small luminance change is effective even when used for motion vector detection, and conversely, a pixel with a large luminance change is determined to be ineffective when used for motion vector detection. Like that. That is, paying attention to the factor of the brightness change, and determining whether the pixel is valid or invalid by the brightness change, the pixel having a small brightness change is used for motion vector detection. Conversely, the brightness change is large. Pixels can be prevented from being used for motion vector detection. Thereby, the accuracy of motion vector detection can be increased.

  Specifically, the effective pixel determination unit 403 sets a predetermined pixel difference value (threshold value) for the pixel difference Δt in the time direction for each pixel in the calculation block, and the pixel difference Δt in the time direction is When the pixel difference value is smaller than the set predetermined pixel difference value, it is determined that the pixel contributes (is effective) to motion vector detection, and is used for the calculation of the gradient method when detecting the motion vector. If the pixel difference Δt in the direction is greater than or equal to the set predetermined pixel difference value, the pixel is excluded from the gradient method calculation when detecting a motion vector because there is a risk of disturbing motion. To do. Thereby, mixing of different motions is suppressed, and more stable gradient method calculation is executed. As a result, a probable motion vector is detected.

  However, as described above, since the gradient method calculation is based on the least square method, which is a statistical solution, it is used for the gradient method calculation by separating the pixels used for the gradient method calculation from the pixels not used. If the number of pixels is extremely small and the gradient method calculation is performed with these extremely small number of pixels, the reliability of the calculation is lowered, and the accuracy of motion vector detection may be reduced. is there.

  Therefore, when the number of pixels used for the gradient method calculation is small, the effective pixel determination unit 403 determines that the calculation in the processing unit of this detection target block (8 × 8) is unstable and is shown in FIG. As described above, the processing is performed by switching to the calculation unit for each pixel (64 pixels) in the 8 × 8 block.

  FIG. 32 shows an example of a calculation block in pixel unit processing. In the example of FIG. 32, the calculation block E of the pixel p at the upper left of the detection target block K when the detection target block (block-based calculation block) K (8 × 8) on the frame is switched to pixel-unit processing. (9 × 9 in the example of FIG. 32) is shown.

  That is, when the motion vector of the pixel p is obtained, a calculation block E (9 × 9 in the example of FIG. 32) centered on the pixel p is used. For example, when obtaining a motion vector of a pixel right next to the pixel p, an arithmetic block 9 × 9 (not shown) centered on the pixel right next to the pixel p is used.

  Therefore, for example, even if most of the detection target block K moves differently from the pixel p, in the detection target block K (8 × 8), the number of pixels used in the gradient method calculation is counted, and the number is When the number is small, the process is switched to the pixel unit process, and as shown in FIG. 32, the calculation of the gradient method can be executed using the calculation block E in the pixel unit process.

  As described above, when the pixels used for the gradient method calculation are separated from the pixels used for the gradient method calculation, and the number of pixels used for the gradient method calculation becomes extremely small, the motion vector for all the pixels in the detection target block K (8 × 8) block. Instead of stopping the detection of pixels, the gradient method calculation is performed by changing the pixel of the calculation block E (9 × 9) for each pixel, thereby mixing pixels that are not used in the calculation. As a result, motion vectors can be detected with higher accuracy depending on the pixel.

  FIG. 33 is a diagram illustrating an example in which pixel unit processing is applied to the pixels of the detection target block (arithmetic block) K in FIG. In the example of FIG. 33, the pixel has a pixel having a diagonally upper left movement indicated by arrow A (thick line circle) and a pixel having two different movements indicated by an arrow B (thin line circle). In the detection target block K (8 × 8) in FIG. 29, processing in units of pixels is performed. In this case, when detecting the motion vector of the detection target pixel p at the upper left of the detection target block K, the calculation block E (9 × 9 in the example of FIG. 33) centering on the detection target pixel p is used.

  The pixels in the computation block E have a diagonally upper left movement indicated by an arrow A except for the lower right three pixels (thin line circles) having a horizontal movement indicated by an arrow B. Here, as in the case of the block unit processing described above, for each pixel in the computation block E, a predetermined threshold value (pixel difference value) is set for the time direction pixel difference Δt, and the time direction pixel difference Δt is When the pixel difference value is smaller than the set predetermined pixel difference value, it is determined that the pixel contributes (is effective) to motion vector detection, and is used for gradient method calculation when detecting a motion vector. When the pixel difference Δt is larger than the set predetermined pixel difference value, the pixel is not used for the gradient method calculation when detecting a motion vector because it may cause a disturbance of motion.

  That is, in the example of FIG. 33, in the calculation block E, the pixels (thick line circles) excluding the lower right three pixels (thin line circles) are considered to be effective pixels for the gradient method calculation of the detection target pixel p. Since the effective pixel is used and the gradient method calculation is executed, the detection target block K (8 × 8) in FIG. 29 is more than the gradient method calculation of the block unit processing executed using the calculation block. A stable gradient method operation is performed and, as a result, a probable motion vector is detected.

  Even in the calculation block for each pixel, if the pixels used for the gradient method calculation are extremely reduced by separating the pixels used for the gradient method calculation from those not used, a motion vector is detected. The operation is aborted. That is, even if the motion vector detection is unstable, if a motion having some size is detected, the motion vector does not match the motion of the object in the actual image (not sure). Because there is a risk of becoming. Subsequent processing (for example, vector allocation processing or vector compensation processing) is executed after using such uncertain motion vectors, and the generated image may have motion discontinuities or block noise. In many cases, it causes a large visual deterioration such as the occurrence of the above.

  Accordingly, when the number of pixels used for pixel-based gradient method calculation becomes extremely small, for example, a zero vector (that is, a stationary state) is used as the detected motion vector. Thereby, in the subsequent processing, this motion vector is suppressed from having an adverse effect, and large visual deterioration of the generated image can be suppressed.

  Next, details of the iterative gradient method calculation processing will be described with reference to the flowchart of FIG. The initial vector V0 is input to the mode selection unit 401 from the previous stage.

  In step S401, the mode selection unit 401 selects a block unit processing mode, sets the block on the frame t as a detection target block (calculation block), and sets the initial vector V0 of the detection target block to the selector 402 and the vector evaluation unit 104. Output, and proceed to step S402. As a result, each unit of the iterative gradient method computing unit 103 executes processing in units of blocks. In step S402, the selector 402 selects the initial vector V0 input from the mode selection unit 401 as an offset vector in accordance with control from the vector evaluation unit 104, and selects the selected offset vector as a time pixel difference calculation unit 411 and The result is output to the time pixel difference calculation unit 421, and the process proceeds to step S403.

  In step S403, the temporal pixel difference calculation unit 411 and the pixel difference value determination unit 412 execute the effective pixel determination process in units of blocks using the selected offset vector (in this case, the initial vector V0 is offset). Then, the process proceeds to step S404. The details of this effective pixel determination process will be described later with reference to FIG. 35. With this effective pixel determination process, the effective pixel number counter 413 has a pixel difference value of a predetermined pixel difference within a block-by-block calculation block. The number of pixels determined to be smaller than the value (that is, determined to be an effective pixel in the subsequent gradient method calculation) is counted.

  In step S404, the gradient method continuation determination unit 414 determines whether or not the number of pixels counted by the effective pixel number counter 413 (effective pixel number) is greater than a predetermined threshold value α, and the effective pixel number is predetermined. If it is determined that the value is larger than the threshold value α, a flag (flg = 1) for executing the gradient method calculation is output to the gradient method calculation unit 404 for each block, and the process proceeds to step S405.

  In step S405, the gradient method calculation unit 404 uses the offset vector from the selector 402 (in this case, the initial vector V0) based on the flag from the gradient method continuation determination unit 414, and performs gradient method calculation processing in units of blocks. Is executed, and the process proceeds to step S406. The details of the gradient method calculation processing will be described later with reference to FIG. 36, but the motion vector Vn is obtained by this gradient method calculation processing and is output to the vector evaluation unit 104 and the delay unit 405. The motion vector Vn is obtained by adding the offset vector Vn−1 to the motion vector vn calculated by the first gradient method calculation (Vn = Vn−1 + vn). For example, the motion vector V1 is obtained by adding an offset vector (initial vector V0) to the motion vector v1 calculated by the first gradient method calculation (V1 = V0 + v1).

  In step S406, the vector evaluation unit 104 evaluates the motion vector Vn obtained by the gradient method calculation process DFD (n) and the evaluation value DFD (n-1) of the motion vector Vn-1 used as the offset vector. , And whether or not the evaluation value DFD (n) is smaller than the evaluation value DFD (n−1), that is, the motion vector obtained by the gradient method calculation processing than the motion vector Vn−1 used as the offset vector It is determined whether or not the reliability of Vn is high. Specifically, for example, in the first process, the obtained evaluation value DFD (1) of the motion vector V1 is compared with the evaluation value DFD (0) of the initial vector V0 used as an offset vector. In the second processing, the obtained evaluation value DFD (2) of the motion vector V2 is compared with the evaluation value DFD (1) of the motion vector V1 used as the offset vector.

  When the vector evaluation unit 104 determines in step S406 that the evaluation value DFD (n) is smaller than the evaluation value DFD (n-1), that is, the reliability of the motion vector Vn is higher than the motion vector Vn-1. In step S407, the calculated motion vector Vn (in the first case, the motion vector V1, in the second case, the motion vector V2) is set as the motion vector V of the detection target block. The process proceeds to S408. In step S408, the vector evaluation unit 104 counts the number of iterations n by one, proceeds to step S409, determines whether or not the number of iterations n has reached the set maximum number of iterations (for example, two times), and If it is determined that the number of iterations n is not the set maximum number of iterations, the selector 402 is controlled to return to step S402 to repeat the subsequent processing.

  That is, in step S402, the selector 402 selects the motion vector V1 output from the vector calculation unit 425 and held in the delay unit 405 as an offset vector, and in step S403, the time pixel difference calculation unit 411 and the pixel difference value are selected. The determination unit 412 executes the effective pixel determination process in units of blocks using the motion vector V1 as an offset vector, and repeats the subsequent processes. Therefore, the motion vector V2 calculated in step S405 is obtained by adding the motion vector V1 used as the offset vector to the motion vector v2 calculated by the second gradient method calculation (that is, V2 = V1 + v2 = V0 + v1 + v2). )

  When the vector evaluation unit 104 determines in step S406 that the evaluation value DFD (n) is not smaller than the evaluation value DFD (n−1), that is, the motion vector Vn−1 is more than the motion vector Vn. If it is determined that the reliability is high, the process proceeds to step S410, and the vector Vn−1 (the initial vector V0 in the first time, the motion vector V1 in the second time) is used as the offset of the gradient method calculation. The motion vector V is set and the process proceeds to step S411. If the vector evaluation unit 104 determines in step S409 that the number of iterations n has reached the set maximum number of iterations, the vector evaluation unit 104 proceeds to step S411.

  In step S411, the vector evaluation unit 104 determines whether or not the motion vector V is within a search area preset as a range for detecting a motion vector, and determines that the motion vector V is within the search area. If so, the process proceeds to step S415, the motion vector V is stored in the detection vector memory 53 in association with the detection target block, and the iterative gradient method process is terminated.

  If the vector evaluation unit 104 determines in step S411 that the motion vector V is not within the search area, the vector evaluation unit 104 proceeds to step S412, sets the motion vector V to 0 vector, proceeds to step S415, and proceeds to step S415. (0 vector) is stored in the detection vector memory 53 in association with the detection target block, and the iterative gradient method processing is terminated.

  On the other hand, if the gradient method continuation determination unit 414 determines in step S404 that the effective pixel number of the effective pixel number counter 413 is equal to or less than the predetermined threshold value α, the process proceeds to step S413 and controls the mode selection unit 401. The pixel unit processing mode is selected, and the process proceeds to step S414. As a result, each unit of the iterative gradient method computing unit 103 executes processing in units of pixels. In step S414, the mode selection unit 401 executes the iterative gradient method calculation process for each pixel, and the process proceeds to step S415. The details of the iterative gradient method computing process in the pixel unit processing mode will be described later with reference to FIG. 37, but the motion vector V of all pixels in the detection target block is obtained by the iterative gradient method computing process in the pixel unit processing mode. Therefore, in step S415, the vector evaluation unit 104 stores the motion vector V in the detection vector memory 53 in association with each pixel of the detection target block, and ends the iterative gradient method processing.

  In step S406 described above, the reliability of the motion vector is evaluated by evaluating the evaluation value DFD (n) of the motion vector Vn obtained by the gradient method processing and the motion vector Vn-1 used as the offset vector. Although the determination is made by comparing the value DFD (n−1), simultaneously with the comparison determination of the evaluation value DFD, the motion vector (motion vector before adding the offset vector) vn obtained by the gradient method calculation It may be determined that the value (| vn |) is smaller than a predetermined size (for example, 16 in the case of the calculation block 8 × 8).

  Here, the norm represents the magnitude of a vector in a certain vector space. For example, the norm of the vector v from the origin (0, 0) to (x, y) is expressed by the following equation (11). expressed.

・・・（１１）

(11)

  That is, in the gradient method calculation, a high-precision result is generally obtained for a minute motion, but the norm value | vn | of the motion vector obtained by the gradient method calculation exceeds a predetermined magnitude. In other words, a highly accurate result is not always obtained for the large motion vector vn. Therefore, in parallel with the evaluation value DFD, it is determined whether or not the norm value | vn | of the motion vector obtained by the gradient method calculation exceeds a predetermined magnitude, and the norm value | vn | By excluding motion vectors that exceed a predetermined size, in this gradient method computation, it is possible to obtain highly accurate results for both small and large motions. The accuracy of detection can be improved.

  In the case of the determination of the evaluation value DFD and the determination based on the norm value, in step S406, when | vn | ≦ 16, and evaluation value DFD (n−1)> evaluation value DFD (n). If there is, the process proceeds to step S407, where V = Vn is set, and if | vn |> 16 in step S406, or evaluation value DFD (n−1) ≦ evaluation value DFD (n). In this case, the process proceeds to step S410, and V = Vn−1 is set.

  Next, the details of the effective pixel determination process will be described with reference to the flowchart of FIG. FIG. 35 shows an example of the effective pixel determination process in step S403 of FIG.

  When the offset vector is input from the selector 402, the temporal pixel difference calculation unit 411 controls the pixel difference value determination unit 412 in step S431, resets the effective pixel number counter 413, proceeds to step S432, and proceeds to block unit. In the calculation block, a pixel is selected, and the process proceeds to step S433. The pixels are selected in the raster scan order from the upper left pixel of the calculation block.

  In step S433, the time pixel difference calculation unit 411 uses the offset vector and the frame t of the image at time t and the frame t + 1 of the image at time t + 1 input via the pre-filters 102-1 and 102-2. Then, the pixel difference Δt in the time direction of the selected pixel is calculated, and the calculated pixel difference Δt in the time direction is output to the pixel difference value determination unit 412 and the process proceeds to step S434.

  In step S434, the pixel difference value determination unit 412 determines whether or not the time direction pixel difference Δt calculated by the time pixel difference calculation unit 411 is smaller than a predetermined pixel difference value. It is determined whether or not the pixel is effective for the legal calculation, and if it is determined that the pixel difference Δt in the time direction is smaller than the predetermined pixel difference value, the process proceeds to step S435, and the effective pixel counter 413 determines the effective pixel number. The number is counted by one, and the process proceeds to step S436. Further, the pixel difference value determination unit 412 determines in step S434 that the time-direction pixel difference Δt is greater than or equal to a predetermined pixel difference value, that is, it is not a pixel that is effective for the subsequent gradient method calculation. Then, the process of step S435 is skipped, and the process proceeds to step S436.

  In step S436, the time pixel difference calculation unit 411 determines whether or not the processing of all the pixels in the calculation block has been completed, and determines that the processing of all the pixels in the calculation block has not ended. Returning to step S432, the next pixel is selected, and the subsequent processing is repeated. On the other hand, if the time pixel difference calculation unit 411 determines in step S436 that all the pixels in the calculation block have been processed, the time pixel difference calculation unit 411 ends the effective pixel determination process and returns to step S404 in FIG.

  Through the above processing, the effective pixel counter 413 counts the number of pixels in the calculation block whose pixel difference Δt is smaller than a predetermined pixel difference value and determined to be effective in the subsequent gradient method calculation. Yes. Therefore, the gradient method continuation determination unit 414 performs the gradient method calculation in units of blocks or performs the gradient method calculation in units of pixels by referring to the number of pixels of the effective pixel counter 413 in step S404 of FIG. Can be determined.

  Next, the details of the gradient method calculation processing in units of blocks will be described with reference to the flowchart of FIG. Note that FIG. 36 illustrates an example of the gradient method calculation processing in units of blocks in step S405 of FIG. Therefore, in the example of FIG. 36, the flag (flg = 1) for executing the gradient method calculation is input from the gradient method continuation determination unit 414 to the time pixel difference calculation unit 421.

  When the flag (flg = 1) is input from the gradient method continuation determination unit 414, the time pixel difference calculation unit 421 selects a pixel in the calculation unit in block units in step S451, and proceeds to step S452. The pixels are selected in the raster scan order from the upper left pixel of the calculation block.

  In step S452, the time pixel difference calculation unit 421 calculates the offset vector from the selector 402, the frame t of the image at time t input via the prefilters 102-1 and 102-2, and the image at time t + 1. The pixel difference Δt in the time direction of the selected pixel is calculated using the frame t + 1, and the calculated pixel difference Δt in the time direction and the offset vector are output to the pixel difference value determination unit 422, and the process proceeds to step S453.

  In step S453, the pixel difference value determination unit 422 determines whether or not the pixel difference Δt in the time direction calculated by the time pixel difference calculation unit 421 is smaller than a predetermined pixel difference value. It is determined whether or not the pixel is effective for legal calculation, and if it is determined that the pixel difference Δt in the time direction is smaller than the predetermined pixel difference value, the process proceeds to step S454 and the vertical horizontal pixel difference calculation unit 423 is controlled. Then, the pixel difference Δx in the horizontal direction of the pixel determined to be effective is calculated. Note that the predetermined pixel difference value in the pixel difference value determination unit 422 may be the same value as or different from the predetermined pixel difference value in the pixel difference value determination unit 412. At this time, the pixel difference value determination unit 422 outputs the pixel difference Δt in the time direction and the offset vector calculated by the time pixel difference calculation unit 421 to the vertical horizontal pixel difference calculation unit 423.

  In step S454, the vertical horizontal pixel difference calculation unit 423 calculates the offset vector and the frame t of the image at time t and the frame t + 1 of the image at time t + 1 input via the prefilters 102-1 and 102-2. The pixel difference Δx in the horizontal direction of the pixel determined to be effective is calculated, and the process proceeds to step S455, where the pixel difference Δy in the vertical direction of the pixel determined to be effective is calculated, and the pixel in the time direction is calculated. The difference Δt, the pixel difference Δx in the horizontal direction, and the pixel difference Δy in the vertical direction are output to the gradient integration unit 424, and the process proceeds to step S456. In step S456, the gradient integration unit 424 integrates the time-direction pixel difference Δt, the horizontal pixel difference Δx, and the vertical pixel difference Δy from the vertical horizontal pixel difference calculation unit 423, and calculates a vector result of the integration. The data is output to the unit 425, and the process proceeds to step S457.

  If the pixel difference value determination unit 422 determines in step S453 that the time-direction pixel difference Δt is greater than or equal to the predetermined pixel difference value, the pixel difference value determination unit 422 skips the processing of steps S454 to S456 and proceeds to step S457. That is, each pixel difference (gradient) of this pixel is not used for the calculation because it is not a pixel effective for the subsequent gradient method calculation.

  In step S457, the time pixel difference calculation unit 421 determines whether or not the processing of all the pixels in the calculation block has been completed, and determines that the processing of all the pixels in the calculation block has not ended. Returning to step S451, the next pixel is selected, and the subsequent processing is repeated. On the other hand, if the time pixel difference calculation unit 421 determines in step S457 that the processing of all the pixels in the calculation block has been completed, the time pixel difference calculation unit 421 outputs an offset vector to the vector calculation unit 425, and proceeds to step S458. When the offset vector is input from the time pixel difference calculation unit 421, the vector calculation unit 425 uses the gradient summation result from the gradient summation unit 424 and the least square sum of the above-described equation (9) to obtain the motion vector vn. Then, the process proceeds to step S459, the motion vector Vn is obtained by adding the calculated motion vector vn to the offset vector from the time pixel difference calculation unit 421, and the obtained motion vector Vn is sent to the vector evaluation unit 104. The gradient method calculation process is terminated, and the process returns to step S406 in FIG.

  As described above, when the pixel difference Δt in the time direction is equal to or larger than a predetermined pixel difference value, the gradient at the time of detecting a motion vector is assumed that the pixel may cause a disturbance in motion. Since it is excluded from the calculation of the method, a more stable gradient method calculation is executed, and as a result, a probable motion vector is detected, thereby improving the accuracy of the motion vector detection.

  Next, details of the iterative gradient method calculation processing in units of pixels will be described with reference to the flowchart of FIG. FIG. 37 shows an example of the iterative gradient method computing process for each pixel in step S414 of FIG. This process is executed for each pixel in the detection target block.

  When the pixel unit processing mode is set, the mode selection unit 401 selects the pixel of the detection target block as the detection target pixel in step S471, and selects the initial vector V0 of the calculation block (for example, 9 × 9) of the pixel as a selector. 402 and the vector evaluation unit 104, and the process proceeds to step S472. The pixels are selected in the raster scan order from the upper left pixel of the detection target block. In step S472, the selector 402 selects the initial vector V0 input from the mode selection unit 401 as an offset vector in accordance with the control from the vector evaluation unit 104, and selects the selected offset vector as a time pixel difference calculation unit 411 and The result is output to the time pixel difference calculation unit 421, and the process proceeds to step S473.

  In step S473, the temporal pixel difference calculation unit 411 and the pixel difference value determination unit 412 execute the effective pixel determination process in units of pixels using the selected offset vector (in this case, using the initial vector V0 as an offset). The process proceeds to step S474. The details of the effective pixel determination process are basically the same except that the effective pixel determination process in units of blocks described above with reference to FIG. 35 is different from the target calculation block (block range and pixels constituting the block). However, by this effective pixel determination process, the effective pixel number counter 413 has a pixel difference value smaller than a predetermined pixel difference value in the calculation unit in units of pixels. (That is, the number of pixels determined to be effective pixels in the subsequent gradient method calculation) is counted. It should be noted that the predetermined pixel difference value for determining the effective pixel in pixel units may be the same value as in the block unit or may be set to a different value.

  In step S474, the gradient method continuation determination unit 414 determines whether or not the number of pixels (effective pixel number) counted by the effective pixel number counter 413 is greater than a predetermined threshold value β, and the effective pixel number is predetermined. If it is determined that the value is larger than the threshold value β, a flag (flg = 1) for executing the gradient method calculation is output to the gradient method calculation unit 404, and the process proceeds to step S475. Note that the threshold value β may be the same value as the threshold value α in the case of a block unit, or a different value may be set.

  In step S475, based on the flag from the gradient method continuation determination unit 414, the gradient method calculation unit 404 uses the offset vector (initial vector V0) from the selector 402 to execute a gradient method calculation process for each pixel, The process proceeds to step S476. The details of the gradient method calculation process are basically the same as the gradient method calculation process in units of blocks described above with reference to FIG. 36 except that the target calculation block (range and pixels constituting the block) is different. Although detailed description thereof is omitted for the purpose of processing, a motion vector Vn of the detection target pixel is obtained by this gradient method calculation processing, and is output to the vector evaluation unit 104 and the delay unit 405. The motion vector Vn is obtained by adding the offset vector to the motion vector vn obtained by the first gradient method calculation (Vn = Vn−1 + vn).

  In step S476, the vector evaluation unit 104 obtains the evaluation value DFD (n) of the motion vector Vn calculated by the gradient method processing and the evaluation value DFD (n−1) of the motion vector Vn−1 used as the offset. , Whether the evaluation value DFD (n) is smaller than the evaluation value DFD (n−1), that is, the reliability of the motion vector Vn calculated by the gradient method processing than the motion vector Vn−1 used as the offset Judge whether or not is high.

  In step S476, the vector evaluation unit 104 determines that the evaluation value DFD (n) is smaller than the evaluation value DFD (n−1), that is, the reliability of the motion vector Vn is higher than the motion vector Vn−1. If it is determined that the value is high, in step S477, the calculated motion vector Vn (the first time, the motion vector V1, and the second time, the motion vector V2) is set as the motion vector V of the detection target pixel, The process proceeds to step S478. In step S478, the vector evaluation unit 104 counts the number of iterations n by one, proceeds to step S479, determines whether or not the number of iterations n has reached the set maximum number of iterations (for example, two times), and If it is determined that the number of iterations n is not the set maximum number of iterations, the selector 402 is controlled to return to step S472 to repeat the subsequent processing.

  That is, in step S472, the selector 402 selects the motion vector V1 output from the vector calculation unit 425 and held in the delay unit 405 as an offset vector, and in step S473, the temporal pixel difference calculation unit 411 and the pixel difference value determination The unit 412 offsets the motion vector V1, which is an offset vector, executes a valid pixel determination process in units of pixels, and repeats the subsequent processes. Therefore, the motion vector V2 obtained in step S475 is obtained by adding the offset motion vector V1 to the motion vector v2 calculated by the second gradient method calculation (that is, V2 = V0 + v1 + v2).

  In step S476, the vector evaluation unit 104 determines that the evaluation value DFD (n) is larger than the evaluation value DFD (n−1), that is, the motion vector Vn−1 is more reliable than the motion vector Vn. When it is determined that the degree is high, the process proceeds to step S480, and the vector Vn−1 (the initial vector V0 in the first case, the motion vector V1 in the second time) is used as the offset of the gradient method calculation. The motion vector V is set, and the process proceeds to step S481. If the vector evaluation unit 104 determines in step S479 that the number of iterations n has reached the set maximum number of iterations, the vector evaluation unit 104 proceeds to step S481.

  In step S481, the vector evaluation unit 104 determines whether or not the motion vector V is within a search area preset as a range for detecting a motion vector, and determines that the motion vector V is within the search area. If so, the process proceeds to step S483. If the vector evaluation unit 104 determines in step S481 that the motion vector V is not within the search area, the vector evaluation unit 104 proceeds to step S482, sets the motion vector V to 0 vector, and proceeds to step S483.

  On the other hand, in step S474, when the gradient method continuation determination unit 414 determines that the effective pixel number of the effective pixel number counter 413 is less than the predetermined threshold value β, the gradient method calculation unit 404 has a flag ( flg = 0) is output, and the flow advances to step S482. Based on the flag, the temporal pixel difference calculation unit 421 controls the vector calculation unit 425 to set the motion vector V to 0 vector, and the process proceeds to step S483.

  When the mode selection unit 401 determines in step S483 whether or not the processing of all the pixels in the detection target block has been completed, and determines that the processing of all the pixels in the detection target block has not been completed, Returning to step S471, the subsequent pixels of the detection target block are set as detection target pixels, and the subsequent processing is repeated. If the mode selection unit 401 determines in step S483 that the processing of all the pixels in the detection target block has been completed, the mode selection unit 401 ends the iterative gradient method calculation processing for each pixel, and returns to step S415 in FIG. That is, since the motion vector V is obtained for all the pixels in the detection target block by the iterative gradient method computing process in units of pixels, the process returns to step S415 in FIG. 34, and the vector evaluation unit 104 converts the motion vector V to the detection target block. Are stored in the detection vector memory 53 in correspondence with each of the pixels, and the iterative gradient method processing is terminated.

  Note that, in the evaluation of the reliability of the motion vector in step S476, the evaluation value DFD determination and the norm value determination may be used in combination, as in the case of step S406.

  As described above, by using only pixels whose pixel difference in the calculation block is smaller than the predetermined pixel difference value for the gradient method calculation, pixels having different motions can be excluded from the calculation target, and the gradient method calculation is performed. As a result, the certainty of the motion vector required by the method improves, and the accuracy of motion vector detection improves.

  In addition, when the pixel difference in the calculation block is larger than the predetermined pixel difference value and the number of pixels that are not used for the gradient method calculation is larger than the predetermined threshold value, the gradient method calculation processing is adaptively performed. Therefore, the accuracy of motion vector detection can be further improved especially at the boundary of the motion of the object, and as a result, the required motion vector can be made more probable.

  Specifically, when the pixel difference in the calculation block is larger than the predetermined pixel difference value and the number of pixels that are not used in the gradient method calculation is larger than the predetermined threshold value, the block unit mode is changed to the pixel unit. Since the calculation block is set for each pixel of the detection target block and the validity determination process and the gradient method calculation process are performed, the motion vector can be detected with higher accuracy depending on the pixel. In addition, when the pixel difference in the calculation block is larger than the predetermined pixel difference value and the number of pixels that are not used for the gradient method calculation is larger than the predetermined threshold value, the gradient method calculation processing is discontinued, Since the zero vector is used, detection of a violent motion vector can be suppressed and a safer motion vector can be detected.

  Next, details of the configuration of the vector allocation unit 54 will be described.

  FIG. 38 is a block diagram showing a configuration of the vector allocating unit 54. The vector allocating unit 54 having the configuration shown in FIG. 38 allocates motion vectors detected on the frame t using the frame t of the image at time t to which the 24P signal is input and the frame t + 1 of the image at time t + 1. A process of assigning to the pixels on the interpolation frame of the 60P signal to be interpolated on the vector memory 55 is performed.

  In the example of FIG. 38, the frame t of the image at time t and the frame t + 1 of the image at time t + 1 are input to the pixel information calculation unit 701, the evaluation value calculation unit 702, and the target pixel difference calculation unit 703.

  The pixel information calculation unit 701 acquires the motion vector detected by the pixel on the frame t in the detection vector memory 53 in the raster scan order from the upper left pixel, and extends the acquired motion vector in the frame t + 1 direction at the next time. The intersection of the extended motion vector and the interpolation frame is calculated. Then, the pixel information calculation unit 701 sets, from the intersection of the calculated motion vector and the interpolation frame, a pixel (hereinafter referred to as an allocation target pixel) to which the motion vector is assigned on the interpolation frame. The motion vector and the position information of the allocation target pixel are output to the vector selection unit 705. The image information calculation unit 701 calculates the position P of the frame t and the position Q on the frame t + 1 that are associated with the allocation target pixel and the motion vector, and the calculated position information on the frame t and the frame t + 1 The result is output to the evaluation value calculation unit 702 and the target pixel difference calculation unit 703.

  When the evaluation value calculation unit 702 receives from the pixel information calculation unit 701 position information on the frame t and the frame t + 1 associated with the allocation target pixel and the motion vector, the evaluation value calculation unit 702 receives the position P of the frame t and the position Q of the frame t + 1. In order to calculate the evaluation value DFD, a certain range of DFD calculation ranges (m × n) centering on the position P and the position Q are respectively set, and it is determined whether or not these DFD calculation ranges are within the image frame. . When the evaluation value calculation unit 702 determines that the DFD calculation range is within the image frame, the evaluation value calculation unit 702 calculates the evaluation value DFD of the allocation target pixel for the motion vector by calculating using the DFD calculation range, and the calculated evaluation value The DFD is output to the vector evaluation unit 704.

  When the pixel-of-interest difference calculation unit 703 receives from the pixel information calculation unit 701 position information on the frame t and the frame t + 1 associated with the allocation target pixel and the motion vector, the position P of the frame t and the position Q of the frame t + 1 Is used to obtain the absolute value of the luminance difference for the allocation target pixel, and the obtained absolute value of the luminance difference is output to the vector evaluation unit 704.

  The vector evaluation unit 704 includes a pixel difference determination unit 711 and an evaluation value determination unit 712. The pixel difference determination unit 711 determines whether the luminance difference absolute value for the allocation target pixel input from the target pixel difference calculation unit 703 is smaller than a predetermined threshold value. The evaluation value determination unit 712 calculates the evaluation value when the pixel difference determination unit 711 determines that the luminance difference absolute value for the allocation target pixel input from the target pixel difference calculation unit 703 is smaller than a predetermined threshold value. It is determined whether the evaluation value DFD of the allocation target pixel input from the unit 702 is smaller than the minimum evaluation value of the DFD table included in the vector selection unit 705. When the evaluation value determination unit 712 determines that the evaluation value DFD of the allocation target pixel is smaller than the minimum evaluation value of the DFD table, the evaluation value determination unit 712 determines that the reliability of the motion vector corresponding to the allocation target pixel is high, The evaluation value DFD of the allocation target pixel is output to the selection unit 705.

  The vector selection unit 705 has a DFD table that holds a minimum evaluation value at each pixel on the interpolation frame, and an evaluation value DFD0 when a 0 vector is assigned to each pixel on the interpolation frame. The DFD table holds the minimum evaluation value for each pixel on the interpolation frame in advance. When the vector selection unit 705 receives the evaluation value DFD of the allocation target pixel from the vector evaluation unit 704, the vector selection unit 705 sets the flag of the allocation flag memory 56 to 1 (based on the position information of the allocation target pixel from the pixel information calculation unit 701). true), and the minimum evaluation value in the DFD table of the allocation target pixel is rewritten to the evaluation value DFD of the allocation target pixel. Further, the vector selection unit 705 allocates the motion vector from the pixel information calculation unit 701 to the allocation target pixel in the allocation vector memory 55 based on the information on the position of the allocation target pixel from the pixel information calculation unit 701.

  Next, the accuracy below the pixel of the motion vector will be described. In the calculation of the evaluation value DFD represented by the above equation (1), the phase p + v on the previous frame t + 1 obtained by shifting the pixel position p of the frame t by the amount of the vector v is actually the frame t + 1 of the 24p signal. In many cases, it does not coincide with the upper pixel position, and the luminance value in that case is not defined. Therefore, in order to perform the calculation of the evaluation value DFD for the motion vector v having subpixel accuracy, it is necessary to generate a luminance value in a phase below the pixel by some method.

  Corresponding to this, there is a method of using the luminance value of the pixel closest to the phase p + v on the previous frame t + 1 obtained by shifting the pixel position p of the frame t by the vector v amount. However, in this method, since the sub-pixel components of the motion vector to be evaluated are rounded, the sub-pixel components of the motion vector are discarded, and the reliability of the evaluation value DFD thus obtained is lowered. End up.

  Therefore, in the present invention, a four-point interpolation process using the luminance values of the surrounding four pixels is used. FIG. 39 is a diagram showing the concept of the four-point interpolation process of the present invention. In FIG. 39, the arrow X indicates the horizontal direction in the frame t + 1, and the arrow Y indicates the vertical direction in the frame t + 1. In this frame t + 1, the white circle represents the pixel position on the frame t + 1, and the black dot represents the position below the pixel. Further, the upper left black point p + v and its surrounding four pixels on the frame t + 1 are shown enlarged in the window E. In window E, the alphabet in the white circle indicates the luminance value of the surrounding four pixels.

If the black point p + v at the upper left in the frame t + 1 is the phase p + v after the pixel position p of the frame t is shifted by the vector v amount, the luminance value F _{t + 1} (p + v) of the phase p + v is the horizontal direction of the phase p + v. The pixel sub-component α and the pixel sub-component β in the vertical direction and the luminance values L0 to L4 of the four surrounding pixels of the phase p + v are used to obtain the sum of the inverse ratios of the distances of the four surrounding pixels. That is, the luminance value F _{t + 1} (p + v) is expressed by the following equation (12).

F _{t + 1} (p + v) = (1-α) (1-β) L0 + α (1-β) L1
+ (1-α) βL2 + αβL3 (12)

As described above, the calculation of the DFD evaluation is performed using the luminance value F _{t + 1} (p + v) obtained by the four-point interpolation process, so that the reliability of the evaluation value DFD can be increased without increasing the hardware implementation cost. A decrease in the degree can be suppressed. In the following, an example in which the four-point interpolation is applied in the calculation of the evaluation value DFD and the luminance difference absolute value at the time of vector allocation will be described. Of course, the above-described initial vector selection processing, vector detection processing, Alternatively, this four-point interpolation is also applied to the calculation of the evaluation value DFD in the case of evaluating a vector such as allocation compensation processing described later.

  Next, the concept of vector allocation processing will be described. For example, when it is assumed that an object is moving at a velocity v on a 24P signal and a constant velocity assumption is made for the motion of the object between any two frames, a new frame is interpolated between the frames of the 24P signal. Think about what to do. In this case, when the motion vector v is extended from the 24P signal object, the intersection of the motion vector v and the interpolation frame is the same object and has the same velocity v.

  Therefore, the motion vector of the frame of the 24P signal detected by the vector detection unit 53 (hereinafter also referred to as an original frame with respect to the interpolation frame) is represented by the motion vector and the interpolation frame of the 60P signal to be interpolated. By assigning them to the intersections, the movement of each pixel on the interpolation frame can be obtained. Conversely, from the allocated motion vector, it can be determined from which position the pixel on the interpolation frame has moved on the 24P signal frame.

  FIG. 40 shows one-dimensional examples of motion vectors detected in the original frame of the 24P signal and pixels on the interpolation frame of the 60P signal. In the example of FIG. 40, between the frame t at the time t and the frame t + 1 at the time t + 1 of the two 24P signals, for example, the interpolation frame F1 at the time t + 0.4 and the interpolation frame F2 at the time t + 0.8 are Two are inserted. As described above, the position of this interpolation frame is the position where the time phase on the 24P signal is 0.0, 0.4, 0.8, 1.2, and 1.6, and the 60P signal. Are set in advance in the signal processing apparatus 1.

  A circle on each frame indicates each pixel. Motion vectors v1, v2, and v3 detected in the frame t by the preceding vector detection unit 52 are extended in the frame t + 1 direction. When these motion vectors are allocated to each pixel of the interpolation frames F1 and F2, the motion vector passing through the vicinity of each pixel on the interpolation frame is a candidate vector (hereinafter also referred to as allocation candidate vector) allocated to that pixel. Is done.

  Accordingly, the motion vector v1 from the pixel on the left side of the frame t to the vicinity of the fourth and fifth pixels from the left of the frame t + 1 is between the second and third pixels from the left on the interpolation frame F1. It passes between the third and fourth pixels from the left on F2. Therefore, the motion vector v1 is a pixel included in the vicinity N1 of the point where the motion vector v1 intersects the interpolation frames F1 and F2 (second and third pixels from the left of the interpolation frame F1 and the left of the interpolation frame F2). To the third and fourth pixels).

  The motion vector v2 from the third pixel from the left of the frame t to the vicinity of the second and third pixels from the left of the frame t + 1 is between the second and third pixels from the left on the interpolation frame F1, It passes between the second and third pixels from the left on the interpolation frame F2. Therefore, the motion vector v2 is a pixel included in the vicinity region N2 of the point where the motion vector v2 intersects the interpolation frames F1 and F2 (second and third pixels from the left of the interpolation frame F1 and the interpolation frame F2). Allocation candidate vectors to be allocated to the second and third pixels from the left).

  Furthermore, the motion vector v3 from the fifth pixel from the left of the frame t to the vicinity of the fourth and fifth pixels from the left of the frame t + 1 is between the fourth and fifth pixels from the left on the interpolation frame F1, It passes between the fourth and fifth pixels from the left on the interpolation frame F2. Therefore, the motion vector v3 includes the pixels included in the vicinity region N3 of the point where the motion vector v3 intersects the interpolation frames F1 and F2 (the fourth and fifth pixels from the left of the interpolation frame F1 and the interpolation frame F2). 4th and 5th pixels from the left).

  That is, the allocation candidate vector of the second pixel from the left of the interpolation frame F2 is the motion vector v2, and the second and third pixels from the left on the interpolation frame F1, and 3 from the left of the interpolation frame F2. The allocation candidate vectors of the th pixel are motion vectors v1 and v2. Further, the allocation candidate vectors of the fourth pixel from the left of the interpolation frame F2 are motion vectors v1 and v3, the fourth and fifth pixels from the left on the interpolation frame F1, and the left of the interpolation frame F2. The fifth pixel allocation candidate vector from is the motion vector v3.

  As described above, an allocation candidate vector to be allocated to each pixel on the interpolation frame is obtained from the motion vectors detected in the original frame. In the leftmost pixel and the rightmost pixel (black circles in the figure) of the interpolation frames F1 and F2, motion vectors passing through the vicinity are not shown. That is, there is no allocation candidate vector to be allocated to the leftmost pixel and the rightmost pixel of the interpolation frames F1 and F2. Therefore, an allocation compensation process is performed on these pixels in an allocation compensation unit 57 in a later stage described later.

Furthermore, with reference to FIG. 41, the motion vector detected in the original frame and the pixels on the interpolation frame of the 60P signal will be described in detail. In the example of FIG. 41, the arrow T indicates the direction of time passage from the frame t at the time t at the left front in the figure to the frame t + 1 at the time t + 1 at the back right. Further, interpolation frame F1 is placed at time t + pos _t between times t and time t + 1.

In the example of FIG. 41, the motion vector v _a (x _va , y _va ) detected by the pixel (x _a , y _a ) on the frame t is extended in the frame t + 1 direction, and the intersection (x _ia , y _ia ) is calculated. Intersection, it is considered that pixels corresponding to end points of the motion vector v _a of the frame t of the 24P signal to a point moved, specifically, is expressed by the equation (13) and (14).

_{x ia = x a + pos t} x va ··· (13)
_{y ia = y a + pos t} y va ··· (14)

Here, as described above, when the motion vector v _a has a less precision pixels, the intersection of the motion vector v _a, pixel positions on the interpolation frame F1 does not always match. Otherwise, as shown in FIG. 41, the motion vector v _a is allocated with respect to the four neighboring pixels G1 to G4 of the intersection on the interpolated frame F1. That is, the motion vector v _a is shifted (parallel movement) on each of the pixels G1 to G4 in the neighborhood, is the allocation candidate vector to be allocated to each of the pixels, the allocation compensating process is executed.

  As described above, since one motion vector may be a candidate assigned to four neighboring pixels, a plurality of motion vectors may be assigned candidate vectors depending on the pixel. In this case, the vector allocating unit 54 calculates, for each motion vector, a pixel on the interpolation frame and an intersection point on the original frame associated with the motion vector, and evaluates each motion vector using the intersection point. Thus, finally, a motion vector to be assigned to the pixel on the interpolation frame is determined.

With reference to FIG. 42, the motion vector evaluation in the vector allocation unit 54 will be described. Figure 42 is a bottom, frame t at time t in FIG. 41, the time t + pos _t interpolated frame F1, and the frame t + 1 at time t + 1, is one-dimensionally illustrated.

In the example of FIG. 42, the motion vector sv _a is the pixel (x _a, y _a) of the frame t detected motion vector v _a at is as allocation candidate vectors of the pixels in the neighborhood G4, shifts on the pixel G4 ( Translated). Here, the intersections of the motion vector sva, shifted onto the pixel G4, and the frame t and the frame t + 1 are defined as _a point P and a point Q, respectively.

The vector allocating unit 54, a first evaluation of motion vector sv _a, first, obtains a DFD operation range around the point P and point Q respectively, DFD operation range determined whether jutting out the image frame Judging. Therefore, when the DFD operation range around the point P and point Q had protruding a picture frame, the motion vector sv _a is excluded from the candidates.

Further, when the point P and the point Q belong to different objects, for example, the difference between the luminance value F _t (P) of the point P and the luminance value F _{t + 1} (Q) of the point Q becomes large. . Accordingly, the vector allocating unit 54, a second evaluation of motion vector sv _a, using the points P and Q, determine the luminance difference absolute value dp of the pixel G4, the luminance difference absolute value dp is greater than a predetermined value Determine whether or not. If the luminance difference absolute value dp is determined to be larger than the predetermined value is a low reliability of the motion vector sv _a in the pixel G4, a motion vector sv _a is excluded from the candidates. The luminance difference absolute value dp is expressed by the following equation (15).

dp = | F _t (P) − F _{t + 1} (Q) | (15)

Then, as the third evaluation of the motion vector sv _a, the vector allocating unit 54 performs evaluation determined by the difference absolute value that represents the correlation value DFD operation range around the point P and the point Q. That is, the vector allocating unit 54, using the DFD operation range around the point P and the point Q, obtains the evaluated value DFD of the motion vector sv _a (absolute differences) in the pixel G4, obtained evaluation value DFD is It is determined whether or not it is smaller than the minimum evaluation value of the DFD table. As a result of the above evaluation, the vector assigning unit 54 assigns a motion vector having the smallest evaluation value DFD among the obtained evaluation values DFD to the pixel G4.

  As described above, since the motion vector of the allocation candidate in the pixel of the interpolation frame is evaluated using not only the evaluation value DFD of the allocation target pixel but also the luminance difference absolute value in the allocation target pixel, the conventional evaluation Rather than using only the value DFD, a reliable motion vector can be allocated to the allocation target pixel. As a result, the accuracy of vector allocation is improved.

  As described above, in order to evaluate the motion vector of the allocation candidate in the pixel of the interpolation frame, the position on the original frame associated with the motion vector is used with reference to the pixel of the interpolation frame. When obtaining the luminance difference absolute value dp and the evaluation value DFD, the motion vector of the allocation candidate is extended with reference to the pixel position of the interpolation frame, so that the intersection of the motion vector and the original frame is the pixel position of the original frame. The pixel values may not match, and the pixel value cannot be obtained as it is. In such a case, the four-point interpolation process described above with reference to FIG. 39 is executed.

  FIG. 43 shows an example of four-point interpolation in the vector allocation process. In FIG. 43, portions corresponding to those in FIGS. 41 and 42 are denoted by the corresponding reference numerals, and the description thereof will be omitted because it will be repeated.

In the example of FIG. 43, the motion vector sv _a assignment candidate, because it is extended pixel position G4 in interpolated frame F1 as a reference, the point of intersection P between the motion vector sv _a frame t, in frame t does not coincide with the pixel position (white circle on the frame t), also intersecting point Q between the motion vector sv _a frame t + 1 also does not match the pixel positions on the frame t + 1 (open circles in frame t). Therefore, in the frame t and the frame t + 1, the above-described four-point interpolation is performed using the four pixels in the vicinity E centered at the intersection point P and the intersection point Q (white circles on the frame t and the frame t + 1), respectively. Pixel values of P and intersection point Q are obtained.

  As described above, in the vector allocation processing of the present invention, the luminance difference absolute value dp and the evaluation value DFD are calculated using the pixel values of the intersection P and the intersection Q obtained by the four-point interpolation processing. The luminance difference absolute value dp and the evaluation value DFD can be obtained with higher accuracy than the method of rounding the sub-pixel components.

  Next, details of the vector allocation processing will be described with reference to the flowchart of FIG. The frame t of the image at time t and the frame t + 1 of the image at time t + 1, which are the original frames of the 24P signal, are input to the pixel information calculation unit 701, the evaluation value calculation unit 702, and the target pixel difference calculation unit 703.

  When a new original frame is input, the pixel information calculation unit 701 controls the vector selection unit 705, initializes the allocation flag of the allocation flag memory 56 to 0 (False) in step S701, and proceeds to step S702. The allocation vector memory 55 is initialized with the 0 vector, and the process proceeds to step S703. As a result, a zero vector is assigned to a pixel to which no motion vector is assigned.

  In step S703, the pixel information calculation unit 701 controls the evaluation value calculation unit 702 to calculate the evaluation value DFD0 using the 0 vector for all the pixels on the interpolation frame, and the vector selection unit 705. And the evaluation value DFD0 of the 0 vector calculated by the evaluation value calculation unit 702 is stored in the DFD table as the minimum evaluation value for each pixel of the interpolation frame, and the process proceeds to step S704. That is, in step S703, the evaluation value calculation unit 702 calculates the evaluation value DFD0 using the 0 vector for all the pixels in the interpolation frame, and the calculated evaluation value DFD0 is passed through the vector evaluation unit 704. The data is output to the vector selection unit 705. Then, the vector selection unit 705 stores the evaluation value DFD0 input via the vector evaluation unit 704 as the minimum evaluation value of the corresponding pixel in the DFD table.

  In step S704, the pixel information calculation unit 701 selects a pixel from the original frame on the detection vector memory 53, and proceeds to step S705. In this case, pixels are selected in the raster scan order from the upper left of the frame.

  In step S705, the pixel information calculation unit 701 performs pixel position calculation processing, and proceeds to step S706. The details of this pixel position calculation process will be described later with reference to FIG. 45, but this pixel position calculation process allows the motion vector detected in the pixel selected in step S704 to be assigned to the interpolation frame. An allocation target pixel is calculated, and a position on the original frame associated with the motion vector is calculated on the basis of the calculated allocation target pixel.

  In step S706, the pixel information calculation unit 701 selects the calculated allocation target pixel, outputs the selected allocation target pixel and its motion vector to the vector selection unit 705, and proceeds to step S707. At the same time, the pixel information calculation unit 701 outputs, to the evaluation value calculation unit 702 and the pixel-of-interest calculation unit 703, information on the position on the original frame associated with the motion vector based on the selected allocation target pixel. To do. In step S706, the pixel information calculation unit 701 selects an upper left pixel when there are a plurality of allocation target pixels.

  In step S707, the pixel information calculation unit 701 executes an allocation vector evaluation process for the selected allocation target pixel, and the process proceeds to step S708. Details of the allocation vector evaluation process will be described later with reference to FIG. 46. By this allocation vector evaluation process, the motion vector evaluation value DFD and the luminance difference absolute value in the allocation target pixel are obtained, and the motion in the allocation target pixel is determined. The reliability of the vector is determined, and the motion vector in the allocation vector memory 55 is rewritten with the motion vector determined to have high reliability as a result of these determinations.

  In step S708, the pixel information calculation unit 701 determines whether or not the processing of all allocation target pixels has been completed. If it is determined that the processing of all allocation target pixels has not yet been completed, Returning to S706, the next allocation target pixel is selected, and the subsequent processing is repeated.

  If it is determined in step S708 that the processing of all the allocation target pixels has been completed, the pixel information calculation unit 701 determines whether or not the processing of all the pixels of the original frame on the detection vector memory 53 has been completed in step S709. Determine whether. If the pixel information calculation unit 701 determines in step S709 that the processing of all the pixels of the original frame on the detection vector memory 53 has not been completed, the process returns to step S704 and the next of the original frame on the detection vector memory 53. This pixel is selected, and the subsequent processing is repeated. If the pixel information calculation unit 701 determines in step S709 that the process for all the pixels in the detection vector memory 53 has been completed, the pixel information calculation unit 701 ends the vector allocation process.

  Next, details of the pixel position calculation process will be described with reference to the flowchart of FIG. FIG. 45 shows an example of the pixel position calculation process in step S705 of FIG.

  In step S721, the pixel information calculation unit 701 acquires the motion vector detected by the pixel selected by the process in step S704 from the detection memory vector 53, and the process proceeds to step S722. If the motion vector of the selected pixel is a 0 vector, the allocation vector memory 55 stores a 0 vector as an initial value in advance, so that the subsequent steps S722 to S724 and step S706 in FIG. 44 are performed. The processes from S708 to S708 are skipped, and the process proceeds to step S709.

  In step S722, the pixel information calculation unit 701 calculates the intersection of the acquired motion vector and the interpolation frame. That is, the pixel information calculation unit 701 extends the acquired motion vector in the direction of the next frame t + 1, calculates the intersection point of the extended motion vector and the interpolation frame, and proceeds to step S723.

  In step S723, the pixel information calculation unit 701 sets an allocation target pixel from the intersection calculated from the motion vector and the interpolation frame, and proceeds to step S724. At this time, if the intersection point coincides with the pixel position on the interpolation frame, the pixel information calculation unit 701 sets the intersection point as the allocation target pixel. On the other hand, when the intersection does not coincide with the pixel position on the interpolation frame, the pixel information calculation unit 701 sets four pixels near the intersection on the interpolation frame as allocation target pixels as described above.

  In step S724, the pixel information calculation unit 701 is acquired with reference to each allocation target pixel, which is necessary for the evaluation value calculation unit 702 and the target pixel difference calculation unit 703 to obtain the evaluation value DFD and the luminance difference absolute value. The position on the original frame associated with the motion vector is calculated. Specifically, in step S724, the pixel information calculation unit 701 shifts (translates) the acquired motion vector to the set allocation target pixel, and the shifted motion vector and the intersection of the original frame The position is obtained, the pixel position calculation process is terminated, and the process returns to step S706 in FIG.

  Next, the details of the allocation vector evaluation process will be described with reference to the flowchart of FIG. 46 shows an example of the allocation vector evaluation process in step S707 of FIG.

  In step S706 of FIG. 44, the pixel information calculation unit 701 obtains the position on the original frame associated with the motion vector with reference to the selected allocation target pixel, and information on the obtained position on the original frame is obtained. The evaluation value calculation unit 702 and the target pixel difference calculation unit 703 are input.

  When the position information on the original frame is input from the pixel information calculation unit 701 to the evaluation value calculation unit 702, in step S741, the evaluation value calculation unit 702 calculates the evaluation value DFD of the motion vector in the allocation target pixel by using the frame t and the frame Each DFD calculation range (m × n) centered on the position on t + 1 is obtained, and the process proceeds to step S742, where it is determined whether or not the obtained DFD calculation range is within the image frame. If the evaluation value calculation unit 702 determines in step S742 that the DFD calculation range is outside the image frame, the evaluation value calculation unit 702 determines that the motion vector is not an allocation candidate vector to be allocated to the allocation target pixel, and steps S743 to S749. Is skipped, the assigned vector evaluation process is terminated, and the process returns to step S708 in FIG.

  If the evaluation value calculation unit 702 determines in step S742 that the obtained DFD calculation range is within the image frame, the evaluation value calculation unit 702 proceeds to step S743 and uses the DFD calculation range determined to be within the image frame, The evaluation value DFD of the target pixel is calculated, and the obtained evaluation value DFD is output to the evaluation value determination unit 712, and the process proceeds to step S744. At this time, if the position on the original frame is equal to or less than the pixel, the evaluation value DFD of the allocation target pixel is obtained by obtaining the luminance value of the intersection on the original frame using the above-described four-point interpolation. Calculated.

  On the other hand, when the position information on the original frame is input from the pixel information calculation unit 701, the target pixel difference calculation unit 703 obtains the luminance difference absolute value dp in the allocation target pixel in step S744, and the obtained luminance The difference absolute value dp is output to the pixel difference determining unit 711, and the process proceeds to step S745. At this time, if the position on the original frame is equal to or less than the pixel, the pixel-of-interest calculation unit 703 obtains the luminance value of the intersection on the original frame using the above-described four-point interpolation. The luminance difference absolute value dp in the allocation target pixel is calculated.

  In step S745, the pixel difference determination unit 711 determines whether the luminance difference absolute value dp of the allocation target pixel from the target pixel difference calculation unit 703 is equal to or less than a predetermined threshold value, and determines the luminance of the allocation target pixel. When it is determined that the difference absolute value dp is larger than the predetermined threshold value, it is determined that there is a high possibility that the intersections of the frame t and the frame t + 1 belong to different objects, that is, the motion vector is determined in the allocation target pixel. It is determined that the reliability is low and the assignment candidate vector to be assigned to the assignment target pixel is not determined, the processes of steps S746 to S749 are skipped, the assignment vector evaluation process is terminated, and the process returns to step S708 of FIG.

  If the pixel difference determination unit 711 determines in step S745 that the luminance difference absolute value dp of the allocation target pixel is equal to or less than the predetermined threshold value, the process proceeds to step S746. In step S746, the evaluation value determination unit 712 refers to the DFD table of the vector selection unit 705, and the evaluation value DFD of the allocation target pixel from the evaluation value calculation unit 702 is the minimum allocation target pixel stored in the DFD table. It is determined whether or not the evaluation value (in this case, the evaluation value DFD0 of the zero vector) is smaller, and the evaluation value DFD of the allocation target pixel from the evaluation value calculation unit 702 is the allocation target pixel stored in the DFD table. If it is determined that the motion vector is equal to or greater than the minimum evaluation value, it is determined that the reliability of the motion vector is not high in the allocation target pixel, the processing in steps S747 to S749 is skipped, and the allocation vector evaluation processing is terminated. The process returns to step S708 of 44.

  On the other hand, when the evaluation value determination unit 712 determines in step S746 that the evaluation value DFD of the allocation target pixel from the evaluation value calculation unit 702 is smaller than the minimum evaluation value of the allocation target pixel stored in the DFD table. The motion vector is determined to have the highest reliability based on the evaluation value DFD among the motion vectors compared so far in the allocation target pixel, and the evaluation value DFD of the allocation target pixel determined to have high reliability. Is output to the vector selection unit 705, and the process proceeds to step S747.

  Upon receiving the evaluation value DFD of the allocation target pixel from the evaluation value determination unit 712, the vector selection unit 705 rewrites the allocation flag of the allocation target pixel in the allocation flag memory 56 to 1 (True) in step S747, and in step S748. The minimum evaluation value corresponding to the pixel to be allocated in the DFD table is rewritten to the evaluation value DFD determined to have high reliability by the evaluation value determination unit 712, and the process proceeds to step S749.

  The vector selection unit 705 receives the allocation target pixel selected from the pixel information calculation unit 701 and its motion vector in step S706. Therefore, in step S749, the vector selection unit 705 rewrites the motion vector allocated to the allocation target pixel in the allocation vector memory 55 with the motion vector corresponding to the evaluation value DFD determined to have high reliability, and allocates the allocation vector. The evaluation process ends, and the process returns to step S708 in FIG.

  As described above, when selecting a motion vector to be allocated to the allocation target pixel of the interpolation frame, not only the evaluation value DFD but also a position on the original frame associated with the motion vector based on the allocation target pixel is obtained. Since the absolute value of the luminance difference of the allocation target pixel is treated separately and evaluated, the most probable motion vector is selected from the allocation candidate vectors rather than using the conventional evaluation value DFD. , It can be assigned to the pixel to be assigned. Thereby, the accuracy of vector allocation is improved, discontinuity of the image generated in the subsequent image interpolation processing, etc. can be suppressed, and the image quality can be improved.

  Furthermore, when obtaining the evaluation value DFD and the luminance difference absolute value, etc., when a pixel value at a position below the pixel is necessary, the value is obtained by linear interpolation based on the distance to the four neighboring pixels at the position below the pixel. As a result, it is possible to perform processing with sub-pixel position accuracy, and more accurately obtain the luminance difference absolute value dp and the evaluation value DFD than the conventional method of rounding off the sub-pixel components. From the allocation candidate vectors, it is possible to allocate a more probable motion vector by the pixel of interest. That is, the accuracy of vector allocation processing is improved.

  In addition, the evaluation value DFD based on the 0 vector is stored as an initial value in the DFD table in advance, and while the motion vector is sequentially processed, the evaluation value DFD based on a certain motion vector becomes the minimum evaluation value at that time. In this case, since the minimum evaluation value of the DFD table and the motion vector allocated to the allocation vector memory 55 are updated as needed, time and resources can be used efficiently.

  Next, consider the evaluation value DFD in the allocation vector evaluation process when selecting a motion vector to be allocated to the allocation target pixel of the interpolation frame. In the vector detection described above with reference to FIG. 25, the one having the smallest evaluation value DFD is obtained in principle. Therefore, using the evaluation value DFD for motion vector detection means that the position at which the frame t + 1 at the time t + 1 is the smallest (pixel) with respect to a certain pixel in the frame t at the time t (that is, for one viewpoint). Therefore, using the evaluation value DFD as the evaluation value is very effective for vector detection.

  On the other hand, in the allocation vector evaluation process, the evaluation value DFD is used as one evaluation value as described above, but a plurality of allocation candidate vectors that are candidates to be allocated to the allocation target pixel are different in the frame t and the frame t + 1, respectively. May pass position.

  Specifically, with reference to FIG. 40 described above, the allocation candidate vectors of the third pixel from the left (allocation target pixel) on the interpolation frame F2 are the motion vectors v1 and v2. The pixel on the frame t of the v1 (first from the left) and the pixel on the frame t of the motion vector v2 (third from the left) are different pixels, and further, the point where the motion vector v1 and the frame t + 1 intersect ( A point near the fourth and fifth pixels from the left of the frame t + 1) and a point where the motion vector v2 and the frame t + 1 intersect (a point near the second and third pixels from the left of the frame t + 1) are different points.

  Therefore, when selecting either the motion vector v1 or the motion vector v2 as the allocation vector of the allocation target pixel, even if an attempt is made to compare using the evaluation value DFD, the evaluation value DFD that is the sum of absolute differences between different points. In this comparison, since the viewpoints are different, the image dependency tends to increase, and there is a possibility of selecting a motion vector that is not certain.

  Correspondingly, when selecting a motion vector to be allocated to the allocation target pixel of the interpolation frame as described above, not only the evaluation value DFD but also the luminance difference absolute value of the allocation target pixel is evaluated. As a result, the accuracy of vector allocation is improved by evaluating the luminance difference absolute value, compared to the case of using only the evaluation value DFD.

  However, even when only the evaluation value DFD is obtained, since pixels of m × n blocks have to be calculated, processing time is required and the hardware scale is required to some extent. When the absolute difference value is also evaluated, there are further problems that the number of processing steps increases, processing time increases, and the circuit scale increases.

  In addition to this, when obtaining the evaluation value DFD and the absolute value of the luminance difference, if a pixel value at a position below the pixel is required, linear interpolation based on the distance from the four neighboring pixels at the position below the pixel is used. Since the value is obtained, the accuracy of the vector allocation process is improved, but there is a problem that it takes a longer processing time.

  Therefore, next, a case will be described in which a comparison is made using an evaluation value other than the evaluation value DFD when selecting a motion vector to be allocated to the allocation target pixel of the interpolation frame.

  FIG. 47 is a block diagram showing another configuration of the vector allocating unit 54. 47 assigns a difference (variation) value (hereinafter also referred to as a difference value) between a motion vector detected on the frame t and a motion vector around the motion vector. By using the evaluation value as an evaluation value and evaluating the reliability, a process of assigning to the pixels on the interpolation frame of the 60P signal to be interpolated on the assignment vector memory 55 is performed.

  That is, unlike the vector assignment unit 54 in FIG. 38, pixel values of frames t and t + 1 are not input to the vector assignment unit 54 in FIG.

  The pixel information calculation unit 751 selects a pixel on the frame t in the detection vector memory 53 in the raster scan order from the upper left pixel, acquires a motion vector detected by the selected pixel, And the intersection of the extended motion vector and the interpolated frame is calculated. Then, the pixel information calculation unit 751 sets an allocation target pixel of the motion vector on the interpolation frame from the intersection of the calculated motion vector and the interpolation frame, and information on the position of the motion vector and the allocation target pixel Is output to the vector selection unit 754.

  When the pixel information calculation unit 751 acquires the motion vector, the evaluation value calculation unit 752 receives the motion vector of the pixel selected by the pixel information calculation unit 751 and pixels around the pixel of the motion vector (for example, the surrounding 8 The detected motion vector is input to (pixel). The evaluation value calculation unit 752 uses the motion vector of the pixel selected by the pixel information calculation unit 751 and the motion vector detected for pixels around the pixel of the motion vector (for example, surrounding 8 pixels), And the difference value of the obtained motion vector is output to the vector evaluation unit 753.

  The vector evaluation unit 753 determines whether or not the difference value of the motion vector input from the evaluation value calculation unit 752 is smaller than the minimum evaluation value of the allocation target pixel from the vector selection unit 754, and the difference of the motion vector is determined. If it is determined that the value is smaller than the minimum evaluation value of the allocation target pixel, it is determined that the reliability of the motion vector is high, and the difference value of the motion vector is output to the vector selection unit 754.

  The vector selection unit 754 has an evaluation value memory (not shown) that holds a minimum evaluation value for each pixel on the interpolation frame. The evaluation value memory has the same configuration as the DFD table included in the vector selection unit 705 in FIG. The vector selection unit 754 rewrites the flag of the allocation flag memory 56 corresponding to the allocation target pixel to which the motion vector is allocated to 1 (true) or based on the position information of the allocation target pixel from the pixel information calculation unit 751. The minimum evaluation value in the evaluation value memory of the allocation target pixel is rewritten with the difference value of the motion vector. In addition, the vector selection unit 754 controls the pixel evaluation unit 753 to assign the pixel information calculation unit to the allocation target pixel of the allocation vector memory 55 based on the position information of the allocation target pixel from the pixel information calculation unit 751. The motion vector from 751 is allocated.

  Next, with reference to FIG. 48, the motion vector evaluation in the vector allocation unit 54 in FIG. 47 will be described. FIG. 48 one-dimensionally shows a frame t at time t, an interpolation frame F1 at a time between time t and time t + 1, and a frame t + 1 at time t + 1 from the bottom. A circle on each frame indicates a pixel on the frame.

  In the example of FIG. 48, the motion vectors v1 and v2 detected in the frame t by the preceding vector detection unit 52 are extended in the frame t + 1 direction.

  The motion vector v1 from the left pixel of the frame t to the vicinity of the third pixel from the left of the frame t + 1 passes through the vicinity of the second pixel G from the left on the interpolation frame F1, and from the left of the frame t The motion vector v1 from the third pixel to the vicinity of the second pixel G from the left in the frame t + 1 passes through the vicinity of the second pixel from the left on the interpolation frame F1. That is, since the motion vectors v1 and v2 pass through the vicinity of the pixel G of the interpolation frame F1, each of the motion vectors v1 and v2 is set as an allocation candidate vector of the pixel G of the interpolation frame F1.

  In order to compare the motion vector v1 and the motion vector v2 which are the allocation candidate vectors, the vector allocation unit 54 obtains a difference value (variation) of each motion vector on the frame t by the following equation (16). Is used as an evaluation value.

Motion vector difference value = Σ | v _i −v | (16)

Here, v is a target motion vector, and v _i is a motion vector (that is, surrounding 8 pixels) detected from the pixels where the motion vector v is detected on the frame t (ie, the surrounding 8 pixels). , Ambient vector). In equation (16), a square value may be used instead of the absolute value.

  The vector allocating unit 54 obtains the difference values between the motion vectors v1 and v2 by using the equation (16), and as a result of comparing the obtained differences, the difference from the surrounding vectors is small. A motion vector having a smaller difference from the surrounding vector is determined to be a motion vector that is stably detected on the frame t, and a motion vector having a smaller difference from the surrounding vector is allocated to the pixel G. .

  As described above, by evaluating motion vectors based on the difference of motion vectors, image allocation is low in vector assignment, and a motion vector that is less accurate than the case of the evaluation value DFD may be selected. It is suppressed. Thereby, for example, problems such as “penetration” that may occur at the time of vector assignment as shown in FIG. 49 can be suppressed.

  In the example of FIG. 49, the image object of the object Ob passes the time t to the time t + 1 (that is, the frame t to the frame t + 1) with the motion vector v1 from the left side to the right side in the drawing. On the other hand, the background passes from time t to time t + 1 with the motion vector v2 from the right side to the left side in the figure.

  Therefore, originally, the motion vector v1 should be selected as the allocation vector for the pixel G at the position where the image object of the object Ob passes in the interpolation frame F1, but for example, from the image object of the object Ob If the evaluation value DFD is obtained using pixels of the block B of m × n, which is a large size, and the motion vectors v1 and v2 are evaluated by the evaluation value DFD, the background is often flat. Since the background evaluation value DFD tends to be small, the background motion vector v2 may be erroneously selected. As a result, the background of the image object of the object Ob of the interpolated frame F1 generated thereby results.

  However, if the background is flat, the motion vector detection process often detects an erroneous motion vector rather than an object having details. In such a case, the difference between the motion vector v2 and the surrounding vector becomes a large value. On the other hand, since an object having details has details, there is a high possibility that a more reliable motion vector is detected than a flat background. Therefore, in this case, the difference between the motion vector v1 and its surrounding vector is smaller than the difference between the motion vector v2 and its surrounding vector.

  From the above, by using the difference between the motion vector and its surrounding vector as an evaluation value, “piercing” in which the background penetrates the object Ob of the object Ob is suppressed in the pixel G of the interpolation frame F1, More probable motion vectors are allocated as allocation vectors.

  Next, the details of the vector allocation processing of the vector allocation unit 54 of FIG. 47 will be described with reference to the flowchart of FIG. That is, this vector allocation process is another example of the vector allocation process of FIG.

  When the motion vector detected in the frame t is stored in the detection vector memory 53 by the preceding vector detection unit 52, the pixel information calculation unit 751 controls the vector selection unit 754, and in step S761, the allocation flag memory The allocation flag of 56 is initialized with 0 (False), and the process proceeds to step S762.

  In step S762, the pixel information calculation unit 751 selects a pixel from the original frame on the detection vector memory 53, and proceeds to step S763. In this case, pixels are selected in the raster scan order from the upper left of the frame.

  In step S763, the pixel information calculation unit 751 executes pixel position calculation processing. Details of the pixel position calculation processing will be described with reference to the flowchart of FIG. The pixel position calculation process in FIG. 51 is different from the pixel position calculation process in FIG. 45 except for the position calculation process in the original frame in step S724. The other processes are the same as those in FIG. A process basically similar to the pixel position calculation process is performed.

  In step S771 in FIG. 51, the pixel information calculation unit 751 acquires the motion vector detected in the pixel selected by the process in step S762 in FIG. 50 from the detection memory vector 53, and proceeds to step S772. At this time, the motion vector of the pixel selected by the pixel information calculation unit 751 and the motion vector detected in pixels around the pixel of the motion vector (for example, the surrounding eight pixels) are sent to the evaluation value calculation unit 752. Entered.

  In step S772, the pixel information calculation unit 751 calculates the intersection of the acquired motion vector and the interpolation frame. That is, the pixel information calculation unit 751 extends the acquired motion vector in the direction of the next frame t + 1, calculates the intersection of the extended motion vector and the interpolated frame, and proceeds to step S773.

  In step S <b> 773, the pixel information calculation unit 751 sets an allocation target pixel from the intersection calculated from the motion vector and the interpolation frame. At this time, when the intersection point coincides with the pixel position on the interpolation frame, the pixel information calculation unit 751 sets the intersection point as the allocation target pixel. On the other hand, when the intersection does not coincide with the pixel position on the interpolation frame, the pixel information calculation unit 751 sets the four neighboring pixels on the interpolation frame as the allocation target pixels as described above. Then, the pixel information calculation unit 751 ends the pixel position calculation process, and returns to step S764 in FIG.

  In step S764, the pixel information calculation unit 751 executes an allocation vector evaluation process, and proceeds to step S765. The details of the allocation vector evaluation process will be described later with reference to FIG. 52. The allocation vector evaluation process obtains a motion vector difference value, and assigns the allocation based on the obtained motion vector difference value. The reliability of the motion vector in the target pixel is determined, and the motion vector in the allocation vector memory 55 is rewritten with the motion vector determined to have high reliability as a result of these determinations.

  In step S765, the pixel information calculation unit 751 determines whether or not the processing of all the pixels of the original frame on the detection vector memory 53 has been completed. If the pixel information calculation unit 751 determines in step S765 that the processing of all the pixels of the original frame on the detection vector memory 53 has not been completed, the process returns to step S762 and the next of the original frame on the detection vector memory 53. This pixel is selected, and the subsequent processing is repeated. On the other hand, if the pixel information calculation unit 751 determines in step S765 that the processing for all the pixels in the detection vector memory 53 has been completed, the vector allocation processing ends.

  Next, details of the allocation vector evaluation process will be described with reference to the flowchart of FIG. FIG. 52 shows an example of the allocation vector evaluation process in step S764 of FIG.

  In step S781 in FIG. 52, the evaluation value calculation unit 752 obtains a difference value between the motion vectors of the pixels selected in step S762 in FIG. That is, in step S771 of FIG. 51, when the pixel information calculation unit 751 acquires a motion vector, the evaluation value calculation unit 752 includes the motion vector of the pixel selected by the pixel information calculation unit 751 and the pixel of the motion vector. The detected motion vector is input to the surrounding pixels (for example, surrounding 8 pixels).

  Therefore, the evaluation value calculation unit 752 detects the motion vector of the pixel selected by the pixel information calculation unit 751 and the pixels around the pixel of the motion vector (for example, the surrounding 8 pixels) in step S781 of FIG. The motion vector difference value shown in Expression (16) is calculated using the obtained motion vector, and the obtained motion vector difference value is output to the vector evaluation unit 753, and the process proceeds to step S782.

  In step S782, the pixel information calculation unit 751 selects the allocation target pixel calculated in step S773 of FIG. 51, and outputs the selected allocation target pixel and its motion vector to the vector selection unit 754, and then proceeds to step S783. move on. In step S782, the pixel information calculation unit 751 selects from the upper left pixels when there are a plurality of allocation target pixels.

  In step S783, the vector selection unit 754 determines whether or not the flag of the allocation flag memory 56 corresponding to the allocation target pixel from the pixel information calculation unit 751 is 1 (true), and the flag of the allocation flag memory 56 is set. If it is determined that it is not 1 (true), the process proceeds to step S784, the flag of the allocation flag memory 56 corresponding to the allocation target pixel is rewritten to 1 (true), and the process proceeds to step S786.

  That is, when the flag of the allocation flag memory 56 is 0 (False), the allocation vector memory 55 of the allocation target pixel has not yet been allocated, and the evaluation value memory built in the vector selection unit 754 No minimum evaluation value is stored. Therefore, when the flag of the allocation flag memory 56 is 0 (False), the vector selection unit 754 rewrites the flag of the allocation flag memory 56 to 1 (true) and does not compare with the minimum evaluation value (that is, step S785). Without executing the process of step S786). The evaluation value memory may store the maximum value stored in the memory as an initial value.

  On the other hand, if it is determined in step S783 that the flag of the allocation flag memory 56 is 1 (true), the allocation vector memory 55 of the allocation target pixel has already been allocated with other allocation vectors. A corresponding minimum evaluation value is stored in the evaluation value memory.

  Therefore, when it is determined in step S783 that the flag of the allocation flag memory 56 is 1 (true), the vector selection unit 754 reads the minimum evaluation value corresponding to the allocation target pixel from the evaluation value memory, and performs vector evaluation. The process proceeds to step S785.

  When the minimum evaluation value is supplied from the vector selection unit 754, the vector evaluation unit 753 determines whether the difference value of the motion vector obtained in step S781 is smaller than the minimum evaluation value of the allocation target pixel from the vector selection unit 754. If the difference value of the motion vector is determined to be smaller than the minimum evaluation value of the allocation target pixel, it is determined that the reliability of the motion vector is high, and the difference value of the motion vector is The data is supplied to the selection unit 754, and the process proceeds to step S786.

  In step S786, the vector selection unit 754 rewrites the corresponding minimum evaluation value in the evaluation value memory with the motion vector difference value from the vector evaluation unit 753, proceeds to step S787, and sets the allocation target pixel in the allocation vector memory 55 as the allocation target pixel. The allocated motion vector is rewritten with a motion vector corresponding to the evaluation value determined to have high reliability, and the process proceeds to step S788.

  In step S785, the difference value of the motion vector is not smaller than the minimum evaluation value of the allocation target pixel in the evaluation value memory (the minimum evaluation value of the allocation target pixel in the evaluation value memory is the difference in motion vector). If it is determined that the motion vector is smaller than the value, the reliability of the motion vector is low, and the reliability of the motion vector already written in the allocation vector memory 55 is higher. Therefore, the processing in steps S786 and S787 is performed. Is skipped, and the process proceeds to step S788.

  In step S788, the pixel information calculation unit 751 determines whether or not all the allocation target pixels (for example, four pixels) have been processed, and determines that all the allocation target pixels have not yet been processed. If so, the process returns to step S782, the next allocation target pixel is selected, and the subsequent processing is repeated. That is, even when there are four allocation target pixels, once the difference value of the motion vector is obtained, it can be used for the four allocation target pixels. In addition, it is not necessary to perform the calculation process in step S781.

  If it is determined in step S788 that the processing of all allocation target pixels has been completed, the pixel information calculation unit 751 ends the allocation vector evaluation processing and returns to step S765 in FIG.

  As described above, when selecting a motion vector to be allocated to the allocation target pixel of the interpolation frame, the difference (variation) of the allocated motion vectors is used as the evaluation value. Therefore, compared to the case where the evaluation value DFD is used. In particular, problems such as “penetration” as described above with reference to FIG. 49 can be suppressed, and more probable motion vectors can be selected from the allocation candidate vectors and allocated to the allocation target pixels. . Thereby, the accuracy of vector allocation is improved, discontinuity of the image generated in the subsequent image interpolation processing, etc. can be suppressed, and the image quality can be improved.

  Further, by using the difference (variation) of the motion vectors to be allocated as an evaluation value, the positions of two original frames are obtained by linear interpolation as compared with the allocation vector evaluation executed by the vector allocation unit 54 in FIG. It is not necessary to execute the process of step S724 in FIG. 51, and it is not necessary to obtain the evaluation value DFD and the luminance difference absolute value dp for each allocation target pixel. Therefore, even if the allocation vector evaluation process of FIGS. 44 to 46 and the allocation vector evaluation process of FIGS. 50 to 52 are simply compared, the allocation vector evaluation process of FIGS. 50 to 52 requires fewer steps. , Processing time is also reduced.

  Further, the image information calculation unit 751 eliminates the need to recalculate the positions of the two original frames necessary for the image information calculation unit 701 and linear interpolation, and the vector allocation unit 54 in FIG. 47 does not require the pixel values of the frames t and t + 1 that are required for the evaluation value calculation unit 702 and the pixel-of-interest calculation unit 703, the vector allocation unit 54 in FIG. 47 requires a memory for storing the pixel values. Since this is eliminated, the circuit scale can be considerably reduced.

  In the above description, the motion vector of the detection vector memory 53 is detected using the gradient method, but a motion vector obtained by another method such as a block matching method may be used.

  Next, details of the configuration of the allocation compensation unit 57 will be described.

  FIG. 53 is a block diagram showing a configuration of the allocation compensator 57. The allocation compensation unit 57 having the configuration shown in FIG. 53 is configured by an allocation vector determination unit 801 and a vector compensation unit 802. Pixels on an interpolation frame of the 60P signal to which no motion vector is allocated by the vector allocation unit 54 A process is performed in which the motion vectors of surrounding pixels are supplemented.

  The motion vector is assigned to the pixel of the interpolation frame on the assigned vector memory 55 by the preceding vector assigning unit 54. Further, 1 (True) is written in the allocation flag of the pixel allocation flag memory 56 to which the motion vector is allocated by the vector allocation unit 54, and the allocation of the allocation flag memory 56 of the pixel to which no motion vector is allocated. 0 (False) is written in the flag.

  The allocation vector determination unit 801 refers to the allocation flag in the allocation flag memory 56 and determines whether or not a motion vector is allocated to the pixel of interest by the vector allocation unit 54. Then, the allocation vector determination unit 801 selects a pixel of interest for which no motion vector has been allocated by the vector allocation unit 54, controls the vector compensation unit 802 for the selected pixel of interest, and sets the surrounding pixels of the pixel of interest. A motion vector is selected and allocated on the interpolation frame of the allocation vector memory 55.

  The vector compensation unit 802 acquires the motion vector assigned to the peripheral pixel of the target pixel from the assigned vector memory 55, and uses the frame t of the image at time t and the frame t + 1 of the image at time t + 1 that are input. The obtained motion vector evaluation value DFD is obtained and compared, and the motion vector having the highest reliability based on the evaluation value DFD among the motion vectors assigned to the peripheral pixels of the target pixel is assigned to the assigned vector memory 55. Assigned to the pixel of interest. Further, the vector compensation unit 802 rewrites the allocation flag of the target pixel to which the motion vector is allocated to 1 (True).

  FIG. 54 is a block diagram showing the configuration of the vector compensation unit 802. 54 includes a compensation processing unit 811 and an evaluation value calculation unit 812.

  The compensation processing unit 811 includes a memory 821 that stores a minimum evaluation value DFD and a motion vector of the minimum evaluation value DFD as candidate vectors (hereinafter also referred to as compensation candidate vectors), and is selected by the allocation vector determination unit 801. As the initial value of the pixel of interest, the zero vector evaluation value DFD is stored in the memory 821 as the minimum evaluation value, and the zero vector is stored in the memory 821 as the compensation candidate vector. The compensation processing unit 811 refers to the allocation flag memory 56 to determine the presence or absence of motion vectors of the peripheral pixels of the target pixel, acquires the motion vector allocated to the peripheral pixels from the allocation vector memory 55, and evaluates the evaluation value. The calculation unit 812 is controlled to calculate the evaluation value DFD of the motion vector.

  Further, the compensation processing unit 811 determines whether or not the evaluation value DFD calculated by the evaluation value calculation unit 812 is smaller than the minimum evaluation value stored in the memory 821, and the calculated evaluation value DFD is the minimum evaluation. When it is determined that the value is smaller than the value, the compensation candidate vector and the minimum evaluation value in the memory 821 are rewritten to the calculated evaluation value DFD and its motion vector, and finally, the peripheral pixels for which the evaluation value DFD is determined to be the smallest Are assigned to the target pixel in the allocation vector memory 55 as the motion vector of the target pixel. Further, the compensation processing unit 811 rewrites the allocation flag of the allocation flag memory 56 of the target pixel to which the motion vector is allocated to 1 (True).

  When the evaluation value calculation unit 812 acquires the motion vector of the surrounding pixels from the allocation vector memory 55, the evaluation vector calculation unit 812 uses the frame t of the 24P signal image at time t and the frame t + 1 of the image at time t + 1 to be input. The motion vector evaluation value DFD from 55 is calculated, and the calculated evaluation value DFD is output to the compensation processing unit 811.

  FIG. 55 is a diagram for explaining the principle of allocation compensation processing. In the example of FIG. 55, each pixel on the interpolation frame is shown. An arrow output from a pixel represents a motion vector assigned to each pixel, and a pixel without an arrow represents a pixel to which no motion vector is assigned.

  Here, based on the evaluation value DFD among the motion vectors assigned to the peripheral pixels in the vicinity of the target pixel P with respect to the central target pixel P to which no motion vector is assigned by the vector allocating unit 54 in the previous stage. Select and assign the one with high reliability. In the case of the example in FIG. 55, the motion vector (thick line arrow) of the pixel above the target pixel P is selected and assigned to the target pixel P. This is a process executed based on the motion correlation described below.

  FIG. 56 is a diagram for explaining the principle of motion correlation. In the example of FIG. 56, an object O1 moving with a motion v1 on a certain frame and an object O2 moving with a motion v2 are shown. The target pixel P1 belonging to the object O1 and its vicinity K1 have substantially the same movement v1 as the object O1. Further, the target pixel P2 belonging to the object O2 and its vicinity K2 have substantially the same movement v2 as the object O2.

  As described above, the motion correlation represents that the motion of the pixels belonging to a certain object is often almost the same in a space at the same time (in the same frame). Therefore, for a pixel for which a motion vector could not be assigned, using such a motion correlation in a space at the same time (in the same frame), from the motion vectors of surrounding pixels, The corresponding motion vector for that pixel can be selected. In addition, although description is abbreviate | omitted, the correlation of a time direction is also the same.

  Next, with reference to FIG. 57 to FIG. 63, motion vector compensation processing executed based on motion correlation will be described. That is, it is a process of selecting a motion vector from the motion vectors of surrounding pixels and supplementing it as the motion vector of the pixel of interest. In the example of FIG. 57, white circles represent pixels on the interpolation frame, and 8 peripheral pixels are shown around the pixel of interest P for which a motion vector is to be obtained. The motion vector of the pixel of interest P is obtained with reference to the motion vectors of the surrounding 8 pixels.

  In the example of FIG. 58, the upper left pixel, the upper right pixel, and the lower right pixel (black circle) among the eight pixels around the target pixel P are obtained by the preceding process (for example, the above-described vector allocation process). The motion vector (arrow) is shown. That is, the compensation candidate vectors for the pixel of interest P in this case are the motion vectors of the upper left pixel, the upper right pixel, and the lower right pixel. On the frame, since the motion vector is obtained in the raster scan order from the upper left pixel on the frame, there may be a pixel for which the motion vector is not yet obtained among the surrounding 8 pixels. Since a motion vector has not yet been obtained, it cannot be a compensation candidate vector.

  Here, as shown in the example of FIG. 59, the peripheral 8 pixels have the motion vector obtained by this process in addition to the pixel (black circle) having the motion vector obtained in the previous process. There are also pixels (hatched circles). That is, in this process, the result of the previous stage of this process itself is also used. Accordingly, the compensation candidate vector of the pixel of interest P in the example of FIG. 59 is a pixel (hatched) having a motion vector of a pixel in which a motion vector already exists (black circle) and a motion vector obtained by this processing in the previous stage. (Circle) motion vector.

  Also, as shown in the example of FIG. 60, a 0 vector (still vector) S0 with a motion amount of 0 can also be used as a compensation candidate vector. In the example of FIG. 59, the pixel having the motion vector obtained by this process and the pixel in which the motion vector already exists are represented separately, but both are the same in that they have a motion vector. In FIG. 60 to FIG. 63, the pixel having the motion vector obtained by this processing is also included in the pixel (black circle) in which the motion vector already exists. Therefore, in the example of FIG. 60, the compensation candidate vector of the target pixel P is configured by a motion vector of a pixel (black circle) in which a motion vector already exists and a zero vector S0.

  In order to compare the reliability (probability) of the compensation candidate vectors configured as described above, an evaluation value DFD, which is a motion vector evaluation method, is obtained as shown in FIGS. 61 to 63 below. It is done. FIG. 61 shows an example in which a zero vector S0 is used as a compensation candidate vector. FIG. 62 shows an example in which the motion vector VK1 of the upper left pixels of the surrounding 8 pixels is used as the compensation candidate vector. FIG. 63 shows an example in which the motion vector VK2 of the upper center pixel of the surrounding 8 pixels is used as the compensation candidate vector.

  In the example of FIG. 61, an example is shown in which the zero vector S0 is selected from the compensation candidate vectors of the pixel of interest P shown on the left side of the drawing, and the evaluation value DFD of the selected zero vector S0 is obtained. That is, the evaluation value DFD for the 0 vector S0 is selected on the basis of the pixel of interest P on the interpolation frame on the frame t and the frame t + 1 of the 24P signal sandwiching the pixel of interest P (an interpolation frame of the 60P signal). An intersection with which the 0 vector S0 is associated is obtained, DFD calculation ranges D1-1 and D1-2 within a predetermined range (m × n) are calculated around the intersection, and the calculated DFD calculation range D1-1 is calculated. And D1-2 are used to calculate equation (1) described above.

  In the example of FIG. 62, the motion vector VK1 of the upper left pixel of the surrounding 8 pixels is selected from the compensation candidate vectors of the target pixel P shown on the left side of the drawing, and the evaluation value DFD of the selected motion vector VK1 is obtained. An example is shown. In other words, the evaluation value DFD for the motion vector VK1 of the upper left pixel of the eight neighboring pixels is selected on the basis of the target pixel P on the interpolation frame on the frame t and the frame t + 1 sandwiching the target pixel P (interpolation frame). The intersection with which the motion vector VK1 is associated is obtained, the DFD calculation ranges D2-1 and D2-2 in a predetermined range (m × n) are calculated around the intersection, and the calculated DFD calculation range D2-1 is calculated. And D2-2 are used to calculate the above-described equation (1).

  In the example of FIG. 63, the motion vector VK2 of the upper center pixel of the surrounding eight pixels is selected as the compensation candidate vector from the compensation candidate vectors of the target pixel P shown on the left side of the drawing, and the selected motion vector VK2 is selected. An example is shown in which the evaluation value DFD is obtained. That is, the evaluation value DFD for the motion vector VK2 of the upper central pixel of the surrounding eight pixels is based on the target pixel P on the interpolation frame on the frame t and the frame t + 1 sandwiching the target pixel P (interpolation frame). An intersection point to which the selected motion vector VK2 is associated is obtained, DFD calculation ranges D3-1 and D3-2 in a predetermined range (m × n) are calculated around this intersection point, and the calculated DFD calculation range D3- It is calculated | required by calculating Formula (1) mentioned above using 1 and D3-2.

  Note that the other compensation candidate vectors shown on the left side in the figure are basically the same processing, and thus the description thereof is omitted. However, as described above, all the compensation candidate vectors for the peripheral pixels of the pixel of interest P are as described above. Evaluation values DFD are obtained, and the obtained evaluation values DFD are compared. Among them, a compensation candidate vector having the smallest evaluation value DFD is assigned to the pixel of interest P as shown in FIG. Selected as the most reliable and probable motion vector.

  In the case of the example in FIG. 64, it is determined that the evaluation value DFD of the motion vector VK1 of the upper left pixel of the eight neighboring pixels is the smallest among the compensation candidate vectors of the peripheral pixels of the target pixel P, and the motion vector VK1 is Is selected and assigned as a motion vector.

  As described above, since the motion vector of the pixel that could not be allocated by the vector allocation unit 54 is compensated from the motion vector of the surrounding pixels using motion correlation, the motion vector is not allocated. , The disturbance of motion can be suppressed as compared with the case where the 0 vector is assigned. Also, the motion vector of the pixel compensated in this way can be used again as a compensation candidate vector for another pixel. That is, since not only motion vectors in the spatial direction but also motion vectors in the temporal direction can be used as compensation candidate vectors, pixels that perform substantially the same motion in the object have substantially the same motion vector. A stable motion vector that is selected and has few errors can be obtained. This improves the accuracy of vector assignment.

  Next, details of the allocation compensation processing will be described with reference to the flowchart of FIG. The motion vector is assigned to the pixel of the interpolation frame on the assigned vector memory 55 by the preceding vector assigning unit 54. Further, 1 (True) is written in the allocation flag of the pixel allocation flag memory 56 to which the motion vector is allocated by the vector allocation unit 54, and the allocation of the allocation flag memory 56 of the pixel to which no motion vector is allocated. 0 (False) is written in the flag.

  In step S801, the allocation vector determination unit 801 selects a pixel of the interpolation frame in the allocation flag memory 56 as a target pixel, and the process proceeds to step S802. At this time, the allocation vector determination unit 801 selects pixels from the upper left pixel of the frame in the raster scan order.

  In step S802, the allocation vector determination unit 801 determines whether the allocation flag of the target pixel in the allocation flag memory 56 is 0 (False), and the allocation flag of the target pixel in the allocation flag memory 56 is If it is determined that the motion vector is 0 (False), it is determined that no motion vector is allocated, and the process proceeds to step S803, the compensation processing unit 811 is controlled to execute the vector compensation process, and the process proceeds to step S804. Details of this vector compensation processing will be described later with reference to FIG. 66. By this vector compensation processing, among the motion vectors assigned to the peripheral pixels, the smallest motion vector of the evaluation value DFD is stored as a compensation candidate vector. 821 is stored.

  In step S804, the compensation processing unit 811 allocates the compensation candidate vector in the memory 821 as the motion vector of the target pixel to the allocation vector memory 55, and proceeds to step S805 to set the target pixel allocation flag in the allocation flag memory 56 to 1 Rewrite to (True), and go to Step S806.

  On the other hand, if the allocation vector determination unit 801 determines in step S802 that the allocation flag of the target pixel in the allocation flag memory 56 is 1 (True), a motion vector has already been allocated to the target pixel. Is determined, the process of steps S803 to S805 is skipped, and the process proceeds to step S806.

  In step S806, the allocation vector determination unit 801 determines whether or not the processing of all the pixels in the interpolation frame of the allocation flag memory 56 has been completed, and determines that the processing of all the pixels has not been completed. Returning to step S801, the next pixel of the interpolation frame in the allocation flag memory 56 is selected as the pixel of interest, and the subsequent processing is executed. If it is determined in step S806 that the processing of all the pixels in the interpolation frame in the allocation flag memory 56 has been completed, the allocation compensation processing is terminated.

  Next, the details of the vector compensation processing will be described with reference to the flowchart of FIG. FIG. 66 shows an example of the vector compensation processing in step S803 of FIG.

  In step S821, the compensation processing unit 811 controls the evaluation value calculation unit 812 to calculate the evaluation value DFD0 using the 0 vector, and the process proceeds to step S822. Specifically, in step S821, the evaluation value calculation unit 812 uses the frame t of the image at time t and the frame t + 1 of the image at time t + 1, for example, as described above with reference to FIG. In addition, for the pixel of interest, an evaluation value DFD0 with a 0 vector is calculated, and the calculated evaluation value DFD0 is output to the compensation processing unit 811.

  In step S822, the compensation processing unit 811 stores the evaluation value DFD0 as the minimum evaluation value in the memory 821, proceeds to step S823, stores the 0 vector as the compensation candidate vector in the memory 821, and proceeds to step S824. In step S824, the compensation processing unit 811 selects one peripheral pixel from the eight peripheral pixels of the target pixel selected by the allocation vector determination unit 801, and the process proceeds to step S825. At this time, the compensation processing unit 811 selects peripheral pixels in the raster scan order from the upper left pixel among the eight peripheral pixels.

  In step S825, the compensation processing unit 811 refers to the allocation flag memory 56 and determines whether there is a motion vector of the selected peripheral pixel. If the allocation flag of the peripheral pixel in the allocation flag memory 56 is 1 (True), the compensation processing unit 811 determines in step S825 that there is a motion vector allocated to the selected peripheral pixel, and the process proceeds to step S826. Then, the motion vectors of the surrounding pixels are acquired from the allocation vector memory 55, and the process proceeds to step S827. At this time, the motion vectors of the surrounding pixels are also output from the allocation vector memory 55 to the evaluation value calculation unit 812.

  When the motion vector of the neighboring pixels is input from the allocation vector memory 55, the evaluation value calculation unit 812 uses the input frame t of the image at time t and the frame t + 1 of the image at time t + 1 in step S827. For the pixel of interest, the motion vector evaluation value DFD from the allocation vector memory 55 is calculated, and the calculated evaluation value DFD is output to the compensation processing unit 811, and the process proceeds to step S 828.

  When the evaluation value DFD is input from the evaluation value calculation unit 812, the compensation processing unit 811 determines whether or not the evaluation value DFD is smaller than the minimum evaluation value of the target pixel stored in the memory 821 in step S828. If it is determined that the evaluation value DFD is smaller than the minimum evaluation value of the target pixel stored in the memory 821, the process proceeds to step S829, and the minimum evaluation value of the memory 821 is determined to be smaller than the minimum evaluation value. The evaluation value DFD is rewritten, and the process proceeds to step S830. The compensation candidate vector in the memory 821 is rewritten to the motion vector of the minimum evaluation value, and the process proceeds to step S831.

  On the other hand, if the allocation flag of the peripheral pixel in the allocation flag memory 56 is 0 (False) in step S825, the compensation processing unit 811 determines that there is no motion vector allocated to the selected peripheral pixel, and step S826. The process of thru | or S830 is skipped and it progresses to step S831. In step S828, when the compensation processing unit 811 determines that the evaluation value DFD is greater than or equal to the minimum evaluation value of the pixel of interest stored in the memory 821, the processing of steps S829 and S830 is skipped, and step S831 is performed. Proceed to

  In step S831, the compensation processing unit 811 determines whether or not the processing has been completed for all eight pixels around the target pixel, and determines that the processing has not been completed for all eight pixels around the target pixel. If so, the process returns to step S824, the next peripheral pixel is selected, and the subsequent processing is repeated. If the compensation processing unit 811 determines in step S831 that the processing has been completed for all eight pixels around the pixel of interest, the compensation processing unit 811 ends the vector compensation processing and returns to step S804 in FIG.

  As described above, the most reliable level based on the evaluation value DFD is obtained from the motion vectors around the pixel using the fact that there is a motion correlation even for the pixel that cannot be assigned in the vector assignment processing. There is a certain motion vector. As a result, the accuracy of vector assignment is improved as compared with the case where a vector is not assigned and a 0 vector is assigned, and discontinuity of an image generated in subsequent image interpolation processing can be suppressed.

  In addition, the allocation flag of the pixel to which the motion vector is allocated by the allocation compensation process described above is rewritten to 1 (True), and the motion vector allocated by the allocation compensation process is also used as a compensation candidate vector for the next pixel. Therefore, almost the same motion vector is selected for pixels that perform substantially the same motion in the object, and a stable motion vector with few errors can be obtained. As a result, it is possible to improve the quality by suppressing block noise, powder noise and the like of the image generated in the subsequent stage.

  In addition, for a pixel of interest for which a motion vector is to be calculated, an evaluation value DFD of 0 vector is calculated in advance and stored in a memory as a minimum evaluation value, so that all compensation candidate vector evaluation values DFD are calculated at once. Thus, time and resources can be used more efficiently than when the smallest evaluation value DFD is selected.

  In the above description, the vector compensation processing is performed on the pixels that are not assigned by the vector assigning unit 54, but there are some kinds of pixels such as pixels that are not detected by the vector detecting unit 52 (the zero vector is detected). You may make it perform a vector compensation process with respect to the pixel in which the motion vector was not calculated | required in the process. Furthermore, vector compensation processing may be performed on pixels for which the detected motion vector or the assigned motion vector is uncertain (low reliability).

  In this embodiment, the evaluation value DFD that is the sum of absolute differences is used as the evaluation value when selecting a motion vector. However, the evaluation value DFD is not limited to the evaluation value DFD, and the reliability of the motion vector is evaluated. If it is a thing, you may make it use another thing.

  In the present embodiment, the case where the motion vector is detected using the gradient method has been described. However, the above-described vector allocation processing and allocation compensation are performed based on the motion vector detected using the block matching method or the like. Processing after motion vector detection such as processing or image interpolation processing may be executed.

  Furthermore, in the present embodiment, the block for performing each process has been described as being configured by, for example, 8 pixels × 8 pixels, 9 pixels × 9 pixels, and the like, but these are examples, and each process is performed. The number of pixels constituting the block to be performed is not limited to the number of pixels.

  In this embodiment, the signal conversion from the 24P signal to the 60P signal has been described as an example. However, the present invention is, for example, an interlace signal or other frame rate as the frame frequency conversion of a moving image. It can also be applied to conversion.

  The series of processes described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program storage medium in a general-purpose personal computer or the like.

  As shown in FIG. 1, a program storage medium that stores a program that is installed in a computer and can be executed by the computer includes a magnetic disk 31 (including a flexible disk), an optical disk 32 (CD-ROM (Compact Disc -Removable recording media (package media) comprising a read only memory (including DVD (Digital Versatile Disc)), magneto-optical disc 33 (including MD (Mini-Disc) (trademark)), or semiconductor memory 34, or The ROM 12 stores the program temporarily or permanently.

  In the present specification, the steps shown in the flowcharts include not only processes performed in time series according to the described order, but also processes executed in parallel or individually even if not necessarily performed in time series. Is included.

It is a block diagram which shows the structural example of the signal processing apparatus of this invention. It is a block diagram which shows the structure of a signal processing apparatus. It is a figure explaining the principle of the process of this invention. It is a figure explaining the process of this invention concretely. It is a figure explaining the evaluation value of the motion vector used in a signal processor. It is a flowchart explaining the frame frequency conversion process of a signal processing apparatus. It is a block diagram which shows the structure of an image interpolation part. It is a flowchart explaining an image interpolation process. It is a block diagram which shows the structure of a vector detection part. It is a figure explaining the gradient method used in a vector detection part. It is a figure explaining the iterative gradient method using an initial vector. It is a flowchart explaining a motion vector detection process. It is a block diagram which shows the structure of a shift initial vector allocation part. It is a figure explaining the concept of a shift initial vector. It is a figure explaining a shift initial vector concretely. It is a figure explaining a shift initial vector concretely. It is a figure explaining the determination method of a shift initial vector. It is a figure explaining the example of allocation of a shift initial vector. It is a figure explaining the example of a shift initial vector in case a motion vector competes. It is a flowchart explaining a shift initial vector allocation process. It is a block diagram which shows the structure of an initial vector selection part. It is a figure which shows the peripheral area | region of an initial vector. It is a figure which shows the candidate block of an initial vector. It is a flowchart explaining an initial vector selection process. It is a block diagram which shows the structure of an iterative gradient method calculating part. It is a block diagram which shows the structure of an effective pixel determination part. It is a block diagram which shows the structure of a gradient method calculating part. It is a figure explaining the detection target block and calculation block of a motion vector. It is a figure explaining the motion of the object in a detection target block. It is a figure explaining the state of the brightness | luminance in the detection target block which has the object of the same motion. It is a figure explaining the state of the brightness | luminance in the detection target block which has an object of a different motion. It is a figure explaining the example of the calculation block of a pixel unit process. It is a figure explaining the pixel unit process in the detection object block of FIG. It is a flowchart explaining an iterative gradient method calculation process. It is a flowchart explaining an effective pixel determination process. It is a flowchart explaining a gradient method calculation process. It is a flowchart explaining the iterative gradient method calculation process of a pixel unit. It is a block diagram which shows the structure of a vector allocation part. It is a figure explaining the concept of the 4-point interpolation process of this invention. It is a figure explaining the outline | summary of a vector allocation process. It is a figure explaining the example of the neighborhood pixel of the intersection of a motion vector and an interpolation frame. It is a figure explaining the evaluation method of the motion vector of an allocation candidate. It is a figure explaining the example of 4-point interpolation in vector allocation. It is a flowchart explaining a vector allocation process. It is a flowchart explaining the pixel position calculation process of step S705 of FIG. It is a flowchart explaining the allocation vector evaluation process of step S707 of FIG. It is a block diagram which shows the other structure of a vector allocation part. It is a figure explaining the evaluation method of the motion vector of the allocation candidate of the vector allocation part of FIG. It is a figure explaining "penetration". It is a flowchart explaining the vector allocation process of the vector allocation part of FIG. 51 is a flowchart for describing pixel position calculation processing in step S763 in FIG. 50. FIG. 51 is a flowchart for describing an allocation vector evaluation process in step S764 of FIG. 50. FIG. It is a block diagram which shows the structure of an allocation compensation part. It is a block diagram which shows the structure of a vector compensation part. It is a figure explaining the principle of an allocation compensation process. It is a figure explaining the principle of motion correlation. It is a figure explaining the structural example of the surrounding pixel of a focused pixel. It is a figure explaining the example of the compensation candidate vector of the motion vector of the focused pixel. It is a figure explaining the example of the compensation candidate vector of the motion vector of the focused pixel. It is a figure explaining the example of the compensation candidate vector of the motion vector of the focused pixel. It is a figure explaining the example which evaluates a compensation candidate vector. It is a figure explaining the example which evaluates a compensation candidate vector. It is a figure explaining the example which evaluates a compensation candidate vector. It is a figure explaining the example which selects a compensation candidate vector as a motion vector of a focused pixel. It is a flowchart explaining an allocation compensation process. It is a flowchart explaining a vector compensation process.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Signal processing apparatus, 11 CPU, 12 ROM, 13 RAM, 31 Magnetic disk, 32 Optical disk, 33 Magneto-optical disk, 34 Semiconductor memory, 51 Frame memory, 52 Vector detection part, 53 Detection vector memory, 54 Vector allocation part, 55 Allocation vector memory, 56 Allocation flag memory, 57 Allocation compensation unit, 58 Image interpolation unit, 101 Initial vector selection unit, 103 Iterative gradient method calculation unit, 104 Vector evaluation unit, 105 Shift initial vector allocation unit, 106 Evaluation value memory, 107 Shift initial vector memory, 201 allocation target position calculation unit, 202 evaluation value comparison unit, 203 shift initial vector replacement unit, 204 evaluation value replacement unit, 251 candidate vector position calculation unit, 252 detection vector acquisition unit, 253 shift initial vector acquisition unit , 254 Offset position calculation unit, 255 evaluation value calculation unit, 256 evaluation value comparison unit, 257 optimum candidate storage register, 401 mode selection unit, 402 selector, 403 effective pixel determination unit, 404 gradient method calculation unit, 405 delay unit, 411 Time pixel difference calculation unit, 412 pixel difference value determination unit, 413 effective pixel number counter, 414 gradient method continuation determination unit, 421 time pixel difference calculation unit, 422 pixel difference value determination unit, 423 horizontal / vertical pixel difference calculation unit, 424 gradient Integration unit, 425 vector calculation unit, 701 pixel information calculation unit, 702 evaluation value calculation unit, 703 focused pixel difference calculation unit, 704 vector evaluation unit, 705 vector selection unit, 711 pixel difference determination unit, 712 evaluation value determination unit, 751 Pixel information calculation section, 752 evaluation value calculation section, 753 vector evaluation section, 54 vector selecting unit, 801 allocated vector determining unit 802 vector allocating unit, 811 compensation processing unit, 812 evaluation value calculation unit, 821 memory

Claims (5)

In an image processing apparatus for detecting a motion vector on a first frame and generating a pixel value on a second frame based on the detected motion vector,
A difference calculating means for calculating a difference between a motion vector detected on the first frame and a surrounding motion vector on the first frame of the motion vector;
If the intersection of the motion vector and the second frame matches a pixel on the second frame, select the pixel that matches the intersection as a pixel of interest to which the motion vector is assigned; Pixel-of-interest selection means for selecting four pixels in the vicinity of the intersection as the pixel of interest when an intersection of the motion vector and the second frame does not coincide with a pixel on the second frame;
For each pixel of interest selected by the pixel-of- interest selection unit, the difference calculated by the difference calculation unit is calculated for other motion vectors that may be assigned to the pixel of interest. Difference determining means for determining whether or not the difference is smaller than,
The motion vector is allocated to the pixel of interest determined that the difference calculated by the difference determination unit is smaller than the difference calculated for the other motion vector, and the difference determination unit An image processing apparatus comprising: a vector allocating unit that allocates the other motion vector to the pixel of interest for which the calculated difference is not determined to be smaller than the difference calculated for the other motion vector . Motion vector detection means for detecting the motion vector on the first frame;
2. The pixel interpolation generation unit that generates by interpolating the pixel value on the second frame based on the motion vector allocated to the pixel of interest by the vector allocation unit. Image processing device. In an image processing method of an image processing apparatus for detecting a motion vector on a first frame and generating a pixel value on a second frame based on the detected motion vector,
The image processing apparatus is
A difference calculating step of calculating a difference between a motion vector detected on the first frame and a surrounding motion vector of the motion vector on the first frame;
If the intersection of the motion vector and the second frame matches a pixel on the second frame, select the pixel that matches the intersection as a pixel of interest to which the motion vector is assigned, A pixel-of-interest selection step of selecting, as the pixel of interest, four pixels in the vicinity of the intersection when an intersection of the motion vector and the second frame does not coincide with a pixel on the second frame;
For each target pixel selected by the process of selecting the target pixel, the difference calculated by the process of calculating the difference is calculated for other motion vectors that may be assigned to the target pixel. A difference determination step for determining whether the difference is smaller than the difference,
The motion vector is allocated to the pixel of interest for which the difference calculated by the process of the difference determination step is determined to be smaller than the difference calculated for the other motion vector, and the difference determination An image including a vector allocating step of allocating the other motion vector to the target pixel that has not been determined that the difference calculated by the processing of the step is smaller than the difference calculated for the other motion vector. Processing method. A computer of an image processing apparatus that detects a motion vector on the first frame and generates a pixel value on the second frame based on the detected motion vector.
A difference calculating step of calculating a difference between a motion vector detected on the first frame and a surrounding motion vector of the motion vector on the first frame;
If the intersection of the motion vector and the second frame matches a pixel on the second frame, select the pixel that matches the intersection as a pixel of interest to which the motion vector is assigned, A pixel-of-interest selection step of selecting, as the pixel of interest, four pixels in the vicinity of the intersection when an intersection of the motion vector and the second frame does not coincide with a pixel on the second frame;
For each target pixel selected by the process of selecting the target pixel, the difference calculated by the process of calculating the difference is calculated for other motion vectors that may be assigned to the target pixel. A difference determination step for determining whether the difference is smaller than the difference,
The motion vector is allocated to the pixel of interest for which the difference calculated by the process of the difference determination step is determined to be smaller than the difference calculated for the other motion vector, and the difference determination A process of assigning the other motion vector to the target pixel that has not been determined that the difference calculated by the processing of the step is smaller than the difference calculated for the other motion vector. A recording medium on which a program for performing the operation is recorded. A computer of an image processing apparatus that detects a motion vector on the first frame and generates a pixel value on the second frame based on the detected motion vector.
A difference calculating step of calculating a difference between a motion vector detected on the first frame and a surrounding motion vector of the motion vector on the first frame;
If the intersection of the motion vector and the second frame matches a pixel on the second frame, select the pixel that matches the intersection as a pixel of interest to which the motion vector is assigned, A pixel-of-interest selection step of selecting, as the pixel of interest, four pixels in the vicinity of the intersection when an intersection of the motion vector and the second frame does not coincide with a pixel on the second frame;
For each target pixel selected by the process of selecting the target pixel, the difference calculated by the process of calculating the difference is calculated for other motion vectors that may be assigned to the target pixel. A difference determination step for determining whether the difference is smaller than the difference,
The motion vector is allocated to the pixel of interest for which the difference calculated by the process of the difference determination step is determined to be smaller than the difference calculated for the other motion vector, and the difference determination A process including: a vector allocating step of allocating the other motion vector to the target pixel that has not been determined that the difference calculated by the processing of the step is smaller than the difference calculated for the other motion vector. A program to let you do.

Images