JP4512984B2 - Image processing apparatus and method, recording medium, and program - Google Patents

Description

  The present invention relates to an image processing apparatus and method, a recording medium, and a program, and more particularly to an image processing apparatus and method, a recording medium, and a program that can improve the accuracy of motion vector detection.

  In an image processing apparatus that converts a frame frequency of an image, a gradient method disclosed in Patent Document 1 is used as one method for detecting a motion vector of a block of interest. In this gradient method, a high-accuracy result can be obtained for a minute motion. However, when a motion is obtained in an actual moving image, the motion is too large to be practical. Therefore, as shown in Patent Document 1, this is dealt with by repeatedly performing the calculation of the gradient method.

  However, if the gradient method is simply repeated, it takes a long time to perform real-time processing. Therefore, in the gradient method calculation, the number of iterations of the gradient method is reduced by using the initial vector of the block of interest as the initial value. The initial vector of the block of interest is obtained from candidate vectors obtained by acquiring motion vectors detected in past frames or the like in the peripheral area of the block of interest.

JP 60-158786 A

  However, conventionally, in order to obtain a more optimal initial vector, the number of candidate vectors acquired in the peripheral region of the block of interest increases, and accordingly, the candidate vector should be obtained. Since the optimal initial vector has a high probability of mixing candidate vectors having different motions, the result is that an initial vector different from the motion to be obtained is given to the gradient method calculation, and the motion There has been a problem of reducing the accuracy of vector detection.

  In addition, with the increase in the number of candidate vectors acquired in the peripheral area of the block, there has been a problem that the amount of processing computation increases when obtaining an optimal initial vector, and hardware implementation is difficult .

  The present invention has been made in view of such a situation, and makes it possible to select an initial vector that improves the accuracy of motion vector detection by the gradient method.

The image processing apparatus according to the present invention includes a motion vector detection unit that detects a motion vector of a target block on a frame by a gradient method using an initial vector as an initial value, and a predetermined block around the target block including the target block. In the past frame, the current vector detected by the motion vector detecting means in the frame, the past vector detected by the motion vector detecting means in the past frame, and a block on the past frame at the same position as the predetermined block are the end points. Based on the spatial correlation with the block of interest among the shifted initial vectors having the same magnitude and direction as the motion vector detected by the motion vector detection means , the position of the predetermined block relative to the block of interest is determined. at least one motion vector predetermined according, the target block A candidate vector setting means for setting a candidate vector in the period vectors, when a plurality of candidate vectors is set at a predetermined block by the candidate vector setting means, the absolute value of the difference between the horizontal and vertical components of the plurality of candidate vectors is given Based on whether or not a plurality of candidate vectors are similar to each other based on whether or not the plurality of candidate vectors are similar to each other , the similarity determining means determines that the plurality of candidate vectors are similar to each other. In this case, if the predetermined block is in the area in contact with the target block, the shift initial vector of the predetermined block is selected as a candidate vector, and if the predetermined block is outside the area in contact with the target block, the current of the predetermined block is selected. vector or a past vector and a candidate vector selecting means for selecting a candidate vector from among the candidate vectors Based on the evaluation value is a correlation value of the region around the position on the two frames that is associated with the candidate vectors, and a initial vector selecting means for selecting the initial vector of the target block.

When selecting the target block in raster scan order, the candidate vector setting means sets the initial shift vector of the target block as the candidate vector, sets the current vector of the block on the left of the target block as the candidate vector, Predetermined block located in the raster scan order, the upper right block of the target block, the left side of the target block, three blocks to the left, the block on the third block, the right side of the target block, the third block, and the upper side of the three blocks A block initial shift vector and a current vector are set as candidate vectors, and are a predetermined block located after the target block in the raster scan order, the lower right block of the target block, and the left side of the target block by three blocks, 3 The block below the block, 3 of the target block A lock right, it is possible to set the shift initial vector and historical vector of under 3 block block as candidate vectors.

When selecting the target block in the raster scan order, the candidate vector setting means sets the initial shift vector of the target block as a candidate vector, and is a predetermined block positioned before the target block in the raster scan order. The next vector, the upper right block of the target block, two blocks to the left of the target block, two blocks above, and two blocks to the right of the target block, and the current vector of the block on the two blocks as a candidate vector Set and a predetermined block located after the target block in the raster scan order, the lower right block of the target block, the two blocks left of the target block, the blocks two blocks below, and the two blocks right of the target block there is, past base of under two blocks block It is possible to set the torque as a candidate vector.

Evaluation value calculating means for calculating an evaluation value that is a correlation value of a region centered on a position on the two frames associated with the candidate vector, and the initial vector selecting means is calculated by the evaluation value calculating means. The candidate vector having the smallest evaluation value can be selected as the initial vector of the block of interest.

The candidate vector setting means also sets priorities for selecting the initial vector of the target vector when there are a plurality of candidate vectors having the smallest evaluation value calculated by the evaluation value calculating means, and selects the target block in the raster scan order. In this case, the candidate vector of the target block is set to the first priority, and the candidate vector of the predetermined block in the area in contact with the target block is given priority over the candidate vector of the predetermined block outside the area in contact with the target block. Can be set high.

The candidate vector setting means sets the candidate vector of the nearest left-most block to the target block in the raster scan order in priority order 2, and sets the priority order first and second candidate vectors in the area in contact with the target block. The candidate vector of the block farthest from the center of the straight line connecting the centers of the blocks and positioned in the raster scan order is set to the priority order 3, and the candidates of the priority orders 1 to 3 are set in the area in contact with the block of interest. The candidate vector of the block farthest from the center of the space defined by the straight line connecting the centers of the vector blocks is set to priority number 4, and outside the area in contact with the block of interest, the block of interest is moved forward in raster scan order. predetermined blanking located after the candidate vectors of a given block located, in the raster scan order in the target block Tsu than the candidate vectors of the click can be set higher priority.

The candidate vector setting means sets the highest priority order of the candidate vector of a predetermined block that is positioned before the target block in the raster scan order and closest to the target block in the raster scan order within the area that is in contact with the target block. The block is positioned later in the raster scan order, the priority vector of the predetermined block farthest in the raster scan order is set to the lowest priority in the target block, and the target block is positioned earlier in the raster scan order outside the area in contact with the target block. It is possible to set the priority vector of the predetermined block to be higher in priority than the candidate vector of the predetermined block located later in the raster scan order in the target block.

  An allocation unit that allocates a motion vector detected by the motion vector detection unit to a pixel on an interpolation frame that is interpolated between frames, and a pixel value on the interpolation frame based on the motion vector allocated by the allocation unit It is possible to further comprise pixel interpolation generation means for generating by interpolation.

The image processing method of the present invention includes a motion vector detection step of detecting a motion vector of a target block on a frame by a gradient method using an initial vector as an initial value, and a predetermined block around the target block including the target block. The current vector detected by the motion vector detection step in the frame, the past vector detected by the motion vector detection step in the past frame, and the block on the past frame at the same position as the predetermined block are set as the end points. Based on the spatial correlation with the block of interest among the initial vectors of the same magnitude and the same direction as the motion vector detected by the motion vector detection step in the past frame, At least predetermined according to the position of the predetermined block 1 The motion vector and a candidate vector setting step of setting as a candidate vector of an initial vector of a target block, when a plurality of candidate vectors in a given block by the processing of the vector setting step is set, and the horizontal component of the plurality of candidate vectors based on whether the absolute value of the difference between the vertical component is smaller than a predetermined threshold value, the similarity determination step of determining whether there is a similarity in the plurality of candidate vectors, the plurality by processing similarity judgment step If it is determined that the candidate vectors are similar, if the predetermined block is in an area in contact with the target block, the shift initial vector of the predetermined block is selected as the candidate vector, and the predetermined block contacts the target block If it is outside the area, the current vector or past vector of the given block is used as the candidate vector. A candidate vector selecting step of selecting and from among the candidate vectors, based on the evaluation value is a correlation value of the region around the position on the two frames that is associated with the candidate vectors, the initial vector of the target block Selecting an initial vector.

The program recorded on the recording medium of the present invention uses an initial vector as an initial value, a motion vector detection step for detecting a motion vector of the target block on the frame by a gradient method, and a target block including the target block. The current vector detected by the motion vector detection step processing in the frame of the predetermined surrounding block, the past vector detected by the motion vector detection step processing in the past frame, and the past frame at the same position as the predetermined block Based on the spatial correlation with the block of interest , among the initial shifted vectors that have the same size and direction as the motion vector detected by the motion vector detection step in the past frame, with the block as the end point Depending on the position of a given block relative to the block At least one motion vector because obtained, and a candidate vector setting step of setting as a candidate vector of an initial vector of a target block, when a plurality of candidate vectors in a given block by the processing of the vector setting step is set, a plurality of candidate A similarity determination step for determining whether or not there is similarity among a plurality of candidate vectors based on whether or not the absolute value of the difference between the horizontal component and the vertical component of the vector is smaller than a predetermined threshold, and a similarity determination step If it is determined that a plurality of candidate vectors are similar by the processing in step (2), if the predetermined block is in an area in contact with the target block, the shifted initial vector of the predetermined block is selected as the candidate vector, and the predetermined block is selected. Is outside the area in contact with the block of interest, the current vector or past vector of the given block A torque, and a candidate vector selecting step of selecting a candidate vector from among the candidate vectors, based on the evaluation value is a correlation value of the region around the position on the two frames that is associated with the candidate vectors, attention Causing the computer to perform a process including an initial vector selection step of selecting an initial vector of the block .

The program of the present invention uses an initial vector as an initial value, a motion vector detection step for detecting a motion vector of a target block on the frame by a gradient method, and a frame of a predetermined block around the target block including the target block. In the past, the current vector detected by the motion vector detection step in the past, the past vector detected by the motion vector detection step in the past frame, and the block on the past frame at the same position as the predetermined block are the end points. Based on the spatial correlation with the block of interest among the shifted initial vectors having the same magnitude and direction as the motion vector detected by the motion vector detection step in the frame, At least one predetermined according to the position of the block A motion vector, and a candidate vector setting step of setting as a candidate vector of an initial vector of a target block, when a plurality of candidate vectors in a given block by the processing of the vector setting step is set, the horizontal component and the vertical of the plurality of candidate vectors Based on whether or not the absolute value of the component difference is smaller than a predetermined threshold, a plurality of candidates are determined by a similarity determination step for determining whether or not there are similarities among a plurality of candidate vectors, and processing of the similarity determination step If it is determined that the vectors are similar, if the predetermined block is in an area in contact with the target block, the shift initial vector of the predetermined block is selected as a candidate vector, and the predetermined block is in contact with the target block If it is outside, the current vector or past vector of the predetermined block is set as a candidate vector. A candidate vector selecting step of selecting Te, from among the candidate vectors, based on the evaluation value is a correlation value around the position on the two frames to be associated with regions in the candidate vector, the initial vector of the target block Causing the computer to perform processing including an initial vector selection step to select .

In the present invention, the current vector detected in the frame, the past vector detected in the past frame, and the block on the past frame at the same position as the predetermined block of the predetermined block around the target block including the target block Is a predetermined block for the target block based on the spatial correlation with the target block among the initial vectors of the same size and the same direction as the motion vector detected in the past frame. At least one motion vector predetermined according to the position of is set as a candidate vector of the initial vector of the block of interest. Then, when a plurality of candidate vectors are set in a predetermined block, similar to the plurality of candidate vectors based on whether or not the absolute value of the difference between the horizontal component and the vertical component of the plurality of candidate vectors is smaller than a predetermined threshold. If a plurality of candidate vectors are determined to be similar, if the predetermined block is in an area in contact with the block of interest, the shifted initial vector of the predetermined block is the candidate vector. If the predetermined block is outside the area in contact with the target block, the current vector or past vector of the predetermined block is selected as a candidate vector . Then, from the candidate vectors, an initial vector of the block of interest is selected based on an evaluation value that is a correlation value of a region centered on a position on two frames associated with the candidate vector .

  According to the present invention, it is possible to select an initial vector that improves the accuracy of motion vector detection by the gradient method. Furthermore, according to the present invention, it is possible to improve the accuracy of the motion vector by the gradient method, particularly in the vicinity of a boundary having motion.

  Embodiments of the present invention will be described below. Correspondences between constituent elements described in the claims and specific examples in the embodiments of the present invention are exemplified as follows. This description is to confirm that specific examples supporting the invention described in the claims are described in the embodiments of the invention. Accordingly, although there are specific examples that are described in the embodiment of the invention but are not described here as corresponding to the configuration requirements, the specific examples are not included in the configuration. It does not mean that it does not correspond to a requirement. On the contrary, even if a specific example is described here as corresponding to a configuration requirement, this means that the specific example does not correspond to a configuration requirement other than the configuration requirement. not.

  Further, this description does not mean that all the inventions corresponding to the specific examples described in the embodiments of the invention are described in the claims. In other words, this description is an invention corresponding to the specific example described in the embodiment of the invention, and the existence of an invention not described in the claims of this application, that is, in the future, a divisional application will be made. Nor does it deny the existence of an invention added by amendment.

The image processing apparatus according to claim 1 is a motion vector detection unit that detects a motion vector of a block of interest on a frame by a gradient method using an initial vector as an initial value (for example, the iterative gradient method computing unit 103 in FIG. 9). ), The current vector detected by the motion vector detecting means in the frame, the past vector detected by the motion vector detecting means in the past frame of the predetermined block around the target block including the target block, and the same as the predetermined block Of the initial shift vector having the same magnitude and the same direction as the motion vector detected by the motion vector detection means in the past frame, with the block on the past frame at the position as the end point , the spatial relationship with the block of interest Based on the correlation, it is predetermined according to the position of the predetermined block with respect to the block of interest. At least one motion vector, the candidate vector setting means for setting as a candidate vector of an initial vector of the target block (e.g., the candidate vector position operation unit 251 of FIG. 25), a plurality of candidate vectors in a given block by the candidate vector setting means Is set, the similarity is determined based on whether the absolute value of the difference between the horizontal component and the vertical component of the plurality of candidate vectors is smaller than a predetermined threshold value. If the plurality of candidate vectors are determined to be similar by the sex determination means (for example, the similarity determination unit 301 in FIG. 25) and the similarity determination means, if the predetermined block is within the area in contact with the block of interest The initial shift vector of a given block is selected as a candidate vector, and the area where the given block touches the block of interest If external, the current vector or past vector of a predetermined block, a candidate vector selecting means for selecting a candidate vector (e.g., a selector 203 in FIG. 25), from among the candidate vectors, two to be associated with the candidate vectors based of the position on the frame evaluation value is a correlation value of the region centered, and a initial vector selecting means for selecting the initial vector of the target block (e.g., evaluation value comparing unit 256 of FIG. 25).

The image processing apparatus according to claim 2, wherein the candidate vector setting means (for example, the candidate vector position calculation unit 251 in FIG. 25 that executes processing based on the candidate block in FIG. 26) selects the target block in the raster scan order. In this case, the initial shift vector of the target block (for example, the target block Bt in FIG. 26) is set as a candidate vector, the current vector of the block adjacent to the left of the target block is set as a candidate vector, Shift of the block on the upper right block of the target block, the left side of the three blocks of the target block, the block on the third block, the right side of the third block of the target block, and the block on the third block. Set the initial vector and the current vector as candidate vectors. Also a predetermined block located after the raster scan order, the block at the lower right of the target block, a third block left block of interest, under 3 block blocks, a 3 block right block of interest, under three blocks The initial shift vector and the past vector of the block are set as candidate vectors.

The image processing apparatus according to claim 3 , wherein the candidate vector setting unit (for example, the candidate vector position calculation unit 251 in FIG. 21 that executes processing based on the candidate block in FIG. 23) selects the block of interest in raster scan order. In this case, the shifted initial vector of the target block (for example, the target block Bt in FIG. 23) is set as a candidate vector, and is a predetermined block located before the target block in the raster scan order, and is a block adjacent to the left of the target block. , Set the current vector of the upper right block of the target block, two blocks to the left of the target block, two blocks on the right, and two blocks on the right of the target block, on the two blocks, as candidate vectors, is a predetermined block located after the raster scan order than the target block, wear Block at the bottom right of the block, a second block left block of interest, under the two blocks blocks, and a 2 block right block of interest, sets a past vector of under 2 block blocks as candidate vectors.

The image processing apparatus according to claim 4 is an evaluation value calculating unit that calculates an evaluation value that is a correlation value of a region centered on a position on two frames that are associated with each other by a candidate vector (for example, the evaluation value in FIG. 25). And an initial vector selection unit selects a candidate vector having the smallest evaluation value calculated by the evaluation value calculation unit as an initial vector of the block of interest.

The image processing apparatus according to claim 5 is configured such that the candidate vector setting unit (for example, the candidate vector position calculation unit 251 in FIG. 21 that executes the processing based on the candidate block in FIG. 23 or FIG. 27) is performed by the evaluation value calculation unit. When there are multiple candidate vectors with the smallest calculated evaluation value, the priority order for selecting the initial vector of the target vector is also set. When selecting the target block in raster scan order, the candidate vector of the target block is given priority. The priority is set to No. 1, and the candidate vector of the predetermined block in the area in contact with the block of interest is set higher in priority than the candidate vector of the predetermined block outside the area in contact with the block of interest.

The image processing apparatus according to claim 8 is an allocation unit that allocates the motion vector detected by the motion vector detection unit to pixels on an interpolation frame interpolated between frames (for example, the vector allocation unit 54 in FIG. 2). ) and, based on the motion vector allocated by the allocation means further comprises a pixel interpolation generating means for generating by interpolation pixel values on the interpolation frame (e.g., the image interpolation unit 58 of FIG. 2).

The image processing method according to claim 9 includes a motion vector detection step (for example, step S103 in FIG. 12) for detecting a motion vector of a target block on a frame by a gradient method using an initial vector as an initial value, Same as the current vector detected by the motion vector detection step processing in the frame, the past vector detected by the motion vector detection step processing in the past frame, and the predetermined block around the target block including the block The space with the block of interest among the initial shifted vectors that have the same magnitude and the same direction as the motion vector detected by the motion vector detection step in the past frame, with the block on the past frame at the position as the end point A predetermined block for the block of interest. Given at least one motion vector which is predetermined in accordance with the position of the click, the candidate vector setting step of setting as a candidate vector of an initial vector of the target block (e.g., step S312 in FIG. 28), by treatment of the vector setting step When a plurality of candidate vectors are set in the block, the plurality of candidate vectors have similarity based on whether the absolute value of the difference between the horizontal component and the vertical component of the plurality of candidate vectors is smaller than a predetermined threshold value. If the similarity determination step (for example, step S353 in FIG. 29) for determining whether or not a plurality of candidate vectors have similarities is determined by the processing of the similarity determination step, the predetermined block becomes the target block. If it is within the tangent region, select the shifted initial vector of the given block as the candidate vector If an area outside the predetermined block is in contact with the target block, the current vector or past vectors of a given block, the candidate vector selecting step of selecting a candidate vector (e.g., step S354 in FIG. 29), from among the candidate vectors An initial vector selection step for selecting the initial vector of the block of interest based on the evaluation value that is the correlation value of the region centered on the position on the two frames associated with the candidate vector (for example, step S318 in FIG. 28) ).

Note that the recording medium according to claim 10 and the program according to claim 11 are basically the same processing as the image processing method according to claim 9 described above, and are therefore repeated, so that the description thereof is omitted. To do.

  Hereinafter, embodiments of the present invention will be described with reference to the drawings.

  FIG. 1 shows a configuration example of a signal processing apparatus 1 to which the present invention is applied. The signal processing device 1 is composed of, for example, a personal computer. In FIG. 1, a CPU (Central Processing Unit) 11 executes various processes according to a program stored in a ROM (Read Only Memory) 12 or a storage unit 18. A RAM (Random Access Memory) 13 appropriately stores programs executed by the CPU 11 and data. The CPU 11, ROM 12, and RAM 13 are connected to each other by a bus 14.

  An input / output interface 15 is also connected to the CPU 11 via the bus 14. The input / output interface 15 is connected to an input unit 16 including a keyboard, a mouse, a microphone, and the like, and an output unit 17 including a display and a speaker. The CPU 11 executes various processes in response to commands input from the input unit 16. Then, the CPU 11 outputs an image, sound, or the like obtained as a result of the processing to the output unit 17.

  The storage unit 18 connected to the input / output interface 15 is composed of, for example, a hard disk, and stores programs executed by the CPU 11 and various data. The communication unit 19 communicates with an external device via the Internet or other networks. A program may be acquired via the communication unit 19 and stored in the storage unit 18.

  The drive 20 connected to the input / output interface 15 drives the magnetic disk 31, the optical disk 32, the magneto-optical disk 33, or the semiconductor memory 34 when they are mounted, and programs and data recorded there. Get etc. The acquired program and data are transferred to and stored in the storage unit 18 as necessary.

  Note that the signal processing device 1 can be, for example, a television receiver, an optical disc player, or a signal processing unit thereof.

  FIG. 2 is a block diagram showing the signal processing apparatus 1.

  It does not matter whether each function of the signal processing device 1 is realized by hardware or software. That is, each block diagram in this specification may be considered as a hardware block diagram or a software functional block diagram.

  In the signal processing device 1 having the configuration shown in FIG. 2, for example, an image of a progressive image signal (hereinafter referred to as a 24P signal) with a frame frequency of 24 Hz is input, and the input image (input image) has a frame frequency of 60 Hz. It is converted into an image of a progressive image signal (hereinafter referred to as 60P signal) and output. That is, FIG. 2 is a diagram illustrating a configuration of a signal processing device that is an image processing device.

  The input image of the 24P signal input to the signal processing device 1 is supplied to the frame memory 51, the vector detection unit 52, the vector allocation unit 54, the allocation compensation unit 57, and the image interpolation unit 58. The frame memory 51 stores input images in units of frames. The frame memory 51 stores a frame at time t one before the input image at time t + 1. The frame at time t stored in the frame memory 51 is supplied to the vector detection unit 52, the vector allocation unit 54, the allocation compensation unit 57, and the image interpolation unit 58. Hereinafter, the frame at time t on the frame memory 51 is referred to as frame t, and the frame of the input image at time t + 1 is referred to as frame t + 1.

  The vector detection unit 52 detects a motion vector between the target block of the frame t on the frame memory 51 and the target block of the frame t + 1 of the input image, and stores the detected motion vector in the detection vector memory 53. As a method for detecting the motion vector between the two frames, a gradient method or a block matching method is used. Details of the configuration of the vector detection unit 52 will be described later. The detection vector memory 53 stores the motion vector detected by the vector detection unit 52 in the frame t.

  The vector allocation unit 54 distinguishes the motion vector obtained on the frame t of the 24P signal from the frame of the 60P signal to be interpolated in the allocation vector memory 55 (hereinafter, the frame of the 60P signal is distinguished from the frame of the 24P signal). , Which is also referred to as an interpolation frame), and the allocation flag of the allocation flag memory 56 of the pixel to which the motion vector is allocated is rewritten to 1 (True). Details of the configuration of the vector allocation unit 54 will be described later.

  The allocation vector memory 55 stores the motion vector allocated by the vector allocation unit 54 in association with each pixel of the interpolation frame. The allocation flag memory 56 stores an allocation flag indicating the presence / absence of a motion vector to be allocated for each pixel of the interpolation frame. For example, an assignment flag of True (1) indicates that a motion vector is assigned to the corresponding pixel, and an assignment flag of False (0) indicates that a motion vector is not assigned to the corresponding pixel. Show.

  The allocation compensation unit 57 refers to the allocation flag of the allocation flag memory 56, and supplements the motion vector of the peripheral pixel of the target pixel with respect to the target pixel for which the motion vector has not been allocated by the vector allocation unit 54. Allocate on 55 interpolation frames. At this time, the allocation compensator 57 rewrites the allocation flag of the target pixel to which the motion vector is allocated to 1 (True). Details of the configuration of the allocation compensator 57 will be described later.

  The image interpolation unit 58 interpolates and generates the pixel value of the interpolation frame using the motion vector allocated to the interpolation frame in the allocation vector memory 55 and the pixel values of the frame t and the next frame t + 1. Then, the image interpolation unit 58 outputs the generated interpolated frame, and then outputs the frame t + 1 as necessary, thereby outputting the 60P signal image to a subsequent stage (not shown). In the following, the pixel value is also referred to as a luminance value as appropriate.

  FIG. 3 is a diagram for explaining the principle of processing in the signal processing apparatus 1 according to the present invention. In the example of FIG. 3, a dotted line represents a frame of a 24P signal that is input to the signal processing device 1 at times t, t + 1, and t + 2, and a solid line is represented by the signal processing device 1 from the input 24P signal. , The 60P signal interpolation frames at the generated times t, t + 0.4, t + 0.8, t + 1.2, t + 1.6, and t + 2.

  Generally, in order to convert a 24P signal into a 60P signal, 5/2 times as many frames are required. That is, five 60P signal images must be generated from two 24P signal images. At this time, the generated interpolated frame of the 60P signal has a time phase of 0.0, 0.4, 0.8, 1.2, and 1 on the 24P signal in order to equalize the frame interval. 6 is arranged at a position. Among these, 4 frames (frames of t + 0.4, t + 0.8, t + 1.2, and t + 1.6) except for one frame at time t with a time phase of 0.0 do not exist on the 24P signal. It is an image. Therefore, when the image of the 24P signal is input, the signal processing device 1 generates four interpolation frames from the two frames at the time t and the time t + 1 of the 24P signal. Therefore, the signal processing apparatus 1 outputs a 60P signal image composed of five frames at times t, t + 0.4, t + 0.8, t + 1.2, and t + 1.6.

  As described above, the signal processing apparatus 1 executes the process of converting the frame frequency from the 24P signal image to the 60P signal image.

  In principle, as described above, from the two frames at time t and time t + 1 of the 24P signal, five frames at times t, t + 0.4, t + 0.8, t + 1.2, and t + 1.6 are obtained. A frame of 60P signal is newly generated. Actually, in the example of FIG. 3, 60P of t, t + 0.4, and t + 0.8 is based on two frames at time t and time t + 1 of the 24P signal. A frame of the signal is generated, and a frame of the 60P signal of t + 1.2, t + 1.6, and t + 2 is generated based on the two frames at the times t + 1 and t + 2 of the 24P signal.

  FIG. 4 is a diagram for more specifically explaining the processing of the present invention. In the example of FIG. 4, a thick arrow represents a transition to each state, and an arrow T represents a time passage direction in the states 81 to 85. The states 81 to 85 are the frame t at the time t of the 24P signal, the frame t + 1 at the time t + 1 next to the time t, or the frame t and the frame t + 1 at the time of input / output to each unit constituting the signal processing device 1. The state of the interpolation frame F of the 60P signal produced | generated during is shown notionally. That is, actually, for example, a frame in which a motion vector is detected as shown in the state 82 is not input, and the frame and the motion vector are input separately.

  The state 81 represents the state of the frame t and the frame t + 1 of the 24P signal input to the vector detection unit 52. A black dot on the frame t in the state 81 represents a pixel on the frame t. The vector detection unit 52 detects the position where the pixel on the frame t in the state 81 moves in the frame t + 1 of the next time, and the movement is indicated as shown on the frame t in the state 82. It outputs as a motion vector corresponding to each pixel. As a method for detecting the motion vector between the two frames, a block matching method or a gradient method is used. At this time, when a plurality of motion vectors are detected in the pixel, the vector detection unit 52 obtains an evaluation value to be described later for each motion vector, and selects a motion vector based on the evaluation value.

  A state 82 represents the state of the frame t and the frame t + 1 input to the vector allocation unit 54. In the state 82, the arrow of each pixel of the frame t represents the motion vector detected by the vector detection unit 52.

  The vector allocating unit 54 extends the motion vector detected for each pixel of the frame t in the state 82 to the next frame t + 1 and sets it to a preset time phase (for example, t + 0.4 in FIG. 3). A position on a certain interpolation frame F is obtained. This is because, assuming that there is a constant motion between the frame t and the frame t + 1, the point where the motion vector passes through the interpolation frame F is the pixel position in that frame. Therefore, the vector allocating unit 54 allocates the passing motion vector to four neighboring pixels on the interpolation frame F in the state 83. At this time, depending on the pixel of the interpolation frame, there may be a case where no motion vector exists, or a plurality of motion vectors may be candidates for allocation. In the latter case, like the vector detection unit 52, the vector allocation unit 54 obtains an evaluation value for each motion vector and selects a motion vector to be allocated based on the evaluation value.

  A state 83 represents the state of the frame t and the frame t + 1 and the interpolation frame F to which the motion vector is assigned, which is input to the assignment compensation unit 57. In the interpolation frame F in the state 83, the pixels to which the motion vector is assigned by the vector assigning unit 54 and the pixels to which the motion vector is not assigned are shown.

  The allocation compensation unit 57 compensates for a pixel to which a motion vector in the state 83 is not allocated by using a motion vector allocated to a peripheral pixel of the pixel. This is because the motion vector of the peripheral pixel of the pixel of interest and the motion vector of the pixel of interest are similar if the assumption that the neighboring region of the pixel of interest has the same motion holds. As a result, a motion vector that is accurate to some extent is also given to pixels for which no motion vector has been assigned, and a motion vector is assigned to all pixels on the interpolation frame F in the state 84. In this case as well, since motion vectors of a plurality of surrounding pixels exist as candidates, the allocation compensator 57 obtains an evaluation value for each motion vector in the same manner as the vector allocation unit 54, and based on the evaluation values Select the motion vector to be assigned.

  A state 84 represents the state of the frame t and the frame t + 1 input to the image interpolation unit 58 and the state of the interpolation frame F in which motion vectors are assigned to all the pixels. Based on the motion vectors assigned to all these pixels, the image interpolation unit 58 can determine the positional relationship between the pixels on the interpolation frame F and the two frames t and t + 1. Therefore, the image interpolation unit 58 uses the motion vector allocated on the interpolation frame F and the pixel values of the frame t and the frame t + 1 to interpolate as indicated by the black dot of the interpolation frame F in the state 85. Pixel values on the frame F are generated by interpolation. Then, the image interpolation unit 58 outputs the generated interpolated frame, and then outputs the frame t + 1 as necessary, thereby outputting the 60P signal image to a subsequent stage (not shown).

  Next, with reference to FIG. 5, the evaluation value of the motion vector used in the signal processing apparatus 1 according to the present invention will be described. As described above with reference to FIG. 4, in each unit (the vector detection unit 52, the vector allocation unit 54, and the allocation compensation unit 57) of the signal processing device 1, an optimal motion vector is selected for subsequent processing. At this time, in each part of the signal processing device 1, as an evaluation value for the motion vector, a sum of absolute differences (DFD (Displaced Frame Difference)) representing a correlation value between the blocks shifted by the vector amount of interest of the two frames is used. Used.

In the example of FIG. 5, the m × n block centered on the pixel position p on the frame t at the time t and the vector amount of the motion vector v of interest are shifted from the pixel position p on the frame t + 1 at the time t + 1. Two blocks of m × n blocks centered on the pixel position p + v are shown. The difference absolute value sum DFD _t (p) obtained between these two blocks is expressed by the following equation (1).

・・・（１）

... (1)

Here, F _t (p) represents the luminance value at the pixel position p at time t, and m × n represents the DFD calculation range (block) for obtaining the sum of absolute differences. Since this difference absolute value sum represents a correlation value between DFD calculation ranges (blocks) in two frames, generally, the smaller the difference absolute value sum, the more the waveform of the block between the frames matches. Thus, it is determined that the reliability of the motion vector v is higher as the difference absolute value sum is smaller. Thereby, this sum of absolute differences is used when the most probable motion vector is selected from a plurality of candidates.

  Therefore, hereinafter, in each unit (vector detection unit 52, vector allocation unit 54, and allocation compensation unit 57) of signal processing device 1, as an evaluation value when a motion vector is selected, a difference is obtained unless otherwise specified. An absolute value sum (hereinafter referred to as an evaluation value DFD) is used.

  Next, the process of converting the frame frequency of the signal processing apparatus 1 will be described with reference to the flowchart of FIG.

  In step S1, the vector detection unit 52 inputs the pixel value of the frame t + 1 of the input image at the time t + 1 and the pixel t of the frame t at the time t immediately before the input image of the frame memory 51, and proceeds to step S2. At this time, the vector allocating unit 54, the allocation compensating unit 57, and the image interpolating unit 58 determine the pixel value of the frame t + 1 of the input image at the time t + 1 and the frame t at the time t immediately before the input image of the frame memory 51. Enter.

  In step S2, the vector detection unit 52 executes a motion vector detection process and proceeds to step S3. That is, the vector detection unit 52 detects a motion vector between the target block of the frame t on the frame memory 51 and the target block of the next frame t + 1 that is the input image, and the detected motion vector is detected by the detection vector memory 53. And proceed to step S3. As a method for detecting the motion vector between the two frames, a gradient method or a block matching method is used. When there are a plurality of motion vector candidates, an evaluation value DFD is obtained for each motion vector, and a highly reliable motion vector based on the obtained evaluation value DFD is detected. That is, in this case, the most probable motion vector is selected and detected in the target block for detecting the motion vector. Details of the motion vector detection process in step S2 will be described later.

  In step S3, the vector allocation unit 54 executes vector allocation processing, and proceeds to step S4. That is, in step S3, the vector allocation unit 54 allocates the motion vector obtained on the frame t to the pixel of interest on the interpolation frame to be interpolated in the allocation vector memory 55, and the pixel to which the motion vector is allocated. The allocation flag in the allocation flag memory 56 is rewritten to 1 (True). For example, an allocation flag that is True indicates that a motion vector is allocated to the corresponding pixel, and an allocation flag that is False indicates that a motion vector is not allocated to the corresponding pixel. When there are a plurality of motion vector candidates in each pixel, an evaluation value DFD is obtained for each motion vector, and a highly reliable motion vector based on the obtained evaluation value DFD is assigned. That is, in this case, the most probable motion vector is selected and assigned in the target pixel to which the motion vector is assigned. Details of the vector allocation processing in step S3 will be described later.

  In step S4, the allocation compensator 57 performs an allocation compensation process, and proceeds to step S5. That is, the allocation compensation unit 57 refers to the allocation flag in the allocation flag memory 56 in step S4, and for the pixel of interest for which no motion vector has been allocated by the vector allocation unit 54, the motion vector of the surrounding pixels of the pixel of interest. Is allocated on the interpolation frame of the allocation vector memory 55. At this time, the allocation compensator 57 compensates the motion vector and rewrites the allocation flag of the allocated pixel of interest to 1 (True). When there are a plurality of motion vectors of peripheral pixels, an evaluation value DFD is obtained for each motion vector, and a highly reliable motion vector is assigned based on the obtained evaluation value DFD. That is, in this case, the most probable motion vector is selected and assigned in the target pixel to which the motion vector is assigned. Details of the allocation compensation processing in step S4 will be described later.

  In step S5, the image interpolation unit 58 executes an image interpolation process. That is, in step S5, the image interpolation unit 58 uses the motion vector allocated to the interpolation frame of the allocation vector memory 55 and the pixel values of the frame t and the frame t + 1 to generate an interpolation frame pixel value. The process proceeds to step S6. Details of the image interpolation processing in step S5 will be described later. In step S6, the image interpolation unit 58 outputs the generated interpolation frame, and then outputs the frame t + 1 as necessary, thereby outputting the 60P signal image to a subsequent stage (not shown). The process proceeds to step S7.

  In step S7, the vector detection unit 52 determines whether or not all the frames have been processed. If the vector detection unit 52 determines that all the frames have not been processed, the process returns to step S1 and the subsequent processing is performed. repeat. On the other hand, when the vector detection unit 52 determines in step S7 that all the frames have been processed, the vector detection unit 52 ends the process of converting the frame frequency.

  As described above, the signal processing apparatus 1 according to the present invention detects a motion vector from a frame of a 24P signal input image, assigns the detected motion vector to a pixel on a frame of a 60P signal, and assigns the assigned motion vector. Based on the above, the pixel value on the frame of the 60P signal is generated. At this time, the signal processing apparatus 1 selects a motion vector with higher reliability based on the evaluation value DFD (sum of absolute differences) in each process, and outputs it to the subsequent stage. Therefore, in the signal processing device 1, it is possible to suppress a failure of the movement and the like and generate a more accurate image.

  Next, details of the configuration of the image interpolation unit 58 will be described.

  FIG. 7 is a block diagram illustrating a configuration of the image interpolation unit 58. The image interpolation unit 58 having the configuration shown in FIG. 7 interpolates and generates the pixel value of the interpolation frame using the motion vector allocated to the interpolation frame in the allocation vector memory 55 and the pixel values of the frame t and the frame t + 1. , 60P signal image processing is performed.

  In the example of FIG. 7, the frame t of the image at time t is input to the spatial filter 92-1, and the frame t + 1 of the image at time t + 1 is input to the spatial filter 92-2 and the buffer 95.

  The interpolation control unit 91 selects the pixel of the interpolation frame in the allocation vector memory 55, and based on the motion vector assigned to the selected pixel, the pixel on the interpolation frame, the two frames t and t + 1 The positional relationship (spatial shift amount) with each pixel is obtained. That is, the interpolation control unit 91 uses, based on the pixels of the interpolated frame, the position on the frame t associated with the motion vector and the position of the pixel on the frame t corresponding to the pixel of the interpolated frame. The spatial shift amount is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-1. Similarly, the interpolation control unit 91 uses, based on the pixels of the interpolation frame, the position on the frame t + 1 associated with the motion vector and the position of the pixel on the frame t + 1 corresponding to the pixel of the interpolation frame. Is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-2.

  Further, the interpolation control unit 91 obtains an interpolation weight between the frame t and the frame t + 1 based on a preset time phase (time) of the interpolation frame, and uses the obtained interpolation weight as a multiplier 93-1. And 93-2. For example, when the time of the interpolation frame is a time that is “k” away from the time t + 1 of the frame t + 1 and a time that is “1−k” away from the time t of the frame t (ie, the time of the interpolation frame is the time t and time t + 1 is “1−k”: generated at a time that is internally divided into “k”), the interpolation control unit 91 sets an interpolation weight of “1-k” in the multiplier 93-1. An interpolation weight of “k” is set in the multiplier 93-2.

  Spatial filters 92-1 and 92-2 are constituted by cubic filters, for example. The spatial filter 92-1 is a pixel value on the frame t corresponding to the pixel of the interpolation frame based on the input pixel value of the pixel on the frame t and the spatial shift amount supplied from the interpolation control unit 91. And the obtained pixel value is output to the multiplier 93-1. The spatial filter 92-2, based on the input pixel value of the pixel on the frame t + 1 and the spatial shift amount supplied from the interpolation control unit 91, the pixel value on the frame t + 1 corresponding to the pixel of the interpolation frame And the obtained pixel value is output to the multiplier 93-2.

  When the position of the pixel in the interpolation frame does not match the position of the pixel on frame t or frame t + 1 (that is, when the position of the pixel in the interpolation frame is a component equal to or less than the pixel in frame t or frame t + 1). The spatial filters 92-1 and 92-2 obtain the sum of the inverse ratios of the distances of the four surrounding pixels by using the pixel values of the four surrounding pixels at the position of the pixel of the interpolation frame in the frame t or the frame t + 1. Then, the pixel value on the frame corresponding to the pixel of the interpolation frame is obtained. That is, the pixel value at the position below the pixel is obtained by linear interpolation based on the distance to the surrounding four pixels (details will be described later).

  The multiplier 93-1 multiplies the pixel value on the frame t input from the spatial filter 92-1 by the interpolation weight “1-k” set by the interpolation control unit 91, and the weighted pixel value is The result is output to the adder 94. The multiplier 93-2 multiplies the pixel value on the frame t + 1 input from the spatial filter 92-2 by the interpolation weight “k” set by the interpolation control unit 91, and adds the weighted pixel value to the adder. Output to 94.

  The adder 94 generates the pixel value of the pixel of the interpolation frame by adding the pixel value input from the multiplier 93-1 and the pixel value input from the multiplier 93-2. The pixel value of the interpolation frame is output to the buffer 95. The buffer 95 buffers the input frame t + 1. The buffer 95 outputs the generated interpolation frame, and then outputs a buffered frame t + 1 as necessary based on a preset time phase (time) of the 60P frame. Thus, an image of the 60P signal is output to a subsequent stage (not shown).

  Details of the image interpolation processing of the image interpolation unit 58 configured as described above will be described with reference to the flowchart of FIG.

  In step S51, the interpolation control unit 91 calculates interpolation weights (for example, “k” and “1-k”) of the interpolation frame between the frame t and the frame t + 1 based on the time phase of the interpolation frame to be processed. The obtained interpolation weights are set in the multipliers 93-1 and 93-2, respectively, and the process proceeds to step S52. In step S52, the interpolation control unit 91 selects a pixel of the interpolation frame in the allocation vector memory 55, and proceeds to step S53. Note that the pixels on the interpolation frame are selected in the raster scan order from the upper left pixel of the frame.

  In step S53, based on the motion vector assigned to the selected pixel, the interpolation control unit 91 determines the positional relationship between the pixels on the interpolation frame and the two frames t and t + 1 (space shift amount). ) And the obtained spatial shift amounts are supplied to the spatial filters 92-1 and 92-2, respectively, and the process proceeds to step S54. Specifically, in step S53, the interpolation control unit 91 uses the pixel of the interpolation frame as a reference, the position on the frame t associated with the motion vector, and the frame t corresponding to the pixel of the interpolation frame. The spatial shift amounts are obtained from the pixel positions, and the obtained spatial shift amounts are supplied to the spatial filter 92-1. Similarly, the interpolation control unit 91 uses, based on the pixels of the interpolation frame, the position on the frame t + 1 associated with the motion vector and the position of the pixel on the frame t + 1 corresponding to the pixel of the interpolation frame. Is obtained, and the obtained spatial shift amount is supplied to the spatial filter 92-2.

  The pixel value of the frame t of the image at time t is input to the spatial filter 92-1, and the pixel value of the frame t + 1 of the image at time t + 1 is input to the spatial filter 92-2. In step S54, the spatial filters 92-1 and 92-2 determine the pixels of the interpolation frame based on the pixel values of the pixels on the input frames t and t + 1 and the spatial shift amount supplied from the interpolation control unit 91. The pixel values on each frame corresponding to are obtained, the obtained pixel values are output to the multipliers 93-1 and 93-2, and the process proceeds to step S55.

  In Step S55, the multipliers 93-1 and 93-2 weight the interpolation weight set by the interpolation control unit 91 to the pixel value on each frame input from the spatial filter 92-1 or 92-2, The weighted pixel value is output to the adder 94, and the process proceeds to step S56. That is, the multiplier 93-1 multiplies the pixel value on the frame t input from the spatial filter 92-1 by the interpolation weight “1-k” set by the interpolation control unit 91, and weighted pixel values. Is output to the adder 94. The multiplier 93-2 multiplies the pixel value on the frame t + 1 input from the spatial filter 92-2 by the interpolation weight “k” set by the interpolation control unit 91, and adds the weighted pixel value to the adder. Output to 94.

  In step S56, the adder 94 adds the pixel value weighted by the multiplier 93-1 and the pixel value weighted by the multiplier 93-2 to generate the pixel value of the pixel of the interpolation frame. The generated pixel value is output to the buffer 95, and the process proceeds to step S57. In step S57, the interpolation control unit 91 determines whether or not the processing for all the pixels on the interpolation frame is completed, and determines that the processing for all the pixels on the interpolation frame is not completed. If so, the process returns to step S52, and the subsequent processing is repeated. If the compatibility control unit 91 determines in step S57 that the processing for all the pixels on the interpolation frame has been completed, the compatibility control unit 91 ends the image interpolation processing.

  As described above, the pixel value of the interpolation frame is generated based on the motion vector assigned to the interpolation frame, and the interpolation frame is output by the buffer 95 in step S6 of FIG. If necessary, the frame t + 1 is output, so that an image of the 60P signal is output to the subsequent stage. Therefore, since the most probable motion vector is assigned to the pixel of the interpolation frame, an accurate interpolation frame can be generated.

  Next, details of the configuration of the vector detection unit 52 will be described.

  FIG. 9 is a block diagram showing the configuration of the vector detection unit 52. The vector detection unit 52 having the configuration shown in FIG. 9 detects a motion vector on the frame t by using the input frame t of the image at time t and the frame t + 1 of the image at time t + 1. The process of detecting the motion vector is executed for each predetermined block composed of a plurality of pixels.

  The initial vector selection unit 101 uses, as an initial vector V0 that is an initial value used in the gradient method, a high-reliability motion vector obtained from the detection result of the past motion vector for each predetermined block, as an iterative gradient method calculation unit. To 103. Specifically, the initial vector selection unit 101 uses a motion vector of a peripheral block obtained in the past stored in the detection vector memory 53 or a shift initial vector stored in the shift initial vector memory 107 as an initial vector. Select as a candidate vector. Then, the initial vector selection unit 101 obtains the evaluation value DFD of the candidate vector using the frame t and the frame t + 1, and selects the candidate vector having the highest reliability based on the obtained evaluation value DFD. And output as an initial vector V0. Details of the configuration of the initial vector selection unit 101 will be described later.

  The pre-filters 102-1 and 102-2 are configured by a low-pass filter and a Gaussian filter, respectively, and remove noise components of the frame t and the frame t + 1 of the input image, and output them to the iterative gradient method computing unit 103.

  The iterative gradient method computing unit 103 uses the initial vector V0 input from the initial vector selection unit 101 and the frame t and the frame t + 1 input via the prefilters 102-1 and 102-2, for each predetermined block. Then, the motion vector Vn is calculated by the gradient method. The iterative gradient method computing unit 103 outputs the initial vector V0 and the calculated motion vector Vn to the vector evaluation unit 104. In addition, the iterative gradient method computing unit 103 repeatedly performs the gradient method computation based on the motion vector evaluation result by the vector evaluation unit 104 to calculate the motion vector Vn. Details of the configuration of the iterative gradient method computing unit 103 will be described later.

  The vector evaluation unit 104 obtains the motion vector Vn−1 (or initial vector V0) from the iterative gradient method computing unit 103 and the evaluation value DFD of the motion vector Vn, and based on the obtained evaluation value DFD, the iterative gradient method The calculation unit 103 is controlled to repeatedly execute the calculation of the gradient method. Finally, a highly reliable one based on the evaluation value DFD is selected and stored in the detection vector memory 53 as a motion vector V. At this time, the vector evaluation unit 104 supplies the evaluation value DFD obtained for the motion vector V together with the motion vector V to the shifted initial vector allocation unit 105.

  When the motion vector V and the evaluation value DFD are supplied from the vector evaluation unit 104, the shifted initial vector allocation unit 105 shifts the motion vector passing through the target block on the next frame to the target block. Set as initial vector. In other words, the shifted initial vector allocation unit 105 shifts a motion vector of the same size and the same direction as the motion vector V, starting from the target block on the next frame at the same position as the end block of the motion vector V. Set as initial vector. Then, the shifted initial vector allocation unit 105 allocates the set shifted initial vector to the shifted initial vector memory 107 in association with the block of interest.

  Specifically, the shifted initial vector allocation unit 105 stores the evaluation value DFD of the motion vector V allocated as the shifted initial vector in the evaluation value memory 106 in association with the target block, and stores the same target block. It is compared with the evaluation value DFD of another motion vector V that passes (that is, the block of the past frame at the same position as the target block). Then, the shifted initial vector allocation unit 105 shifts the motion vector V determined to have high reliability based on the evaluation value DFD to the target block, and allocates it to the shifted initial vector memory 107 as the shifted initial vector of the target block. Details of the configuration of the shifted initial vector allocation unit 105 will be described later.

Next, the principle of the gradient method used in the vector detection unit 52 will be described. First, a luminance value of a pixel represented by coordinates (x, y, t) using a horizontal, vertical, and time axis in a moving image is assumed to be g (x, y, t). Here, when the pixel of interest (x ₀ , y ₀ , t ₀ ) is displaced by (dx, dy, dt) during a minute time, the horizontal (vertical) and time axis gradients (difference differences) are respectively expressed as gx. When expressed as (x ₀ , y ₀ , t ₀ ), gy (x ₀ , y ₀ , t ₀ ), and gt (x ₀ , y ₀ , t ₀ ), the luminance value of the pixel after displacement is approximated by Taylor expansion. Is expressed by the following equation (2).

g (x ₀ + dx, y ₀ + dy, t ₀ + dt)
≒ g (x ₀ , y ₀ , t ₀ ) + gx (x ₀ , y ₀ , t ₀ ) dx
+ Gy (x ₀ , y ₀ , t ₀ ) dy + gt (x ₀ , y ₀ , t ₀ ) dt
... (2)

  Here, when a pixel of interest in a moving image moves by horizontal vx and vertical vy after one frame (hereinafter, expressed as (vx, vy)), the luminance value of the pixel is expressed by the following equation (3). Is done.

_{g (x 0 + vx, y} 0 + vy, t 0 +1) = g (x 0, y 0, t 0)
... (3)

  When Expression (2) is substituted into Expression (3), it is expressed by the following Expression (4).

gx (x ₀ , y ₀ , t ₀ ) vx + gy (x ₀ , y ₀ , t ₀ ) vy
+ Gt (x ₀ , y ₀ , t ₀ ) = 0
... (4)

  Since the equation (4) is a two-variable equation of vx and vy, the solution cannot be obtained with a single equation for the target pixel. Therefore, as described below, a block that is a peripheral region of the target pixel is considered as one processing unit, and it is assumed that all pixels in the block (peripheral region) have the same movement (vx, vy). Build a similar formula. Assuming an assumption, an expression for the number of neighboring pixels is obtained for two variables. Therefore, these equations are combined to obtain (vx, vy) that minimizes the sum of squares of motion compensation frame differences of all pixels in the block.

  When the pixel (x, y, t) moves by (vx, vy) by one frame, the motion compensation inter-frame difference d is expressed by the following equation (5).

d = g (x + vx, y + vy, t + 1) −g (x, y, t)
= Δxvx + Δyvy + Δt
... (5)

  In equation (5), Δx = gx (x, y, t), which represents a horizontal gradient, Δy = gy (x, y, t), which represents a vertical gradient, Δt = gt ( x, y, t), which represents the gradient in the time direction. Using these, assuming that the square sum of the difference between motion compensation frames is E, it is expressed by Equation (6).

E = Σd ²
= Σ (Δx ² vx ² + Δy ² vy ² + 2ΔxΔyvxvy
+ 2ΔxΔtvx + 2ΔyΔtvy + Δt ² )
= Vx ² ΣΔx ² + vy ² ΣΔy ² +2 vxvy ΣΔxΔy
+ 2vxΣΔxΔt + 2vyΣΔyΔt + ΣΔt ²
... (6)

  Here, E becomes the minimum (vx, vy) when the partial differential value in each variable becomes 0, that is, when the condition of δE / δvx = δE / δvy = 0 is satisfied. Thus, the following equations (7) and (8) are obtained.

vxΣΔx ² + vyΣΔxΔy + ΣΔxΔt = 0
... (7)
vyΣΔy ² + vxΣΔxΔy + ΣΔyΔt = 0
... (8)

  From these equations (7) and (8), (vx, vy) which is the desired motion can be obtained by calculating the following equation (9).

・・・（９）

... (9)

  Here, it demonstrates concretely with reference to FIG. In the example of FIG. 10, an arrow X indicates the horizontal direction, and an arrow Y indicates the vertical direction. In addition, an arrow T indicates the direction of time passage from the frame t at the time t in the back right to the frame t + 1 at the time t + 1 in the left front. In the example of FIG. 10, each frame shows only an area of 8 pixels × 8 pixels used for the gradient method calculation as a peripheral area (block) of the pixel of interest p.

  In the frame t, when the motion vector V (vx, vy) of the pixel of interest p, which is the fifth pixel down from the upper left pixel and the fifth pixel on the right, is obtained using the gradient method described above, the motion vector V (vx , Vy) is the difference in luminance (ie, gradient) Δx and Δy between adjacent pixels px and py obtained for the x and y directions of the pixel of interest p, and the same phase of the pixel of interest p obtained in frame t + 1. A difference in luminance (gradient) Δt in the time direction with respect to the pixel q located is obtained for all pixels in the peripheral region (8 pixels × 8 pixels) of the pixel of interest p, and the difference between them is expressed by Equation (9). It can obtain | require by calculating using.

  That is, the gradient method calculates gradients Δx, Δy, and Δt between two frames, and statistically calculates the motion vector V (vx, Δt from the obtained Δx, Δy, and Δt using the sum of squares of differences. vy).

  Generally, in a motion vector detection method using such a gradient method, a highly accurate result can be obtained for a minute motion. However, when trying to obtain a motion in an actual moving image, this gradient method is not practical because the amount of motion is too large. Correspondingly, a method of repeating this gradient method a plurality of times is conceivable. By repeatedly executing the gradient method, the amount of motion obtained by each calculation converges, so that a correct motion is gradually obtained.

  However, it is not practical from the viewpoint of calculation time when trying to perform real-time processing only by repeating the gradient method. Therefore, the vector detection unit 52 reduces the number of iterations of the gradient method by using, as an initial value, an initial vector obtained based on the motion of surrounding pixels in the past frame and the current frame. In other words, if a rough motion is calculated by adding an offset in advance from the pixel of interest that is the starting point of the motion to the destination indicated by the initial vector, an operation using the gradient method is performed from the position where the offset is added. Fine adjustment including movement below the pixel can be performed. Thereby, an accurate motion vector can be detected without increasing the calculation time.

  FIG. 11 is a diagram specifically explaining the iterative gradient method executed using the initial vector. In the example of FIG. 11, the arrow T indicates the passage of time from the frame t at the time t on the left front side to the frame t + 1 at the time t + 1 on the right back in the drawing. A block centered on each pixel p, q0, q1, q2, and q3 represents a peripheral region (block) used for the gradient method calculation of that pixel.

  In the case of the example in FIG. 11, the initial vector v0 obtained in advance is offset (moved) instead of the pixel q0 located in the same phase of the target pixel p in the frame t + 1 with respect to the target pixel p in the frame t. The first gradient method calculation is performed with the position (pixel) q1 calculated in this way as the starting point, and as a result, a motion vector v1 is obtained.

  Next, the second gradient method calculation is performed with the position (pixel) q2 calculated by offsetting v0 + v1 from the pixel q0 as a starting point, and as a result, a motion vector v2 is obtained. Thereby, the motion vector V is finally obtained as Expression (10).

V = v0 + v1 + v2
... (10)

  As described above, by performing the iterative gradient method using the initial vector, it is possible to obtain a highly accurate motion vector while reducing the calculation time.

  Next, details of the motion vector detection process will be described with reference to the flowchart of FIG. The vector detection unit 52 receives the input frame t of the image at time t and input frame t + 1 of the image at time t + 1.

  In step S101, the initial vector selection unit 101 selects a block to be processed on the frame t as a target block, and the process proceeds to step S102. On the frame, the processing is executed in the raster scan order from the upper left block.

  In step S102, the initial vector selection unit 101 performs an initial vector selection process. In step S101, the initial vector selection unit 101 selects a motion vector with high reliability from the detection result of the past motion vector for each predetermined block, and uses the selected motion vector as an initial value used in the gradient method. The initial vector V0 is output to the iterative gradient method computing unit 103, and the process proceeds to step S103.

  That is, the initial vector selection unit 101 obtains the motion vector of the peripheral blocks obtained in the past gradient method calculation evaluation process (step S103 described later) and stored in the detection vector memory 53, and the past shifted initial vector allocation process (described later). In step S104), the shifted initial vector stored in the shifted initial vector memory 107 is selected as an initial vector candidate vector. Then, the initial vector selection unit 101 obtains the evaluation value DFD of the candidate vector using the frame t and the frame t + 1, and selects a candidate vector having a high reliability based on the obtained evaluation value DFD. The selected candidate vector is output as the initial vector V0. Details of the initial vector selection process in step S102 will be described later.

  In step S103, the iterative gradient method calculation unit 103 and the vector evaluation unit 104 execute an iterative gradient method calculation evaluation process (also referred to as an iterative gradient method calculation process), and the process proceeds to step S104. Specifically, in step S103, the iterative gradient method computing unit 103 receives the initial vector V0 input from the initial vector selecting unit 101, the frame t and the frame input via the prefilters 102-1 and 102-2. Based on the motion vector evaluation result by the vector evaluation unit 104 using t + 1, the gradient method is repeatedly calculated to calculate the motion vector Vn. Further, the vector evaluation unit 104 obtains the motion vector Vn−1 from the iterative gradient method computing unit 103 and the evaluation value DFD of the motion vector Vn, and selects the most reliable one based on the obtained evaluation value DFD. And stored in the detected vector memory 53 as a motion vector V. At this time, the vector evaluation unit 104 supplies the evaluation value DFD obtained for the motion vector V together with the motion vector V to the shifted initial vector allocation unit 105. Details of the iterative gradient method calculation processing in step S103 will be described later.

  In step S104, the shifted initial vector allocation unit 105 executes a shifted initial vector allocation process, and proceeds to step S105. When the motion vector V and its evaluation value DFD are supplied from the vector evaluation unit 104, the shifted initial vector allocation unit 105 shifts the motion vector passing through the target block on the next frame to the target block in step S104. The shifted initial vector is set. That is, in other words, a motion vector having the same size and the same direction as the motion vector V is set as the initial shift vector, starting from the block of interest on the next frame at the same position as the end block of the motion vector V. . Then, the shifted initial vector allocation unit 105 allocates the set shifted initial vector to the shifted initial vector memory 107 in association with the block of interest.

  Specifically, the shifted initial vector allocation unit 105 stores the evaluation value DFD of the motion vector V allocated as the shifted initial vector in the evaluation value memory 106 in association with the target block, so that the same target Compared with the evaluation value DFD of another motion vector V that passes through the block (that is, the block of the past frame at the same position as the target block is the end point), the motion determined to have high reliability based on the evaluation value DFD The vector V is shifted to that block and set as a shifted initial vector, and is allocated to the shifted initial vector memory 107 in correspondence with the shifted block. Details of the configuration of the shifted initial vector allocation unit 105 will be described later.

  In step S105, the initial vector selection unit 101 determines whether or not all blocks have been processed in the frame t. If it is determined that all blocks have not been processed, the initial vector selection unit 101 returns to step S101. The subsequent processing is repeated. In step S105, the initial vector selection unit 101 determines that the motion vectors V have been detected in all the blocks on the frame t when it is determined that all the blocks have been processed in the frame t. Then, the motion vector detection process ends.

  As described above, an initial vector is selected from motion vectors detected in the past, an iterative gradient method is calculated based on the selected initial vector, and a repeated motion vector is calculated. Among them, a motion vector having a high reliability (that is, most likely) based on the evaluation value DFD is detected. As a result, the motion vector V corresponding to all the blocks on the frame t is stored in the detection vector memory 53.

  Next, the details of the configuration of the shifted initial vector allocation unit 105 will be described.

  FIG. 13 is a block diagram showing a configuration of the shifted initial vector allocating unit 105. The shifted initial vector allocation unit 105 having the configuration shown in FIG. 13 sets a shifted initial vector that is a candidate vector of the initial vector based on the motion vector V detected by the vector evaluation unit 104 in the previous (past) frame, A process of assigning to the shifted initial vector memory 107 is performed. The shifted initial vector allocation unit 105 receives the motion vector V detected by the vector evaluation unit 104 and the evaluation value DFD of the motion vector V.

  The allocation target position calculation unit 201 detects the position of the block through which the motion vector V detected by the vector evaluation unit 104 passes on the frame at the next time (that is, the end point of the motion vector V detected on the current frame). The position of the block on the next frame at the same position as the block) is calculated, and the calculated position of the block is supplied to the evaluation value memory 106 and the shifted initial vector replacement unit 203.

  When the evaluation value comparison unit 202 receives the motion vector V and the evaluation value DFD of the motion vector V, the evaluation value comparison unit 202 reads the evaluation value DFD of the block position from the allocation target position calculation unit 201 from the evaluation value memory 106. . Then, the evaluation value comparison unit 202 compares the evaluation value DFD read from the evaluation value memory 106 with the evaluation value DFD of the motion vector V detected by the vector evaluation unit 104, and detects the detected motion vector V When the evaluation value DFD is smaller (that is, the reliability is higher), the shifted initial vector replacement unit 203 is controlled to determine the block of the block supplied by the shifted initial vector allocation unit 105 of the shifted initial vector memory 107. The position initial shift vector is rewritten with the motion vector V determined to have high reliability based on the evaluation value DFD. At the same time, the evaluation value comparison unit 202 controls the evaluation value replacement unit 204, and the evaluation value memory 106 selects the evaluation value DFD of the position of the block selected by the allocation target position calculation unit 201 in the evaluation value memory 106. Rewrite with evaluation value DFD.

  The shift initial vector replacement unit 203 uses the motion vector V (that is, the motion) supplied from the evaluation value comparison unit 202 as the shift initial vector of the block position supplied from the allocation target position calculation unit 201 in the shift initial vector memory 107. The motion vector having the same magnitude as the vector V and the same direction) is rewritten. The evaluation value replacement unit 204 converts the evaluation value DFD of the position of the block selected by the allocation target position calculation unit 201 into the evaluation value DFD of the motion vector V in the evaluation value memory 106 under the control of the evaluation value comparison unit 202. Rewrite with.

  The evaluation value memory 106 stores, for each block, an evaluation value DFD of a shifted initial vector candidate assigned to each block on the next frame. The shifted initial vector memory 107 stores the motion vector having the smallest evaluation value DFD (that is, the most reliable) in each block in the next frame as a shifted initial vector in association with the block.

  FIG. 14 is a diagram for describing one-dimensionally the shifted initial vectors allocated by the shifted initial vector allocating unit 105. In the example of FIG. 14, a frame t−1 at time T = t−1, a frame t at time T = t, and a frame t + 1 at time T = t + 1 are shown in order from the top. The partition indicates a block boundary in each frame.

  In the case of the example in FIG. 14, the motion vector detected in the block B on the frame t−1 is the motion vector V (solid arrow in the figure), and the motion on the frame t by the motion vector V starting from the block B. A compensation destination (hereinafter also referred to as an offset destination) is a block Bt. A motion vector obtained by shifting the motion vector V on the frame t-1 to the block Bt on the frame t is defined as a shifted initial vector SV (dotted line arrow in the figure). In this case, the offset destination represents the block Bt on the frame t at the same position as the end block of the motion vector V detected on the frame t−1, and the shift means the frame t The start point of a motion vector having the same magnitude and the same direction as the motion vector V detected on -1 is a block Bt on the frame t at the same position as the end point block of the motion vector V on the frame t-1. That means. That is, the shifted initial vector SV is the same size and the same direction as the motion vector V, starting from the block Bt on the frame t at the same position as the end block of the motion vector V detected on the frame t−1. It is a motion vector.

  In general, there is a certain degree of continuity in the amount of movement of the moving object between consecutive frames, and the change in the amount of movement is considered to be small. Therefore, in the example of FIG. When the motion is at or close to it, the motion vector in the block Bt is often the motion vector V or a motion vector close thereto.

  Therefore, when this shifted initial vector SV is used as a candidate for an initial vector given to the gradient method calculation when detecting a motion vector of the block of interest Bt on the frame t, only the motion vectors of the peripheral blocks are used as initial vector candidates. It is possible to obtain a more appropriate initial vector for motion vector detection by the gradient method calculation than in the case of using as.

  The above will be specifically described with reference to FIGS. 15 and 16. In FIGS. 15 and 16, nine blocks are shown on each frame.

  FIG. 15 shows an example of a motion vector of a block through which a large motion object passes and its neighboring blocks. In the case of the example in FIG. 15, the image object of a certain object O1 consisting of one block is a motion vector V1 of a large motion, and is on the frame t from the rightmost block (the ninth from the left) on the frame t-1. It moves to the leftmost block on frame t + 1 via the fifth block from the left. On the other hand, the image object of the object O2 composed of four blocks is a motion vector V2 having a motion smaller than the motion vector V1, and the fourth to fourth blocks from the third to sixth from the left on the frame t−1 and 4 from the left on the frame t. Through the 4th to 7th blocks, the frame moves to the 5th to 8th 4 blocks from the left on the frame t + 1. That is, in the fifth block from the left on the frame t, the image object of the object O1 and the image object of the object O2 intersect each other with the image object of the object O1 in front.

  At this time, in order to detect the motion vector of the fifth block from the left on which the image object of the object O1 passes by the motion vector V1 on the frame t, the initial vector is extracted from the motion of the peripheral blocks. Since the image object of the object O2 passes through the block (the fourth or sixth block from the left) with a motion vector V2 different from that of the image object of the object O1, there is no appropriate motion vector. On the other hand, the shift initial vector SV1 (dotted arrow in the figure) is the fifth motion vector from the left on the frame t, even in such a case, with the motion vector V1 detected on the past frame T-1. Motion vector shifted to a block (that is, the same size and the same direction as the motion vector V1, starting from a block on the frame t at the same position as the end block of the motion vector V1 detected on the frame t-1) Therefore, the motion vector can be reliably acquired.

  FIG. 16 shows an example of a motion vector of a block at the boundary of a moving object and its surrounding blocks. In the case of the example of FIG. 16, the image object of a certain object O3 consisting of one block is a motion vector V3, and the sixth block from the left on the frame t-1 to the fifth block from the left on the frame t. Via, it moves to the fourth block from the left on the frame t + 1. On the other hand, the image object of the object O4 composed of 9 blocks or more is a motion vector V4 different from the motion vector V1, and from the 9 blocks on the frame t-1 to the 2nd to 9th 8 blocks from the left on the frame t. Via, it moves to the third to ninth seventh blocks from the left on the frame t + 1. That is, the image object of the object O3 is in front of the image object of the object O4 in the sixth block from the left of the frame t-1, the fifth block from the left on the frame t, and the fourth block from the left of the frame t + 1. Is going through.

  At this time, in order to detect the motion vector of the fifth block from the left on which the image object of the object O3 passes with the motion vector V3 on the frame t, the initial vector is extracted from the motion of the peripheral blocks. Since the image object of the object O4 passes through the block (the fourth or sixth block from the left) with a motion vector V4 different from that of the image object of the object O3, there is no appropriate motion vector. However, in this case as well, as in the case of FIG. 15, the shifted initial vector SV3 (indicated by a dotted arrow in the figure) represents the motion vector V3 detected on the past frame T-1 as the right on the frame t. To the fifth block (ie, the same size as the motion vector V3, starting from the block on the frame t at the same position as the end block of the motion vector V3 detected on the frame t-1) The motion vectors in the same direction in FIG.

  As described above, at the boundary of an object having a large movement as shown in FIG. 15 or an object moving as shown in FIG. There are many cases where the optimal motion vector does not exist as the initial vector of the block of interest in the peripheral block. On the other hand, the initial shift vector has the same size and the same size as the motion vector starting from the block on the frame t at the same position as the end block of the motion vector detected on the past frame t-1. Since it is a motion vector in the direction, it does not depend on the spatial distance between the blocks, so even if there is no appropriate motion vector as the initial vector of the block of interest in the surrounding blocks, it can be obtained reliably. It can be used as an optimal candidate.

  Here, returning to FIG. 14, a method for determining the shifted initial vector will be described. In the example of FIG. 14, with reference to the target block Bt on the frame t on which the motion vector detection process is performed, the motion vector passing from the frame t-1 through the target block Bt (that is, at the same position as the block Bt on the frame t). If an attempt is made to search for a motion vector whose end point is a block of a certain frame t-1, the motion vectors of all the blocks on the frame t-1 must be searched, and the processing becomes enormous. Therefore, in this case, when detecting a motion vector, it is necessary to deal with a huge amount of processing, and it is difficult to implement (hardware).

  Therefore, the shifted initial vector allocating unit 105 detects the position of the block through which the motion vector V detected in the block of interest Bt passes on the frame t at the next time (that is, when detecting the motion vector in the frame t−1). A block on the frame t at the same position as the end point block of the motion vector V) is obtained, and the motion vector V is assigned in advance as a shift initial vector SV for the block Bt. As a result, it is possible to reduce the amount of calculation for searching for the initial shift vector necessary for detecting the motion vector on the frame t, and to realize hardware implementation. It can be used as a candidate for an initial vector.

  With reference to FIG. 17, a method for determining the shifted initial vector by shifted initial vector allocating unit 105 will be specifically described. In the example of FIG. 17, the arrow T indicates the passage of time from the frame t-1 at time t-1 on the left front side to the frame t at time t on the right back in the drawing. A circle on each frame represents a pixel.

  On the frame t−1, the block B0 is composed of 4 × 4 pixels. Assuming that the motion vector Vb detected in the block B0, an offset destination (a block on the frame t at the same position as the end block of the motion vector Vb of the block B0) A0 obtained by offsetting (motion compensation) the block B0 on the frame t is In general, there is an overlap with four blocks Bt1 to Bt4 configured by 4 × 4 pixels on the frame t. Therefore, in the example of FIG. 17, all of the four blocks Bt1 to Bt4 that overlap with the offset destination A0 are set as target blocks to which the motion vector Vb is allocated as the shifted initial vector.

  The motion vector detected in the previous frame as described above is assigned as the shifted initial vector. In this case, the position of the block to be allocated depends on the magnitude and direction of the motion vector. Therefore, depending on the distribution of motion vectors detected on the frame t−1, as shown in FIG. 18, on the frame t, a block that is a target for allocation of a plurality of shift initial vectors, or conversely, a shift Blocks that do not have an initial vector assignment will occur.

  FIG. 18 shows an example of allocation of shifted initial vectors. In the example of FIG. 18, the partition of each frame indicates the boundary of the block, and is composed of blocks B1 to B5 on the frame t−1 from the left in the figure, and on the frame t on the left in the figure. Thus, blocks Bt1 to Bt5 are shown.

  In the block B1, the motion vector V1 is detected, and the offset destination A1 of the block B1 overlaps with the blocks Bt1 and Bt2 on the frame t. In block B2, motion vector V2 is detected, and offset destination A2 of block B2 overlaps with blocks Bt1 and Bt2 on frame t. In block B3, motion vector V3 is detected, and offset destination A3 of block B3 overlaps with blocks Bt4 and Bt5 on frame t. In block B4, motion vector V4 is detected, and offset destination A4 of block B4 overlaps with blocks Bt4 and Bt5 on frame t. In the block B5, the motion vector V5 is detected, and the offset destination A5 of the block B5 overlaps with the block Bt5 on the frame t and an adjacent block (not shown).

  That is, in the example of FIG. 18, one of the motion vectors V1 and V2 is allocated as the shifted initial vector to the blocks Bt1 and Bt2 on the frame t. Also, one of the motion vectors V3 and V4 is assigned as the initial shift vector to the block Bt4 on the frame t, and any one of the motion vectors V3, V4, and V5 is assigned to the block Bt5 on the frame t. Are assigned as shifted initial vectors. However, in the block Bt3 on the frame t, there is no motion vector that is a candidate for the shifted initial vector, and the shifted initial vector is not allocated.

  Therefore, the shifted initial vector allocating unit 105 allocates the zero vector as the shifted initial vector to a block such as the block Bt3 where allocation of the shifted initial vector does not occur. On the other hand, the shifted initial vector allocating unit 105 assigns a motion vector having high reliability based on the above-described evaluation value DFD to a block in which a plurality of motion vectors compete, such as the blocks Bt1, Bt2, Bt4, and Bt5. The selected motion vector is assigned to each block as a shifted initial vector.

  In the example of FIG. 18, for convenience of explanation, the offset destination A is shown slightly shifted from the frame t.

  FIG. 19 is a diagram illustrating an example of competing motion vectors. In the example of FIG. 19, the motion vectors V1 and V2 detected in the blocks B1 and B2 on the frame t-1 respectively pass the block Bt on the frame t. That is, the block Bt is a block on the frame t at the same position as the end point on the frame t-1 of the motion vector V1, and is a block on the frame t at the same position as the end point on the frame t-1 of the motion vector V2. is there. Thus, when a plurality of motion vectors compete for the same block, the evaluation value DFD1 for the motion vector V1 is compared with the evaluation value DFD2 for the motion vector V2, and the evaluation value DFD is smaller (that is, , With higher reliability) is selected as the shifted initial vector SV2 corresponding to the block Bt. As a result, a highly reliable shifted initial vector based on the evaluation value DFD can be obtained.

  Next, the details of the shifted initial vector allocation processing will be described with reference to the flowchart of FIG. In the preceding stage, when the vector evaluation unit 104 detects the motion vector V of the block of interest on the frame t−1, the vector evaluation unit 104 uses the detected motion vector V and the evaluation value DFD obtained for the motion vector V as the shifted initial vector. This is supplied to the allocation unit 105.

  In step S <b> 201, the evaluation value comparison unit 202 inputs the evaluation value DFD of the motion vector V together with the motion vector V from the vector evaluation unit 104. At this time, the allocation target position calculation unit 201 also receives the motion vector V and proceeds to step S202. The allocation target position calculation unit 201 obtains the position of the allocation target block that is the offset (motion compensation) destination of the motion vector V in the frame t. That is, the allocation target position calculation unit 201 obtains the position of the block on the frame t at the same position as the end point block of the motion vector V detected on the frame t−1, and proceeds to step S203. In step S203, the allocation target position calculation unit 201 selects one allocation target block from the obtained allocation target blocks, and sets the position of the selected allocation target block to the evaluation value memory 106 and the shifted initial vector replacement unit 203. To proceed to step S204. In step S203, the blocks to be allocated are selected in order from the upper left block on the frame t.

  In step S204, the evaluation value comparison unit 202 acquires the evaluation value DFD of the allocation target block selected by the allocation target position calculation unit 201 from the evaluation value memory 106, proceeds to step S205, and the motion input in step S201. Whether the evaluation value DFD of the vector V is smaller than the evaluation value DFD of the evaluation value memory 106 (that is, whether the evaluation value DFD of the motion vector V is more reliable than the evaluation value DFD of the evaluation value memory 106). If the evaluation value DFD of the motion vector V is determined to be smaller than the evaluation value DFD of the evaluation value memory 106, the process proceeds to step S206.

  In step S206, the evaluation value comparison unit 202 controls the shift initial vector replacement unit 203 to determine the shift initial vector of the allocation target block in the shift initial vector memory 107 selected by the allocation target position calculation unit 201 as the motion vector V ( That is, the motion vector V is rewritten to a motion vector having the same magnitude and the same direction as the motion vector V, and the process proceeds to step S207 where the evaluation value replacement unit 204 is controlled and the evaluation value of the allocation target block selected by the allocation target position calculation unit 201 The DFD is rewritten with the evaluation value DFD of the motion vector V, and the process proceeds to step S208.

  If the evaluation value comparison unit 202 determines in step S205 that the evaluation value DFD of the motion vector V input in step S201 is not smaller than the evaluation value DFD stored in the evaluation value memory 106, step S206 is performed. And the process of S207 is skipped and it progresses to step S208. That is, in this case, since the evaluation value DFD of the evaluation value memory 106 is determined to have higher reliability than the evaluation value DFD of the motion vector V, the values of the evaluation value memory 106 and the shifted initial vector memory 107 are rewritten. There is nothing.

  In step S208, the allocation target position calculation unit 201 determines whether all the allocation target blocks of the motion vector V have been processed, and if it determines that all the allocation target blocks have not been processed, step S203. Return to and repeat the subsequent processing. If the allocation target position calculation unit 201 determines in step S208 that all the allocation target blocks of the motion vector V have been processed, the shifted initial vector allocation process ends.

  In the first process, the shifted initial vector corresponding to the selected allocation target block is not yet stored in the shifted initial vector memory 107. Therefore, if the evaluation value DFD of the shifted initial vector is not yet stored in the corresponding allocation target block of the evaluation value memory 106, the evaluation value DFD is obtained from the selected allocation target block in step S204. Since it is not carried out, it will be judged as Yes in Step S205 and processing of Steps S206 and S207 will be performed.

  As described above, when the motion vector detection of the block of interest is processed, the motion amount of the moving object has a certain degree of continuity between successive frames, and the change in the motion amount is small. A motion vector that passes through the target block of the frame at the next time from the frame at the time (past) (that is, a block on the next frame at the same position as the end block of the motion vector V detected on the past frame) As a conventional method, a shift initial vector that is a motion vector having the same magnitude as the motion vector V and having the same direction as a starting point is used as a candidate vector of the initial vector, and thus is obtained in the past in the peripheral block as in the past. Compared to the case where only the motion vector is the candidate vector of the initial vector, especially at the boundary of the moving object, the gradient method It can give an appropriate motion vector as an initial value for calculation, which makes it possible to improve the accuracy of motion vector detection by computation of the gradient method.

  Further, when the initial shift vector is obtained, the motion vector detected in the frame at the previous time passes in the frame at the next time (that is, the block at the end of the motion vector V detected on the frame t−1). And a motion vector passing through the block of interest is assigned to all the frames of the previous time. Compared to the search from the motion vector of the block, the calculation amount of processing is reduced, so that it is possible to realize hardware that is difficult to realize due to the huge amount of calculation.

  Next, details of the configuration of the initial vector selection unit 101 will be described.

  FIG. 21 is a block diagram illustrating a configuration of the initial vector selection unit 101. The initial vector selection unit 101 having the configuration shown in FIG. 21 uses a motion vector with high reliability from a motion vector detected in the previous (past) frame and a candidate vector such as a shifted initial vector (hereinafter also referred to as an initial candidate vector). Is selected as the initial vector. The initial vector selection unit 101 receives the frame t of the image at time t and the frame t + 1 of the image at time t + 1.

  When the frame t is input, the candidate vector position calculation unit 251 selects a target block to be processed on the frame t, and obtains an initial candidate vector of the target block from the peripheral region of the target block. The position, the type of motion vector to be the initial candidate vector and the priority order are obtained, and the position information of the candidate block and the type information of the initial candidate vector are obtained in the order of the obtained priority order, and the detected vector acquisition unit 252 and the shifted initial vector acquisition unit 253. The candidate vector position calculation unit 251 also supplies the position information of the candidate block to the offset position calculation unit 254.

  In the signal processing device 1, the number of initial candidate vectors is set to a predetermined number based on the balance between the accuracy of the initial vector and the hardware capability, and further, the position of the candidate block and the initial candidate vector The type and priority are also set in advance as will be described later with reference to FIG. In addition, as a type of the initial candidate vector, a motion vector obtained by shifting a motion vector passing through a predetermined block in the past frame to the predetermined block (that is, a block at the end point of the motion vector detected on the past frame) A shift initial vector SV, which is a motion vector having the same magnitude and the same direction as the motion vector V, starting from a block on the next frame at the same position, and a motion vector detected in the past frame (hereinafter, past vector) PV), a motion vector detected in a block before the target block in the current frame (also referred to as current vector CV), and a zero vector.

  Accordingly, when the preset initial candidate vector type information is a past vector or a current vector, the candidate vector position calculation unit 251 displays the candidate block position information and the initial candidate vector type information as the detected vector acquisition unit 252. If the type information of the obtained initial candidate vector is a shifted initial vector, the position information of the candidate block and the type information of the initial candidate vector are supplied to the shifted initial vector acquisition unit 253, which is neither of them. In the case (for example, when the type information of the initial candidate vector is 0 vector), the 0 vector is set, and the position information of the candidate block is supplied to the offset position calculation unit 254 together with the 0 vector.

  The detection vector acquisition unit 252 acquires the motion vector corresponding to the position information of the candidate block and the type information of the initial candidate vector supplied from the candidate vector position calculation unit 251 from the detection vector memory 53, and the acquired motion vector is The initial candidate vector is output to the offset position calculation unit 254.

  The shifted initial vector acquisition unit 253 generates the shifted initial vector corresponding to the position information of the candidate block according to the position information of the candidate block and the type information of the initial candidate vector supplied by the candidate vector position calculating unit 251. Obtained from the memory 107 and output to the offset position calculation unit 254 as an initial candidate vector. In addition, the shifted initial vector acquisition unit 253 outputs a zero vector to the offset position calculating unit 254 when the shifted initial vector is not allocated to the position of the block designated by the candidate vector position calculating unit 251. If no shifted initial vector is assigned, the zero vector may be stored in advance in the shifted initial vector memory 107.

  When the initial position candidate vector (or 0 vector from the candidate vector position calculation section 251) is input from the detection vector acquisition section 252 or the shifted initial vector acquisition section 253, the offset position calculation section 254 is supplied by the candidate vector position calculation section 251. Based on the position information of the candidate block, an offset destination block position obtained by offsetting (motion compensation) the target block of frame t to frame t + 1 is calculated for each initial candidate vector. Then, the offset position calculation unit 254 outputs the candidate block position information and the offset destination block position information to the evaluation value calculation unit 255 together with the initial candidate vector.

  When the evaluation value calculation unit 255 inputs the position information of the candidate block and the information of the offset destination block position together with the initial candidate vector from the offset position calculation unit 254, the evaluation value of the initial candidate vector is used using the frame t and the frame t + 1. Find the DFD. Then, the evaluation value calculation unit 255 outputs the obtained evaluation value DFD together with the initial candidate vector to the evaluation value comparison unit 256.

  The evaluation value comparison unit 256 compares the evaluation value DFD input by the evaluation value calculation unit 255 with the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 257 and inputs the evaluation value DFD by the evaluation value calculation unit 255. When the evaluation value DFD of the initial candidate vector is smaller than the evaluation value DFD of the optimal candidate vector, that is, when it is determined that the initial candidate vector has higher reliability than the optimal candidate vector, the optimal candidate storage register 257 The optimal candidate vector and its evaluation value DFD are replaced with the initial candidate vector and its evaluation value DFD that are determined to have high reliability. Finally, the evaluation value comparison unit 256 controls the optimum candidate storage register 257 to select the optimum candidate vector determined to have the highest reliability based on the evaluation value DFD from among all candidate vectors. The iterative gradient method computing unit 103 outputs the initial vector V0.

  In the optimum candidate storage register 257, an initial candidate vector whose evaluation value DFD is made small (high reliability) by the evaluation value comparison unit 256 is stored as an optimum candidate vector together with the evaluation value DFD. Then, under the control of the evaluation value comparison unit 256, the optimal candidate storage register 257 outputs the finally stored optimal candidate vector to the iterative gradient method computing unit 103 as the initial vector V0.

  FIG. 22 shows a peripheral region that can be a candidate block of the initial vector. In the example of FIG. 22, the arrow T indicates the direction of time passage from the past (previous) frame t−1 at the time t−1 on the left front side to the current frame t at the time t on the right rear side. . In the example of FIG. 22, the peripheral region that can be a candidate for the initial vector is configured by 7 blocks × 7 blocks centered on the block Bt of interest, and each block is configured by 4 pixels × 4 pixels. Has been.

  As described above, the motion vector detection process is executed in the raster scan order from the upper left block of the frame. Therefore, the initial vector selection unit 101 can use the result of motion vector detection up to the immediately preceding block as an initial vector candidate in the motion vector detection processing of the block Bt of interest in the current frame t.

  That is, the peripheral area of the block of interest Bt is composed of a block CVB processed before the block of interest Bt and a block PVB processed after the block of interest Bt. Therefore, when obtaining the initial vector of the block of interest Bt, the initial candidate vector is detected on the motion vector (current vector CV) detected on the current frame t of the block CVB and on the past frame t-1 of the block PVB. Motion vectors (past vectors PV) can be selected. At this time, a shifted initial vector assigned to a block in the same peripheral area can also be a candidate for an initial candidate vector.

  FIG. 23 illustrates an example of initial vector candidate blocks set in the candidate vector position calculation unit 251. In the example of FIG. 23, eight predetermined blocks each having an “alphabet / number” symbol are set as candidate blocks from which an initial candidate vector is acquired in the peripheral area of the block of interest Bt. Yes. The symbols before and after the hatched lines shown in these eight predetermined blocks represent “initial candidate vector type” and “priority”, respectively. Of the types of initial candidate vectors, P represents the past vector PV detected in the past frame t−1, C represents the current vector detected in the current frame t, and S is The shifted initial vector SV is represented.

  “S / 1” of the target block Bt indicates that the shifted initial vector SV assigned to the target block Bt is used as the initial candidate vector in the priority “1”. “C / 2” of the block adjacent to the left of the block of interest Bt indicates that the current vector CV of the block detected in the current frame t is used as the initial candidate vector in the “2” priority order. “C / 3” in the upper right block of the block of interest Bt indicates that the current vector CV of the block detected in the current frame t is used as the initial candidate vector with the priority “3”. “P / 4” in the lower right block of the target block Bt indicates that the past vector PV of the block detected in the past frame t−1 is used as the initial candidate vector with the priority “4”. Yes.

  As for “C / 6” of the lower right block at the upper left corner of the peripheral area 7 × 7, the current vector CV of the block detected in the current frame t is used as the initial candidate vector with the priority “6”. Represents. “C / 5” of the lower left block in the upper right corner of the peripheral area 7 × 7 indicates that the current vector CV of the block detected in the current frame t is used as the initial candidate vector with the priority “5”. Represents. For the upper right block “P / 7” in the lower left corner of the peripheral area 7 × 7, the past vector PV of the block detected in the past frame t−1 is used as the initial candidate vector with the priority “7”. Represents that. In the upper left block “P / 8” in the lower right corner of the peripheral area 7 × 7, the past vector PV of the block detected in the past frame t−1 is used as the initial candidate vector with the priority “8”. Represents that.

  As described above, in the example of FIG. 23, the initial candidate vectors include the block of interest Bt, the block adjacent to the block of interest Bt, the block next to the block of interest Bt, the block adjacent to the left of the block of interest Bt, the upper right, and the lower right. Among the blocks that are not adjacent to the block of interest Bt, the block is selected from the blocks at positions one inside each from the four corners of the peripheral area 7 × 7.

  That is, in the case of the example in FIG. 23, a block adjacent to the block of interest Bt (a block that is spatially and temporally close) has a higher priority than a block slightly apart, and the current vector detected in the current frame t. The priority is set higher than the past vector detected in the past frame t-1. Note that the positions, types, and priorities of these initial vector candidates are not only in the vicinity of the block of interest Bt but also slightly apart so that a reliable initial candidate vector can be obtained with as few candidates as possible. These blocks are set on the basis of spatial correlation and temporal correlation so that there is no bias. Therefore, using the motion vector of a predetermined block set in advance as an initial candidate vector in this way prevents the number of initial candidate vectors from increasing and the amount of computation of processing from becoming enormous. .

  Note that the types of candidate blocks and the positions of the candidate blocks are not limited to the example in FIG. 23, and other types of vectors and other positions may be used. In consideration of the case where the optimum vector does not exist in the peripheral blocks, a zero vector that suppresses deterioration in such a case may be used. Also, the number of candidate blocks may be more or less than eight.

  Next, the details of the initial vector selection process will be described with reference to the flowchart of FIG.

  In step S251, the candidate vector position calculation unit 251 determines the position of the candidate block, the type of the initial candidate vector, and the priority order from which the initial candidate vector of the target block set in advance is acquired from the peripheral area of the selected target block. In step S252, it is determined whether the type information of the initial candidate vectors of the candidate block is a past vector or a current vector in the order of the obtained priority. In step S252, when the candidate vector position calculation unit 251 determines that the type of the initial candidate vector of the candidate block is the past vector or the current vector, the process proceeds to step S253, where the candidate block position information and the type of initial candidate vector are determined. Information is supplied to the detection vector acquisition unit 252, and the motion vector (past vector PV or current vector CV) according to the position information of the candidate block and the type information of the initial candidate vector is detected in the detection vector memory. 53, the acquired motion vector is output to the offset position calculation unit 254, and the process proceeds to step S257.

  In step S252, when the candidate vector position calculation unit 251 determines that the type of the initial candidate vector of the candidate block is not a past vector or a current vector, the process proceeds to step S254, where the candidate vector position calculation unit 251 It is determined whether or not the type of candidate vector is a shifted initial vector. In step S254, if the candidate vector position calculation unit 251 determines that the type of the initial candidate vector of the candidate block is a shifted initial vector, the process proceeds to step S255, where the candidate block position information and the initial candidate vector type information are obtained. The shifted initial vector acquisition unit 253 supplies the shifted initial vector acquisition unit 253 with the shifted initial vector corresponding to the position information of the candidate block from the shifted initial vector memory 107, and the acquired shifted initial vector is offset. The position calculation unit 254 is output, and the process proceeds to step S257.

  In step S254, when the candidate vector position calculation unit 251 determines that the type of the initial candidate vector of the candidate block is not the shifted initial vector (that is, determines that the type of the initial candidate vector of the candidate block is the 0 vector). In step S256, a zero vector is set as the initial candidate vector, and the position information of the candidate block is supplied to the offset position calculation unit 254 together with the zero vector, and the process proceeds to step S257. Also in steps S253 and S255, the candidate vector position calculation unit 251 supplies the position information of the candidate block to the offset position calculation unit 254.

  In step S257, the offset position calculation unit 254 receives the initial candidate vector from the detection vector acquisition unit 252 or the shifted initial vector acquisition unit 253, and based on the position information of the candidate block supplied by the candidate vector position calculation unit 251, For each initial candidate vector, an offset destination block position obtained by offsetting the target block of frame t to frame t + 1 is calculated. Then, the offset position calculation unit 254 outputs the position information of the candidate block and the information of the offset destination block position together with the initial candidate vector to the evaluation value calculation unit 255, and proceeds to step S258.

  When the evaluation value calculation unit 255 inputs the position information of the candidate block and the information of the offset destination block position together with the initial candidate vector from the offset position calculation unit 254, in step S258, using the frame t and the frame t + 1, the initial candidate An evaluation value DFD of the vector is obtained, and the obtained evaluation value DFD is output to the evaluation value comparison unit 256 together with the initial candidate vector, and the process proceeds to S259.

  In step S259, the evaluation value comparison unit 256 determines whether or not the evaluation value DFD obtained by the evaluation value calculation unit 255 is smaller than the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 257. The evaluation value DFD obtained by the evaluation value calculator 255 is smaller than the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 257, that is, the initial candidate vector is more reliable than the optimal candidate vector. Is determined to be high, in step S260, the optimal candidate vector of the optimal candidate storage register 257 and its evaluation value DFD are rewritten with the initial candidate vector and its evaluation value DFD that are determined to have high reliability, and the process returns to step S261. move on.

  In step S259, the evaluation value comparison unit 256 determines that the evaluation value DFD obtained by the evaluation value calculation unit 255 is not smaller than the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 257. If so, the process of step S260 is skipped, and the process proceeds to step S261.

  In step S261, the candidate vector position calculation unit 251 determines whether all the initial candidate vectors (eight vectors in the example of FIG. 23) have been processed, and all the initial candidate vectors have been processed. If it is determined that it is not, the process returns to step S252 and the subsequent processing is repeated.

  When it is determined in step S261 that all the initial candidate vectors have been processed, the process proceeds to step S262, and the evaluation value comparison unit 256 controls the optimal candidate storage register 257 to control all the initial candidate vectors. The optimal candidate vector having the highest reliability based on the evaluation value DFD is output to the iterative gradient method computing unit 103 as the initial vector V0, and the initial vector selection process is terminated.

  As described above, in the target block, the evaluation value DFD of a plurality of initial candidate vectors is obtained, and the initial candidate vector having the smallest evaluation value DFD, that is, the highest reliability is selected as the initial vector. Therefore, it is possible to provide an optimal initial vector for the subsequent motion vector detection, and as a result, it is possible to improve the accuracy of the subsequent motion vector detection. In addition, since a predetermined motion vector of a predetermined block is used as an initial candidate vector, the number of initial candidate vectors is increased, and the amount of calculation of processing is suppressed from becoming enormous.

  Furthermore, based on the fact that there is a certain degree of continuity in the amount of movement of moving objects between consecutive frames and the change in the amount of movement is small, an initial shift vector that is a motion vector that passes through the block of interest from the previous frame Since the vector candidates are used, it is possible to perform motion detection with higher accuracy than in the case where only the motion vectors obtained in the past in the peripheral blocks are used as the initial vector candidates. This is particularly effective at the boundaries of moving objects.

  Next, another configuration of the initial vector selection unit 101 will be described.

  FIG. 25 is a block diagram illustrating a configuration of the initial vector selection unit 101. The initial vector selection unit 101 in FIG. 25 is basically the same as the case in FIG. 21 except that a similarity determination unit 301 and a selector 302 are added, and therefore the description thereof is omitted. Will be omitted as appropriate.

  In the example of FIG. 25, when the frame t is input, the candidate vector position calculation unit 251 selects a target block to be processed on the frame t, and sets a target of interest set in advance from the peripheral region of the target block. Obtain the position of the candidate block for obtaining the initial candidate vector of the block, the type and priority of the motion vector to be the initial candidate vector, and in the order of the obtained priority, the position information of the candidate block and the type information of the initial candidate vector, The detection vector acquisition unit 252 and the shifted initial vector acquisition unit 253 are supplied. Although illustration is omitted, the candidate vector position calculation unit 251 supplies the 0 vector to the similarity determination unit 301 when the type of the initial candidate vector is 0 vector. The candidate vector position calculation unit 251 also supplies the position information of the candidate block to the offset position calculation unit 254. Note that the position of the candidate block from which the initial candidate vector is acquired, the type and priority of the motion vector serving as the initial candidate vector will be described later with reference to FIGS. 26 and 27.

  The detection vector acquisition unit 252 acquires the motion vector corresponding to the position information of the candidate block and the type information of the initial candidate vector supplied from the candidate vector position calculation unit 251 from the detection vector memory 53, and the acquired motion vector is The initial candidate vector is output to the similarity determination unit 301 and the selector 302.

  The shifted initial vector acquisition unit 253 generates the shifted initial vector corresponding to the position information of the candidate block according to the position information of the candidate block and the type information of the initial candidate vector supplied by the candidate vector position calculating unit 251. Obtained from the memory 107 and output to the similarity determination unit 301 and the selector 302 as initial candidate vectors.

  The similarity determination unit 301 inputs initial candidate vectors from at least one of the detection vector acquisition unit 252 and the shifted initial vector acquisition unit 253, and when there are a plurality of initial candidate vectors in the same candidate block, a predetermined conditional expression described later is And comparing a plurality of initial candidate vectors, determining whether or not the initial candidate vectors are similar, and based on the determination result, a predetermined initial candidate vector among the plurality of initial candidate vectors, The selector 302 is selected. Note that the similarity determination unit 301 can also compare the 0 vector as one of the initial candidate vectors from the candidate vector position calculation unit 251.

  The selector 302 receives an initial candidate vector from at least one of the detection vector acquisition unit 252 and the shifted initial vector acquisition unit 253, and when there are a plurality of initial candidate vectors in the same candidate block, under the control of the similarity determination unit 301 Then, any one of the initial candidate vectors is selected, and the selected initial candidate vector is output to the offset position calculation unit 254.

  26 and 27 show examples of initial vector candidate blocks set in the candidate vector position calculation unit 251. FIG. 26 and 27 are other examples of candidate blocks in FIG. Accordingly, the peripheral region that can be a candidate for the initial vector is configured by 7 blocks × 7 blocks centered on the block Bt of interest, and each block is configured by 4 pixels × 4 pixels. Note that FIG. 26 shows that among the peripheral region including the block of interest Bt itself, eight blocks in which alphabetic symbols are shown are set as candidate blocks from which initial candidate vectors are acquired. FIG. 27 shows priorities in which the motion vectors of the candidate blocks in FIG. 26 are used (evaluated) in the initial vector selection unit 101.

  In the example of FIG. 26, the symbols shown in the eight predetermined blocks represent “types of initial candidate vectors”. Of the types of the initial candidate vectors, P represents the past vector PV detected in the past frame t-1, C represents the current vector detected in the current frame t, and S represents the shift. An initial vector SV is represented. In the two types of symbols before and after the hatched line, two types of motion vectors are acquired in the block, the two types of motion vectors are compared by the similarity determination unit 301, and the two types of motion vectors are similar. If it is determined that the motion vector before the oblique line is selected as the initial candidate vector of the block, and if it is determined that there is a difference between the two types of motion vectors, the motion vector after the oblique line is This indicates that the block is selected as an initial candidate vector.

  Note that the similarity determination unit 301 obtains the similarity of motion vectors by the following equation (11).

  | Vbx-vsx | <th1 && | vby-vsy | <th1 (11)

  Where && represents the logical product, vbx and vby represent the horizontal and vertical components of the past or current vector, vsx and vsy represent the horizontal and vertical components of the shifted initial vector, and th1 is Represents a predetermined threshold. As an example, the value of th1 is 5. However, other values may be effective.

  That is, the similarity determination unit 301 determines that the difference (absolute value) between the horizontal and vertical components of the past vector or the current vector and the horizontal and vertical components of the shifted initial vector is smaller than the predetermined threshold th1. Determines that there is a similarity between the past vector or current vector and the shifted initial vector, and the difference between the horizontal and vertical components of the past or current vector and the horizontal or vertical component of the shifted initial vector (the absolute value of ) Are equal to or greater than a predetermined threshold th1, respectively, it is determined that there is no similarity (ie, there is a difference) between the past vector or the current vector and the shifted initial vector.

  Therefore, in the example of FIGS. 26 and 27, “S” and “1” of the target block Bt are used as the initial candidate vector with the shifted initial vector SV assigned to the target block Bt as the first candidate vector. Represents that. “C” and “2” of the block adjacent to the left of the target block Bt indicate that the current vector CV of the block detected in the current frame t is used as the initial candidate vector in the priority “2” th. Yes.

  “C / S” and “3” in the upper right block of the target block Bt are similar to the current vector CV of the block detected in the current frame t and the shifted initial vector SV assigned to this block. If it is determined that the current vector CV is used as the initial candidate vector in the priority “3” order, and if it is determined that there is a difference between the current vector CV and the shifted initial vector SV, the shifted initial vector SV is the initial candidate vector. This indicates that the vector is used in the priority order “3”. “P / S” and “4” in the lower right block of the target block Bt are similar to the past vector PV of the block detected in the past frame t−1 and the shifted initial vector SV assigned to this block. If it is determined that there is a difference, the past vector PV is used as the initial candidate vector in the priority “4” order, and if it is determined that there is a difference between the past vector PV and the shifted initial vector SV, the shifted initial vector SV Is used as the initial candidate vector in the priority “4”.

  “S / C” and “5” of the block in the upper left corner of the peripheral area 7 × 7 are similar to the current vector CV of the block detected in the current frame t and the shifted initial vector SV assigned to this block If it is determined that there is a difference, the shifted initial vector SV is used as the initial candidate vector in the priority “5” th order, and if it is determined that there is a difference between the current vector CV and the shifted initial vector SV, the current vector CV Is used as the initial candidate vector in the priority “5”. “S / C” and “6” of the block in the upper right corner of the peripheral area 7 × 7 are similar to the current vector CV of the block detected in the current frame t and the shifted initial vector SV assigned to this block. If it is determined that there is a difference, the shifted initial vector SV is used as the initial candidate vector in the priority “6” order, and if it is determined that there is a difference between the current vector CV and the shifted initial vector SV, the current vector CV Is used as the initial candidate vector in the priority order “6”.

  “S / P” and “7” of the block in the lower left corner of the peripheral area 7 × 7 are the past vector PV of the block detected in the past frame t−1 and the shifted initial vector SV assigned to this block. Are determined to be similar to each other, the shifted initial vector SV is used as the initial candidate vector in the priority “7” th order, and when it is determined that there is a difference between the past vector PV and the shifted initial vector SV, It represents that the vector PV is used as the initial candidate vector in the priority “7”. “S / P” and “8” of the block in the lower right corner of the peripheral area 7 × 7 are the past vector PV of the block detected in the past frame t−1 and the shifted initial vector SV assigned to this block. Are determined to be similar to each other, the shifted initial vector SV is used as the initial candidate vector in the priority “8” order, and when it is determined that there is a difference between the past vector PV and the shifted initial vector SV, This indicates that the vector PV is used as the initial candidate vector in the priority order “8”.

  As described above, in the example of FIG. 26, the initial candidate vector is an area adjacent to the block of interest Bt including the block of interest Bt (a block in contact with any of the four sides and the four vertices of the block of interest Bt, that is, the block of interest Of the target block Bt, the block adjacent to the left, the upper right, and the lower right of the target block Bt, and the peripheral region 7 × 7 among the blocks of the region not in contact with the target block Bt. It is assumed to be a block at the four corners. That is, in the example of FIG. 26, the priority order of the initial candidate vector of the block in the area in contact with the block of interest Bt is set higher than the initial candidate vector of the block in the area of no contact with the block of attention Bt.

  Here, the region in contact with the target block Bt will be described in detail. The initial shift vector of the target block Bt is the same size as the motion vector having the block on the frame t−1 at the same position as the target block Bt as the end point. Since the motion vectors are in the same direction, the motion amount of the moving object has a certain degree of continuity between successive frames, and the initial candidate vector of the block of interest Bt is surely based on the small change in the motion amount. This is set to the candidate vector with the priority “1”. The current vector of the block adjacent to the left of the target block Bt is immediately before the target block Bt in the raster scan order, and is detected immediately before detecting the motion vector of the target block Bt. The candidate vector is assumed to be most likely next to the shifted initial vector of the block of interest Bt, and is set to the candidate vector with the priority “2”.

  In order to obtain a more reliable initial vector with fewer initial candidate vectors based on the statistics of the experimental results conducted by the present applicant, the position of the candidate block is at the right or left of the target block Bt, or above and below. It is considered that it is desirable to set it to be moderately scattered, and it is considered that the current vector obtained in the frame t is more likely than the past vector detected in the frame t-1. That is, since the initial candidate vector of the block adjacent to the left of the block of interest Bt is set to the priority “2”, the initial candidate vector of the upper right block of the block of interest Bt is the initial candidate vector of the priority “2”. This is the right side opposite to the block, and is positioned in front of the block of interest Bt in the raster scan order. Further, since the positions are scattered in the upward direction, the priority is set as “3”.

  As a result, from the left side of the block of interest Bt, the initial candidate vector with the highest priority “2” is acquired, and from the right side of the block of interest Bt, the initial candidate vector with the highest priority of “3” is acquired. Is done. That is, the probability that a probable initial candidate vector is acquired is higher on the left side of the block of interest Bt, and therefore the initial candidate vector of the lower right block of the block of interest Bt has different characteristics based on spatial correlation. In order to complement the vectors, the priority is set as “4th”.

  In other words, for candidate blocks with priority levels “3” and “4”, in the area in contact with the block of interest Bt, from the midpoint of the straight line connecting the centers of the priority blocks “1” and “2”, It can be said that the initial candidate vector of the upper right block of the target block Bt, which is located in the far block and before the target block Bt in the raster scan order, is set to the priority “3”. Similarly, in the area that is in contact with the block of interest Bt, the center of a space (a circle with three points on the circumference) defined by a straight line connecting the centers of the candidate blocks of priority “1” to “3”. That is, it can be said that the initial candidate vector of the lower right block of the block of interest Bt that is the farthest block from the center of gravity of the triangle connecting the three points is set to the priority “4”.

  On the other hand, the region that is not adjacent to the block of interest Bt will be described in detail. Since the current vector obtained in the frame t is more likely than the past vector detected in the frame t-1, the block of interest is In an area that is not adjacent to Bt, the priority order of the initial candidate vector of the block positioned before the target block Bt is higher than the initial candidate vector of the block positioned after the target block Bt in the raster scan order. Is set high.

  About the above, since the candidate block of FIG. 23 is also fundamentally the same structure, it can be said also about the candidate block of FIG. On the other hand, the difference from the candidate block of FIG. 23 is that in the example of FIG. 26, a block one block away from the block of interest Bt than the candidate block of FIG. ) Is a candidate block. This is because, based on the spatial correlation, the motion vector from the candidate block located as far as possible from the block of interest Bt is adjacent to the block of interest Bt within a range where the calculation load is not applied ( This is because a motion vector having a different property from the motion vector obtained from the block in the region is acquired, and it is effective to obtain a reliable initial vector candidate with as few candidates as possible.

  Further, in the example of FIG. 26, when it is determined that there is a difference between the two types of motion vectors in the block adjacent to the target block Bt, the shifted initial vector is selected, whereas the target block In the four corner blocks of the peripheral area 7 × 7 that are slightly apart from Bt, when it is determined that there is a difference between the two types of motion vectors, the past vector (or the current vector) is set to be selected. .

  That is, in a block in an area adjacent to the block of interest Bt, an initial shift vector (a motion vector having a block on the frame t−1 at the same position as the block as an end point) and a frame t−1 or When there is a difference from the past or current vector detected on t (that is, when the movement of the shifted initial vector changes to the movement of the past or current vector), the same movement may occur in the vicinity of the block of interest. The block is included in the boundary of objects with different motions, based on the spatial correlation of many, the inertia of the motion of the object, and the statistics of the experimental results by the applicant, etc. Since it is assumed that the disappearance area and generation area of the object described later have occurred, Than the past or present vector might have, it is better to select a shift initial vector is the probable.

  On the other hand, in a block slightly apart from the target block Bt, when it is determined that there is a difference between the two types of motion vectors, motion vectors (adjacent to the motion vector obtained from blocks adjacent to the target block Bt) are different. It is considered that it is more effective to obtain a reliable initial vector candidate with as few candidates as possible, so that the motion vector that complements the motion vector obtained from the block to be processed is a candidate. Instead, the past or present vector is selected.

  Also in the candidate block of FIG. 23, as in the case of the candidate block of FIG. 26, a plurality of motion vectors may be acquired and the motion vector based on the similarity determination result may be used as the initial candidate vector. .

  As described above, the position, type, and priority of these initial vector candidates are the smallest possible number of candidates (in this case, 8) so that a reliable initial candidate vector can be obtained. In addition to not only the area that touches the block, but also blocks that are located a little away from each other, and the initial vector selection over the candidate block of FIG. Is set so as to improve the accuracy. Therefore, using the motion vector of a predetermined block set in advance as an initial candidate vector in this way prevents the number of initial candidate vectors from increasing and the amount of computation of processing from becoming enormous. .

  Note that the similarity of motion vectors is not limited to equation (11), but may be obtained by other conditional equations. As another conditional expression, for example, it can be obtained by evaluating a norm (vector norm). The norm represents the magnitude of a vector in a certain vector space, and is obtained by the following equation (12).

・・・（１２）

(12)

  Here, vbx and vby represent the horizontal and vertical components of the past vector or the current vector, vsx and vsy represent the horizontal and vertical components of the shifted initial vector, and th2 represents a predetermined threshold value. Yes.

  In this case, the similarity determination unit 301 resembles the past vector or the current vector and the shifted initial vector when the norm of the past vector or the current vector and the shifted initial vector is smaller than the predetermined threshold th2. If the norms of the past vector or the current vector and the shifted initial vector are each greater than or equal to a predetermined threshold th2, there is no similarity between the past vector or the current vector and the shifted initial vector ( That is, there is a difference.

  Next, details of the initial vector selection process will be described with reference to the flowchart of FIG. FIG. 28 shows an example of the initial vector selection process in step S102 of FIG.

  In step S311, the candidate vector position calculation unit 251 obtains the initial candidate vector of the target block set in advance from the peripheral region of the selected target block, for example, as described above with reference to FIGS. The position of the candidate block to be acquired, the type and priority of the initial candidate vector are obtained, and the position information of the candidate block and the type information of the initial candidate vector to be acquired are detected in the order of the obtained priority, and the detected vector acquisition unit 252 and The initial value is supplied to the shifted initial vector acquisition unit 253, and the process proceeds to step S312. At this time, the candidate vector position calculation unit 251 also supplies the position information of the candidate block to the offset position calculation unit 254.

  In step S312, the detection vector acquisition unit 252 detects a motion vector (past vector PV or current vector CV) according to the position information of the candidate block and the type information of the initial candidate vector supplied by the candidate vector position calculation unit 251. The motion vector acquired from the vector memory 53 is output to the similarity determination unit 301 and the selector 302, and the process proceeds to step S313.

  In step S313, the shifted initial vector acquisition unit 253 acquires, from the shifted initial vector memory 107, the shifted initial vector corresponding to the position information of the candidate block and the type information of the initial candidate vector supplied by the candidate vector position calculation unit 251. The obtained initial shift vector is output to the similarity determination unit 301 and the selector 302, and the process proceeds to step S314. The shifted initial vector acquisition unit 253 outputs a 0 vector to the similarity determination unit 301 and the selector 302 when the shifted initial vector is not assigned to the block position designated by the candidate vector position calculation unit 251. .

  When the similarity determination unit 301 receives an initial candidate vector from at least one of the detection vector acquisition unit 252 and the shifted initial vector acquisition unit 253, in step S314, the similarity determination unit 301 performs operation selection processing of the initial candidate vector, and the process proceeds to step S315. This initial candidate vector calculation / selection process will be described later with reference to FIG. 29, but when multiple types (for example, two types) of initial candidate vectors exist in the same candidate block by the process of step S314. Two types of similarity are determined, and one initial candidate vector based on the determined result is selected by the selector 302 and output to the offset position calculation unit 254.

  In step S315, the offset position calculation unit 254 receives the initial candidate vector from the selector 302. Based on the position information of the candidate block supplied by the candidate vector position calculation unit 251, the offset position calculation unit 254 performs frame t The block position of the offset destination obtained by offsetting the block of interest in frame t + 1 is calculated. Then, the offset position calculation unit 254 outputs the position information of the candidate block and the information of the offset destination block position together with the initial candidate vector to the evaluation value calculation unit 255, and the process proceeds to step S316.

  When the evaluation value calculation unit 255 inputs the position information of the candidate block and the information of the offset destination block position together with the initial candidate vector from the offset position calculation unit 254, in step S316, using the frame t and the frame t + 1, the initial candidate The vector evaluation value DFD is obtained, and the obtained evaluation value DFD is output to the evaluation value comparison unit 256 together with the initial candidate vector, and the process proceeds to S317.

  In step S317, the evaluation value comparison unit 256 determines whether or not the evaluation value DFD obtained by the evaluation value calculation unit 255 is smaller than the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 257. The evaluation value DFD obtained by the evaluation value calculator 255 is smaller than the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 257, that is, the initial candidate vector is more reliable than the optimal candidate vector. Is determined to be high, in step S318, the optimal candidate vector in the optimal candidate storage register 257 and its evaluation value DFD are rewritten with the initial candidate vector and its evaluation value DFD that are determined to have high reliability, and the process proceeds to step S319. move on.

  In step S317, the evaluation value comparison unit 256 determines that the evaluation value DFD obtained by the evaluation value calculation unit 255 is not smaller than the evaluation value DFD of the optimal candidate vector stored in the optimal candidate storage register 257. If so, the process of step S318 is skipped and the process proceeds to step S319.

  In step S319, the candidate vector position calculation unit 251 determines whether or not the processing of the initial candidate vectors in all candidate blocks (eight vectors in the case of FIGS. 26 and 27) has been completed, and all the initial candidates are determined. If it is determined that the vector processing has not ended, the process returns to step S312 and the subsequent processing is repeated.

  If it is determined in step S319 that all the initial candidate vectors have been processed, the process proceeds to step S320, and the evaluation value calculation unit 255 controls the optimal candidate storage register 257 to control all the initial candidate vectors. The optimal candidate vector having the highest reliability based on the evaluation value DFD is output to the iterative gradient method computing unit 103 as the initial vector V0, and the initial vector selection process is terminated.

  Next, with reference to the flowchart of FIG. 29, the calculation selection process of the initial candidate vector will be described. In steps S312 and 313 in FIG. 28, the initial candidate vector set in the candidate block in FIG. 26 is input to the similarity determination unit 301 and the selector 302 from at least one of the detection vector acquisition unit 252 and the shifted initial vector acquisition unit 253. Is done.

  In step S351, the similarity determination unit 301 determines whether there are a plurality of initial candidate vectors in the same candidate block. For example, in the case of the upper right block of the block of interest Bt in FIG. 26, the similarity determination unit 301 receives the current vector CV of the block detected in the current frame t from the detection vector acquisition unit 252 and shifts the initial vector. From the acquisition unit 253, the shifted initial vector SV assigned to the block is input. In this case, the similarity determination unit 301 determines that there are a plurality of initial candidate vectors, and proceeds to step S352. In step S352, the similarity determination unit 301 compares the current vector CV with the shifted initial vector SV using Expression (11), and proceeds to step S353, based on the comparison result between the current vector CV and the shifted initial vector SV. It is determined whether or not two initial candidate vectors (current vector CV and shifted initial vector SV) are similar.

  As a result of comparing the current vector CV and the shifted initial vector SV, if the expression (11) is satisfied, it is determined in step S353 that the two initial candidate vectors (the current vector CV and the shifted initial vector SV) are similar. The sex determination unit 301 proceeds to step S354, and causes the selector 302 to select the initial candidate vector (that is, the current vector CV) in the case of similarity set in the candidate block of FIG. Under the control of the similarity determination unit 301, the selector 302 outputs the selected initial candidate vector to the offset position calculation unit 254, ends the initial candidate vector calculation selection process, and returns to step S315 in FIG.

  As a result of comparing the current vector CV and the shifted initial vector SV, if the expression (11) is not satisfied, it is determined in step S353 that the two initial candidate vectors are not similar (the two initial candidate vectors are different). Then, the similarity determination unit 301 proceeds to step S355, and causes the selector 302 to select an initial candidate vector (that is, a shifted initial vector SV) when there is a difference set in the candidate block of FIG. Under the control of the similarity determination unit 301, the selector 302 outputs the selected initial candidate vector to the offset position calculation unit 254, ends the initial candidate vector calculation selection process, and returns to step S315 in FIG.

  On the other hand, for example, in the case of the target block Bt in FIG. 26, the similarity determination unit 301 receives the shifted initial vector SV assigned to the block from the shifted initial vector acquisition unit 253. That is, only one initial candidate vector is input to the similarity determination unit 301. In this case, the similarity determination unit 301 determines that a plurality of initial candidate vectors do not exist in the candidate block, proceeds to step S356, and causes the selector 302 to select the input initial candidate vector. Under the control of the similarity determination unit 301, the selector 302 outputs the selected initial candidate vector to the offset position calculation unit 254, ends the initial candidate vector calculation selection process, and returns to step S315 in FIG.

  As described above, a plurality of types of motion vectors are set as initial candidate vectors in the blocks around the target block for detecting the motion vectors, the similarities of these motion vectors are determined, and the similarity determination results are used. Since one initial candidate vector is selected in the block, the initial vector is determined in a predetermined block as compared with the case where one type of motion vector is set in the surrounding blocks. The range of choices can be expanded. In other words, since the selection range of the initial vector can be expanded without increasing the number of candidate blocks, a more probable initial vector can be supplied to the subsequent motion vector detection without increasing the computational load so much. Can do.

  Further, when setting an initial candidate vector, in addition to the motion vector detected in the past or the current frame, a shifted initial vector that is a motion vector passing through the block of interest from the past frame is set as the initial candidate vector. Therefore, especially at the boundary of a moving object, it is more reliable for motion vector detection in the subsequent stage than in the case where only the motion vector obtained in the past in the peripheral block is used as the initial vector candidate as in the past. A new initial vector can be supplied. Thereby, highly accurate motion detection can be performed.

  Furthermore, in the block adjacent to the target block Bt, when it is determined that there is a difference between the two types of motion vectors, the shifted initial vector is selected, whereas in the block slightly away from the target block Bt, When it is determined that there is a difference between the two types of motion vectors, the past vector (or the current vector) is selected. Therefore, the motion vector obtained from the block adjacent to the block of interest Bt and the nature of the motion vector is different. Since the (complementary motion vector) becomes the initial candidate vector, a more reliable initial vector can be obtained with as few candidates as possible.

  Next, details of the configuration of the iterative gradient method computing unit 103 will be described.

  FIG. 30 is a block diagram illustrating a configuration of the iterative gradient method computing unit 103. The iterative gradient method computing unit 103 having the configuration shown in FIG. 30 performs processing for detecting an optimal motion vector using the input frame t of the image at time t and the frame t + 1 of the image at time t + 1. The process of detecting the motion vector is a process executed for each predetermined block composed of a plurality of pixels, and the iterative gradient method computing unit 103 performs gradients for each block or for each pixel in each block. By repeatedly executing the calculation using the method, an optimal motion vector with high reliability based on the evaluation value DFD is output.

  The iterative gradient method computing unit 103 receives the frame t of the image at time t and the frame t + 1 of the image at time t + 1 via the pre-filters 102-1 and 102-2. The initial vector V0 is input.

  The mode selection unit 401 selects a gradient method calculation processing mode for each predetermined block under the control of the effective pixel determination unit 403, and selects the initial vector V0 from the initial vector selection unit 102 with the selector 402 and the vector evaluation. Output to the unit 104. The gradient method processing mode consists of a block unit processing mode in which the motion vector detection target is a block and a pixel unit processing mode in which the motion vector detection target is a pixel, and the block unit processing mode is selected as an initial value. The Under the control from the vector evaluation unit 104, the selector 402 uses either the initial vector V0 from the mode selection unit 401 or the motion vector Vn output from the gradient method calculation unit 404 as an initial value of the gradient method calculation. The motion vector to be used (hereinafter referred to as an offset vector) is output to the effective pixel determination unit 403 and the gradient method calculation unit 404.

  When the effective pixel determination unit 403 receives the offset vector selected by the selector 402, the frame t of the image at time t and the frame t + 1 of the image at time t + 1 input via the pre-filters 102-1 and 102-2. , Using the position calculated by offsetting the offset vector as a starting point, it is determined whether or not the number of pixels effective for the calculation of the gradient method is greater than the threshold in the calculation block of each unit processing mode, The processing method is switched according to the result. That is, when the effective pixel determination unit 403 determines that the number of pixels effective for the calculation of the gradient method is equal to or less than the threshold value, the effective pixel determination unit 403 controls the mode selection unit 401 and sets the processing mode for the predetermined block. Then, change to the pixel unit processing mode or stop the calculation. When the effective pixel determination unit 403 determines that the number of pixels effective for gradient method calculation is greater than the threshold value in the calculation block of each unit processing mode, the effective pixel determination unit 403 controls the gradient method calculation unit 404 to The gradient method calculation process is executed.

  When the gradient vector calculation unit 404 receives the offset vector input from the selector 402, the frame t of the image at time t and the frame t + 1 of the image at time t + 1 input via the prefilters 102-1 and 102-2. Using the offset vector from the selector 402 as a starting point, the gradient method calculation of each unit processing mode is executed, and the calculated motion vector Vn is used as the delay unit 405 and the vector evaluation unit 104. Output to.

  The delay unit 405 holds the motion vector Vn output from the gradient method calculation unit 404 until the next processing cycle of the effective pixel determination unit 403 and the gradient method calculation unit 404, and according to the control of the selector 402, The held motion vector Vn is output to the effective pixel determination unit 403.

  FIG. 31 is a block diagram illustrating a detailed configuration of the effective pixel determination unit 403. In the example of FIG. 31, the effective pixel determination unit 403 includes a time pixel difference calculation unit 411, a pixel difference value determination unit 412, an effective pixel counter 413, and a gradient method continuation determination unit 414.

  When the offset vector selected by the selector 402 is input, the temporal pixel difference calculation unit 411 selects a pixel in an arithmetic block in units of blocks, and inputs the offset vector and the pre-filters 102-1 and 102-2. The pixel difference Δt in the time direction of the selected pixel is calculated using the frame t of the image at time t and the frame t + 1 of the image at time t + 1, and the calculated pixel difference Δt in the time direction is calculated as the pixel difference. The data is output to the value determination unit 412.

  The pixel difference value determination unit 412 determines whether or not the pixel difference Δt in the time direction calculated by the time pixel difference calculation unit 411 is smaller than a predetermined threshold (hereinafter referred to as a pixel difference value). When it is determined that the pixel difference Δt in the time direction is smaller than the predetermined pixel difference value, the number of effective pixels of the effective pixel number counter 413 is counted by one. The effective pixel number counter 413 counts the value of pixels determined to be effective by the pixel difference value determination unit 412 for each calculation block. The value of the effective pixel number counter 413 is reset for each calculation block.

  The gradient method continuation determination unit 414 determines whether or not the number of effective pixels for the gradient method calculation is greater than a threshold value in each unit processing mode calculation block, and the unit processing mode calculation block includes a gradient. If it is determined that the number of pixels effective for the arithmetic operation is greater than the threshold value, a flag (flg = 1) that causes the gradient operation unit 404 to execute the gradient operation for each unit processing mode is output. If the gradient method continuation determination unit 414 determines that the number of pixels effective for gradient method calculation is less than the threshold in the calculation block of each unit processing mode, the gradient method continuation determination unit 414 controls the mode selection unit 401 to perform other unit processing. A mode is selected, or a flag (flg = 0) for stopping the gradient method calculation is output to the gradient method calculation unit 404.

  FIG. 32 is a block diagram showing a detailed configuration of the gradient method computing unit 404. In the example of FIG. 32, the gradient method calculation unit 404 includes a time pixel difference calculation unit 421, a pixel difference value determination unit 422, a horizontal / vertical pixel difference calculation unit 423, a gradient integration unit 424, and a vector calculation unit 425.

  The temporal pixel difference calculation unit 421 controls each unit of the gradient method calculation unit 404 based on the flag input from the effective pixel determination unit 403. In other words, when the flag indicates 1 (flg = 1), the temporal pixel difference calculation unit 421 causes each unit of the gradient method calculation unit 404 to execute gradient method calculation processing. Specifically, when the time pixel difference calculation unit 421 receives a flag from the effective pixel determination unit 403, the frame t of the image at time t input via the prefilters 102-1 and 102-2, and the time t + 1 The pixel in the calculation block of each unit processing mode is selected around the pixel calculated by offsetting the offset vector from the selector 402 using the frame t + 1 of the image of the selected pixel, and the pixel in the time direction of the selected pixel is selected. The difference Δt is calculated, and the calculated pixel difference Δt and offset vector in the time direction are output to the pixel difference value determination unit 422. Also, the time pixel difference calculation unit 421 outputs the offset vector to the vector calculation unit 425 when the processing of the pixels in the calculation block is completed, and controls the vector calculation unit 425 to obtain the motion vector of the detection target block. Let it be calculated.

  On the other hand, when the flag indicates 0 (flg = 0), the time pixel difference calculation unit 421 does not calculate the pixel difference Δt in the time direction of the pixels in the calculation block. The gradient method calculation process is not executed, and the process is terminated. At this time, the temporal pixel difference calculation unit 421 controls the vector calculation unit 425 to set the motion vector V to 0 vector.

  The pixel difference value determination unit 422 determines whether or not the pixel difference Δt in the time direction calculated by the time pixel difference calculation unit 421 is smaller than a predetermined threshold (hereinafter referred to as a pixel difference value). When it is determined that the pixel difference Δt in the time direction is smaller than the predetermined pixel difference value, the pixel is subjected to the gradient method calculation, and the pixel difference Δt in the time direction and the offset vector are input to the horizontal / vertical pixel difference calculation unit 423. Output. That is, the pixel of the pixel difference Δt determined to be smaller than the predetermined pixel difference value is set as a calculation target in the horizontal / vertical pixel difference calculation unit 423 and the gradient integration unit 424. In addition, when the pixel difference value determination unit 422 determines that the pixel difference Δt in the time direction is equal to or greater than a predetermined pixel difference value, the pixel difference value determination unit 422 causes the horizontal / vertical pixel difference calculation unit 423 and the gradient integration unit 424 to process the pixel. Prohibit.

  When the horizontal / vertical pixel difference calculation unit 423 receives the offset vector input from the pixel difference value determination unit 422, the frame t of the image at time t input through the prefilters 102-1 and 102-2, and the time The pixel difference Δt in the time direction is smaller than a predetermined pixel difference value in the calculation block of each unit processing mode centered on the pixel calculated by offsetting the offset vector using the frame t + 1 of the t + 1 image. The pixel difference Δx in the horizontal direction and the pixel difference Δy in the vertical direction of the pixel determined to be calculated are calculated. The horizontal / vertical pixel difference calculation unit 423 outputs the pixel difference Δt in the time direction, the pixel difference Δx in the horizontal direction, and the pixel difference Δy in the vertical direction to the gradient integration unit 424. The gradient integration unit 424 integrates the gradients of pixels that are determined to have a pixel difference Δt in the time direction smaller than a predetermined pixel difference value. That is, the gradient integrating unit 424 is configured to calculate the temporal pixel difference Δt calculated by the temporal pixel difference calculating unit 421, the horizontal pixel difference Δx calculated by the horizontal / vertical pixel difference calculating unit 423, and the vertical pixel difference. Δy is integrated, and the integrated gradient value is output to the vector calculation unit 425.

  When the offset vector is input from the time pixel difference calculation unit 421, the vector calculation unit 425 uses the value of the gradient accumulated by the gradient accumulation unit 424 and the least square sum of Equation (9) described above to use the motion vector. vn is calculated. Further, the vector calculation unit 425 adds the offset vector from the temporal pixel difference calculation unit 421 to the calculated motion vector vn to obtain a motion vector Vn, and outputs it to the vector evaluation unit 104 and the delay unit 405.

  Next, the motion vector detection target block and the calculation block will be described. When considering the movement of an object in an image, the same object often moves the same. Therefore, the detection processing unit for detecting a motion vector is usually performed in units of blocks as shown in FIG.

  In the example of FIG. 33, the arrow X indicates the horizontal direction, and the arrow Y indicates the vertical direction. In addition, an arrow T indicates the direction of time passage from the frame t at the time t in the back right to the frame t + 1 at the time t + 1 in the left front.

  In the case of the example of FIG. 33, a block of 8 pixels × 8 pixels (hereinafter referred to as 8 × 8 block) is shown as the detection target block K of each frame. The detection target block K is a block in which the same motion is obtained for the 8 × 8 block when the motion vector is detected for the 8 × 8 block.

  On the other hand, motion vector detection (that is, calculation) by the gradient method has a feature that a motion vector can be detected for each pixel. However, when the motion vector is calculated in units of pixels, the gradient method using the least square method, which is a statistical solution, has a drawback that the detection accuracy is significantly reduced. Therefore, in general, when calculating a motion vector not only for each block but also for each pixel, the pixels used for the calculation of the gradient method are configured in units of blocks (calculation blocks). There are many.

  As described above, the pixels used for the calculation of the gradient method are the pixels of the calculation block of 8 × 8 blocks. That is, in the example of FIG. 33, the motion vector obtained as a result of the gradient method computation using the computation block (8 × 8) is the motion corresponding to all the pixels of the detection target block K (8 × 8). It will be a vector.

  However, as described above, as a problem that occurs when the pixels used for the calculation of the gradient method are handled in units of calculation blocks as described above, different motions are generated in the pixels constituting the detection target block as shown in FIG. An example is a case where pixels of objects that are held are mixed.

  In the example of FIG. 34, in the detection target block (= calculation block) K composed of 8 × 8 pixels (64 pixels), 28 pixels (thick line circles) from the upper left pixel to the diagonally upper left indicated by the arrow A. In the 8 × 8 detection target block K, 36 pixels (thin line circles) from the lower right pixel have a rightward movement indicated by an arrow B.

  Therefore, in the detection target block K, the assumption that the movements are almost the same as described above is lost, and if the calculation of the gradient method is performed using a calculation block that includes pixels with different movements, the movement is remarkably increased. The vector detection accuracy is reduced. That is, this is a problem of a decrease in detection accuracy regarding the disappearance region (covered background region) and the generation region (uncovered background region) of the object that occurs at the boundary between objects having different movements. The disappearance region is a mixed region at a position corresponding to all end portions in the advancing direction of the foreground object with respect to the foreground region, and is a region where the background component is covered with the foreground as time passes. On the other hand, the generation region is a mixed region at a position corresponding to the rear end of the foreground object in the traveling direction with respect to the foreground region, and refers to a region where a background component appears as time passes.

  FIG. 35 is a diagram for explaining the luminance state of the detection target block in the case of the same object. On the left side in the figure, the arrow X indicates the horizontal direction, and the arrow Y indicates the vertical direction. In addition, an arrow T indicates the direction of time passage from the frame t at the time t in the back right to the frame t + 1 at the time t + 1 in the left front. On the right side in the figure, the arrow X indicates the horizontal direction, and the arrow L indicates the luminance value. That is, the luminance values Lp1, Lp2, Lp3,..., Lpi (3 <i <7),..., Lp7 on the right side in the figure are, for example, the upper left in the 8 × 8 detection target block of the frame t on the left side in the figure. .., Pi (3 <i <7),..., P7,..., And similarly, the luminance values Lq1, Lq2, and Lq3. ,..., Lqi (3 <i <7),..., Lq7 are pixels p1, p2, p3,..., Pi (3 <i <7) in the 8 × 8 detection target block in the left frame t + 1 in the drawing. ,..., P7,... Are pixels located in the same phase as the pixels q1, q2, q3,..., Qi (3 <i <7),. Represents the luminance value. The arrows between the luminance values Lp1 to Lp7 and Lq1 to Lq7 indicate the time direction difference Δt between the pixel on the frame t and the pixel on the frame t + 1.

  Therefore, in the example of FIG. 35, the time direction difference Δt between the pixel p1 and the pixel q1 is represented by the difference between the luminance value Lp1 and the luminance value Lq1, and the time direction difference Δt between the pixel p2 and the pixel q2 is the luminance value Lp2. The time direction difference Δt between the pixel p3 and the pixel q3 is represented by the difference between the luminance value Lp3 and the luminance value Lq3, and the time direction difference Δt between the pixel pi and the pixel qi is represented by the luminance value Lpi. The time direction difference Δt between the pixel p7 and the pixel q7 is represented by the difference between the luminance value Lp7 and the luminance value Lq7, and each luminance value difference is a curve connecting the luminance values Lp1 to Lp7. And the change in luminance is small so that the shape of the curve connecting the luminance values Lq1 to Lq7 is shown as substantially the same shape.

  As described above, when the pixels in the detection target block are the same object (that is, the same motion), the luminance change is not so large for the corresponding pixel in the time direction.

  FIG. 36 is a diagram for explaining a luminance state of a detection target block when pixels of objects having different motions are mixed. In FIG. 35, portions corresponding to those in FIG. 36 are denoted by the corresponding reference numerals, and the description thereof will be repeated and will be omitted. However, in the example of FIG. , Has a different motion from other pixels.

  For example, when the pixel p7 on the frame t has the same movement as other pixels, the luminance value of the pixel q7 located in the same phase as the pixel p7 on the frame t + 1 is the luminance value Lq7-1. In the example of FIG. 36, the pixel p7 has a different movement from the other pixels, so that the luminance value of the pixel q7 is represented by the luminance value Lq7-2. The luminance value Lq7-1 is greatly changed to the lower side in the figure. Therefore, the time direction difference Δt (the difference between the luminance value Lp7 and the luminance value Lq7-2) between the pixel p7 and the pixel q7 becomes large.

  Thus, for example, when pixels of an object having different movements are mixed in the detection target block due to the disappearance, generation, or deformation of the above-described object, the luminance value of the corresponding pixel greatly changes. This often leads to disturbances in movement. Therefore, in the effective pixel determination unit 403, a pixel with a small luminance change is effective even when used for motion vector detection, and conversely, a pixel with a large luminance change is determined to be ineffective when used for motion vector detection. Like that. That is, paying attention to the factor of the brightness change, and determining whether the pixel is valid or invalid by the brightness change, the pixel having a small brightness change is used for motion vector detection. Conversely, the brightness change is large. Pixels can be prevented from being used for motion vector detection. Thereby, the accuracy of motion vector detection can be increased.

  Specifically, the effective pixel determination unit 403 sets a predetermined pixel difference value (threshold value) for the pixel difference Δt in the time direction for each pixel in the calculation block, and the pixel difference Δt in the time direction is When the pixel difference value is smaller than the set predetermined pixel difference value, it is determined that the pixel contributes (is effective) to motion vector detection, and is used for the calculation of the gradient method when detecting the motion vector. If the pixel difference Δt in the direction is greater than or equal to the set predetermined pixel difference value, the pixel is excluded from the gradient method calculation when detecting a motion vector because there is a risk of disturbing motion. To do. Thereby, mixing of different motions is suppressed, and more stable gradient method calculation is executed. As a result, a probable motion vector is detected.

  However, as described above, since the gradient method calculation is based on the least square method, which is a statistical solution, it is used for the gradient method calculation by separating the pixels used for the gradient method calculation from the pixels not used. If the number of pixels is extremely small and the gradient method calculation is performed with these extremely small number of pixels, the reliability of the calculation is lowered, and the accuracy of motion vector detection may be reduced. is there.

  Therefore, when the number of pixels used in the gradient method calculation is small, the effective pixel determination unit 403 determines that the calculation in the processing unit of this detection target block (8 × 8) is unstable, and is shown in FIG. As described above, the processing is performed by switching to the calculation unit for each pixel (64 pixels) in the 8 × 8 block.

  FIG. 37 shows an example of a calculation block in pixel unit processing. In the example of FIG. 37, the calculation block E of the pixel p at the upper left of the detection target block K when the detection target block (block-based calculation block) K (8 × 8) on the frame is switched to pixel-unit processing. (9 × 9 in the example of FIG. 37) is shown.

  That is, when the motion vector of the pixel p is obtained, the calculation block E (9 × 9 in the example of FIG. 37) centered on the pixel p is used. For example, when obtaining a motion vector of a pixel right next to the pixel p, an arithmetic block 9 × 9 (not shown) centered on the pixel right next to the pixel p is used.

  Therefore, for example, even if most of the detection target block K moves differently from the pixel p, in the detection target block K (8 × 8), the number of pixels used in the gradient method calculation is counted, and the number is When the number is small, the process is switched to the pixel unit process, and as shown in FIG. 37, the calculation of the gradient method can be executed using the calculation block E in the pixel unit process.

  As described above, when the pixels used for the gradient method calculation are separated from the pixels used for the gradient method calculation, and the number of pixels used for the gradient method calculation becomes extremely small, the motion vector for all the pixels in the detection target block K (8 × 8) block. Instead of stopping the detection of pixels, the gradient method calculation is performed by changing the pixel of the calculation block E (9 × 9) for each pixel, thereby mixing pixels that are not used in the calculation. As a result, motion vectors can be detected with higher accuracy depending on the pixel.

  FIG. 38 is a diagram illustrating an example in which pixel unit processing is applied to the pixels of the detection target block (calculation block) K in FIG. In the example of FIG. 38, the pixel includes a pixel having a diagonally upper left movement indicated by an arrow A (thick line circle) and a pixel having two different movements indicated by an arrow B (thin line circle). In the detection target block K (8 × 8) in FIG. 34, processing in units of pixels is performed. In this case, when the motion vector of the detection target pixel p at the upper left of the detection target block K is detected, a calculation block E (9 × 9 in the case of the example in FIG. 38) centering on the detection target pixel p is used.

  The pixels in the computation block E have a diagonally upper left movement indicated by an arrow A except for the lower right three pixels (thin line circles) having a horizontal movement indicated by an arrow B. Here, as in the case of the block unit processing described above, for each pixel in the computation block E, a predetermined threshold value (pixel difference value) is set for the time direction pixel difference Δt, and the time direction pixel difference Δt is When the pixel difference value is smaller than the set predetermined pixel difference value, it is determined that the pixel contributes (is effective) to motion vector detection, and is used for gradient method calculation when detecting a motion vector. When the pixel difference Δt is larger than the set predetermined pixel difference value, the pixel is not used for the gradient method calculation when detecting a motion vector because it may cause a disturbance of motion.

  That is, in the example of FIG. 38, in the calculation block E, the pixels (thick line circles) excluding the lower right three pixels (thin line circles) are effective pixels for the gradient method calculation of the detection target pixel p. Since the gradient method calculation is executed using the effective pixels, the detection target block K (8 × 8) in FIG. 34 is more than the gradient method calculation of the block unit processing executed using the calculation block. A stable gradient method operation is performed and, as a result, a probable motion vector is detected.

  Even in the calculation block for each pixel, if the pixels used for the gradient method calculation are extremely reduced by separating the pixels used for the gradient method calculation from those not used, a motion vector is detected. The operation is aborted. That is, even if the motion vector detection is unstable, if a motion having some size is detected, the motion vector does not match the motion of the object in the actual image (not sure). Because there is a risk of becoming. Subsequent processing (for example, vector allocation processing or vector compensation processing) is executed after using such uncertain motion vectors, and the generated image may have motion discontinuities or block noise. In many cases, it causes a large visual deterioration such as the occurrence of the above.

  Accordingly, when the number of pixels used for pixel-based gradient method calculation becomes extremely small, for example, a zero vector (that is, a stationary state) is used as the detected motion vector. Thereby, in the subsequent processing, this motion vector is suppressed from having an adverse effect, and large visual deterioration of the generated image can be suppressed.

  Next, details of the iterative gradient method calculation processing will be described with reference to the flowchart of FIG. The initial vector V0 is input to the mode selection unit 401 from the previous stage.

  In step S401, the mode selection unit 401 selects a block unit processing mode, sets the block on the frame t as a detection target block (calculation block), and sets the initial vector V0 of the detection target block to the selector 402 and the vector evaluation unit 104. Output, and proceed to step S402. As a result, each unit of the iterative gradient method computing unit 103 executes processing in units of blocks. In step S402, the selector 402 selects the initial vector V0 input from the mode selection unit 401 as an offset vector in accordance with control from the vector evaluation unit 104, and selects the selected offset vector as a time pixel difference calculation unit 411 and The result is output to the time pixel difference calculation unit 421, and the process proceeds to step S403.

  In step S403, the temporal pixel difference calculation unit 411 and the pixel difference value determination unit 412 execute the effective pixel determination process in units of blocks using the selected offset vector (in this case, the initial vector V0 is offset). Then, the process proceeds to step S404. The details of the effective pixel determination process will be described later with reference to FIG. 40. By this effective pixel determination process, the effective pixel number counter 413 has a pixel difference value of a predetermined pixel difference within the calculation unit in block units. The number of pixels determined to be smaller than the value (that is, determined to be an effective pixel in the subsequent gradient method calculation) is counted.

  In step S404, the gradient method continuation determination unit 414 determines whether or not the number of pixels counted by the effective pixel number counter 413 (effective pixel number) is greater than a predetermined threshold value α, and the effective pixel number is predetermined. If it is determined that the value is larger than the threshold value α, a flag (flg = 1) for executing the gradient method calculation is output to the gradient method calculation unit 404 for each block, and the process proceeds to step S405.

  In step S405, the gradient method calculation unit 404 uses the offset vector from the selector 402 (in this case, the initial vector V0) based on the flag from the gradient method continuation determination unit 414, and performs gradient method calculation processing in units of blocks. Is executed, and the process proceeds to step S406. The details of the gradient method calculation process will be described later with reference to FIG. 41, and the gradient vector calculation process determines the motion vector Vn and outputs it to the vector evaluation unit 104 and the delay unit 405. The motion vector Vn is obtained by adding the offset vector Vn−1 to the motion vector vn calculated by the first gradient method calculation (Vn = Vn−1 + vn). For example, the motion vector V1 is obtained by adding an offset vector (initial vector V0) to the motion vector v1 calculated by the first gradient method calculation (V1 = V0 + v1).

  In step S406, the vector evaluation unit 104 evaluates the motion vector Vn obtained by the gradient method calculation process DFD (n) and the evaluation value DFD (n-1) of the motion vector Vn-1 used as the offset vector. , And whether or not the evaluation value DFD (n) is smaller than the evaluation value DFD (n−1), that is, the motion vector obtained by the gradient method calculation processing than the motion vector Vn−1 used as the offset vector It is determined whether or not the reliability of Vn is high. Specifically, for example, in the first process, the obtained evaluation value DFD (1) of the motion vector V1 is compared with the evaluation value DFD (0) of the initial vector V0 used as an offset vector. In the second processing, the obtained evaluation value DFD (2) of the motion vector V2 is compared with the evaluation value DFD (1) of the motion vector V1 used as the offset vector.

  When the vector evaluation unit 104 determines in step S406 that the evaluation value DFD (n) is smaller than the evaluation value DFD (n-1), that is, the reliability of the motion vector Vn is higher than the motion vector Vn-1. In step S407, the calculated motion vector Vn (in the first case, the motion vector V1, in the second case, the motion vector V2) is set as the motion vector V of the detection target block. The process proceeds to S408. In step S408, the vector evaluation unit 104 counts the number of iterations n by one, proceeds to step S409, determines whether or not the number of iterations n has reached the set maximum number of iterations (for example, two times), and If it is determined that the number of iterations n is not the set maximum number of iterations, the selector 402 is controlled to return to step S402 to repeat the subsequent processing.

  That is, in step S402, the selector 402 selects the motion vector V1 output from the vector calculation unit 425 and held in the delay unit 405 as an offset vector, and in step S403, the time pixel difference calculation unit 411 and the pixel difference value are selected. The determination unit 412 executes the effective pixel determination process in units of blocks using the motion vector V1 as an offset vector, and repeats the subsequent processes. Therefore, the motion vector V2 calculated in step S405 is obtained by adding the motion vector V1 used as the offset vector to the motion vector v2 calculated by the second gradient method calculation (that is, V2 = V1 + v2 = V0 + v1 + v2). )

  When the vector evaluation unit 104 determines in step S406 that the evaluation value DFD (n) is not smaller than the evaluation value DFD (n−1), that is, the motion vector Vn−1 is more than the motion vector Vn. If it is determined that the reliability is high, the process proceeds to step S410, and the vector Vn−1 (the initial vector V0 in the first time, the motion vector V1 in the second time) is used as the offset of the gradient method calculation. The motion vector V is set and the process proceeds to step S411. If the vector evaluation unit 104 determines in step S409 that the number of iterations n has reached the set maximum number of iterations, the vector evaluation unit 104 proceeds to step S411.

  In step S411, the vector evaluation unit 104 determines whether or not the motion vector V is within a search area preset as a range for detecting a motion vector, and determines that the motion vector V is within the search area. If so, the process proceeds to step S415, the motion vector V is stored in the detection vector memory 53 in association with the detection target block, and the iterative gradient method process is terminated.

  If the vector evaluation unit 104 determines in step S411 that the motion vector V is not within the search area, the vector evaluation unit 104 proceeds to step S412, sets the motion vector V to 0 vector, proceeds to step S415, and proceeds to step S415. (0 vector) is stored in the detection vector memory 53 in association with the detection target block, and the iterative gradient method processing is terminated.

  On the other hand, if the gradient method continuation determination unit 414 determines in step S404 that the effective pixel number of the effective pixel number counter 413 is equal to or less than the predetermined threshold value α, the process proceeds to step S413 and controls the mode selection unit 401. The pixel unit processing mode is selected, and the process proceeds to step S414. As a result, each unit of the iterative gradient method computing unit 103 executes processing in units of pixels. In step S414, the mode selection unit 401 executes the iterative gradient method calculation process for each pixel, and the process proceeds to step S415. The details of the iterative gradient method computing process in the pixel unit processing mode will be described later with reference to FIG. 42, but the motion vector V of all pixels in the detection target block is obtained by the iterative gradient method computing process in the pixel unit processing mode. Therefore, in step S415, the vector evaluation unit 104 stores the motion vector V in the detection vector memory 53 in association with each pixel of the detection target block, and ends the iterative gradient method processing.

  In step S406 described above, the reliability of the motion vector is evaluated by evaluating the evaluation value DFD (n) of the motion vector Vn obtained by the gradient method processing and the motion vector Vn-1 used as the offset vector. Although the determination is made by comparing the value DFD (n−1), simultaneously with the comparison determination of the evaluation value DFD, the motion vector (motion vector before adding the offset vector) vn obtained by the gradient method calculation It may be determined that the value (| vn |) is smaller than a predetermined size (for example, 16 in the case of the calculation block 8 × 8).

  Here, the norm represents the magnitude of a vector in a certain vector space. For example, the norm of the vector v from the origin (0, 0) to (x, y) is expressed by the following equation (13). expressed.

・・・（１３）

... (13)

  That is, in the gradient method calculation, a high-precision result is generally obtained for a minute motion, but the norm value | vn | of the motion vector obtained by the gradient method calculation exceeds a predetermined magnitude. In other words, a highly accurate result is not always obtained for the large motion vector vn. Therefore, in parallel with the evaluation value DFD, it is determined whether or not the norm value | vn | of the motion vector obtained by the gradient method calculation exceeds a predetermined magnitude, and the norm value | vn | By excluding motion vectors that exceed a predetermined size, in this gradient method computation, it is possible to obtain highly accurate results for both small and large motions. The accuracy of detection can be improved.

  In the case of the determination of the evaluation value DFD and the determination based on the norm value, in step S406, when | vn | ≦ 16, and evaluation value DFD (n−1)> evaluation value DFD (n). If there is, the process proceeds to step S407, where V = Vn is set, and if | vn |> 16 in step S406, or evaluation value DFD (n−1) ≦ evaluation value DFD (n). In this case, the process proceeds to step S410, and V = Vn−1 is set.

  Next, details of the effective pixel determination process will be described with reference to the flowchart of FIG. FIG. 40 shows an example of the effective pixel determination process in step S403 of FIG.

  When the offset vector is input from the selector 402, the temporal pixel difference calculation unit 411 controls the pixel difference value determination unit 412 in step S431, resets the effective pixel number counter 413, proceeds to step S432, and proceeds to block unit. In the calculation block, a pixel is selected, and the process proceeds to step S433. The pixels are selected in the raster scan order from the upper left pixel of the calculation block.

  In step S433, the time pixel difference calculation unit 411 uses the offset vector and the frame t of the image at time t and the frame t + 1 of the image at time t + 1 input via the pre-filters 102-1 and 102-2. Then, the pixel difference Δt in the time direction of the selected pixel is calculated, and the calculated pixel difference Δt in the time direction is output to the pixel difference value determination unit 412 and the process proceeds to step S434.

  In step S434, the pixel difference value determination unit 412 determines whether or not the time direction pixel difference Δt calculated by the time pixel difference calculation unit 411 is smaller than a predetermined pixel difference value. It is determined whether or not the pixel is effective for the legal calculation, and if it is determined that the pixel difference Δt in the time direction is smaller than the predetermined pixel difference value, the process proceeds to step S435, and the effective pixel counter 413 determines the effective pixel number. The number is counted by one, and the process proceeds to step S436. Further, the pixel difference value determination unit 412 determines in step S434 that the time-direction pixel difference Δt is greater than or equal to a predetermined pixel difference value, that is, it is not a pixel that is effective for the subsequent gradient method calculation. Then, the process of step S435 is skipped, and the process proceeds to step S436.

  In step S436, the time pixel difference calculation unit 411 determines whether or not the processing of all the pixels in the calculation block has been completed, and determines that the processing of all the pixels in the calculation block has not ended. Returning to step S432, the next pixel is selected, and the subsequent processing is repeated. On the other hand, if the time pixel difference calculation unit 411 determines in step S436 that all the pixels in the calculation block have been processed, the time pixel difference calculation unit 411 ends the effective pixel determination process and returns to step S404 in FIG.

  Through the above processing, the effective pixel counter 413 counts the number of pixels in the calculation block whose pixel difference Δt is smaller than a predetermined pixel difference value and determined to be effective in the subsequent gradient method calculation. Yes. Therefore, the gradient method continuation determination unit 414 performs the gradient method calculation in units of blocks or performs the gradient method calculation in units of pixels by referring to the number of pixels of the effective pixel counter 413 in step S404 of FIG. Can be determined.

  Next, the details of the gradient method calculation processing in units of blocks will be described with reference to the flowchart of FIG. FIG. 41 shows an example of the gradient method calculation processing in units of blocks in step S405 of FIG. Therefore, in the example of FIG. 41, the flag (flg = 1) for executing the gradient method calculation is input from the gradient method continuation determination unit 414 to the time pixel difference calculation unit 421.

  When the flag (flg = 1) is input from the gradient method continuation determination unit 414, the time pixel difference calculation unit 421 selects a pixel in the calculation unit in block units in step S451, and proceeds to step S452. The pixels are selected in the raster scan order from the upper left pixel of the calculation block.

  In step S452, the time pixel difference calculation unit 421 calculates the offset vector from the selector 402, the frame t of the image at time t input via the prefilters 102-1 and 102-2, and the image at time t + 1. The pixel difference Δt in the time direction of the selected pixel is calculated using the frame t + 1, and the calculated pixel difference Δt in the time direction and the offset vector are output to the pixel difference value determination unit 422, and the process proceeds to step S453.

In step S453, the pixel difference value determination unit 422 determines whether or not the time direction pixel difference Δt calculated by the time pixel difference calculation unit 421 is smaller than a predetermined pixel difference value. It is determined whether or not the pixel is effective for legal calculation, and if it is determined that the pixel difference Δt in the time direction is smaller than the predetermined pixel difference value, the process proceeds to step S454 and the vertical horizontal pixel difference calculation unit 423 is controlled. Then, the pixel difference Δx in the horizontal direction of the pixel determined to be effective is calculated. Note that the predetermined pixel difference value in the pixel difference value determination unit 422 may be the same value as or different from the predetermined pixel difference value in the pixel difference value determination unit 412. At this time, the pixel difference value determination unit 422 outputs the pixel difference Δt in the time direction and the offset vector calculated by the time pixel difference calculation unit 421 to the vertical horizontal pixel difference calculation unit 423.

  In step S454, the vertical horizontal pixel difference calculation unit 423 calculates the offset vector and the frame t of the image at time t and the frame t + 1 of the image at time t + 1 input via the prefilters 102-1 and 102-2. The pixel difference Δx in the horizontal direction of the pixel determined to be effective is calculated, and the process proceeds to step S455, where the pixel difference Δy in the vertical direction of the pixel determined to be effective is calculated, and the pixel in the time direction is calculated. The difference Δt, the pixel difference Δx in the horizontal direction, and the pixel difference Δy in the vertical direction are output to the gradient integration unit 424, and the process proceeds to step S456. In step S456, the gradient integration unit 424 integrates the time-direction pixel difference Δt, the horizontal pixel difference Δx, and the vertical pixel difference Δy from the vertical horizontal pixel difference calculation unit 423, and calculates a vector result of the integration. The data is output to the unit 425, and the process proceeds to step S457.

  If the pixel difference value determination unit 422 determines in step S453 that the time-direction pixel difference Δt is greater than or equal to the predetermined pixel difference value, the pixel difference value determination unit 422 skips the processing of steps S454 to S456 and proceeds to step S457. That is, each pixel difference (gradient) of this pixel is not used for the calculation because it is not a pixel effective for the subsequent gradient method calculation.

  In step S457, the time pixel difference calculation unit 421 determines whether or not the processing of all the pixels in the calculation block has been completed, and determines that the processing of all the pixels in the calculation block has not ended. Returning to step S451, the next pixel is selected, and the subsequent processing is repeated. On the other hand, if the time pixel difference calculation unit 421 determines in step S457 that the processing of all the pixels in the calculation block has been completed, the time pixel difference calculation unit 421 outputs an offset vector to the vector calculation unit 425, and proceeds to step S458. When the offset vector is input from the time pixel difference calculation unit 421, the vector calculation unit 425 uses the gradient summation result from the gradient summation unit 424 and the least square sum of the above-described equation (9) to obtain the motion vector vn. Then, the process proceeds to step S459, the motion vector Vn is obtained by adding the calculated motion vector vn to the offset vector from the time pixel difference calculation unit 421, and the obtained motion vector Vn is sent to the vector evaluation unit 104. Output, and finish the gradient method calculation process, and return to step S406 in FIG.

  As described above, when the pixel difference Δt in the time direction is equal to or larger than a predetermined pixel difference value, the gradient at the time of detecting a motion vector is assumed that the pixel may cause a disturbance in motion. Since it is excluded from the calculation of the method, a more stable gradient method calculation is executed, and as a result, a probable motion vector is detected, thereby improving the accuracy of the motion vector detection.

  Next, the details of the iterative gradient method calculation processing in units of pixels will be described with reference to the flowchart of FIG. FIG. 42 shows an example of the iterative gradient method computing process for each pixel in step S414 of FIG. This process is executed for each pixel in the detection target block.

  When the pixel unit processing mode is set, the mode selection unit 401 selects the pixel of the detection target block as the detection target pixel in step S471, and selects the initial vector V0 of the calculation block (for example, 9 × 9) of the pixel as a selector. 402 and the vector evaluation unit 104, and the process proceeds to step S472. The pixels are selected in the raster scan order from the upper left pixel of the detection target block. In step S472, the selector 402 selects the initial vector V0 input from the mode selection unit 401 as an offset vector in accordance with the control from the vector evaluation unit 104, and selects the selected offset vector as a time pixel difference calculation unit 411 and The result is output to the time pixel difference calculation unit 421, and the process proceeds to step S473.

  In step S473, the temporal pixel difference calculation unit 411 and the pixel difference value determination unit 412 execute the effective pixel determination process in units of pixels using the selected offset vector (in this case, using the initial vector V0 as an offset). The process proceeds to step S474. The details of the effective pixel determination process are basically the same except that the effective pixel determination process in units of blocks described above with reference to FIG. 40 is different from the target calculation block (block range and pixels constituting the block). However, by this effective pixel determination process, the effective pixel number counter 413 has a pixel difference value smaller than a predetermined pixel difference value in the calculation unit in units of pixels. (That is, the number of pixels determined to be effective pixels in the subsequent gradient method calculation) is counted. It should be noted that the predetermined pixel difference value for determining the effective pixel in pixel units may be the same value as in the block unit or may be set to a different value.

  In step S474, the gradient method continuation determination unit 414 determines whether or not the number of pixels (effective pixel number) counted by the effective pixel number counter 413 is greater than a predetermined threshold value β, and the effective pixel number is predetermined. If it is determined that the value is larger than the threshold value β, a flag (flg = 1) for executing the gradient method calculation is output to the gradient method calculation unit 404, and the process proceeds to step S475. Note that the threshold value β may be the same value as the threshold value α in the case of a block unit, or a different value may be set.

  In step S475, based on the flag from the gradient method continuation determination unit 414, the gradient method calculation unit 404 uses the offset vector (initial vector V0) from the selector 402 to execute a gradient method calculation process for each pixel, The process proceeds to step S476. The details of the gradient method calculation processing are basically the same as those described above with reference to FIG. 41 except that the block unit gradient method calculation processing is different from the calculation block (range and pixels constituting the block). Although detailed description thereof is omitted for the purpose of processing, a motion vector Vn of the detection target pixel is obtained by this gradient method calculation processing, and is output to the vector evaluation unit 104 and the delay unit 405. The motion vector Vn is obtained by adding the offset vector to the motion vector vn obtained by the first gradient method calculation (Vn = Vn−1 + vn).

  In step S476, the vector evaluation unit 104 obtains the evaluation value DFD (n) of the motion vector Vn calculated by the gradient method processing and the evaluation value DFD (n−1) of the motion vector Vn−1 used as the offset. , Whether the evaluation value DFD (n) is smaller than the evaluation value DFD (n−1), that is, the reliability of the motion vector Vn calculated by the gradient method processing than the motion vector Vn−1 used as the offset Judge whether or not is high.

  In step S476, the vector evaluation unit 104 determines that the evaluation value DFD (n) is smaller than the evaluation value DFD (n−1), that is, the reliability of the motion vector Vn is higher than the motion vector Vn−1. If it is determined that the value is high, in step S477, the calculated motion vector Vn (the first time, the motion vector V1, and the second time, the motion vector V2) is set as the motion vector V of the detection target pixel, The process proceeds to step S478. In step S478, the vector evaluation unit 104 counts the number of iterations n by one, proceeds to step S479, determines whether or not the number of iterations n has reached the set maximum number of iterations (for example, two times), and If it is determined that the number of iterations n is not the set maximum number of iterations, the selector 402 is controlled to return to step S472 to repeat the subsequent processing.

  That is, in step S472, the selector 402 selects the motion vector V1 output from the vector calculation unit 425 and held in the delay unit 405 as an offset vector, and in step S473, the temporal pixel difference calculation unit 411 and the pixel difference value determination The unit 412 offsets the motion vector V1, which is an offset vector, executes a valid pixel determination process in units of pixels, and repeats the subsequent processes. Therefore, the motion vector V2 obtained in step S475 is obtained by adding the offset motion vector V1 to the motion vector v2 calculated by the second gradient method calculation (that is, V2 = V0 + v1 + v2).

  In step S476, the vector evaluation unit 104 determines that the evaluation value DFD (n) is larger than the evaluation value DFD (n−1), that is, the motion vector Vn−1 is more reliable than the motion vector Vn. When it is determined that the degree is high, the process proceeds to step S480, and the vector Vn−1 (the initial vector V0 in the first case, the motion vector V1 in the second time) is used as the offset of the gradient method calculation. The motion vector V is set, and the process proceeds to step S481. If the vector evaluation unit 104 determines in step S479 that the number of iterations n has reached the set maximum number of iterations, the vector evaluation unit 104 proceeds to step S481.

  In step S481, the vector evaluation unit 104 determines whether or not the motion vector V is within a search area preset as a range for detecting a motion vector, and determines that the motion vector V is within the search area. If so, the process proceeds to step S483. If the vector evaluation unit 104 determines in step S481 that the motion vector V is not within the search area, the vector evaluation unit 104 proceeds to step S482, sets the motion vector V to 0 vector, and proceeds to step S483.

  On the other hand, in step S474, when the gradient method continuation determination unit 414 determines that the effective pixel number of the effective pixel number counter 413 is less than the predetermined threshold value β, the gradient method calculation unit 404 has a flag ( flg = 0) is output, and the flow advances to step S482. Based on the flag, the temporal pixel difference calculation unit 421 controls the vector calculation unit 425 to set the motion vector V to 0 vector, and the process proceeds to step S483.

  When the mode selection unit 401 determines in step S483 whether or not the processing of all the pixels in the detection target block has been completed, and determines that the processing of all the pixels in the detection target block has not been completed, Returning to step S471, the subsequent pixels of the detection target block are set as detection target pixels, and the subsequent processing is repeated. If the mode selection unit 401 determines in step S483 that all the pixels in the detection target block have been processed, the mode selection unit 401 ends the iterative gradient method calculation process for each pixel and returns to step S415 in FIG. That is, since the motion vector V is obtained for all the pixels in the detection target block by the iterative gradient method computing process in units of pixels, the process returns to step S415 in FIG. 39, and the vector evaluation unit 104 converts the motion vector V to the detection target block. Are stored in the detection vector memory 53 in correspondence with each of the pixels, and the iterative gradient method processing is terminated.

  Note that, in the evaluation of the reliability of the motion vector in step S476, the evaluation value DFD determination and the norm value determination may be used in combination, as in the case of step S406.

  As described above, by using only pixels whose pixel difference in the calculation block is smaller than the predetermined pixel difference value for the gradient method calculation, pixels having different motions can be excluded from the calculation target, and the gradient method calculation is performed. As a result, the certainty of the motion vector required by the method improves, and the accuracy of motion vector detection improves.

  In addition, when the pixel difference in the calculation block is larger than the predetermined pixel difference value and the number of pixels that are not used for the gradient method calculation is larger than the predetermined threshold value, the gradient method calculation processing is adaptively performed. Therefore, the accuracy of motion vector detection can be further improved especially at the boundary of the motion of the object, and as a result, the required motion vector can be made more probable.

  Specifically, when the pixel difference in the calculation block is larger than the predetermined pixel difference value and the number of pixels that are not used in the gradient method calculation is larger than the predetermined threshold value, the block unit mode is changed to the pixel unit. Since the calculation block is set for each pixel of the detection target block and the validity determination process and the gradient method calculation process are performed, the motion vector can be detected with higher accuracy depending on the pixel. In addition, when the pixel difference in the calculation block is larger than the predetermined pixel difference value and the number of pixels that are not used for the gradient method calculation is larger than the predetermined threshold value, the gradient method calculation processing is discontinued, Since the zero vector is used, detection of a violent motion vector can be suppressed and a safer motion vector can be detected.

  Next, details of the configuration of the vector allocation unit 54 will be described.

  FIG. 43 is a block diagram showing the configuration of the vector allocating unit 54. 43 assigns motion vectors detected on the frame t using the frame t of the image at time t to which the 24P signal is input and the frame t + 1 of the image at time t + 1. A process of assigning to the pixels on the interpolation frame of the 60P signal to be interpolated on the vector memory 55 is performed.

  In the example of FIG. 43, the frame t of the image at time t and the frame t + 1 of the image at time t + 1 are input to the pixel information calculation unit 701, the evaluation value calculation unit 702, and the target pixel difference calculation unit 703.

  The pixel information calculation unit 701 acquires the motion vector detected by the pixel on the frame t in the detection vector memory 53 in the raster scan order from the upper left pixel, and extends the acquired motion vector in the frame t + 1 direction at the next time. The intersection of the extended motion vector and the interpolation frame is calculated. Then, the pixel information calculation unit 701 sets, from the intersection of the calculated motion vector and the interpolation frame, a pixel (hereinafter referred to as an allocation target pixel) to which the motion vector is assigned on the interpolation frame. The motion vector and the position information of the allocation target pixel are output to the vector selection unit 705. The image information calculation unit 701 calculates the position P of the frame t and the position Q on the frame t + 1 that are associated with the allocation target pixel and the motion vector, and the calculated position information on the frame t and the frame t + 1 The result is output to the evaluation value calculation unit 702 and the target pixel difference calculation unit 703.

  When the evaluation value calculation unit 702 receives from the pixel information calculation unit 701 position information on the frame t and the frame t + 1 associated with the allocation target pixel and the motion vector, the evaluation value calculation unit 702 receives the position P of the frame t and the position Q of the frame t + 1. In order to calculate the evaluation value DFD, a certain range of DFD calculation ranges (m × n) centering on the position P and the position Q are respectively set, and it is determined whether or not these DFD calculation ranges are within the image frame. . When the evaluation value calculation unit 702 determines that the DFD calculation range is within the image frame, the evaluation value calculation unit 702 calculates the evaluation value DFD of the allocation target pixel for the motion vector by calculating using the DFD calculation range, and the calculated evaluation value The DFD is output to the vector evaluation unit 704.

  When the pixel-of-interest difference calculation unit 703 receives from the pixel information calculation unit 701 position information on the frame t and the frame t + 1 associated with the allocation target pixel and the motion vector, the position P of the frame t and the position Q of the frame t + 1 Is used to obtain the absolute value of the luminance difference for the allocation target pixel, and the obtained absolute value of the luminance difference is output to the vector evaluation unit 704.

  The vector evaluation unit 704 includes a pixel difference determination unit 711 and an evaluation value determination unit 712. The pixel difference determination unit 711 determines whether the luminance difference absolute value for the allocation target pixel input from the target pixel difference calculation unit 703 is smaller than a predetermined threshold value. The evaluation value determination unit 712 calculates the evaluation value when the pixel difference determination unit 711 determines that the luminance difference absolute value for the allocation target pixel input from the target pixel difference calculation unit 703 is smaller than a predetermined threshold value. It is determined whether the evaluation value DFD of the allocation target pixel input from the unit 702 is smaller than the minimum evaluation value of the DFD table included in the vector selection unit 705. When the evaluation value determination unit 712 determines that the evaluation value DFD of the allocation target pixel is smaller than the minimum evaluation value of the DFD table, the evaluation value determination unit 712 determines that the reliability of the motion vector corresponding to the allocation target pixel is high, The evaluation value DFD of the allocation target pixel is output to the selection unit 705.

  The vector selection unit 705 has a DFD table that holds a minimum evaluation value at each pixel on the interpolation frame, and an evaluation value DFD0 when a 0 vector is assigned to each pixel on the interpolation frame. The DFD table holds the minimum evaluation value for each pixel on the interpolation frame in advance. When the vector selection unit 705 receives the evaluation value DFD of the allocation target pixel from the vector evaluation unit 704, the vector selection unit 705 sets the flag of the allocation flag memory 56 to 1 (based on the position information of the allocation target pixel from the pixel information calculation unit 701). true), and the minimum evaluation value in the DFD table of the allocation target pixel is rewritten to the evaluation value DFD of the allocation target pixel. Further, the vector selection unit 705 allocates the motion vector from the pixel information calculation unit 701 to the allocation target pixel in the allocation vector memory 55 based on the information on the position of the allocation target pixel from the pixel information calculation unit 701.

  Next, the accuracy below the pixel of the motion vector will be described. In the calculation of the DFD evaluation expressed by the above formula (1), the phase p + v on the previous frame t + 1 obtained by shifting the pixel position p of the frame t by the amount of the vector v is actually on the frame t + 1 of the 24p signal. In many cases, the luminance value is not defined. Therefore, in order to perform the calculation of the evaluation value DFD for the motion vector v having subpixel accuracy, it is necessary to generate a luminance value in a phase below the pixel by some method.

  Corresponding to this, there is a method of using the luminance value of the pixel closest to the phase p + v on the previous frame t + 1 obtained by shifting the pixel position p of the frame t by the vector v amount. However, in this method, since the sub-pixel components of the motion vector to be evaluated are rounded, the sub-pixel components of the motion vector are discarded, and the reliability of the evaluation value DFD thus obtained is lowered. End up.

  Therefore, in the present invention, a four-point interpolation process using the luminance values of the surrounding four pixels is used. FIG. 44 is a diagram showing the concept of the four-point interpolation process of the present invention. In FIG. 44, the arrow X indicates the horizontal direction in the frame t + 1, and the arrow Y indicates the vertical direction in the frame t + 1. In this frame t + 1, the white circle represents the pixel position on the frame t + 1, and the black dot represents the position below the pixel. Further, the upper left black point p + v and its surrounding four pixels on the frame t + 1 are shown enlarged in the window E. In window E, the alphabet in the white circle indicates the luminance value of the surrounding four pixels.

If the black point p + v at the upper left in the frame t + 1 is the phase p + v after the pixel position p of the frame t is shifted by the vector v amount, the luminance value F _{t + 1} (p + v) of the phase p + v is the horizontal direction of the phase p + v. The pixel sub-component α and the pixel sub-component β in the vertical direction and the luminance values L0 to L4 of the four surrounding pixels of the phase p + v are used to obtain the sum of the inverse ratios of the distances of the four surrounding pixels. That is, the luminance value F _{t + 1} (p + v) is expressed by the following equation (14).

F _{t + 1} (p + v) = (1-α) (1-β) L0 + α (1-β) L1
+ (1-α) βL2 + αβL3 (14)

As described above, the calculation of the DFD evaluation is performed using the luminance value F _{t + 1} (p + v) obtained by the four-point interpolation process, so that the reliability of the evaluation value DFD can be increased without increasing the hardware implementation cost. A decrease in the degree can be suppressed. In the following, an example in which the four-point interpolation is applied in the calculation of the evaluation value DFD and the luminance difference absolute value at the time of vector allocation will be described. Of course, the above-described initial vector selection processing, vector detection processing, Alternatively, this four-point interpolation is also applied to the calculation of the evaluation value DFD in the case of evaluating a vector such as allocation compensation processing described later.

  Next, the concept of vector allocation processing will be described. For example, when it is assumed that an object is moving at a velocity v on a 24P signal and a constant velocity assumption is made for the motion of the object between any two frames, a new frame is interpolated between the frames of the 24P signal. Think about what to do. In this case, when the motion vector v is extended from the 24P signal object, the intersection of the motion vector v and the interpolation frame is the same object and has the same velocity v.

  Therefore, the motion vector of the frame of the 24P signal detected by the vector detection unit 53 (hereinafter also referred to as an original frame with respect to the interpolation frame) is represented by the motion vector and the interpolation frame of the 60P signal to be interpolated. By assigning them to the intersections, the movement of each pixel on the interpolation frame can be obtained. Conversely, from the allocated motion vector, it can be determined from which position the pixel on the interpolation frame has moved on the 24P signal frame.

  FIG. 45 shows one-dimensional examples of motion vectors detected in the original frame of the 24P signal and pixels on the interpolation frame of the 60P signal. In the example of FIG. 45, between the frame t at the time t and the frame t + 1 at the time t + 1 of the two 24P signals, for example, the interpolation frame F1 at the time t + 0.4 and the interpolation frame F2 at the time t + 0.8 are Two are inserted. As described above, the position of this interpolation frame is the position where the time phase on the 24P signal is 0.0, 0.4, 0.8, 1.2, and 1.6, and the 60P signal. Are set in advance in the signal processing apparatus 1.

  A circle on each frame indicates each pixel. Motion vectors v1, v2, and v3 detected in the frame t by the preceding vector detection unit 52 are extended in the frame t + 1 direction. When these motion vectors are allocated to each pixel of the interpolation frames F1 and F2, the motion vector passing through the vicinity of each pixel on the interpolation frame is a candidate vector (hereinafter also referred to as allocation candidate vector) allocated to that pixel. Is done.

  Accordingly, the motion vector v1 from the pixel on the left side of the frame t to the vicinity of the fourth and fifth pixels from the left of the frame t + 1 is between the second and third pixels from the left on the interpolation frame F1. It passes between the third and fourth pixels from the left on F2. Therefore, the motion vector v1 is a pixel included in the vicinity N1 of the point where the motion vector v1 intersects the interpolation frames F1 and F2 (second and third pixels from the left of the interpolation frame F1 and the left of the interpolation frame F2). To the third and fourth pixels).

  The motion vector v2 from the third pixel from the left of the frame t to the vicinity of the second and third pixels from the left of the frame t + 1 is between the second and third pixels from the left on the interpolation frame F1, It passes between the second and third pixels from the left on the interpolation frame F2. Therefore, the motion vector v2 is a pixel included in the vicinity region N2 of the point where the motion vector v2 intersects the interpolation frames F1 and F2 (second and third pixels from the left of the interpolation frame F1 and the interpolation frame F2). Allocation candidate vectors to be allocated to the second and third pixels from the left).

  Furthermore, the motion vector v3 from the fifth pixel from the left of the frame t to the vicinity of the fourth and fifth pixels from the left of the frame t + 1 is between the fourth and fifth pixels from the left on the interpolation frame F1, It passes between the fourth and fifth pixels from the left on the interpolation frame F2. Therefore, the motion vector v3 includes the pixels included in the vicinity region N3 of the point where the motion vector v3 intersects the interpolation frames F1 and F2 (the fourth and fifth pixels from the left of the interpolation frame F1 and the interpolation frame F2). 4th and 5th pixels from the left).

  That is, the allocation candidate vector of the second pixel from the left of the interpolation frame F2 is the motion vector v2, and the second and third pixels from the left on the interpolation frame F1, and 3 from the left of the interpolation frame F2. The allocation candidate vectors of the th pixel are motion vectors v1 and v2. Further, the allocation candidate vectors of the fourth pixel from the left of the interpolation frame F2 are motion vectors v1 and v3, the fourth and fifth pixels from the left on the interpolation frame F1, and the left of the interpolation frame F2. The fifth pixel allocation candidate vector from is the motion vector v3.

  As described above, an allocation candidate vector to be allocated to each pixel on the interpolation frame is obtained from the motion vectors detected in the original frame. In the leftmost pixel and the rightmost pixel (black circles in the figure) of the interpolation frames F1 and F2, motion vectors passing through the vicinity are not shown. That is, there is no allocation candidate vector to be allocated to the leftmost pixel and the rightmost pixel of the interpolation frames F1 and F2. Therefore, an allocation compensation process is performed on these pixels in an allocation compensation unit 57 in a later stage described later.

Furthermore, with reference to FIG. 46, the motion vector detected in the original frame and the pixels on the interpolation frame of the 60P signal will be described in detail. In the example of FIG. 46, the arrow T indicates the direction of passage of time from the frame t at the left front time t to the frame t + 1 at the right rear time t + 1. Further, interpolation frame F1 is placed at time t + pos _t between times t and time t + 1.

In the example of FIG. 46, the motion vector v _a (x _va , y _va ) detected by the pixel (x _a , y _a ) on the frame t is extended in the frame t + 1 direction, and the intersection (x _ia , y _ia ) is calculated. Intersection, it is considered that pixels corresponding to end points of the motion vector v _a of the frame t of the 24P signal to a point moved, specifically, is expressed by the equation (15) and (16).

_{x ia = x a + pos t} x va ··· (15)
_{y ia = y a + pos t} y va ··· (16)

Here, as described above, when the motion vector v _a has a less precision pixels, the intersection of the motion vector v _a, pixel positions on the interpolation frame F1 does not always match. Otherwise, as shown in FIG. 46, the motion vector v _a is allocated with respect to the four neighboring pixels G1 to G4 of the intersection on the interpolated frame F1. That is, the motion vector v _a is shifted (parallel movement) on each of the pixels G1 to G4 in the neighborhood, is the allocation candidate vector to be allocated to each of the pixels, the allocation compensating process is executed.

  As described above, since one motion vector may be a candidate assigned to four neighboring pixels, a plurality of motion vectors may be assigned candidate vectors depending on the pixel. In this case, the vector allocating unit 54 calculates, for each motion vector, a pixel on the interpolation frame and an intersection point on the original frame associated with the motion vector, and evaluates each motion vector using the intersection point. Thus, finally, a motion vector to be assigned to the pixel on the interpolation frame is determined.

With reference to FIG. 47, the motion vector evaluation in the vector allocation unit 54 will be described. Figure 47 is a bottom, frame t at time t in FIG. 46, the time t + pos _t interpolated frame F1, and the frame t + 1 at time t + 1, is one-dimensionally illustrated.

In the example of FIG. 47, the motion vector sv _a is the pixel (x _a, y _a) of the frame t detected motion vector v _a at is as allocation candidate vectors of the pixels in the neighborhood G4, shifts on the pixel G4 ( Translated). Here, the intersections of the motion vector sva, shifted onto the pixel G4, and the frame t and the frame t + 1 are defined as _a point P and a point Q, respectively.

The vector allocating unit 54, a first evaluation of motion vector sv _a, first, obtains a DFD operation range around the point P and point Q respectively, DFD operation range determined whether jutting out the image frame Judging. Therefore, when the DFD operation range around the point P and point Q had protruding a picture frame, the motion vector sv _a is excluded from the candidates.

Further, when the point P and the point Q belong to different objects, for example, the difference between the luminance value F _t (P) of the point P and the luminance value F _{t + 1} (Q) of the point Q becomes large. . Accordingly, the vector allocating unit 54, a second evaluation of motion vector sv _a, using the points P and Q, determine the luminance difference absolute value dp of the pixel G4, the luminance difference absolute value dp is greater than a predetermined value Determine whether or not. If the luminance difference absolute value dp is determined to be larger than the predetermined value is a low reliability of the motion vector sv _a in the pixel G4, a motion vector sv _a is excluded from the candidates. The luminance difference absolute value dp is expressed by the following equation (17).

dp = | F _t (P) − F _{t + 1} (Q) | (17)

Then, as the third evaluation of the motion vector sv _a, the vector allocating unit 54 performs evaluation determined by the difference absolute value that represents the correlation value DFD operation range around the point P and the point Q. That is, the vector allocating unit 54, using the DFD operation range around the point P and the point Q, obtains the evaluated value DFD of the motion vector sv _a (absolute differences) in the pixel G4, obtained evaluation value DFD is It is determined whether or not it is smaller than the minimum evaluation value of the DFD table. As a result of the above evaluation, the vector assigning unit 54 assigns a motion vector having the smallest evaluation value DFD among the obtained evaluation values DFD to the pixel G4.

  As described above, since the motion vector of the allocation candidate in the pixel of the interpolation frame is evaluated using not only the evaluation value DFD of the allocation target pixel but also the luminance difference absolute value in the allocation target pixel, the conventional evaluation Rather than using only the value DFD, a reliable motion vector can be allocated to the allocation target pixel. As a result, the accuracy of vector allocation is improved.

  As described above, in order to evaluate the motion vector of the allocation candidate in the pixel of the interpolation frame, the position on the original frame associated with the motion vector is used with reference to the pixel of the interpolation frame. When obtaining the luminance difference absolute value dp and the evaluation value DFD, the motion vector of the allocation candidate is extended with reference to the pixel position of the interpolation frame, so that the intersection of the motion vector and the original frame is the pixel position of the original frame. The pixel values may not match, and the pixel value cannot be obtained as it is. In such a case, the four-point interpolation process described above with reference to FIG. 44 is executed.

  FIG. 48 shows an example of four-point interpolation in the vector allocation process. In FIG. 48, portions corresponding to those in FIGS. 46 and 47 are denoted by corresponding reference numerals, and the description thereof will be omitted because it will be repeated.

In the example of FIG. 48, the motion vector sv _a assignment candidate, because it is extended pixel position G4 in interpolated frame F1 as a reference, the point of intersection P between the motion vector sv _a frame t, in frame t does not coincide with the pixel position (white circle on the frame t), also intersecting point Q between the motion vector sv _a frame t + 1 also does not match the pixel positions on the frame t + 1 (open circles in frame t). Therefore, in the frame t and the frame t + 1, the above-described four-point interpolation is performed using the four pixels in the vicinity E centered at the intersection point P and the intersection point Q (white circles on the frame t and the frame t + 1), respectively. Pixel values of P and intersection point Q are obtained.

  As described above, in the vector allocation processing of the present invention, the luminance difference absolute value dp and the evaluation value DFD are calculated using the pixel values of the intersection P and the intersection Q obtained by the four-point interpolation processing. The luminance difference absolute value dp and the evaluation value DFD can be obtained with higher accuracy than the method of rounding the sub-pixel components.

  Next, details of the vector allocation processing will be described with reference to the flowchart of FIG. The frame t of the image at time t and the frame t + 1 of the image at time t + 1, which are the original frames of the 24P signal, are input to the pixel information calculation unit 701, the evaluation value calculation unit 702, and the target pixel difference calculation unit 703.

  When a new original frame is input, the pixel information calculation unit 701 controls the vector selection unit 705, initializes the allocation flag of the allocation flag memory 56 to 0 (False) in step S701, and proceeds to step S702. The allocation vector memory 55 is initialized with the 0 vector, and the process proceeds to step S703. As a result, a zero vector is assigned to a pixel to which no motion vector is assigned.

  In step S703, the pixel information calculation unit 701 controls the evaluation value calculation unit 702 to calculate the evaluation value DFD0 using the 0 vector for all the pixels on the interpolation frame, and the vector selection unit 705. And the evaluation value DFD0 of the 0 vector calculated by the evaluation value calculation unit 702 is stored in the DFD table as the minimum evaluation value for each pixel of the interpolation frame, and the process proceeds to step S704. That is, in step S703, the evaluation value calculation unit 702 calculates the evaluation value DFD0 using the 0 vector for all the pixels in the interpolation frame, and the calculated evaluation value DFD0 is passed through the vector evaluation unit 704. The data is output to the vector selection unit 705. Then, the vector selection unit 705 stores the evaluation value DFD0 input via the vector evaluation unit 704 as the minimum evaluation value of the corresponding pixel in the DFD table.

  In step S704, the pixel information calculation unit 701 selects a pixel from the original frame on the detection vector memory 53, and proceeds to step S705. In this case, pixels are selected in the raster scan order from the upper left of the frame.

  In step S705, the pixel information calculation unit 701 performs pixel position calculation processing, and proceeds to step S706. The details of the pixel position calculation process will be described later with reference to FIG. 50. However, the pixel position calculation process allows the motion vector detected in the pixel selected in step S704 to be assigned to the interpolation frame. An allocation target pixel is calculated, and a position on the original frame associated with the motion vector is calculated on the basis of the calculated allocation target pixel.

  In step S706, the pixel information calculation unit 701 selects the calculated allocation target pixel, outputs the selected allocation target pixel and its motion vector to the vector selection unit 705, and proceeds to step S707. At the same time, the pixel information calculation unit 701 outputs, to the evaluation value calculation unit 702 and the pixel-of-interest calculation unit 703, information on the position on the original frame associated with the motion vector based on the selected allocation target pixel. To do. In step S706, the pixel information calculation unit 701 selects an upper left pixel when there are a plurality of allocation target pixels.

  In step S707, the pixel information calculation unit 701 executes an allocation vector evaluation process for the selected allocation target pixel, and the process proceeds to step S708. The details of the allocation vector evaluation process will be described later with reference to FIG. 51. By this allocation vector evaluation process, the motion vector evaluation value DFD and the luminance difference absolute value in the allocation target pixel are obtained, and the motion in the allocation target pixel is determined. The reliability of the vector is determined, and the motion vector in the allocation vector memory 55 is rewritten with the motion vector determined to have high reliability as a result of these determinations.

  In step S708, the pixel information calculation unit 701 determines whether or not the processing of all allocation target pixels has been completed. If it is determined that the processing of all allocation target pixels has not yet been completed, Returning to S706, the next allocation target pixel is selected, and the subsequent processing is repeated.

  If it is determined in step S708 that the processing of all the allocation target pixels has been completed, the pixel information calculation unit 701 determines whether or not the processing of all the pixels of the original frame on the detection vector memory 53 has been completed in step S709. Determine whether. If the pixel information calculation unit 701 determines in step S709 that the processing of all the pixels of the original frame on the detection vector memory 53 has not been completed, the process returns to step S704 and the next of the original frame on the detection vector memory 53. This pixel is selected, and the subsequent processing is repeated. If the pixel information calculation unit 701 determines in step S709 that the process for all the pixels in the detection vector memory 53 has been completed, the pixel information calculation unit 701 ends the vector allocation process.

  Next, details of the pixel position calculation process will be described with reference to the flowchart of FIG. FIG. 50 shows an example of the pixel position calculation process in step S705 of FIG.

  In step S721, the pixel information calculation unit 701 acquires the motion vector detected by the pixel selected by the process in step S704 from the detection memory vector 53, and the process proceeds to step S722. If the motion vector of the selected pixel is a 0 vector, the allocation vector memory 55 stores the 0 vector as an initial value in advance, so that the subsequent steps S722 to S724 and step S706 in FIG. 49 are performed. The processes from S708 to S708 are skipped, and the process proceeds to step S709.

  In step S722, the pixel information calculation unit 701 calculates the intersection of the acquired motion vector and the interpolation frame. That is, the pixel information calculation unit 701 extends the acquired motion vector in the direction of the next frame t + 1, calculates the intersection point of the extended motion vector and the interpolation frame, and proceeds to step S723.

  In step S723, the pixel information calculation unit 701 sets an allocation target pixel from the intersection calculated from the motion vector and the interpolation frame, and proceeds to step S724. At this time, if the intersection point coincides with the pixel position on the interpolation frame, the pixel information calculation unit 701 sets the intersection point as the allocation target pixel. On the other hand, when the intersection does not coincide with the pixel position on the interpolation frame, the pixel information calculation unit 701 sets four pixels near the intersection on the interpolation frame as allocation target pixels as described above.

  In step S724, the pixel information calculation unit 701 is acquired with reference to each allocation target pixel, which is necessary for the evaluation value calculation unit 702 and the target pixel difference calculation unit 703 to obtain the evaluation value DFD and the luminance difference absolute value. The position on the original frame associated with the motion vector is calculated. Specifically, in step S724, the pixel information calculation unit 701 shifts (translates) the acquired motion vector to the set allocation target pixel, and the shifted motion vector and the intersection of the original frame The position is obtained, the pixel position calculation process is terminated, and the process returns to step S706 in FIG.

  Next, the details of the allocation vector evaluation process will be described with reference to the flowchart of FIG. 51 shows an example of the allocation vector evaluation process in step S707 of FIG.

  In step S706 of FIG. 49, the pixel information calculation unit 701 obtains the position on the original frame associated with the motion vector with reference to the selected allocation target pixel, and information on the obtained position on the original frame is obtained. The evaluation value calculation unit 702 and the target pixel difference calculation unit 703 are input.

  When the position information on the original frame is input from the pixel information calculation unit 701 to the evaluation value calculation unit 702, in step S741, the evaluation value calculation unit 702 calculates the evaluation value DFD of the motion vector in the allocation target pixel by using the frame t and the frame Each DFD calculation range (m × n) centered on the position on t + 1 is obtained, and the process proceeds to step S742, where it is determined whether or not the obtained DFD calculation range is within the image frame. If the evaluation value calculation unit 702 determines in step S742 that the DFD calculation range is outside the image frame, the evaluation value calculation unit 702 determines that the motion vector is not an allocation candidate vector to be allocated to the allocation target pixel, and steps S743 to S749. Is skipped, the assigned vector evaluation process is terminated, and the process returns to step S708 in FIG.

  If the evaluation value calculation unit 702 determines in step S742 that the obtained DFD calculation range is within the image frame, the evaluation value calculation unit 702 proceeds to step S743 and uses the DFD calculation range determined to be within the image frame, The evaluation value DFD of the target pixel is calculated, and the obtained evaluation value DFD is output to the evaluation value determination unit 712, and the process proceeds to step S744. At this time, if the position on the original frame is equal to or less than the pixel, the evaluation value DFD of the allocation target pixel is obtained by obtaining the luminance value of the intersection on the original frame using the above-described four-point interpolation. Calculated.

  On the other hand, when the position information on the original frame is input from the pixel information calculation unit 701, the target pixel difference calculation unit 703 obtains the luminance difference absolute value dp in the allocation target pixel in step S744, and the obtained luminance The difference absolute value dp is output to the pixel difference determining unit 711, and the process proceeds to step S745. At this time, if the position on the original frame is equal to or less than the pixel, the pixel-of-interest calculation unit 703 obtains the luminance value of the intersection on the original frame using the above-described four-point interpolation. The luminance difference absolute value dp in the allocation target pixel is calculated.

  In step S745, the pixel difference determination unit 711 determines whether the luminance difference absolute value dp of the allocation target pixel from the target pixel difference calculation unit 703 is equal to or less than a predetermined threshold value, and determines the luminance of the allocation target pixel. When it is determined that the difference absolute value dp is larger than the predetermined threshold value, it is determined that there is a high possibility that the intersections of the frame t and the frame t + 1 belong to different objects, that is, the motion vector is determined in the allocation target pixel. It is determined that the reliability is low and the assignment candidate vector is not assigned to the assignment target pixel, the processing of steps S746 to S749 is skipped, the assignment vector evaluation processing is terminated, and the process returns to step S708 of FIG.

  If the pixel difference determination unit 711 determines in step S745 that the luminance difference absolute value dp of the allocation target pixel is equal to or less than the predetermined threshold value, the process proceeds to step S746. In step S746, the evaluation value determination unit 712 refers to the DFD table of the vector selection unit 705, and the evaluation value DFD of the allocation target pixel from the evaluation value calculation unit 702 is the minimum allocation target pixel stored in the DFD table. It is determined whether or not the evaluation value (in this case, the evaluation value DFD0 of the zero vector) is smaller, and the evaluation value DFD of the allocation target pixel from the evaluation value calculation unit 702 is the allocation target pixel stored in the DFD table. If it is determined that the motion vector is equal to or greater than the minimum evaluation value, it is determined that the reliability of the motion vector is not high in the allocation target pixel, the processing in steps S747 to S749 is skipped, and the allocation vector evaluation processing is terminated. It returns to step S708 of 49.

  On the other hand, when the evaluation value determination unit 712 determines in step S746 that the evaluation value DFD of the allocation target pixel from the evaluation value calculation unit 702 is smaller than the minimum evaluation value of the allocation target pixel stored in the DFD table. The motion vector is determined to have the highest reliability based on the evaluation value DFD among the motion vectors compared so far in the allocation target pixel, and the evaluation value DFD of the allocation target pixel determined to have high reliability. Is output to the vector selection unit 705, and the process proceeds to step S747.

  Upon receiving the evaluation value DFD of the allocation target pixel from the evaluation value determination unit 712, the vector selection unit 705 rewrites the allocation flag of the allocation target pixel in the allocation flag memory 56 to 1 (True) in step S747, and in step S748. The minimum evaluation value corresponding to the pixel to be allocated in the DFD table is rewritten to the evaluation value DFD determined to have high reliability by the evaluation value determination unit 712, and the process proceeds to step S749.

  The vector selection unit 705 receives the allocation target pixel selected from the pixel information calculation unit 701 and its motion vector in step S706. Therefore, in step S749, the vector selection unit 705 rewrites the motion vector allocated to the allocation target pixel in the allocation vector memory 55 with the motion vector corresponding to the evaluation value DFD determined to have high reliability, and allocates the allocation vector. The evaluation process ends, and the process returns to step S708 in FIG.

  As described above, when selecting a motion vector to be allocated to the allocation target pixel of the interpolation frame, not only the evaluation value DFD but also a position on the original frame associated with the motion vector based on the allocation target pixel is obtained. Since the absolute value of the luminance difference of the allocation target pixel is treated separately and evaluated, the most probable motion vector is selected from the allocation candidate vectors rather than using the conventional evaluation value DFD. , It can be assigned to the pixel to be assigned. Thereby, the accuracy of vector allocation is improved, discontinuity of the image generated in the subsequent image interpolation processing, etc. can be suppressed, and the image quality can be improved.

  Furthermore, when obtaining the evaluation value DFD and the luminance difference absolute value, etc., when a pixel value at a position below the pixel is necessary, the value is obtained by linear interpolation based on the distance to the four neighboring pixels at the position below the pixel. As a result, it is possible to perform processing with sub-pixel position accuracy, and more accurately obtain the luminance difference absolute value dp and the evaluation value DFD than the conventional method of rounding off the sub-pixel components. From the allocation candidate vectors, it is possible to allocate a more probable motion vector by the pixel of interest. That is, the accuracy of vector allocation processing is improved.

  In addition, the evaluation value DFD based on the 0 vector is stored as an initial value in the DFD table in advance, and while the motion vector is sequentially processed, the evaluation value DFD based on a certain motion vector becomes the minimum evaluation value at that time. In this case, since the minimum evaluation value of the DFD table and the motion vector allocated to the allocation vector memory 55 are updated as needed, time and resources can be used efficiently.

  Next, details of the configuration of the allocation compensation unit 57 will be described.

  FIG. 52 is a block diagram showing the configuration of the allocation compensator 57. 52 is configured by an allocation vector determination unit 801 and a vector compensation unit 802. The allocation compensation unit 57, which has a configuration shown in FIG. A process is performed in which the motion vectors of surrounding pixels are supplemented.

  The motion vector is assigned to the pixel of the interpolation frame on the assigned vector memory 55 by the preceding vector assigning unit 54. Further, 1 (True) is written in the allocation flag of the pixel allocation flag memory 56 to which the motion vector is allocated by the vector allocation unit 54, and the allocation of the allocation flag memory 56 of the pixel to which no motion vector is allocated. 0 (False) is written in the flag.

  The allocation vector determination unit 801 refers to the allocation flag in the allocation flag memory 56 and determines whether or not a motion vector is allocated to the pixel of interest by the vector allocation unit 54. Then, the allocation vector determination unit 801 selects a pixel of interest for which no motion vector has been allocated by the vector allocation unit 54, controls the vector compensation unit 802 for the selected pixel of interest, and sets the surrounding pixels of the pixel of interest. A motion vector is selected and allocated on the interpolation frame of the allocation vector memory 55.

  The vector compensation unit 802 acquires the motion vector assigned to the peripheral pixel of the target pixel from the assigned vector memory 55, and uses the frame t of the image at time t and the frame t + 1 of the image at time t + 1 that are input. The obtained motion vector evaluation value DFD is obtained and compared, and the motion vector having the highest reliability based on the evaluation value DFD among the motion vectors assigned to the peripheral pixels of the target pixel is assigned to the assigned vector memory 55. Assigned to the pixel of interest. Further, the vector compensation unit 802 rewrites the allocation flag of the target pixel to which the motion vector is allocated to 1 (True).

  FIG. 53 is a block diagram showing the configuration of the vector compensation unit 802. The vector compensation unit 802 having the configuration shown in FIG. 53 includes a compensation processing unit 811 and an evaluation value calculation unit 812.

  The compensation processing unit 811 includes a memory 821 that stores a minimum evaluation value DFD and a motion vector of the minimum evaluation value DFD as candidate vectors (hereinafter also referred to as compensation candidate vectors), and is selected by the allocation vector determination unit 801. As the initial value of the pixel of interest, the zero vector evaluation value DFD is stored in the memory 821 as the minimum evaluation value, and the zero vector is stored in the memory 821 as the compensation candidate vector. The compensation processing unit 811 refers to the allocation flag memory 56 to determine the presence or absence of motion vectors of the peripheral pixels of the target pixel, acquires the motion vector allocated to the peripheral pixels from the allocation vector memory 55, and evaluates the evaluation value. The calculation unit 812 is controlled to calculate the evaluation value DFD of the motion vector.

  Further, the compensation processing unit 811 determines whether or not the evaluation value DFD calculated by the evaluation value calculation unit 812 is smaller than the minimum evaluation value stored in the memory 821, and the calculated evaluation value DFD is the minimum evaluation. When it is determined that the value is smaller than the value, the compensation candidate vector and the minimum evaluation value in the memory 821 are rewritten to the calculated evaluation value DFD and its motion vector, and finally, the peripheral pixels for which the evaluation value DFD is determined to be the smallest Are assigned to the target pixel in the allocation vector memory 55 as the motion vector of the target pixel. Further, the compensation processing unit 811 rewrites the allocation flag of the allocation flag memory 56 of the target pixel to which the motion vector is allocated to 1 (True).

  When the evaluation value calculation unit 812 acquires the motion vector of the surrounding pixels from the allocation vector memory 55, the evaluation vector calculation unit 812 uses the frame t of the 24P signal image at time t and the frame t + 1 of the image at time t + 1 to be input. The motion vector evaluation value DFD from 55 is calculated, and the calculated evaluation value DFD is output to the compensation processing unit 811.

  FIG. 54 is a diagram for explaining the principle of allocation compensation processing. In the example of FIG. 54, each pixel on the interpolation frame is shown. An arrow output from a pixel represents a motion vector assigned to each pixel, and a pixel without an arrow represents a pixel to which no motion vector is assigned.

  Here, based on the evaluation value DFD among the motion vectors assigned to the peripheral pixels in the vicinity of the target pixel P with respect to the central target pixel P to which no motion vector is assigned by the vector allocating unit 54 in the previous stage. Select and assign the one with high reliability. In the case of the example in FIG. 54, the motion vector (thick arrow) of the pixel above the target pixel P is selected and assigned to the target pixel P. This is a process executed based on the motion correlation described below.

  FIG. 55 is a diagram for explaining the principle of motion correlation. In the example of FIG. 55, an object O1 moving with a motion v1 on a certain frame and an object O2 moving with a motion v2 are shown. The target pixel P1 belonging to the object O1 and its vicinity K1 have substantially the same movement v1 as the object O1. Further, the target pixel P2 belonging to the object O2 and its vicinity K2 have substantially the same movement v2 as the object O2.

  As described above, the motion correlation represents that the motion of the pixels belonging to a certain object is often almost the same in a space at the same time (in the same frame). Therefore, for a pixel for which a motion vector could not be assigned, using such a motion correlation in a space at the same time (in the same frame), from the motion vectors of surrounding pixels, The corresponding motion vector for that pixel can be selected. In addition, although description is abbreviate | omitted, the correlation of a time direction is also the same.

  Next, with reference to FIGS. 56 to 62, the motion vector compensation processing executed based on the motion correlation will be described. That is, it is a process of selecting a motion vector from the motion vectors of surrounding pixels and supplementing it as the motion vector of the pixel of interest. In the example of FIG. 56, white circles represent pixels on the interpolation frame, and 8 peripheral pixels are shown around the pixel of interest P for which a motion vector is to be obtained. The motion vector of the pixel of interest P is obtained with reference to the motion vectors of the surrounding 8 pixels.

  In the example of FIG. 57, the upper left pixel, the upper right pixel, and the lower right pixel (black circle) among the eight pixels around the target pixel P are obtained by the preceding process (for example, the above-described vector allocation process). The motion vector (arrow) is shown. That is, the compensation candidate vectors for the pixel of interest P in this case are the motion vectors of the upper left pixel, the upper right pixel, and the lower right pixel. On the frame, since the motion vector is obtained in the raster scan order from the upper left pixel on the frame, there may be a pixel for which the motion vector is not yet obtained among the surrounding 8 pixels. Since a motion vector has not yet been obtained, it cannot be a compensation candidate vector.

  Here, as shown in the example of FIG. 58, the peripheral 8 pixels have the motion vector obtained by this process in addition to the pixel (black circle) having the motion vector obtained in the previous process. There are also pixels (hatched circles). That is, in this process, the result of the previous stage of this process itself is also used. Therefore, the compensation candidate vector of the pixel of interest P in the example of FIG. 58 is a pixel (hatched) having a motion vector of a pixel in which a motion vector already exists (black circle) and a motion vector obtained by this processing in the previous stage. (Circle) motion vector.

  As shown in the example of FIG. 59, a 0 vector (still vector) S0 with a motion amount of 0 can also be used as a compensation candidate vector. In the example of FIG. 58, the pixel having the motion vector obtained by this processing and the pixel having the motion vector already existed separately, but both are the same in that they have a motion vector. 59 to 62, the pixel having the motion vector obtained by this processing is also included in the pixel (black circle) in which the motion vector already exists. Therefore, in the example of FIG. 59, the compensation candidate vector of the pixel of interest P is composed of the motion vector of the pixel (black circle) in which the motion vector already exists and the zero vector S0.

  In order to compare the reliability (probability) of the compensation candidate vectors configured as described above, an evaluation value DFD, which is a motion vector evaluation method, is obtained as shown in FIGS. It is done. FIG. 60 shows an example in which a zero vector S0 is used as a compensation candidate vector. FIG. 61 shows an example in which the motion vector VK1 of the upper left pixels of the surrounding 8 pixels is used as the compensation candidate vector. FIG. 62 shows an example in which the motion vector VK2 of the upper center pixel of the surrounding 8 pixels is used as the compensation candidate vector.

  In the example of FIG. 60, an example is shown in which the zero vector S0 is selected from the compensation candidate vectors of the pixel of interest P shown on the left side of the drawing, and the evaluation value DFD of the selected zero vector S0 is obtained. That is, the evaluation value DFD for the 0 vector S0 is selected on the basis of the pixel of interest P on the interpolation frame on the frame t and the frame t + 1 of the 24P signal sandwiching the pixel of interest P (the interpolation frame of the 60P signal). An intersection with which the 0 vector S0 is associated is obtained, DFD calculation ranges D1-1 and D1-2 within a predetermined range (m × n) are calculated around the intersection, and the calculated DFD calculation range D1-1 is calculated. And D1-2 are used to calculate equation (1) described above.

  In the example of FIG. 61, the motion vector VK1 of the upper left pixel of the surrounding eight pixels is selected from the compensation candidate vectors of the target pixel P shown on the left side in the drawing, and the evaluation value DFD of the selected motion vector VK1 is obtained. An example is shown. In other words, the evaluation value DFD for the motion vector VK1 of the upper left pixel of the eight neighboring pixels is selected on the basis of the target pixel P on the interpolation frame on the frame t and the frame t + 1 sandwiching the target pixel P (interpolation frame). The intersection with which the motion vector VK1 is associated is obtained, the DFD calculation ranges D2-1 and D2-2 in a predetermined range (m × n) are calculated around the intersection, and the calculated DFD calculation range D2-1 is calculated. And D2-2 are used to calculate the above-described equation (1).

  In the example of FIG. 62, the motion vector VK2 of the upper center pixel of the surrounding 8 pixels is selected as the compensation candidate vector from the compensation candidate vectors of the target pixel P shown on the left side of the drawing, and the selected motion vector VK2 is selected. An example is shown in which the evaluation value DFD is obtained. That is, the evaluation value DFD for the motion vector VK2 of the upper central pixel of the surrounding eight pixels is based on the target pixel P on the interpolation frame on the frame t and the frame t + 1 sandwiching the target pixel P (interpolation frame). An intersection point to which the selected motion vector VK2 is associated is obtained, DFD calculation ranges D3-1 and D3-2 in a predetermined range (m × n) are calculated around this intersection point, and the calculated DFD calculation range D3- It is calculated | required by calculating Formula (1) mentioned above using 1 and D3-2.

  Note that the other compensation candidate vectors shown on the left side in the figure are basically the same processing, and thus the description thereof is omitted. However, as described above, all the compensation candidate vectors for the peripheral pixels of the pixel of interest P are as described above. Evaluation values DFD are obtained, and the obtained evaluation values DFD are compared. Among them, a compensation candidate vector having the smallest evaluation value DFD is assigned to the target pixel P as shown in FIG. Selected as the most reliable and probable motion vector.

  In the case of the example of FIG. 63, it is determined that the evaluation value DFD of the motion vector VK1 of the upper left pixel of the eight neighboring pixels is the smallest among the compensation candidate vectors of the peripheral pixels of the target pixel P, and the motion vector VK1 is Is selected and assigned as a motion vector.

  As described above, since the motion vector of the pixel that could not be allocated by the vector allocation unit 54 is compensated from the motion vector of the surrounding pixels using motion correlation, the motion vector is not allocated. , The disturbance of motion can be suppressed as compared with the case where the 0 vector is assigned. Also, the motion vector of the pixel compensated in this way can be used again as a compensation candidate vector for another pixel. That is, since not only motion vectors in the spatial direction but also motion vectors in the temporal direction can be used as compensation candidate vectors, pixels that perform substantially the same motion in the object have substantially the same motion vector. A stable motion vector that is selected and has few errors can be obtained. This improves the accuracy of vector assignment.

  Next, the details of the allocation compensation processing will be described with reference to the flowchart of FIG. The motion vector is assigned to the pixel of the interpolation frame on the assigned vector memory 55 by the preceding vector assigning unit 54. Further, 1 (True) is written in the allocation flag of the pixel allocation flag memory 56 to which the motion vector is allocated by the vector allocation unit 54, and the allocation of the allocation flag memory 56 of the pixel to which no motion vector is allocated. 0 (False) is written in the flag.

  In step S801, the allocation vector determination unit 801 selects a pixel of the interpolation frame in the allocation flag memory 56 as a target pixel, and the process proceeds to step S802. At this time, the allocation vector determination unit 801 selects pixels from the upper left pixel of the frame in the raster scan order.

  In step S802, the allocation vector determination unit 801 determines whether the allocation flag of the target pixel in the allocation flag memory 56 is 0 (False), and the allocation flag of the target pixel in the allocation flag memory 56 is If it is determined that the motion vector is 0 (False), it is determined that no motion vector is allocated, and the process proceeds to step S803, the compensation processing unit 811 is controlled to execute the vector compensation process, and the process proceeds to step S804. Details of the vector compensation processing will be described later with reference to FIG. 65. By this vector compensation processing, the smallest motion vector of the evaluation value DFD is stored as a compensation candidate vector among the motion vectors assigned to the peripheral pixels. 821 is stored.

  In step S804, the compensation processing unit 811 allocates the compensation candidate vector in the memory 821 as the motion vector of the target pixel to the allocation vector memory 55, and proceeds to step S805 to set the target pixel allocation flag in the allocation flag memory 56 to 1 Rewrite to (True), and go to Step S806.

  On the other hand, if the allocation vector determination unit 801 determines in step S802 that the allocation flag of the target pixel in the allocation flag memory 56 is 1 (True), a motion vector has already been allocated to the target pixel. Is determined, the process of steps S803 to S805 is skipped, and the process proceeds to step S806.

  In step S806, the allocation vector determination unit 801 determines whether or not the processing of all the pixels in the interpolation frame of the allocation flag memory 56 has been completed, and determines that the processing of all the pixels has not been completed. Returning to step S801, the next pixel of the interpolation frame in the allocation flag memory 56 is selected as the pixel of interest, and the subsequent processing is executed. If it is determined in step S806 that the processing of all the pixels in the interpolation frame in the allocation flag memory 56 has been completed, the allocation compensation processing is terminated.

  Next, details of the vector compensation processing will be described with reference to the flowchart of FIG. FIG. 65 shows an example of the vector compensation processing in step S803 of FIG.

  In step S821, the compensation processing unit 811 controls the evaluation value calculation unit 812 to calculate the evaluation value DFD0 using the 0 vector, and the process proceeds to step S822. Specifically, in step S821, the evaluation value calculation unit 812 uses the frame t of the image at time t and the frame t + 1 of the image at time t + 1, for example, as described above with reference to FIG. In addition, for the pixel of interest, an evaluation value DFD0 with a 0 vector is calculated, and the calculated evaluation value DFD0 is output to the compensation processing unit 811.

  In step S822, the compensation processing unit 811 stores the evaluation value DFD0 as the minimum evaluation value in the memory 821, proceeds to step S823, stores the 0 vector as the compensation candidate vector in the memory 821, and proceeds to step S824. In step S824, the compensation processing unit 811 selects one peripheral pixel from the eight peripheral pixels of the target pixel selected by the allocation vector determination unit 801, and the process proceeds to step S825. At this time, the compensation processing unit 811 selects peripheral pixels in the raster scan order from the upper left pixel among the eight peripheral pixels.

  In step S825, the compensation processing unit 811 refers to the allocation flag memory 56 and determines whether there is a motion vector of the selected peripheral pixel. If the allocation flag of the peripheral pixel in the allocation flag memory 56 is 1 (True), the compensation processing unit 811 determines in step S825 that there is a motion vector allocated to the selected peripheral pixel, and the process proceeds to step S826. Then, the motion vectors of the surrounding pixels are acquired from the allocation vector memory 55, and the process proceeds to step S827. At this time, the motion vectors of the surrounding pixels are also output from the allocation vector memory 55 to the evaluation value calculation unit 812.

  When the motion vector of the neighboring pixels is input from the allocation vector memory 55, the evaluation value calculation unit 812 uses the input frame t of the image at time t and the frame t + 1 of the image at time t + 1 in step S827. For the pixel of interest, the motion vector evaluation value DFD from the allocation vector memory 55 is calculated, and the calculated evaluation value DFD is output to the compensation processing unit 811, and the process proceeds to step S 828.

  When the evaluation value DFD is input from the evaluation value calculation unit 812, the compensation processing unit 811 determines whether or not the evaluation value DFD is smaller than the minimum evaluation value of the target pixel stored in the memory 821 in step S828. If it is determined that the evaluation value DFD is smaller than the minimum evaluation value of the target pixel stored in the memory 821, the process proceeds to step S829, and the minimum evaluation value of the memory 821 is determined to be smaller than the minimum evaluation value. The evaluation value DFD is rewritten, and the process proceeds to step S830. The compensation candidate vector in the memory 821 is rewritten to the motion vector of the minimum evaluation value, and the process proceeds to step S831.

  On the other hand, if the allocation flag of the peripheral pixel in the allocation flag memory 56 is 0 (False) in step S825, the compensation processing unit 811 determines that there is no motion vector allocated to the selected peripheral pixel, and step S826. The process of thru | or S830 is skipped and it progresses to step S831. In step S828, when the compensation processing unit 811 determines that the evaluation value DFD is greater than or equal to the minimum evaluation value of the pixel of interest stored in the memory 821, the processing of steps S829 and S830 is skipped, and step S831 is performed. Proceed to

  In step S831, the compensation processing unit 811 determines whether or not the processing has been completed for all eight pixels around the target pixel, and determines that the processing has not been completed for all eight pixels around the target pixel. If so, the process returns to step S824, the next peripheral pixel is selected, and the subsequent processing is repeated. If the compensation processing unit 811 determines in step S831 that the processing has been completed for all eight pixels around the pixel of interest, the compensation processing unit 811 ends the vector compensation processing and returns to step S804 in FIG.

  As described above, the most reliable level based on the evaluation value DFD is obtained from the motion vectors around the pixel using the fact that there is a motion correlation even for the pixel that cannot be assigned in the vector assignment processing. There is a certain motion vector. As a result, the accuracy of vector assignment is improved as compared with the case where a vector is not assigned and a 0 vector is assigned, and discontinuity of an image generated in subsequent image interpolation processing can be suppressed.

  In addition, the allocation flag of the pixel to which the motion vector is allocated by the allocation compensation process described above is rewritten to 1 (True), and the motion vector allocated by the allocation compensation process is also used as a compensation candidate vector for the next pixel. Therefore, almost the same motion vector is selected for pixels that perform substantially the same motion in the object, and a stable motion vector with few errors can be obtained. As a result, it is possible to improve the quality by suppressing block noise, powder noise and the like of the image generated in the subsequent stage.

  In addition, for a pixel of interest for which a motion vector is to be calculated, an evaluation value DFD of 0 vector is calculated in advance and stored in a memory as a minimum evaluation value, so that all compensation candidate vector evaluation values DFD are calculated at once. Thus, time and resources can be used more efficiently than when the smallest evaluation value DFD is selected.

  In the above description, the vector compensation processing is performed on the pixels that are not assigned by the vector assigning unit 54, but there are some kinds of pixels such as pixels that are not detected by the vector detecting unit 52 (the zero vector is detected). You may make it perform a vector compensation process with respect to the pixel in which the motion vector was not calculated | required in the process. Furthermore, vector compensation processing may be performed on pixels for which the detected motion vector or the assigned motion vector is uncertain (low reliability).

  In this embodiment, the evaluation value DFD that is the sum of absolute differences is used as the evaluation value when selecting a motion vector. However, the evaluation value DFD is not limited to the evaluation value DFD, and the reliability of the motion vector is evaluated. If it is a thing, you may make it use another thing.

  In the present embodiment, the block for performing each process has been described as being configured by, for example, 8 pixels × 8 pixels, 9 pixels × 9 pixels, and the like. However, these are examples, and each process is performed. The number of pixels constituting the block to be performed is not limited to the number of pixels.

  Furthermore, in the present embodiment, the signal conversion from the 24P signal to the 60P signal has been described as an example. However, the present invention can be applied to, for example, interlaced signals and other frame rates as frame frequency conversion of moving images. It can also be applied to conversion.

  The series of processes described above can be executed by hardware, but can also be executed by software. When a series of processing is executed by software, a program constituting the software may execute various functions by installing a computer incorporated in dedicated hardware or various programs. For example, it is installed from a program storage medium in a general-purpose personal computer or the like.

  As shown in FIG. 1, a program storage medium that stores a program that is installed in a computer and can be executed by the computer includes a magnetic disk 31 (including a flexible disk), an optical disk 32 (CD-ROM (Compact Disc -Removable recording media (package media) comprising a read only memory (including DVD (Digital Versatile Disc)), magneto-optical disc 33 (including MD (Mini-Disc) (trademark)), or semiconductor memory 34, or The ROM 12 stores the program temporarily or permanently.

  In the present specification, the steps shown in the flowcharts include not only processes performed in time series according to the order described, but also processes executed in parallel or individually even if not necessarily performed in time series. Is included.

It is a block diagram which shows the structural example of the signal processing apparatus of this invention. It is a block diagram which shows the structure of a signal processing apparatus. It is a figure explaining the principle of the process of this invention. It is a figure explaining the process of this invention concretely. It is a figure explaining the evaluation value of the motion vector used in a signal processor. It is a flowchart explaining the frame frequency conversion process of a signal processing apparatus. It is a block diagram which shows the structure of an image interpolation part. It is a flowchart explaining an image interpolation process. It is a block diagram which shows the structure of a vector detection part. It is a figure explaining the gradient method used in a vector detection part. It is a figure explaining the iterative gradient method using an initial vector. It is a flowchart explaining a motion vector detection process. It is a block diagram which shows the structure of a shift initial vector allocation part. It is a figure explaining the concept of a shift initial vector. It is a figure explaining a shift initial vector concretely. It is a figure explaining a shift initial vector concretely. It is a figure explaining the determination method of a shift initial vector. It is a figure explaining the example of allocation of a shift initial vector. It is a figure explaining the example of a shift initial vector in case a motion vector competes. It is a flowchart explaining a shift initial vector allocation process. It is a block diagram which shows the structure of an initial vector selection part. It is a figure which shows the peripheral area | region of an initial vector. It is a figure which shows the candidate block of an initial vector. It is a flowchart explaining an initial vector selection process. It is a block diagram which shows the other structure of an initial vector selection part. It is a figure which shows the candidate block of an initial vector. It is a figure which shows the priority of the candidate block of an initial vector. It is a flowchart explaining the other example of an initial vector selection process. It is a flowchart explaining an initial candidate vector calculation selection process. It is a block diagram which shows the structure of an iterative gradient method calculating part. It is a block diagram which shows the structure of an effective pixel determination part. It is a block diagram which shows the structure of a gradient method calculating part. It is a figure explaining the detection target block and calculation block of a motion vector. It is a figure explaining the motion of the object in a detection target block. It is a figure explaining the state of the brightness | luminance in the detection target block which has the object of the same motion. It is a figure explaining the state of the brightness | luminance in the detection target block which has an object of a different motion. It is a figure explaining the example of the calculation block of a pixel unit process. It is a figure explaining the pixel unit process in the detection object block of FIG. It is a flowchart explaining an iterative gradient method calculation process. It is a flowchart explaining an effective pixel determination process. It is a flowchart explaining a gradient method calculation process. It is a flowchart explaining the iterative gradient method calculation process of a pixel unit. It is a block diagram which shows the structure of a vector allocation part. It is a figure explaining the concept of the 4-point interpolation process of this invention. It is a figure explaining the outline | summary of a vector allocation process. It is a figure explaining the example of the neighborhood pixel of the intersection of a motion vector and an interpolation frame. It is a figure explaining the evaluation method of the motion vector of an allocation candidate. It is a figure explaining the example of 4-point interpolation in vector allocation. It is a flowchart explaining a vector allocation process. It is a flowchart explaining a pixel position calculation process. It is a flowchart explaining the allocation vector evaluation process. It is a block diagram which shows the structure of an allocation compensation part. It is a block diagram which shows the structure of a vector compensation part. It is a figure explaining the principle of an allocation compensation process. It is a figure explaining the principle of motion correlation. It is a figure explaining the structural example of the surrounding pixel of a focused pixel. It is a figure explaining the example of the compensation candidate vector of the motion vector of the focused pixel. It is a figure explaining the example of the compensation candidate vector of the motion vector of the focused pixel. It is a figure explaining the example of the compensation candidate vector of the motion vector of the focused pixel. It is a figure explaining the example which evaluates a compensation candidate vector. It is a figure explaining the example which evaluates a compensation candidate vector. It is a figure explaining the example which evaluates a compensation candidate vector. It is a figure explaining the example which selects a compensation candidate vector as a motion vector of a focused pixel. It is a flowchart explaining an allocation compensation process. It is a flowchart explaining a vector compensation process.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Signal processing apparatus, 11 CPU, 12 ROM, 13 RAM, 31 Magnetic disk, 32 Optical disk, 33 Magneto-optical disk, 34 Semiconductor memory, 51 Frame memory, 52 Vector detection part, 53 Detection vector memory, 54 Vector allocation part, 55 Allocation vector memory, 56 Allocation flag memory, 57 Allocation compensation unit, 58 Image interpolation unit, 101 Initial vector selection unit, 103 Iterative gradient method calculation unit, 104 Vector evaluation unit, 105 Shift initial vector allocation unit, 106 Evaluation value memory, 107 Shift initial vector memory, 201 allocation target position calculation unit, 202 evaluation value comparison unit, 203 shift initial vector replacement unit, 204 evaluation value replacement unit, 251 candidate vector position calculation unit, 252 detection vector acquisition unit, 253 shift initial vector acquisition unit , 254 Offset position calculation unit, 255 evaluation value calculation unit, 256 evaluation value comparison unit, 257 optimum candidate storage register, 301 similarity determination unit, 302 selector, 401 mode selection unit, 402 selector, 403 effective pixel determination unit, 404 gradient Legal operation unit, 405 delay unit, 411 time pixel difference calculation unit, 412 pixel difference value determination unit, 413 effective pixel number counter, 414 gradient method continuation determination unit, 421 time pixel difference calculation unit, 422 pixel difference value determination unit, 423 Horizontal / vertical pixel difference calculation unit, 424 gradient integration unit, 425 vector calculation unit, 701 pixel information calculation unit, 702 evaluation value calculation unit, 703 focused pixel difference calculation unit, 704 vector evaluation unit, 705 vector selection unit, 711 pixel difference determination Part, 712 evaluation value judgment part, 801 assigned vector judgment part, 8 2 vector allocating unit, 811 compensation processing unit, 812 evaluation value calculation unit, 821 memory

Claims (11)

In an image processing apparatus that detects a motion vector and generates a pixel value based on the detected motion vector,
Using an initial vector as an initial value, a motion vector detecting means for detecting a motion vector of a block of interest on the frame by a gradient method;
The current vector detected by the motion vector detecting means in the frame, the past vector detected by the motion vector detecting means in the past frame, and the predetermined vector around the target block including the target block; Among the shifted initial vectors having the same magnitude and the same direction as the motion vector detected by the motion vector detecting means in the past frame, the end point being the block on the past frame at the same position as the block , Based on a spatial correlation with the block of interest, at least one motion vector predetermined according to the position of the predetermined block with respect to the block of interest is set as a candidate vector for the initial vector of the block of interest. Candidate vector setting means;
When a plurality of candidate vectors are set in the predetermined block by the candidate vector setting means , based on whether the absolute value of the difference between the horizontal component and the vertical component of the plurality of candidate vectors is smaller than a predetermined threshold, Similarity determination means for determining whether the plurality of candidate vectors have similarity;
When the similarity determination unit determines that the plurality of candidate vectors are similar, if the predetermined block is in an area in contact with the block of interest, the shifted initial vector of the predetermined block is A candidate vector selecting means for selecting as a candidate vector and, if the predetermined block is outside the area in contact with the target block, selecting the current vector or the past vector of the predetermined block as the candidate vector;
An initial vector selection that selects an initial vector of the block of interest based on an evaluation value that is a correlation value of a region centered on a position on the two frames associated with the candidate vector from the candidate vectors image processing and means. When the candidate vector setting means selects the block of interest in raster scan order,
Setting the shifted initial vector of the block of interest as the candidate vector;
Setting the current vector of the block to the left of the block of interest as the candidate vector;
The predetermined block located before the target block in the raster scan order, the upper right block of the target block, the left three blocks of the target block, the block above the three blocks, and the right side of the three blocks of the target block And setting the shifted initial vector and the current vector of blocks on three blocks as the candidate vectors,
The predetermined block located after the target block in the raster scan order, the lower right block of the target block, the left three blocks of the target block, the block three blocks below, and the right of the three blocks of the target block The image processing apparatus according to claim 1, wherein the initial shift vector and the past vector of a block three blocks below are set as the candidate vectors. When the candidate vector setting means selects the block of interest in raster scan order,
Setting the shifted initial vector of the block of interest as the candidate vector;
A predetermined block located before the target block in the raster scan order, a block adjacent to the left of the target block, an upper right block of the target block, and two blocks to the left of the target block, on the two blocks Set the current vector of the block and the block on the two blocks on the right side of the block and the block of interest as the candidate vector,
The predetermined block located after the target block in the raster scan order, the lower right block of the target block, two blocks to the left of the target block, two blocks below the target block, and two blocks of the target block The image processing apparatus according to claim 1, wherein the past vector of a block on the right side and two blocks below is set as the candidate vector. Evaluation value calculating means for calculating an evaluation value that is a correlation value of a region centered on a position on two frames associated with the candidate vector ;
The image processing apparatus according to claim 1, wherein the initial vector selection unit selects a candidate vector having the smallest evaluation value calculated by the evaluation value calculation unit as an initial vector of the block of interest. The candidate vector setting means also sets priorities for selecting an initial vector of the target vector when there are a plurality of candidate vectors having the smallest evaluation value calculated by the evaluation value calculating means, and the target block When selecting in raster scan order,
Set the candidate vector of the block of interest to priority number 1,
The candidate vector of a predetermined block in a region in contact with the target block is set to have a higher priority than the candidate vector of a predetermined block outside the region in contact with the target block
The image processing apparatus according to claim 4 . The candidate vector setting means includes:
Set the candidate vector of the block on the left most immediately before the target block in the raster scan order to priority order 2;
The candidate vector of the block that is farthest from the center of the straight line connecting the centers of the blocks of the candidate vectors of the first priority and the second priority in the region that is in contact with the block of interest and that is positioned in the raster scan order in the order of priority 3 Set,
The candidate vector of the block farthest from the center of the space defined by the straight line connecting the centers of the candidate vectors of the priority Nos. 1 to 3 is set to the priority No. 4 within the area in contact with the block of interest. And
Outside the area in contact with the block of interest, the candidate vector of the predetermined block located before the block of interest in the raster scan order is set to be more than the candidate vector of the predetermined block located behind the block of interest in the raster scan order. Set higher priority
The image processing apparatus according to claim 5 . The candidate vector setting means includes:
In the region that is in contact with the target block, the priority order of the candidate vector of the predetermined block that is located before the target block in the raster scan order and is closest to the target block in the raster scan order is set highest. Is set later in the raster scan order, and the priority of the candidate vector of the predetermined block farthest in the raster scan order in the target block is set to be the lowest,
Outside the area in contact with the block of interest, the candidate vector of the predetermined block located before the block of interest in the raster scan order is set to be more than the candidate vector of the predetermined block located behind the block of interest in the raster scan order. Set higher priority
The image processing apparatus according to claim 5 . Allocating means for allocating motion vectors detected by the motion vector detecting means to pixels on an interpolation frame interpolated between the frames;
On the basis of the motion vector allocated by the allocation unit, the image processing apparatus according to claim 1, further comprising a pixel interpolation generating means for generating and interpolating the pixel values on the interpolation frame. In an image processing method of an image processing apparatus for detecting a motion vector and generating a pixel value based on the detected motion vector,
A motion vector detection step of detecting a motion vector of a block of interest on a frame by a gradient method using an initial vector as an initial value;
A current vector detected by the motion vector detecting step in the frame, a past vector detected by the motion vector detecting step in a past frame, of a predetermined block around the target block including the target block, And a shift having the same size and the same direction as the motion vector detected by the processing of the motion vector detection step in the past frame, with the block on the past frame at the same position as the predetermined block as an end point Of the initial vectors , based on a spatial correlation with the block of interest, at least one motion vector predetermined according to the position of the predetermined block with respect to the block of interest is determined as an initial vector of the block of interest. Candidate vector setting step And-flops,
When a plurality of candidate vectors are set in the predetermined block by the processing of the vector setting step , based on whether the absolute value of the difference between the horizontal component and the vertical component of the plurality of candidate vectors is smaller than a predetermined threshold a similarity determining step of determining whether there is a similarity in the plurality of candidate vectors,
If it is determined by the similarity determination step that the plurality of candidate vectors are similar, if the predetermined block is in an area in contact with the block of interest, the shifted initial vector of the predetermined block is Selecting the candidate vector, and if the predetermined block is outside the area in contact with the block of interest, a candidate vector selecting step of selecting the current vector or the past vector of the predetermined block as the candidate vector;
An initial vector selection that selects an initial vector of the block of interest based on an evaluation value that is a correlation value of a region centered on a position on the two frames associated with the candidate vector from the candidate vectors An image processing method including steps. A computer of an image processing apparatus that detects a motion vector and generates a pixel value based on the detected motion vector.
A motion vector detection step of detecting a motion vector of a block of interest on a frame by a gradient method using an initial vector as an initial value;
A current vector detected by the motion vector detecting step in the frame, a past vector detected by the motion vector detecting step in a past frame, of a predetermined block around the target block including the target block, And a shift having the same size and the same direction as the motion vector detected by the processing of the motion vector detection step in the past frame, with the block on the past frame at the same position as the predetermined block as an end point Of the initial vectors , based on a spatial correlation with the block of interest, at least one motion vector predetermined according to the position of the predetermined block with respect to the block of interest is determined as an initial vector of the block of interest. Candidate vector setting step And-flops,
When a plurality of candidate vectors are set in the predetermined block by the processing of the vector setting step , based on whether the absolute value of the difference between the horizontal component and the vertical component of the plurality of candidate vectors is smaller than a predetermined threshold a similarity determining step of determining whether there is a similarity in the plurality of candidate vectors,
If it is determined by the similarity determination step that the plurality of candidate vectors are similar, if the predetermined block is in an area in contact with the block of interest, the shifted initial vector of the predetermined block is Selecting the candidate vector, and if the predetermined block is outside the area in contact with the block of interest, a candidate vector selecting step of selecting the current vector or the past vector of the predetermined block as the candidate vector;
An initial vector selection that selects an initial vector of the block of interest based on an evaluation value that is a correlation value of a region centered on a position on the two frames associated with the candidate vector from the candidate vectors Tei Ru recording medium program is recorded for causing a process including the step. A computer of an image processing apparatus that detects a motion vector and generates a pixel value based on the detected motion vector.
A motion vector detection step of detecting a motion vector of a block of interest on a frame by a gradient method using an initial vector as an initial value;
A current vector detected by the motion vector detecting step in the frame, a past vector detected by the motion vector detecting step in a past frame, of a predetermined block around the target block including the target block, And a shift having the same size and the same direction as the motion vector detected by the processing of the motion vector detection step in the past frame, with the block on the past frame at the same position as the predetermined block as an end point Of the initial vectors , based on a spatial correlation with the block of interest, at least one motion vector predetermined according to the position of the predetermined block with respect to the block of interest is determined as an initial vector of the block of interest. Candidate vector setting step And-flops,
When a plurality of candidate vectors are set in the predetermined block by the processing of the vector setting step , based on whether the absolute value of the difference between the horizontal component and the vertical component of the plurality of candidate vectors is smaller than a predetermined threshold a similarity determining step of determining whether there is a similarity in the plurality of candidate vectors,
If it is determined by the similarity determination step that the plurality of candidate vectors are similar, if the predetermined block is in an area in contact with the block of interest, the shifted initial vector of the predetermined block is Selecting the candidate vector, and if the predetermined block is outside the area in contact with the block of interest, a candidate vector selecting step of selecting the current vector or the past vector of the predetermined block as the candidate vector;
An initial vector selection that selects an initial vector of the block of interest based on an evaluation value that is a correlation value of a region centered on a position on the two frames associated with the candidate vector from the candidate vectors A program for performing processing including steps and.

Images