WO2010023982A1 - Motion vector detection device and image processing device - Google Patents
Motion vector detection device and image processing device Download PDFInfo
- Publication number
- WO2010023982A1 WO2010023982A1 PCT/JP2009/055423 JP2009055423W WO2010023982A1 WO 2010023982 A1 WO2010023982 A1 WO 2010023982A1 JP 2009055423 W JP2009055423 W JP 2009055423W WO 2010023982 A1 WO2010023982 A1 WO 2010023982A1
- Authority
- WO
- WIPO (PCT)
- Prior art keywords
- motion vector
- fitness
- candidate
- pixel
- detection device
- Prior art date
Links
Images
Classifications
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N5/00—Details of television systems
- H04N5/14—Picture signal circuitry for video frequency region
- H04N5/144—Movement detection
- H04N5/145—Movement estimation
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T7/00—Image analysis
- G06T7/20—Analysis of motion
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20021—Dividing image into blocks, subimages or windows
-
- G—PHYSICS
- G06—COMPUTING; CALCULATING OR COUNTING
- G06T—IMAGE DATA PROCESSING OR GENERATION, IN GENERAL
- G06T2207/00—Indexing scheme for image analysis or image enhancement
- G06T2207/20—Special algorithmic details
- G06T2207/20036—Morphological image processing
- G06T2207/20044—Skeletonization; Medial axis transform
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
- H04N7/0127—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level by changing the field or frame frequency of the incoming video signal, e.g. frame rate converter
-
- H—ELECTRICITY
- H04—ELECTRIC COMMUNICATION TECHNIQUE
- H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
- H04N7/00—Television systems
- H04N7/01—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level
- H04N7/0135—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes
- H04N7/014—Conversion of standards, e.g. involving analogue television standards or digital television standards processed at pixel level involving interpolation processes involving the use of motion vectors
Definitions
- the present invention relates to a motion vector detection device that detects a motion vector as a movement amount and distance information of an object by associating the same object in the frame between a plurality of frames, and an image processing device including the motion vector detection device.
- motion vectors are calculated by a pattern matching method, a gradient method, a method using a cross-correlation function, a method using a Fourier transform, and the like (for example, see Non-Patent Document 1).
- motion vectors obtained using multiple frames at different times can be used to improve coding efficiency in interframe coding, frame interpolation (frame number conversion, frame addition, deinterlace conversion), resolution conversion, etc. Can be used for motion detection.
- motion vectors obtained using a plurality of frames taken at spatially different viewpoint positions are used for distance detection such as creation of distance images and three-dimensional images, object distance measurement, and three-dimensional image coding. It can be used for
- the following equation (1) is obtained between two frame signals g 0 (x, y) and g 1 (x, y).
- the candidate for fitness e ( ⁇ , ⁇ ) which is an index for matching the movement amount ( ⁇ , ⁇ ) is obtained.
- T lim is a predetermined threshold when calculating the fitness candidate by inverting the polarity of the error between the two frame signals, and a luminance signal is used for the two frame signals.
- the calculation is performed using the absolute value of the difference between the two frame signals, but the square of the difference or a non-linear characteristic may be used.
- both the luminance signal and the color difference signal may be used as the two frame signals.
- the range B to be added is a block of about 8 ⁇ 8 pixels. Equation (1) is calculated for each movement amount ( ⁇ , ⁇ ) within the search range of the motion vector candidate, and the movement amount ( ⁇ , ⁇ ) that maximizes the fitness candidate e ( ⁇ , ⁇ ) among the calculation results. ) As motion vector candidates for the block.
- the movement amount ( ⁇ , ⁇ ) that maximizes h ( ⁇ , ⁇ ) in the following equation (2) is obtained as a motion vector candidate.
- a block to which an inappropriate motion vector candidate is assigned or a block to which a motion vector candidate is not assigned is generated (for example, see Patent Document 1).
- motion vector candidate interpolation processing is performed to determine a motion vector.
- Interpolation processing is also performed on the fitness candidates, and the fitness is determined.
- the motion vector detecting device and the image processing device of the prior art in order to determine the motion vector and the fitness, the same number of motion vector candidates and fitness candidates existing within the same predetermined range are used, and the motion vector and the fitness are used. Is determined. For example, in Patent Document 2, the motion vector and the fitness are obtained by the following equations (3) and (4).
- a i (x 1 , y 1 , x 2 , y 2 ) has the same motion vector at two points (x 1 , y 1 ) and (x 2 , y 2 ).
- the degree of correlation representing the degree of movement estimated in the direction, r i (x 2 , y 2 ) indicates how reliable the motion vector candidate at that point is, or the degree to which a predetermined condition is met. This is a reliability candidate to be represented (corresponding to a fitness candidate), and v i (x 2 , y 2 ) is a motion vector candidate input to the interpolation process.
- v i + 1 (x 1 , y 1 ) in Expression (3) is a motion vector determined by interpolation processing
- r i + 1 (x 1 , y 1 ) in Expression (4) is determined by interpolation processing. It is the determined reliability (equivalent to goodness of fit).
- Equation (3) and Equation (4) for both motion vectors and fitness, the average number of motion vector candidates and fitness candidates around (x 2 , y 2 ) ⁇ (x 1 , y 1 ) is used. Thus, the motion vector and the fitness are determined.
- JP 2002-27414 A page 4, lines 6-20
- Japanese Patent No. 3935500 page 13, number 1, number 2, page 20, FIG. 10
- Takahiko Fukiki “Multi-dimensional TV signal processing”, first edition, 1st edition, Nikkan Kogyo Shimbun, November 15, 1988, p. 193-212
- the search accuracy is coarsened, matching is performed with an accuracy of several pixels, and the search accuracy is gradually increased in the vicinity of the obtained motion vector.
- a so-called multi-stage search method that repeats matching while approaching a pixel unit is often used.
- a motion vector detection device and an image processing device that can solve the above-described problems and suppress the motion vector detection or the processing time while suppressing erroneous detection of the motion vector.
- the purpose is to provide.
- the motion vector detection apparatus detects a motion vector that associates one or more pixels included in the first and second input frames between the first and second input frames.
- An edge detection unit for detecting an edge component of each input frame, and a plurality of pixels included in a predetermined attention area of each input frame based on the edge component detected by the edge detection unit, Using the thinning means for thinning out one representative pixel and outputting this pixel as the representative one pixel, and the representative one pixel of the first and second input frames output from the thinning means,
- a fitness candidate calculation means for calculating a fitness candidate that is a fitness candidate that represents the adaptation of the movement amount between the first and second input frames; The motion vector between the first and second input frames based on the position of the representative pixel output from the thinning unit in each input frame and the fitness candidate calculated by the fitness candidate calculation unit
- a motion vector candidate calculation unit that calculates a motion vector candidate that is a candidate for the motion vector, the fitness candidate calculated by the fitness candidate calculation unit, and the motion vector candidate calculated
- the image processing apparatus includes, based on the motion vector detection device and the pixels included in the first and second input frames, based on the motion vector detected by the motion vector detection device and the fitness.
- Frame rate conversion means for converting a frame rate of an image by generating pixels included in an interpolation frame to be interpolated between the first and second input frames.
- the image processing apparatus of the present invention is based on the motion vector detection device and the pixels included in the first and second input frames based on the motion vector detected by the motion vector detection device and the fitness.
- a resolution conversion unit configured to generate a new image having a different resolution from the image formed by the first and second input frames by generating a new pixel.
- the motion vector detection device of the present invention after thinning out a plurality of pixels included in the attention area of the first and second input frames to one representative pixel by the thinning unit, the representative one pixel is used for matching.
- the fitness candidate calculation means calculates fitness candidates. As a result, it is possible to reduce the amount of calculation for calculating the suitability level candidates, so that the circuit scale or processing time for calculating the suitability level candidates can be reduced. Since motion vector candidates are calculated using the fitness candidates and motion vectors are detected, the circuit scale or processing time for motion vector detection can be kept small.
- the thinning-out means thins out a plurality of pixels in the region of interest to one representative pixel based on the edge component of each input frame detected by the edge detecting means, it is possible to suppress a decrease in motion vector detection accuracy due to the thinning-out. it can.
- the storage means calculates the storage address of the motion vector candidate and the fitness candidate using the position information of one representative pixel at the time of the thinning, and stores the motion vector candidate and the fitness candidate at the storage address. It is possible to suppress the occurrence of position errors due to. Further, since the motion vector determining means determines the motion vector and the fitness from the motion vector candidate and the fitness candidate, it is possible to suppress the occurrence of an error when determining the motion vector.
- the image processing apparatus is configured to include the above-described excellent motion vector detection device and frame rate conversion means.
- the motion vector detection device can suppress the erroneous detection of the motion vector and suppress the circuit scale or the processing time for detecting the motion vector, so that the pixels included in the interpolation frame can be generated with high accuracy. . Therefore, an image processing apparatus capable of converting the frame rate while suppressing deterioration in image quality is realized.
- the image processing apparatus is configured to include the above-described excellent motion vector detection device and resolution conversion means.
- the motion vector detection device can reduce the circuit size or processing time for motion vector detection while suppressing false detection of motion vectors, so it can suppress the degradation of image quality and generate new images with different resolutions. An image processing apparatus that can do this is realized.
- FIG. 3 is a block diagram illustrating a configuration of a motion vector candidate calculation unit 14.
- FIG. It is a figure for demonstrating the phase Fph of an interpolation frame.
- 3 is a block diagram showing a configuration of a storage unit 15.
- FIG. It is a figure for demonstrating the pair at the time of fixing the point on one frame and shaking only the point on the other frame. It is a figure for demonstrating the pair at the time of shaking the point on both frames to a point symmetrical type.
- 3 is a block diagram illustrating a configuration of a motion vector determination unit 16.
- FIG. 6 is a diagram for explaining the operation of a resolution conversion unit 82.
- FIG. 6 is a diagram for explaining the operation of a resolution conversion unit 82.
- FIG. 1 is a block diagram showing a configuration of a motion vector detection apparatus 1 according to the first embodiment of the present invention.
- FIG. 2 is a diagram illustrating an example of one representative pixel.
- the motion vector detection device 1 detects a motion vector that associates one or more pixels included in each of a plurality of frames between a plurality of input frames.
- the motion vector detection device 1 is configured to input the first two input frames of interest among the input frames (hereinafter sometimes referred to as “input frames”), that is, between the first and second input frames.
- input frames hereinafter sometimes referred to as “input frames”
- the motion vector detection device 1 detects, for example, an image motion vector representing a motion of a subject in a screen in a moving image, and calculates a motion vector between a previous frame and a subsequent frame that are in a time series relationship. To detect.
- the motion vector detected by the motion vector detection device 1 is used for camera shake correction of the video camera.
- the image signal of the frame input to the motion vector detection device 1 is read out by performing horizontal pixel scanning and vertical line scanning.
- the motion vector detection device 1 includes an input terminal 10, an edge detection unit 11, a thinning unit 12, a fitness candidate calculation unit 13, a motion vector candidate calculation unit 14, a storage unit 15, a motion vector determination unit 16, A first output terminal 17 and a second output terminal 18 are provided.
- the input terminal 10 receives image data such as moving image data acquired by an image sensor in a video camera, for example.
- the image data input to the input terminal 10 includes a plurality of frames, and is input to the edge detection unit 11 and the thinning unit 12 in units of frames.
- the edge detection unit 11 detects horizontal and vertical edge components from the input image signals of a plurality of frames.
- the edge detection unit 11 gives the detected edge component to the thinning unit 12 and the fitness candidate calculation unit 13.
- the edge detection unit 11 corresponds to an edge detection unit.
- the thinning unit 12 thins out a plurality of pixels included in a predetermined attention area of the input frame, and outputs one representative pixel as a representative pixel. Specifically, the thinning unit 12 calculates an edge strength value using the edge component given from the edge detection unit 11, and the edge strength value is calculated among a plurality of pixels of the input frame as shown in FIG. The largest pixel is output as one representative pixel. The thinning unit 12 outputs the position of one representative pixel. The thinning unit 12 supplies the representative pixel and the image signal representing the position thereof to the fitness candidate calculation unit 13, the motion vector candidate calculation unit 14, and the storage unit 15. The thinning unit 12 corresponds to a thinning unit.
- the goodness-of-fit candidate calculation unit 13 obtains goodness-of-fit candidates based on two representative one-pixel image signals given from the thinning-out unit 12 for two input frames. More specifically, the fitness candidate calculation unit 13 uses the representative one pixel output from the thinning unit 12, and serves as a fitness candidate that represents the fitness of the movement amount between two input frames. Candidates are calculated. The fitness candidate calculation unit 13 provides the calculated fitness candidates to the motion vector candidate calculation unit 14 and the storage unit 15. The fitness candidate calculation unit 13 corresponds to fitness candidate calculation means.
- the motion vector candidate calculation unit 14 is a motion vector that is a candidate for a motion vector between two input frames, based on the fitness candidate given from the fitness candidate calculation unit 13 and the position of one representative pixel given from the thinning unit 12. Candidates are calculated.
- the motion vector candidate calculation unit 14 gives the calculated motion vector candidates to the storage unit 15.
- the motion vector candidate calculation unit 14 corresponds to motion vector candidate calculation means.
- the storage unit 15 calculates a storage address from the motion vector candidate given from the motion vector candidate calculation unit 14, the position of one representative pixel given from the thinning unit 12, and the phase of the interpolation frame.
- the storage unit 15 stores the motion vector candidate given from the motion vector candidate calculation unit 14 and the fitness candidate given from the fitness candidate calculation unit 13 at the calculated storage address.
- the storage unit 15 outputs a motion vector candidate and a fitness candidate in a predetermined range every time the storage process in the predetermined range is completed. Furthermore, the storage unit 15 stores a motion vector and a fitness level given from a motion vector determination unit 16 described later.
- the storage unit 15 corresponds to a storage unit.
- the motion vector determination unit 16 determines the motion vector and the fitness based on the motion vector candidate and the fitness candidate given from the storage unit 15.
- the motion vector determined by the motion vector determination unit 16 is supplied to the storage unit 15 and is output from the first output terminal 17.
- the fitness determined by the motion vector determination unit 16 is given to the storage unit 15 and is output from the second output terminal 18.
- the motion vector determination unit 16 corresponds to a motion vector determination unit.
- the edge detection unit 11 detects edge components in the horizontal direction and the vertical direction from the input image signals of a plurality of frames, and provides them to the thinning unit 12.
- the edge component may be detected using a primary differential filter such as a Prewitt filter or a Sobel filter, or a primary differential filter is used in combination with a secondary differential filter or the like. You may go.
- the edge component is obtained by calculating according to the following equation (5).
- g (x, y) is an image signal of the coordinates (x, y) of the input frame, and for example, luminance is used.
- E ′ x is a horizontal edge component detected by the primary differential filter (hereinafter referred to as “horizontal primary differential edge component”)
- E ′ y is a vertical edge detected by the primary differential filter.
- Component hereinafter referred to as “vertical first-order differential edge component”.
- E ′′ x is a horizontal edge component detected by the secondary differential filter (hereinafter referred to as “horizontal second differential edge component”)
- E ′′ y is a vertical direction detected by the secondary differential filter.
- Edge components hereinafter referred to as “second-order differential edge components in the vertical direction”).
- the thinning unit 12 calculates an edge strength value using the edge component given from the edge detection unit 11, and, as shown in FIG. 2, the pixel having the largest edge strength value among a plurality of pixels of the input frame is calculated. Output as one representative pixel.
- the edge intensity value among a plurality of pixels in a region of interest consisting of M M is an integer of 1 or more) pixels in the horizontal direction and N (N is an integer of 1 or more) pixels in the vertical direction of the input frame. Only the largest pixel is left and output as one representative pixel. Other pixels of the region of interest consisting of M pixels in the horizontal direction and N pixels in the vertical direction are discarded.
- the edge strength value is obtained by, for example, C1 (
- C1 and C2 are predetermined constants.
- the thinning unit 12 outputs a position RP (O x , O y ) of one representative pixel moved from the offset reference point PO by O x in the horizontal direction and O y in the vertical direction.
- a pixel at a predetermined position in the attention area for example, the upper left pixel of the attention area may be used as one representative pixel. In this case, in terms of detection accuracy, the pixel having the largest edge intensity value may be inferior to the case where the representative one pixel is used, but the effect of reducing the calculation amount by thinning out the representative one pixel is the same. Is recognized.
- the fitness candidate calculation unit 13 uses two representative one-pixel image signals g 0 (x, y) and g 1 (x, x) given from the thinning unit 12 for two input frames.
- a fitness candidate e ( ⁇ M , ⁇ N ) which is an index for matching the movement amount ( ⁇ M , ⁇ N ) between y), is obtained and given to the motion vector candidate calculation unit 14 and the storage unit 15.
- ( ⁇ M , ⁇ N ) is a reference mark of M and N at the lower right of ⁇ and ⁇ in order to indicate that it is a coordinate after being thinned out for one representative pixel given from the thinning unit 12. Is added. The same applies to the method using the cross-correlation function.
- pixel matching is performed. For example, p pixels in the horizontal direction, q pixels in the vertical direction (p and q are integers, one of p and q is 2 or more, and the other is
- p and q are integers, one of p and q is 2 or more, and the other is
- block matching is performed. Normally, block matching can reduce false detections more than pixel matching, but if p and q, which are the number of pixels, are increased too much, there is a problem that false detections are likely to occur at the boundary of an object. As the number of pixels increases, the amount of calculation also increases.
- the fitness candidate calculation unit 13 can also calculate fitness candidates using the edge components given from the edge detection unit 11. Thereby, since the fitness candidates are calculated from both the image signal and the edge component, the detection accuracy of the motion vector candidates can be improved. For example, two image signals and edge components (g 0 (x, y), E ′ x0 (x, y), E ′ y0 (x, y), E ′′ x0 (x, y), E ′′ y0 ( x, y)), (G 1 (x, y), E ′ x1 (x, y), E ′ y1 (x, y), E ′′ x1 (x, y), E ′′ y1 (x, y)) Using the following equation (6), a fitness candidate e ( ⁇ M , ⁇ N ) that is an index for matching the movement amount ( ⁇ M , ⁇ N ) is obtained.
- C my , C m1 , and C m2 are predetermined constants. Further, (x, y) is the coordinates after thinning out one representative pixel given from the thinning unit 12.
- the fitness candidates are calculated using the edge information as well, in matching between a small number of pixels, such as pixel matching between one pixel or a size of a p ⁇ q block is small.
- the detection accuracy of the motion vector can be improved.
- F x and F y shown in the following formula (7) indicating the size of the edge component are added to the fitness candidate e ( ⁇ M , ⁇ N ), and a calculation formula that is easily adapted when the edge component is large: You can also
- C ex , C ey , T ex , and T ey are predetermined constants.
- F x ( ⁇ M , ⁇ N ), F y ( ⁇ M , ⁇ N ) are both limited to 0 or less, and then added to the fitness candidate e ( ⁇ M , ⁇ N )
- the horizontal component e x ( ⁇ M , ⁇ N ) and the vertical component e y ( ⁇ M , ⁇ N ) of the fitness candidates are calculated according to the following equation (8).
- the motion vector candidate calculation unit 14 calculates and outputs a motion vector candidate from the fitness candidate given from the fitness candidate calculation unit 13 and the position of one representative pixel given from the thinning unit 12.
- FIG. 3 is a block diagram illustrating a configuration of the motion vector candidate calculation unit 14.
- the motion vector candidate calculation unit 14 includes a first input terminal 20, a second input terminal 21, a selection unit 22, a motion vector calculation unit 23, and an output terminal 24.
- the candidate for suitability given from the suitability candidate calculation unit 13 is given to the selection unit 22 via the first input terminal 20.
- the selection unit 22 selects a fitness level candidate having the highest fitness level, outputs a movement amount ( ⁇ M , ⁇ N ) corresponding to the selected fitness level candidate, and gives it to the motion vector calculation unit 23.
- the selection of the one with the highest fitness is selected with the highest fitness candidate e ( ⁇ M , ⁇ N ).
- the one with the largest ( ⁇ M , ⁇ N ) is selected.
- the motion vector calculation unit 23 receives the movement amount ( ⁇ M , ⁇ N ) given from the selection unit 22 and the position RP (O x , O y) of the representative one pixel given from the thinning unit 12 via the second input terminal 21. ) To calculate and output motion vector candidates.
- the position of one representative pixel for a first input frame (hereinafter sometimes simply referred to as a “first frame”) among a plurality of input frames to the motion vector detection device 1, in this embodiment, two input frames.
- Is (O 1x , O 1y ) and the position of one representative pixel for the second input frame (hereinafter sometimes simply referred to as “second frame”) is (O 2x , O 2y ), the first frame
- the motion vector candidates ( ⁇ , ⁇ ) from 1 to the second frame are calculated according to the following equation (9), for example.
- the storage unit 15 calculates a storage address from the motion vector candidate given from the motion vector candidate calculation unit 14, the position of one representative pixel given from the thinning unit 12, and the phase of the interpolation frame.
- the storage unit 15 stores the motion vector candidate and the fitness candidate given from the fitness candidate calculation unit 13 at the calculated storage address.
- the storage unit 15 outputs the motion vector candidate and the fitness candidate for the predetermined area every time the motion vector candidate and the fitness candidate for the predetermined area in the frame are stored. Furthermore, the storage unit 15 also stores the motion vector and the degree of matching given from the motion vector determination unit 16.
- FIG. 4 is a diagram for explaining the phase Fph of the interpolation frame.
- the coordinates CI in the interpolation frame IF are calculated, the corresponding storage address is calculated, and the motion vector is calculated to the calculated storage address.
- the candidate and the fitness candidate are stored.
- FIG. 5 is a block diagram showing the configuration of the storage unit 15.
- FIG. 6 is a diagram for explaining a pair when a point on one frame is fixed and only a point on the other frame is shaken.
- FIG. 6A is a diagram showing a case where only points on the second frame are shaken while fixing points on the first frame
- FIG. 6B is a diagram showing points fixed on the second frame. It is a figure which shows the case where only the point on the 1st frame is shaken.
- FIG. 7 is a diagram for explaining a pair when the points on both frames are shaken in a point-symmetric manner.
- the storage unit 15 includes a first input terminal 30, a second input terminal 31, a third input terminal 32, a fourth input terminal 33, a fifth input terminal 34, a sixth input terminal 35, a storage address calculation unit 36, and a candidate storage unit. 37, a first output terminal 38 and a second output terminal 39.
- the storage address calculation unit 36 includes a motion vector candidate given from the motion vector candidate calculation unit 14 via the first input terminal 30, a representative one pixel position given from the thinning unit 12 via the fourth input terminal 33, and From the phase Fph of the interpolation frame IF given through the third input terminal 32, the coordinates of the motion vector candidate at the phase Fph of the interpolation frame IF are calculated, and the corresponding storage address is calculated and output.
- the candidate storage unit 37 has a motion address candidate given from the motion vector candidate calculation unit 14 via the first input terminal 30 and a fitness level via the second input terminal 31 to the storage address given from the storage address calculation unit 36.
- the fitness candidate given from the candidate calculation unit 13 is stored.
- the candidate storage unit 37 outputs the motion vector candidate for the predetermined region of interest from the first output terminal 38 every time the motion vector candidate and the fitness candidate for the predetermined region of interest in the frame are stored.
- a suitability degree candidate for a predetermined attention area is output from the second output terminal 39.
- the candidate storage unit 37 also stores the motion vector given from the motion vector determination unit 16 via the fifth input terminal 34 and the fitness given from the motion vector determination unit 16 via the sixth input terminal 35. This will be described later.
- the storage address calculation unit 36 calculates coordinates in the interpolation frame from the input motion vector candidate, the position of one representative pixel, and the phase of the interpolation frame, and calculates and outputs a storage address corresponding to the coordinates. To do.
- the coordinates (R 1x , R 1y ) and (R 2x , R 2y ) in the interpolation frame are obtained, for example, according to the following equation (10).
- (x, y) is the coordinates of one representative pixel output from the thinning unit 12
- (O 1x , O 1y ) is the position of one representative pixel for the first frame F1
- ( O 2x , O 2y ) is the position of one representative pixel for the second frame F2
- Fph is the phase of the interpolation frame IF
- ( ⁇ , ⁇ ) is a motion vector candidate.
- FIG. 6 shows a case in which a point on one frame is fixed and only a point on the other frame is shaken to search for a motion vector. The same applies to the case where the motion vector is searched by shaking this point, considering the amount of movement doubled.
- the storage addresses A 1 and A 2 are calculated according to the following equation (11).
- HSZ is the number of horizontal pixels, for example 320.
- the storage addresses A1 and A2 can be calculated by thinning out the M pixels in the horizontal direction to the representative one pixel using the coordinates generated by thinning out the representative pixel. In this case, the circuit scale of the candidate storage unit 37 can be reduced.
- FIG. 8 is a block diagram illustrating a configuration of the motion vector determination unit 16.
- FIG. 9 is a diagram for explaining a central pixel CP1 that stores a motion vector.
- FIG. 10 is a diagram for explaining the central pixel CP2 that stores the fitness.
- the motion vector determination unit 16 determines and outputs the motion vector and the fitness from the motion vector candidate and the fitness candidate given from the storage unit 15.
- the motion vector determination unit 16 includes a first input terminal 40, a second input terminal 41, a motion vector candidate interpolation unit 42, a fitness candidate interpolation unit 43, a first output terminal 44, and a second output terminal 45. .
- the motion vector candidate interpolation unit 42 performs an interpolation process from a motion vector candidate given from the storage unit 15 via the first input terminal 40 and a fitness candidate given from the storage unit 15 via the second input terminal 41. A motion vector is determined. The motion vector determined by the motion vector candidate interpolation unit 42 is output from the first output terminal 44. The goodness-of-fit candidate interpolation unit 43 determines the goodness of fit by interpolating the goodness-of-fit candidates given from the storage unit 15 via the second input terminal 41. The fitness determined by the fitness candidate interpolation unit 43 is output from the second output terminal 45.
- the motion vector candidate interpolation unit 42 determines a motion vector by interpolation processing from the motion vector candidate and the fitness candidate given from the storage unit 15.
- the motion vector candidate interpolation unit 42 is a motion vector interpolation range which is an area of horizontal M1 (M1 is an integer of 1 or more) pixels and vertical direction N1 (N1 is an integer of 1 or more) pixels shown in FIG.
- the motion vector of the center pixel CP1 in the motion vector interpolation range IA1 is determined using the motion vector candidate and the fitness candidate of IA1.
- the motion vector of the center pixel CP1 is determined, for example, by performing an interpolation process according to the following equation (13).
- ⁇ i (x, y) is the horizontal component of the motion vector at the center pixel (x, y)
- ⁇ i (x, y) is the motion vector at the center pixel (x, y)
- ⁇ (u, v) is the horizontal component of the motion vector candidate at coordinates (u, v)
- ⁇ (u, v) is the vertical of the motion vector candidate at coordinates (u, v).
- E (u, v) is a candidate for the degree of fitness at coordinates (u, v).
- the motion vector interpolation range IA1 is an area of horizontal M1 pixels and vertical N1 pixels.
- S is the sum of fitness candidates e (u, v) existing in the region of the horizontal direction M1 pixel and the vertical direction N1 pixel which are the motion vector interpolation range IA1, as shown in the following equation (14). .
- a motion vector is interpolated for a storage address in which no motion vector candidate is stored in the storage unit 15, and an error due to an inappropriate motion vector candidate is suppressed by interpolation. It is possible to determine a motion vector with high accuracy. Further, since the division of Expression (13) is performed according to the number of fitness candidates e (u, v) existing in the region of the horizontal direction M1 pixel and the vertical direction N1 pixel which are the motion vector interpolation range IA1, When there are few motion vector candidates and goodness-of-fit candidates existing in the area of the direction M1 pixel and the vertical direction N1 pixel, it is possible to prevent the determined motion vector from becoming small. Thereby, a motion vector with higher accuracy can be determined.
- the motion vector can also be determined using the fitness candidates having the horizontal and vertical components shown in Expression (8).
- the horizontal component of the motion vector is calculated from the horizontal component of the motion vector candidate and the horizontal component of the fitness candidate
- the vertical component of the motion vector is calculated from the vertical component of the motion vector candidate and the vertical component of the fitness candidate.
- the motion vector is determined.
- the motion vector is determined according to the following equation (15) using the horizontal component e x (u, v) of the fitness candidate and the vertical component e y (u, v) of the fitness candidate.
- the horizontal component of the motion vector is weighted by the horizontal component of the edge component of the input frame
- the vertical component of the motion vector is weighted by the vertical component of the edge component of the input frame.
- the motion vector can be determined using the position (ux, vy) from the center.
- the motion vector is determined according to the following equation (17).
- the goodness-of-fit candidate interpolation unit 43 interpolates the goodness-of-fit candidates given from the storage unit 15 and determines the goodness of fit.
- the fitness candidate of the fitness interpolation range IA2 which is an area of the horizontal direction M2 (M2 is an integer of 1 or more) pixels and the vertical direction N2 (N2 is an integer of 1 or more) pixels shown in FIG. Determine the goodness of fit.
- the fitness of the center pixel CP2 is determined by performing an interpolation process according to the following equation (19), for example.
- e i (x, y) is the fitness at the center pixel (x, y)
- e (u, v) is a fitness candidate at the coordinates (u, v).
- S e in equation (19) and the horizontal direction M2 pixels, the number of pixels of the M2 ⁇ N2 in the vertical direction N2 pixels in the region.
- the interpolation process is a simple average, and the smaller the fitness level candidates that exist in the horizontal direction M2 pixel and vertical direction N2 pixel regions that are the fitness level interpolation range IA2, the smaller the fitness level to be determined. Therefore, when there are few motion vector candidates that can be used for determination of motion vectors in the surrounding area, the degree of matching of the pixel can be reduced, so that the degree of matching with higher accuracy can be determined.
- FIG. 11 is a diagram showing the relationship between the obtained motion vector and the interpolation process.
- FIG. 12 is a diagram illustrating the relationship between the obtained fitness and interpolation processing.
- the storage unit 15 of the motion vector detection device 1 shown in FIG. 1 is configured to also store the motion vector and the fitness given from the motion vector determination unit 16.
- the candidate storage unit 37 constituting the storage unit 15 shown in FIG. 5 receives the motion vector supplied from the motion vector determination unit 16 via the fifth input terminal 34 and the motion vector determination unit 16 via the sixth input terminal 35. Memorize the goodness of fit given.
- the motion vector is stored in the storage address corresponding to the center pixel CP1 shown in FIG. 9 by overwriting the stored motion vector candidate.
- the fitness is stored by overwriting the stored fitness candidates at the storage address corresponding to the center pixel CP2 shown in FIG.
- the motion vector obtained for the center pixel CP1 (a) of the first motion vector interpolation range IA1 (a) corresponds to the interpolation frame IF.
- the stored address is overwritten and stored.
- This motion vector is used when determining the motion vector of the center pixel CP1 (b) of the second motion vector interpolation range IA1 (b) in the next pixel processing, as shown in FIG.
- FIG. 11 (c) it is used when determining the motion vector of the center pixel CP1 (c) of the third motion vector interpolation range IA1 (c) in the processing of the next pixel.
- the motion vector of the center pixel CP1 (b) in the second motion vector interpolation range IA1 (b) is also used.
- the fitness obtained for the center pixel CP2 (a) of the first fitness interpolation range IA2 (a) is the interpolation frame IF.
- the degree of fitness of the center pixel CP2 (b) in the second fitness interpolation range IA2 (b) shown in FIG. 12 (b) is obtained, and FIG. 12 (c).
- the fitness of the center pixel CP2 (b) in the second fitness interpolation range IA2 (b) is used when obtaining the fitness of the center pixel CP2 (c) in the third fitness interpolation range IA2 (c). It is done.
- the motion vector and the fitness are written back, that is, returned to the storage unit 15 and overwritten, as shown in FIGS. 11 and 12, the stored motion vector and the fitness are processed by the subsequent pixels. Used for. Therefore, the accuracy of the motion vector and the fitness is improved.
- the thinning unit 12 has M (M is an integer of 1 or more) pixels in the horizontal direction and N (N is an integer of 1 or more) in the vertical direction at each coordinate of the input frame. Since the fitness candidate calculation unit 13 calculates fitness candidates after thinning out a plurality of pixels in a pixel area to a representative pixel, it is possible to reduce the amount of calculation for calculating fitness candidates. Become. As a result, the circuit scale or processing time for calculating the fitness candidates can be reduced. Since motion vector candidates are calculated using the fitness candidates and motion vectors are detected, the circuit scale or processing time for motion vector detection can be kept small.
- the thinning unit 12 thins a plurality of pixels in the attention area into one representative pixel based on the edge component of each input frame detected by the edge detection unit 11, it suppresses a decrease in motion vector detection accuracy due to the thinning. be able to.
- the storage unit 15 calculates the storage address of the motion vector candidate using the position information of the representative one pixel when the thinning is performed, and stores the motion vector candidate at the storage address. Can be suppressed.
- the motion vector determination part 16 determines a motion vector and a fitness using a motion vector candidate and a fitness candidate, it can suppress the generation
- a motion vector detection device according to a second embodiment of the present invention will be described. Since the configuration of the motion vector detection device of the present embodiment is similar to that of the motion vector detection device 1 of the first embodiment described above, the same reference numerals are assigned to corresponding portions, and the first The description common to the embodiment is omitted.
- the motion vector detection device of the present embodiment has a configuration in which the storage unit 15 is replaced with a storage unit 15A shown in FIG. 13 in the motion vector detection device 1 of the first embodiment shown in FIG.
- FIG. 13 is a block diagram showing a configuration of the storage unit 15A of the motion vector detection device according to the second embodiment of the present invention.
- the storage unit 15A includes a first input terminal 30, a second input terminal 31, a third input terminal 32, a fourth input terminal 33, a fifth input terminal 34, a storage address calculation unit 36, a candidate storage unit 50, and a center selection unit 51.
- the candidate storage unit 50 stores motion vector candidates given from the motion vector candidate calculation unit 14 via the first input terminal 30 and fitness candidates given from the fitness degree candidate calculation unit 13 via the second input terminal 31. To do.
- the candidate storage unit 50 outputs the motion vector candidate for the predetermined region of interest from the first output terminal 54 every time the motion vector candidate and the fitness candidate for the predetermined region of interest in the frame are stored.
- a suitability degree candidate for a predetermined attention area is output from the second output terminal 55. Further, the candidate storage unit 50 provides the center selection unit 51 with the suitability candidates of the predetermined attention area.
- the center selection unit 51 selects a fitness candidate located in the center of a predetermined fitness interpolation range, for example, the horizontal M2 pixel and vertical pixel N2 regions shown in FIG. And output.
- the center pixel suitability candidate that is the suitability candidate output from the center selection unit 51 is given to the suitability comparison unit 53.
- the suitability threshold storage unit 52 stores a preset suitability threshold for comparison with the center pixel suitability candidate output from the center selection unit 51.
- the fitness level threshold value storage unit 52 provides the fitness level threshold value to the fitness level comparison unit 53.
- the fitness level comparison unit 53 compares the center pixel fitness level candidate given from the center selection unit 51 with the fitness level threshold value given from the fitness level threshold storage unit 52, and outputs a fitness level equal to or higher than the fitness level threshold value. Specifically, the fitness level comparison unit 53 compares the center pixel fitness level candidate given from the center selection unit 51 with the fitness level threshold value given from the fitness level threshold storage unit 52 to obtain a center pixel fitness level candidate.
- the center pixel fitness candidate is output as the fitness, and if the center pixel fitness candidate is determined to be less than the fitness threshold, the fitness threshold is output as the fitness.
- the fitness level comparison unit 53 gives the output fitness level to the candidate storage unit 50.
- the candidate storage unit 50 overwrites the stored fitness candidates to the storage address corresponding to the center of the area of the horizontal M2 pixel and the vertical pixel N2 shown in FIG. 10 with the fitness given from the fitness comparison unit 53. And remember.
- the storage unit 15A uses the storage address corresponding to the center pixel CP2 located in the center of the region of the horizontal direction M2 pixel and the vertical direction N2 pixel shown in FIG.
- the stored center pixel suitability candidate is compared with the suitability threshold, and if the center pixel suitability candidate is equal to or greater than the suitability threshold, the center pixel that is the suitability candidate stored in the storage address corresponding to the center pixel CP2 If the fitness candidate is held and the center pixel fitness candidate is less than the fitness threshold, the fitness threshold is stored instead of the center pixel fitness candidate.
- the motion vector determining unit 16 interpolates the motion vector using the motion vector candidate and the fitness candidate, it is possible to suppress the occurrence of an error due to a motion vector candidate whose fitness is less than the fitness threshold. Can do. In addition, it is possible to give a degree of fitness equal to or higher than the threshold of fitness to all pixels.
- the fitness threshold stored in the fitness threshold storage unit 52 is a non-zero minimum value, that is, a value other than 0 among possible values of the fitness candidate calculated by the fitness candidate calculation unit 13.
- the goodness-of-fit candidate with a high goodness of fit remains as it is, and the goodness-of-fit candidate with a low goodness of fit is replaced with a goodness-of-fit threshold.
- FIG. 14 is a block diagram showing a configuration of an image processing apparatus 60 according to the third embodiment of the present invention.
- the image processing device 60 includes an input terminal 61, a motion vector detection device 62, a frame rate conversion unit 63, and an output terminal 64.
- the motion vector detection device 62 according to the present embodiment is the motion vector detection device according to the first embodiment or the second embodiment described above.
- a plurality of input frames are input to the motion vector detection device 62 and the frame rate conversion unit 63 via the input terminal 61.
- the motion vector detection device 62 generates and outputs a motion vector and fitness from a plurality of input frames.
- the motion vector and the fitness level output from the motion vector detection device 62 are given to the frame rate conversion unit 63.
- the frame rate conversion unit 63 interpolates between the two input frames from a plurality of input frames, for example, pixels included in the two input frames, based on the motion vector and the fitness given from the motion vector detecting device 62. Generate and output a pixel of the interpolation frame. As a result, the frame rate conversion unit 63 converts the frame rate of the image. The pixel of the interpolation frame output from the frame rate conversion unit 63 is output from the output terminal 64.
- the frame rate conversion unit 63 corresponds to a frame rate conversion unit.
- FIG. 15 is a block diagram showing a configuration of the frame rate conversion unit 63.
- the frame rate conversion unit 63 includes a first input terminal 70, a second input terminal 71, a third input terminal 72, a delay unit 73, a reference coordinate generation unit 74, an interframe interpolation unit 75, and an output terminal 76. .
- a plurality of input frames are given to the delay unit 73 via the first input terminal 70.
- the reference coordinate generation unit 74 is given a motion vector via the second input terminal 71 from the motion vector detection device 62 and given a fitness level via the third input terminal 72.
- the reference coordinate generation unit 74 calculates a reference coordinate from the motion vector given from the motion vector detection device 62 via the second input terminal 71 and the fitness given from the motion vector detection device 62 via the third input terminal 72. Is generated and output. The reference coordinates output from the reference coordinate generation unit 74 are given to the delay unit 73.
- the delay unit 73 holds a plurality of input frames given via the first input terminal 70, and corresponds to the reference coordinates given from the reference coordinate generation unit 74, specifically the reference coordinates of the first frame and the second frame.
- the pixel to be output is output. Pixels corresponding to the reference coordinates of the first frame and the second frame output from the delay unit 73 are provided to the inter-frame interpolation unit 75.
- the inter-frame interpolation unit 75 generates and outputs a pixel of an interpolation frame from the pixels corresponding to the reference coordinates of the first frame and the second frame given from the delay unit 73.
- the pixels of the interpolation frame output from the inter-frame interpolation unit 75 are output from the output terminal 76.
- the reference coordinate generation unit 74 refers to the motion vector given from the motion vector detection device 62 via the second input terminal 71 and the fitness given from the motion vector detection device 62 via the third input terminal 72. Calculate and output coordinates. Specifically, the fitness is compared with a predetermined fitness threshold for calculating reference coordinates (hereinafter referred to as “reference coordinate fitness threshold”). As a result of the comparison, if the fitness is larger than the fitness threshold for reference coordinates, it is determined that the frame interpolation can be performed using the motion vector, and the reference coordinates of the first frame and the second frame are calculated based on the motion vector. And output. When the fitness is equal to the reference coordinate fitness threshold or smaller than the reference coordinate fitness threshold, it is determined that the motion vector cannot be used, and the motion vector is regarded as (0, 0). The reference coordinates of the first frame and the second frame are output.
- FIG. 16 is a diagram for explaining the reference coordinates.
- the point at which the motion vector passing through the position GP of the pixel (Q x , Q y ) generated in the interpolation frame IF and the first frame F1 intersect is the reference coordinates RC1 (P 1x , P 1x , P 1y ), and a point where the motion vector passing through the position GP and the second frame F2 intersect is a reference coordinate RC2 (P 2x , P 2y ) of the second frame F2.
- the reference coordinates can be generated according to the following equation (20), for example.
- Equation (20) ( ⁇ , ⁇ ) is a motion vector, and Fph is the phase of the interpolation frame IF.
- FIG. 17 is a diagram for explaining generation of pixels of the interpolation frame IF.
- the inter-frame interpolation unit 75 generates pixels of the interpolation frame IF from the pixels given from the delay unit 73 as shown in FIG.
- the pixels of the interpolation frame are generated, for example, according to the following equation (22).
- Y 1 is the image information of the pixel of the first frame F1 given from the delay unit 73
- Y 2 is the pixel information of the pixel of the second frame F2 given from the delay unit 73
- Y 0 Is the pixel image information generated by the interpolation frame IF
- Fph is the phase of the interpolation frame IF.
- the pixel information for example, using the luminance and the color difference, the pixel of the interpolation frame IF is generated according to the equation (22) for each.
- a plurality of interpolation frames can be generated from a plurality of frames, so that frame rate conversion can be performed.
- the image processing device 60 includes the excellent motion vector detection device 62 and the frame rate conversion unit 63 as described above.
- the motion vector detection device 62 can suppress the erroneous detection of the motion vector and reduce the circuit scale or processing time for detecting the motion vector, so that the pixels included in the interpolation frame can be generated with high accuracy. it can. Therefore, the image processing apparatus 60 that can convert the frame rate while suppressing the deterioration of the image quality is realized.
- FIG. 18 is a block diagram showing a configuration of an image processing apparatus 80 according to the fourth embodiment of the present invention.
- the image processing device 80 includes an input terminal 81, a motion vector detection device 62, a resolution conversion unit 82, and an output terminal 83.
- the motion vector detection device 62 is the motion vector detection device of the first or second embodiment described above, similarly to the image processing device 60 of the third embodiment described above.
- a linear interpolation method that performs interpolation from four surrounding pixels and a bicubic method that performs interpolation from 16 surrounding pixels are known. These perform interpolation processing from surrounding points within one frame.
- the image processing apparatus 80 detects motion vectors from two frames, finds a plurality of corresponding pixels between the two frames, and calculates pixels having different resolutions from the plurality of pixels by interpolation processing. .
- the number of pixels corresponding between two frames may be four as in the linear interpolation method, or may be 16 as in the bicubic method.
- the motion vector detecting device 62 and the resolution converting unit 82 are provided with a plurality of input frames from the input terminal 81. As described in the first and second embodiments, the motion vector detection device 62 generates and outputs a motion vector and a fitness degree from a plurality of input frames. The motion vector and the fitness level output from the motion vector detection device 62 are given to the resolution conversion unit 82.
- the resolution conversion unit 82 generates and outputs a new pixel whose resolution is converted from a plurality of input frames, for example, pixels included in two input frames, based on the motion vector and the degree of matching given from the motion vector detection device 62. . As a result, a new image having a resolution different from that of the image composed of the plurality of input frames is generated.
- the resolution-converted pixels output from the resolution converter 82 are output from the output terminal 83.
- the resolution conversion unit 82 corresponds to a resolution conversion unit.
- FIG. 19 is a diagram for explaining the operation of the resolution conversion unit 82.
- FIG. 19A is a diagram showing a grid before resolution conversion
- FIG. 19B is a diagram showing a grid after resolution conversion.
- FIG. 19A shows nine grids
- FIG. 19B shows 16 grids.
- FIG. 20 is a diagram for explaining the operation of the resolution conversion unit 82. A case where the input frame is enlarged 4/3 times as shown in FIG. 19 will be described as an example.
- the resolution conversion unit 82 searches for four adjacent motion vectors having similar movement amounts from among the motion vectors supplied from the motion vector detection device 62, and the first frame F1 and the second frame. Find a set of 4 pixels corresponding to F2.
- a motion vector having a fitness less than or equal to a search fitness threshold that is a predetermined motion vector search fitness threshold is excluded from the search target.
- the found set of four pixels is interpolated in each of the first frame F1 and the second frame F2, and the interpolation pixel IP1 of the first frame F1 and the interpolation pixel IP2 of the second frame F2 are calculated.
- the resolution conversion unit 82 calculates and outputs a pixel whose resolution has been converted by interpolation processing from the interpolation pixel IP1 of the first frame F1 and the interpolation pixel IP2 of the second frame F2. For the pixels where four adjacent motion vectors having similar movement amounts are not found, interpolation processing is performed from the surrounding four pixels in the second frame F2, and the resolution-converted pixels are calculated and output.
- resolution conversion can be performed by obtaining pixels with different resolutions.
- the case of performing resolution conversion using four corresponding pixels of the first frame F1 and the second frame F2 has been described.
- the number of corresponding pixels is not limited to four, for example, sixteen. But you can.
- the image processing apparatus 80 includes the excellent motion vector detection apparatus 62 and the resolution conversion unit 83 as described above.
- the motion vector detection device 62 can suppress the detection of the motion vector and reduce the circuit scale or processing time for detecting the motion vector.
- An image processing apparatus 80 that can be generated is realized.
Landscapes
- Engineering & Computer Science (AREA)
- Multimedia (AREA)
- Signal Processing (AREA)
- Computer Vision & Pattern Recognition (AREA)
- Physics & Mathematics (AREA)
- General Physics & Mathematics (AREA)
- Theoretical Computer Science (AREA)
- Television Systems (AREA)
- Image Analysis (AREA)
Abstract
The objective is to provide a motion vector detection device and an image processing device, with which the circuit scale or the processing time required for detecting motion vectors can be reduced, while suppressing the erroneous detection of motion vectors. After decimation to one representative pixel from multiple pixels in a targeted area of an input frame by a decimation unit (12), a suitable candidate is calculated by a suitable-candidate calculation unit (13). The suitable candidate is used to calculate a motion-vector-candidate by a motion vector candidate calculation unit (14), and the motion vector is detected by a motion vector determination unit (16).
Description
本発明は、複数のフレーム間において、フレームの中の同じオブジェクトを対応付けることで、オブジェクトの移動量および距離情報となる動きベクトルを検出する動きベクトル検出装置およびこれを備える画像処理装置に関する。
The present invention relates to a motion vector detection device that detects a motion vector as a movement amount and distance information of an object by associating the same object in the frame between a plurality of frames, and an image processing device including the motion vector detection device.
従来技術の動きベクトル検出装置および画像処理装置では、パターンマッチング法、勾配法、相互相関関数による方法、フーリエ変換による方法などによって、動きベクトルを算出している(たとえば、非特許文献1参照)。
In conventional motion vector detection devices and image processing devices, motion vectors are calculated by a pattern matching method, a gradient method, a method using a cross-correlation function, a method using a Fourier transform, and the like (for example, see Non-Patent Document 1).
これらの方法によって、時間の異なる複数のフレームを用いて求めた動きベクトルは、フレーム間符号化における符号化効率向上、フレーム内挿(フレーム数変換、フレーム追加、デインターレース変換)、および解像度変換などの動き検出に用いることができる。また、視点の位置が空間的に異なる位置で撮影した複数のフレームを用いて求めた動きベクトルは、距離画像および3次元画像の作成、オブジェクトの距離測定、ならびに3次元画像符号化などの距離検出の用途に用いることができる。
With these methods, motion vectors obtained using multiple frames at different times can be used to improve coding efficiency in interframe coding, frame interpolation (frame number conversion, frame addition, deinterlace conversion), resolution conversion, etc. Can be used for motion detection. In addition, motion vectors obtained using a plurality of frames taken at spatially different viewpoint positions are used for distance detection such as creation of distance images and three-dimensional images, object distance measurement, and three-dimensional image coding. It can be used for
これらの用途において、特に広く実用的に用いられている前記パターンマッチング法では、たとえば2つのフレーム信号g0(x,y)、g1(x,y)の間で以下の式(1)を用いて移動量(ξ,η)のマッチングの指標である適合度候補e(ξ,η)を求める。
In these applications, in the pattern matching method that is particularly widely used practically, for example, the following equation (1) is obtained between two frame signals g 0 (x, y) and g 1 (x, y). Using this, the candidate for fitness e (ξ, η), which is an index for matching the movement amount (ξ, η), is obtained.
ここで、Tlimは2つのフレーム信号間の誤差を極性反転して適合度候補を計算するときの所定の閾値であり、2つのフレーム信号には輝度信号を用いる。また式(1)では、2つのフレーム信号について差の絶対値を用いて計算しているが、差の自乗、または非線形特性などを用いてもよい。また、2つのフレーム信号としても、輝度信号以外に、輝度信号および色差信号の両方を用いてもよい。
Here, T lim is a predetermined threshold when calculating the fitness candidate by inverting the polarity of the error between the two frame signals, and a luminance signal is used for the two frame signals. Further, in the expression (1), the calculation is performed using the absolute value of the difference between the two frame signals, but the square of the difference or a non-linear characteristic may be used. In addition to the luminance signal, both the luminance signal and the color difference signal may be used as the two frame signals.
加算する範囲Bとしては、8×8画素程度のブロックとする。動きベクトル候補の探索範囲内の移動量(ξ,η)それぞれについて式(1)の計算を行い、計算結果の中で適合度候補e(ξ,η)が最大となる移動量(ξ,η)をそのブロックの動きベクトル候補とする。
The range B to be added is a block of about 8 × 8 pixels. Equation (1) is calculated for each movement amount (ξ, η) within the search range of the motion vector candidate, and the movement amount (ξ, η) that maximizes the fitness candidate e (ξ, η) among the calculation results. ) As motion vector candidates for the block.
また、相互相関関数による方法では、以下の式(2)のh(ξ,η)を最大にする移動量(ξ,η)を動きベクトル候補として求める。
In the method using the cross-correlation function, the movement amount (ξ, η) that maximizes h (ξ, η) in the following equation (2) is obtained as a motion vector candidate.
前記の方法によって動きベクトル候補および適合度候補を算出した結果、不適切な動きベクトル候補が割り当てられるブロック、または動きベクトル候補が割り当てられないブロックが発生する(たとえば、特許文献1参照)。動きベクトル候補が割り当てられないブロックを補間し、かつ不適切な動きベクトル候補による誤差を抑えるために、動きベクトル候補の補間処理を行い、動きベクトルを決定する。また適合度候補についても補間処理を行い、適合度を決定する。従来技術の動きベクトル検出装置および画像処理装置では、動きベクトルおよび適合度を決定するために、所定の同じ範囲内に存在する動きベクトル候補と適合度候補とを同数用いて、動きベクトルおよび適合度を決定している。たとえば特許文献2では、動きベクトルおよび適合度を、以下の式(3)および式(4)によって求めている。
As a result of calculating the motion vector candidate and the fitness candidate by the above method, a block to which an inappropriate motion vector candidate is assigned or a block to which a motion vector candidate is not assigned is generated (for example, see Patent Document 1). In order to interpolate a block to which no motion vector candidate is assigned and to suppress an error due to an inappropriate motion vector candidate, motion vector candidate interpolation processing is performed to determine a motion vector. Interpolation processing is also performed on the fitness candidates, and the fitness is determined. In the motion vector detecting device and the image processing device of the prior art, in order to determine the motion vector and the fitness, the same number of motion vector candidates and fitness candidates existing within the same predetermined range are used, and the motion vector and the fitness are used. Is determined. For example, in Patent Document 2, the motion vector and the fitness are obtained by the following equations (3) and (4).
式(3)および式(4)において、ai(x1,y1,x2,y2)は2つの点(x1,y1)、(x2,y2)の動きベクトルが同じ向きに動いていると推定される度合いを表す相関度であり、ri(x2,y2)はその点の動きベクトル候補がどれくらい信頼できるか、または所定の条件に合っているかの度合いを表す信頼度候補(適合度候補に相当)であり、vi(x2,y2)は補間処理に入力される動きベクトル候補である。また、式(3)のvi+1(x1,y1)は補間処理によって決定された動きベクトルであり、式(4)のri+1(x1,y1)は補間処理によって決定された信頼度(適合度に相当)である。式(3)および式(4)では、動きベクトルおよび適合度の両方について、(x2,y2)∈(x1,y1)の周囲の動きベクトル候補および適合度候補を同数用いて平均化することによって、動きベクトルおよび適合度を決定している。
In the equations (3) and (4), a i (x 1 , y 1 , x 2 , y 2 ) has the same motion vector at two points (x 1 , y 1 ) and (x 2 , y 2 ). The degree of correlation representing the degree of movement estimated in the direction, r i (x 2 , y 2 ) indicates how reliable the motion vector candidate at that point is, or the degree to which a predetermined condition is met. This is a reliability candidate to be represented (corresponding to a fitness candidate), and v i (x 2 , y 2 ) is a motion vector candidate input to the interpolation process. In addition, v i + 1 (x 1 , y 1 ) in Expression (3) is a motion vector determined by interpolation processing, and r i + 1 (x 1 , y 1 ) in Expression (4) is determined by interpolation processing. It is the determined reliability (equivalent to goodness of fit). In Equation (3) and Equation (4), for both motion vectors and fitness, the average number of motion vector candidates and fitness candidates around (x 2 , y 2 ) ∈ (x 1 , y 1 ) is used. Thus, the motion vector and the fitness are determined.
パターンマッチング法および相互相関関数による方法では、動きベクトルの探索範囲を広くすると、探索範囲の面積に応じて演算回数が増加し、回路規模または計算時間が増大するという問題がある。そのため、フレーム間符号化にパターンマッチング法を適用する場合には、まず探索の精度を粗くし、数画素単位の精度でマッチングを行い、求めた動きベクトルの近辺で、徐々に探索の精度を1画素単位に近づけながらマッチングを繰り返す、いわゆる多段階探索法が多く用いられている。
In the pattern matching method and the method using the cross-correlation function, if the search range of the motion vector is widened, there is a problem that the number of operations increases according to the area of the search range, and the circuit scale or calculation time increases. Therefore, when applying the pattern matching method to inter-frame coding, first, the search accuracy is coarsened, matching is performed with an accuracy of several pixels, and the search accuracy is gradually increased in the vicinity of the obtained motion vector. A so-called multi-stage search method that repeats matching while approaching a pixel unit is often used.
しかしながら、このような多段階探索法による動きベクトル検出では、全探索をしていないので、局所最適な動きベクトルが求まることとなり、フレーム全体としては、必ずしも最適な動きベクトルが求まるとは限らない。フレームの内容によっては、全探索をした場合と比較すると、動きベクトルとして大きい誤差が出てしまうという問題がある。
However, since motion vector detection by such a multi-step search method does not perform full search, a locally optimal motion vector is obtained, and an optimal motion vector is not necessarily obtained for the entire frame. Depending on the contents of the frame, there is a problem that a large error occurs as a motion vector as compared with the case of full search.
フレーム間符号化に多段階探索法を適用する場合には、局所最適な動きベクトルを用いても画質的に余り問題にならないことが多い。しかし、フレーム内挿に多段階探索法を適用する場合には、画質的な劣化の点で問題となることがあり、また距離検出に多段階探索法を適用する場合には、距離の誤検出が問題となることがある。
When applying a multi-step search method to inter-frame coding, it is often not a problem in terms of image quality even if a locally optimal motion vector is used. However, when the multi-step search method is applied to frame interpolation, there may be a problem in terms of image quality degradation, and when the multi-step search method is applied to distance detection, a false detection of distance may occur. May be a problem.
本発明では、上記のような問題点を解決し、動きベクトルの誤検出を抑えつつ、動きベクトルの検出のための回路規模または処理時間を小さく抑えることが可能な動きベクトル検出装置および画像処理装置を提供することを目的とする。
According to the present invention, a motion vector detection device and an image processing device that can solve the above-described problems and suppress the motion vector detection or the processing time while suppressing erroneous detection of the motion vector. The purpose is to provide.
本発明の動きベクトル検出装置は、第1および第2の入力フレーム間で、前記第1および第2の入力フレームにそれぞれ含まれる1個以上の画素同士を対応付ける動きベクトルを検出する動きベクトル検出装置であって、各前記入力フレームのエッジ成分を検出するエッジ検出手段と、各前記入力フレームの予め定める注目領域に含まれる複数の画素を、前記エッジ検出手段によって検出されるエッジ成分に基づいて、代表となる1個の画素に間引いて、この画素を代表1画素として出力する間引き手段と、前記間引き手段から出力される前記第1および第2の入力フレームの前記代表1画素を用いて、前記第1および第2の入力フレーム間の移動量の適合度合いを表す適合度の候補となる適合度候補を算出する適合度候補算出手段と、前記間引き手段から出力される前記代表1画素の各入力フレームにおける位置、および前記適合度候補算出手段によって算出される前記適合度候補に基づいて、前記第1および第2の入力フレーム間の前記動きベクトルの候補となる動きベクトル候補を算出する動きベクトル候補算出手段と、前記適合度候補算出手段によって算出される前記適合度候補、および前記動きベクトル候補算出手段によって算出される前記動きベクトル候補を記憶する記憶手段と、前記記憶手段に記憶される前記適合度候補および前記動きベクトル候補に基づいて、前記適合度および前記動きベクトルを決定する動きベクトル決定手段とを備え、前記記憶手段は、前記間引き手段から出力される前記代表1画素の各入力フレームにおける位置に基づいて、前記動きベクトル候補および前記適合度候補を記憶する記憶アドレスを算出し、算出した前記記憶アドレスに前記動きベクトル候補および前記適合度候補を記憶することを特徴とする。
The motion vector detection apparatus according to the present invention detects a motion vector that associates one or more pixels included in the first and second input frames between the first and second input frames. An edge detection unit for detecting an edge component of each input frame, and a plurality of pixels included in a predetermined attention area of each input frame based on the edge component detected by the edge detection unit, Using the thinning means for thinning out one representative pixel and outputting this pixel as the representative one pixel, and the representative one pixel of the first and second input frames output from the thinning means, A fitness candidate calculation means for calculating a fitness candidate that is a fitness candidate that represents the adaptation of the movement amount between the first and second input frames; The motion vector between the first and second input frames based on the position of the representative pixel output from the thinning unit in each input frame and the fitness candidate calculated by the fitness candidate calculation unit A motion vector candidate calculation unit that calculates a motion vector candidate that is a candidate for the motion vector, the fitness candidate calculated by the fitness candidate calculation unit, and the motion vector candidate calculated by the motion vector candidate calculation unit Storage means; and motion vector determination means for determining the fitness and the motion vector based on the fitness candidate and the motion vector candidate stored in the storage means, and the storage means is the thinning means Based on the position of the representative pixel output from each input frame. Torr candidate and calculates a memory address for storing the adaptation degree candidates, and to store the motion vector candidate and the adaptation degree candidates calculated the storage address.
本発明の画像処理装置は、前記動きベクトル検出装置と、前記動きベクトル検出装置によって検出される前記動きベクトルおよび前記適合度に基づいて、前記第1および第2の入力フレームに含まれる画素から、前記第1および第2の入力フレーム間に内挿する内挿フレームに含まれる画素を生成することによって、画像のフレームレートを変換するフレームレート変換手段とを備えることを特徴とする。
The image processing apparatus according to the present invention includes, based on the motion vector detection device and the pixels included in the first and second input frames, based on the motion vector detected by the motion vector detection device and the fitness. Frame rate conversion means for converting a frame rate of an image by generating pixels included in an interpolation frame to be interpolated between the first and second input frames.
また本発明の画像処理装置は、前記動きベクトル検出装置と、前記動きベクトル検出装置によって検出される前記動きベクトルおよび前記適合度に基づいて、前記第1および第2の入力フレームに含まれる画素から新たな画素を生成することによって、前記第1および第2の入力フレームで構成される画像とは異なる解像度を有する新たな画像を生成する解像度変換手段とを備えることを特徴とする。
Further, the image processing apparatus of the present invention is based on the motion vector detection device and the pixels included in the first and second input frames based on the motion vector detected by the motion vector detection device and the fitness. And a resolution conversion unit configured to generate a new image having a different resolution from the image formed by the first and second input frames by generating a new pixel.
本発明の動きベクトル検出装置によれば、間引き手段によって、第1および第2の入力フレームの注目領域に含まれる複数の画素から代表1画素に間引いた後に、この代表1画素を用いて、適合度候補算出手段によって適合度候補が算出される。これによって、適合度候補を算出するための演算量を削減することが可能になるので、適合度候補を算出するための回路規模または処理時間を小さく抑えることができる。この適合度候補を用いて動きベクトル候補が算出され、動きベクトルが検出されるので、動きベクトルの検出のための回路規模または処理時間を小さく抑えることができる。
According to the motion vector detection device of the present invention, after thinning out a plurality of pixels included in the attention area of the first and second input frames to one representative pixel by the thinning unit, the representative one pixel is used for matching. The fitness candidate calculation means calculates fitness candidates. As a result, it is possible to reduce the amount of calculation for calculating the suitability level candidates, so that the circuit scale or processing time for calculating the suitability level candidates can be reduced. Since motion vector candidates are calculated using the fitness candidates and motion vectors are detected, the circuit scale or processing time for motion vector detection can be kept small.
また間引き手段は、エッジ検出手段によって検出される各入力フレームのエッジ成分に基づいて注目領域の複数の画素を代表1画素に間引くので、間引いたことによる動きベクトルの検出精度の低下を抑えることができる。また記憶手段は、間引いたときの代表1画素の位置情報を用いて、動きベクトル候補および適合度候補の記憶アドレスを算出し、その記憶アドレスに動きベクトル候補および適合度候補を記憶するので、間引きによる位置誤差の発生を抑えることができる。また動きベクトル決定手段は、動きベクトル候補および適合度候補から動きベクトルおよび適合度を決定するので、動きベクトルを決定するときの誤差の発生を抑えることができる。
Further, since the thinning-out means thins out a plurality of pixels in the region of interest to one representative pixel based on the edge component of each input frame detected by the edge detecting means, it is possible to suppress a decrease in motion vector detection accuracy due to the thinning-out. it can. In addition, the storage means calculates the storage address of the motion vector candidate and the fitness candidate using the position information of one representative pixel at the time of the thinning, and stores the motion vector candidate and the fitness candidate at the storage address. It is possible to suppress the occurrence of position errors due to. Further, since the motion vector determining means determines the motion vector and the fitness from the motion vector candidate and the fitness candidate, it is possible to suppress the occurrence of an error when determining the motion vector.
したがって、動きベクトルの誤検出を抑えつつ、動きベクトルの検出のための回路規模または処理時間を小さく抑えることができる。
Therefore, it is possible to reduce the circuit scale or processing time for detecting the motion vector while suppressing erroneous detection of the motion vector.
本発明の画像処理装置によれば、前述のような優れた動きベクトル検出装置とフレームレート変換手段とを備えて画像処理装置が構成される。動きベクトル検出装置は、動きベクトルの誤検出を抑えつつ、動きベクトルの検出のための回路規模または処理時間を小さく抑えることができるので、内挿フレームに含まれる画素を精度良く生成することができる。したがって、画質の低下を抑えて、フレームレートを変換することのできる画像処理装置が実現される。
According to the image processing apparatus of the present invention, the image processing apparatus is configured to include the above-described excellent motion vector detection device and frame rate conversion means. The motion vector detection device can suppress the erroneous detection of the motion vector and suppress the circuit scale or the processing time for detecting the motion vector, so that the pixels included in the interpolation frame can be generated with high accuracy. . Therefore, an image processing apparatus capable of converting the frame rate while suppressing deterioration in image quality is realized.
また本発明の画像処理装置によれば、前述のような優れた動きベクトル検出装置と解像度変換手段とを備えて画像処理装置が構成される。動きベクトル検出装置は、動きベクトルの誤検出を抑えつつ、動きベクトルの検出のための回路規模または処理時間を小さく抑えることができるので、画質の低下を抑えて、解像度の異なる新たな画像を生成することのできる画像処理装置が実現される。
Further, according to the image processing apparatus of the present invention, the image processing apparatus is configured to include the above-described excellent motion vector detection device and resolution conversion means. The motion vector detection device can reduce the circuit size or processing time for motion vector detection while suppressing false detection of motion vectors, so it can suppress the degradation of image quality and generate new images with different resolutions. An image processing apparatus that can do this is realized.
この発明の目的、特徴、局面、および利点は、以下の詳細な説明と添付図面とによって、より明白となる。
The objects, features, aspects and advantages of the present invention will become more apparent from the following detailed description and the accompanying drawings.
<第1の実施の形態>
図1は、本発明の第1の実施の形態である動きベクトル検出装置1の構成を示すブロック図である。図2は、代表1画素の一例を示す図である。動きベクトル検出装置1は、入力される複数のフレーム間において、複数のフレームにそれぞれ含まれる1個以上の画素同士を対応付ける動きベクトルを検出するものである。具体的には、動きベクトル検出装置1は、入力されるフレーム(以下「入力フレーム」という場合がある)のうち、注目する2つの入力フレーム、すなわち第1および第2の入力フレーム間で、第1および第2の入力フレームにそれぞれ含まれる1個以上の画素同士を対応付ける動きベクトルを検出する。 <First Embodiment>
FIG. 1 is a block diagram showing a configuration of a motionvector detection apparatus 1 according to the first embodiment of the present invention. FIG. 2 is a diagram illustrating an example of one representative pixel. The motion vector detection device 1 detects a motion vector that associates one or more pixels included in each of a plurality of frames between a plurality of input frames. Specifically, the motion vector detection device 1 is configured to input the first two input frames of interest among the input frames (hereinafter sometimes referred to as “input frames”), that is, between the first and second input frames. A motion vector that associates one or more pixels included in each of the first and second input frames is detected.
図1は、本発明の第1の実施の形態である動きベクトル検出装置1の構成を示すブロック図である。図2は、代表1画素の一例を示す図である。動きベクトル検出装置1は、入力される複数のフレーム間において、複数のフレームにそれぞれ含まれる1個以上の画素同士を対応付ける動きベクトルを検出するものである。具体的には、動きベクトル検出装置1は、入力されるフレーム(以下「入力フレーム」という場合がある)のうち、注目する2つの入力フレーム、すなわち第1および第2の入力フレーム間で、第1および第2の入力フレームにそれぞれ含まれる1個以上の画素同士を対応付ける動きベクトルを検出する。 <First Embodiment>
FIG. 1 is a block diagram showing a configuration of a motion
動きベクトル検出装置1は、たとえば動画像において画面内の被写体の動きを表す画像動きベクトルを検出するものであり、時系列的に前後の関係となる前フレームと後フレームとの間の動きベクトルを検出する。この動きベクトル検出装置1によって検出された動きベクトルは、ビデオカメラの手振れ補正などに利用される。動きベクトル検出装置1に入力されるフレームの画像信号は、水平方向の画素走査および垂直方向のライン走査を行うことによって読み出される。
The motion vector detection device 1 detects, for example, an image motion vector representing a motion of a subject in a screen in a moving image, and calculates a motion vector between a previous frame and a subsequent frame that are in a time series relationship. To detect. The motion vector detected by the motion vector detection device 1 is used for camera shake correction of the video camera. The image signal of the frame input to the motion vector detection device 1 is read out by performing horizontal pixel scanning and vertical line scanning.
本実施の形態の動きベクトル検出装置1は、入力端子10、エッジ検出部11、間引き部12、適合度候補算出部13、動きベクトル候補算出部14、記憶部15、動きベクトル決定部16、第1出力端子17および第2出力端子18を備えて構成される。
The motion vector detection device 1 according to the present embodiment includes an input terminal 10, an edge detection unit 11, a thinning unit 12, a fitness candidate calculation unit 13, a motion vector candidate calculation unit 14, a storage unit 15, a motion vector determination unit 16, A first output terminal 17 and a second output terminal 18 are provided.
入力端子10には、たとえばビデオカメラ内の撮像素子で取得された動画像データなどの画像データが入力される。入力端子10に入力された画像データは、複数のフレームを含んでおり、フレーム単位で、エッジ検出部11と間引き部12とに入力される。
The input terminal 10 receives image data such as moving image data acquired by an image sensor in a video camera, for example. The image data input to the input terminal 10 includes a plurality of frames, and is input to the edge detection unit 11 and the thinning unit 12 in units of frames.
エッジ検出部11は、入力された複数のフレームの画像信号から、水平方向および垂直方向のエッジ成分を検出する。エッジ検出部11は、検出したエッジ成分を間引き部12および適合度候補算出部13に与える。エッジ検出部11は、エッジ検出手段に相当する。
The edge detection unit 11 detects horizontal and vertical edge components from the input image signals of a plurality of frames. The edge detection unit 11 gives the detected edge component to the thinning unit 12 and the fitness candidate calculation unit 13. The edge detection unit 11 corresponds to an edge detection unit.
間引き部12は、入力フレームの予め定める注目領域に含まれる複数の画素を間引いて、代表となる1個の画素を代表1画素として出力する。具体的には、間引き部12は、エッジ検出部11から与えられるエッジ成分を用いてエッジ強度値を計算し、図2に示すように、入力フレームの複数の画素の中で、エッジ強度値が一番大きい画素を代表1画素として出力する。また間引き部12は、代表1画素の位置を出力する。間引き部12は、前記代表1画素およびその位置を表す画像信号を、適合度候補算出部13、動きベクトル候補算出部14および記憶部15に与える。間引き部12は、間引き手段に相当する。
The thinning unit 12 thins out a plurality of pixels included in a predetermined attention area of the input frame, and outputs one representative pixel as a representative pixel. Specifically, the thinning unit 12 calculates an edge strength value using the edge component given from the edge detection unit 11, and the edge strength value is calculated among a plurality of pixels of the input frame as shown in FIG. The largest pixel is output as one representative pixel. The thinning unit 12 outputs the position of one representative pixel. The thinning unit 12 supplies the representative pixel and the image signal representing the position thereof to the fitness candidate calculation unit 13, the motion vector candidate calculation unit 14, and the storage unit 15. The thinning unit 12 corresponds to a thinning unit.
適合度候補算出部13は、2つの入力フレームについて間引き部12から与えられる2つの代表1画素の画像信号に基づいて適合度候補を求める。さらに具体的には、適合度候補算出部13は、間引き部12から出力される前記代表1画素を用いて、2つの入力フレーム間の移動量の適合度合いを表す適合度の候補となる適合度候補を算出する。適合度候補算出部13は、算出した適合度候補を、動きベクトル候補算出部14および記憶部15に与える。適合度候補算出部13は、適合度候補算出手段に相当する。
The goodness-of-fit candidate calculation unit 13 obtains goodness-of-fit candidates based on two representative one-pixel image signals given from the thinning-out unit 12 for two input frames. More specifically, the fitness candidate calculation unit 13 uses the representative one pixel output from the thinning unit 12, and serves as a fitness candidate that represents the fitness of the movement amount between two input frames. Candidates are calculated. The fitness candidate calculation unit 13 provides the calculated fitness candidates to the motion vector candidate calculation unit 14 and the storage unit 15. The fitness candidate calculation unit 13 corresponds to fitness candidate calculation means.
動きベクトル候補算出部14は、適合度候補算出部13から与えられる適合度候補と、間引き部12から与えられる代表1画素の位置とから、2つの入力フレーム間の動きベクトルの候補となる動きベクトル候補を算出する。動きベクトル候補算出部14は、算出した動きベクトル候補を記憶部15に与える。動きベクトル候補算出部14は、動きベクトル候補算出手段に相当する。
The motion vector candidate calculation unit 14 is a motion vector that is a candidate for a motion vector between two input frames, based on the fitness candidate given from the fitness candidate calculation unit 13 and the position of one representative pixel given from the thinning unit 12. Candidates are calculated. The motion vector candidate calculation unit 14 gives the calculated motion vector candidates to the storage unit 15. The motion vector candidate calculation unit 14 corresponds to motion vector candidate calculation means.
記憶部15は、動きベクトル候補算出部14から与えられる動きベクトル候補と、間引き部12から与えられる代表1画素の位置と、内挿フレームの位相とから記憶アドレスを算出する。記憶部15は、算出した記憶アドレスに、動きベクトル候補算出部14から与えられる動きベクトル候補、および適合度候補算出部13から与えられる適合度候補を記憶する。また記憶部15は、所定の範囲の記憶処理が終了する毎に、所定の範囲の動きベクトル候補および適合度候補を出力する。さらに記憶部15は、後述する動きベクトル決定部16から与えられる動きベクトルおよび適合度を記憶する。記憶部15は、記憶手段に相当する。
The storage unit 15 calculates a storage address from the motion vector candidate given from the motion vector candidate calculation unit 14, the position of one representative pixel given from the thinning unit 12, and the phase of the interpolation frame. The storage unit 15 stores the motion vector candidate given from the motion vector candidate calculation unit 14 and the fitness candidate given from the fitness candidate calculation unit 13 at the calculated storage address. The storage unit 15 outputs a motion vector candidate and a fitness candidate in a predetermined range every time the storage process in the predetermined range is completed. Furthermore, the storage unit 15 stores a motion vector and a fitness level given from a motion vector determination unit 16 described later. The storage unit 15 corresponds to a storage unit.
動きベクトル決定部16は、記憶部15から与えられる動きベクトル候補および適合度候補に基づいて、動きベクトルおよび適合度を決定する。動きベクトル決定部16によって決定された動きベクトルは、記憶部15に与えられるとともに、第1出力端子17から出力される。動きベクトル決定部16によって決定された適合度は、記憶部15に与えられるとともに、第2出力端子18から出力される。動きベクトル決定部16は、動きベクトル決定手段に相当する。
The motion vector determination unit 16 determines the motion vector and the fitness based on the motion vector candidate and the fitness candidate given from the storage unit 15. The motion vector determined by the motion vector determination unit 16 is supplied to the storage unit 15 and is output from the first output terminal 17. The fitness determined by the motion vector determination unit 16 is given to the storage unit 15 and is output from the second output terminal 18. The motion vector determination unit 16 corresponds to a motion vector determination unit.
次に、動きベクトル検出装置1の各部の動作を具体的に説明する。まず、図1に示すエッジ検出部11の動作について説明する。エッジ検出部11は、入力された複数のフレームの画像信号から、水平方向および垂直方向のエッジ成分を検出し、間引き部12に与える。エッジ成分の検出は、たとえばプリウィット(Prewitt)フィルタ、ソーベル(Sobel)フィルタのような1次微分フィルタを用いて行ってもよいし、1次微分フィルタを2次微分フィルタなどと組合せて用いて行ってもよい。具体的には、以下の式(5)に従って計算することによってエッジ成分が求められる。
Next, the operation of each part of the motion vector detection device 1 will be specifically described. First, the operation of the edge detection unit 11 shown in FIG. 1 will be described. The edge detection unit 11 detects edge components in the horizontal direction and the vertical direction from the input image signals of a plurality of frames, and provides them to the thinning unit 12. The edge component may be detected using a primary differential filter such as a Prewitt filter or a Sobel filter, or a primary differential filter is used in combination with a secondary differential filter or the like. You may go. Specifically, the edge component is obtained by calculating according to the following equation (5).
式(5)において、g(x,y)は入力フレームの座標(x,y)の画像信号であり、たとえば輝度を用いる。E’xは1次微分フィルタによって検出される水平方向のエッジ成分(以下「水平方向の1次微分エッジ成分」という)であり、E’yは1次微分フィルタによって検出される垂直方向のエッジ成分(以下「垂直方向の1次微分エッジ成分」という)である。E’’xは2次微分フィルタによって検出される水平方向のエッジ成分(以下「水平方向の2次微分エッジ成分」という)であり、E’’yは2次微分フィルタによって検出される垂直方向のエッジ成分(以下「垂直方向の2次微分エッジ成分」という)である。
In Expression (5), g (x, y) is an image signal of the coordinates (x, y) of the input frame, and for example, luminance is used. E ′ x is a horizontal edge component detected by the primary differential filter (hereinafter referred to as “horizontal primary differential edge component”), and E ′ y is a vertical edge detected by the primary differential filter. Component (hereinafter referred to as “vertical first-order differential edge component”). E ″ x is a horizontal edge component detected by the secondary differential filter (hereinafter referred to as “horizontal second differential edge component”), and E ″ y is a vertical direction detected by the secondary differential filter. Edge components (hereinafter referred to as “second-order differential edge components in the vertical direction”).
次に、図1に示す間引き部12の動作について説明する。間引き部12は、エッジ検出部11から与えられるエッジ成分を用いてエッジ強度値を計算し、図2に示すように、入力フレームの複数の画素の中で、エッジ強度値が一番大きい画素を代表1画素として出力する。具体的には、入力フレームの水平方向にM(Mは1以上の整数)画素、垂直方向にN(Nは1以上の整数)画素からなる注目領域の複数の画素の中で、エッジ強度値が一番大きい画素のみを残して、代表1画素として出力する。水平方向M画素、垂直方向N画素からなる注目領域の他の画素は破棄する。エッジ強度値は、たとえばC1(|E’x|+|E’y|)+C2(|E’’x|+|E’’y|)によって求める。ここで、C1、C2は所定の定数である。また間引き部12は、図2に示すように、オフセット基準点POから、水平方向にOx、垂直方向にOyだけ移動した代表1画素の位置RP(Ox、Oy)を出力する。なお、エッジ強度値が一番大きい画素を代表1画素とする代わりに、注目領域の所定の位置の画素、たとえば注目領域の左上の画素を代表1画素としてもよい。この場合、検出精度の点では、エッジ強度値が一番大きい画素を代表1画素とする場合に比して劣ることもあるが、代表1画素に間引いたことによる演算量削減の効果は同様に認められる。
Next, the operation of the thinning unit 12 shown in FIG. 1 will be described. The thinning unit 12 calculates an edge strength value using the edge component given from the edge detection unit 11, and, as shown in FIG. 2, the pixel having the largest edge strength value among a plurality of pixels of the input frame is calculated. Output as one representative pixel. Specifically, the edge intensity value among a plurality of pixels in a region of interest consisting of M (M is an integer of 1 or more) pixels in the horizontal direction and N (N is an integer of 1 or more) pixels in the vertical direction of the input frame. Only the largest pixel is left and output as one representative pixel. Other pixels of the region of interest consisting of M pixels in the horizontal direction and N pixels in the vertical direction are discarded. The edge strength value is obtained by, for example, C1 (| E ′ x | + | E ′ y |) + C2 (| E ″ x | + | E ″ y |). Here, C1 and C2 are predetermined constants. Further, as shown in FIG. 2, the thinning unit 12 outputs a position RP (O x , O y ) of one representative pixel moved from the offset reference point PO by O x in the horizontal direction and O y in the vertical direction. Instead of the pixel having the largest edge intensity value as one representative pixel, a pixel at a predetermined position in the attention area, for example, the upper left pixel of the attention area may be used as one representative pixel. In this case, in terms of detection accuracy, the pixel having the largest edge intensity value may be inferior to the case where the representative one pixel is used, but the effect of reducing the calculation amount by thinning out the representative one pixel is the same. Is recognized.
次に、図1に示す適合度候補算出部13の動作について説明する。適合度候補算出部13は、従来技術のパターンマッチング法と同様に、2つの入力フレームについて間引き部12から与えられる2つの代表1画素の画像信号g0(x,y)、g1(x,y)の間で移動量(ξM,ηN)のマッチングの指標である適合度候補e(ξM,ηN)を求め、動きベクトル候補算出部14および記憶部15に与える。ここで(ξM,ηN)は、間引き部12から与えられる代表1画素についての間引かれた後の座標であることを示すために、ξ、ηの右下にM、Nの参照符を付加している。相互相関関数による方法でも同様である。
Next, the operation of the fitness candidate calculation unit 13 shown in FIG. 1 will be described. Similar to the conventional pattern matching method, the fitness candidate calculation unit 13 uses two representative one-pixel image signals g 0 (x, y) and g 1 (x, x) given from the thinning unit 12 for two input frames. A fitness candidate e (ξ M , η N ), which is an index for matching the movement amount (ξ M , η N ) between y), is obtained and given to the motion vector candidate calculation unit 14 and the storage unit 15. Here, (ξ M , η N ) is a reference mark of M and N at the lower right of ξ and η in order to indicate that it is a coordinate after being thinned out for one representative pixel given from the thinning unit 12. Is added. The same applies to the method using the cross-correlation function.
ここで、1画素同士で適合度候補eを算出した場合は画素マッチングとなり、たとえば水平方向p画素、垂直方向q画素(p,qは整数。p,qのいずれか一方は2以上、他方は1以上)の複数の画素同士で適合度候補eを算出した場合はブロックマッチングとなる。通常、画素マッチングよりもブロックマッチングの方が誤検出を減らすことができるが、画素数であるpおよびqを増やしすぎると、オブジェクトの境界で誤検出が発生しやすくなる弊害が起こる。また画素数が増えると、計算量も増える。
Here, when the fitness candidate e is calculated for each pixel, pixel matching is performed. For example, p pixels in the horizontal direction, q pixels in the vertical direction (p and q are integers, one of p and q is 2 or more, and the other is When the fitness candidate e is calculated between a plurality of pixels (one or more), block matching is performed. Normally, block matching can reduce false detections more than pixel matching, but if p and q, which are the number of pixels, are increased too much, there is a problem that false detections are likely to occur at the boundary of an object. As the number of pixels increases, the amount of calculation also increases.
適合度候補算出部13は、エッジ検出部11から与えられるエッジ成分を用いて適合度候補を算出することもできる。これによって、画像信号とエッジ成分との両方から適合度候補を算出するので、動きベクトル候補の検出精度を向上することができる。たとえば2つの画像信号とエッジ成分
(g0(x,y),E’x0(x,y),E’y0(x,y),E’’x0(x,y),E’’y0(x,y))、
(g1(x,y),E’x1(x,y),E’y1(x,y),E’’x1(x,y),E’’y1(x,y))
の間で以下の式(6)を用いて、移動量(ξM,ηN)のマッチングの指標である適合度候補e(ξM,ηN)を求める。 The fitnesscandidate calculation unit 13 can also calculate fitness candidates using the edge components given from the edge detection unit 11. Thereby, since the fitness candidates are calculated from both the image signal and the edge component, the detection accuracy of the motion vector candidates can be improved. For example, two image signals and edge components (g 0 (x, y), E ′ x0 (x, y), E ′ y0 (x, y), E ″ x0 (x, y), E ″ y0 ( x, y)),
(G 1 (x, y), E ′ x1 (x, y), E ′ y1 (x, y), E ″ x1 (x, y), E ″ y1 (x, y))
Using the following equation (6), a fitness candidate e (ξ M , η N ) that is an index for matching the movement amount (ξ M , η N ) is obtained.
(g0(x,y),E’x0(x,y),E’y0(x,y),E’’x0(x,y),E’’y0(x,y))、
(g1(x,y),E’x1(x,y),E’y1(x,y),E’’x1(x,y),E’’y1(x,y))
の間で以下の式(6)を用いて、移動量(ξM,ηN)のマッチングの指標である適合度候補e(ξM,ηN)を求める。 The fitness
(G 1 (x, y), E ′ x1 (x, y), E ′ y1 (x, y), E ″ x1 (x, y), E ″ y1 (x, y))
Using the following equation (6), a fitness candidate e (ξ M , η N ) that is an index for matching the movement amount (ξ M , η N ) is obtained.
式(6)において、Cmy,Cm1,Cm2は所定の定数である。また(x,y)は、間引き部12から与えられる代表1画素についての間引かれた後の座標である。
In formula (6), C my , C m1 , and C m2 are predetermined constants. Further, (x, y) is the coordinates after thinning out one representative pixel given from the thinning unit 12.
このような構成によれば、エッジ情報も用いて適合度候補を演算することになるので、1画素同士の画素マッチング、あるいはp×qブロックのサイズが小さい場合等の、少ない画素同士のマッチングにおける動きベクトルの検出精度を向上することができる。
According to such a configuration, since the fitness candidates are calculated using the edge information as well, in matching between a small number of pixels, such as pixel matching between one pixel or a size of a p × q block is small. The detection accuracy of the motion vector can be improved.
さらに、適合度候補e(ξM,ηN)にエッジ成分の大きさを示す以下の式(7)に示すFx,Fyを加算し、エッジ成分が大きい場合に適合しやすい計算式とすることもできる。
Further, F x and F y shown in the following formula (7) indicating the size of the edge component are added to the fitness candidate e (ξ M , η N ), and a calculation formula that is easily adapted when the edge component is large: You can also
式(7)において、Cex,Cey,Tex,Teyは所定の定数である。Fx(ξM,ηN),Fy(ξM,ηN)を、いずれも0以下となるようにリミットをかけた後、適合度候補e(ξM,ηN)に加算し、適合度候補の水平成分ex(ξM,ηN)と垂直成分ey(ξM,ηN)とを以下の式(8)に従って算出する。
In Expression (7), C ex , C ey , T ex , and T ey are predetermined constants. F x (ξ M , η N ), F y (ξ M , η N ) are both limited to 0 or less, and then added to the fitness candidate e (ξ M , η N ) The horizontal component e x (ξ M , η N ) and the vertical component e y (ξ M , η N ) of the fitness candidates are calculated according to the following equation (8).
次に、図1に示す動きベクトル候補算出部14の動作について説明する。動きベクトル候補算出部14は、適合度候補算出部13から与えられる適合度候補と、間引き部12から与えられる代表1画素の位置とから、動きベクトル候補を算出して出力する。
Next, the operation of the motion vector candidate calculation unit 14 shown in FIG. 1 will be described. The motion vector candidate calculation unit 14 calculates and outputs a motion vector candidate from the fitness candidate given from the fitness candidate calculation unit 13 and the position of one representative pixel given from the thinning unit 12.
図3は、動きベクトル候補算出部14の構成を示すブロック図である。動きベクトル候補算出部14は、第1入力端子20、第2入力端子21、選択部22、動きベクトル計算部23および出力端子24を備えて構成される。適合度候補算出部13から与えられた適合度候補は、第1入力端子20を介して選択部22に与えられる。選択部22は、適合度候補の中で最も適合度の大きいものを選択し、選択した適合度候補に対応する移動量(ξM,ηN)を出力し、動きベクトル計算部23に与える。最も適合度の大きいものの選択は、パターンマッチング法であれば、適合度候補e(ξM,ηN)が最も大きいものを選択し、相互相関関数による方法であれば、同様に適合度候補h(ξM,ηN)が最も大きいものを選択する。
FIG. 3 is a block diagram illustrating a configuration of the motion vector candidate calculation unit 14. The motion vector candidate calculation unit 14 includes a first input terminal 20, a second input terminal 21, a selection unit 22, a motion vector calculation unit 23, and an output terminal 24. The candidate for suitability given from the suitability candidate calculation unit 13 is given to the selection unit 22 via the first input terminal 20. The selection unit 22 selects a fitness level candidate having the highest fitness level, outputs a movement amount (ξ M , η N ) corresponding to the selected fitness level candidate, and gives it to the motion vector calculation unit 23. In the case of the pattern matching method, the selection of the one with the highest fitness is selected with the highest fitness candidate e (ξ M , η N ). The one with the largest (ξ M , η N ) is selected.
動きベクトル計算部23は、選択部22から与えられる移動量(ξM,ηN)と、第2入力端子21を介して間引き部12から与えられる代表1画素の位置RP(Ox、Oy)とから、動きベクトル候補を算出して出力する。動きベクトル検出装置1への複数の入力フレーム、本実施の形態では2つの入力フレームのうち、第1の入力フレーム(以下、単に「第1フレーム」という場合がある)についての代表1画素の位置を(O1x,O1y)、第2の入力フレーム(以下、単に「第2フレーム」という場合がある)についての代表1画素の位置を(O2x,O2y)とした場合、第1フレームから第2フレームへの動きベクトル候補(ξ,η)を、たとえば以下の式(9)に従って計算する。
The motion vector calculation unit 23 receives the movement amount (ξ M , η N ) given from the selection unit 22 and the position RP (O x , O y) of the representative one pixel given from the thinning unit 12 via the second input terminal 21. ) To calculate and output motion vector candidates. The position of one representative pixel for a first input frame (hereinafter sometimes simply referred to as a “first frame”) among a plurality of input frames to the motion vector detection device 1, in this embodiment, two input frames. Is (O 1x , O 1y ) and the position of one representative pixel for the second input frame (hereinafter sometimes simply referred to as “second frame”) is (O 2x , O 2y ), the first frame The motion vector candidates (ξ, η) from 1 to the second frame are calculated according to the following equation (9), for example.
次に、図1に示す記憶部15の動作について説明する。記憶部15は、動きベクトル候補算出部14から与えられる動きベクトル候補と、間引き部12から与えられる代表1画素の位置と、内挿フレームの位相とから記憶アドレスを算出する。記憶部15は、算出した記憶アドレスに、動きベクトル候補と、適合度候補算出部13から与えられる適合度候補とを記憶する。また記憶部15は、フレーム内の所定の領域の動きベクトル候補と適合度候補とを記憶完了する度に、前記所定の領域の動きベクトル候補および適合度候補を出力する。さらに記憶部15は、動きベクトル決定部16から与えられる動きベクトルおよび適合度も記憶する。
Next, the operation of the storage unit 15 shown in FIG. 1 will be described. The storage unit 15 calculates a storage address from the motion vector candidate given from the motion vector candidate calculation unit 14, the position of one representative pixel given from the thinning unit 12, and the phase of the interpolation frame. The storage unit 15 stores the motion vector candidate and the fitness candidate given from the fitness candidate calculation unit 13 at the calculated storage address. The storage unit 15 outputs the motion vector candidate and the fitness candidate for the predetermined area every time the motion vector candidate and the fitness candidate for the predetermined area in the frame are stored. Furthermore, the storage unit 15 also stores the motion vector and the degree of matching given from the motion vector determination unit 16.
図4は、内挿フレームの位相Fphを説明するための図である。動きベクトル検出装置1に入力された第1フレームF1および第2フレームF2に対して、動きベクトルを求める相対的な時間を、内挿フレームIFの位相Fphとする。第1フレームF1の位相をFph=0、第2フレームF2の位相をFph=1とすると、内挿の場合は0≦Fph≦1、外挿の場合は内挿の場合の範囲外となる。第1フレームF1から第2フレームF2への動きベクトル候補MVC(ξ,η)について、内挿フレームIFでの座標CIを算出し、それに対応した記憶アドレスを算出し、算出した記憶アドレスへ動きベクトル候補と適合度候補とを記憶する。
FIG. 4 is a diagram for explaining the phase Fph of the interpolation frame. A relative time for obtaining a motion vector with respect to the first frame F1 and the second frame F2 input to the motion vector detection device 1 is defined as a phase Fph of the interpolation frame IF. If the phase of the first frame F1 is Fph = 0 and the phase of the second frame F2 is Fph = 1, then 0 ≦ Fph ≦ 1 in the case of interpolation, and out of the range in the case of extrapolation. For the motion vector candidate MVC (ξ, η) from the first frame F1 to the second frame F2, the coordinates CI in the interpolation frame IF are calculated, the corresponding storage address is calculated, and the motion vector is calculated to the calculated storage address. The candidate and the fitness candidate are stored.
図5は、記憶部15の構成を示すブロック図である。図6は、一方のフレーム上の点を固定して他方のフレーム上の点のみを振った場合のペアを説明するための図である。図6(a)は、第1フレーム上の点を固定して第2フレーム上の点のみを振った場合を示す図であり、図6(b)は、第2フレーム上の点を固定して第1フレーム上の点のみを振った場合を示す図である。図7は、点対称型に両フレーム上の点を振った場合のペアを説明するための図である。記憶部15は、第1入力端子30、第2入力端子31、第3入力端子32、第4入力端子33、第5入力端子34、第6入力端子35、記憶アドレス算出部36、候補記憶部37、第1出力端子38および第2出力端子39を備えて構成される。
FIG. 5 is a block diagram showing the configuration of the storage unit 15. FIG. 6 is a diagram for explaining a pair when a point on one frame is fixed and only a point on the other frame is shaken. FIG. 6A is a diagram showing a case where only points on the second frame are shaken while fixing points on the first frame, and FIG. 6B is a diagram showing points fixed on the second frame. It is a figure which shows the case where only the point on the 1st frame is shaken. FIG. 7 is a diagram for explaining a pair when the points on both frames are shaken in a point-symmetric manner. The storage unit 15 includes a first input terminal 30, a second input terminal 31, a third input terminal 32, a fourth input terminal 33, a fifth input terminal 34, a sixth input terminal 35, a storage address calculation unit 36, and a candidate storage unit. 37, a first output terminal 38 and a second output terminal 39.
記憶アドレス算出部36は、第1入力端子30を介して動きベクトル候補算出部14から与えられる動きベクトル候補と、第4入力端子33を介して間引き部12から与えられる代表1画素の位置と、第3入力端子32を介して与えられる内挿フレームIFの位相Fphとから、内挿フレームIFの位相Fphでの動きベクトル候補の座標を算出し、それに対応した記憶アドレスを算出して出力する。
The storage address calculation unit 36 includes a motion vector candidate given from the motion vector candidate calculation unit 14 via the first input terminal 30, a representative one pixel position given from the thinning unit 12 via the fourth input terminal 33, and From the phase Fph of the interpolation frame IF given through the third input terminal 32, the coordinates of the motion vector candidate at the phase Fph of the interpolation frame IF are calculated, and the corresponding storage address is calculated and output.
候補記憶部37は、記憶アドレス算出部36から与えられる記憶アドレスに、第1入力端子30を介して動きベクトル候補算出部14から与えられる動きベクトル候補、および第2入力端子31を介して適合度候補算出部13から与えられる適合度候補を記憶する。また候補記憶部37は、フレーム内の所定の注目領域の動きベクトル候補と適合度候補とを記憶完了する度に、前記所定の注目領域の動きベクトル候補を第1出力端子38から出力し、前記所定の注目領域の適合度候補を第2出力端子39から出力する。さらに候補記憶部37は、第5入力端子34を介して動きベクトル決定部16から与えられる動きベクトル、および第6入力端子35を介して動きベクトル決定部16から与えられる適合度も記憶するが、これについては後述する。
The candidate storage unit 37 has a motion address candidate given from the motion vector candidate calculation unit 14 via the first input terminal 30 and a fitness level via the second input terminal 31 to the storage address given from the storage address calculation unit 36. The fitness candidate given from the candidate calculation unit 13 is stored. The candidate storage unit 37 outputs the motion vector candidate for the predetermined region of interest from the first output terminal 38 every time the motion vector candidate and the fitness candidate for the predetermined region of interest in the frame are stored. A suitability degree candidate for a predetermined attention area is output from the second output terminal 39. Further, the candidate storage unit 37 also stores the motion vector given from the motion vector determination unit 16 via the fifth input terminal 34 and the fitness given from the motion vector determination unit 16 via the sixth input terminal 35. This will be described later.
次に、図5に示す記憶アドレス算出部36の具体的な動作について説明する。記憶アドレス算出部36は、入力される動きベクトル候補と、代表1画素の位置と、内挿フレームの位相とから、内挿フレームでの座標を算出し、それに対応した記憶アドレスを算出して出力する。内挿フレームでの座標(R1x,R1y)、(R2x,R2y)を、たとえば以下の式(10)に従って求める。
Next, a specific operation of the storage address calculation unit 36 shown in FIG. 5 will be described. The storage address calculation unit 36 calculates coordinates in the interpolation frame from the input motion vector candidate, the position of one representative pixel, and the phase of the interpolation frame, and calculates and outputs a storage address corresponding to the coordinates. To do. The coordinates (R 1x , R 1y ) and (R 2x , R 2y ) in the interpolation frame are obtained, for example, according to the following equation (10).
式(10)において、(x,y)は間引き部12から出力される代表1画素の座標であり、(O1x,O1y)は第1フレームF1についての代表1画素の位置であり、(O2x,O2y)は第2フレームF2についての代表1画素の位置であり、Fphは内挿フレームIFの位相であり、(ξ,η)は動きベクトル候補である。
In Expression (10), (x, y) is the coordinates of one representative pixel output from the thinning unit 12, (O 1x , O 1y ) is the position of one representative pixel for the first frame F1, ( O 2x , O 2y ) is the position of one representative pixel for the second frame F2, Fph is the phase of the interpolation frame IF, and (ξ, η) is a motion vector candidate.
内挿フレームでの座標が(R1x,R1y)と(R2x,R2y)との2組あるのは、図6(a)に示すように第1フレーム上の点を固定点として第2フレーム上の点を探索して求めた動きベクトル候補と、図6(b)に示すように第2フレーム上の点を固定点として第1フレーム上の点を探索して求めた動きベクトル候補とが存在するためである。図6には、一方のフレーム上の点を固定して他方のフレーム上の点のみを振って動きベクトルを探索した場合を示しているが、図7に示すような点対称型に両フレーム上の点を振って動きベクトルを探索した場合も移動量が2倍になる分を考慮すれば同様である。
There are two sets of coordinates (R 1x , R 1y ) and (R 2x , R 2y ) in the interpolation frame, as shown in FIG. 6 (a), with the points on the first frame as fixed points. Motion vector candidates obtained by searching for points on two frames, and motion vector candidates obtained by searching for points on the first frame with points on the second frame as fixed points as shown in FIG. This is because there exists. FIG. 6 shows a case in which a point on one frame is fixed and only a point on the other frame is shaken to search for a motion vector. The same applies to the case where the motion vector is searched by shaking this point, considering the amount of movement doubled.
内挿フレームでの座標(R1x,R1y)、(R2x,R2y)から、たとえば以下の式(11)に従って記憶アドレスA1、A2を算出する。
From the coordinates (R 1x , R 1y ) and (R 2x , R 2y ) in the interpolation frame, for example, the storage addresses A 1 and A 2 are calculated according to the following equation (11).
式(11)において、HSZは水平画素数であり、たとえば320である。ここで以下の式(12)のように、注目領域である水平方向M画素、垂直方向N画素からなる領域に含まれる複数の画素のうち、第1および第2フレームの水平方向に対応する画素を代表1画素に間引いて生成される座標を用いて、具体的には、水平方向のM画素から代表1画素に間引いて記憶アドレスA1、A2を算出することもできる。この場合、候補記憶部37の回路規模を小さくすることができる。
In Equation (11), HSZ is the number of horizontal pixels, for example 320. Here, as in the following Expression (12), pixels corresponding to the horizontal direction of the first and second frames among the plurality of pixels included in the region including the horizontal direction M pixels and the vertical direction N pixels as the attention region Specifically, the storage addresses A1 and A2 can be calculated by thinning out the M pixels in the horizontal direction to the representative one pixel using the coordinates generated by thinning out the representative pixel. In this case, the circuit scale of the candidate storage unit 37 can be reduced.
次に、図1に示す動きベクトル決定部16の動作について説明する。図8は、動きベクトル決定部16の構成を示すブロック図である。図9は、動きベクトルを記憶する中心画素CP1を説明するための図である。図10は、適合度を記憶する中心画素CP2を説明するための図である。動きベクトル決定部16は、記憶部15から与えられる動きベクトル候補および適合度候補から、動きベクトルおよび適合度を決定して出力する。動きベクトル決定部16は、第1入力端子40、第2入力端子41、動きベクトル候補補間部42、適合度候補補間部43、第1出力端子44および第2出力端子45を備えて構成される。
Next, the operation of the motion vector determination unit 16 shown in FIG. 1 will be described. FIG. 8 is a block diagram illustrating a configuration of the motion vector determination unit 16. FIG. 9 is a diagram for explaining a central pixel CP1 that stores a motion vector. FIG. 10 is a diagram for explaining the central pixel CP2 that stores the fitness. The motion vector determination unit 16 determines and outputs the motion vector and the fitness from the motion vector candidate and the fitness candidate given from the storage unit 15. The motion vector determination unit 16 includes a first input terminal 40, a second input terminal 41, a motion vector candidate interpolation unit 42, a fitness candidate interpolation unit 43, a first output terminal 44, and a second output terminal 45. .
動きベクトル候補補間部42は、第1入力端子40を介して記憶部15から与えられる動きベクトル候補と、第2入力端子41を介して記憶部15から与えられる適合度候補とから、補間処理によって動きベクトルを決定する。動きベクトル候補補間部42によって決定された動きベクトルは、第1出力端子44から出力される。適合度候補補間部43は、第2入力端子41を介して記憶部15から与えられる適合度候補を補間処理することによって、適合度を決定する。適合度候補補間部43によって決定された適合度は、第2出力端子45から出力される。
The motion vector candidate interpolation unit 42 performs an interpolation process from a motion vector candidate given from the storage unit 15 via the first input terminal 40 and a fitness candidate given from the storage unit 15 via the second input terminal 41. A motion vector is determined. The motion vector determined by the motion vector candidate interpolation unit 42 is output from the first output terminal 44. The goodness-of-fit candidate interpolation unit 43 determines the goodness of fit by interpolating the goodness-of-fit candidates given from the storage unit 15 via the second input terminal 41. The fitness determined by the fitness candidate interpolation unit 43 is output from the second output terminal 45.
次に、図8に示す動きベクトル候補補間部42の具体的な動作について説明する。動きベクトル候補補間部42は、記憶部15から与えられる動きベクトル候補と適合度候補とから、補間処理によって動きベクトルを決定する。具体的には、動きベクトル候補補間部42は、図9に示す水平方向M1(M1は1以上の整数)画素、垂直方向N1(N1は1以上の整数)画素の領域である動きベクトル補間範囲IA1の動きベクトル候補と適合度候補とを用いて、動きベクトル補間範囲IA1の中心画素CP1の動きベクトルを決定する。中心画素CP1の動きベクトルは、たとえば以下の式(13)に従って補間処理をすることによって決定する。
Next, a specific operation of the motion vector candidate interpolation unit 42 shown in FIG. 8 will be described. The motion vector candidate interpolation unit 42 determines a motion vector by interpolation processing from the motion vector candidate and the fitness candidate given from the storage unit 15. Specifically, the motion vector candidate interpolation unit 42 is a motion vector interpolation range which is an area of horizontal M1 (M1 is an integer of 1 or more) pixels and vertical direction N1 (N1 is an integer of 1 or more) pixels shown in FIG. The motion vector of the center pixel CP1 in the motion vector interpolation range IA1 is determined using the motion vector candidate and the fitness candidate of IA1. The motion vector of the center pixel CP1 is determined, for example, by performing an interpolation process according to the following equation (13).
式(13)において、ξi(x,y)は中心画素(x,y)での動きベクトルの水平成分であり、ηi(x,y)は中心画素(x,y)での動きベクトルの垂直成分であり、ξ(u,v)は座標(u,v)での動きベクトル候補の水平成分であり、η(u,v)は座標(u,v)での動きベクトル候補の垂直成分であり、e(u,v)は座標(u,v)での適合度候補である。動きベクトル補間範囲IA1は、水平方向M1画素、垂直方向N1画素の領域である。また、Sは以下の式(14)に示すように、動きベクトル補間範囲IA1である水平方向M1画素、垂直方向N1画素の領域内に存在する適合度候補e(u,v)の和である。
In equation (13), ξ i (x, y) is the horizontal component of the motion vector at the center pixel (x, y), and η i (x, y) is the motion vector at the center pixel (x, y). Ξ (u, v) is the horizontal component of the motion vector candidate at coordinates (u, v), and η (u, v) is the vertical of the motion vector candidate at coordinates (u, v). E (u, v) is a candidate for the degree of fitness at coordinates (u, v). The motion vector interpolation range IA1 is an area of horizontal M1 pixels and vertical N1 pixels. Further, S is the sum of fitness candidates e (u, v) existing in the region of the horizontal direction M1 pixel and the vertical direction N1 pixel which are the motion vector interpolation range IA1, as shown in the following equation (14). .
前述のように動きベクトルを決定することによって、記憶部15に動きベクトル候補が記憶されていない記憶アドレスについて動きベクトルが補間され、また不適切な動きベクトル候補による誤差が補間によって抑えられるので、より精度の高い動きベクトルを決定することが可能である。また、動きベクトル補間範囲IA1である水平方向M1画素、垂直方向N1画素の領域内に存在する適合度候補e(u,v)の数に応じて式(13)の除算が行われるので、水平方向M1画素、垂直方向N1画素の領域内に存在する動きベクトル候補と適合度候補とが少ない場合に、決定する動きベクトルが小さくなることを防ぐことができる。これによって、より精度の高い動きベクトルを決定することができる。
By determining a motion vector as described above, a motion vector is interpolated for a storage address in which no motion vector candidate is stored in the storage unit 15, and an error due to an inappropriate motion vector candidate is suppressed by interpolation. It is possible to determine a motion vector with high accuracy. Further, since the division of Expression (13) is performed according to the number of fitness candidates e (u, v) existing in the region of the horizontal direction M1 pixel and the vertical direction N1 pixel which are the motion vector interpolation range IA1, When there are few motion vector candidates and goodness-of-fit candidates existing in the area of the direction M1 pixel and the vertical direction N1 pixel, it is possible to prevent the determined motion vector from becoming small. Thereby, a motion vector with higher accuracy can be determined.
また、式(8)に示した水平方向および垂直方向の成分を持つ適合度候補を用いて動きベクトルを決定することもできる。このとき、動きベクトル候補の水平成分と適合度候補の水平成分とから動きベクトルの水平成分を算出し、動きベクトル候補の垂直成分と適合度候補の垂直成分とから動きベクトルの垂直成分を算出することによって、動きベクトルを決定する。たとえば、動きベクトルは、適合度候補の水平成分ex(u,v)と適合度候補の垂直成分ey(u,v)とを用いて以下の式(15)に従って決定する。
The motion vector can also be determined using the fitness candidates having the horizontal and vertical components shown in Expression (8). At this time, the horizontal component of the motion vector is calculated from the horizontal component of the motion vector candidate and the horizontal component of the fitness candidate, and the vertical component of the motion vector is calculated from the vertical component of the motion vector candidate and the vertical component of the fitness candidate. Thus, the motion vector is determined. For example, the motion vector is determined according to the following equation (15) using the horizontal component e x (u, v) of the fitness candidate and the vertical component e y (u, v) of the fitness candidate.
式(15)におけるSx、Syは、以下の式(16)によって決定する。
S x and S y in equation (15) are determined by equation (16) below.
以上のようにすることによって、動きベクトルの水平成分は入力フレームのエッジ成分の水平成分で重み付けされ、動きベクトルの垂直成分は入力フレームのエッジ成分の垂直成分で重み付けされるので、水平方向および垂直方向それぞれの輪郭の動きに基づいた、より精度の高い動きベクトルを決定することができる。
By doing so, the horizontal component of the motion vector is weighted by the horizontal component of the edge component of the input frame, and the vertical component of the motion vector is weighted by the vertical component of the edge component of the input frame. A more accurate motion vector can be determined based on the motion of the contour in each direction.
また中心からの位置(u-x,v-y)を用いて、動きベクトルを決定することもできる。たとえば、以下の式(17)に従って動きベクトルを決定する。
Also, the motion vector can be determined using the position (ux, vy) from the center. For example, the motion vector is determined according to the following equation (17).
式(17)におけるrは、以下の式(18)によって決定される。
R in equation (17) is determined by equation (18) below.
以上のことから、中心画素からの距離も動きベクトルに反映されるので、中心画素の近辺を重視した、より精度の高い動きベクトルが得られる。
From the above, since the distance from the center pixel is also reflected in the motion vector, a more accurate motion vector that emphasizes the vicinity of the center pixel can be obtained.
ここで、r=|u-x|+|u-y|とすることもできる。式(17)では、rの逆数を乗算しているが、rが増加するに伴い減少するような他の関数を乗算してもよい。さらに、水平成分、垂直成分ともにrの逆数を乗算する代わりに、水平方向ξiの計算では|u-x|の増加に伴い減少する関数を、垂直方向ηiの計算では|u-y|の増加に伴い減少する関数を個別に乗算してもよい。
Here, r = | u−x | + | u−y |. In equation (17), the inverse of r is multiplied, but other functions that decrease as r increases may be multiplied. Further, instead of multiplying the inverse of the horizontal component, r both the vertical component, in the calculation of the horizontal xi] i | a decreasing function with increasing, in the calculation of the vertical η i | | u-x u -y | A function that decreases with increasing may be multiplied individually.
次に、図8に示す適合度候補補間部43の具体的な動作について説明する。適合度候補補間部43は、記憶部15から与えられる適合度候補を補間処理して適合度を決定する。図10に示す水平方向M2(M2は1以上の整数)画素、垂直方向N2(N2は1以上の整数)画素の領域である適合度補間範囲IA2の適合度候補を用いて、中心画素CP2の適合度を決定する。中心画素CP2の適合度は、たとえば以下の式(19)に従って補間処理をすることによって決定する。
Next, a specific operation of the suitability candidate interpolation unit 43 shown in FIG. 8 will be described. The goodness-of-fit candidate interpolation unit 43 interpolates the goodness-of-fit candidates given from the storage unit 15 and determines the goodness of fit. Using the fitness candidate of the fitness interpolation range IA2 which is an area of the horizontal direction M2 (M2 is an integer of 1 or more) pixels and the vertical direction N2 (N2 is an integer of 1 or more) pixels shown in FIG. Determine the goodness of fit. The fitness of the center pixel CP2 is determined by performing an interpolation process according to the following equation (19), for example.
式(19)において、ei(x,y)は中心画素(x,y)での適合度であり、e(u,v)は座標(u,v)での適合度候補である。また式(19)におけるSeを、水平方向M2画素、垂直方向N2画素の領域のM2×N2の画素数とする。これによって、補間処理は単純平均となるので、適合度補間範囲IA2である水平方向M2画素、垂直方向N2画素の領域内に存在する適合度候補が少ないほど、決定する適合度が小さくなる。したがって、周囲に動きベクトルの決定に用いることのできる動きベクトル候補が少ないときは、その画素の適合度を小さくできるので、より精度の高い適合度を決定することができる。
In equation (19), e i (x, y) is the fitness at the center pixel (x, y), and e (u, v) is a fitness candidate at the coordinates (u, v). Also the S e in equation (19), and the horizontal direction M2 pixels, the number of pixels of the M2 × N2 in the vertical direction N2 pixels in the region. As a result, the interpolation process is a simple average, and the smaller the fitness level candidates that exist in the horizontal direction M2 pixel and vertical direction N2 pixel regions that are the fitness level interpolation range IA2, the smaller the fitness level to be determined. Therefore, when there are few motion vector candidates that can be used for determination of motion vectors in the surrounding area, the degree of matching of the pixel can be reduced, so that the degree of matching with higher accuracy can be determined.
図11は、求めた動きベクトルと補間処理との関係を示す図である。図12は、求めた適合度と補間処理との関係を示す図である。図1に示す動きベクトル検出装置1の記憶部15では、動きベクトル決定部16から与えられる動きベクトルおよび適合度も記憶するように構成されている。図5に示す記憶部15を構成する候補記憶部37は、第5入力端子34を介して動きベクトル決定部16から与えられる動きベクトル、および第6入力端子35を介して動きベクトル決定部16から与えられる適合度を記憶する。このとき、動きベクトルについては、図9に示す中心画素CP1に対応する記憶アドレスへ、記憶されている動きベクトル候補に上書きして記憶する。適合度については、図10に示す中心画素CP2に対応する記憶アドレスへ、記憶されている適合度候補に上書きして記憶する。
FIG. 11 is a diagram showing the relationship between the obtained motion vector and the interpolation process. FIG. 12 is a diagram illustrating the relationship between the obtained fitness and interpolation processing. The storage unit 15 of the motion vector detection device 1 shown in FIG. 1 is configured to also store the motion vector and the fitness given from the motion vector determination unit 16. The candidate storage unit 37 constituting the storage unit 15 shown in FIG. 5 receives the motion vector supplied from the motion vector determination unit 16 via the fifth input terminal 34 and the motion vector determination unit 16 via the sixth input terminal 35. Memorize the goodness of fit given. At this time, the motion vector is stored in the storage address corresponding to the center pixel CP1 shown in FIG. 9 by overwriting the stored motion vector candidate. The fitness is stored by overwriting the stored fitness candidates at the storage address corresponding to the center pixel CP2 shown in FIG.
具体的に述べると、たとえば図11(a)に示すように第1の動きベクトル補間範囲IA1(a)の中心画素CP1(a)に対して求められた動きベクトルは、内挿フレームIFの対応する記憶アドレスに上書きして記憶される。この動きベクトルは、図11(b)に示すように、次の画素の処理において、第2の動きベクトル補間範囲IA1(b)の中心画素CP1(b)の動きベクトルを求めるときに用いられ、さらに図11(c)に示すように、その次の画素の処理において、第3の動きベクトル補間範囲IA1(c)の中心画素CP1(c)の動きベクトルを求めるときに用いられる。第3の動きベクトル補間範囲IA1(c)の中心画素CP1(c)の動きベクトルを求めるときには、第2の動きベクトル補間範囲IA1(b)の中心画素CP1(b)の動きベクトルも用いられる。
Specifically, for example, as shown in FIG. 11 (a), the motion vector obtained for the center pixel CP1 (a) of the first motion vector interpolation range IA1 (a) corresponds to the interpolation frame IF. The stored address is overwritten and stored. This motion vector is used when determining the motion vector of the center pixel CP1 (b) of the second motion vector interpolation range IA1 (b) in the next pixel processing, as shown in FIG. Further, as shown in FIG. 11 (c), it is used when determining the motion vector of the center pixel CP1 (c) of the third motion vector interpolation range IA1 (c) in the processing of the next pixel. When obtaining the motion vector of the center pixel CP1 (c) in the third motion vector interpolation range IA1 (c), the motion vector of the center pixel CP1 (b) in the second motion vector interpolation range IA1 (b) is also used.
同様に、適合度については、たとえば図12(a)に示すように第1の適合度補間範囲IA2(a)の中心画素CP2(a)に対して求められた適合度は、内挿フレームIFの対応する記憶アドレスに上書きして記憶され、図12(b)に示す第2の適合度補間範囲IA2(b)の中心画素CP2(b)の適合度を求めるとき、および図12(c)に示す第3の適合度補間範囲IA2(c)の中心画素CP2(c)の適合度を求めるときに用いられる。また第2の適合度補間範囲IA2(b)の中心画素CP2(b)の適合度は、第3の適合度補間範囲IA2(c)の中心画素CP2(c)の適合度を求めるときに用いられる。
Similarly, for the fitness, for example, as shown in FIG. 12A, the fitness obtained for the center pixel CP2 (a) of the first fitness interpolation range IA2 (a) is the interpolation frame IF. When the degree of fitness of the center pixel CP2 (b) in the second fitness interpolation range IA2 (b) shown in FIG. 12 (b) is obtained, and FIG. 12 (c). Is used when obtaining the fitness of the center pixel CP2 (c) in the third fitness interpolation range IA2 (c) shown in FIG. The fitness of the center pixel CP2 (b) in the second fitness interpolation range IA2 (b) is used when obtaining the fitness of the center pixel CP2 (c) in the third fitness interpolation range IA2 (c). It is done.
このように動きベクトルおよび適合度は書き戻し、すなわち記憶部15に戻されて上書きされるので、図11および図12に示すように、記憶した動きベクトルおよび適合度は、次以降の画素の処理に用いられる。したがって、動きベクトル、適合度の精度が向上する。
Thus, since the motion vector and the fitness are written back, that is, returned to the storage unit 15 and overwritten, as shown in FIGS. 11 and 12, the stored motion vector and the fitness are processed by the subsequent pixels. Used for. Therefore, the accuracy of the motion vector and the fitness is improved.
以上のように本実施の形態によれば、間引き部12で、入力フレームの各座標において、水平方向にM(Mは1以上の整数)画素、垂直方向にN(Nは1以上の整数)画素からなる領域の複数の画素から代表1画素に間引いた後で、適合度候補算出部13によって適合度候補を算出するので、適合度候補を算出するための演算量を削減することが可能となる。これによって、適合度候補を算出するための回路規模または処理時間を小さく抑えることができる。この適合度候補を用いて動きベクトル候補が算出され、動きベクトルが検出されるので、動きベクトルの検出のための回路規模または処理時間を小さく抑えることができる。
As described above, according to the present embodiment, the thinning unit 12 has M (M is an integer of 1 or more) pixels in the horizontal direction and N (N is an integer of 1 or more) in the vertical direction at each coordinate of the input frame. Since the fitness candidate calculation unit 13 calculates fitness candidates after thinning out a plurality of pixels in a pixel area to a representative pixel, it is possible to reduce the amount of calculation for calculating fitness candidates. Become. As a result, the circuit scale or processing time for calculating the fitness candidates can be reduced. Since motion vector candidates are calculated using the fitness candidates and motion vectors are detected, the circuit scale or processing time for motion vector detection can be kept small.
また間引き部12は、エッジ検出部11によって検出される各入力フレームのエッジ成分に基づいて注目領域の複数の画素を代表1画素に間引くので、間引いたことによる動きベクトルの検出精度の低下を抑えることができる。また、記憶部15は、間引いたときの代表1画素の位置情報を用いて、動きベクトル候補の記憶アドレスを算出し、その記憶アドレスに動きベクトル候補を記憶するので、間引きによる位置誤差の発生を抑えることができる。また動きベクトル決定部16は、動きベクトル候補および適合度候補を用いて動きベクトルおよび適合度を決定するので、動きベクトルを決定するときの誤差の発生を抑えることができる。
Further, since the thinning unit 12 thins a plurality of pixels in the attention area into one representative pixel based on the edge component of each input frame detected by the edge detection unit 11, it suppresses a decrease in motion vector detection accuracy due to the thinning. be able to. Further, the storage unit 15 calculates the storage address of the motion vector candidate using the position information of the representative one pixel when the thinning is performed, and stores the motion vector candidate at the storage address. Can be suppressed. Moreover, since the motion vector determination part 16 determines a motion vector and a fitness using a motion vector candidate and a fitness candidate, it can suppress the generation | occurrence | production of the error at the time of determining a motion vector.
したがって、動きベクトルの誤検出を抑えつつ、動きベクトルの検出のための回路規模または処理時間を小さく抑えることができる。
Therefore, it is possible to reduce the circuit scale or processing time for detecting the motion vector while suppressing erroneous detection of the motion vector.
<第2の実施の形態>
次に、本発明の第2の実施の形態の動きベクトル検出装置について説明する。本実施の形態の動きベクトル検出装置は、前述の第1の実施の形態の動きベクトル検出装置1と構成が類似しているので、対応する箇所に同一の参照符を付して、第1の実施の形態と共通する説明を省略する。本実施の形態の動きベクトル検出装置は、図1に示す第1の実施の形態の動きベクトル検出装置1において、記憶部15を、図13に示す記憶部15Aに置き換えた構成である。 <Second Embodiment>
Next, a motion vector detection device according to a second embodiment of the present invention will be described. Since the configuration of the motion vector detection device of the present embodiment is similar to that of the motionvector detection device 1 of the first embodiment described above, the same reference numerals are assigned to corresponding portions, and the first The description common to the embodiment is omitted. The motion vector detection device of the present embodiment has a configuration in which the storage unit 15 is replaced with a storage unit 15A shown in FIG. 13 in the motion vector detection device 1 of the first embodiment shown in FIG.
次に、本発明の第2の実施の形態の動きベクトル検出装置について説明する。本実施の形態の動きベクトル検出装置は、前述の第1の実施の形態の動きベクトル検出装置1と構成が類似しているので、対応する箇所に同一の参照符を付して、第1の実施の形態と共通する説明を省略する。本実施の形態の動きベクトル検出装置は、図1に示す第1の実施の形態の動きベクトル検出装置1において、記憶部15を、図13に示す記憶部15Aに置き換えた構成である。 <Second Embodiment>
Next, a motion vector detection device according to a second embodiment of the present invention will be described. Since the configuration of the motion vector detection device of the present embodiment is similar to that of the motion
図13は、本発明の第2の実施の形態である動きベクトル検出装置の記憶部15Aの構成を示すブロック図である。記憶部15Aは、第1入力端子30、第2入力端子31、第3入力端子32、第4入力端子33、第5入力端子34、記憶アドレス算出部36、候補記憶部50、中心選択部51、適合度閾値記憶部52、適合度比較部53、第1出力端子54および第2出力端子55を備えて構成される。
FIG. 13 is a block diagram showing a configuration of the storage unit 15A of the motion vector detection device according to the second embodiment of the present invention. The storage unit 15A includes a first input terminal 30, a second input terminal 31, a third input terminal 32, a fourth input terminal 33, a fifth input terminal 34, a storage address calculation unit 36, a candidate storage unit 50, and a center selection unit 51. A fitness threshold storage unit 52, a fitness comparison unit 53, a first output terminal 54, and a second output terminal 55.
候補記憶部50は、第1入力端子30を介して動きベクトル候補算出部14から与えられる動きベクトル候補、および第2入力端子31を介して適合度候補算出部13から与えられる適合度候補を記憶する。また候補記憶部50は、フレーム内の所定の注目領域の動きベクトル候補と適合度候補とを記憶完了する度に、前記所定の注目領域の動きベクトル候補を第1出力端子54から出力し、前記所定の注目領域の適合度候補を第2出力端子55から出力する。さらに候補記憶部50は、前記所定の注目領域の適合度候補を中心選択部51に与える。
The candidate storage unit 50 stores motion vector candidates given from the motion vector candidate calculation unit 14 via the first input terminal 30 and fitness candidates given from the fitness degree candidate calculation unit 13 via the second input terminal 31. To do. The candidate storage unit 50 outputs the motion vector candidate for the predetermined region of interest from the first output terminal 54 every time the motion vector candidate and the fitness candidate for the predetermined region of interest in the frame are stored. A suitability degree candidate for a predetermined attention area is output from the second output terminal 55. Further, the candidate storage unit 50 provides the center selection unit 51 with the suitability candidates of the predetermined attention area.
中心選択部51は、候補記憶部50から与えられる適合度候補について、予め定める適合度補間範囲、たとえば図10に示す水平方向M2画素、垂直画素N2の領域の中心に位置する適合度候補を選択して出力する。中心選択部51から出力される前記適合度候補である中心画素適合度候補は、適合度比較部53に与えられる。
The center selection unit 51 selects a fitness candidate located in the center of a predetermined fitness interpolation range, for example, the horizontal M2 pixel and vertical pixel N2 regions shown in FIG. And output. The center pixel suitability candidate that is the suitability candidate output from the center selection unit 51 is given to the suitability comparison unit 53.
適合度閾値記憶部52は、中心選択部51から出力される前記中心画素適合度候補と比較するための予め定める適合度閾値を記憶する。適合度閾値記憶部52は、適合度閾値を適合度比較部53に与える。適合度比較部53は、中心選択部51から与えられる中心画素適合度候補と、適合度閾値記憶部52から与えられる適合度閾値とを比較して、適合度閾値以上の適合度を出力する。具体的に述べると、適合度比較部53は、中心選択部51から与えられる中心画素適合度候補と、適合度閾値記憶部52から与えられる適合度閾値とを比較して、中心画素適合度候補が適合度閾値以上であると判断すると、中心画素適合度候補を適合度として出力し、中心画素適合度候補が適合度閾値未満であると判断すると、適合度閾値を適合度として出力する。適合度比較部53は、出力される前記適合度を候補記憶部50に与える。候補記憶部50は、適合度比較部53から与えられる適合度を、図10に示す水平方向M2画素、垂直画素N2の領域の中心に相当する記憶アドレスへ、記憶されている適合度候補に上書きして記憶する。
The suitability threshold storage unit 52 stores a preset suitability threshold for comparison with the center pixel suitability candidate output from the center selection unit 51. The fitness level threshold value storage unit 52 provides the fitness level threshold value to the fitness level comparison unit 53. The fitness level comparison unit 53 compares the center pixel fitness level candidate given from the center selection unit 51 with the fitness level threshold value given from the fitness level threshold storage unit 52, and outputs a fitness level equal to or higher than the fitness level threshold value. Specifically, the fitness level comparison unit 53 compares the center pixel fitness level candidate given from the center selection unit 51 with the fitness level threshold value given from the fitness level threshold storage unit 52 to obtain a center pixel fitness level candidate. Is determined to be greater than or equal to the fitness threshold, the center pixel fitness candidate is output as the fitness, and if the center pixel fitness candidate is determined to be less than the fitness threshold, the fitness threshold is output as the fitness. The fitness level comparison unit 53 gives the output fitness level to the candidate storage unit 50. The candidate storage unit 50 overwrites the stored fitness candidates to the storage address corresponding to the center of the area of the horizontal M2 pixel and the vertical pixel N2 shown in FIG. 10 with the fitness given from the fitness comparison unit 53. And remember.
このように本実施の形態では、記憶部15Aは、適合度補間範囲IA2である図10に示す水平方向M2画素、垂直方向N2画素の領域の中心に位置する中心画素CP2に対応する記憶アドレスに記憶された中心画素適合度候補と適合度閾値とを比較し、中心画素適合度候補が適合度閾値以上であると、中心画素CP2に対応する記憶アドレスに記憶される適合度候補である中心画素適合度候補を保持し、中心画素適合度候補が適合度閾値未満であると、中心画素適合度候補に代えて、適合度閾値を記憶する。これによって、動きベクトル決定部16において、動きベクトル候補と適合度候補とを用いて動きベクトルを補間するときに、適合度が適合度閾値未満と小さい動きベクトル候補によって誤差が発生することを抑えることができる。また全ての画素に適合度閾値以上の適合度を与えることができる。
As described above, in the present embodiment, the storage unit 15A uses the storage address corresponding to the center pixel CP2 located in the center of the region of the horizontal direction M2 pixel and the vertical direction N2 pixel shown in FIG. The stored center pixel suitability candidate is compared with the suitability threshold, and if the center pixel suitability candidate is equal to or greater than the suitability threshold, the center pixel that is the suitability candidate stored in the storage address corresponding to the center pixel CP2 If the fitness candidate is held and the center pixel fitness candidate is less than the fitness threshold, the fitness threshold is stored instead of the center pixel fitness candidate. Accordingly, when the motion vector determining unit 16 interpolates the motion vector using the motion vector candidate and the fitness candidate, it is possible to suppress the occurrence of an error due to a motion vector candidate whose fitness is less than the fitness threshold. Can do. In addition, it is possible to give a degree of fitness equal to or higher than the threshold of fitness to all pixels.
具体的には適合度閾値記憶部52に記憶する適合度閾値を0ではない最小の値、すなわち適合度候補算出部13によって算出される適合度候補のとり得る値のうち、0以外の値であって、かつ最小の値とすることによって、適合度の大きい適合度候補はそのままの値が残り、適合度の小さい適合度候補は、適合度閾値に置き換わる。これによって、適合度の小さい動きベクトル候補が、動きベクトル決定部16での補間のときに発生させる誤差を抑えることを可能にしつつ、全ての画素に適合度閾値以上の適合度を与えることができる。
Specifically, the fitness threshold stored in the fitness threshold storage unit 52 is a non-zero minimum value, that is, a value other than 0 among possible values of the fitness candidate calculated by the fitness candidate calculation unit 13. In addition, by setting the value to the minimum value, the goodness-of-fit candidate with a high goodness of fit remains as it is, and the goodness-of-fit candidate with a low goodness of fit is replaced with a goodness-of-fit threshold. As a result, it is possible to give a degree of fitness that is equal to or higher than the fitness threshold value to all the pixels while enabling the motion vector candidate having a low fitness to suppress an error that occurs during interpolation by the motion vector determination unit 16. .
<第3の実施の形態>
次に、本発明の第3の実施の形態の画像処理装置について説明する。図14は、本発明の第3の実施の形態である画像処理装置60の構成を示すブロック図である。画像処理装置60は、入力端子61、動きベクトル検出装置62、フレームレート変換部63および出力端子64を備えて構成される。本実施の形態の動きベクトル検出装置62は、前述の第1の実施の形態または第2の実施の形態の動きベクトル検出装置である。 <Third Embodiment>
Next, an image processing apparatus according to a third embodiment of the present invention will be described. FIG. 14 is a block diagram showing a configuration of animage processing apparatus 60 according to the third embodiment of the present invention. The image processing device 60 includes an input terminal 61, a motion vector detection device 62, a frame rate conversion unit 63, and an output terminal 64. The motion vector detection device 62 according to the present embodiment is the motion vector detection device according to the first embodiment or the second embodiment described above.
次に、本発明の第3の実施の形態の画像処理装置について説明する。図14は、本発明の第3の実施の形態である画像処理装置60の構成を示すブロック図である。画像処理装置60は、入力端子61、動きベクトル検出装置62、フレームレート変換部63および出力端子64を備えて構成される。本実施の形態の動きベクトル検出装置62は、前述の第1の実施の形態または第2の実施の形態の動きベクトル検出装置である。 <Third Embodiment>
Next, an image processing apparatus according to a third embodiment of the present invention will be described. FIG. 14 is a block diagram showing a configuration of an
動きベクトル検出装置62およびフレームレート変換部63には、入力端子61を介して複数の入力フレームが入力される。動きベクトル検出装置62は、前述の第1および第2の実施の形態に示したように、複数の入力フレームから動きベクトルと適合度とを生成して出力する。動きベクトル検出装置62から出力される動きベクトルおよび適合度は、フレームレート変換部63に与えられる。
A plurality of input frames are input to the motion vector detection device 62 and the frame rate conversion unit 63 via the input terminal 61. As described in the first and second embodiments, the motion vector detection device 62 generates and outputs a motion vector and fitness from a plurality of input frames. The motion vector and the fitness level output from the motion vector detection device 62 are given to the frame rate conversion unit 63.
フレームレート変換部63は、動きベクトル検出装置62から与えられる動きベクトルおよび適合度に基づいて、複数の入力フレーム、たとえば2つの入力フレームに含まれる画素から、前記2つの入力フレーム間に内挿する内挿フレームの画素を生成して出力する。これによって、フレームレート変換部63は、画像のフレームレートを変換する。フレームレート変換部63から出力される前記内挿フレームの画素は、出力端子64から出力される。フレームレート変換部63は、フレームレート変換手段に相当する。
The frame rate conversion unit 63 interpolates between the two input frames from a plurality of input frames, for example, pixels included in the two input frames, based on the motion vector and the fitness given from the motion vector detecting device 62. Generate and output a pixel of the interpolation frame. As a result, the frame rate conversion unit 63 converts the frame rate of the image. The pixel of the interpolation frame output from the frame rate conversion unit 63 is output from the output terminal 64. The frame rate conversion unit 63 corresponds to a frame rate conversion unit.
図15は、フレームレート変換部63の構成を示すブロック図である。フレームレート変換部63は、第1入力端子70、第2入力端子71、第3入力端子72、遅延部73、参照座標生成部74、フレーム間補間部75および出力端子76を備えて構成される。
FIG. 15 is a block diagram showing a configuration of the frame rate conversion unit 63. The frame rate conversion unit 63 includes a first input terminal 70, a second input terminal 71, a third input terminal 72, a delay unit 73, a reference coordinate generation unit 74, an interframe interpolation unit 75, and an output terminal 76. .
遅延部73には、第1入力端子70を介して複数の入力フレームが与えられる。参照座標生成部74には、動きベクトル検出装置62から、第2入力端子71を介して動きベクトルが与えられ、第3入力端子72を介して適合度が与えられる。
A plurality of input frames are given to the delay unit 73 via the first input terminal 70. The reference coordinate generation unit 74 is given a motion vector via the second input terminal 71 from the motion vector detection device 62 and given a fitness level via the third input terminal 72.
参照座標生成部74は、第2入力端子71を介して動きベクトル検出装置62から与えられる動きベクトルと、第3入力端子72を介して動きベクトル検出装置62から与えられる適合度とから、参照座標を生成して出力する。参照座標生成部74から出力される参照座標は、遅延部73に与えられる。
The reference coordinate generation unit 74 calculates a reference coordinate from the motion vector given from the motion vector detection device 62 via the second input terminal 71 and the fitness given from the motion vector detection device 62 via the third input terminal 72. Is generated and output. The reference coordinates output from the reference coordinate generation unit 74 are given to the delay unit 73.
遅延部73は、第1入力端子70を介して与えられる複数の入力フレームを保持し、参照座標生成部74から与えられる参照座標、具体的には第1フレームおよび第2フレームの参照座標に対応する画素を出力する。遅延部73から出力される前記第1フレームおよび第2フレームの参照座標に対応する画素は、フレーム間補間部75に与えられる。
The delay unit 73 holds a plurality of input frames given via the first input terminal 70, and corresponds to the reference coordinates given from the reference coordinate generation unit 74, specifically the reference coordinates of the first frame and the second frame. The pixel to be output is output. Pixels corresponding to the reference coordinates of the first frame and the second frame output from the delay unit 73 are provided to the inter-frame interpolation unit 75.
フレーム間補間部75は、遅延部73から与えられる前記第1フレームおよび第2フレームの参照座標に対応する画素から、内挿フレームの画素を生成して出力する。フレーム間補間部75から出力される前記内挿フレームの画素は、出力端子76から出力される。
The inter-frame interpolation unit 75 generates and outputs a pixel of an interpolation frame from the pixels corresponding to the reference coordinates of the first frame and the second frame given from the delay unit 73. The pixels of the interpolation frame output from the inter-frame interpolation unit 75 are output from the output terminal 76.
次に、図15に示す参照座標生成部74の具体的な動作について説明する。参照座標生成部74は、第2入力端子71を介して動きベクトル検出装置62から与えられる動きベクトル、および第3入力端子72を介して動きベクトル検出装置62から与えられる適合度に基づいて、参照座標を算出して出力する。具体的には、適合度を、予め定める参照座標算出用の適合度閾値(以下「参照座標用適合度閾値」という)と比較する。比較した結果、適合度が参照座標用適合度閾値よりも大きければ、動きベクトルを用いてフレーム内挿を実施できると判断し、動きベクトルに基づいて第1フレームおよび第2フレームの参照座標を計算して出力する。適合度が参照座標用適合度閾値と等しいか、または参照座標用適合度閾値に比べて小さい場合は、動きベクトルの使用は不可能と判断し、動きベクトルを(0,0)とみなして、第1フレームおよび第2フレームの参照座標を出力する。
Next, a specific operation of the reference coordinate generation unit 74 shown in FIG. 15 will be described. The reference coordinate generation unit 74 refers to the motion vector given from the motion vector detection device 62 via the second input terminal 71 and the fitness given from the motion vector detection device 62 via the third input terminal 72. Calculate and output coordinates. Specifically, the fitness is compared with a predetermined fitness threshold for calculating reference coordinates (hereinafter referred to as “reference coordinate fitness threshold”). As a result of the comparison, if the fitness is larger than the fitness threshold for reference coordinates, it is determined that the frame interpolation can be performed using the motion vector, and the reference coordinates of the first frame and the second frame are calculated based on the motion vector. And output. When the fitness is equal to the reference coordinate fitness threshold or smaller than the reference coordinate fitness threshold, it is determined that the motion vector cannot be used, and the motion vector is regarded as (0, 0). The reference coordinates of the first frame and the second frame are output.
次に、動きベクトルを用いてフレーム内挿を実施できると判断した場合の参照座標の計算について説明する。図16は、参照座標を説明するための図である。図16において、内挿フレームIFに生成する画素(Qx,Qy)の位置GPを通る動きベクトルと、第1フレームF1とが交差する点を第1フレームF1の参照座標RC1(P1x,P1y)とし、前記位置GPを通る動きベクトルと第2フレームF2とが交差する点を第2フレームF2の参照座標RC2(P2x,P2y)とする。前記参照座標は、たとえば以下の式(20)に従って生成することができる。
Next, calculation of reference coordinates when it is determined that frame interpolation can be performed using a motion vector will be described. FIG. 16 is a diagram for explaining the reference coordinates. In FIG. 16, the point at which the motion vector passing through the position GP of the pixel (Q x , Q y ) generated in the interpolation frame IF and the first frame F1 intersect is the reference coordinates RC1 (P 1x , P 1x , P 1y ), and a point where the motion vector passing through the position GP and the second frame F2 intersect is a reference coordinate RC2 (P 2x , P 2y ) of the second frame F2. The reference coordinates can be generated according to the following equation (20), for example.
式(20)において、(ξ,η)は動きベクトルであり、Fphは内挿フレームIFの位相である。
In Equation (20), (ξ, η) is a motion vector, and Fph is the phase of the interpolation frame IF.
動きベクトルの使用が不可能と判断した場合は、以下の式(21)に示す参照座標を出力する。
When it is determined that the motion vector cannot be used, the reference coordinates shown in the following formula (21) are output.
次に、図15に示すフレーム間補間部75の具体的な動作について説明する。図17は、内挿フレームIFの画素の生成を説明するための図である。フレーム間補間部75は、遅延部73から与えられる画素から、図17に示すように内挿フレームIFの画素を生成する。内挿フレームの画素は、たとえば以下の式(22)に従って生成する。
Next, a specific operation of the interframe interpolation unit 75 shown in FIG. 15 will be described. FIG. 17 is a diagram for explaining generation of pixels of the interpolation frame IF. The inter-frame interpolation unit 75 generates pixels of the interpolation frame IF from the pixels given from the delay unit 73 as shown in FIG. The pixels of the interpolation frame are generated, for example, according to the following equation (22).
式(22)において、Y1は遅延部73から与えられる第1フレームF1の画素の画像情報であり、Y2は遅延部73から与えられる第2フレームF2の画素の画素情報であり、Y0は内挿フレームIFの生成する画素の画像情報であり、Fphは内挿フレームIFの位相である。画素情報としては、たとえば輝度と色差とを用いて、それぞれについて式(22)に従って内挿フレームIFの画素を生成する。これによって複数のフレーム間から複数の内挿フレームを生成できるので、フレームレート変換を行うことができる。
In Expression (22), Y 1 is the image information of the pixel of the first frame F1 given from the delay unit 73, Y 2 is the pixel information of the pixel of the second frame F2 given from the delay unit 73, and Y 0 Is the pixel image information generated by the interpolation frame IF, and Fph is the phase of the interpolation frame IF. As the pixel information, for example, using the luminance and the color difference, the pixel of the interpolation frame IF is generated according to the equation (22) for each. As a result, a plurality of interpolation frames can be generated from a plurality of frames, so that frame rate conversion can be performed.
以上のように本実施の形態によれば、画像処理装置60は、前述のような優れた動きベクトル検出装置62とフレームレート変換部63とを備えて構成される。動きベクトル検出装置62は、動きベクトルの誤検出を抑えつつ、動きベクトルの検出のための回路規模または処理時間を小さく抑えることができるので、内挿フレームに含まれる画素を精度良く生成することができる。したがって、画質の低下を抑えて、フレームレートを変換することのできる画像処理装置60が実現される。
As described above, according to the present embodiment, the image processing device 60 includes the excellent motion vector detection device 62 and the frame rate conversion unit 63 as described above. The motion vector detection device 62 can suppress the erroneous detection of the motion vector and reduce the circuit scale or processing time for detecting the motion vector, so that the pixels included in the interpolation frame can be generated with high accuracy. it can. Therefore, the image processing apparatus 60 that can convert the frame rate while suppressing the deterioration of the image quality is realized.
<第4の実施の形態>
次に、本発明の第4の実施の形態の画像処理装置について説明する。本実施の形態の画像処理装置は、前述の第3の実施の形態の画像処理装置60と構成が類似しているので、対応する箇所に同一の参照符を付して、第3の実施の形態と共通する説明を省略する。 <Fourth embodiment>
Next, an image processing apparatus according to a fourth embodiment of the present invention will be described. Since the image processing apparatus according to the present embodiment is similar in configuration to theimage processing apparatus 60 according to the third embodiment described above, the same reference numerals are assigned to corresponding portions, and the third embodiment is described. The description common to the form is omitted.
次に、本発明の第4の実施の形態の画像処理装置について説明する。本実施の形態の画像処理装置は、前述の第3の実施の形態の画像処理装置60と構成が類似しているので、対応する箇所に同一の参照符を付して、第3の実施の形態と共通する説明を省略する。 <Fourth embodiment>
Next, an image processing apparatus according to a fourth embodiment of the present invention will be described. Since the image processing apparatus according to the present embodiment is similar in configuration to the
図18は、本発明の第4の実施の形態である画像処理装置80の構成を示すブロック図である。画像処理装置80は、入力端子81、動きベクトル検出装置62、解像度変換部82および出力端子83を備えて構成される。動きベクトル検出装置62は、前述の第3の実施の形態の画像処理装置60と同様に、前述の第1または第2の実施の形態の動きベクトル検出装置である。
FIG. 18 is a block diagram showing a configuration of an image processing apparatus 80 according to the fourth embodiment of the present invention. The image processing device 80 includes an input terminal 81, a motion vector detection device 62, a resolution conversion unit 82, and an output terminal 83. The motion vector detection device 62 is the motion vector detection device of the first or second embodiment described above, similarly to the image processing device 60 of the third embodiment described above.
解像度変換時の補間方法として、周囲の4個の画素から補間を行う線形補間法、および周囲の16個の画素から補間を行うバイキュービック法が知られている。これらは、1つのフレーム内で周囲の点から補間処理を行う。本実施の形態の画像処理装置80では、2つのフレームから動きベクトルを検出し、2つのフレーム間で対応する複数の画素を見つけて、前記複数の画素から補間処理によって解像度の異なる画素を算出する。2つのフレーム間で対応する複数の画素の数は、たとえば線形補間法のように4個でもよいし、バイキュービック法のように16個でもよい。
As an interpolation method at the time of resolution conversion, a linear interpolation method that performs interpolation from four surrounding pixels and a bicubic method that performs interpolation from 16 surrounding pixels are known. These perform interpolation processing from surrounding points within one frame. The image processing apparatus 80 according to the present embodiment detects motion vectors from two frames, finds a plurality of corresponding pixels between the two frames, and calculates pixels having different resolutions from the plurality of pixels by interpolation processing. . The number of pixels corresponding between two frames may be four as in the linear interpolation method, or may be 16 as in the bicubic method.
動きベクトル検出装置62および解像度変換部82には、図18に示すように、入力端子81から複数の入力フレームが与えられる。動きベクトル検出装置62は、前述の第1および第2の実施の形態に示したように、複数の入力フレームから動きベクトルおよび適合度を生成して出力する。動きベクトル検出装置62から出力される動きベクトルおよび適合度は、解像度変換部82に与えられる。
As shown in FIG. 18, the motion vector detecting device 62 and the resolution converting unit 82 are provided with a plurality of input frames from the input terminal 81. As described in the first and second embodiments, the motion vector detection device 62 generates and outputs a motion vector and a fitness degree from a plurality of input frames. The motion vector and the fitness level output from the motion vector detection device 62 are given to the resolution conversion unit 82.
解像度変換部82は、動きベクトル検出装置62から与えられる動きベクトルおよび適合度に基づいて、複数の入力フレーム、たとえば2つの入力フレームに含まれる画素から解像度変換した新たな画素を生成して出力する。これによって、前記複数の入力フレームで構成される画像とは異なる解像度を有する新たな画像が生成される。解像度変換部82から出力される前記解像度変換した画素は、出力端子83から出力される。解像度変換部82は、解像度変換手段に相当する。
The resolution conversion unit 82 generates and outputs a new pixel whose resolution is converted from a plurality of input frames, for example, pixels included in two input frames, based on the motion vector and the degree of matching given from the motion vector detection device 62. . As a result, a new image having a resolution different from that of the image composed of the plurality of input frames is generated. The resolution-converted pixels output from the resolution converter 82 are output from the output terminal 83. The resolution conversion unit 82 corresponds to a resolution conversion unit.
図19は、解像度変換部82の動作を説明するための図である。図19(a)は、解像度変換前のグリッドを示す図であり、図19(b)は、解像度変換後のグリッドを示す図である。図19(a)には、9個のグリッドを示しており、図19(b)には、16個のグリッドを示している。図20は、解像度変換部82の動作を説明するための図である。図19に示すように、入力フレームを4/3倍に拡大する場合を例にとり説明する。解像度変換部82は、動きベクトル検出装置62から与えられる動きベクトルのうち、図20に示すように、移動量が似ている4個の隣接した動きベクトルを探し、第1フレームF1と第2フレームF2との間で対応する4個の画素の組を見つける。このとき、予め定める動きベクトル探索用の適合度閾値である探索用適合度閾値以下の適合度を持つ動きベクトルは、探す対象から外す。次に、見つけた4個の画素の組を、第1フレームF1および第2フレームF2のそれぞれで補間し、第1フレームF1の補間画素IP1および第2フレームF2の補間画素IP2を算出する。さらに解像度変換部82は、前記第1フレームF1の補間画素IP1と前記第2フレームF2の補間画素IP2とから、補間処理によって解像度変換した画素を算出して出力する。移動量が似ている4個の隣接した動きベクトルが見つからない画素については、第2フレームF2において周囲の4個の画素から補間処理を行い、解像度変換した画素を算出して出力する。
FIG. 19 is a diagram for explaining the operation of the resolution conversion unit 82. FIG. 19A is a diagram showing a grid before resolution conversion, and FIG. 19B is a diagram showing a grid after resolution conversion. FIG. 19A shows nine grids, and FIG. 19B shows 16 grids. FIG. 20 is a diagram for explaining the operation of the resolution conversion unit 82. A case where the input frame is enlarged 4/3 times as shown in FIG. 19 will be described as an example. As shown in FIG. 20, the resolution conversion unit 82 searches for four adjacent motion vectors having similar movement amounts from among the motion vectors supplied from the motion vector detection device 62, and the first frame F1 and the second frame. Find a set of 4 pixels corresponding to F2. At this time, a motion vector having a fitness less than or equal to a search fitness threshold that is a predetermined motion vector search fitness threshold is excluded from the search target. Next, the found set of four pixels is interpolated in each of the first frame F1 and the second frame F2, and the interpolation pixel IP1 of the first frame F1 and the interpolation pixel IP2 of the second frame F2 are calculated. Further, the resolution conversion unit 82 calculates and outputs a pixel whose resolution has been converted by interpolation processing from the interpolation pixel IP1 of the first frame F1 and the interpolation pixel IP2 of the second frame F2. For the pixels where four adjacent motion vectors having similar movement amounts are not found, interpolation processing is performed from the surrounding four pixels in the second frame F2, and the resolution-converted pixels are calculated and output.
このようにして異なる解像度の画素を得ることによって、解像度変換を行うことができる。本実施の形態では、第1フレームF1と第2フレームF2の対応する4個の画素を用いて解像度変換を行う場合について述べたが、対応する画素の数は4個に限らず、たとえば16個でもよい。
In this way, resolution conversion can be performed by obtaining pixels with different resolutions. In the present embodiment, the case of performing resolution conversion using four corresponding pixels of the first frame F1 and the second frame F2 has been described. However, the number of corresponding pixels is not limited to four, for example, sixteen. But you can.
以上のように本実施の形態によれば、画像処理装置80は、前述のような優れた動きベクトル検出装置62と解像度変換部83とを備えて構成される。動きベクトル検出装置62は、動きベクトルの誤検出を抑えつつ、動きベクトルの検出のための回路規模または処理時間を小さく抑えることができるので、画質の低下を抑えて、解像度の異なる新たな画像を生成することのできる画像処理装置80が実現される。
As described above, according to the present embodiment, the image processing apparatus 80 includes the excellent motion vector detection apparatus 62 and the resolution conversion unit 83 as described above. The motion vector detection device 62 can suppress the detection of the motion vector and reduce the circuit scale or processing time for detecting the motion vector. An image processing apparatus 80 that can be generated is realized.
この発明は詳細に説明されたが、上記した説明は、すべての局面において、例示であって、この発明がそれに限定されるものではない。例示されていない無数の変形例が、この発明の範囲から外れることなく想定され得るものと解される。
Although the present invention has been described in detail, the above description is illustrative in all aspects, and the present invention is not limited thereto. It is understood that countless variations that are not illustrated can be envisaged without departing from the scope of the present invention.
Claims (15)
- 第1および第2の入力フレーム間で、前記第1および第2の入力フレームにそれぞれ含まれる1個以上の画素同士を対応付ける動きベクトルを検出する動きベクトル検出装置(1,62)であって、
各前記入力フレームのエッジ成分を検出するエッジ検出手段(11)と、
各前記入力フレームの予め定める注目領域に含まれる複数の画素を、前記エッジ検出手段(11)によって検出されるエッジ成分に基づいて、代表となる1個の画素に間引いて、この画素を代表1画素として出力する間引き手段(12)と、
前記間引き手段(12)から出力される前記第1および第2の入力フレームの前記代表1画素を用いて、前記第1および第2の入力フレーム間の移動量の適合度合いを表す適合度の候補となる適合度候補を算出する適合度候補算出手段(13)と、
前記間引き手段(12)から出力される前記代表1画素の各入力フレームにおける位置、および前記適合度候補算出手段(13)によって算出される前記適合度候補に基づいて、前記第1および第2の入力フレーム間の前記動きベクトルの候補となる動きベクトル候補を算出する動きベクトル候補算出手段(14)と、
前記適合度候補算出手段(13)によって算出される前記適合度候補、および前記動きベクトル候補算出手段(14)によって算出される前記動きベクトル候補を記憶する記憶手段(15)と、
前記記憶手段(15)に記憶される前記適合度候補および前記動きベクトル候補に基づいて、前記適合度および前記動きベクトルを決定する動きベクトル決定手段(16)とを備え、
前記記憶手段(15)は、
前記間引き手段(12)から出力される前記代表1画素の各入力フレームにおける位置に基づいて、前記動きベクトル候補および前記適合度候補を記憶する記憶アドレスを算出し、算出した前記記憶アドレスに前記動きベクトル候補および前記適合度候補を記憶することを特徴とする動きベクトル検出装置。 A motion vector detection device (1, 62) for detecting a motion vector that associates one or more pixels included in each of the first and second input frames between the first and second input frames,
Edge detection means (11) for detecting an edge component of each input frame;
Based on the edge component detected by the edge detection means (11), a plurality of pixels included in a predetermined attention area of each input frame are thinned out to one representative pixel, and this pixel is represented as a representative 1 Thinning means (12) for outputting as pixels;
Using the representative one pixel of the first and second input frames output from the thinning-out means (12), candidates for the degree of adaptation representing the degree of adaptation of the movement amount between the first and second input frames Fitness level candidate calculation means (13) for calculating fitness level candidates to be
Based on the position of the representative one pixel output from the thinning-out means (12) in each input frame and the fitness candidate calculated by the fitness candidate calculation means (13), the first and second Motion vector candidate calculating means (14) for calculating a motion vector candidate that is a candidate for the motion vector between input frames;
Storage means (15) for storing the fitness candidate calculated by the fitness candidate calculation means (13) and the motion vector candidate calculated by the motion vector candidate calculation means (14);
Motion vector determining means (16) for determining the fitness and the motion vector based on the fitness candidate and the motion vector candidate stored in the storage means (15),
The storage means (15)
Based on the position of the representative pixel output from the thinning means (12) in each input frame, a storage address for storing the motion vector candidate and the fitness candidate is calculated, and the motion is stored in the calculated storage address. A motion vector detecting apparatus for storing a vector candidate and the fitness candidate. - 前記適合度候補算出手段(13)によって算出される前記適合度候補は、各前記入力フレームの水平方向に対応する水平成分と、各前記入力フレームの垂直方向に対応する垂直成分とを含み、
前記動きベクトル候補算出手段(14)によって算出される前記動きベクトル候補は、各前記入力フレームの水平方向に対応する水平成分と、各前記入力フレームの垂直方向に対応する垂直成分とを含むことを特徴とする請求項1に記載の動きベクトル検出装置。 The fitness candidate calculated by the fitness candidate calculation means (13) includes a horizontal component corresponding to the horizontal direction of each input frame and a vertical component corresponding to the vertical direction of each input frame;
The motion vector candidates calculated by the motion vector candidate calculation means (14) include a horizontal component corresponding to the horizontal direction of each input frame and a vertical component corresponding to the vertical direction of each input frame. The motion vector detection device according to claim 1, wherein - 前記動きベクトル決定手段(16)は、前記適合度候補の水平成分および前記動きベクトルの水平成分に基づいて、前記動きベクトルの水平成分を算出し、前記適合度候補の垂直成分および前記動きベクトルの垂直成分に基づいて、前記動きベクトルの垂直成分を算出することを特徴とする請求項2に記載の動きベクトル検出装置。 The motion vector determination means (16) calculates the horizontal component of the motion vector based on the horizontal component of the fitness candidate and the horizontal component of the motion vector, and calculates the vertical component of the fitness candidate and the motion vector. The motion vector detection device according to claim 2, wherein a vertical component of the motion vector is calculated based on a vertical component.
- 前記動きベクトル決定手段(16)は、前記記憶手段(15)に記憶される動きベクトル候補および適合度候補のうち、予め定める動きベクトル補間範囲に含まれる画素の動きベクトル候補および適合度候補を用いて、前記動きベクトル補間範囲の中心に位置する中心画素の動きベクトルを算出することを特徴とする請求項1~3のいずれか1つに記載の動きベクトル検出装置。 The motion vector determination means (16) uses a motion vector candidate and a fitness candidate for pixels included in a predetermined motion vector interpolation range among the motion vector candidates and the fitness candidates stored in the storage means (15). The motion vector detection device according to any one of claims 1 to 3, wherein a motion vector of a central pixel located at the center of the motion vector interpolation range is calculated.
- 前記動きベクトル決定手段(16)は、前記動きベクトル補間範囲に含まれる画素のうち、動きベクトル候補および適合度候補が記憶されている画素を用いて、その画素の動きベクトル候補および適合度候補をそれぞれ、その画素の個数で平均化することによって、前記中心画素の動きベクトルを算出することを特徴とする請求項4に記載の動きベクトル検出装置。 The motion vector determination means (16) uses a pixel in which a motion vector candidate and a fitness candidate are stored among the pixels included in the motion vector interpolation range, and determines a motion vector candidate and a fitness candidate of the pixel. The motion vector detection device according to claim 4, wherein the motion vector of the central pixel is calculated by averaging the number of pixels.
- 前記記憶手段(15)は、前記動きベクトル決定手段(16)によって前記中心画素の動きベクトルが算出されると、前記中心画素に対応する記憶アドレスに記憶される動きベクトル候補に代えて、前記動きベクトル決定手段(16)によって算出された前記中心画素の動きベクトルを記憶することを特徴とする請求項4に記載の動きベクトル検出装置。 When the motion vector determination unit (16) calculates the motion vector of the central pixel, the storage unit (15) replaces the motion vector candidate stored in the storage address corresponding to the central pixel with the motion vector. 5. The motion vector detecting device according to claim 4, wherein the motion vector of the central pixel calculated by the vector determining means (16) is stored.
- 前記動きベクトル決定手段(16)は、前記記憶手段(15)に記憶される動きベクトル候補および適合度候補のうち、予め定める動きベクトル補間範囲に含まれる画素の動きベクトル候補および適合度候補に、前記動きベクトル補間範囲の中心からの距離に応じた重みを付けることによって、前記動きベクトルを算出することを特徴とする請求項1~3のいずれか1つに記載の動きベクトル検出装置。 The motion vector determination unit (16) includes, among the motion vector candidates and fitness candidates stored in the storage unit (15), pixel motion vector candidates and fitness candidates included in a predetermined motion vector interpolation range. The motion vector detection device according to any one of claims 1 to 3, wherein the motion vector is calculated by applying a weight according to a distance from a center of the motion vector interpolation range.
- 前記動きベクトル決定手段(16)は、前記記憶手段(15)に記憶される適合度候補のうち、予め定める適合度補間範囲に含まれる画素の適合度候補を用いて、前記適合度補間範囲の中心に位置する中心画素の適合度を算出することを特徴とする請求項1に記載の動きベクトル検出装置。 The motion vector determination means (16) uses the fitness degree candidates of the pixels included in a predetermined fitness degree interpolation range among the fitness degree candidates stored in the storage means (15), and uses the fitness degree interpolation range. The motion vector detection device according to claim 1, wherein the fitness of a central pixel located at the center is calculated.
- 前記動きベクトル決定手段(16)は、前記適合度補間範囲に含まれる画素の適合度候補を単純平均することによって、前記中心画素の適合度を算出することを特徴とする請求項8に記載の動きベクトル検出装置。 The said motion vector determination means (16) calculates the fitness of the said center pixel by carrying out the simple average of the fitness candidates of the pixel included in the said fitness interpolation range. Motion vector detection device.
- 前記記憶手段(15)は、前記動きベクトル決定手段(16)によって前記中心画素の適合度が算出されると、前記中心画素に対応する記憶アドレスに記憶される適合度候補に代えて、前記動きベクトル決定手段(16)によって算出された前記中心画素の適合度を記憶することを特徴とする請求項8または9に記載の動きベクトル検出装置。 When the fitness of the center pixel is calculated by the motion vector determination unit (16), the storage means (15) replaces the fitness candidate stored in the storage address corresponding to the center pixel with the motion. The motion vector detection device according to claim 8 or 9, characterized in that the fitness of the central pixel calculated by the vector determination means (16) is stored.
- 前記記憶手段(15)は、前記記憶手段(15)に記憶される適合度候補のうち、予め定める適合度補間範囲の中心に位置する中心画素に対応する記憶アドレスに記憶された中心画素適合度候補と、予め定める適合度閾値とを比較し、前記中心画素適合度候補が前記適合度閾値以上であると、前記中心画素に対応する記憶アドレスに記憶される適合度候補を保持し、前記中心画素適合度候補が前記適合度閾値未満であると、前記中心画素適合度候補に代えて、前記適合度閾値を記憶することを特徴とする請求項1に記載の動きベクトル検出装置。 The storage means (15) is a central pixel fitness stored in a storage address corresponding to a central pixel located at the center of a predetermined fitness interpolation range among the fitness candidates stored in the storage means (15). The candidate is compared with a predetermined fitness threshold, and when the center pixel fitness candidate is equal to or higher than the fitness threshold, the fitness candidate stored in the storage address corresponding to the center pixel is held, and the center The motion vector detection device according to claim 1, wherein when the pixel suitability candidate is less than the suitability threshold, the fitness threshold is stored instead of the center pixel suitability candidate.
- 前記適合度閾値は、前記適合度候補算出手段(13)によって算出される前記適合度候補のとり得る値のうち、0以外の値であって、かつ最小の値であることを特徴とする請求項11に記載の動きベクトル検出装置。 The fitness threshold value is a value other than 0 and a minimum value among the possible values of the fitness candidate calculated by the fitness candidate calculation means (13). Item 12. The motion vector detection device according to Item 11.
- 前記記憶手段(15)は、前記注目領域に含まれる複数の画素のうち、各前記入力フレームの水平方向に対応する画素を前記代表1画素に間引いて生成される座標を用いて、前記記憶アドレスを算出することを特徴とする請求項1に記載の動きベクトル検出装置。 The storage means (15) uses the coordinates generated by thinning out the pixels corresponding to the horizontal direction of each input frame among the plurality of pixels included in the attention area to the representative pixel, and stores the storage address. The motion vector detection device according to claim 1, wherein:
- 請求項1~13のいずれか1つに記載の動きベクトル検出装置(1,62)と、
前記動きベクトル検出装置(1,62)によって検出される前記動きベクトルおよび前記適合度に基づいて、前記第1および第2の入力フレームに含まれる画素から、前記第1および第2の入力フレーム間に内挿する内挿フレームに含まれる画素を生成することによって、画像のフレームレートを変換するフレームレート変換手段(63)とを備えることを特徴とする画像処理装置(60)。 A motion vector detection device (1,62) according to any one of claims 1 to 13;
Based on the motion vector detected by the motion vector detection device (1, 62) and the fitness, between the first and second input frames from the pixels included in the first and second input frames. An image processing apparatus (60) comprising: frame rate conversion means (63) for converting a frame rate of an image by generating pixels included in an interpolation frame to be interpolated into the frame. - 請求項1~13のいずれか1つに記載の動きベクトル検出装置(1,62)と、
前記動きベクトル検出装置(1,62)によって検出される前記動きベクトルおよび前記適合度に基づいて、前記第1および第2の入力フレームに含まれる画素から新たな画素を生成することによって、前記第1および第2の入力フレームで構成される画像とは異なる解像度を有する新たな画像を生成する解像度変換手段(82)とを備えることを特徴とする画像処理装置(80)。 A motion vector detection device (1,62) according to any one of claims 1 to 13;
Based on the motion vector detected by the motion vector detection device (1, 62) and the fitness, a new pixel is generated from the pixels included in the first and second input frames. An image processing apparatus (80) comprising resolution conversion means (82) for generating a new image having a different resolution from the image composed of the first and second input frames.
Priority Applications (1)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
EP09809637A EP2317763A4 (en) | 2008-08-27 | 2009-03-19 | Motion vector detection device and image processing device |
Applications Claiming Priority (2)
Application Number | Priority Date | Filing Date | Title |
---|---|---|---|
JP2008-217866 | 2008-08-27 | ||
JP2008217866A JP4385077B1 (en) | 2008-08-27 | 2008-08-27 | Motion vector detection device and image processing device |
Publications (1)
Publication Number | Publication Date |
---|---|
WO2010023982A1 true WO2010023982A1 (en) | 2010-03-04 |
Family
ID=41549789
Family Applications (1)
Application Number | Title | Priority Date | Filing Date |
---|---|---|---|
PCT/JP2009/055423 WO2010023982A1 (en) | 2008-08-27 | 2009-03-19 | Motion vector detection device and image processing device |
Country Status (3)
Country | Link |
---|---|
EP (1) | EP2317763A4 (en) |
JP (1) | JP4385077B1 (en) |
WO (1) | WO2010023982A1 (en) |
Cited By (1)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2016185832A1 (en) * | 2015-05-21 | 2016-11-24 | オリンパス株式会社 | Image processing device, image processing method, and program |
Families Citing this family (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JP5448983B2 (en) * | 2010-04-05 | 2014-03-19 | 三菱電機株式会社 | Resolution conversion apparatus and method, scanning line interpolation apparatus and method, and video display apparatus and method |
SG11201400429RA (en) | 2011-09-08 | 2014-04-28 | Paofit Holdings Pte Ltd | System and method for visualizing synthetic objects withinreal-world video clip |
JP5975791B2 (en) * | 2012-08-27 | 2016-08-23 | 三菱電機株式会社 | Image processing apparatus and method, and image display apparatus and method |
CA2949478A1 (en) | 2014-05-30 | 2015-12-03 | Paofit Technology Pte Ltd | Systems and methods for motion-vector-aided video interpolation using real-time smooth video playback speed variation |
US9704298B2 (en) | 2015-06-23 | 2017-07-11 | Paofit Holdings Pte Ltd. | Systems and methods for generating 360 degree mixed reality environments |
Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH08163573A (en) * | 1994-12-09 | 1996-06-21 | Matsushita Electric Ind Co Ltd | Motion vector detector and successive scanning converter using the detector |
JP2002027414A (en) | 2000-06-13 | 2002-01-25 | Samsung Electronics Co Ltd | Method and device for format conversion using bidirectional motion vector |
JP3534551B2 (en) * | 1996-09-20 | 2004-06-07 | シャープ株式会社 | Motion detection device |
JP2006186504A (en) * | 2004-12-27 | 2006-07-13 | Sony Corp | Apparatus and method of image processing, recording medium, and program |
JP3935500B2 (en) | 2005-01-14 | 2007-06-20 | 株式会社モルフォ | Motion vector calculation method and camera shake correction device, imaging device, and moving image generation device using this method |
Family Cites Families (3)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
EP0358196B1 (en) * | 1988-09-09 | 1997-07-30 | Canon Kabushiki Kaisha | Automatic image stabilization device |
US6002431A (en) * | 1993-03-03 | 1999-12-14 | Goldstar Co., Ltd. | Video correction apparatus for camcorder |
US7667778B2 (en) * | 2004-04-09 | 2010-02-23 | Sony Corporation | Image processing apparatus and method, and recording medium and program used therewith |
-
2008
- 2008-08-27 JP JP2008217866A patent/JP4385077B1/en not_active Expired - Fee Related
-
2009
- 2009-03-19 EP EP09809637A patent/EP2317763A4/en not_active Withdrawn
- 2009-03-19 WO PCT/JP2009/055423 patent/WO2010023982A1/en active Application Filing
Patent Citations (5)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
JPH08163573A (en) * | 1994-12-09 | 1996-06-21 | Matsushita Electric Ind Co Ltd | Motion vector detector and successive scanning converter using the detector |
JP3534551B2 (en) * | 1996-09-20 | 2004-06-07 | シャープ株式会社 | Motion detection device |
JP2002027414A (en) | 2000-06-13 | 2002-01-25 | Samsung Electronics Co Ltd | Method and device for format conversion using bidirectional motion vector |
JP2006186504A (en) * | 2004-12-27 | 2006-07-13 | Sony Corp | Apparatus and method of image processing, recording medium, and program |
JP3935500B2 (en) | 2005-01-14 | 2007-06-20 | 株式会社モルフォ | Motion vector calculation method and camera shake correction device, imaging device, and moving image generation device using this method |
Non-Patent Citations (3)
Title |
---|
MASAYUKI KOBAYASHI ET AL.: "Ryoiki Bunkatsu to Warping Yosoku ni Motozuita Ugoki Hosho Hoshiki", NEN IEICE INFORMATION AND SYSTEM SOCIETY CONFERENCE, pages 111, XP008141742 * |
See also references of EP2317763A4 * |
TAKAHIKO FUKINUKI: "Multi-dimensional signal processing for TV image", 19 April 1122, NIKKAN KOGYO SIMBUN SHA, pages: 193 - 212 |
Cited By (2)
Publication number | Priority date | Publication date | Assignee | Title |
---|---|---|---|---|
WO2016185832A1 (en) * | 2015-05-21 | 2016-11-24 | オリンパス株式会社 | Image processing device, image processing method, and program |
US10516825B2 (en) | 2015-05-21 | 2019-12-24 | Olympus Corporation | Image processing device, image processing method, and computer program product for correcting a motion vector of a low-reliability block |
Also Published As
Publication number | Publication date |
---|---|
EP2317763A4 (en) | 2012-04-11 |
JP4385077B1 (en) | 2009-12-16 |
EP2317763A1 (en) | 2011-05-04 |
JP2010055252A (en) | 2010-03-11 |
Similar Documents
Publication | Publication Date | Title |
---|---|---|
JP4883223B2 (en) | Motion vector generation apparatus and motion vector generation method | |
JP5534299B2 (en) | Method and apparatus for dealing with periodic structures in motion compensation | |
CN106851046A (en) | Video dynamic super-resolution processing method and system | |
JP4869045B2 (en) | Interpolation frame creation method and interpolation frame creation apparatus | |
US10818018B2 (en) | Image processing apparatus, image processing method, and non-transitory computer-readable storage medium | |
JP4385077B1 (en) | Motion vector detection device and image processing device | |
US20120093231A1 (en) | Image processing apparatus and image processing method | |
US8264600B2 (en) | Image processing apparatus for converting a lower resolution image into a higher resolution image using cyclic coefficients | |
JPWO2019092844A1 (en) | Image processing apparatus, image processing method, and image processing program | |
JP2011509617A (en) | Sparse geometry for super-resolution video processing | |
JP2012034327A5 (en) | ||
JP2012095048A (en) | Image processor, image processing method, image display device, and image display method | |
US9106926B1 (en) | Using double confirmation of motion vectors to determine occluded regions in images | |
JP6604783B2 (en) | Image processing apparatus, imaging apparatus, and image processing program | |
JP5448983B2 (en) | Resolution conversion apparatus and method, scanning line interpolation apparatus and method, and video display apparatus and method | |
JP2507178B2 (en) | Motion detection method and motion detection device | |
JP2013126123A (en) | Image processing device, imaging apparatus, and image processing method | |
JPWO2013011797A1 (en) | Degradation restoration system, degradation restoration method and program | |
US9432690B2 (en) | Apparatus and method for video processing | |
JP5975791B2 (en) | Image processing apparatus and method, and image display apparatus and method | |
JP2009065283A (en) | Image shake correction apparatus | |
JP2013192094A (en) | Video magnification device and video magnification method | |
JP4250806B2 (en) | Field frequency conversion device and conversion method | |
JP3020299B2 (en) | Motion vector detection device | |
JPH08186802A (en) | Interpolation picture element generating method for interlace scanning image |
Legal Events
Date | Code | Title | Description |
---|---|---|---|
121 | Ep: the epo has been informed by wipo that ep was designated in this application |
Ref document number: 09809637 Country of ref document: EP Kind code of ref document: A1 |
|
WWE | Wipo information: entry into national phase |
Ref document number: 2009809637 Country of ref document: EP |
|
NENP | Non-entry into the national phase |
Ref country code: DE |