JP2008199097A - Image processing apparatus, image processing method, program, recording medium, portable terminal, and receiver - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an image processing apparatus capable of improving the accuracy of an enlarged interpolation frame by simultaneously executing frame interpolation processing and image enlargement processing, capable of achieving the enlargement processing of received frames and capable of generating smooth moving images. <P>SOLUTION: The image processing apparatus calculates moving vector values of images of received frames k(51x), k+1(52x) which are continuously received, adjusts the moving vectors on the basis of the number of frames interpolated between the received frames, and generates an interpolation enlargement frame 52m by referring to the received frames k(51x), k+1(52x) on the basis of the adjusted moving vector values and enlargement magnification M. <P>COPYRIGHT: (C)2008,JPO&INPIT

Description

  The present invention relates to a video processing device, a video processing method, a program, a recording medium, a portable terminal, and a receiving device, and in particular, a video processing device having a frame interpolation function and an image enlargement function, a video processing method, and the video processing method The present invention relates to a program for recording, a recording medium storing the program, a mobile terminal including the video processing device, and a receiving device.

  Currently, various mobile terminals capable of viewing so-called one-segment broadcasting (one-segment partial reception service for mobile terminals) have been developed. In this one-segment broadcasting, the number of frames transmitted is 15 frames per second, which is less than that of normal television broadcasting, and when watching sports programs such as soccer, frames are dropped, so frame interpolation is performed. It is necessary to display a smooth moving image. A technique related to such frame interpolation is disclosed in Patent Document 1.

  One-seg broadcasting has a resolution of up to 320 × 240 pixels (QVGA). However, in recent years, the display screen of mobile terminals has tended to increase in size, and one-seg broadcasting can be viewed on the enlarged display screen. However, since the displayed image is small, it is necessary to enlarge the displayed image. Patent Document 2 discloses a technique relating to such video enlargement.

  That is, both the frame interpolation process and the image enlargement process must be performed in order to comfortably view a one-segment broadcast on a mobile terminal having a large display screen.

FIG. 7 is a schematic diagram for explaining frame interpolation processing and image enlargement processing.
As shown in FIG. 7, conventionally, an interpolated frame I (102x) is generated from the received frame k (101x) and the received frame k + 1 (103x), an enlargement process is performed on each frame, and the enlarged frame 101m to 103m are generated. Note that the enlarged frames 104m and 105m are also generated from the interpolation frame I + 1 (104x) and the reception frame k + 2 (105x).
JP 2005-268912 A JP 2000-184187 A

However, when the interpolation frame I (102x) is generated and the enlarged frame 102m is generated from the interpolation frame I (102x), there arises a problem that a smooth moving image cannot be generated.
Here, this problem will be described with reference to FIG.
8A is a pixel array diagram 111 of 10 × 10 pixels indicating a part (block) of the received frame k (101x), and FIG. 8B is the same pixel array diagram of the received frame k + 1 (103x). FIG. 8C is a pixel array diagram 113 of the interpolation frame I (102x).

In the pixel array diagram 111 of FIG. 8A, it is assumed that a pixel of interest is set at coordinates (2, 2) (black portion). Similarly, it is assumed that the target pixel moved to the coordinates (7, 2) in the pixel array diagram 112 of FIG. 8B is set (black portion).
In such a case, the motion vector mv of the pixel of interest is calculated as (5, 0) ((7, 2)-(2, 2)). The motion vector referred to when generating the interpolation frame is a value obtained by dividing the motion vector mv by (number of interpolation frames + 1), that is, mv / 2 ((5,0) / 2 = (2.5,0). ) And non-integer multiples.

Therefore, as shown in FIG. 8C, the position of the pixel to be interpolated to either the coordinate (4, 2) or the coordinate (5, 2) in the pixel array diagram 113 must be determined. This pixel position has an error.
That is, when the motion vector calculated when generating the interpolation frame is a non-integer multiple (half-pixel accuracy), an error occurs when determining the pixel position of the interpolation frame (the interpolation accuracy is low). ).

  When an enlargement process is performed on an interpolation frame having such an error, the error further increases, and a smooth moving image cannot be generated.

The present invention has been made in view of such circumstances, and by executing the frame interpolation process and the image enlargement process at the same time, the accuracy of the enlarged interpolation frame is improved, and the enlargement process of the received frame is realized, so that smoothness is achieved. An object of the present invention is to provide a video processing apparatus, a video processing method, a program, and a recording medium on which the program is recorded.
It is another object of the present invention to provide a receiving device and a portable terminal provided with the video processing device.

  A first technical means is a video processing device that performs frame interpolation and enlarges an image on a received video, calculates a motion vector value of images of two received frames that are received consecutively, and interpolates between the received frames Adjusting the motion vector value based on the number of frames to be performed and the enlargement magnification of the image, and generating an interpolated frame image on the enlarged frame with reference to the received frame based on the adjusted motion vector value and the enlargement magnification Is a video processing apparatus characterized by the above.

  According to a second technical means, in the first technical means, the adjustment of the motion vector value is calculated by multiplying the calculated motion vector value by a reciprocal of a number obtained by adding 1 to the number of interpolation frames and an enlargement factor. Is a video processing apparatus characterized by the above.

  A third technical means generates a received frame image on an enlarged frame by referring to a value obtained by multiplying the calculated motion vector value by an enlargement magnification in the first technical means. It is.

  A fourth technical means is a video processing method for performing frame interpolation and enlarging an image on a received video, the step of calculating a motion vector value of images of two received frames successively received, and between the received frames Adjusting the motion vector value based on the number of frames to be interpolated and the enlargement magnification of the image, and referring to the received frame based on the adjusted motion vector value and the enlargement magnification, an image of the interpolation frame on the enlargement frame A video processing method characterized by comprising the steps of:

  According to a fifth technical means, in the fourth technical means, the step of adjusting the motion vector value is calculated by multiplying the calculated motion vector value by an inverse number obtained by adding 1 to the number of interpolation frames and an enlargement ratio. The video processing method is characterized by:

  According to a sixth technical means, in the fourth technical means, an image of a received frame is generated on an enlarged frame with reference to a value obtained by multiplying the calculated motion vector value by an enlargement factor. It is.

  A seventh technical means is a portable terminal comprising any one of the first to third video processing devices.

  The eighth technical means is a receiving device comprising any one of the first to third video processing devices.

  A ninth technical means is a program that executes any one of the fourth to sixth video processing methods.

  The tenth technical means is a recording medium in which the program of the ninth technical means is recorded.

  According to the present invention, by executing the frame interpolation process and the image enlargement process at the same time, the accuracy of the enlarged interpolation frame is improved, and a smooth moving image is generated by a simple process by performing the image enlargement process of the received frame. it can.

FIG. 1 is a schematic diagram for explaining processing for simultaneously executing frame interpolation processing and image enlargement processing.
As shown in FIG. 1, according to the present invention, based on the received frame k (51x) and the received frame k + 1 (52x), the interpolated enlarged frame 52m is generated by simultaneously executing the frame interpolation and the image enlarging process. Further, the enlarged frames 51m and 53m of the reception frame k / the reception frame k + 1 are generated by this interpolation expansion frame generation processing method.
Similarly, an interpolation expansion frame 54m and an expansion frame 55m are generated from the reception frame k + 1 (52x) and the reception frame k + 2 (53x).

  FIG. 2 is a block diagram of the video processing apparatus 1 having a function of simultaneously executing such frame interpolation processing and image enlargement processing.

  Reference numeral 11 denotes a received frame memory, which stores received video frames received by a video signal receiving apparatus (not shown). It is assumed that the reception frame memory 11 stores a reception frame k (11a), a reception frame k + 1 (11b), and the like.

  Reference numeral 12 denotes a pixel-of-interest extraction unit, which analyzes a received frame image stored in the reception frame memory 11 and extracts (detects) a pixel of interest. Various known methods can be applied to this method. Usually, an edge between the target pixel and another image (pixel) is detected, and a control line (a line segment that provides correspondence of a feature such as an edge between images) is set from the detected edge.

Reference numeral 13 denotes a motion vector calculation unit that estimates the motion of the control line set by the pixel-of-interest extraction unit 12 and calculates a motion vector mv1.
That is, as shown in (Equation 1), the pixel position of the target pixel in the reception frame k + 1 corresponding to the target pixel in the reception frame k extracted by the target pixel extraction unit 12 and the target pixel in the reception frame k. The motion vector mv1 is calculated by taking the difference from the pixel position.
mv1 = pixel position p ′ of the target pixel in the reception frame k + 1 corresponding to the target pixel in the reception frame k−pixel position p of the target pixel in the reception frame k (Expression 1)

Here, the process of calculating the motion vector will be specifically described with reference to FIG.
3A is a pixel array diagram 61 of 10 × 10 pixels indicating a part (block) of the received frame k (11a), and FIG. 3B is the same pixel array diagram of the received frame k + 1 (11b). 62.

Assume that a pixel of interest is set at coordinates (2, 2), (2, 3), (2, 4) in the pixel array diagram 61 of FIG. 3A (black portion). As described above, the control line is set based on the pixel, but in the following description, the pixel corresponds to a part of the control line. Similarly, it is assumed that the target pixel moved to the coordinates (7, 2), (7, 3), (7, 4) in the pixel array diagram 62 of FIG. 3B is set (black portion).
As is apparent from the figure, the target pixel moves in the right direction.

  The motion vector calculation unit 13 calculates the motion vector of the pixel of interest in the reception frame k and the reception frame k + 1. In the example of FIG. 3, the motion vector mv1 (p′−p) is calculated as (5, 0) ((7, 2) − (2, 2)).

  Returning to the block diagram of FIG.

  Reference numeral 14 denotes a parameter calculation unit, which calculates the reciprocal of (interpolation frame number L + 1 = N) and outputs it to the motion vector adjustment unit 15 when generating an interpolation enlarged frame.

The motion vector adjustment unit 15 calculates the motion vector mv1 calculated by the motion vector calculation unit 13, C indicating the number of interpolated frames L and 1, C corresponding to the frame to be interpolated, and the parameter calculation unit 14 Formula for calculating the motion vector mv2 by multiplying the parameter (1 / N, N = L + 1) and the input enlargement factor M mv2 = mv1 × C × (1 / N) × M (Expression 2)
Based on the above, the value of the motion vector mv2 is adjusted for each interpolation enlarged frame.

For example, when generating three interpolation frames (number of interpolation frames L = 3),
For the first interpolated enlarged frame, mv2 ′ = mv1 × (1) × (1/4) × M
Mv2 ″ = mv1 × (2) × (1/4) × M for the second interpolation expansion frame
Mv2 ′ ″ = mv1 × (3) × (1/4) × M for the third interpolation expansion frame
As described above, the value of the motion vector mv2 is adjusted for each interpolation enlarged frame.

In this embodiment, when generating an interpolation enlarged frame, the value of the motion vector mv2 to be adjusted is mv1 = (5,0), C = 1, N = 2, M = 2, and (5,0 ).
Note that the enlargement magnification is arbitrary.

  The output frame generation unit 16 generates an image of an interpolation frame and an image of a reception frame on the enlarged frame based on the value of the motion vector mv2 adjusted by the motion vector adjustment unit 15 and the magnification M.

When generating an interpolated enlarged frame, based on a pixel position p and a motion vector mv2 constituting a part of a control line in the received frame, an interpolated enlarged frame in the received frame corresponding to the pixel position p is received. A pixel position P is calculated. In this case, the pixel position P is, for example,
P = p × M + mv2 (Formula 3)
Is calculated by Then, an enlarged interpolation frame is generated by performing a predetermined enlargement process on pixels located in the vicinity of the calculated pixel position P.

  The calculation of the pixel position P is not limited to (Equation 3), and various calculation methods can be employed by referring to the adjusted motion vector mv2.

  Further, the pixel position in the region of the control line (or the vicinity of the control line) composed of the pixel position P in the interpolation enlarged frame and the pixel value of the pixel position are the reception frame (for example, the reception frame) It is possible to calculate with reference to a pixel position in k / reception frame k + 1), a pixel value of the pixel position, and the like, and various conventionally proposed methods can be adopted as this calculation method.

  The interpolation enlargement process is executed for each frame to be interpolated and enlarged.

Further, when generating an enlarged frame of the reception frame k + 1, for example, the motion vector mv2 in (Expression 3) may be replaced with (motion vector mv1 × enlargement factor M). In the enlarged frame generation process, as described above, various methods conventionally proposed can be adopted by referring to the motion vector mv1 / enlargement factor.
In this way, since the enlarged frame of the received frame can be generated using the interpolation enlarged frame generation method, the processing load can be reduced.
It should be noted that various conventionally proposed methods can be employed as the pixel interpolation method in the enlargement processing.

  The output frame memory 17 is a memory used at the time of execution of the interpolation enlarged frame generation process and the reception frame enlargement process in the output frame generation unit 16, and the interpolation enlarged frame and the like are expanded in the memory. The generated interpolated enlarged frame and the enlarged frame of the received frame are output to a display device (not shown).

  Here, an example in which one frame is interpolated with respect to the received frame of the one-segment broadcasting and at the same time the QVGA video (320 × 240) is doubled will be described with reference to FIGS.

(Interpolation enlargement frame generation processing)
FIG. 4A is a pixel array diagram 63 of 20 × 20 pixels showing a part (block) of an interpolation enlarged frame generated by the output frame generation unit 16. This is a pixel array diagram in which the pixel array diagram of FIG.
The output frame generation unit 16 determines the pixel position of the interpolation enlarged frame using, for example, the above-described equation (Equation 3), P = p × M + mv2.

  That is, the pixel position p (2, 2), (2, 3), (2, 4) constituting a part of the control line in FIG. By adding the motion vector mv2 (5, 0), pixel positions ((9, 4), (9, 6), (9, 8)) constituting a part of the control line of the interpolation enlarged frame are determined ( Black part).

  Note that the pixel positions (gray portions) on the right, top, and top right of the determined pixels are pixels that are interpolated with enlargement, and the pixel value (luminance value) at the pixel position is also the pixel in FIG. It is determined with reference to the pixel value (luminance value) at the position p.

  An interpolation expansion frame is generated by executing such processing on other blocks of the interpolation expansion frame.

(Enlargement processing of received frame k + 1 (11b))
FIG. 4B shows a pixel array diagram 64 of 20 × 20 pixels in which the pixel array diagram of FIG.
The output frame generation unit 16 determines the pixel position of the enlarged frame using the above-described equation (Equation 3) and the equation in which the motion vector mv2 of P = p × M + mv2 is replaced with (motion vector mv1 × enlargement magnification M). .

  That is, the pixel position p (2,2), (2,3), (2,4) in FIG. 3A is multiplied by the enlargement magnification M (2), and the motion vector mv1 (5,0) is enlarged. The pixel position (14, 4), (14, 6), (14, 8) of the enlarged frame is determined by adding the value obtained by multiplying M (2) (black portion).

  As described above, the right, upper, and upper right pixel positions (gray portions) of the determined pixels are determined according to the enlargement magnification, and the pixel value (luminance value) of the pixel position is also shown in FIG. / Determined with reference to the pixel value (luminance value) at the pixel position p in FIG. For this enlargement process, various conventionally proposed enlargement processes can be employed.

  By executing such processing for other blocks of the received frame k (11a), an enlarged frame is generated.

(Enlargement process of received frame k (11a))
FIG. 4C shows a pixel array diagram 65 of 20 × 20 pixels obtained by doubling the pixel array diagram of FIG.
This enlargement process is similarly generated by the interpolation enlargement process of the reception frame k-1 and the reception frame k.

As described in the related art, when an enlargement process is performed on a frame in which an error has occurred (see FIG. 8C), the error is further increased as shown in the pixel arrangement diagram 66 of FIG. portion).
However, by executing the frame interpolation process and the image enlargement process simultaneously as in the present invention, it is possible to generate an interpolation enlargement frame with higher accuracy than the interpolation enlargement frame generated by the conventional method, and a smooth moving image can be generated. .

FIG. 6 is a flowchart for explaining the interpolation enlarged frame generation process and the received frame enlargement process.
The received frame k and the received frame k + 1 are stored in the received frame memory 11 (step S1).
The target pixel extraction unit 12 extracts a target pixel from the stored images in the received frame k and the received frame k + 1, and sets a control line (step S2).
Next, the motion vector calculation unit 13 calculates the motion vector mv1 of the pixel of the control line set by the target pixel extraction unit 12 (step S3).

  The parameter calculation unit 14 calculates the reciprocal (1 / N) of N (number of frames to be interpolated L + 1) (step S4), sets the input magnification M (step S5), and the motion vector adjustment unit 15 A motion vector adjustment value mv2 is calculated based on the motion vector mv1 calculated in step S3, the reciprocal number of N (1 / N) calculated by the parameter calculation unit 14, the set enlargement magnification M (see Formula 2), and the like (step 2) S6)

  The output frame generation unit 16 calculates the number of frames to be interpolated based on the target pixel extracted by the target pixel extraction unit 12, the magnification (M), and the value of the motion vector mv2 adjusted by the motion vector adjustment unit 15 (see Equation 3). Minute interpolation enlarged frames are generated (step S7).

  In addition, the output frame generation unit 16 replaces the motion vector mv2 with (motion vector mv1 × enlargement factor M) to generate an enlarged frame of the reception frame k + 1 (Step S8).

  Next, 1 is added to the counter k (step S9), and it is determined whether the reception process of the video frame is completed (step S10). If not finished (step S10: NO), the process of step S1 is executed. If completed (step S10: YES), the process ends. Note that the enlarged frame of the reception frame k is generated by the interpolation enlargement process in the reception frame k-1 and the reception frame k as described above.

  As described above, by executing the frame interpolation process and the image enlargement process at the same time, it is possible to generate an enlarged interpolation frame with high accuracy and to generate a smooth moving image by performing the enlargement process of the image of the received frame.

  In this embodiment, the interpolation expansion frame is generated with reference to the reception frame k. However, the interpolation expansion frame can be generated with reference to the reception frame k + 1.

The video processing apparatus according to the present invention can also be mounted on a mobile terminal such as a mobile phone or a PDA, or a video receiving apparatus.
Furthermore, the video processing function according to the present invention can be executed by a program, the program can be recorded on a recording medium, and the recorded program can be installed and executed on a portable terminal or an information processing apparatus.

It is the schematic for demonstrating the process which performs a frame interpolation process and an image expansion process simultaneously. 1 is a block diagram of a video processing apparatus according to the present invention. It is a figure for demonstrating the process which calculates a motion vector. It is a pixel arrangement | sequence figure which shows a part (block) of an expansion frame. It is a pixel arrangement | sequence diagram which shows a part (block) of the flame | frame in which the error produced. It is a flowchart for demonstrating the production | generation process of an interpolation expansion frame, and the expansion process of a received frame. It is the schematic for demonstrating a frame interpolation process and an image expansion process. It is a pixel arrangement | sequence diagram which shows a part (block) of a received frame.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 ... Video processing apparatus, 11 ... Reception frame memory, 12 ... Interesting pixel extraction part, 13 ... Vector calculation part, 14 ... Parameter calculation part, 15 ... Vector adjustment part, 16 ... Output frame generation part, 17 ... Output frame memory, 51m, 53m, 55m ... enlarged frame, 52m, 54m ... interpolated enlarged frame, 51x to 53x ... received frame, 61-66 ... pixel array diagram, 101m-105m ... enlarged frame, 101x, 103x, 105x ... received frame, 102x, 104x... Interpolation frame, 111 to 113.

Claims (10)

A video processing device that performs frame interpolation and enlarges an image for a received video,
Calculating the motion vector value of the images of two received frames received in succession;
Adjusting the motion vector value based on the number of frames to be interpolated between the received frames and the magnification of the image;
An image processing apparatus, wherein an image of an interpolation frame is generated on an enlarged frame by referring to the received frame based on the adjusted motion vector value and an enlargement magnification.   The video processing apparatus according to claim 1, wherein the adjustment of the motion vector value is calculated by multiplying the calculated motion vector value by a reciprocal of a number obtained by adding 1 to the number of interpolation frames and an enlargement magnification.   2. The video processing apparatus according to claim 1, wherein, in the adjustment of the motion vector value, an image of a received frame is generated on an enlarged frame with reference to a value obtained by multiplying the calculated motion vector value by an enlargement factor. . A video processing method for performing frame interpolation and enlarging an image for a received video,
Calculating a motion vector value of images of two received frames received in succession;
Adjusting the motion vector value based on the number of frames to be interpolated between the received frames and the magnification of the image;
A video processing method comprising: generating an image of an interpolated frame on an enlarged frame by referring to the received frame based on the adjusted motion vector value and an enlarged magnification.   5. The video processing according to claim 4, wherein the step of adjusting the motion vector value is calculated by multiplying the calculated motion vector value by a reciprocal of a number obtained by adding 1 to the number of interpolation frames and an enlargement ratio. Method.   5. The video processing according to claim 4, wherein, in the step of adjusting the motion vector, an image of a received frame is generated on an enlarged frame with reference to a value obtained by multiplying the calculated motion vector value by an enlargement factor. Method.   A portable terminal comprising the video processing device according to claim 1.   A receiving apparatus comprising the video processing apparatus according to claim 1.   The program which performs the video processing method of any one of Claims 4-6.   A recording medium on which the program according to claim 9 is recorded.

Images