JP2000148130A - Sprite formation method and device and recording medium recording the method - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide a method and a device capable of forming a sprite having fewer blurrs. SOLUTION: A motion parameter extraction means 2 extracts a global motion parameter from a dynamic image including camera operations, such as panning, zooming or the like, from a dynamic image input means 1, and a reference frame determination means 3 determines a reference frame at the time of sprite formation in the dynamic image. Then, a predictive image formation means 4 forms a predictive image by influencing the global motion parameter on a frame other than the reference frame. Then, a sampling point pixel value calculation means 5 calculates the pixel value of a sampling point by using pixel values of other plural frames distributing on the sampling position of pixels of the reference frame or on its periphery in the state where a sampling point interval is different from the reference frame. Finally, a positioning means 6 arranges the image of a frame including other plural predictive images relative to a background of the reference frame by using the pixel value of the calculated sampling point, so that patterns are connected continuously, to thereby form the sprite.

Description

DETAILED DESCRIPTION OF THE INVENTION

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a technique for generating a sprite from a moving image.

[0002]

2. Description of the Related Art FIG. 7 is a conceptual diagram of a general sprite. A sprite is obtained by aligning all the frames in the coordinate system of the reference frame and pasting them to one image.

FIG. 8 shows the concept of positioning in a sprite. FIG. 9A shows a state in which the respective frames are aligned as viewed from the xy plane direction, and FIG. 9B shows this state as viewed from the tx plane direction. To convert this into a single plane as a sprite, the coordinates (x,
In y), the average value or the median value (median value) of the pixels overlapping in the time t direction is used as the value of the coordinates.

FIG. 9 shows a method of complementing pixels at the time of alignment in the prior art. When an arbitrary frame of interest is aligned with the coordinates of the reference frame, the pixels of the frame of interest are generally not on the grid points of the reference frame as shown in FIG. Therefore, four pixels i1-i4 of the frame of interest surrounding the pixel of interest ip of the reference frame are selected,
The value of the pixel of interest ip is estimated using these four values.
Generally, linear interpolation is used, and the value of ip is calculated using the following equation (1).

[0005] ip = i1 × (1-a) × b + i2 × a × b + i3 × (1-a) × (1-b) + i4 × (1-b) × a (1) As described above, In the sprite generation method from a moving image, a reference frame is determined in the moving image, and the obtained global motion parameters are applied to frames other than the reference frame to generate a predicted image. The interpolation is performed so that the position matches the sample position of the reference image, and a sprite is generated using these interpolated images. That is, in the conventional method, a candidate point of a sprite pixel for a certain pixel position in the reference frame is selected from any one frame other than the reference frame. Specifically, a median (center) value or an average value of a plurality of frames at the time of alignment in the time direction is used. At this time, for the frames other than the reference frame, the pixel values have already been calculated by interpolation processing so as to match the sample interval of the reference frame.

[0006]

However, in the conventional sprite generation method, a sprite is generated using pixels of a frame already interpolated, and interpolation is performed by interpolation from four surrounding pixels. The interpolation from the four surrounding pixels corresponds to a low-pass filter.
Moreover, since interpolation with an accuracy smaller than the sample interval is impossible, there is a disadvantage that an image generated as a sprite is blurred. That is, since the ip calculated by Expression (1) does not satisfy the sampling theorem, it has the same effect as passing through the low-pass filter, and thus the obtained predicted image is blurred.

There is also a problem that when a selected pixel uses an average value of pixels aligned in the time direction, or when the pixel is selected at random, it is affected by an object other than the background (the median value is This problem can be avoided if the background occupies more than half of all pixels in the time direction).

An object of the present invention is to provide a method and an apparatus capable of generating a sprite with little blur and a sprite excluding a moving area.

[0009]

The present invention solves the above-mentioned problems by the inventions listed below.

The method according to the first aspect of the present invention includes the steps of extracting global motion parameters between frames from a moving image including a camera operation such as panning and zooming, determining a reference frame in the moving image, Generating a predicted image by applying the extracted global motion parameters to a frame other than the reference frame, and generating a predicted image at a sample position of a pixel of the reference frame or at a periphery thereof at a sample point interval different from that of the reference frame. Calculating a pixel value of a sample point using pixel values of a plurality of other frames including the predicted image distributed (hereinafter, may be only the predicted image); Using a plurality of other frame images including the predicted image against the background of the reference frame,
Arranging pictures so as to be connected continuously.

A method according to a second aspect of the present invention includes the steps of extracting global motion parameters between frames from a moving image including a camera operation such as panning and zooming, determining a reference frame in the moving image, Generating a predicted image by applying the extracted global motion parameter to a frame other than the reference frame; and calculating sample values of a plurality of other frames including the predicted image existing in the time direction of the coordinates of the reference frame. A median (center) value is calculated, and a frame at which the median value is to be given and a plurality of frames around the frame are grid coordinates that are sample positions of pixels of the reference frame or positions near the grid coordinates and close to the grid coordinates. Sampling points are determined by using pixel values of a plurality of other frames including the predicted image around which sampling point intervals are distributed in a state different from the reference frame. Calculating the pixel value, and using the calculated pixel value of the sample point, images of a plurality of other frames including the predicted image with respect to the background of the reference frame, so that the pattern is continuously connected. And arranging the sprites.

The apparatus according to the first aspect of the present invention includes a global motion parameter extracting means for extracting global motion parameters between frames from a moving image including a camera operation such as panning and zooming, and determining a reference frame in the moving image. Reference frame determination means, predicted image generation means for generating a predicted image by applying the extracted global motion parameters to frames other than the reference frame, and sample point intervals at or around sample positions of pixels of the reference frame Sample point pixel value calculation means for calculating pixel values of sample points using pixel values of a plurality of other frames including the predicted image distributed in a state different from the reference frame; and Using the pixel values, the image of the plurality of other frames including the predicted image is connected to the background of the reference frame so that the pattern is continuously connected. And positioning means for positioning a sprite generating apparatus characterized by comprising.

The apparatus according to the second aspect of the present invention includes a global motion parameter extracting means for extracting global motion parameters between frames from a moving image including a camera operation such as panning and zooming, and determining a reference frame in the moving image. Reference frame determining means, predicted image generating means for generating the predicted image by applying the extracted global motion parameter to a frame other than the reference frame, and calculating the predicted image existing in the time direction of the coordinates of the reference frame. Calculate the median (median) value of the sample values of multiple other frames,
In the frame giving the median value and a plurality of frames around the frame, the sampling point interval is set around the grid coordinate which is the sampling position of the pixel of the reference frame or around the position close to the grid coordinate and close to the grid coordinate. A sample point pixel value calculating means for calculating a pixel value of a sample point using pixel values of a plurality of other frames including the predicted image distributed in a state different from the above, and using the pixel value of the calculated sample point. And a positioning means for arranging images of a plurality of other frames including the predicted image with respect to the background of the reference frame so that pictures are continuously connected. It is.

The steps in the sprite generation method according to the first or second aspect of the present invention can be executed by a computer, and a program for that is recorded on a computer-readable recording medium, and the sprite generation method is recorded. It is possible to use the recording medium.

According to the first aspect of the present invention, when an image of another frame is arranged so that a picture is continuously connected to the background of the reference frame, the sample point interval of the pixel of the reference frame or the surroundings thereof is different from that of the reference frame. The pixel value of the sample point is calculated using the pixel values of a plurality of other frames distributed in a state different from the reference frame. That is, in the present invention, a point serving as a sprite pixel candidate for a certain pixel position in the reference frame is selected from a plurality of frames other than the reference frame. At this time, pixels in frames other than the reference frame have not been subjected to the interpolation processing. Therefore, when viewed from the sample intervals of the reference frame, distribution occurs between those sample intervals. In order to determine the value of the pixel of the sprite corresponding to a certain pixel position in the reference frame, interpolation from the plurality of frames distributed around this pixel is performed according to the distance.

These methods are implemented by hardware or a program running on a computer.

According to the first aspect of the present invention, in order to determine the value of the pixel of the sprite corresponding to a certain pixel position of the reference frame,
By using pixels from a plurality of frames distributed around this pixel and performing interpolation according to the distance, calculating the pixels of the sprite using the pixels closer to the sample interval of the reference frame Can be.

In the present invention 2, in consideration of the characteristic that the moving object is removed when the median value is taken in the time direction, the median is set in the time direction from a plurality of frames aligned with the coordinates in the grid coordinates of the reference frame. Calculate the value. From the N non-interpolated frames around the frame giving the median value, candidate points for calculating sprite pixels with high accuracy are selected, and the selected points are interpolated to obtain sprite values. This interpolation method employs the following method.

When an image of another frame is arranged so that the picture is connected continuously to the background of the reference frame, the frame giving the calculated median value and a plurality of frames around the frame give the calculated frame value. The pixel value of the sample point is calculated using the pixel values of a plurality of other frames in which the sample point interval is distributed around the pixel sample position or a position outside the pixel in a state different from the reference frame. That is, in the present invention, a point serving as a sprite pixel candidate for a certain pixel position in the reference frame is selected from a plurality of frames other than the reference frame. At this time, pixels in frames other than the reference frame have not been subjected to the interpolation processing.
Therefore, when viewed from the sample intervals of the reference frame, distribution occurs between those sample intervals. In order to determine the value of the pixel of the sprite corresponding to a certain pixel position in the reference frame, interpolation from the plurality of frames distributed around this pixel is performed according to the distance.

These methods are realized by hardware or a program running on a computer.

In the present invention 2, in order to determine the value of the pixel of the sprite corresponding to a certain pixel position of the reference frame,
By using pixels from a plurality of frames distributed around this pixel and performing interpolation according to the distance, calculating the pixels of the sprite using the pixels closer to the sample interval of the reference frame Can be. Further, since it is possible to select a pixel candidate for interpolation only in a frame around a frame to which a median value is initially given, a sprite excluding a moving region can be generated.

[0022]

Embodiments of the present invention will be described below in detail with reference to the drawings.

FIG. 1 shows the configuration of an embodiment of the apparatus according to the present invention. 1 is a moving image input unit, 2 is a motion parameter extracting unit, 3 is a reference frame determining unit, 4 is a predicted image frame generating unit, 5 is a sample point pixel value calculating unit, and 6 is a positioning unit.

FIG. 2 shows one embodiment of the present invention 1 in FIG.
2 shows a flowchart of an embodiment of the method of the present invention 1 together with an operation example of the apparatus of the present invention. Hereinafter, according to FIGS. 1 and 2,
The above operation example and the sprite generation method in this embodiment will be described.

First, a moving image including a camera operation such as pan / zoom is input from the moving image input means 1. The motion parameter extracting means 2 extracts a global motion parameter from the input moving image, and the reference frame determining means 3
A reference frame serving as a reference when generating sprites is determined in the input moving image.

Next, the predicted image generating means 4 generates a predicted image by applying the global motion parameter to a frame other than the reference frame.

Next, in the sample point pixel value calculating means 5,
The pixel value of the sample point is calculated using the pixel values of a plurality of other frames in which the sample point interval is distributed at or around the sample position of the pixel of the reference frame in a state different from the reference frame.

Finally, the image of a frame including a plurality of other predicted images with respect to the background of the reference frame is obtained by the positioning means 6 using the pixel values of the sample points calculated above.
The sprites are arranged so that the pictures are continuously connected to generate a sprite with little blur.

FIG. 3 is an explanatory diagram of a pixel complementing method at the time of position alignment at the time of sprite generation according to this embodiment.

FIG. 3A shows the state of pixels of a plurality of frames mapped to the coordinate system of the reference frame. In the present embodiment, four pixels that provide the shortest distance surrounding the target pixel ip are selected. FIG. 3B shows an arrangement of four pixels used for estimating a pixel of interest in the related art, and FIG. 3C shows an arrangement of the same four pixels in this embodiment. In the prior art, the pixels used for interpolation are far apart, whereas in the present embodiment, the pixel of interest ip is estimated from pixels at a shorter distance than in the prior art. The value of ip is calculated by the sample point pixel value calculation means 4 using the luminance values i1-i4 of the four pixels and the distances d1-d4 between the four pixels and the pixel of interest, as in the following equation (2). Is calculated.

Ip = k × (i1 / d1 + i2 / d2 + i3 / d3 + i4 / d4) ∵k = (d1 × d2 × d3 × d4) / (d2 × d3 × d4 + d1 × d3 × d4 + d1 × d2 × d4 + d1 × d2 × d3 Next, an embodiment of the present invention 2 will be described. The configuration is the same as that of FIG. 1, but the method of interpolating pixels at the time of alignment, that is, the method of calculating the pixel value of the sample point by the sample point pixel value calculation means 5 is different from that of the first embodiment of the present invention. .

FIG. 4 shows a flowchart of an embodiment of the method of the present invention 2 together with an operation example of this embodiment.
Hereinafter, an operation example of the apparatus according to the present embodiment and a sprite generation method will be described with reference to FIGS.

First, a moving image including camera operations such as panning and zooming is input from the moving image input means 1. The motion parameter extracting means 2 extracts a global motion parameter from the input moving image, and the reference frame determining means 3
A reference frame serving as a reference when generating sprites is determined in the input moving image.

Next, the predicted image generating means 4 generates a predicted image by applying the global motion parameter to a frame other than the reference frame.

Next, in the sample point pixel value calculating means 5,
A median (center) value of sample values of a plurality of other frames existing in the time direction of the coordinates of the reference frame is calculated, and a frame giving the median value and a plurality of frames around the frame are searched. Then, in the frame giving the median value and a plurality of frames around it, the sample coordinates of the pixels of the reference frame are grid coordinates, or the sample point interval around the position near the grid coordinates and close to the grid coordinates is the reference frame. The pixel values of the sample points are calculated using the pixel values of other plural prediction image frames distributed in different states.

Finally, the image of a frame including a plurality of other predicted images with respect to the background of the reference frame is obtained by the positioning means 6 using the pixel values of the sample points calculated above.
The sprites are arranged so that the pictures are continuously connected to generate a sprite with little blur and no moving area.

FIG. 5 shows a method for searching for a median frame (of a frame providing a median value) and its surrounding frames in the present invention. A pixel corresponding to the grid coordinate of the reference frame is obtained from each frame using a conventional linear interpolation method, and a median value is calculated by searching in the time t direction at the reference coordinate p (x, y). At the same time, a frame number n giving a median value of the coordinates (x, y)
(X, y) is stored. A similar process is performed for all coordinates. Next, at each coordinate, pixel interpolation is performed in N frames around n (x, y).

FIG. 6 shows a pixel interpolation method at the time of positioning according to the present invention. FIG. 6A shows the state of pixels of a plurality of frames mapped to the coordinate system of the reference frame. In the present embodiment, four pixels that provide the shortest distance belonging to four quadrants surrounding the target pixel ip are selected. FIG. 6B shows an arrangement of four pixels used for estimating a pixel of interest in the related art, and FIG. 6C shows an arrangement of the same four pixels in the present embodiment. In the prior art, the pixels used for interpolation are far apart, whereas in the present embodiment, the pixel of interest ip is estimated from pixels at a shorter distance than in the prior art. The value of ip is calculated by using the luminance values of four pixels, i1-i4, and the distance d1-d4 between the four pixels and the pixel of interest, as in Expression (3).
It is calculated by taking the weighted weighted sum by the distance dn.

Ip = Σ _{j = 1} ⁿ wj × ij wj = f (dj) (3) These samples need not be four points, and are similarly calculated from two points and three points. As a specific example of the weight, it is calculated as in the following equation (4). Where d1 ≠ 0, d2 ≠ 0,
It is assumed that d3 ≠ 0 and d4 ≠ 0.

Ip = k × (i1 / d1 + i2 / d2 + i3 / d3 + i4 / d4) ∵k = (d1 × d2 × d3 × d4) / (d2 × d3 × d4 + d1 × d3 × d4 + d1 × d2 × d4 + d1 × d2 × d3 (4) In addition, as a second specific example, it can be calculated as in the following equation (5).

T = d1 + d2 + d3 + d4 ip = ((t−d1) / t) i1 + ((t−d2) / t) i2 + ((t−d3) / t) i3 + ((t−d4) / t) i4 ... ( 5) It goes without saying that some or all of the means shown in FIG. 1 can be made to function using a computer, or the steps of the processing shown in FIGS. 2 and 4 can be executed by a computer. In addition, a program for causing a computer to function as the means or a program for causing a computer to execute the processing steps is stored in a computer-readable recording medium such as an FD (Floppy (registered trademark) disk) or , MO, ROM, memory card, CD, DV
D, recorded on a removable disk or the like, provided, and distributed.

[0042]

According to the first aspect of the present invention, since a pixel of a sprite can be calculated by using a pixel located at a position closer than the sample interval of the reference frame, a sprite with less blur can be generated.

According to the second aspect of the present invention, since the pixels of the sprite can be calculated using the pixels located at positions closer than the sample interval of the reference frame, a sprite with less blur can be generated. In addition, by selecting a candidate of an interpolation pixel from a frame around a frame to which a median value is first given, a sprite excluding a moving region can be generated.

[Brief description of the drawings]

FIG. 1 is a configuration diagram showing an embodiment of an apparatus according to the present invention 1. FIG.

FIG. 2 is a flowchart showing an example of an operation of an embodiment of the apparatus according to the present invention 1 and an embodiment of the method according to the present invention 1;

FIGS. 3A, 3B, and 3C are diagrams for explaining a method of complementing pixels at the time of alignment according to the first embodiment of the present invention; FIGS.

FIG. 4 is a flowchart showing an example of an operation of an embodiment of the apparatus according to the present invention 2 and an embodiment of the method according to the present invention 2;

FIG. 5 is a diagram for explaining a method of searching for a median frame and its surrounding frames in the embodiment 2 of the present invention.

FIGS. 6A, 6B, and 6C are diagrams for explaining a method of complementing pixels at the time of position alignment according to the second embodiment of the present invention.

FIG. 7 is a conceptual diagram of a general sprite.

FIG. 8 is a conceptual diagram of alignment in sprite generation in the related art.

FIG. 9 is a diagram illustrating a method of complementing pixels at the time of alignment according to the related art.

[Explanation of symbols]

 REFERENCE SIGNS LIST 1 moving image input means 2 motion parameter extracting means 3 reference frame determining means 4 predicted image frame generating means 5 sample point pixel value calculating means 6 positioning means

──────────────────────────────────────────────────続 き Continued on the front page (51) Int.Cl. ⁷ Identification symbol FI Theme coat ゛ (Reference) H04N 7/18 G06F 15/66 470J F-term (Reference) 5B057 BA02 CE08 DA07 DC22 DC30 5C054 FC12 FC13 FE12 5C082 AA27 BA41 BA43 CA21 CA31 CA52 CA74 CA81 CB01 DA87 MM10

Claims (5)

[Claims] 1. Extracting global motion parameters between frames from a moving image including a camera operation such as panning and zooming; determining a reference frame in the moving image; Is applied to a frame other than the reference frame to generate a predicted image, and the predicted image whose sample point interval is distributed in a state different from the reference frame at or around a sample position of a pixel of the reference frame. Calculating the pixel value of the sample point using the pixel values of the other plurality of frames including the plurality of other frames including the predicted image with respect to the background of the reference frame using the pixel value of the calculated sample point. Arranging the images of the frames so that the pictures are connected continuously. 2. Extracting a global motion parameter between frames from a moving image including a camera operation such as panning and zooming; determining a reference frame in the moving image; Is applied to a frame other than the reference frame to generate a predicted image, and a median value of sample values of a plurality of other frames including the predicted image existing in the time direction of the coordinates of the reference frame is calculated. In the frame giving the median value and a plurality of frames around the frame, the sampling point interval is set around the grid coordinate which is the sampling position of the pixel of the reference frame or around the position close to the grid coordinate and close to the grid coordinate. Calculating the pixel value of the sample point using the pixel values of the other plurality of frames including the predicted image distributed in a different state, Arranging images of a plurality of other frames including the predicted image with respect to the background of the reference frame using the calculated pixel values of the sample points so that the pictures are connected continuously. A sprite generation method characterized by the following. 3. Global motion parameter extracting means for extracting global motion parameters between frames from a moving image including a camera operation such as pan / zoom, and reference frame determining means for determining a reference frame in the moving image. Predictive image generating means for generating a predictive image by applying the extracted global motion parameter to a frame other than the reference frame; and a sample point interval at or around a sample position of a pixel of the reference frame. Sample point pixel value calculation means for calculating pixel values of sample points using pixel values of a plurality of other frames including the predicted image distributed in different states; and Positioning for arranging images of a plurality of other frames including the predicted image with respect to the background of the reference frame so that the pictures are connected continuously. Sprite generating apparatus characterized by the means comprises. 4. Global motion parameter extracting means for extracting global motion parameters between frames from a moving image including a camera operation such as panning and zooming; reference frame determining means for determining a reference frame in the moving image; Predicted image generating means for generating a predicted image by applying the extracted global motion parameter to a frame other than the reference frame; and a plurality of other frames including the predicted image existing in the time direction of the coordinates of the reference frame. Calculates the median value of the sample value of, the grid coordinates which are the sample positions of the pixels of the reference frame in the frame giving the median value and a plurality of frames around the frame, or a position near the grid coordinates and close to the grid coordinates And a plurality of other frames including the predicted image in which sampling point intervals are distributed in a state different from the reference frame around the reference frame. Sample point pixel value calculating means for calculating the pixel value of the sample point using the pixel value of the frame; and other pluralities including the predicted image with respect to the background of the reference frame using the pixel value of the calculated sample point. And a positioning means for arranging the images of the frames in such a way that the pictures are continuously connected. 5. A recording recording the sprite generation method, wherein a program for executing the steps of the sprite generation method according to claim 1 on a computer is recorded on a recording medium readable by the computer. Medium.