KR100522171B1 - Generator of the arbitrary shape frame and method for generating the arbitrary shape frame using the same - Google Patents

Abstract

The present invention relates to an apparatus for inserting a new frame between two frames having shape information and texture information in an image decoding system for processing arbitrary shapes. Arbitrary shape frames usually have different image sizes. In such a case, a frame of a small size is updated in accordance with a larger one of the frames before and after, thereby making it convenient to insert a new frame. As a method of updating a small frame, a matching point that minimizes the difference between two frames through frame matching using shape information, which is easy to process data, is used. After moving the frame, the rest is padded with pixel values outside the image to fit the larger frame. As padding, horizontal padding is performed first, followed by vertical padding. New frames are inserted by linear interpolation for before and after frames that have been expanded to the same size. In addition, the arbitrary shape image has the position information of the arbitrary shape image in the background image screen to be combined with the background image, and the position information of the inserted frame is obtained by using the linear interpolation method to obtain the position information of the before and after frame. And you can see the natural transition in the video.

Description

Generator of the arbitrary shape frame and method for generating the arbitrary shape frame using the same

The present invention relates to an image decoding system for processing arbitrary shapes. More particularly, the present invention relates to an arbitrary shape frame generator for inserting a new frame between two frames having different shape information and texture information, and an arbitrary shape using the same. It relates to a shape frame generation method.

Recently, research on object-based image encoding technology has been actively conducted. In the case of object-based image, the shape of the image is arbitrary shape and the size of the image is not fixed with time. It is added. The same is true for arbitrary shapes in order to send as little information as possible in order to improve the transmission efficiency of the transmission medium and to use it efficiently to obtain better image quality. Therefore, screen insertion in a random shape decoding system can lead to a more natural video switching process, and when the amount of the result stream is controlled in accordance with the capacity of the transmission channel in the video encoding system, it is processed as a frame skip. It is possible to generate a screen that is skipped relatively naturally on the decoder side for the frame.

MPEG 4 video is used as a standard for efficient storage, transmission, and manipulation in video data processing in a multimedia environment. MPEG 4 aims to meet standards of use, such as efficient image data compression, image object processing, spatial resolution, and temporal flexibility. For this purpose, the context based visual data representation approach is basically used. This represents a screen as a set of video objects (VO), and one video object has shape, motion, and texture information. The MPEG 4 coded decoding unit is a video object (VO), that is, a video object plane (hereinafter referred to as a VOP) at a unit time, and the VOP may have an arbitrary shape, and thus, for one video object (VO). The size of the VOP varies over time. The structure of an MPEG 4 encoder is shown in FIG. 1.

The shape coding block 1 encodes the shape information of the VOP. The operation reads the shape information of the VOP, and if the shape information of the corresponding block is '0' for each macro block (MB, 16x16 pixels), that is, if it is a macro block to be transparently processed on the screen, Shape information is encoded after separation into 255 '(when the pixel depth is 8 bits), that is, a macro block to be opaquely processed. Shape characteristics such as transparency and opacity imparted to each macro block MB by the shape coding block 1 may be determined by a motion estimation block 2, a motion comparison block 3, and texture coding ( Texture Coding) is sent to block (4) to disable motion and texture coding for transparent blocks.

The texture coding block 4 of FIG. 1 encodes Y, U, and V information of a VOP, and uses an intra picture encoding method and an inter picture encoding method. In the case of using the intra coding method, DCT (discret consine transform) is performed on the Y, U, and V information of the VOP, and then quantized, and the result is subjected to variable length coding (VLC). In the case of using the inter-screen encoding method, only the difference from the current video is exported with reference to the previous video. Referring to the previous image stored in the previously reconstructed VOP 5 in FIG. 1, the motion estimation block 2 may select a region having the smallest difference from the corresponding macro block among the previous image regions for each macro block. The motion vector pointing is obtained, and the motion comparison block 3 completes the prediction VOP using the previous image and the motion vector. In the code between pictures, the texture coding block 4 receives the difference between the prediction VOP and the current VOP as an input, encodes the result, and then outputs the result. Reference numeral 6 denotes a multiplexer MUX for multiplexing shape information, motion information, and texture information, and reference numeral 7 denotes an output buffer of the multiplexer 6.

In particular, shape and alpha map coding are used to realize arbitrary shape VOP, and alpha map is used to express shape information of the screen in addition to texture information of the image. . The alpha map can be divided into binary alpha and gray scale alpha. The former represents the shape information in two levels, '0' and '255' (when the pixel resolution is 8 bits). On the other hand, gray scale alpha may have all level values that can be expressed as the resolution of a pixel, thereby representing transmittance. Arbitrary shape VOP is shown in FIG.

The upper left reference point of the VOP (the upper right corner of the rectangle (part A) shown in FIG. 2) contains all of the shape objects, with the smallest number of macro blocks (16x16 pixels) in MB. It becomes a horizontal and vertical even-numbered pixel that can be included, and the lower right side is the maximum value (right max, bottom max) of the right and bottom of any object. In Fig. 2, the rectangle A of the thick line becomes an arbitrary shape VOP. As a result, the arbitrary shape VOP has a width, height size Y (luminance), alpha (alpha map) and width / 2, length / 2 size (4: 2: 0 format) determined as shown in FIG. If U), V (color signal). Examples of arbitrary shape VOPs are the same as in FIGS. 3A and 3B.

FIG. 3A illustrates a VOP including both shapes and textures. FIG. 3B illustrates a shape portion (alpha map). When the pixel resolution is 8 bits, the alpha map value of the pixel is '255' (opaque). ) Are black and '0' (transparent) is white.

In object-based digital image compression and reconstruction, the size of a VOP representing an object image at a specific time, that is, the size of a frame, is not constant over time. Therefore, if there is a need for screen insertion in the decoder, a technique for generating a new frame through two arbitrary shape images having different sizes is needed.

Therefore, the present invention updates the frame so that the frame size is the same before and after the frame through simple frame matching using simple shape information, and generates the insertion frame by simple linear interpolation using the frame before and after the same size. Accordingly, an object of the present invention is to provide an arbitrary shape frame generator and a method of generating an arbitrary shape frame using the same, which can solve the above disadvantages.

According to an aspect of the present invention, there is provided an arbitrary shape frame generator for inserting a new frame between two frames having different shape information and texture information, comprising: an arbitrary shape decoder for outputting a restored arbitrary shape VOP; The VOP stored in the previous frame buffer and the after frame buffer for storing the restored arbitrary shape VOP output from the arbitrary shape decoder, and the VOP stored in the previous frame buffer and the after frame buffer, the shape of the VOP when the size of the two frames are different An arbitrary shape frame matcher which finds a matching point of two frames through frame matching using information, and outputs the matching point and mode value, the maximum width and the maximum length of two frames required for updating, and the random shape frame matcher. Receive the data and according to the mode, the previous frame buffer and the next frame A linear shape interpolation is performed between the arbitrary shape frame updater for updating and outputting the VOP stored in the buffer, the temporary buffer for storing the updated VOP output from the arbitrary shape frame updater, and the data stored in the previous frame buffer and the after frame buffer. It characterized in that it comprises a frame generator for generating a frame.

In addition, the present invention for achieving the above object is an arbitrary shape frame generation method for inserting a new frame between two frames of different sizes having shape information and texture information, the arbitrary shape VOP restored in the arbitrary shape decoder Storing the previous frame buffer and the next frame buffer, and inputting the VOPs stored in the previous frame buffer and the next frame buffer to an arbitrary shape frame matcher. Finding a matching point of two frames through frame matching using the frame matching, and outputting the matching point, the mode value, the maximum width and the maximum length of two frames required for the update to an arbitrary shape frame updater, and the arbitrary shape frame updater VO stored in the previous frame buffer and the after frame buffer according to the mode Updating and storing P in a temporary buffer; and generating a new frame by linearly interpolating data stored in the previous frame buffer and the next frame buffer.

The frame of T-1 time is stored in the previous frame buffer, the frame of T time is stored in the post frame buffer, and the VOP includes shape information, texture information, and position information in a background screen.

In the frame matching, the VOP size of the VOP (T-1) is referred to as X (T-1) (horizontal pixels, short width), Y (T-1) (vertical pixels, short length), and VOP ( When the VOP size of T) is X (T) (long width) and Y (T) (long length), x + X (T-1) at x of the VOP (T-1) and VOP (T). Difference in the alpha map from y to y + Y (T-1) (x: 0 to X (T) -X (T-1), y: 0 to Y (T) -Y (T-1)) And a matching point x, y for minimizing the difference between the alpha maps.

In the frame matching, the VOP size of the VOP (T-1) is referred to as X (T-1) (horizontal pixels, long width), Y (T-1) (vertical pixels, long length), and VOP ( When the VOP size of T) is X (T) (short width) and Y (T) (short length), from x to x + X (T) of the VOP (T) and VOP (T-1), calculating the difference between y + Y (T) (x: 0 ~ X (T-1) -X (T), y: 0 ~ Y (T-1) -Y (T)) in y And obtaining matching points x and y that minimize the difference between the alpha maps.

And storing the updated VOP stored in the temporary buffer in the previous frame buffer and the after frame buffer.

Since a frame of an arbitrary image does not have a fixed screen size, a different method must be used for frame insertion. The present invention proposes a technique for generating a frame to be inserted in a simple manner by matching the size of the frame before and after through frame matching using shape information that is relatively simple to process.

In the present invention, a frame matching algorithm using an alpha map is proposed, and a method of generating the frame matching algorithm using a small buffer in a short time is implemented. The present invention is expected to be utilized in an application field for reconstructing arbitrary images in real time.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

4 is a block diagram of an arbitrary shape frame generator according to the present invention.

The arbitrary shape VOP restored in the arbitrary shape decoder 101, that is, the frame, is stored in the previous frame buffer 105 and the after frame buffer 104. The frame of time T-1 is stored in the previous frame buffer 105, and the frame of time T is stored in the frame buffer 104 afterwards. As in the case of a normal arbitrary shape decoder, it is assumed that the VOP stored in the buffer includes both the shape information and the texture information, so that the location information (horizontal spatial reference, vertical spatial reference) in the background of the current arbitrary shape VOP is included. .

The VOPs stored in the before frame buffer 105 and the after frame buffer 104 are input to the arbitrary shape frame matcher 102. If the size of the two frames is different, the shape matching of the VOP, that is, frame matching using the alpha map is performed. The matching point of the two frames is found, and the arbitrary shape frame updater 103 outputs the matching point, the mode value to be described later, the maximum width and the maximum length of the two frames required for the update.

The matching point extraction operation of the arbitrary shape frame matcher 102 is shown in FIG. 5. For example, if the alpha map of the VOP at time T-1 is equal to A (short width, short length) in FIG. 5, and the alpha map of the VOP at time T is equal to B (long width, long length), then To insert a frame into the frame, the average motion of two VOPs must be expressed for a natural screen transition. For an arbitrary shape VOP, this can be obtained relatively simply using alpha map data. In the case of alpha map data, it can be expressed as '255' and '0'. If the two frames are matched by simply mapping them to '1' and '0', the operation becomes quite simple. In the case of gray scale alpha, the same method can be used through preprocessing that separates the case of '0' and the case of '0' and '1'.

The frame matching method is performed by the same steps as C of FIG. 5. The VOP size of the VOP (T-1) is called X (T-1) (horizontal pixels, short width), Y (T-1) (vertical pixels, short length), and the VOP of VOP (T) If the size is X (T) (long width), Y (T) (long length), then x to x + X (T-1) of VOP (T-1) and VOP (T), y to y Find the difference in the alpha map of + Y (T-1) (x: 0 ~ X (T) -X (T-1), y: 0 ~ Y (T) -Y (T-1)) Find the smallest matching point (x, y). However, since 4: 2: 0 format is mainly used here, x and y are performed only for even numbers to make x / 2 and y / 2 integers to be applied to U and V. If it is obtained by hardware, it can be implemented simply by AND of the pixel-by-pixel AND of the alpha maps of two VOPs, and can be easily implemented. As described above, the mode 1 is used when the frame is large both horizontally and vertically.

Unlike the above example, when VOP (T-1) is larger in width and length than VOP (T), that is, the VOP size of VOP (T-1) is X (T-1) (horizontal pixels, long). Width), Y (T-1) (vertical number of pixels, long length), and the VOP size of VOP (T) is X (T) (short width), Y (T) (short length), Contrary to the above case, from x to x + X (T) of VOP (T) and VOP (T-1), y to y + Y (T) (x: 0 ~ X (T-1) -X (T) , y: 0 to Y (T-1) -Y (T)) to find the alpha map difference and find the matching points x and y that minimize the difference. Mode 2 is used when all the frames are large in both the horizontal and vertical directions.

In addition, if there is no larger frame in both the horizontal and vertical directions in both frames as shown in FIG. As shown in FIG. 6, the difference in the shape information of the two frames is added to a region where two frames overlap each other, that is, a semi-transparent rectangular area D in FIG. 6C, and the position where the result is the smallest is a matching point. . That is, for two VOPs of VOP1 (l_width, s_length) and VOP2 (s_width, l_length), VOP1 is vertically y: 0 ~ (l_length-s_length), and VOP2 is horizontal. Move x: 0 ~ (1_width-s_width) in the direction, and find the minimum x and y by overlapping the overlapping rectangle, that is, the translucent rectangular area (D) of Figure C. Mode 3 is set as the case where the VOP1 type, that is, the long width and the short length type, is the previous frame.

The arbitrary shape frame updater 103 receives the mode information, the maximum VOP width of the two VOPs, the maximum VOP length, the matching pointers x and y from the arbitrary shape frame matcher 102, and the entire frame buffer 105 according to the mode. After updating the VOP using the data of the frame buffer 104, the updated VOP is stored in the temporary buffer 107. The result stored in the temporary buffer 107 is stored in the before and after frame buffers 105 and 104 as necessary. An operation example of the arbitrary shape frame updater 103 is shown in FIG.

Mode 1: The previous frame is smaller than the next frame

The arbitrary shape frame updater 103 corrects and positions the data of the previous frame by the matching point in the temporary buffer 107 and pads the data to fit the next frame size, that is, the maximum VOP width and the maximum VOP length. Padding uses the normal outermost pixel values. For alpha and Y (luminance) components, compensate with x, y and pad with max width and max length, for U and V with x / 2 and y / 2 and pad with max width / 2 and max length / 2 do. The value of the temporary buffer 107 is written to the previous frame buffer 105. As a result, the size of the updated previous frame is the same as the size of the next frame, so that it can be easily processed.

Mode 2: The previous frame is larger than the next frame

The arbitrary shape frame updater 103 corrects and positions the data of the next frame in the temporary buffer 107 by a matching point, and pads to fit the previous frame size, that is, the maximum VOP width and the maximum VOP length. The value of the temporary buffer 107 is written to the later frame buffer 104.

Mode 3: The previous frame is wider and shorter than the next frame

The maximum width is the width of the previous frame and the maximum length is the length of the next frame. In mode 3, both before and after frames are updated to the maximum width and the maximum length to fit the size. First, the data of the previous frame buffer 105 is corrected in the temporary buffer 107 using the y component of the matching point, and padded up and down according to the maximum length, and then stored in the previous frame buffer 105. After the previous frame processing is completed, the next frame is corrected in the temporary buffer 107 using the x component of the matching point and padded left and right according to the maximum width, and stored in the later frame buffer 104.

Mode 4: The previous frame is narrower and longer than the next frame

The maximum width is the width of the next frame, and the maximum length is the length of the previous frame. First, the data of the previous frame buffer 105 is corrected using the x component of the matching point in the temporary buffer 107 and padded left and right to fit the maximum width, and then stored in the previous frame buffer 105. After the previous frame processing, the data of the later frame buffer 104 is corrected using the y component of the matching point in the temporary buffer 107, and padded up and down according to the maximum length, and stored in the subsequent frame buffer 104. .

The frame generator 106 linearly interpolates the data of the pre-frame buffer 105 and the post-frame buffer 104 newly stored by the arbitrary shape frame updater 103 with the same size of the pre- and post-frame. To create a new frame. In addition, the position information in the background screen of the new frame is set as follows by using the mode information (mode 1 to mode 4) and the matching point information (x, y) from the arbitrary shape frame matcher 102.

In case of Mode 1, the average value of (horizontal spatial reference-x), (vertical spatial reference-y) of the previous frame and the horizontal spatial reference and vertical spatial reference of the next frame is used.

In mode 2, the average values of the horizontal spatial reference, the vertical spatial reference of the previous frame, and the (horizontal spatial reference-x) and (vertical spatial reference-y) of the previous frame are used.

In mode 3, the average of the horizontal spatial reference (vertical spatial reference-y) of the previous frame and the horizontal spatial reference (x) of the previous frame is used.

In mode 4, the average value of the previous frame (horizontal spatial reference-x), the vertical spatial reference-y and the later frame horizontal spatial reference, (vertical spatial reference-y) is used.

As described above, the present invention uses the shape information that is simple to process, and updates the frame so that the size of the frame before and after is the same through simple frame matching. An insertion frame is generated by simple linear interpolation using before and after frames updated to the same size.

In addition, by setting the position of the object image in the background image of the embedded image in consideration of the position information of the front and rear frames, it is possible to realize a natural screen switching of the embedded image in the background image.

When the frame matcher of the present invention is implemented in hardware using the shape information (which can be mapped to two level values of '0' and '1'), the shape matcher can be easily implemented in an application field requiring real time.

1 is a block diagram of an MPEG 4 video object plane (VOP) encoder.

2 is an exemplary diagram for explaining extraction of arbitrary shape VOP.

3A and 3B are exemplary views for explaining an arbitrary shape VOP and an alpha map of the VOP.

4 is a block diagram of an arbitrary shape frame generator of the present invention.

5 and 6 are exemplary diagrams for explaining the operation of the arbitrary shape frame matcher of FIG.

7 is an exemplary diagram for explaining the operation of the arbitrary shape frame updater of FIG.

<Explanation of symbols for the main parts of the drawings>

1: shape coding block 2: motion estimation block

3: motion comparison block 4: texture coding block

5: Restored VOP 6: Multiplexer

7: buffer 101: arbitrary shape image decoder

102: arbitrary shape frame matcher 103: arbitrary shape frame updater

104: after frame buffer 105: before frame buffer

106: frame generator 107: temporary buffer

Claims (8)

In the arbitrary shape frame generator for inserting a new frame between two frames having different shape information and texture information, An arbitrary shape decoder for outputting the restored arbitrary shape VOP, A pre-frame buffer and a post-frame buffer for storing the restored arbitrary shape VOP output from the random shape decoder; VOPs stored in the previous frame buffer and the next frame buffer are input, and when the sizes of the two frames are different, the matching points of the two frames are found through frame matching using the shape information of the VOP, and the matching points, the mode values, and the update are performed. An arbitrary shape frame matcher that outputs the maximum width and the maximum length of the two frames required for An arbitrary shape frame updater which receives data from the arbitrary shape frame matcher and updates and outputs VOPs stored in the previous frame buffer and the next frame buffer according to a mode; A temporary buffer for storing the updated VOP output from the arbitrary shape frame updater; And a frame generator for linearly interpolating data stored in the previous frame buffer and the next frame buffer to generate a new frame. In the arbitrary shape frame generating method for inserting a new frame between two frames having different shape information and texture information, A random shape VOP restored by the random shape decoder is stored in the previous frame buffer and the after frame buffer, Inputting a VOP stored in the previous frame buffer and the next frame buffer to an arbitrary shape frame matcher, If the size of the two frames is different, the shape frame matcher finds a matching point of the two frames through frame matching using the shape information of the VOP, and the matching point and mode value, the maximum width and the maximum length of the two frames required for updating. Outputting an arbitrary shape frame updater, The arbitrary shape frame updater updating a VOP stored in the previous frame buffer and the after frame buffer according to a mode and storing the VOP in a temporary buffer; And a frame generator linearly interpolating data stored in the before frame buffer and the after frame buffer to generate a new frame. The method of claim 2, wherein a frame of T-1 time is stored in the previous frame buffer, and a frame of T time is stored in the post frame buffer. The method of claim 2, wherein the VOP includes shape information, texture information, and location information in a background screen. 3. The frame matching of claim 2, wherein the frame matching sets the VOP size of the VOP (T-1) to X (T-1) (horizontal pixels, short width), Y (T-1) (vertical pixels, short length). ) And the VOP size of VOP (T) is X (T) (long width) and Y (T) (long length), From x to x + X (T-1) of VOP (T-1) and VOP (T), y to y + Y (T-1) (x: 0 to X (T) -X (T-1) , y: calculating a difference between alpha maps of 0 to Y (T) -Y (T-1)), And obtaining matching points x and y for minimizing the difference between the alpha maps. 6. The method of claim 5, wherein x and y are even. 3. The frame matching method of claim 2, wherein the frame matching sets the VOP size of the VOP (T-1) to X (T-1) (horizontal pixels, long width), Y (T-1) (vertical pixels, long length). ) And the VOP size of VOP (T) is X (T) (short width) and Y (T) (short length), From x to x + X (T) of the VOP (T) and VOP (T-1), y to y + Y (T) (x: 0 ~ X (T-1)-X (T), y: Obtaining a difference between alpha maps of 0 to Y (T-1) -Y (T)), And obtaining matching points x and y for minimizing the difference between the alpha maps. 3. The method of claim 2, further comprising storing the updated VOP stored in the temporary buffer in the pre frame buffer and the post frame buffer.