KR100878640B1 - Image frame processing method and device for displaying moving images to a variety of displays - Google Patents

Abstract

The rendering process for rendering the image frame and the post-processing for changing the image frame to suit the display device are separated. The rendering processing unit 42 generates an image frame sequence by performing rendering at a predetermined frame rate irrespective of the condition that the image frame must be satisfied to output to the display device. The post-processing unit 50 merges the image frame sequence generated by the rendering processing unit to generate and output an updated image frame sequence satisfying the above condition. Since the rendering process and the post-processing are separated, the image frame sequence can be generated regardless of the display device's specifications such as the display device's resolution and frame rate.

Image Frames, Rendering, Post-Processing, Image Frame Sequences, Moving Pictures, Fast Forward

Description

Image frame processing method and device for displaying moving images to a variety of displays}

TECHNICAL FIELD The present invention relates to an image frame processing technique, and more particularly, to an image frame processing technique for generating an image frame sequence modified to suit various rendering conditions.

BACKGROUND ART With the improvement of manufacturing technology and lowering the price of thin display devices such as liquid crystal display and plasma display, various display devices for reproducing moving images are currently around us.

The specification of displayable image data including frame rate and resolution varies depending on the type of display device. According to the prior art, image data output from an image rendering device is processed in a display device to generate an image adapted to be suitable for the display device. In these adjustments, there is an increase in manpower required to develop circuits or software due to the increase in various frame rates and resolutions that must be adapted to the display. The processing load on the display device is also increased in connection with this.

The present invention has been made in view of the above problems, and an object of the present invention is to provide an image frame processing technique suitable for various kinds of display devices.

One aspect of the present invention relates to an image frame processing method. Such a method includes a rendering process of rendering an image frame sequence and a post process of changing the image frame sequence generated by the rendering process to be suitable for the display device. In the rendering process, an image frame sequence is generated by performing rendering at a predetermined frame rate irrespective of a condition that an image frame must satisfy to output to a display device. In post-processing, predetermined processing is applied to the image frame sequence generated by the rendering process to generate and output an image frame sequence satisfying the above conditions.

Since the rendering process and the post-processing are separated, the image frame sequence can be generated regardless of the display device's specifications such as the display device's resolution and frame rate.

Another aspect of the present invention is a moving picture display method of reading an image frame sequence from a memory, and performing a predetermined process to display the image frame sequence, wherein the image is displayed as a quick turn picture from the memory when a quick turn request occurs. Selectively reading the image frame, generating an updated image frame by performing a predetermined integration process on the image frame read image frame, and displaying the updated image frame.

According to this aspect, since the quick turn frame is generated by integrating a plurality of image frames, a quick turn image having added value can be obtained. Here, the "integration process" refers to a process of generating an updated image frame using some or all of the image information included in the plurality of image frames.

Embodiments of the present invention in the form of methods, apparatuses, systems, computer programs and recording media may be practiced as additional methods of the present invention.

1 is a diagram illustrating a hardware structure of an entertainment device.

2 is a functional block diagram of an image frame processing apparatus.

3 is a diagram illustrating a primitive coordinate system.

4 is a diagram showing how the offset value changes every four frames in the image frame sequence output from the rendering processing unit.

5 is a diagram illustrating point sampling including four image frames.

FIG. 6 is a diagram showing the generation of an image frame four times the size of the original four image frames.

FIG. 7 is a diagram showing motion blur using four picture frames. FIG.

8 is a diagram showing merging four image frames to produce one image frame of the same size.

FIG. 9 is a diagram illustrating generating an enlarged image by performing point linear sampling on four image frames and then performing bilinear interpolation.

FIG. 10 is a diagram illustrating bilinear sampling. FIG.

11 is a flowchart illustrating determining a merging method in a merging condition setting unit.

12 is a diagram illustrating a hardware configuration of the image frame processing apparatus 200 according to the second embodiment of the present invention.

13 is a functional block diagram of an image frame processing apparatus according to the fourth embodiment.

14 is a diagram conceptually illustrating a process of extracting an image frame from an image frame sequence and generating a quick turn frame.

15 is a diagram conceptually illustrating a process of generating a quick turn frame by merging a plurality of image frames.

16 is a diagram conceptually illustrating a process of generating a rotated image by reducing the number of merged image frames and at a reduced rotational speed.

17 is a functional block diagram of an image frame processing apparatus according to the fifth embodiment.

18 is a diagram conceptually illustrating a process of extracting an image frame based on luminance information.

19 is a functional block diagram of an image frame processing apparatus according to the sixth embodiment.

20 is a diagram conceptually illustrating a process of separating an image frame into a specific region and a non-specific region.

21 is a functional block diagram of an image frame processing apparatus according to the seventh embodiment.

22 is a diagram conceptually illustrating a path display process.

23 is a functional block diagram of an image frame processing apparatus capable of executing a process according to the fourth to seventh embodiments.

(1st embodiment)

An image frame processing apparatus according to the present invention will be described by introducing an embodiment of an inventive apparatus applied to an entertainment apparatus for rendering a three-dimensional computer graphics (CG) image.

1 shows a hardware structure of the entertainment device 100. The entertainment apparatus 100 includes a graphics chip 18, and by performing rendering, it is possible to display a three-dimensional image on the display device 26 in real time.

The entertainment apparatus 100 includes a main CPU 12, a main memory 14, a shape processor 16, a graphics chip 18, a display controller 28, and an input / output port (I / O port) 30. These blocks may be connected to each other through the graphic bus 36 to transmit and receive data with each other. The display controller 28 may be connected to one of various display devices 26 each having different display conditions and specifications.

The input / output port 30 is connected to an external storage device 32 such as a CD-ROM drive, a DVD-ROM drive, or a hard disk drive, and a keyboard or mouse for inputting key input data and coordinate data to the entertainment device 100. It is connected to an input device 34 such as. The input / output port 30 controls data entering and exiting both the external storage device 32 and the input device 34. The input / output port 30 reads rendering data or programs stored in the external storage device 32 and provides them to the main CPU 12 and the shape processor 16. For example, the rendering data may be object data for the rendered object. The input / output port 30 may be configured to communicate with another device to receive rendering data or a program.

The main CPU 12 controls the entertainment device 100 as a whole and executes a rendering program stored in the external storage device 32. When the program is executed, the main CPU 12 controls the image display by controlling the graphics chip 18 in accordance with an input from a user using the input device 34.

The main CPU 12 controls the entertainment device 100 by controlling the data transfer between the component devices. For example, the main CPU 12 controls the transfer of the shape data generated by the shape processor 16 to the graphics chip 18 using the main memory 14 as a buffer. The main CPU 12 also manages data transfer synchronization between the graphics chip 18, the external storage device 32, the input device 34 and the display device 26. In this embodiment, the shape processor 16 and the main CPU 12 are provided separately. In other embodiments, these components may be incorporated to allow the main CPU 12 to perform the functions of the shape processor 16.

  The main memory 14 stores the object type data and the rendering program read from the external storage device 32. Each object data includes vertex data of a plurality of polygons constituting a related object. The main memory 14 has a texture buffer area used for texture mapping.

Under the control of the main CPU 12, the shape processor 16 performs shape processing on object data stored in the main memory 14 such as coordinate shifting or transformation to define a position or shape, or the light source brightens vertices. Do the processing related to doing The shape data obtained as a result of the shape processing includes object attributes such as vertex coordinates of the object, texture coordinates at the vertices, and vertex luminance.

The graphics chip 18 includes a rendering processor 20, a memory interface 22, and an image memory 24 such as EDRAM. The rendering processor 20 sequentially reads shape data generated by the shape processor 16 under the control of the main CPU 12, and performs rendering processing on the shape data to generate an image frame. RGB values and alpha values indicating transparency of pixels in the image frame are stored in the image memory 24. The Z value representing the depth of the pixel is stored in the Z buffer area (not shown). The Z buffer area may exist in the picture memory 24.

The rendering processor 20 of the graphics chip 18 renders an image frame in the image memory 24 via the memory interface 22 according to a rendering instruction provided from the main CPU 12. A high-speed bus connection is formed between the rendering processor 20 and the memory interface 22 and between the memory interface 22 and the image memory 24 so that the rendering processor 20 Rendering processing can be performed in the image memory 24 at a high speed. As an example, the rendering processor 20 renders an image frame of 640 * 480 pixels.

The picture frame rendered by the rendering processor 20 is temporarily stored in the picture memory 24. The main CPU 12 retrieves the image frame from the image memory 24 via the memory interface 22 and writes the image frame to another memory such as the main memory 14. The main CPU 12 converts the image frame into an image frame which can be displayed on the display device 26. The display controller 28 then receives the image frame via the bus 36 and displays it on the display device 26.

2 is a functional block diagram of the image frame processing apparatus 10. The functions of FIG. 2 are mainly implemented by the graphics chip 18, the main CPU 12, and the main memory 14. 2 is a diagram focusing on functionality. Thus, these functions may be variously implemented by hardware only, software only, or a combination of hardware and software.

The object data reading unit 40 reads shape data of an object for rendering. The rendering processing unit 42 sequentially renders image frames having an object having a predetermined resolution at a predetermined image ratio. The rendered picture frame is stored in the first memory 44 as a buffer. The rendering processing unit 42 renders the image frame at a frame rate equal to or higher than the highest frame rate of the display device 26 to be used. For example, the rendering processing unit 42 corresponds to the rendering processor 20 of FIG. 1.

The transfer controller 46 reads an image frame stored in the first memory 44 and stores it in the second memory 48. The second memory 48 may store a plurality of image frames to identify a time-order between the plurality of image frames. For example, the first memory 44 corresponds to the image memory 24 of FIG. 1, and the second memory 48 corresponds to the main memory 14. In another embodiment, the second memory may be a memory such as the external storage device 32 included in the image frame processing apparatus 10 or any storage device. In another embodiment, each of the first memory 44 or the second memory 48 may correspond to another memory area in the physically the same memory.

The interface unit 52 obtains resolution or frame rate information of the display device 26 connected to the image frame processing apparatus 10. The interface unit 52 may obtain content information such as information indicating whether the image is a still image or a moving image from the main CPU 12 or the rendering program. The interface unit 52 may obtain resolution or frame rate information of the display device from the user through the input device 34. The obtained information is transferred to the post processing unit 50. For example, the post processing unit 50 may correspond to the main CPU 12 of FIG. 1.

The post-processing unit 50 includes a merging condition setting unit 54, a frame sequence obtaining unit 56, and a merging execution unit 58. The post-processing unit 50 performs post-processing on the image frame sequence rendered by the rendering unit 42 and stored in the second memory 48 to generate an image that can be displayed on the display device.

In particular, the merging condition setting unit 54 sets an appropriate merging condition for the image frame sequence based on the information received from the interface unit 52. The description thereof will be described with reference to FIG. 10 to be described later. The frame sequence acquiring unit 56 retrieves a sequence of image frames according to the conditions set by the merging condition setting unit 54 and transfers the retrieved image frames to the merging execution unit 58. The merge execution unit 58 executes merge processing on the received image frame. "Merge process" here means to generate one image frame from a plurality of image frames.

The image frame generated by the merging process by the merging execution unit 58 is output to the image display unit 60 at the frame rate set by the merging condition setting unit 54.

As described above, one of the features of the first embodiment is that "rendering" and "post-processing" of the image frame sequence are not executed on the same chip but are executed separately. An advantage of separating rendering and post-processing is that it eliminates the dependency on the type of display device connected to the image frame processing apparatus.

In particular, even if the interface unit 52 obtains a condition that the image frame output to the display device must satisfy, the rendering processing unit 42 generates the image frame sequence regardless of the condition obtained by the interface unit 52. do. As a result, the post-processing unit 50 applies predetermined processing to the image frame sequence generated by the rendering processing unit 42 according to the conditions obtained by the interface unit 52 to update the updated image frame suitable for the display device. Create a sequence. As a result, the rendering method in the rendering processing unit 42 does not need to be changed depending on the condition that the image frame must satisfy. Therefore, the rendering processing unit 42 only needs to have a general structure. When the display device connected to the image frame processing apparatus 10 is changed to another kind, the change is adjusted by changing the processing in the post-processing unit 50. Therefore, more various kinds of display devices can be connected to ensure a high level of compatibility. This will eventually reduce the manpower required to develop circuits or software for modifying the processing device to suit various displays.

In addition, according to the prior art, even if the rendering processor has a high rendering capability, it is necessary to limit the rendering capability of the rendering processor because the specification of the display device does not change the rendering capability. However, according to the first embodiment, the capability of the rendering processor is not limited and can be fully utilized. Post-processing is adjusted so that the ability to render at high frame rates is assigned to improving resolution, and the like. By ensuring that the post-processing has a high versatility, flexibility in the design of the image frame processing apparatus will be improved. In addition, since the display device does not perform image frame processing, the processing load applied to the display device will also be reduced.

An additional consideration is that when the frame rate of rendering a moving image, such as animation video, changes, an animated image representing movement different from that intended by the image creator may be displayed on the display device. According to the prior art, one approach to dealing with this problem is to prepare a plurality of image sequences in consideration of the frame rate of the image frame finally displayed on the display device, so that an optimal viewing experience can be enjoyed at any frame rate. will be. In contrast, according to the first embodiment, it is sufficient that only one image frame sequence having a high frame rate is prepared regardless of the frame rate of the display device.

The first memory 44 operates as a frame buffer that stores the image frame sequence rendered by the rendering processing unit 42 into one frame. The picture frame temporarily stored in the first memory 44 is eventually transferred to the second memory 48. Thus, the second memory 48 primarily acts as a work area for the post processing unit 50. The first memory 44, which acts as a frame buffer, is generally implemented with expensive EDRAM or the like. If there is no second memory in this embodiment, as described below, the first memory needs to be a large memory because a maximum of four frames must be stored for the merge process for four frame times. By having the second memory 48, the first memory 44 only needs to have a capacity for storing at least one image frame rendered by the rendering processing unit 42. Therefore, it is advantageous to provide the second memory 48 separately from the first memory 44 as a work area for the post-processing unit 50. The first memory 44 is formed in the same semiconductor circuit element as the semiconductor circuit element in which the rendering processing unit 42 is formed.

3 shows a primitive coordinate system in which a pixel of each image frame is located in the rendering processing unit 42. The horizontal axis is represented by x and the vertical axis is represented by y. The arrangement of pixel coordinates is represented by (x, y). Each of the x and y coordinate values is a fixed-point value represented by an integer part of 12 bits and a fraction part of 4 bits. As shown, the center of each pixel is located at the integer point of the coordinate in the primitive coordinate system. The rendered image frame is stored in a pixel by pixel in the first memory. The coordinate system for indicating a location in the first memory is called a window coordinate system. Memory address calculation is done using this coordinate system. The window coordinate system is a coordinate system for indicating a position in the frame buffer using the upper left point of the rectangular area of the buffer as the origin.

Given primitive coordinates are (Px, Py) and offsets are (Offx, Offy), the window coordinates are given by

Wx = Px-Offx

Wy = Py-Offy

Some embodiments in which the image frame processing apparatus shown in FIG. 2 is used to generate an image frame sequence modified to suit various display conditions in post processing will be described. The rendering processing unit 42 is assumed to render an image of 640 * 480 at a frame rate of 240 frames per second (hereinafter referred to as fps).

(First embodiment)

4 shows how the offset value changes every four frames in the image frame sequence output from the rendering processing unit 42. For convenience of explanation, it is assumed that picture frames F1, F2, F3 and F4 are generated by rendering in the above-mentioned order. The rendering processing unit 42 renders the first picture frame F1 without an offset, renders the second picture frame F2 at an offset of (0.5, 0), and renders the third picture frame F3 at an offset of (0, 0.5). Then, the fourth image frame F4 is rendered at an offset of (0.5, 0.5). The offset in rendering by the rendering processing unit 42 is affected by the coordinates continuously displaced as the starting point of the rendering in the rendering space. In the following, this processing is referred to as "pixel displacement processing".

5 and 6 show a first embodiment of the merging process. In the first embodiment, the sequence of frames is merged to produce an image frame four times the size of the rendered image frame.

5 is a schematic diagram showing how pixels rendered by the rendering processing unit 42 with pixel displacements are located in the same window coordinate system. 5 and 6, circles indicated by "1" indicate pixels of the first image frame F1, circles indicated by "2" indicate pixels of the second image frame F2, and circles indicated by "3". Denotes a pixel of the third image frame, and a circle denoted by "4" denotes a pixel of the fourth image frame F4.

The distance between the centers of adjacent pixels in each frame is "1" in both the x and y directions. As a result of the pixel displacement processing by the rendering processing unit 42, the pixels of the picture frame F2 are displaced by 0.5 in the x direction with respect to the combined pixels of the picture frame F1, and the pixels of the picture frame F3 are combined in the picture frame F1. It is displaced by 0.5 in the y direction with respect to the pixel, and the pixel of the picture frame F4 is displaced by 0.5 in the x direction and by 0.5 in the y direction with respect to the combined pixel of the picture frame F1. Therefore, when four image frames are located in the same coordinate system, as shown in Fig. 5, the pixels of each image frame are distanced by 0.5 in both the x direction and the y direction.

By lattice sampling in 0.5 pixel units instead of 1 pixel units in the window coordinate system, an image frame having twice the pixels in the x direction and the y direction can be generated. This is described with reference to FIG. 6. 6 is a schematic diagram showing how pixels are aligned. 6 shows only 4 pixels horizontally and 3 pixels vertically to simplify the picture frames 102, 104, 106, and 108, but in practice there are 640 pixels horizontally and 480 pixels vertically in each frame. . The entire pixel is aligned in the relationship shown in image frame 110 in FIG. By aligning the 640 × 480 pixel image frames 102, 104, 106, and 108 in the grid as described above, the image frame 110 is 1280 × 960 pixels, which is four times the size of the 640 × 480 pixel image frame. Is made. In the following, this sampling method is referred to as "point sampling".

According to the first embodiment, a plurality of image frames are generated by rendering with spatial displacement in the rendering processing unit 42. As a result, the post-processing unit 50 performs a merge process on the image frames maintaining the displacement between the image frames, thereby processing the image frame sequence at a higher spatial resolution than the image frame sequence output from the rendering processing unit 42. Create The phrase "keeping displacement between picture frames" means using the pixels of each offset picture frame without any modification to obtain the final picture frame. As a result, an image frame sequence changed to be suitable for different resolutions of different display devices is generated by post-processing.

The first embodiment can also be understood as a means for reducing the frame rate. For example, by generating an image frame from four image frames as shown in FIG. 3, the frame rate will be reduced to one quarter. In a situation where the maximum frame rate of the display device is lower than the frame rate of the rendering processing unit 42, there is an advantage that a low frame rate and a high resolution image can be obtained by performing point sampling.

The present embodiment is not limited to generating one image frame from four image frames. In another embodiment, by generating one image frame from nine image frames, it is possible to generate an image frame having nine times more pixels than the original image frame. The same is true when including more image frames. The larger the number of image frames, the lower the frame rate of the finally obtained image frame.

(2nd Example)

7 and 8 show a second embodiment of the merging process. In the second embodiment, the motion blur effect is achieved by merging consecutive frames.

As shown in Fig. 7, open circles represent pixels of image frames F1 to F4 subjected to pixel displacement processing. In this embodiment, the average RGB value is determined by the RGB values of four adjacent pixels, and the resulting value is used as the new RGB value. An image frame is created centering on the midpoint between the centers of the four empty circle pixels, with each pixel shown as diagonally hatched circles in FIG.

8 schematically illustrates this alignment. In other words, a 680x480 image frame having an RGB value obtained by multiplying the RGB value of the pixels of the 680x480 image frames F1-F4 by 0.25 is generated.

According to the second embodiment, the plurality of image frames are generated by rendering the plurality of image frames with spatial displacement in the rendering processing unit 42. As a result, the post-processing unit 50 performs merge processing on the image frames to cancel the displacement between the image frames. As a result, the updated picture frame sequence is generated at the same spatial resolution as the picture frame sequence output from the rendering processing unit 42. The phrase " undo the displacement between image frames " means to finally obtain a non-offset image frame by blending the offset rendering the image frame. In this embodiment, four pixels are mixed to generate one pixel. Cancellation of displacement between picture frames produces an image obtained by dividing into a time frame between two temporally successive picture frames, and uses the averaged picture of the divided pictures as the picture of the target frame. Is substantially the same as Therefore, if the content of the image frame sequence rendered by the rendering processing unit is a moving picture, cancellation of the displacement can be applied to the effect of motion blur on the moving picture. Like the first embodiment, the second embodiment can be understood as a means for reducing the frame rate. More specifically, the second embodiment can output an image frame sequence having a quarter frame rate of the frame rate of the rendering processing unit 42 while maintaining the resolution of the original image frame.

The second embodiment can also be used when the content is a still picture. In this case, an antialiasing effect is applied to the still picture obtained by the merging process. In this case, the merging process is similar to "super sampling" in which the original pixel is divided into subpixels to obtain data for the target pixel in the image, and the data averaged in the subpixels is used as pixel data.

(Third Embodiment)

9 shows a merge process according to the third embodiment. In the third embodiment, the image frame is generated at a horizontal to vertical ratio different from the horizontal to vertical ratio of the original image frame generated by the rendering processing unit 42. For example, the rendering processing unit 42 has a first image frame 112, a second image frame 114, a third image frame 116, and a fourth image frame 118, each having a resolution of 720 × 480. And the target picture frame 122 is a picture frame having a resolution of 1920 × 1080 pixels, which is a vertical to horizontal ratio that is different from the vertical to horizontal ratio of the original picture frames 112-118. 112-118). In Fig. 9, numerals " 1 ", " 2 ", " 3 " It is a pixel of a 4th image frame.

In a first step 130, the point sampling described in the first embodiment is performed. As a result, an image frame 120 of 1440x960, which is four times the size of each of the original image frames 112-118, is generated. Next, bilinear sampling is performed in a second step 132 to produce an image frame 122 having a resolution of 1920x1080 pixels. In this embodiment, the color of the rendered pixel is determined by linear interpolation of the RGB values of the four pixels that surround it.

A second linear sampling will be described with reference to FIG. 10. When the image frame 112 (1920 × 1080) is reduced to the image frame 120 (1440 × 960), the coordinates of the pixel center of the image frame 122 are calculated in the coordinate system of the image frame 120. FIG. 10 shows a portion 124 of the image frame 120 of FIG. The empty circle 140 represents the center of the pixel in the image frame 122. To determine the color used to render the pixel at this coordinate, linear interpolation is applied to the RGB value according to the coordinate displacement from the center of the pixels 124a, 124b, 124c and 124d of FIG. 10. Pixel 124a is the pixel of the first picture frame 112 of FIG. 9, pixel 124b is the pixel of the second picture frame 114, pixel 124c is the pixel of the third picture frame 116, and pixel 124d is the fourth Note that it is a pixel of the picture frame 118. If the horizontal displacement from the center of the four pixels 124a-124d is α and the vertical displacement is given by β (see FIG. 10), the RGB value of the empty circle 140 calculated by linear interpolation is Given by

R = (1-α) (1-β) Rs1 + α (1-β) Rs2 + (1-α) βRs3 + αβRs4 (1)

G = (1-α) (1-β) Gs1 + α (1-β) Gs2 + (1-α) βGs3 + αβGs4 (2)

B = (1-α) (1-β) Bs1 + α (1-β) Bs2 + (1-α) βBs3 + αβBs4 (3)

Where Rs, Gs, and Bs represent the RGB values of four pixels 124a-124d, respectively, and the subscripts s1, s2, s3, and s4 represent the components of pixels 124a, 124b, 124c, and 124d pixels, respectively. By calculating the formulas (1)-(3) for all the pixels included in the image frame 122 to determine the color, the image frame 122 is generated. The formulas (1) to (3) are based on the same concept as the calculation of general secondary linear sampling. The difference from ordinary second order linear sampling is that color components are obtained from pixels of different picture frames.

If the resolution of the image frame sequence rendered by the rendering processing unit 42 and the resolution displayed on the display device are not a relationship in which one is an integer multiple of the other, that is, for example, the rendered image frame is 720x480, When the resolution of the image frame displayed on the display device is 1920x1080, the target resolution cannot be obtained by simply performing pixel displacement processing and point sampling. In this case, an intermediate image frame of 1440x960 is generated by pixel displacement processing and point sampling, and then second-order linear sampling is performed to obtain 1920x1080 image frame. As a result, the frame rate drops to 1 / (the number of image frames used for point sampling).

Pixel displacement processing does not necessarily have to be performed. More specifically, second order linear sampling is performed directly on the original image frame rendered by the rendering processing unit 42 to obtain the final image frame. However, by performing pixel displacement processing and point sampling prior to the second order linear sampling, a less deteriorated image can be obtained when enlarged. In another embodiment, each of the four image frames (e.g., 720x480 size) in the image frame sequence is enlarged to an image frame (e.g., 1920x1080) that has been modified to suit the display device, and then four magnifications. The final image can be obtained by mixing the obtained image frames.

The user may have the opportunity to select one of the first to third embodiments described above. In one approach to implementing this embodiment, the image frame processing apparatus can automatically make this determination. 11 is a flowchart showing automatic determination performed by the merging condition setting unit 54. In this flowchart, according to the frame rate, the resolution and the content, it is determined which of the post-processes described in the first to third embodiments is to be performed on the image frame sequence rendered by the rendering processing unit 42. .

The merging condition setting unit 54 compares the frame rate information of the display device obtained through the interface unit 52 with the frame rate of the image frame rendered by the rendering processing unit 42 to determine whether these frame rates match. Determine (S10). If the frame rates match (Yes in S10), the transfer controller 46 transfers the image frame from the first memory 44 to the second memory 48. The frame sequence obtaining unit 56 reads out the image frame from the second memory 48 at the same time interval as the frame rate of the rendering processing unit 42. The merge execution unit 58 outputs an image to the image display unit 60 without performing post-processing such as merge process S12. With this, the rendering performance of the rendering processing unit 42 can be fully utilized, and the image can be rendered at a full-spec frame rate. In another embodiment, the picture frame sequence may be output without being temporarily stored in the second memory 48. More specifically, the frame sequence obtaining unit 56 directly reads the image frame sequence rendered by the rendering processing unit 42 using the first memory 44 as a buffer, and the image is displayed on the image display unit 60. Output the frame.

If the frame rates do not match (No in S10), the merging condition setting unit 54 has the image frame rendered by the rendering processing unit 42 with the resolution information of the display device obtained through the interface unit 52. In the following, the resolutions of " rendered image frames " If the resolution of the display device is higher than the resolution of the rendered image frame (Yes in S14), the merging condition setting unit 54 determines whether the content of the image frame displayed on the display device is a still image or a moving image (S16). This decision is made by reading the information registered in such as the header or the header of the program. In another embodiment, the determination as to whether a still picture or a moving picture is made based on the magnitude of the motion component, which is the difference of adjacent picture frames calculated by the motion determining unit (not shown). If the content is a still picture such as a screen picture of a word processor document or an HTML document (Yes in S16), the merging condition setting unit 54 determines whether the resolution of the display device is an integer multiple of the resolution of the rendered image. The determination of an "integer multiple" is when both the vertical and horizontal pixel counts of one of the compared resolutions are multiplied by the same integer of the vertical and horizontal pixel counts of the other resolutions, i.e., the rendered image frame is 640. And the resolution of the display device is lowered to 1280x960 or 1920x1440. If the resolution of the display device is an integer multiple of the resolution of the rendered image frame (Yes in S18), the first embodiment described with reference to FIGS. 5 and 6 is performed to obtain an image frame sequence of a desired resolution (S22). Therefore, when the resolution of the display device is doubled, the merging condition setting unit 54 instructs the frame sequence acquisition unit 56 to search four image frames. When the resolution of the display device is tripled, the merging condition setting unit 54 instructs the frame sequence acquisition unit 56 to search for nine image frames. After the frame sequence acquisition unit 56 passes the obtained image frame to the merge execution unit 58, the merge execution unit 58 performs point sampling on the passed image frame to display an image frame having a desired resolution. Output to 60.

If it is determined in S18 that the resolution of the display device is not an integer multiple of the rendered image frame, the third embodiment described with reference to Figs. 9 and 10 is executed to obtain an image frame sequence of a desired resolution (S20). More specifically, the merging condition setting unit 54 generates an image frame having a resolution closest to the desired resolution and which is an integer multiple of the resolution of the rendered image frame. By performing the second order linear sampling on the generated image frame, the merging condition setting unit 54 generates an image frame in which the horizontal to vertical ratio is not maintained.

If it is determined that the content is a moving picture such as a motion CG or a movie (No in S16), the merging condition setting unit 54 executes the second embodiment described with reference to Figs. 7 and 8 to execute a moving picture having a motion blur effect. (S24). More specifically, the merging condition setting unit 54 executes merging processing using the number of image frames determined by (frame rate of rendered image frames) / (frame rate of display devices). For example, if the frame rate of the rendered image frame is 240 fps and the frame rate of the display device is 60 fps, 4 (= 240/60) frames are used for the merging process. If the result of this division is not an integer, for example, if the frame rate of the rendered image is 240 fps and the frame rate of the display device is 70 fps, the result of division is 3.4 (= 240/70). In this case, the decimal point portion is discarded, and the merging process is executed for three image frames to output the resulting image frame at 70 fps.

In any case, the frame rate of the image frame sequence output from the post processing unit 50 is lower than the frame rate of the rendered image frame produced by the rendering processing unit 42. For example, the execution of point sampling produces twice as high a resolution as the frame rate drops to 1/4, and the execution of point sampling produces a resolution three times as high as the frame rate drops to 1/9. Thus, even when a desired resolution is obtained as a result of S20 or S22 processing, flickering or the like may occur on the screen due to the low frame rate. In another embodiment, a user prompt unit (not shown) may be provided that warns the user on screen that the frame rate will drop significantly when obtaining the desired resolution, and prompts the user to accept. . If the user accepts, point sampling is performed. If not, no point sampling is performed. In another embodiment, the merging condition setting unit 54 refers to the (resolution and frame rate) specification of the display device and the post-processing that can be performed in the post-processing unit 50, and allows for a possible pair of resolution and frame rate. A list of can be displayed on the screen. A user prompt unit (not shown) may prompt the user to select the desired pair and deliver the selected pair to the merge condition setting unit 54. In response, the merging condition setting unit 54 instructs the frame sequence obtaining unit 56 and the merging execution unit 58.

The above processing may be performed prior to the display of the rendered image frame. In another embodiment, image frames subjected to merge processing may be displayed according to a predetermined algorithm so that a user viewing the displayed results may decide whether to continue the above-mentioned processing according to the user's taste.

Returning back to S14, when the resolution of the display device is equal to or lower than the resolution of the rendered image frame (No in S14), the merging condition setting unit 54 determines whether the content is a still image (S26). If the content is a still image (Yes in S26), the merging condition setting unit 54 skips and displays some image frames. More specifically, the merging condition setting unit 54 instructs the frame sequence obtaining unit 56 to obtain one picture frame from a specific number of rendered picture frames. The image frame is output to the image display unit 60 without processing. For example, if the frame rate of the rendered image frame is 240 fps and the frame rate of the display device is 60 fps, every fourth image frame is output.

If it is determined in S26 that the content is a moving image (No in S26), as described above, the second embodiment is implemented to obtain a motion blurred moving image (S30).

Therefore, the merging condition setting unit 54 compares the frame rate or resolution of the image frame sequence rendered by the rendering processing unit 42 with the frame rate or resolution of the display device connected to the image frame processing apparatus, and the condition for post-processing. Can be determined automatically.

As described above, the rendering processing unit performs rendering at a predetermined frame rate to generate an image frame sequence irrespective of a condition that an image frame output to the display device must satisfy. The post-processing unit performs predetermined processing on the image frame sequence generated by the rendering process and outputs an updated image frame sequence complying with the above-mentioned condition.

Since the rendering process and the post-processing are executed separately, it is possible to generate an image frame sequence by performing rendering at a predetermined frame rate regardless of the specification of the display device such as resolution or frame rate.

The rendering processing unit 42 is described as rendering 640x480 image frames at 240 fps. Image frames of other pixel counts may also be rendered. Of course, the rendering speed of the image frame may be low or high. For example, the picture frame sequence can be rendered at 300 fps. In this case, the picture frame sequence may be generated to be changed to suit both the 50 Hz display or the 60 Hz display.

In describing this embodiment, the merge process is described as performing to pixels of four image frames. In another embodiment, the merging process may be performed on multiple pixels. For example, six image frames may be rendered with pixel displacement such that a pixel of each frame is located at a vertex of a corresponding hexagon, and one image frame may be generated in the form of pixels having an average RGB value of six pixels. .

In describing this embodiment, the image frame processing apparatus is described as being formed in an entertainment apparatus that renders a CG image. However, the image frame processing technique according to the present invention can be applied to DVD players, personal computers, digital video cameras and the like.

(2nd embodiment)

In the first embodiment, image frames equal to or greater than those required for displaying a moving image on the display device are rendered. Then, predetermined processing is performed on the rendered image frame to output an image frame for display. In contrast, when a moving image is provided in advance, an embodiment is also conceived in which a plurality of image frames are selected from the moving images, a predetermined process is applied, and image frames below the read image frame are output. By the latter embodiment, a fast-turned image of the original moving image can be generated. In addition, fast rewinding moving pictures can also be generated. The fast rewound moving picture is an image output in the reverse direction on the time axis of the moving picture. Hereinafter, "fast turn" also includes a fast rewind operation.

These two embodiments appear to be different at first glance. However, the two embodiments have the same concept in that predetermined processing is applied to the image frames of the frames or more finally displayed to the user and the updated image frames are output. In other words, the difference between the two embodiments is only the length of the interval of the image frames to be output.

Recently, digital moving picture recorders such as HDD (Hard Disk Drive) video recording devices have become widespread. Therefore, a large amount of moving image data can be made personally and easily recorded or reproduced. In such a device, the user uses the quick turn function to find the portion of interest of the recorded moving image data. However, when a moving picture is rotated quickly, the user often misses the part of interest while searching and sometimes feels inconvenient to find.

Therefore, in the second embodiment, an image frame processing technique is provided for outputting moving images that are easily viewed even when the moving images are rotated quickly.

12 shows hardware settings of the image frame processing apparatus 200 according to the second embodiment. The main CPU 12, the main memory 14, the display device 26, the display device controller 28, the input / output (I / O) port 30 and the external storage device 32 are shown according to the first embodiment. Same as the blocks shown in 1, therefore, these blocks are given the same reference numerals and further detailed description is omitted. The input / output port 30 is connected to a camera device 38 such as a digital video camera. The moving picture captured by the camera device 38 is stored as digital data in an external storage device 32 such as a DVD (Digital Versatile Disc) drive and a hard disk drive. The graphic processor 80 selects an image frame sequence from the moving image data stored in the external storage device 32 and stores the image frame sequence in the main memory 14. Then, the graphic processor 80 performs predetermined processing on the image frame sequence to generate the updated image frame sequence, and outputs the updated sequence to the display device 26.

The image frame processing apparatus 200 may be coupled to various kinds of moving image display mechanisms for displaying on a display device 26 a moving image composed of an image frame sequence. Such moving picture display mechanisms may include various mechanisms for storing or playing back movie contents such as DVD players and HDD video recorders. In addition, the moving picture display mechanism may be coupled to a personal computer, a digital video camera or an entertainment apparatus.

The input device 84 makes some input to the image frame processing apparatus 200. As the input device 84, various types of devices can be used depending on the shape of the moving picture display mechanism. For example, assuming that the moving picture display mechanism is a DVD player or an HDD video recorder, the input device 84 may be various buttons provided to the remote control or the moving picture display mechanism. Assuming that the moving picture display mechanism is a general-purpose computer, the input device 84 can be a keyboard or a mouse.

In the second embodiment, it will be described that a quick turn image is created when a quick turn request for already made movie content is received from a user and recorded in a mass storage device such as a DVD drive or an HDD drive. Similar to the first embodiment, the second embodiment can be applied to an entertainment apparatus that performs rendering processing for generating a new image frame sequence for display on a display device.

Hereinafter, in the image frame processing apparatus 200 shown in FIG. 12, a method for generating a fast-turning moving image having added value will be described with reference to some examples.

(Example 4)

13 is a functional block diagram of the image frame processing apparatus 200 according to the fourth embodiment. 13 is mainly implemented with a graphics processor 80, a main CPU 12 and a main memory 14. In the fourth embodiment, a method for providing a smooth quick turn moving picture in response to a quick turn request from a user will be described.

The interface unit 202 obtains a quick turn request through the user's input device 84. For example, assuming that the image frame processing apparatus 200 is coupled to a DVD player, this fast turn request is made as fast as "2X" or "4X" specified by a turn button or dial provided to the main unit or the remote controller. Corresponds to the rotation speed information. The fast turn request may be specified in the header portion of moving image data instead of being given by the user. The interface unit 202 transfers the obtained information to the transmission frame number determination unit 206. The transmission frame number determination unit 206 determines the number of image frames necessary for realizing a fast-turning moving picture according to the received fast-turn speed information. The frame transfer unit 208 reads out the image frame number determined by the transmission frame number determination unit 206 from the image frame sequence stored in the storage unit 250 at a constant timing. The frame transfer unit 208 then sends the frame to the fast turn unit 220. As an example, the storage unit 250 corresponds to the main memory 14 of FIG. 12. However, the storage unit 250 may be a memory such as an external storage device 32 provided to any storage unit or the image frame processing apparatus 200. In addition, the picture frame in the storage unit 250 may be an uncompressed picture. The picture frame in storage unit 250 may also be a compressed picture using Discrete Cosine Transformation (DCT).

The quick turn unit 220 includes a frame sequence acquisition unit 222 and a merge execution unit 224. The frame sequence obtaining unit 222 acquires and temporarily stores the transmitted picture frame. The merge execution unit 224 performs a merge process of generating one updated image frame from the plurality of image frames stored in the frame sequence acquisition unit 222. This merging process may be the mixing process described in the first embodiment. The updated picture frame is referred to as "quick turn frame".

The quick turn frames generated by the merge execution unit 224 are transmitted to the image construction unit 240 in the order created. The image construction unit 240 outputs the quick turn frame at a predetermined frame rate that can be displayed on the display device 26. In this way, the user can view the desired fast rotating image on the display device 26.

When the merge process is performed on a plurality of image frames, false afterimages are generated in the quick turn frame. By sequentially outputting these quick turn frames, a fast turn image having a motion blur effect is obtained. Thus, the user can enjoy a natural and smooth video.

However, this processing extracts the image frame from the image frame sequence in the storage unit 250 every predetermined number of frames, thereby making the extracted frame at a predetermined frame rate without any merging process for generating a fast-turning image. You can print In order to understand the effect of the fourth embodiment, the shortcomings of the quick turn image generated by this process will be described with reference to FIG.

FIG. 14 shows the concept of a process including generating a fast-turn frame sequence 310 by estimating an appropriate number of picture frames from a previously prepared picture frame sequence 300.

Picture frame sequence 300 includes picture frames 301-309 and many other frames before and after these picture frames. The image frames 301-309 represent a moving picture in which the circular object 400 moves from the upper left to the lower right on the screen. In practice, it is impossible to smoothly move the circular object 400 without using more image frames than the frames 301-309. However, for simplicity of explanation in FIG. 14, it is assumed that the smooth movement of the circular object 400 is realized only with the image frames 301-309.

The star shape 401 shown in the picture frames 303 and 304 indicates the flicker of the circular object 400. In the picture frame sequence 300, the circular object 400 is shown in the upper left corner of the screen and flickers twice, moving downward.

In this embodiment, one picture frame is extracted every three picture frames. More specifically, picture frames 301, 304, and 307 are extracted every three picture frames in the picture frame sequence 300. Then, these extracted image frames become quick turn frames 311, 312 and 313, respectively, without any processing. Therefore, the fast-turning image can be generated by extracting one picture frame for every appropriate number of picture frames to generate the fast-turning frame sequence 310 and outputting the sequence 310 at a predetermined frame rate. In the present embodiment shown in FIG. 14, a 3 times faster rotation image can be obtained.

However, in this process, when the difference between the picture frames extracted from the original picture frame sequence is large, the image may be discontinuous in frame by frame, especially at high speed and fast rotation. Therefore, the image is difficult for the user to recognize. In addition, the image frame sequence 300 includes an image frame 303 indicating that the circular object 400 is blinking. However, the fast-turn frame sequence 310 does not include this picture frame 303. Therefore, a user who views the quick turn image composed of the frame sequence 310 may not recognize that the circular object 400 blinks twice.

As can be seen from this, in this process, an image frame having important information is not included in the fast-turning image except for flickering of the object in FIG. In other words, no matter what event occurs in the original picture frame sequence, it is possible for the user to not see the event if he / she sees the fast spinning image. Thus, when the user searches for a scene of interest from the quick turn image with the specific information as a clue, if the user lacks specific information from the fast turn image, the user cannot find the scene.

Next, a method of generating a quick-turn image according to the fourth embodiment will be described with reference to FIG. 15. Instead of extracting one image frame for every three image frames, the merge execution unit 224 performs merge processing on three image frames in the image frame sequence 300 to generate one quick turn frame. More specifically, the merge execution unit 224 performs merge processing on the image frames 301-303 to generate the quick turn frame 321. The merge execution unit 224 executes merge processing on the image frames 304-306 to generate the quick turn frame 322. The merge execution unit 224 executes merge processing on the image frames 307-309 to generate the quick turn frame 323.

This merging process corresponds to generating an image frame having each pixel of pixels that has been weighted averaged of pixels located at the same position in the image frames. More specifically, when n pieces of image frame F _m (m = 1,…, n, n are positive integers) are used to generate one fast-turn frame F _f ,

F _f = ∑α _m / F _m (4)

Where α _m is a weighting coefficient for each image frame and satisfies Σ α _m = 1. As can be seen from equation (4), the weight ratio is not the same for each image frame. For example, a high weight ratio may be applied to an image frame adjacent to a particular image frame, and a low weight ratio may be applied to an image frame located farther from the specific image frame. How the value of α _m is distributed depends on the characteristics of the fast-turn frame F _f .

By the merging process described above, quick turn frames 321, 322, and 323 having false afterimages of the circular object 400 moving between the image frames are obtained. In FIG. 15, an afterimage of the circular object 400 is represented by an empty circle or star shape. As a result, when the quick turn frame sequence 320 having the quick turn frames 321 to 323 is reproduced, motion blur and a smooth moving picture can be obtained. Therefore, the viewing restriction of the user is relaxed. In addition, as can be seen from the fast-turn frames 321 and 323, an object flashing an image remains in such a frame as a false afterimage. Thus, the information in the original picture frame will not be lost from the frame due to creating the quick turn frame. In other words, part of the information of the original picture frame always remains in the fast-turn frame. Therefore, when a user searches for a scene of interest from a fast-turning image with specific information as a clue, the user feels easy to find the scene because of the remaining information.

Fig. 16 shows the concept of a process of increasing or decreasing the number of image frames for merging to generate a fast-turned image having a different turn speed. The picture frame sequence 350 includes picture frames 351-362 and many frames before and after these picture frames. When generating a normal quick turn image, the merge execution unit 224 performs a merge process on four image frames to generate one quick turn frame. More specifically, the merge execution unit 224 executes merge processing on the image frames 351-354 to generate the quick turn frame 371, and the merge execution unit 224 generates the image frames 355-358. ), The fast rotation frame 372 is generated.

When a specific picture frame having a specific condition is retrieved by the frame sequence acquisition unit 222 while generating the quick turn frame, the merge execution unit 224 executes a merge process for every two picture frames to turn one fast turn. Create a frame. In FIG. 16, when the image frame 359 satisfies a specific condition, the merge execution unit 224 performs a merge process on the image frames 359 and 360 to generate a quick turn frame 373, and executes merge. The unit 224 performs a merge process on the image frames 361 and 362 to generate the fast turn frame 374.

The quick turn image consisting of the fast turn frame sequence 370 including the fast turn frames 371-374 initially has a 4 times turn speed, but after the fast turn frame 373, the turn speed decreases by 2 times. Therefore, by merging and increasing or decreasing the number of image frames appropriately, a fast-turning image whose speed changes at any point in time can be obtained.

In order to search for a specific picture frame, any known technique can be used. For example, a scene change search technique can be used to search for a particular picture frame whose scene changes. With this, you can get a fast turn image with a reduced turn speed in some particular scenes. In another embodiment, a motion vector between picture frames is calculated and then a particular picture frame with an absolute value of the motion vector greater than a predetermined value can be retrieved. In this way, by searching for an image frame in which the movement of an object increases in the screen, a fast-turning image having a reduced rotational speed can be obtained after a specific frame.

Even if the user rotates the image quickly, the fast rotation speed is automatically reduced at a given point, so that the user can easily find an interesting or important scene. In addition, if a user searches for a scene of interest from a quick turn image with specific information as a clue, the fast turn speed will automatically decrease in the frame with this information. Therefore, the user can find the scene more easily. The practical use is described below. Assuming that the image content is a drama, the scene in which a particular actor appears can be reproduced at a reduced speed during a fast turn. Assuming the video content is a soccer game broadcast, the scoring scene can be reproduced at a reduced speed during a fast turn.

(Example 5)

In the fourth embodiment, without considering the characteristics of every image frame, a predetermined number of image frames are extracted from an image frame sequence, and merge processing is performed on the extracted image frames to generate a quick turn frame. This treatment preferably produces a smooth, fast-moving image. In some cases, however, it is more desirable to first extract some image frames having certain characteristics to produce a fast-turning image. In the fifth embodiment, there is provided an image frame processing apparatus for generating a fast-turned image having high viewing efficiency by first extracting some image frames satisfying a specific condition.

17 is a functional block diagram of an image frame processing apparatus according to the fifth embodiment. The interface unit 202, the transmission frame number determination unit 206, the frame transmission unit 208, the frame sequence acquisition unit 222, the image construction unit 240, and the storage unit 250 are the blocks shown in FIG. It is the same, and therefore, these blocks are given the same reference numerals and further detailed description is omitted.

The quick turn unit 220 includes a frame sequence acquisition unit 222 and a characteristic frame extraction unit 226. The characteristic frame extraction unit 226 extracts an image frame that satisfies a predetermined condition from the image frames transmitted from the frame transmission unit 208 based on its luminance information as the characteristic frame. For example, the characteristic frame extraction unit 226 calculates an average of a pixel for each pixel included in 10 frames before and after including a specific image frame, and then has a pixel having a value 50% larger than the average value. Extract an image frame comprising a as a characteristic frame. The characteristic frame extraction unit 226 extracts a suitable number of image frames in addition to these characteristic frames, generates a quick-turn frame, and delivers them to the image construction unit 240. The image construction unit 240 outputs a quick turn frame that can be displayed on the display device 26 at a predetermined frame rate.

Hereinafter, the extraction of the characteristic frame according to the fifth embodiment will be described in more detail. 18 shows a concept of a process of extracting some image frames based on luminance information from an image frame sequence. The picture frame sequence 300 includes picture frames 301-309 and many other frames before and after these frames as shown in FIG. The characteristic frame extraction unit 226 extracts an image frame including a pixel of higher luminance than other adjacent image frames into the characteristic image frame. As described above, the circular object 400 blinks in the image frames 303 and 307. Therefore, the image frames 303 and 307 are extracted as characteristic frames because they contain pixels of greater luminance than adjacent image frames. These distinctive frames become fast-running frames 331 and 332, respectively, without any processing.

It is possible that the frames needed to construct a fast-turn image at the requested turn rate are not extracted only by extracting the characteristic frames from the picture frame sequence 300. Thus, if there are no characteristic frames extracted from a predetermined number of frames, it is preferable that the characteristic frame extraction unit 226 extracts one frame from the predetermined number of image frames regardless of the luminance information. In contrast, if there are a plurality of characteristic frames extracted from a predetermined number of frames, it is preferable that the characteristic frame extraction unit 226 extracts only one frame. In this manner, a fast turn frame sequence 300 can be created.

In another embodiment, when there are a plurality of characteristic frames extracted based on luminance information in a predetermined number of image frames, all image frames determined as characteristic frames may be fast turning frames regardless of the turning speed information. . In this way, image frames containing pixels of high luminance are continuously extracted for a certain period of time. Thus, during this particular period, it is possible to obtain a fast-rotating image with a reduced speed with a normal playback speed. As a result, even in a fast-turning image, a moving picture that is almost similar to normal playback in a characteristic scene can be obtained, thereby reducing the possibility of a user missing important information in the characteristic scene. It is desirable to set the conditions for the characteristic frames according to the kind of information the user wants to obtain.

In the fifth embodiment, since the quick-turn image is generated by extracting the characteristic frame based on the luminance information, the number of image frames having important information missed from the fast-turn image is reduced.

The information for extracting the characteristic frame is not limited to the luminance information. For example, motion information between picture frames may be used to first extract any picture frame with a particular condition.

Referring to FIG. 17, the motion information detecting unit 210 receives an image frame transmitted from the frame transmitting unit 209 and calculates motion information between these image frames. For example, the motion information sensing unit 210 obtains corresponding points between picture frames using a known block matching method. Next, the motion information detecting unit 210 calculates a motion vector from the difference between the corresponding points. The motion vector is used for motion information. If some motion information for each area or object in the picture frame is prepared as data in advance, this data can also be used as motion information.

The characteristic frame extraction unit 226 extracts, as a characteristic frame, an image frame satisfying a predetermined condition from the transmitted image frame based on the motion information. For example, the condition is that the absolute value of the motion vector is greater than the predetermined value. The characteristic frame extraction unit 226 extracts an appropriate number of image frames other than these characteristic frames, generates a quick turn frame, and transfers the quick turn frame to the image construction unit 240. The image construction unit 240 outputs the quick turn frame to the display device 26 at a predetermined frame rate.

In another embodiment, the characteristic frame extraction unit 226 receives the information recorded in the header portion of the moving image data from the interface unit 202 and extracts the characteristic frame based on this information. For example, assuming that the content of a moving picture is drama, bits are used in the header portion of several tens or hundreds of picture frames before and after the point at which the scene changes to indicate the scene change. The characteristic frame extraction unit 226 can extract the image frame represented by the characteristic frame by these bits. In this way, even in a fast-turning image, the turning speed is equal to the normal playback speed. Therefore, the user can more easily recognize the content in the fast-rotating image.

(Example 6)

In the fifth embodiment, it has been described that an image frame satisfying a specific condition is extracted into a characteristic frame using luminance information or motion information. In other words, in the fifth embodiment, the picture frames in the picture frame sequence are divided into two groups. One group includes picture frames that are beneficial to the user (i.e., picture frames with a lot of information). Another group includes picture frames that are less beneficial to the user (ie, picture frames with less information). Next, more image frames are selected from the first group to produce a fast turn image.

In the sixth embodiment, an image frame processing apparatus is provided for dividing one image frame into two portions, one portion having more information and the other portion having less information. Either part is reinforced or inconspicuous. This makes it easier for the user to get the information from the quick turn image.

19 is a functional block diagram of an image frame processing apparatus according to the sixth embodiment. The interface unit 202, the transmission frame number determination unit 206, the frame transmission unit 208, the motion information sensing unit 210, and the storage unit 250 are the same as the blocks shown in Fig. 17, and thus these blocks The same reference numerals are used, and detailed description thereof will be omitted.

The quick turn unit 220 includes a separation unit 228, a merge execution unit 230 and a frame reconstruction unit 232. The separating unit 228 receives the image frame transmitted from the frame transmitting unit 208. Separation unit 228 divides each image frame into " specific image region " and " non-specific image region ". This separation is performed based on the motion information received from the motion information sensing unit 210. The specific image area is an area in which the absolute value of the motion vector is larger than a predetermined boundary. The non-specific image area is an area other than the specific image area. The merge execution unit 230 performs merge processing on the non-specific image region between the image frames. On the other hand, the merge execution unit 230 selects any one specific image area from the images.

The frame reconstruction unit 232 synthesizes the non-specific picture areas merged with the selected specific picture area to generate an updated picture frame. The updated picture frame is delivered to the image construction unit 240 as a quick turn frame. The image construction unit 240 outputs the quick turn frame to the display device 26 at a predetermined frame rate.

20 shows the concept of dividing image frames into specific image regions and non-specific image regions. The picture frame sequence 380 includes picture frames 381-384 and many picture frames before and after these picture frames. The picture frame sequence 380 includes a picture of someone. Human images are detected as follows. The user can specify the color and pattern of clothes worn by the person. The human image area is then sensed using known image matching techniques with clues to color and pattern.

The separating unit 228 separates the image frames 381-384 into non-specific images for the background image and specific images for the human image. The merge execution unit 230 performs merge processing on the non-specific image regions of the image frames 381-384. The merging unit 230 selects one specific image area from the image frames 381-384. In FIG. 20, the specific image area of the image frame 382 is selected by the merging execution unit 230. The frame reconstruction unit 232 then arranges the non-specific picture area merged with the specific picture area selected by the merge execution unit 230 to generate the quick turn frame 385. The quick turn frame 385 has a blurred background image due to the merging process. Therefore, the quick-rotated image including the frame 385 can display a person with a motion blurred background, so that the user can recognize the person more easily.

As described above, according to the sixth embodiment, an important part of the image frame can be clearly displayed in the fast-turning image. In other words, according to the sixth embodiment, less important parts of the picture frame can be made unrecognizable due to motion blur.

In this manner, when the content of the moving picture is a drama or a sports broadcast, a user who the user likes may be displayed in a recognizable manner on a fast-turning image.

In another embodiment, when the non-specific picture area is a still picture, the merging unit 230 uses a plurality of non-specific picture areas to enhance its picture quality.

(Example 7)

21 shows a functional block diagram of an image frame processing apparatus according to the seventh embodiment. In the seventh embodiment, the path (trace) of the object in the picture frame is displayed in the fast-turning image.

The interface unit 202, the transmission frame number determination unit 206, the frame transmission unit 208, the motion information detection unit 210, the image construction unit 240, and the storage unit 250 are divided into blocks shown in FIG. 17. It is the same, and therefore, these blocks are given the same reference numerals and further detailed description is omitted.

The quick turn unit 220 includes a path generation unit 236 and a frame reconstruction unit 232. The path generation unit 236 generates a path image using the motion information received from the motion information detection unit 210. This path image is an image which displays a flow line of a predetermined object in the image frame transmitted from the frame transfer unit 208. The frame reconstruction unit 232 overwrites the path picture with the original picture frame to generate a quick turn frame.

22 shows the concept of a route generation process according to the seventh embodiment. The picture frame sequence 300 includes picture frames 301-309 and many picture frames before and after these pictures, as in FIG.

The path generation unit 236 generates a path image 411 from the difference between the picture frame 301 and the picture frame 302. The path generation unit 236 generates the path picture 412 from the difference between the picture frame 302 and the picture frame 303. The frame reconstruction unit 232 arranges the path images 411 and 412 in the image frame 303 to generate the quick turn frame 341. Similarly, the path generation unit 236 generates a path picture 413 from the difference between the picture frame 304 and the picture frame 305. The path generation unit 236 generates a path picture 414 from the difference between the picture frame 305 and the picture frame 306. The frame reconstruction unit 232 places the path images 413 and 414 in the image frame 306 to generate the quick turn frame 342. The same processing is repeated at or after the frame image 307.

The image construction unit 240 outputs the quick turn frame sequence 340 including the quick turn frames 341 and 342 at a predetermined frame rate. Therefore, a quick turn image with a path representing the movement of the circular object can be obtained.

In order to determine which of the objects present in the picture frame is an object that indicates its path, various known methods can be used. For example, a predetermined object (eg, soccer ball) may be detected in each image frame using a known image recognition technique, and the path of the soccer ball may be displayed in a fast spinning image.

According to the seventh embodiment, it is possible to display information which does not appear in the original image frame in the fast-rotating image. In other words, by using different information between picture frames, it is possible to enhance the information in the picture frame.

(Example 8)

By selecting one process according to the fourth to seventh embodiments described above, a quick turn image suitable for the content or the purpose of the user can be produced. For example, according to the contents of the moving picture recorded in the storage device, an appropriate fast forwarding may be selected.

Hereinafter, assuming that the image frame processing apparatus is included in the moving picture reproducing mechanism, the fast-turning image generation processing will be described.

23 shows a functional block diagram of an image frame processing mechanism capable of realizing all the processes according to the fourth to seventh embodiments. The interface unit 202, the transmission frame number determination unit 206, the frame transmission unit 208, the motion information detection unit 210, the image construction unit 240, and the storage unit 250 may be divided into blocks shown in FIG. 13. It is the same, and therefore, these blocks are given the same reference numerals and further detailed description is omitted.

The quick turn unit 220 is set to execute all the processes described in the fourth to seventh embodiments. The image frame processing apparatus further includes a content determination unit 214. The content determining unit 214 determines the type of content of the moving image stored in the storage unit 250. This determination is made based on the header information of the moving picture data. In another embodiment, the determination may be based on movement information from the user's input or motion information sensing unit 210. The determined content type is transmitted to the quick turn unit 220. The quick turn unit 220 receives the image frames transmitted from the frame transfer unit 208 and executes one process according to the fourth to seventh embodiments according to the type of content.

Hereinafter, when the type of content determined by the content judging unit 214 is a sports broadcast, a drama or a movie, or a user's original movie, the specific processing executed by the quick turn unit 9220 will be described.

end. Sports broadcast

When the type of content is a video recording a soccer game, it is possible to reduce the speed of turning the fast-turning image only in the scoring scene. The scoring scene may be sensed as follows. Motion pictures are captured by a fixed position camera pointing to the goalpost. The bone mouth area is predefined in the image captured by the camera. When the soccer ball image enters the goal mouse area, it is detected by the image matching method, and the quick turn unit 220 judges it as the scoring scene and extracts a plurality of image frames before and after the time point. In addition, the unit 220 determines a specific player as a clue using an image matching technique such as color, pattern, or the number of the uniform. Then, a quick turn image that enhances the movement of a particular player can be obtained by applying motion blur to other players. In addition, the soccer ball in the picture frame is recognized using an image matching technique, and a fast-turning image with the path of the soccer ball can be obtained.

I. Drama / Movie

For example, if the type of content is a drama program, the unit prompts the user to input the color or pattern of his / her favorite actor in the drama. The quick turn unit 220 then detects an object having an area corresponding to the input color and pattern by a known image matching technique. By this, the unit 220 may identify a scene in which the favorite actor is visible and generate a quick turn image of the identified scene at a reduced turn speed.

All. User's original movie

When the moving image stored in the storage unit 250 is an image captured by the user by the portable camera, the quick turn unit 220 detects a short circuit of the scene using a known scene change extraction technique. By including the image frames before and after the paragraph of the scene in the quick turn image, the contents in the quick turn image can be easily grasped. In addition, objects tracked with a portable camera are detected using optical flow. Then, the background image except the tracked object is motion blurred in the fast rotating image. With this, a quick turn image can be generated as easy to view with the tracked object.

As described above, according to the image frame processing apparatus shown in FIG. 23, a fast-running image may be obtained as soon as the processing appropriate to the type of the moving image or the user's preferred content is obtained.

According to the second embodiment of the present invention, when a fast turn request is received based on a picture frame sequence previously stored in a DVD drive or a hard disk drive, a predetermined process is performed on the picture frame sequence to generate a fast turn frame. . Then, the quick turn frame is output at a predetermined frame rate necessary to display the quick turn image for the quick turn request. One of the features of the second embodiment is that various fast-forwarding images can have various added values by performing various processes on the image frame sequence. Value added includes that important information of the original image frame sequence remains in the image to be rotated as quickly as possible (fourth and fifth embodiments), and unnecessary information of the image frame sequence is omitted from the image to be rotated quickly (sixth embodiment) do.

In the second embodiment, after receiving the quick turn request signal, the quick turn frame is generated almost in real time and output as a quick turn image. Therefore, whenever a fast turn request signal is received according to a condition or a user's designation, it is possible to output various fast turn images generated by other processing. In other words, in the second embodiment, various quick turn images having high versatility in the quick turn unit corresponding to the post-processing and having other advantages can be provided.

The image frame sequence of the moving image stored in the storage unit 250 is generated at a predetermined frame rate irrespective of the rotation speed. However, the fast turn unit executes processing according to the type of the fast turn request signal or the content to generate an updated picture frame sequence for fast turn. As can be seen from the foregoing, the first and second embodiments have a common concept that frames displayed to the user are generated by sampling an image frame prepared at a rate higher than the display ratio. The second embodiment corresponds to the special case of the first embodiment in which the time axis for sampling is extended.

The application of the second embodiment is not limited to generating a fast-turn image or a fast-rewind image. For example, using a moving image captured by a high speed camera, a normal reproduced image having any of the above effects can be generated according to the second embodiment. In this case, the following conditions must be satisfied.

N _s ≥ N _p ≥ N _o

Where N _s denotes the number of image frames captured by the high speed camera during the unit time, N _p denotes the number of image frames stored in the storage unit during the unit time, and N _o denotes the image frame finally displayed on the display unit during the unit time. Indicates a number.

The present invention has been described based on several embodiments. Such embodiments are illustrative, and it will be apparent to those skilled in the art that various modified components and treatments are possible within the scope of the invention. Any combination of the components described in the embodiments and implementing the invention in the form of methods, instruments, systems, computer programs and recording media can also be implemented as additional aspects of the invention.

Claims (39)

Rendering a sequence of picture frames in the memory based on the given shape data; And And a post-processing step of changing the image frame sequence generated by the rendering step to be suitable for a display device. The rendering step generates an image frame sequence at a predetermined frame rate irrespective of a condition that the image frame must satisfy to output to a display device, The post-processing step applies a process of two-dimensionally merging pixels in an image frame according to the condition to the image frame sequence generated by the rendering step, thereby generating an updated image frame sequence satisfying the condition, And outputting the image frame processing method. Rendering a picture frame sequence in memory based on the given shape data; And And a post-processing step of changing the image frame sequence generated by the rendering step to be suitable for a display device. The rendering step generates an image frame sequence at a predetermined frame rate regardless of the specification of the display device, The post-processing step applies a process of two-dimensionally merging the pixels in the image frame according to the specification of the display device to the image frame sequence generated by the rendering step, thereby updating the modified to conform to the specification of the display device. And generating and outputting a sequence of image frames. Rendering a picture frame sequence in memory based on the given shape data; And And a post-processing step of changing the image frame sequence generated by the rendering step to be suitable for a display device. The rendering step generates an image frame sequence at a predetermined frame rate regardless of the program being executed, The post-processing step applies a process of two-dimensionally merging pixels in an image frame in accordance with the executed program to the image frame sequence generated by the rendering step, thereby updating the updated image frame sequence to be suitable for the program. And generating and outputting the image frame. An interface unit for obtaining a condition that an image frame must satisfy in order to output to a display device; A rendering processing unit for generating an image frame sequence at a predetermined frame rate regardless of the condition; And According to the condition obtained by the interface unit, a process of two-dimensionally merging the pixels in the image frame is applied to the image frame sequence generated by the rendering processing unit to produce an updated image frame sequence for the display device. And a post-processing unit for generating. The method of claim 4, wherein And the condition is an output resolution of the display device. The method of claim 4, wherein And the condition is a frame rate of the display device. The method of claim 4, wherein And the condition is an output resolution defined according to the type of content displayed on the display device. The method of claim 4, wherein And the rendering processing unit starts rendering one set of picture frames in the picture frame sequence at different offsets of coordinates in rendering space. The method of claim 8, The post-processing unit merges the image frame set generated by the offset rendering while maintaining the displacement between the image frame sets, and has a spatial resolution higher than the spatial resolution of the image frame sequence output from the rendering processing unit. An image frame processing apparatus, characterized by generating an updated image frame sequence. The method of claim 9, The image frame set generated by the offset rendering is located in one coordinate system, and the updated image frame sequence is less than the spatial resolution of the image frame sequence output from the rendering processing unit by sampling pixels in the coordinate system. An image frame processing apparatus, characterized in that it is generated at a high spatial resolution. The method of claim 8, The post-processing unit merges the image frame set generated by the offset rendering such as canceling the displacement between the image frame sets, thereby updating the update having the same spatial resolution as the image frame sequence output from the rendering processing unit. And generate an image frame sequence. The method of claim 11, The image frame set generated by offset rendering is located in one coordinate system, and the updated image frame sequence is generated at the same spatial resolution as the image frame sequence output from the rendering processing unit by mixing pixels in the coordinate system. An image frame processing apparatus, characterized in that. The method according to any one of claims 4 to 12, A first memory for storing the image frame generated by the rendering processing unit; And And a second memory used as a work area by the post-processing unit. The first memory has a capacity to store at least one image frame generated by the rendering processing unit, The image frame temporarily stored in the first memory is transferred to the second memory, And the post-processing unit reads out a plurality of image frames from the second memory for the merging process. The method according to any one of claims 4 to 12, A first memory for storing the image frame generated by the rendering processing unit; And And a second memory used as a work area by the post-processing unit. And the first memory is formed in at least the same semiconductor circuit element as the rendering processing unit. The method of claim 14, And a main memory is used as the second memory. Rendering a picture frame sequence in memory based on the given shape data; And A post-processing step of changing the image frame sequence generated by the rendering step to be suitable for a display device; a recording medium having recorded thereon a program for causing a computer to execute, wherein the program The rendering step generates an image frame sequence at a predetermined frame rate irrespective of a condition that the image frame must satisfy to output to a display device, The post-processing step applies a process of two-dimensionally merging pixels in an image frame according to the condition to the image frame sequence generated by the rendering step, thereby generating an updated image frame sequence satisfying the condition, Outputting a recording medium. A rendering processor that generates a sequence of image frames in a memory based on shape data given at a predetermined frame rate regardless of a condition that an image frame must satisfy for display on a display device. The image frame sequence is subjected to a process of two-dimensionally merging the pixels in the image frame according to the condition to the image frame sequence generated by the rendering processor in a post-processing step, so as to update the image frame to satisfy the condition. A rendering processor, which is converted. The method of claim 17, And the condition is an output resolution of the display device. The method of claim 17, And the condition is a frame rate of the display device. The method of claim 17, And the condition is an output resolution defined according to a type of content displayed on the display device. The method according to any one of claims 17 to 20, And rendering of one set of picture frames in the picture frame sequence at different offsets of coordinates in rendering space. The method according to any one of claims 17 to 20, And a memory for storing the image frame generated by the rendering is formed in the same semiconductor circuit element as the rendering processor. An image frame processing method for displaying an externally supplied image frame sequence on a display device, Obtaining a condition that an image frame must satisfy to output to the display device; Receiving the picture frame sequence at a predetermined frame rate regardless of the condition; And And subjecting the received image frame sequence to two-dimensionally merging the pixels in the image frame according to the condition, to generate an updated image frame sequence that satisfies the condition. An interface unit for obtaining a condition that an image frame must satisfy in order to output to a display device; An image frame receiving unit for receiving an image frame sequence at a predetermined frame rate regardless of the above condition; And According to the condition obtained by the interface unit, a process of two-dimensionally merging pixels in the picture frame according to the condition is applied to the received picture frame sequence to generate an updated picture frame sequence for the display device. And a post-processing unit for performing. A recording medium having recorded thereon a program for causing a computer to display an externally supplied image frame sequence on a display device. Obtaining a condition that an image frame must satisfy to output to the display device; Receiving the picture frame sequence at a predetermined frame rate regardless of the condition; And Subjecting the image frame sequence to the two-dimensional merge of pixels in the image frame according to the condition, according to the condition obtained by the interface unit, to generate an updated image frame sequence for the display device; And a recording medium comprising; delete delete delete delete delete delete delete delete delete delete delete delete delete delete

Images