CN101123692A - Method and apparatus for generating new images by using image data that vary along time axis - Google Patents

Abstract

A rectangular-parallelopiped space (box space) expresses the moving images by use of a virtual space. A plurality of frames contained in the moving images appear continuously along time axis. The box space is cut through by a desired surface, and an image projected on this cut surface is projected onto a plane parallel in the direction of time axis. Images sequentially projected onto the plane are outputted as new moving images.

Description

Method and apparatus for generating new image using image data changed along time axis

The present application is a divisional application entitled "method and apparatus for generating new images using image data changed along a time axis" (application No.: 200380101976.8; application date: 10/22/2003).

Technical Field

The present invention relates to a method and apparatus for generating an image, and more particularly, to a technique of processing a moving picture taken by a camera and thus re-outputting the processed moving picture.

Background

With the remarkable progress and development of computer technology in recent years, the image processing capability provided by computers has improved remarkably. Even home PCs (personal computers) and game machines available to general users can realize various processes that were once realized by high-end workstations dedicated to image processing.

Improvements in the image processing performance and capabilities of PCs provide another potential application for home PCs and gaming machines. That is, there are inexpensive tools for movie editing, image processing, authoring, and the like, which are directed to general users. Thus, professional skills are no longer a prerequisite for operating complex image processing, and even amateur users can operate the processing of moving images by using these available tools.

Disclosure of Invention

In view of the foregoing, the inventors of the present invention have proceeded with the search for innovative image processing methods by which novel and special-effect images can be obtained. The inventors of the present invention invented the present invention based on the above recognition, and an object of the present invention is to obtain an interesting image. Further, the present invention has been developed in consideration of the following objects or other objects understood through the description of the present patent specification. That is, the targets include new proposals to improve the efficiency of image processing, to reduce the load caused by image processing, to improve the image processing technique, and the like.

A preferred embodiment according to the present invention relates to an image generation method. The method comprises the following steps: an original moving image is regarded as a two-dimensional image that changes along a time axis, and when a moving image is expressed in a virtual manner as a box space formed by the two-dimensional image and the time axis, the box space is cut with a curved surface containing a plurality of points that differ from each other in time value. The image appearing on the section is projected to a plane in the time axis direction, and the image appearing on the plane is output as a new live image by changing the section with time. The changed contents are determined by setting curved surfaces in various ways, and a new moving image different from the contents of the original moving image is output.

Here, the "original moving picture" may be a picture taken by a camera on the spot, or may be a picture encoded in a format such as MPEG stored in advance in a recording medium. "projected onto a plane" is: if projected onto a plane on the time axis, the image to be projected onto that plane is the result of the "projection onto plane". Specifically, it refers to: when a section is viewed directly from the time axis direction, an image to be projected to the section is equal to an image projected to a plane.

For example, "changing a cross section over time" can be achieved by moving the cross section along the time axis while keeping the curved shape of the cross section intact. By moving the cross-section over time, a smooth and continuous new live image can be obtained. The surface shape may change over time. If the position of a point contained in a curved surface on the time axis is represented by t and the coordinates of the point contained in the two-dimensional image are represented by (x, y), t can be defined by a function t = f (x, y) represented by a general equation. The curved surface may be a plane. The projected image varies depending on the type of curved surface shape set.

Another preferred embodiment according to the present invention relates to an image generating apparatus. The device comprises: an image memory for sequentially storing original moving images along a time axis; an image conversion unit that treats an original moving image stored in the image memory as a two-dimensional image changing along a time axis, and when the moving image is expressed in a virtual manner as a box space formed by the two-dimensional image and the time axis, cuts the box space with a curved surface containing a plurality of points different from each other in time value, and projects an image appearing on a section onto a plane in the time axis direction; and an image data output unit that sets an image appearing on the plane, which is obtained by changing the cross section with time in the image conversion unit, as a new moving image frame. The image memory serves as a buffer for temporarily storing a plurality of frames in a fixed period until a frame contained in an original moving image has been converted into a new frame.

The image generating apparatus may further include an image input unit for obtaining images taken by the camera as original moving images and transmitting these obtained images to the image memory. Accordingly, the real-time image processes the photographed image, so that an image unique, mysterious, or special effect different from the actual state of the object can be displayed on the screen.

The image conversion unit may cut the box space with a curved surface defined as a function of coordinates of image areas constituting the two-dimensional image. Here, the "image area" may be an area covering a single pixel or an area covering a block of pixels. The curved surface may be defined by a function that does not depend on the coordinates of the two-dimensional image in the horizontal direction. The "horizontal direction" may be the direction of the scanning line. The image conversion unit may cut the box space with a curved surface defined by a function on the attribute values of the image areas constituting the two-dimensional image. The "attribute value" that decides the display content of each pixel may be various attributes such as a pixel value, a depth value (depth value), an approximation order (order) with a specific mode, a degree of change with respect to other frames, and the like. The attribute value may be an average value or a center value of the image region.

The time value of a point contained in the above-described curved surface can be determined from any parameters in the pixel value, the depth value, the approximation order with the specific pattern, and the degree of change. Further, which image area is to be projected to the above-described plane can be determined according to any parameters in pixel values, depth values, approximation orders with a specific pattern, and degrees of change.

Another preferred embodiment according to the present invention relates to an image generating method. The method comprises the following steps: reading out data corresponding to a position in a picture from at least one frame of a plurality of frames contained in an original moving image for each position in the picture of an image contained in a target frame in the original moving image; synthesizing the read data; and a new moving image is formed by sequentially outputting the frames formed in the composition. Data is read out from a past frame in units of pixels or pixel rows and then synthesized, thereby obtaining an image different from an original moving image. These are so-called patchy images that contain temporally different data mixed and synthesized in units of pixels or pixel rows, so that a unique and mysterious image that cannot exist in the real world can be obtained.

The "target frame" is a frame that is used as a reference when displayed, and may change as time passes. For example, in the existing scanning method, the "target frame" corresponds to the current frame to be output at the present timing. From the target frame, it is determined from which frame the actual data is read out and output. The "in-picture position" may be a position of a pixel line as a scan line, or may be a position of a pixel. The respective data corresponding thereto can be read out from the frame, and the data is synthesized in units of pixels or pixel lines. "Synthesis" may be overlapping, mixing, displacing and bonding.

Another preferred embodiment according to the present invention is directed to an image generating apparatus including an image memory, an image converting unit, and an image data outputting unit. The image memory sequentially records the original moving image for each frame. The image conversion unit reads out data corresponding to a position in the figure from at least one frame recorded in the image memory for each in-figure position of the image contained in the target frame, and synthesizes the data. The image data output unit sequentially outputs the frames synthesized and reconstructed by the image conversion unit. The image memory serves to temporarily store a plurality of frames for a predetermined period of time until the frames contained in the original moving image have been converted into new frames and are no longer used. A "pixel" is a point that constitutes an image displayed on a display screen, and may be a pixel represented by a set of RGB colors.

It should be noted that any combination of the above-described structural elements and expressions which change between methods, apparatuses, systems, computer programs, recording media storing computer programs, data structures, and the like, is effective and encompassed by the embodiments.

Furthermore, the summary of the invention does not necessarily describe all necessary features, so the invention may also be sub-combinations of these described features.

Drawings

Fig. 1 illustrates and expresses in a virtual manner a state in which frames of an original moving image appear continuously along a time axis in the first embodiment according to the present invention.

Fig. 2A and 2B are provided to compare a screen showing a photographed object (object shot) with a screen showing actually displayed contents.

Fig. 3 is a block diagram showing functions of the image generating apparatus according to the first embodiment.

Fig. 4 is a flowchart showing the steps of converting an original moving image into a new moving image according to the first embodiment;

fig. 5 illustrates a moving image as a box space in a virtual manner according to the second embodiment.

Fig. 6A and 6B are provided to compare a screen showing a photographed target with a screen showing actually displayed contents according to the second embodiment.

Fig. 7 is a flowchart showing steps of generating a new moving image by reading out data from a frame in accordance with a Z value in the second embodiment.

Fig. 8 illustrates a moving image as a box space in a virtual manner according to the third embodiment.

Fig. 9A and 9B compare a screen showing a photographed target with a screen showing actually displayed contents according to the third embodiment.

Fig. 10 is a flowchart showing steps of generating a moving image from which a desired color section is extracted, from generating an original moving image, according to the third embodiment.

Fig. 11 illustrates an original moving image as a box space according to the fourth embodiment.

Fig. 12 is a functional block diagram showing the structure of the image generating apparatus.

Fig. 13 shows an example of a screen of a monitor displaying a graph of functions determined by the setting input unit.

FIG. 14 is a flowchart showing steps to generate a direction and manipulation (directing and manipulating) target.

Fig. 15 is a flowchart showing steps of applying orientation and operational effects to a current frame.

Detailed Description

The invention will be described in terms of embodiments, which are not meant to limit the invention but rather to exemplify the invention. All of the features and their combinations described in the embodiments are not essential.

First embodiment

According to the first embodiment of the present invention, a plurality of frames contained in an original moving image are sequentially stored in a ring buffer (see fig. 3), and data is read out from different frames for each scanning line, so that such read-out data is displayed as a new frame on a screen. Specifically, data on pixels on a scan line located at an upper edge of the screen is read out from the updated frame, and data on pixels on a scan line at a lower edge of the screen is read out from a temporally older previous frame. On the screen, what is displayed is a strange and mysterious image different from the actual object.

Fig. 1 illustrates and expresses a state in which frames of an original moving image appear continuously along a time axis in a virtual manner. The original live image is taken as and captured as a two-dimensional image that changes along the time axis. A rectangular parallelepiped (rectangular-shaped) space 10 (hereinafter also referred to as "box space") expands in the direction of the time axis t as time passes. A cross section perpendicular to the time axis t represents a frame. A frame is a set of pixels represented by the coordinates of a plane formed by the x-axis and the y-axis. The box space 10 is cut by a curved surface having a desired shape. As shown in FIG. 1, according to the first embodiment, the box space 10 is defined by a space parallel to the x-axis at a time t ₀ Elapsed time t ₁ And a bevel cut in the direction from above the x-axis to a line below. When the image appearing on the curved surface 14 is projected onto a plane in the time axis direction, the image projected onto the plane on the time axis is output as an actual frame, instead of outputting the current frame 12.

The sections 14 move along the time axis t as time passes. The section 14 is defined in such a way that it has a continuous width in the direction of the time axis t. The images contained in the width are synthesized, and these synthesized images are used as frames actually displayed on the screen.

The current frame 12 corresponds to a frame that should have been located at the timing of the present scanning in the normal display mode. Let the current position of the current frame 12 on the time axis be at time t ₀ . Then, at time t ₀ Previous frames (e.g. respectively at time t) ₁ 、t ₂ 、t ₃ And t ₄ The frame) corresponds to the frame that has been displayed at the normal display timing. However, in the first embodiment, it is actually displayed prior to the time t ₀ The frame of (2). The data of the pixels included in each frame is output in such a manner that the output of the data is sequentially displayed for each pixel row in the horizontal direction. Data of each pixel included in a single pixel row is read out at the same timing, and thenAnd (6) displaying.

The pixel row of the highest point is output at the normal scanning timing. Thereafter, a pixel row of one pixel below the pixel row at the highest point is output with a delay of one frame. Therefore, the lower the order of the pixel rows, the more delayed the timing is output.

The data for each pixel on the screen is read out from the past frames, and the extent to which these frames should be backed off can be represented by a function of pixel coordinates, such as t = t ₀ -y. Function t = t ₀ Y is a function of the y coordinate of the pixel row only and is independent of the x coordinate of the pixel row.

When the resolution of the current frame 12 is 720 × 480, the coordinates of the top-left pixel are (0, 0) and the coordinates of the bottom-right pixel are (719, 479). In this case, the maximum value of the coordinate y is 479, and the scanning timing of the pixel row of the lowest order is delayed by 479 frames. In the box space 10, at time t ₀ And time t ₂ Between which 480 frames are placed.

Fig. 2A and 2B illustrate a screen showing a photographed target and a screen showing an actually displayed target, respectively, and are provided to compare the former with the latter. Fig. 2A shows an image shot, the image of which is equivalent to the current frame 12. Here, the subject swings his/her hand 16 slowly. Fig. 2B is an image appearing on the cross section 14 shown in fig. 1, and is an image actually displayed on the screen of the object of fig. 2A. In other words, the position of the hand 16 changes in order from the left to the midpoint, the right, the midpoint, and the left from the past frame to the current frame, and therefore, by reading out data from different past frames for each scan line, the image of the hand 16 at the left position and the image at the right position appear alternately. Since data on the same data line is read out from frames of the same scanning timing in time, no bending or deformation is caused in the horizontal direction. However, in the vertical direction, the hand 16 is displayed in such a manner that the shape of the hand 16 is meandering and curved on the left and right.

In other words, the object of swinging the hand 16 hardly moves other than the hand 16. Therefore, even if images read out from temporally different frames are synthesized, since there is no difference in their display positions, warping or deformation hardly occurs.

Fig. 3 is a functional block diagram showing an image generating apparatus according to the first embodiment. In terms of hardware, the configuration of the image generating apparatus 50 can be realized by a CPU such as an arbitrary computer and the like. In terms of software, it is implemented by a program or the like having data storage, image processing, and drawing functions, but functional blocks implemented in connection with them are illustrated and described in fig. 3. Accordingly, these functional blocks can be implemented in various forms by hardware only, software only, or a combination thereof.

The image generating apparatus 50 includes an image input unit 52 that obtains an image taken by a camera as an original moving image and sends a frame contained in the original moving image to an image memory; a ring buffer 56 serving as an image memory for sequentially storing original moving images along a time axis; a buffer control unit 54 that controls read frames from the ring buffer 56 and write frames to the ring buffer 56; an image conversion unit 60 that converts the frames stored in the ring buffer 56 into frames for display; a function memory 70 that stores functions referenced at frame conversion; and a display buffer 74 that stores frames for display.

The image input unit 52 may include a CCD that captures a digital image, and a conversion unit that obtains the digital image by a-D conversion. The image input unit 52 may be implemented as a device provided externally in a detachable manner and mounted on the image generating unit 50. The buffer control unit 54 sequentially records the frames of the original moving images input by the image input unit 52 to the area indicated by the write pointer of the ring buffer 56.

For each pixel contained in current frame 12, image conversion unit 60 reads out data corresponding to the pixel from the frame recorded in ring buffer 56, and synthesizes the data. The image conversion unit 60 includes a decision processing unit 62 for determining from which frame the data should be read out for each pixel; a data obtaining unit 64 for reading out data from the frame determined by the decision processing unit 62; and an image forming unit 66 for forming a frame by synthesizing data for each of the read out pixel rows.

In the decision processing unit 62, a decision from which frame the data should be read out is defined and derived from equation (1) below.

P _Fr (x，y，t ₀ )＝P(x，y，t ₀ -y)---(1)

Where x and y are pixel coordinates for current frame 12, and t is the coordinate of the pixel, as shown in FIG. 1 ₀ Is a time value on the time axis t. P is _Fr Is the pixel value of each pixel in the frame that is actually output. It is apparent from equation (1) that the time value of the frame to be output is a function of the y-coordinate only. Therefore, a decision is made for each pixel row from which frame of the plurality of frames stored in the ring buffer 56 the data should be read out, and this decision is not dependent on the x-coordinate.

The function represented by equation (1) is stored in the function memory 70. Other optional functions are also stored in the function memory 70. Which function the user can set to employ via the command obtaining unit 72.

The data for each pixel read out by the data obtaining unit 64 is sequentially written to the display buffer 74 by the image forming unit 66 having an image chip function, thereby constituting a frame.

The image generating unit 50 further includes a command obtaining unit 72 that receives a command from a user; an image data output unit 76 for outputting the frames stored in the buffer 74; and a monitor 78 for displaying the output frames on the screen. The monitor 78 may be a display provided externally to the image-producing device 50.

The image data output unit 76 reads out image data from the display buffer 74 that stores image data for one frame, then converts them into analog signals and sends them to the monitor 78. The image data output unit 76 sequentially outputs the frames stored in the display buffer 74 to output a new moving image.

Fig. 4 is a flowchart showing the steps of converting an original moving image into a new moving image according to the first embodiment. First, a write pointer t indicating the next writing position in the ring buffer 56 is initialized, i.e., t =0 is set (S10) so that the frame is stored from the top area of the ring buffer 56. The frame included in the original moving image is recorded in the t-th area of the ring buffer 56 (S12). Thus, T of one frame is provided ₀ The sum of the areas.

The pixel line number n in the display buffer 74 is initialized, i.e., n =0 is set (S14), so that the data of the pixel lines starting from those corresponding to the top line of the screen are sequentially copied to the display buffer 74. Calculated is a read pointer T which specifies a data read position corresponding to the line number n (S16). Here, T is obtained by T = T-n. The row number increases and the read pointer T returns further to the past frame. Initially, T =0-0=0, and therefore, the readout pointer T indicates the 0 th area.

If the read pointer T is smaller than 0 (S18Y), such a read pointer does not actually exist. Accordingly, the readout pointer moves to the end of the ring buffer 56 (S20). More specifically, the region T of the annular buffer 56 ₀ Is added to the read pointer T. The data obtaining unit 64 reads out the line number n from the frame of the area of the readout pointer stored in the ring buffer 56, and the image forming unit 66 copies the data corresponding to the readout number to the area of the line number n of the display buffer 74.

When the line number N is not the last line in the display buffer 74 (S24N), the line number is incremented by "1" (S26). The line number n continues to be incremented and the processing of S16 to S24 is repeated until the line number reaches the last line. When the line number becomes a number corresponding to the last line, the image data of one frame is stored to the display buffer 74 (S24Y), and the write pointer is incremented by "1" (S28). When the write pointer t indicates the end area of the ring buffer 56 (S30Y), the write pointer t returns to the start area of the ring buffer 56 (S32).

The image data output unit reads out the frame from the display buffer 74, outputs the frame as video data, and causes the monitor 78 to display the frame on the screen (S34). The processing of S12 to S34 is repeated until termination of display is commanded (S36). In this way, data is read out from the same frame in units of pixel rows, and written to the display buffer. However, the pixel row is first a plurality of pixel sets that are the same as the scanning lines arranged in the horizontal direction, and thus the pixel row is data that should be read out at the same scanning timing in the normal setting. Therefore, reading and writing are effectively handled in the scanning process, and an excessive increase in load due to image conversion in the present embodiment can be prevented.

As a modified example of the present embodiment, the decision processing unit 62 may determine a frame to be read out from the x coordinate. For example, for the pixel row located on the left hand side of the screen, the data thereof is read out from the left hand side of the current frame 12, and for the pixel row located on the right hand side of the screen, from the time t shown in fig. 1 ₂ Of a frameThe right-hand row of pixels reads out its data. Then, its cross-section will be defined by the time t parallel to the y-axis ₀ To time t ₂ The inclined plane of (a).

As another modification, the decision processing unit 62 may determine the frame to be read out based on the x and y coordinates. For example, with respect to the pixel row located at the upper left of the screen, the data thereof is read out from the upper left of the current frame 12, and with respect to the pixel row located at the lower right of the screen, from time t shown in fig. 1 ₂ The lower right pixel of the upper frame reads out its data.

As another modification, the scan lines may be disposed in a vertical direction instead of a horizontal direction. In this case, by determining the frame to be read out from the x-coordinate, more efficient image conversion can be achieved.

Second embodiment

In the second embodiment, data is read out from different frames according to a depth value (Z value) specified for each pixel. Thus, in this respect, the processing performed by the second embodiment is different from the processing of the first embodiment in which the frame is determined from the y-coordinate of the pixel. For example, among objects photographed in an original moving image, an object closer to the camera is read out from an older frame. Therefore, the closer the distance between the camera and the target, the more delayed the display timing thereof.

Fig. 5 illustrates a moving image as a box space in a virtual manner according to the second embodiment. Between the objects photographed in the current frame 12, the Z value of the first image 20 is set to "120" and the Z value of the second image 24 is set to "60". A larger Z value indicates that the target is closer to the camera. The delay amount of the display timing is proportional to the Z value. Each pixel of a frame actually displayed on the screen is defined by the following equation (2).

P _Fr (x，y，t ₀ )＝P(x，y，t ₀ -Z(x，y，t ₀ ))---(2)

Wherein Z (x, y, t) ₀ ) Is the Z value of the current pixel cell. As the Z value increases, the frame from which the pixel data is read out is t on the time axis ₀ To t ₁ And t ₂ Is retreated. From time t ₂ The data corresponding to the first image 20 is read out in the area indicated as the third image 22 in the upper frame. From time t ₁ The data corresponding to the second image 24 is read out in the area indicated as fourth image 26 in the upper frame.

In a cross-section of the box space 10, the third image area 22 occupies a time value t ₂ And the fourth image area 26 occupies a time value t ₁ . Time value t of other zone occupation ₀ . Therefore, the points included in the cross section are dispersed at t ₀ 、t ₁ And t ₂ So that its cross section has a discrete width in the direction of the time axis.

The pixels constituting the first image 20 have larger Z values than the pixels of the second image, and the data thereof is read out from the temporally older frame. That is, since the pixels constituting the second image have smaller Z values than the pixels of the first image 20, the time to return to the older frame is shorter.

Fig. 6A and 6B are provided to compare a screen displaying a photographed object (object shot) with a screen showing an object actually displayed. Fig. 6A shows a photographed object, in this case a person 30 who lifts his/her hand and starts to slowly swing the hand left and right and a car 32 traveling behind. Fig. 6B shows an image of the object shown in fig. 6A actually projected onto a screen. The target is displayed on the screen in a state different from the normal setting. That is, the closer the region is to the camera, the more delayed the display timing thereof. Now, in particular, the person 30 is the closest target to the camera, and therefore the amount of delay in the display timing is the largest. Displaying an older image for its pixels will result in an almost identical image of the object, with respect to the part or area that hardly moves. On the other hand, with respect to an image of an area that moves frequently or largely in the left-right or up-down direction, its display portion on the frame is moved. Thus, as shown in fig. 6B, even when its data is read out from an older frame at coordinates corresponding to those of fig. 6A, the image of the area will be transparent or in a diffused (permeated) state. In fig. 6B, the person 30 is displayed on the screen in a state where the hand that frequently moves disappears. The following vehicle 32 is located relatively far from the camera and its Z value is small, so there is little difference between the display states of the vehicle shown in fig. 6A and fig. 6B.

The image generating apparatus 50 according to this second embodiment has substantially the same structure as the apparatus shown in fig. 3. From equation (2) above, the decision processing unit 62 calculates the amount of time to return to the past from each Z value, and then determines from which frame the data is to be read out for each pixel unit. Hereinafter, a frame from which data is to be read out is also referred to as a source frame. The image input unit 52 according to the second embodiment includes a distance measuring sensor for detecting a Z value of each pixel unit. The distance measuring method may be a laser method, an infrared illumination method, a phase detection method, or the like.

Fig. 7 is a flowchart showing steps of generating a new moving image by reading out data from a frame according to a Z value in the second embodiment. First, a write pointer t indicating the next write position in the ring buffer 56 is initialized, i.e., t =0 is set (S100), so that the frame is stored from the top area of the buffer 56. The frame contained in the original moving image is recorded in the t-th area of the ring buffer 56 (S102).

The positions x and y of the target pixel are initialized in the display buffer 74, that is, the positions x and y are set to x =0 and y =0 (S104), so that the pixels starting from the top line of the screen are sequentially copied to the display buffer 74. Calculated is a readout pointer T that specifies readout positions of data corresponding to the pixels x and y (S106). The decision processing unit 62 calculates a readout pointer T from the Z value of each pixel. The data obtaining unit 64 reads out the pixel P from the frame stored in the readout pointer area of the ring buffer 56 _x，y The data of (1). Then, the image forming unit 66 copies the readout data to the pixel P in the display buffer 74 _x，y On the region (S108).

When P is present _x，y Not yet the last pixel of the display buffer 74, i.e. when the pixel P _x，y When the pixel is not the lower right edge (S110N), the pixel P _x，y Move to the next pixel (S112). Repeating S106 to S112Processing up to pixel P _x，y Until the last pixel. When pixel P _x，y When it becomes the last pixel, the image for one frame is written in the display buffer 74 (S110Y) and it is drawn by the image forming unit 66 (S114).

"1" is added to the write pointer t (S116). When the write pointer t indicates the end region of the ring buffer 56 (S118Y), the write pointer t returns to the top region of the ring buffer 56 (S120). The image output unit 76 outputs the drawn image to the display. The processing of S102 to S122 is repeated until the termination of the display is commanded (S124). In this manner, data is read out from the separated frames in units of pixels and then written into the display buffer 74. It should be noted here that the determination of which frame the data is read out from is made separately for each pixel, and the data may be read out from different or the same frames.

Third embodiment

The third embodiment according to the present invention is different from the first and second embodiments in that an image is synthesized by reading out data of pixels having desired parameters from a plurality of frames. The above-described attribute values are pixel values, and for example, when an image is synthesized by reading out image values having only a red component, an image of a mystery and special effect is obtained in which only a desired color as if it were an afterimage is left.

Fig. 8 illustrates a moving image as a box space in a virtual manner according to the third embodiment. In the box space 10, the projection is at time t ₀ 、t ₁ 、t ₂ And t ₃ The person 30 on the frame (2) is a target of holding the red material and slowly waving his or her hand. The respective displayed portions of the red material object images 34, 35, 36 and 37 are projected on respective mutually different frames.

According to the third embodiment, a frame to be used for image synthesis or image synthesis is determined in advance among a plurality of old frames stored in the ring buffer 56. In the case of FIG. 8, at time t ₀ 、t ₁ 、t ₂ And t ₃ For image synthesis. These four frames are a plurality of curved surfaces arranged at fixed time intervals in the box space 10.

Fig. 9A and 9B are provided to compare a screen displaying a photographed object with a screen showing an actually displayed object. Fig. 9A shows a photographed target. Fig. 9B shows an image of the target shown in fig. 9A actually projected onto the screen. Only the red component is extracted and synthesized on the screen, and thus only the images 34, 35, 36, and 37 of the red material object are displayed, and the background thereof is white or black.

Each pixel of the frame actually displayed on the screen is defined by the following equation (3).

Where the alpha or alpha value indicating the composite ratio is represented by equation (4) below.

α＝P _R (x，y，t ₀ -const*i)---(4)

Wherein P is _R Is the red component value of the pixel.

The data obtaining unit 64 reads data for each pixel according to equation (3), and determines whether to combineAnd (5) imaging. In this way, pixel extraction by color is achieved. Although the pixel value of the red component is set to the alpha value in equation (4), the setting thereof is not limited. If the alpha value is set to P _G Or P _B Only the green component or the blue component is extracted and synthesized. Therefore, if any specific color component is contained in the target, only a specific portion containing the specific color component is displayed as if it were a residual image.

The image generating apparatus 50 according to this third embodiment has substantially the same structure as the apparatus shown in fig. 3. The decision processing unit 62 selects a plurality of frames over a fixed time interval according to equations (3) and (4) above. The data obtaining unit 64 and the image forming unit 66 synthesize readout data at a ratio determined by the pixel value of each pixel.

Fig. 10 is a flowchart showing steps of generating a moving image from which a desired color portion is extracted from generating an original moving image according to the third embodiment. First, a write pointer t indicating the next writing position in the ring buffer 56 is initialized, i.e., t =0 is set (S50), and a frame composition number i is initialized, i.e., i =0 is set (S51), so that a frame is stored from the top end area of the ring buffer 56. The frame contained in the original moving image is recorded in the t-th area of the ring buffer 56 (S52).

The positions x and y of the target pixel are initialized in the display buffer 74, that is, the positions x and y are set to x =0 and y =0 (S54), so that the pixels starting from the upper left edge of the screen are sequentially copied to the display buffer 74. The position of the readout pointer T in the ring buffer 56 is calculated as the readout position of the data corresponding to the pixels x and y (S56). The read pointer T is passed T = T ₀ -const t, andindicating a number of past frames returned by fixed time intervals along the time axis. The data obtaining unit 64 reads out the pixel P from the frame stored in the readout pointer area of the ring buffer 56 _x，y The data of (2). Then, the image forming unit 66 copies the readout data to the pixel P in the display buffer 74 _x，y On the region (S58).

Calculating and setting a pixel P _x，y Alpha value of (a) _x，y (S60). When the pixel P _x，y Not yet the last pixel in the display buffer 74, i.e. when pixel P is _x，y When it is not the lower right edge pixel (S62N), the pixel P _x，y Move to the next pixel (S64). The processing of S56 to S62 is repeated until the pixel P _x，y Becomes the last pixel in the display buffer 74. When the pixel P _x，y When it becomes the last pixel in the display buffer 74, the image for one frame is written in the display buffer 74 (S62Y) and drawn by the image forming unit 66 (S66).

If the frame composition number I has not reached the predetermined number I (S68N), 1 is added to the number I and the processing of S54 to S66 is repeated. In the third embodiment, the predetermined number I is "3" and the synthesis is repeated 4 times until the number I of frame synthesis counts from "0" to "3". When the number i of the frame composition reaches the predetermined number i (S68Y), the write pointer t is incremented by "1" (S70). When the write pointer t indicates the end region of the ring buffer 56, the write pointer t returns to the top region of the ring buffer 56. The image data output unit 76 outputs the drawn image data to the display 78 (S76). The processing of S52 to S76 is repeated until termination of the display is commanded (S78). In this way, data of a desired color component is read out from a past frame only in units of pixels and then written to the display buffer.

For example, as a modified example of the third embodiment, the alpha value of the frame corresponds to the composite number i =0, i.e. the alpha value of the current frame 12 may be set instead of P _R Set to P. In this case, the three colors RGB are extracted together so that not only the red material objects 34-37 but also the person 30 appear simultaneously on the display screen of fig. 9B.

Fourth embodiment

The fourth embodiment according to the present invention is different from the third embodiment in that the attribute value in the fourth embodiment is an approximate order (order) indicating between a desired image pattern and an actual image. As a result of the pattern matching, the more closely and closely the image is to the desired image pattern, the older the frame from which its data is read. Therefore, a desired partial image contained solely in the original moving image can be displayed at delayed timing.

Fig. 11 illustrates an original moving image as a box space in a virtual manner according to the fourth embodiment. The current frame 20 contained in box space 10 contains a first image 40. Now assume that the matching is calculated by the image pattern so as to approximate the first image 40, and then the pixels constituting the first image 40 have a higher order approximation to the image pattern than the pixels in the other regions. Therefore, by going further back along the time axis, data corresponding thereto is read out from the past frame according to the approximate order. Here, by returning along the time axis to time t ₂ To have a time value t ₂ The data is read out at the position of the second image 42 in the frame of (a). The cross-section of the box space 10 only occupies the time value t in the region of the second image 42 ₂ And occupying time value t in other areas ₁ . Therefore, the cross section has a discrete width in the time axis direction.

The image generating apparatus 50 according to the fourth embodiment has substantially the same structure as the apparatus shown in fig. 3. The user designates an image pattern via the command obtaining unit 72, and the decision processing unit 62 processes matching between the image pattern and the frame image. As a result of this, pixel-by-pixel detection and approximation order of the image pattern. The decision processing unit 62 determines from which frame the data should be read out for each pixel, based on its approximation order.

A process flow according to the fourth embodiment will be described with reference to fig. 7. First, prior to step S100, the user specifies an image pattern as a target for which matching is calculated, and calculates matching between the current frame 12 and the image pattern so as to detect an approximation order denoted by "S" for each pixel. That is, the approximation rank of the image area is set with respect to the pixels in the image area that approximate the image pattern. Steps S100 to S104 are the same as those of the second embodiment. In step S106, the readout pointer T is determined from the approximation order "S". The read-out pointer T is obtained, for example, by T = T-s (x, y). The subsequent steps are also the same as those of the second embodiment.

Fifth embodiment

In the fifth embodiment, too, data is read out from separate frames and synthesized according to pixel attribute values. The attribute value is different from the third and fourth embodiments in that it is a value representing the degree of time variation of an image area. For example, between objects, areas that move quickly or greatly have large image changes in time, so data is read out from older frames. Therefore, the display of the area having a large image change included in the original moving image can be delayed, so that the larger the image change is, the more delayed the display of the image area is.

The image generating apparatus 50 according to the fifth embodiment has substantially the same structure as the apparatus shown in fig. 3. The degree of temporal change between the target frame and the frame immediately preceding the target frame is detected for each pixel by the detection processing unit 62. The decision processing unit 62 determines from which frame the data is read out, according to the degree of change thereof.

A process flow according to the fifth embodiment will be described with reference to fig. 7. At S106, the T-th frame and the (T-1) -th frame are compared pixel by pixel, and the degree of change indicated by "c" is detected. The read pointer T is determined according to its degree of change "c". The read-out pointer T is obtained, for example, by T = T-c (x, y). When the degree of change "c" increases, the time value representing the degree of returning to the past time along the time axis increases.

Sixth embodiment

The sixth embodiment according to the present invention is different from the first embodiment in that a user can freely determine or define for each scan line from which frame data is read out by using an interface on a screen. The sixth embodiment is described below by emphasizing the difference from the first embodiment.

Fig. 12 is a functional block diagram showing the structure of the image generating apparatus. The image generating apparatus 50 mainly differs from the image generating apparatus 50 according to the first embodiment shown in fig. 3 in that the apparatus shown in fig. 12 includes a setting input unit 80. The setting input unit 80 obtains an input for defining a setting value of the cross section 14 of fig. 1 via the command obtaining unit 72 operated by the user. In the sixth embodiment, as a function of defining from which frame the data of each pixel row is read out, t = t ₀ -y is a predetermined value as a default setting. That is, the function defines the extent to which frames return to the past. Setting input section 80 transmits the display at t = t to display buffer 74 ₀ -an image represented graphically by the relation of the time values on y and the coordinates y of the pixel rows. The image data output unit 76 displays on the display 78 an image generated by the setting input unit 80, which also displays the relationship between the time t and the coordinate y. When viewing the diagram displayed on the display 78, the user operates the command obtaining unit 72, and modifies the shape of the diagram to make the function t = t ₀ -y is changed to another function. For example, the command obtaining unit 72 may be a touch screen connected to a display screen. In this case, a value indicating a position where the user presses the touch panel is input as the operation content.

The command obtaining unit 72 sends the user operation content to the setting input unit 80 to change the diagram displayed on the display 78. The input unit changing function t = t is set according to the operation content obtained from the command obtaining unit 72 ₀ Y, to set a new function and thus store the newly set function in the function memory 70. The decision processing unit 62 reads out the function set by the setting input unit 80 from the function storage unit, and determines from which frame the data is read out for each pixel row according to the new function. As a result thereof, the box space 10 shown in FIG. 1 is cut by a curved surface defined by the function set by the setting input unit 80, and an image appearing on the cross section is output instead of the current frame 12 asThe actual frame. By implementing the above configuration, the user can utilize the image generating apparatus 50 as an authorization tool, and can generate a mystery and unique image by freely changing the map displayed on the screen.

Fig. 13 shows a screen of a monitor displaying a graph of functions determined by the setting input unit. At the beginning, the indication function t = t is displayed on the setting screen 82 ₀ A line 84 of the relation between time t and coordinate y in-y. The user may change the line 84 to a bezier curve 86. Bezier curve 86 is a curve connecting first end point 88 and second end point 90, the shape of which is determined by the positions of first control point 96 and second control point 98. The location of the first control point 96 and the location of the second control point 98 are determined by the user changing the location and length of the first handle 92 and the second handle 94. If the function set by the setting input unit 80 is specified by the bezier curve 86, an image in which data read out from a frame adjacent to the current frame and data read out from a past frame are mixed together for each pixel row is obtained. For example, via the setting input unit 80, the user can specify a periodic curve, such as a sine wave curve, by the bezier curve 86. Although in this embodiment, the function is specified by the bezier curve 86, a structure may be provided as an improved example in which the function is specified by other curves such as B-splines or the like.

Seventh embodiment

The seventh embodiment according to the present invention is different from the sixth embodiment in that the setting input unit 80 obtains the coordinates of a feature point in the current frame 12 as one of the setting values, and defines a function by the coordinates of the feature point. The present embodiment is described below by emphasizing its difference from the sixth embodiment.

The command obtaining unit 72 is also a touch screen connected to a display 78. When the user presses the touch screen at a desired position and moves the touch point in a circle, a plurality of continuous values representing the coordinates of the touch point are transmitted to the setting input unit 80. Based on the obtained coordinate values, the setting input unit 80 recognizes an area surrounded and covered by a plurality of touch points, and generates a function for determining the surrounded area to be recorded into the function memory 70. Data on pixels contained in the surrounding area is read out from the past frame, and data on pixels not contained in the surrounding area is read out from the current frame 12, according to the function read out from the function memory 70. As a result thereof, the box space 10 shown in fig. 1 is cut by a curved surface defined by the coordinate function obtained by setting the input unit 80, and an image appearing on a cross section is output as an actual frame instead of the current frame 12. By implementing the above configuration, the user can utilize the image generating unit 50 as an authorization tool, and can generate a mystery and unique image by designating an arbitrary area on the touch screen.

Eighth embodiment

The eighth embodiment according to the present invention is different from the other embodiments in that a function is defined in advance in such a manner that a predetermined changed shape appears on a screen, and data is read out from a frame determined according to the function. According to this eighth embodiment, a function is defined in advance in such a manner that a waveform change shape such as a water ring appears on the screen.

The decision processing unit 62 determines feature points in the current frame 12. Similar to the sixth and seventh embodiments, the feature points are specified by the user via a touch screen of a screen connected to the display 78, which serves as the command obtaining unit 72. The decision processing unit 62 determines the source frame and pixel coordinates so that the waveform of the water ring appears from the feature point as its center. Here, the source frame is a frame from which data is to be read out. For example, in order to represent a stereogram of a water ring, assuming that circles are displayed in the radial direction from the feature points, the decision processing unit 62 determines a source frame using a fade time value for each radial circle. The change in the fade time value is defined such that it is a periodic change. So that unevenness of the water ring can be expressed. Further, the decision processing unit 62 moves the pixel coordinates to be read out in a predetermined direction by a predetermined amount. So that the refraction of light by the water ring can be represented.

Ninth embodiment

The ninth embodiment according to the present invention is different from the seventh and eighth embodiments of the command obtaining unit 72 in which a touch screen is used as an input feature point in that a feature point is determined based on information contained in the current frame 12.

The decision processing unit 62 according to the ninth embodiment determines a feature point from the pixel value of each pixel included in the current frame 12. For example, an LED that blinks at a high speed is synthesized onto the target to become a part of the target, and the decision processing unit 62 recognizes the blinking part by specifying an area where the pixel value intermittently changes between two values in the current frame 12 that is continuously input. The decision processing unit 62 determines the blinking part as the coordinates of the feature point. As a modified example, the decision processing unit 62 may determine the feature points using fixed coordinates. As another modified example, the decision processing unit 62 may determine that the pixel is a feature point if any of the following factors, i.e., the pixel value, the Z value, the approximation order from the desired pattern, and the change in the pixel value of the pixel, falls within a predetermined range.

Tenth embodiment

The tenth embodiment according to the present invention is different from the other embodiments in that the image input device 52 obtains not only the original moving image but also audio data. The audio data obtained by the image input device 52 is input in synchronization with the original moving image so as to be sent to the ring buffer 56. The decision processing unit 62 determines at least one of a source frame, readout timing, an alpha value, and a feature point according to the frequency distribution, volume change, and the like of the audio data. For example, when the volume change of the audio data exceeds a threshold, the decision processing unit 62 may determine the source frame and the read-out pixel in such a manner that the shape described in the eighth embodiment appears on the screen. For example, if the change in the volume exceeds the threshold value of the frequency domain section, the decision processing unit 62 may determine the feature points in the eighth and ninth embodiments from the frequency domain.

Eleventh embodiment

According to the eleventh embodiment of the present invention, a predetermined pattern is combined in the vicinity of the position of a pixel in accordance with the attribute value of the pixel contained in the target frame. Here, the attribute value is a digital value indicating a degree of time variation of the image area. For example, a fast or largely moving region in the target is continuously displayed so that an artistic and vividly represented target having a particle form (form of particle) is deformed in a manner spreading from a time-varying larger pixel toward its periphery. In this way, it is possible to produce an orientation and an operation effect such as a paper-snowfall-like effect displayed on a screen at the periphery of a main object such as a moving area or a trajectory in an original moving image.

The image generating unit 50 according to the eleventh embodiment has a structure similar to that of the apparatus shown in fig. 3. The decision processing unit 62 detects, for each pixel, the degree of time variation of the pixel values of the image areas constituting the frame between the current frame 12 and the frame temporally preceding the current frame 12. If the degree of change of this pixel exceeds a predetermined threshold, the decision processing unit 62 regards the pixel as the center position for orienting and manipulating the target. If there are a plurality of pixels whose degree of change exceeds the threshold value and are arranged adjacent to each other, one of them having the largest degree of change may be determined as a center position, and a plurality of orientation and manipulation targets may be displayed in a diffused manner around the center position. Further, the decision processing unit 62 may determine the moving direction of the orientation and the operation target based on the pixel in the current frame 12 and the pixel in the previous frame, each of which has the largest degree of change in each frame.

FIG. 14 is a flowchart showing steps for generating orientation and operational goals. First, the current frame 12 is input as a processing target (S150). If the reproduction process is not performed immediately after the input of the current frame 12 is started (S152N), the change in pixel value is extracted between the current frame 12 and the temporally preceding frame thereof (S154), the position where the pixel value change is the largest is detected (S156), and the vector of the position where the pixel value change is the largest is determined as the direction of movement of the orientation and operation target (S158). If the reproduction processing is executed immediately after the input of the current frame 12 is started, there is no such previous frame, and thus the processing S154 to S158 (S152Y) is omitted. The current frame 12 is stored separately to be compared with the next frame (S160). An image of the orientation and the operation target to be displayed is generated around the position detected at S156 as the center (S162). The orientation and operation target thus generated are overlapped with the current frame 12 so as to process the picture to be displayed (S164). By repeating the processes S150 to S164 until the display is terminated (S166Y), the orientation and the operation target are displayed when moving along the moving method determined in S158.

In the above example, the structure of determining the display orientation and the position of the operation target according to the pixel value change from the frame immediately before that is described. As another example, the decision processing unit 62 may determine a location to perform an orientation and operational effect based on color components, contours, brightness, Z-values, motion trajectories, and the like. For example, a position where an orientation and operation effect is to be produced may be determined according to the size of a pixel value such as "a position containing the most red component in the figure", or an outline whose difference in pixel value between itself and other adjacent outlines is the largest in a single frame may be determined as an orientation and operation portion. Here, the "position where the orientation and operation effect is to be produced" will also be simply referred to as "orientation position" hereinafter. Further, for example, the difference between pixel values adjacent to the "red contour" is larger than a threshold value, and a portion whose color component is larger than the threshold value may be determined as the orientation position. Further, a portion having a luminance greater than a threshold may be determined as an oriented portion, and a portion having a specific Z value range may be determined as an oriented portion. If a plurality of past frames are stored within a fixed time limit, a trajectory (lorus) of feature points extracted according to a certain criterion may be detected. Thus, orientation and manipulation effects can be produced along such a trajectory. As an orientation and operation effect, the decision processing unit 62 may display a linear object or a character to which a blinking color or other features are applied, or an object such as a symbol. Further, the decision processing unit 62 may produce an orientation and an operation effect in which the transparency of the characteristic region extracted from the current frame 12 becomes a translucent state so as to be overlapped to the past frame.

The decision processing unit 62 may determine the size and moving speed of the orientation and operation target to be displayed, based on attribute values such as coordinates, Z values, and pixel values of each pixel, approximation order with a desired image pattern, and change rate in pixel values. The alpha value for combining the orientation and the manipulation target may be a fixed value or different for each pixel. For example, the alpha value may be set according to attribute values such as coordinates, a Z value, and a pixel value of each pixel, an approximation order from a desired image pattern, a degree of change in pixel values, and the like.

Fig. 15 is a flowchart showing steps of applying the orientation and the operational effect to current frame 12. First, the type of orientation and operation effect to be applied to the current frame 12 is determined according to a command from the user (S200). Then, the current frame 12 generating the orientation and operation effect is input (S202). Then, a portion where the difference in pixel values of the neighboring pixels in the current frame 12 exceeds the threshold is extracted as a contour (S204), and a portion where the difference in pixel values is the largest is determined as a portion where the orientation and operation effect is generated (S206). The orientation and manipulation effects are then applied to the determined portions (S208), and a map for displaying an image that produces the orientation and manipulation effects is processed (S210). The above processes S202 to S210 (S212N) are repeatedly performed on the current frame until the display is terminated (S212Y) so that the orientation and the operation effect are applied thereto.

Twelfth embodiment

According to the twelfth embodiment, the reproduction frame rate is locally changed in accordance with a change in the pixel attribute value contained in the target frame. That is, the section 14 changes with time at different rates for each image area according to the attribute value of the image area constituting the two-dimensional image, so that the frame rate of the new moving image to be output from the image data output unit 76 locally changes. For example, for a portion in which the degree of change in pixel value is larger than the threshold value in time, the time interval for reading out data from a frame becomes longer, thereby lowering the reproduction frame rate. Thus, a mysterious and unique image will be produced in which the faster the area actually moves, the slower the area displayed on the display moves, and thus a part of the object in the moving image moves at a different rate from the normal speed.

The image generating apparatus 50 according to the twelfth embodiment has substantially the same structure as the apparatus shown in fig. 3. The decision processing unit 62 according to this embodiment can change the frame rate on a pixel-by-pixel basis or change the frame in units of objects extracted according to their pixel values. The decision processing unit 62 may extract the object in such a way that some pixels surrounding the pixel are included as part of the range in question. In this case, a portion such as an edge of the object whose pixel value gradually changes may be included as a part of the object and also processed.

Regarding the area whose frame rate is to be changed, the decision processing unit 62 determines the frame rate after the change. For example, the frame rate may be set according to the degree of time variation of the pixel values of the regions, and the frame rate of the region whose rate is changed more may be set to a lower value. Based on the frame rate thus determined, the decision processing unit 62 determines the time interval between the reading source frame and the next frame for each pixel. The decision processing unit 62 may change the frame rate over time. For example, the decision processing unit 62 first sets the frame rate to a low rate and gradually increases the frame rate so that it follows the display timing of other pixels surrounding the pixel.

As a modified example, a structure may be such that the user, via the command obtaining unit 72 shown in fig. 12, can set whether or not to execute processing in such a manner that the edge of the target is included in the range of the target. The frame rate of pixels having a predetermined range of Z values in current frame 12 may be changed. The position in the current frame having a predetermined approximation order to the desired image pattern may be changed. In other words, the frame rate is controlled on a pixel-by-pixel basis.

Thirteenth embodiment

According to the thirteenth embodiment of the present invention, data of pixels having a predetermined attribute value in a target frame is read out from a temporally previous frame other than the target frame. For example, data corresponding to pixel values of black is read out from an old frame so that a vivid effect can be produced as if a part of a past image is viewed from a trimming shape (trimming shape) window.

The image generating unit 50 according to the thirteenth embodiment has substantially the same structure as the apparatus shown in fig. 3. The decision processing unit 62 according to the present embodiment extracts a region having a predetermined pixel value range from the current frame 12, and at the same time determines a source frame of the region. The source frame may be a past frame obtained by returning for a predetermined period along the time axis, or may be determined from attribute values such as coordinates, a Z value, a pixel value of each pixel, an approximation order with a desired image pattern, a change amplitude of the pixel value, and the like.

As an improved example, the older the frame, the more sequentially the frames are arranged (gradation) so that the older can be emphasized and vividly represented. The decision processing unit 62 may extract a region that also includes some surrounding pixels. For example, regions corresponding to the mouth and some pixels around the mouth are extracted together from the face of a person, thereby positively extracting a portion where the pixel value gradually changes, such as an edge of an object. In the above embodiment, data is read out from a temporally preceding frame. However, if temporally future frames are also stored in the ring buffer 56, data can be read out from these future frames.

Fourteenth embodiment

In the fourteenth embodiment of the present invention, a pixel value is added to a pixel in accordance with a change in an attribute value of the pixel contained in a target frame, thereby changing a color. For example, the orientation and manipulation effects are applied to the original image in such a manner that a largely moved region in the target is displayed in red or the like.

An image generating apparatus 50 according to the fourteenth embodiment has substantially the same structure as the apparatus shown in fig. 3. The decision processing unit 62 according to this embodiment adds a predetermined value to the pixel value of the pixel, so that the pixel of the current frame 12 that has changed greatly is displayed in red. Thereafter, with respect to the pixel to which the predetermined value is added, the pixel value to be added to the pixel is gradually decreased with time, and as a result thereof, an afterimage in which a red tail is left can be displayed.

As a modified example of the image generating apparatus 50, the configuration may be such that data of a pixel which changes drastically in time may be synthesized with a predetermined alpha value in such a manner that the pixel remains on the screen even after the pixel has been displayed. The pixel values may also be added to the synthesized data so that an image thereof can be displayed in a desired color. Thereafter, the alpha value of the data to be synthesized gradually decreases with time, and as a result thereof, an afterimage that leaves a tail can be displayed. As another modified example, the structure may be such that: by using a predetermined alpha value, pixel data whose degree of change is high is synthesized with a screen in which the alpha values of all pixels with respect to the screen are set to zero, and the entire screen is gradually displayed. As another modified example, the pixel value added to each pixel may be changed, and the color of the pixel may be changed by adding the pixel value of the pixel in an accumulated manner.

Fifteenth embodiment

In a fifteenth embodiment of the present invention, a target frame is synthesized using a predetermined target in accordance with the attribute values of pixels contained in a future frame. That is, between frames contained in the original moving image stored in advance, the granular object as in the eleventh embodiment is displayed on an area close to a predetermined image pattern in the frame to be displayed. Thus, a directional effect like broadcasting (announcement) can be produced. For example, an image such as paper-snowfall is displayed in front of a main person who is targeted here and is exposed on the screen.

An image generating apparatus 50 according to the fifteenth embodiment has substantially the same structure as the apparatus shown in fig. 3. The decision processing unit 62 according to the present embodiment detects, from the frames included in the original moving image stored in advance in the ring buffer 56, an area falling within a predetermined range with respect to the approximation rank of a predetermined image pattern in a frame to be displayed temporally later. The decision processing unit 62 synthesizes granular objects around the detected region. The method of combining and composing images is similar to that described in the eleventh embodiment.

As an improved example, the composition of objects may be applied to a real-time live image, which is taken in parallel with the current reproduction of the image. That is, moving images obtained immediately after shooting are temporarily stored in a buffer, and then each frame is reproduced at a timing delayed from the shooting timing. A predetermined image pattern is extracted from a current frame obtained immediately after photographing while reproducing the frame thereof at a timing delayed from the photographing timing, whereby a directional effect similar to broadcasting can be produced.

The present invention has been described in terms of embodiments that are exemplary only. It will be understood by those skilled in the art that there are other various modifications regarding the combination of each of the components and processes described above, and that such modifications are encompassed by the scope of the present invention.

In the second embodiment, from which frame data is read out is determined according to the Z value. However, in a modified example, a plurality of frames are set as source frames at fixed time intervals, and the plurality of frames can be synthesized by a ratio according to the Z value. Here again, the source frame is a frame from which data is read out. In this case, the alpha value is determined based on the Z value. A region having a relatively large Z value in the target, that is, a region close to the camera may be set in such a manner that its alpha value is set larger. In this case, the area close to the camera will be projected more clearly, while the severely moved area will be displayed in a residual manner as if it were an afterimage.

In the third embodiment, the alpha value is set according to the pixel value. In yet another improved example, the source frame may be determined from pixel values. For example, when a pixel value having a red component is extracted, data on an area having more red components is read out from an older frame so that display of an area containing more red components is further delayed.

In a fourth embodiment, the source frames are determined according to an approximation order to the desired image pattern. In yet another refinement, the alpha value may be set based on the approximation order. In this case, the area closer to the image pattern is displayed more clearly, and the area that moves quickly and largely is displayed in a manner of further remaining afterimages. Further, a plurality of different image modes are prepared in advance, and the source frame can be read out with an approximation order according to which specific mode is to be used. Alternatively, the alpha value may be determined using an approximation order based on which particular mode is to be used. The recognition of the image may be not only recognition of each frame but also recognition of a gesture across a plurality of frames.

In the fifth embodiment, a source frame is determined according to the degree of time-variation in an image region. In yet another modified example, the alpha value may be set according to the degree of change. In this case, the region having a larger degree of change is displayed more clearly, and the region having a larger degree of change is also displayed in a manner of remaining afterimages.

In each of the first to fifteenth embodiments, the correspondence relationship of pixels in a plurality of frames is determined by the same coordinates (x, y). However, according to another modification, the correspondence may be determined by shifting the coordinates of a specific pixel, or whether the shift should be made may be determined according to the attribute or the width of the pixel.

In the second to fifth embodiments, the source frame is determined based on each single attribute value or alpha value. However, in another modification, the source frame or alpha value may be determined based on a plurality of attribute values in the Z value, pixel value, approximation order, and degree of change. For example, after a source frame has been determined for a particular pixel based on its Z value, a pattern match may be calculated between the frame and current frame 12, and then multiple frames may be synthesized based on the alpha values corresponding to their approximation orders. In this case, if the target is closer to the camera, data is read out from an older frame, and further, a largely moving area is displayed in such a manner that an afterimage remains.

In each of the first to fifteenth embodiments, a structure is provided such that source frames corresponding to a certain period contained in an original moving image are stored in the ring buffer 56. In yet another modification, the image input unit 52 may read out the frames determined by the decision processing unit 62 from the original moving image compressed in the MPEG format, and the buffer control unit 54 may cause the frames to be stored in the ring buffer 54. Further, the buffer control unit 54 may refer to frames before and after the frame.

Hereinafter, further modifications will be described.

1-1. In the first embodiment, a configuration is provided such that data is read out from different frames for each pixel row. In this modified example 1-1, a structure may be such that some past frames as source frames are set, and data is read out from any of these set frames for each pixel row. For example, a configuration may be such that two frames a and B are set as source frames, and data is read out from a and B according to whether the order of pixel rows is odd or even. For example, the structure may be such that six frames a, B, C, D, E, and F are set as source frames, data of 0 th to 79 th pixel rows are read out from the frame a, and data of 80 th to 159 th pixel rows are read out from the frame B, and so on. In other words, the pixel rows are divided in units of 80 lines and data for each cell of the division line is read out from different past frames. When data is output from respective different frames for each unit composed of 50 pixel rows on the screen, a band pattern or the like appears on the moving area.

1-2. In the second embodiment, a structure is provided such that data is read out from different frames according to the Z value. In this modified example 1-2, a structure may be such that data of only pixels having a predetermined Z value range is read out from a past frame. For example, at least one of the upper limit and the lower limit of the Z value is set in advance, and one or more past frames from which data is read out are set in advance. With respect to pixels having Z values falling in a set range (below the upper limit and above the lower limit), data is read out from the set past frame. With respect to pixels having Z values outside the set range, data is read out from the current frame 12. The number of source frames to be set here is fixed to one or more frames, or the source frames may be past frames according to pixel coordinates of the source frames.

1-3 in the third embodiment, a structure is provided such that data of pixels having predetermined pixel values are read out from a plurality of past frames and synthesized with the current frame 12. In this modified example 1-3, a structure may be such that: data is read out from a predetermined past frame with respect to pixels of the current frame 12 having a predetermined pixel value, and data is read out from the current frame 12 with respect to other pixels. Further, the past frame serving as the source frame may be set in a fixed manner, or the source frame may be the past frame according to its pixel value.

1-4. In the fourth embodiment, a configuration is provided such that data is read out from a past frame corresponding to an approximate order of a desired image pattern. In this modified example 1-4, a configuration may be such that only data of pixels whose approximate order from the desired image pattern falls within a predetermined range is read out from the past frame, and other pixels are read out from the current frame 12. As the range of the approximation order, at least one of the upper limit and the lower limit thereof may be set in advance. The past frame used as the source frame may be set in a fixed manner, or the source frame may be a past frame obtained by returning along the time axis according to the approximation order.

1-5 in the fifth embodiment, a structure is provided such that data is read out from a past frame in accordance with a change in pixel value. In this modified example 1-5, a configuration may be such that only data of pixels whose pixel value changes fall within a predetermined range is read out from the past frame, while data of other pixels are read out from the current frame 12. The past frame serving as the source frame may be set in a fixed manner, or the source frame may be a past frame obtained by returning along the time axis according to a pixel value change.

1-6. In a first embodiment, a function t = t is defined in terms of a time value t and a pixel coordinate y ₀ -y. In this modified example 1-6, the relationship between time t and pixel coordinate y can be defined as using threeT = siny for the angular function, etc. In the graph described in fig. 13, a pixel row whose data is read out from the past frame further returning along the time axis and another pixel row whose data is read out from the updated frame are periodically mixed. Further, as in modified example 1-1, a format may be such that some past frames as source frames are set in advance, and data of each pixel row is read out from any of these set frames corresponding to the time value t.

2-1. In the first embodiment, a structure is provided such that data is read out from a past frame for each pixel row, and these data are arranged in a vertical direction so as to construct one frame. In this modified example 2-1, a structure may be such that data read out from the past frame for each pixel row is synthesized with the current frame 12 to form one frame. In this case, the alpha value may be a fixed value or may be different for each pixel row. For example, the alpha value may be set based on coordinates, a Z value, a pixel value of each pixel, its approximate order from a desired image pattern, a change in its pixel value, and the like.

2-2. In the second embodiment, a structure is provided such that data is read out from different frames according to the Z value. In this modified example 2-2, a structure may be such that data read out from different frames according to the Z value is synthesized with the current frame 12 to generate one frame. Alternatively, data of only pixels having a predetermined range of Z values in the current frame 12 is read out from the past frame, and such data is synthesized with the current frame 12 to form one frame. The alpha value in this case may be a fixed value or may be different for each pixel row. For example, the alpha value may be set according to the coordinates, the Z value, the pixel value of each pixel, its approximate order from the desired image pattern, the change in its pixel value, and the like.

2-3 in the third embodiment, a structure is provided such that data of pixels having predetermined pixel values are read out from a plurality of past frames to be synthesized with the current frame 12. In this modified example, a structure may be such that: as for the pixels having a predetermined pixel value in the current frame 12, data thereof is read out from a predetermined past frame, and the data is synthesized with the current frame 12. The alpha value in this case may be a fixed value or may be different for each pixel row. For example, the alpha value may be set according to coordinates, a Z value, a pixel value of each pixel, an approximation order thereof to a desired image pattern, a change in a pixel value thereof, or the like.

2-4. In the fourth embodiment, a configuration is provided such that data is read out from a past frame corresponding to an approximate order of a desired image pattern. In this modified example 2-4, a structure may be such that data read out from a past frame corresponding to an approximation order with a desired image pattern is synthesized with the current frame 12. Alternatively, data of only pixels whose approximate order with the desired image pattern falls within a predetermined range is read out from the past frame, and such data is synthesized with the current frame 12. The alpha value in this case may be a fixed value or may be different for each pixel row. For example, the alpha value may be set based on coordinates, the Z value, the pixel value of each pixel, its approximate order to the desired image pattern, the change in its pixel value, and the like.

2-5. In the fifth embodiment, a configuration is provided such that data is read out from past frames returned along the time axis in accordance with a change in pixel value. In modified examples 2 to 5, a structure may be such that data read out from a past frame is synthesized with a current frame 12 according to a pixel value change. Alternatively, data of only pixels whose pixel value changes fall within a predetermined range is read out from the past frame, and such data is synthesized with the current frame 12. The alpha value in this case may be a fixed value or may be different for each pixel row. For example, the alpha value may be set according to coordinates, a Z value, a pixel value of each pixel, an approximation order thereof to a desired image pattern, a change in a pixel value thereof, or the like.

2-6. In modified examples 1-6, the relationship between time t and pixel coordinate y is defined as t = sin y or the like using a trigonometric function. As a further modification thereof, data read out from the current to past frames is synthesized with the current frame 12 according to a function using a triangle function such as t = sin y. The alpha value in this case may be a fixed value or may be different for each pixel row. For example, the alpha value may be set according to the coordinates, the Z value, the pixel value of each pixel, its approximate order from the desired image pattern, the change in its pixel value, and the like.

2-7 in the sixth embodiment, a structure is provided such that data is read out from a frame corresponding to the bezier curve 86 set by the user on the setting screen 82. In this modified example 2-7, the structure may be such that the data read out from the frame is synthesized with the current frame 12 in accordance with the bezier curve 86 set by the user on the setting screen 82. The alpha value in this case may be a fixed value or may be different for each pixel row. For example, the alpha value may be set according to coordinates, a Z value, a pixel value of each pixel, an approximation order thereof to a desired image pattern, a change in a pixel value thereof, or the like.

3-1. In this modified example, a structure may be such that two or more copies of data read out for each pixel in at least two or more embodiments or modified examples of the first to fifteenth embodiments, 1-1 to 1-6 modified examples, and 2-1 to 2-7 modified examples are synthesized. The alpha value in this case may be a fixed value or may be different for each pixel row. For example, the alpha value may be set according to the coordinates, the Z value, the pixel value of each pixel, its approximate order from a desired image pattern, a change in its pixel value, and the like.

While the invention has been shown and described with reference to certain preferred embodiments thereof, it will be understood by those skilled in the art that various changes in form and details may be made therein without departing from the spirit and scope of the invention as defined by the appended claims.

Claims (20)

1. An image generation method, comprising:

reading out data corresponding to a position in a picture from at least one frame of a plurality of frames contained in an original moving picture for each position in the picture contained in a target frame of the original moving picture;

synthesizing the read data; and

a new moving image is formed by sequentially outputting frames formed at the time of the composition.

2. An image generating method according to claim 1, wherein said readout is such that data to be read out from at least one frame of the plurality of frames is determined for each in-picture position in accordance with its coordinates.

3. An image generating method according to claim 1, wherein said synthesizing is such that the read data is synthesized at a ratio according to the attribute value of the image contained in at least one of the plurality of frames.

4. An image generating apparatus includes an image memory, an image converting unit, and an image data outputting unit,

wherein the image memory sequentially records original moving images of each frame, and the image conversion unit reads out data corresponding to positions in the figure from at least one frame recorded in the image memory for each position in the figure of the image contained in the target frame and synthesizes the data, and the image data output unit sequentially outputs the frames synthesized and reconstructed by the image conversion unit.

5. The image generating apparatus according to claim 4, wherein said image conversion unit determines at least one frame for each in-picture position according to its coordinates.

6. An image generating apparatus according to claim 5, wherein the coordinates are such that they are orthogonal to the scan lines.

7. The image generating apparatus according to claim 4, wherein said image converting unit determines at least one frame for each in-picture position according to its attribute value.

8. An image generating apparatus according to claim 4, wherein said image converting unit determines a plurality of frames as said at least one frame at predetermined time intervals, and said image converting unit synthesizes said plurality of frames for each in-picture position at a ratio according to the attribute value thereof.

9. The image generating apparatus according to claim 4, wherein for each in-picture position of the image contained in the target frame, said image conversion unit applies a directional effect according to an attribute value of its position.

10. The image generating apparatus according to claim 7, wherein said image converting unit sets the time intervals of the determination frames to separate time intervals for each in-picture position according to the attribute value thereof.

11. An image generating apparatus according to claim 4, wherein the target frame or said at least one frame is at least one of a temporally preceding frame or a temporally subsequent frame with respect to a reference frame that should have been naturally output from said image memory by said image data output unit.

12. The image generating apparatus according to claim 4, wherein said image converting unit adds a predetermined pixel value according to an attribute value thereof for each in-picture position of the image contained in the target frame.

13. An image generating apparatus according to claim 7, wherein the attribute value is a depth value.

14. An image generating apparatus according to claim 7, wherein the attribute value is a value indicating an approximation order with respect to a desired image mode.

15. The image generating apparatus according to claim 7, wherein the attribute value is a value indicating a degree to which the image area temporally changes.

16. An image generating apparatus according to claim 7, wherein the attribute value is a pixel value.

17. An image generating apparatus according to claim 4, further comprising an image input unit for obtaining images taken by the camera as original moving images and sending the images to said image memory.

18. The image generating apparatus according to claim 4, further comprising a setting input unit operable to obtain, via a user operation, an input for determining the setting value of the at least one frame, wherein the image converting unit determines the at least one frame based on the setting value obtained by the setting input unit.

19. The image generating apparatus according to claim 13, wherein the setting value obtained by said setting input unit is represented by a curve indicating a relationship between coordinates of a point contained in the two-dimensional image and a time value thereof when displayed on the screen.

20. The image generating apparatus according to claim 13, wherein the setting input unit obtains coordinates of the feature point in the two-dimensional image as the setting value, and the image converting unit determines the at least one frame based on the coordinates of the feature point.

Images