JP4285422B2 - Moving image generation system, moving image generation apparatus, moving image generation method, program, and recording medium - Google Patents

Description

  The present invention relates to a technique for generating a smooth moving image using a plurality of images obtained by photographing a subject in advance from various directions.

  In recent years, three-dimensional CG has been actively used for presentation of products and the like with the improvement of multimedia processing capability of computers and the spread of broadband. A good example is MediaPlayer from Viewpoint. The user can browse the product he / she wants to see from his / her favorite direction and angle on the computer screen. At this time, it is most important to display an image smoothly according to a change in the direction in which the object is viewed.

  A general three-dimensional CG is generated by a polygon-based method. By generating a geometric model by this method, a smooth moving image viewed from an arbitrary position and direction can be obtained. However, in the case of generating a highly dynamic moving image based on a polygon, there is a problem that modeling takes too much time and time.

  In general, 3D CG uses geometry-based data representing a shape with a plurality of polygons. By this method, a smooth moving image viewed from an arbitrary position and direction can be obtained.

  As a technique for automatically generating this geometry-based data, there is a technique called Shape from Silhouette. In this method, as shown in FIG. 1, a subject object is photographed from n places, and silhouettes (binary data) of n subject objects are extracted from the photographed image. This silhouette (1,..., L,..., M,..., N) is projected from the points photographed in the virtual space, and the three-dimensional shape data of the subject object can be obtained by obtaining the overlapping portion. it can. By adding color data of the surface of the subject object obtained from the photographed image to this shape, three-dimensional model data can be obtained.

  However, with this method, details of the subject cannot be accurately represented, and indentations cannot be represented. Therefore, manual work is required to obtain high-quality data, and modeling takes cost and time.

  For this reason, attention has been paid to a three-dimensional CG of an image-based method for reconstructing a scene in a three-dimensional space using a two-dimensional image obtained by photographing a subject from a plurality of directions in advance. Since this method does not require modeling work, a highly realistic three-dimensional CG can be generated easily and in a relatively short time. Apple's QuickTimeVR is a typical example of this.

  Patent Document 2 describes a three-dimensional model generation method capable of accurately performing segmentation. Patent Document 3 describes an image processing method that reproduces translucency and reflectivity to obtain a natural image.

  In the above method, in order to display a three-dimensional CG as a smooth moving image that follows the position and direction of the viewpoint that changes every moment, since the subject must be photographed from all directions in advance, data handled in comparison with shape modeling The amount becomes enormous. For example, when photographing a subject from all directions at 5 ° intervals that can be felt smoothly, it is necessary to photograph about 600 images, which is practical from the viewpoint of not only the number of images to be photographed but also the amount of data and the time required for photographing. It can no longer be said to be a natural thing.

  FIG. 2 shows a processing procedure in the conventional image-based method. As shown in FIG. 2, in the conventional image-based method, a subject is first photographed, and then a moving image is generated by performing image processing and editing. In this case, shooting conditions such as the interval at which shooting is performed are manually specified. However, in order to create a moving image that feels smooth as described above, it is necessary to shoot a large number of images, and in the time required for about 30 minutes, the number of original captured images is inevitably insufficient, so a jerky display The conventional image-based method is not suitable for use as a smooth moving image.

On the other hand, as another example of a three-dimensional CG generation method using an image-based method, a method of interpolating and drawing an image between two live-action images has been proposed. Non-patent document 1 is a typical example of such research. Further, this technique is used in the image composition display method described in Patent Document 1, and this technique is described. In the method using the two actual images, as shown in FIG. 3, corresponding points common to the two actual images I ₀ and I ₁ are obtained in advance, and the two images are deformed based on the corresponding points. and, to synthesize a split image I _s've seen from between. This method can accurately draw an image from an arbitrary viewpoint, and can reduce the number of actual images required.

However, in this method, the corresponding points must be manually specified, and an effective algorithm for obtaining the corresponding points has not yet been established. In addition, this method is not suitable for a complicated subject for which feature points are difficult to obtain, and depending on the subject, the quality of the generated interpolation image may be reduced. Furthermore, since the calculation amount is enormous and a considerable amount of time is required for processing, it cannot be displayed in real time. Therefore, even with this method, it is difficult to generate a smooth moving image and it has not been put into practical use.
Japanese Patent Laid-Open No. 2001-34788 JP 2001-148021 A JP 2004-46776 A Steven M.M. Seitz, Charles R .; Dyer, “View Morphing”, SIGGRAPH '96, pp. 21-29

  The present invention has been made in view of the above points, and an object of the present invention is to solve the above-described problems of the prior art and provide a technique for smoothly displaying an image-based moving image with a practical number of shots.

  The above-described problem is a moving image generation system that generates a moving image using a plurality of images obtained by photographing a subject from various directions, and a virtual space information generation unit that generates three-dimensional virtual space information including a camera path. A shooting condition extracting unit that extracts shooting conditions from the three-dimensional virtual space information, a subject shooting unit that captures a subject based on the shooting conditions and obtains a plurality of two-dimensional images, and the plurality of two-dimensional images. A three-dimensional object generating means for generating a three-dimensional object, and a moving image generating means for generating a two-dimensional moving image from the three-dimensional object using the three-dimensional virtual space information generated by the virtual space information generating means This can be solved by a moving image generation system characterized by comprising:

  The virtual space information generation unit generates a temporary object generation unit that generates a temporary object that represents an approximate shape of the subject, and movement information of a position of a virtual camera that captures the temporary object as the camera path. And a preview for generating a two-dimensional moving image from the temporary object using the three-dimensional virtual space information before the three-dimensional object is generated. You may make it provide an image generation means.

  The imaging condition extraction unit may include a coordinate conversion unit that converts a relative position of the virtual camera with respect to the temporary object into a plurality of coordinate data based on the movement information.

  Further, the three-dimensional object can be configured by subject dimensions, a plurality of two-dimensional images of the subject, a plurality of mask images representing a subject portion on the two-dimensional image, and photographing conditions when the subject is photographed. .

  The present invention is also a moving image generation method executed by a moving image generation system that generates a moving image using a plurality of images obtained by photographing a subject from various directions, and generates three-dimensional virtual space information including a camera path. A virtual space information generation step, a shooting condition extraction step for extracting a shooting condition from the three-dimensional virtual space information, a subject shooting step of shooting a subject based on the shooting condition to obtain a plurality of two-dimensional images, A three-dimensional object generating step for generating a three-dimensional object using a plurality of two-dimensional images, and a two-dimensional moving image using the three-dimensional virtual space information generated by the virtual space information generating step from the three-dimensional object It is also possible to configure as a moving image generation method characterized by having a moving image generation step for generating.

  The present invention is also a moving image generation apparatus for generating a moving image using a three-dimensional object generated from a plurality of images obtained by photographing a subject from various directions, and includes a three-dimensional virtual space including a camera path. Virtual space information generating means for generating information, shooting condition extracting means for extracting shooting conditions from the three-dimensional virtual space information, and a plurality of two-dimensional images obtained by shooting a subject based on the shooting conditions A moving image generating apparatus comprising: a moving image generating unit that generates a two-dimensional moving image from a three-dimensional object using the three-dimensional virtual space information generated by the virtual space information generating unit. Is also possible. In the present invention, the computer can also be configured as a program that functions as the moving image generating apparatus or a computer-readable recording medium that records the program.

  According to the present invention, first, three-dimensional virtual space information including a camera path is generated, shooting conditions are extracted from the three-dimensional virtual space information, and a subject is shot based on the extracted shooting conditions. Since a two-dimensional moving image is generated from a three-dimensional object generated using a two-dimensional image using three-dimensional virtual space information, an image-based moving image can be smoothly displayed with a practical number of shots. It becomes possible.

Embodiments of the present invention will be described below with reference to the drawings. The case where the fixation point in the virtual space is fixed will be described as the first embodiment, and the case where the fixation point in the virtual space is variable will be described as the second embodiment.
(First embodiment)
[System configuration]
FIG. 4 shows the configuration of the moving image generation system according to the first embodiment. As shown in FIG. 4, the moving image generation system includes an editing device AD, a photographing device CD, and a display device DD.

  The editing device AD includes a virtual space construction unit VP, a shooting condition extraction unit EP, and a moving image generation unit MP. The photographing apparatus CD includes a subject photographing unit SP and a three-dimensional object composition unit IP. The display device DD has a moving image display unit DP. The editing device AD and the display device DD can each be realized using a computer system. Further, the subject photographing unit SP in the photographing apparatus CD can be realized by a computer system for performing photographing position control, a camera, a table on which a subject is placed, a background device, and the like, and the three-dimensional object composition unit IP is a three-dimensional object composition etc. It can be realized by a computer system equipped with a program for

  As shown in FIG. 4, each device can be configured as a bidirectional system that communicates with each other via a network. For example, the editing device AD and the display device DD may be constructed as one computer system, The three-dimensional object composition unit IP, the editing device AD, and the display device DD of the photographing device CD can be constructed on one computer system.

  FIG. 5 shows an example of a computer system. As shown in FIG. 5, this computer system is a general computer system having a CPU 1, a memory 2, a hard disk 3, an input device 4, an output device 5, and a CD-ROM drive 6. For example, the editing apparatus AD can be realized by mounting a program for executing the processing of the editing apparatus AD on the computer system. The program may be recorded and stored in a computer-readable recording medium such as an FD (flexible disk), MO, ROM, memory card, CD-ROM, DVD, removable disk, or distributed. Is possible. The program can also be provided through a network such as the Internet or e-mail.

[Outline of processing]
FIG. 6 shows a processing flow in the moving image generation system of the present embodiment. Hereinafter, an overview of the moving image generation processing in the present embodiment will be described with reference to the processing flow of FIG.

  First, the virtual space construction unit VP of the editing apparatus AD generates a temporary object in response to an operation from the user, and generates a camera path for the temporary object as space information (step 1). At this time, the user can confirm the video of the temporary object viewed along the camera path. That is, it is possible to preview the final video.

  Next, the imaging condition extraction unit EP extracts the imaging conditions when the subject is actually captured by performing coordinate conversion of the camera path in the space information of the virtual space (step 2). Then, the subject photographing unit SP of the photographing apparatus CD photographs the subject from a plurality of directions according to the extracted photographing conditions, and outputs a plurality of two-dimensional images (step 3).

  Subsequently, the three-dimensional object composition unit IP of the photographing apparatus CD composes a three-dimensional object from the plurality of two-dimensional images (step 4). The synthesized three-dimensional object is sent to the editing device AD. In the editing device AD, the moving image generation unit MP uses the received three-dimensional object and spatial information such as a camera path created by the virtual space construction unit VP. A two-dimensional moving image is generated (step 5). The two-dimensional moving image is sent to the display device DD, and the moving image display unit DP of the display device DD displays the two-dimensional moving image (step 6).

  As in the prior art, when a two-dimensional image with fine angular increments obtained by photographing the subject from all directions of 360 degrees is created, the amount of data becomes very large and the photographing time becomes very long. On the other hand, according to the processing in the present system as described above, a two-dimensional image in fine angular increments is taken from the camera position corresponding to each time position on the camera path defined in the virtual space instead of all directions of 360 degrees. By acquiring the image, a smooth moving image can be created, the amount of data can be reduced, and the shooting time can be shortened.

[Detailed processing]
Hereinafter, functions and processes of each processing unit of the moving image generation system according to the present embodiment illustrated in FIG. 4 will be described in detail. Each unit described below is a unit realized by executing a program on a computer system.
(1) Virtual space construction unit VP
FIG. 7 is a diagram illustrating a functional configuration of the virtual space construction unit VP. As shown in FIG. 7, the virtual space construction unit VP includes temporary object generation means 10, camera path generation means 11, space information editing means 12, and video confirmation means 13. FIG. 7 shows information generated by each means. Processing in the virtual space construction unit VP is as follows. The following processing is performed based on the user's operation.

  First, the temporary object generation unit 10 generates a temporary object in the three-dimensional virtual space. Further, the camera path generation unit 11 generates a camera path in a three-dimensional virtual space. Spatial information is constituted by the temporary object and the camera path. The generated spatial information is stored in a storage device.

  Further, the spatial information editing unit 12 edits the spatial information into a form that can be displayed as a video, and the video confirmation unit 13 displays a preview video based on the spatial information on a display or the like. The user can see the preview image and grasp the completed image before completion. When the confirmation is performed in the preview, if the movement of the object is different from what is assumed, it can be appropriately corrected by changing the camera path.

  The temporary object is rough rectangular parallelepiped data that represents the outline of the subject and also represents the position and occupied range of the subject. Since the temporary object does not represent the exact size of the subject, correction is required in the moving image generation process described later, but at this point, it is sufficient to grasp the rough completed video image before photographing the subject. And trying to generate a simple cuboid.

  As the temporary object, a rectangular parallelepiped representing the outline of the subject as described above is used, data of another object having the same size is used, or the subject is easily photographed in advance by a photographing device. Data of a three-dimensional object may be created and used as a temporary object.

  The camera path is a path indicating from what viewpoint position the 3D object in the 3D virtual space is photographed, and the trajectory / path of the virtual camera movement in the 3D virtual space is arbitrarily set. These are represented by line segments, curves, figures, etc., and arranged in the same three-dimensional virtual space. In the present embodiment, since there is only one temporary object, the gazing point of the camera is set to a fixed position (for example, the center position of the temporary object). FIG. 8 shows a setting example of the camera path on the operation interface screen.

  The technology for setting the camera path in the three-dimensional virtual space is a conventional standard technology for constructing a scene image in a polygon-based three-dimensional CG, and should be similarly realized in an image-based system. Can do. The shape, number, and combination of camera paths can be freely defined by using shape input means.

  In addition, since the camera path represents a change in the camera position as shape information in the three-dimensional virtual space, it includes time information in addition to the position information. In order to support editing of time information of a camera path in a three-dimensional virtual space, an interface such as a timeline is generally used. A timeline is a two-dimensional interface that manages objects having time attributes on a time axis for each layer, and is generally used in the past when constructing a scene image in a three-dimensional CG. By using the timeline, the camera path can be easily managed. In this embodiment, it is assumed that such a timeline interface is used.

  FIG. 9 shows an example of the timeline setting screen. As shown in FIG. 9, the camera path time length (camera path time) is set for each camera path. In addition, the number of frames per second (FPS) in the camera path can be set, and in this embodiment, 1 ≦ FPS ≦ 30. That is, time information of a maximum of 30 frames can be provided per second. In the example of FIG. 9, the camera pass time of the camera pass 1 is 10 seconds, and the FPS is set to 12. In this case, the total number of frames in camera path 1 is 120.

Spatial information is constituted by the position information of the temporary object and the position information and time information of the camera path. Here, the relationship between the temporary object and the camera path in the present embodiment will be described. Since this embodiment is a case where there is only one temporary object as the spatial information, the virtual camera always looks at the center of the temporary object as a direction of viewing the temporary object. In this case, the usage pattern is limited, but for example, when it is desired to show a piece of a museum or a highly individual product such as jewelry as an independent high-quality image, spatial information can be quickly constructed.
(2) Shooting condition extraction unit EP
Next, the imaging condition extraction unit EP that converts the space information generated by the virtual space construction unit VP into imaging conditions will be described. As shown in FIG. 10, the shooting condition extraction unit EP includes coordinate conversion means 20, shooting distance calculation means 21, vertical angle calculation means 22, horizontal angle calculation means 23, and shooting procedure generation means 24. The photographing condition extraction unit EP having these means converts a camera path arranged as a shape in a three-dimensional virtual space into a plurality of coordinate values, and obtains a photographing distance (D) and a photographing angle (V) in the photographing apparatus CD. H) • Extraction is performed using the photographing procedure (AL) as a condition. Hereinafter, the processing content in the imaging condition extraction unit EP will be described in detail.

  First, the coordinate conversion means 20 obtains the relative position at each time of the camera path with respect to the coordinates representing the center of the temporary object as a plurality of coordinate data. For example, when the center of the temporary object, that is, the origin of the object coordinate system is (2, 2, 2) in the virtual space coordinate system, the camera coordinates at two points with the camera path are (10, 2, 10), (12 , 2, 8), the output from the coordinate conversion means 20 is (8, 0, 8) (12, 0, 6).

  Since camera coordinates in the camera path exist for each frame time position, coordinate data for the total number of frames is output. In addition, the number of coordinate data to be obtained can be adjusted by designating the frame interval (FI). Here, the frame interval is an index indicating at what ratio the coordinate data is extracted with respect to the total number of frames of a certain camera path. In this case, since coordinate data is output at every frame interval, a value obtained by dividing the total number of frames in the camera path by the number of frame intervals is the number of coordinate data to be output. That is, if the number of coordinate data obtained by the coordinate conversion means 20 is (n) and the time length of the camera path is (T), n = (T × FPS) / FI. Where T is in seconds.

  For example, if the camera pass time is 10 seconds and the camera pass has an FPS of 12, the total number of frames is 120. If the frame interval is specified as 12 in this camera pass, the number of obtained coordinate data is 120/12 = 10.

  If it is desired to generate the video more smoothly, the number of coordinate transformations can be increased by reducing the frame interval value. Since the camera path is obtained by cutting out a part of the space in the three-dimensional virtual space, when the total number of shots in the conventional method when shooting the subject from all directions is (Pm), the total number of shots according to the present embodiment. Between (Pn), Pm ≧ Pn holds. Therefore, when a smooth moving image is generated, the number of necessary images can be reduced as compared with the conventional case.

  FIG. 11 shows an example of camera coordinates obtained by the coordinate conversion means. As shown in FIG. 11, unlike the conventional integer-based imaging conditions that have been manually set, the amount of change in coordinates for each viewpoint is quantified in real numbers. Therefore, changes in camera coordinates can be described more flexibly.

  When coordinate data is obtained by the coordinate conversion means 20, the shooting distance calculation means 21 calculates a shooting distance in mm based on the coordinate data. The shooting distance is the distance between the camera and the subject center.

  Since the camera path arranged in the virtual space has a free shape, various variable distances are taken with respect to the temporary object. In the three-dimensional virtual space, the distance variable can take any arbitrary value, but the actual photographing apparatus has physical restrictions such as the movable range of the arm that supports the camera. There are cases where the distance is limited (for example, shooting must be performed at a fixed distance). Therefore, distance correction is performed as necessary by inputting such constraints as data in advance.

  Note that three-dimensional distance measurement in actual shooting may be distance information measurement by stereo processing a plurality of images from the same direction, which is a commonly used method, or by an optical method such as a range finder. The position of the pattern light projected on the target object may be measured, and distance information may be obtained from the geometric positional relationship. These depend on the accuracy of the required shooting distance correction.

  Next, for each camera coordinate in the three-dimensional virtual space obtained by the coordinate conversion means 20, a vector calculation is performed based on the camera coordinates and the origin of the temporary object in the object coordinate system, and the vertical angle calculation means 22 Find the vertical angle. Further, the horizontal angle calculation means 23 obtains the horizontal angle.

  The vertical angle is an angle formed between a line connecting the camera and the subject center and the horizontal plane, and the horizontal angle is a line connecting the subject center and a predetermined position of the subject and a line connecting the subject center and the camera position on the horizontal plane. The angle between

  Note that numerical calculation of parameters such as the shooting distance, the vertical angle, and the horizontal angle can be easily realized with existing technology.

  After obtaining the photographing distance, the vertical angle, and the horizontal angle as described above, the photographing procedure generating unit 24 generates a photographing procedure. As shown in FIG. 12, the shooting procedure is generated as a set of shooting conditions including a shooting distance, a vertical angle, and a horizontal angle, and describing the shooting conditions in the order of shooting. The photographing apparatus CD performs photographing by reading these photographing conditions in order from the top. In general, it is difficult to obtain the shortest time for a series of shooting paths, and a method for obtaining an optimal solution has not yet been established. However, an approximate solution is generally obtained using a known algorithm, and a sequence of shooting conditions is arranged. Decide the order.

Conventionally, these parameters are set manually one by one before photographing. However, it was difficult to manually specify various conditions while considering the shooting procedure, and it was difficult to create various shooting patterns. On the other hand, by using the means in the present embodiment, it is possible to obtain all shooting conditions from the coordinate data based on the real number based on the setting in the virtual space, so that flexible conditions given to the shooting apparatus CD can be set. It can be determined automatically.
(3) Subject photographing unit SP
Subsequently, the subject photographing unit SP receives the above photographing procedure, and based on this, the subject is photographed. The subject photographing unit SP has a camera and includes means for performing photographing while changing the relative position between the camera and the subject in accordance with the received photographing procedure.

For example, as shown in FIG. 13, a configuration including an image shooting device CM, a table TB, a background display device BD, a switch S, and a control device C can be used as the subject shooting unit SP (in this configuration). (See Patent Document 3 for a method of photographing a subject). In the configuration of FIG. 13, the control device C performs control according to the photographing procedure. The configuration in FIG. 13 is an example in which the vertical angle is fixed, and shooting is performed while changing the horizontal angle by rotating the table according to the shooting procedure. Thereby, a plurality of two-dimensional images photographed from many directions can be generated.
(4) 3D object composition unit IP
The three-dimensional object composition unit IP can be realized by installing a three-dimensional object composition program in the control device C in the example of FIG.

The three-dimensional object includes a subject size, a plurality of two-dimensional images of the subject, a plurality of mask images representing a subject portion on the two-dimensional image, and shooting conditions (distance to subject, angle, resolution) 3D object composition unit IP generates these data. The subject size can also be obtained by using the Shape from Silhouette method, and the front image and the image from the side of the subject can be estimated. The mask image can be generated by using the chroma key method or the method of Patent Document 3.
(5) Moving image generation unit MP
The moving image generation unit MP receives the spatial information created by the virtual space construction unit VP and the 3D object created by the 3D object synthesis unit IP of the photographing apparatus CD. When a three-dimensional object is input, the temporary object is deleted and the camera path deviation is corrected. A preview is performed as necessary, and a moving image is generated by outputting spatial information as an image sequence.

  FIG. 14 shows a functional configuration of the moving image generation unit MP. As illustrated in FIG. 14, the moving image generation unit MP includes a comparison unit 30, a correction unit 31, and an image sequence processing unit 32. Hereinafter, processing of the moving image generation unit MP having such means will be described.

  First, the comparison means 30 performs a size comparison process between the temporary object created by the virtual space construction unit VP and the three-dimensional object. In this embodiment, since a rectangular parallelepiped having a rough size is used as the temporary object, the above comparison processing is performed in order to resize the three-dimensional object so that the rectangular parallelepiped circumscribes the three-dimensional object. Note that when using data obtained by roughly photographing a subject as a temporary object, the size is already the same, so there is no need to perform comparison processing. The temporary object is deleted when the role is finished.

  Next, if the size of the temporary object and the three-dimensional object are different as a result of the comparison process, resizing is performed, and in order to suppress a deviation between the preview before the creation of the three-dimensional object and the preview in this processing stage, Is corrected by the correction means 31. If necessary, the camera path itself may be edited again by a user operation.

Finally, the image sequence processing means 32 performs processing for converting spatial information including a three-dimensional object into an image sequence using the frame interval set in the timeline, and outputs a moving image. Note that the image sequence processing means 32 can output one piece of data collectively as a moving image, or can output one piece at a time for each image frame.
(6) Moving image display unit DP
The generated moving image is sent to the moving image display unit DP, and the user can view the moving image displayed on the moving image display unit DP. The moving image display unit DP is a display device such as a CRT monitor, a liquid crystal display, a plasma display, or a television.

  The moving image displayed on the moving image display unit DP is based on flexible spatial information generated by the processing of this system and a three-dimensional object based on shooting conditions, so it can be used as a high-quality video image with a high frame rate. it can. It can also be used as a lightweight multimedia format with a reduced rate for applications such as e-commerce.

(Second Embodiment)
In the first embodiment, the gazing point of the camera in the virtual space is fixed, but in the second embodiment, the gazing point of the camera is variable. When there are a plurality of objects by making the gazing point variable, the gazing point can be directed to each object.

  The configuration and overall processing flow of the moving image generation system in the second embodiment are the same as those in the first embodiment. The difference from the first embodiment is that in the virtual space construction unit VP, another parameter is increased in order for the camera to freely change the direction. That is, in the first embodiment, as shown in FIG. 15 (a), the gazing point of the camera is fixed, but in the second embodiment, the object is displayed as shown in FIG. 15 (b). When there are a plurality of objects, it is possible to set a gazing point for each object. It is also possible to look at an arbitrary point on a three-dimensional space without an object.

  The configuration of the virtual space construction unit VP in the second embodiment is shown in FIG. As illustrated in FIG. 16, a configuration in which a gazing point processing unit 40 is added to the configuration of the virtual space construction unit VP in the first embodiment is adopted.

  In the first embodiment, spatial information is created by generating a temporary object and generating a camera path. In the second embodiment, the gazing point information acquired by the gazing point processing means 40 is used. Spatial information is generated.

  For example, at the stage of editing the camera path, a first camera path is created using the first gazing point, and then the gazing point is changed from the first gazing point to the second gazing point by the gazing point processing means 40. To create a second camera path. As the spatial information, for example, spatial information for the first gazing point and spatial information for the second gazing point are created. Thereby, it is possible to configure spatial information corresponding to a plurality of objects. The video confirmation means 13 can provide a preview video before completion based on the spatial information in the same manner as in the first embodiment.

  Actual subject photographing can be performed in the same manner as in the first embodiment. That is, a shooting procedure is generated from spatial information including gazing point information, and a plurality of two-dimensional images are generated by controlling the position, angle, orientation, and the like of the camera according to the shooting procedure.

  Note that even in the case of the photographing apparatus configured for one subject as shown in FIG. 13, the subject is replaced for each gazing point, a two-dimensional image is acquired, a three-dimensional object is generated for each subject, By synthesizing a plurality of three-dimensional objects in the image generation unit MP, a moving image having a plurality of gazing points can be generated.

  In the second embodiment, since the gazing point is variable, the direction in which the camera views the temporary object is arbitrary. In other words, since the camera's point of interest can take an arbitrary position in the three-dimensional virtual space, it is possible to construct a virtual space freely even when there are a plurality of temporary objects, not only a single product, This makes it possible to express more realistic space, such as using a combination of products and looking inside the room.

(About the effect of this system)
According to the moving image generation system according to the present embodiment as described above, a three-dimensional virtual space is constructed by finally imagining a target video before photographing a subject, and based on the three-dimensional virtual space information. Since the shooting conditions are automatically extracted and the subject is shot using the shooting conditions, it is smooth just to shoot the necessary amount for display without shooting the subject from any direction. A moving image can be obtained. As a result, not only is it possible to manually specify the shooting conditions, but also a three-dimensional object can be obtained in a practical time. Accordingly, a smooth three-dimensional moving image based on the image base can be generated in a practical time.

  FIG. 17 shows a conventional processing procedure and a processing procedure in this system. As shown in FIG. 17, the processing of the virtual space construction unit VP in this system does not exist in the conventional processing procedure. Conventionally, the phase of editing a three-dimensional object in order to construct a virtual space is performed after shooting of a subject or image processing is completed, and on the three-dimensional virtual space layout system, the arrangement change, rotation, roll, I could only zoom.

  Therefore, with the conventional method, even if the number of shots was relatively small, there was no practical problem.However, in order to obtain a moving image that can be used for video applications, it was necessary to manually specify the conditions at the shooting stage, Not only is it very time-consuming to obtain, but it is also difficult to realize a high-quality moving image.

  On the other hand, in the processing method according to the present system, since the phase for constructing the virtual space is set before the subject is photographed, bidirectional feedback can be obtained between the editing device AD and the photographing device CD. In other words, this system has a distinctive feature that does not exist in the prior art in the present system that enables interactive input / output exchange between devices by first performing the virtual space construction phase. As a result, more flexible shooting conditions can be automatically extracted, and a smooth moving image can be generated in a practical time.

  The present invention is not limited to the above-described embodiment, and various modifications and applications can be made within the scope of the claims.

It is a figure for demonstrating the example of the method of automatically producing | generating geometry base data. It is a figure which shows the process sequence by the conventional image base method. It is a figure which shows the view morphing method which produces | generates a middle image. It is a block diagram of the moving image generation system in embodiment of this invention. It is a figure which shows an example of a computer system. It is a processing flow in the moving image production | generation system of embodiment of this invention. It is a figure which shows the function structure of the virtual space construction part VP. It is a figure which shows the example of a camera path setting on an operation interface screen. It is a figure which shows an example of a timeline setting screen. It is a figure which shows the function structure of the imaging condition extraction part EP. It is a figure which shows the example of the camera coordinate calculated | required by the coordinate conversion means. It is a figure which shows the example of the imaging | photography condition produced | generated from the camera coordinate of FIG. It is a figure which shows an example of subject imaging | photography part SP. It is a figure which shows the function structure of the moving image generation part MP. It is a figure for demonstrating a gaze point. It is a figure which shows the function structure of the virtual space construction part VP in the 2nd Embodiment of this invention. It is a figure which shows the conventional process sequence and the process sequence in the system which concerns on this invention.

Explanation of symbols

AD editing device VP Virtual space construction unit EP Shooting condition extraction unit MP Moving image generation unit CD Shooting device SP Subject shooting unit IP 3D object synthesis unit DD Display device DP Moving image display unit CM Image shooting device C Control device TB Table O Subject BD Background device S Switch

Claims (11)

A moving image generation system that generates a moving image using a plurality of images obtained by photographing a subject from various directions,
Virtual space information generating means for generating three-dimensional virtual space information including a camera path;
Shooting condition extraction means for extracting shooting conditions from the three-dimensional virtual space information;
Subject photographing means for photographing a subject based on the photographing conditions and obtaining a plurality of two-dimensional images;
Three-dimensional object generation means for generating a three-dimensional object using the plurality of two-dimensional images;
A moving image generation system comprising: a moving image generation unit configured to generate a two-dimensional moving image from the three-dimensional object using the three-dimensional virtual space information generated by the virtual space information generation unit. The virtual space information generating means
Temporary object generating means for generating a temporary object representing the approximate shape of the subject;
The moving image generation system according to claim 1, further comprising: camera path generation means for generating movement information of a position of a virtual camera that captures the temporary object as the camera path. The virtual space information generating means
The preview image generating means for generating a two-dimensional moving image from the temporary object using the three-dimensional virtual space information before the three-dimensional object is generated. Video generation system. The photographing condition extracting means includes
The moving image generating system according to claim 2, further comprising coordinate conversion means for converting a relative position of the virtual camera with respect to the temporary object into a plurality of coordinate data based on the movement information.   The three-dimensional object has a subject size, a plurality of two-dimensional images of the subject, a plurality of mask images representing a subject portion on the two-dimensional image, and a photographing condition when photographing the subject. Item 4. The moving image generation system according to Item 1. A moving image generation method executed by a moving image generation system that generates a moving image using a plurality of images obtained by photographing a subject from various directions,
A virtual space information generation step for generating three-dimensional virtual space information including a camera path;
A shooting condition extraction step for extracting shooting conditions from the three-dimensional virtual space information;
Subject photographing step of photographing a subject based on the photographing conditions to obtain a plurality of two-dimensional images;
A three-dimensional object generation step of generating a three-dimensional object using the plurality of two-dimensional images;
A moving image generating method comprising: generating a two-dimensional moving image from the three-dimensional object using the three-dimensional virtual space information generated by the virtual space information generating step. A moving image generating device for generating a moving image using a three-dimensional object generated from a plurality of images obtained by photographing a subject from various directions,
Virtual space information generating means for generating three-dimensional virtual space information including a camera path;
Shooting condition extraction means for extracting shooting conditions from the three-dimensional virtual space information;
A two-dimensional moving image is generated from a three-dimensional object generated from a plurality of two-dimensional images obtained by shooting a subject based on the shooting conditions, using the three-dimensional virtual space information generated by the virtual space information generating means. And a moving image generating means. The virtual space information generating means
Temporary object generating means for generating a temporary object representing the approximate shape of the subject;
The moving image generation apparatus according to claim 7, further comprising: camera path generation means for generating movement information of a position of a virtual camera that captures the temporary object as the camera path. The photographing condition extracting means includes
9. The moving image generating apparatus according to claim 8, further comprising coordinate conversion means for converting a relative position of the virtual camera with respect to the temporary object into a plurality of coordinate data based on the movement information.   The computer program for making a computer implement | achieve each means in the moving image generation apparatus of any one of Claims 7 thru | or 9.   The computer-readable recording medium which recorded the computer program of Claim 10.

Images