JP4395082B2 - Video generation apparatus and program - Google Patents

Description

  The present invention belongs to the technical field of free viewpoint video.

Free viewpoint video technology is a technology that integrates video obtained from a plurality of cameras, performs three-dimensional analysis, and obtains video from a free viewpoint based on the analysis result.
If a shooting system that can obtain free-viewpoint images is installed in a meeting room in the company, the behavior of attendees at the company meeting can be shot from a free viewpoint. Moreover, if it is installed at a medical site, it is possible to take pictures of medical practices of doctors and nurses from a free viewpoint.

The free viewpoint video obtained in this way can be used later as a video of the minutes and workplace improvement materials. Since it can be applied to the minutes and materials for improvement, free viewpoint video technology is expected to develop in the future. The applicant recognizes that there is a prior art described in the following non-patent literature regarding the technique of free viewpoint video.
Takeo Kanade, Peter Rander, and PJ Narayanan.Virtualized reality: Constructing virtual worlds from real scenes.IEEE MultiMedia, Vol. 4, No.1, pp. 34-47, 1997. Ryofumi Inoue, Ryuji Yoshida, Atsuko Hiraishi, Hiroshi Shigeno, Kenichi Okada, Atsushi Matsushita, Automatic Shooting Method for Face-to-face Meeting Scenes Based on Movie Image Theory, IPSJ Transactions, Vol.45, No.1, pp.212 -221, January 2004.

  By the way, with respect to video generation from a free viewpoint, although some achievements are recognized in the above-mentioned non-patent literature, there is still room for research on how to change the free viewpoint in time. . This is because a change in viewpoint position may give viewers a great sense of incongruity depending on how the change is made. On the other hand, in the field of video production such as movies, various shooting techniques have been produced to prevent such a sense of incongruity.

  For this reason, even today, when research on free viewpoints is progressing, in order to pursue ease of viewing like a movie, it is still necessary to borrow the power of shooting techniques by professional shooting staff. However, the cooperation of professional photography staff requires a large amount of investment, so it is very difficult to produce a documentary that will leave the impression of the user like a movie, even though free-viewpoint technology has developed. It is said that it is difficult.

  An object of the present invention is to provide an image generation device and a program that can obtain an easy-to-view image such as a movie without depending on the skill of a professional shooting staff.

  The above object is a video generation device that obtains a moving image by changing the viewpoint position based on the scene description information, and the scene description information includes camera control information corresponding to each of a plurality of candidate positions, Evaluation means for evaluating how much each candidate position corresponds to the current viewpoint position by applying a predetermined rule to the set of the previous viewpoint position and one candidate position, and the evaluation result Selecting the highest candidate position as the current viewpoint position, and using the camera control information corresponding to the current viewpoint position, generating means for generating a moving image when the subject is viewed from the current viewpoint position, evaluation means, And a control unit that causes the generation unit to repeatedly execute the process.

In addition, the predetermined rule includes a constraint on a boundary that divides the subject space, and based on the constraint, when a straight line connecting a previous viewpoint position and a candidate position intersects the boundary, The evaluation for the candidate position may be made low, and if the straight line connecting the immediately preceding viewpoint position and the candidate position does not intersect the boundary, the evaluation for the candidate position may be made high.
Furthermore, the subject space may include two or more persons, and the boundary may include a straight line connecting two persons included in the subject space, or an arbitrary straight line, and the subject space may be divided into two.

  Also, some camera control information is intended for camera pan control. According to the predetermined rule, camera control information associated with any candidate position is displayed from the previous viewpoint position to the candidate position. When the pan control is indicated, the evaluation value of the candidate position is increased, and the camera control information associated with any candidate position does not indicate the pan control from the previous viewpoint position to the candidate position. In some cases, the evaluation value of the candidate position may be lowered.

Further, the predetermined rule includes a constraint on a boundary that divides the subject space, and based on the constraint, when a straight line connecting a previous viewpoint position and a candidate position intersects the boundary, The evaluation for the candidate position may be made low, and if the straight line connecting the immediately preceding viewpoint position and the candidate position does not intersect the boundary, the evaluation for the candidate position may be made high.
Furthermore, when the camera control information associated with any candidate position indicates pan control from the previous viewpoint position to the candidate position, the evaluation value for the candidate position is the previous viewpoint position, the candidate The straight line connecting the positions may be higher than the evaluation for the candidate position when the line does not intersect the boundary.

The camera control information, the procedure view angle θ that determines the candidate position is in which direction the subject, and the subject comprises a framing condition information indicating a ratio should occupy on the screen, to obtain the moving image, the framing A procedure for determining the distance r between the subject and the viewpoint position based on the condition information and obtaining the three-dimensional coordinates of the candidate position from the viewpoint angle θ and the distance r may be included.
The above object is a program for causing a computer to generate a moving image whose viewpoint position is changed based on scene description information. The scene description information includes camera control information corresponding to each of a plurality of candidate positions. And by applying a predetermined rule to the set of the previous viewpoint position and one candidate position, and evaluating how each candidate position corresponds to the current viewpoint position The candidate position with the highest result is selected as the current viewpoint position, and using the camera control information corresponding to the current viewpoint position, the step of obtaining a moving image when the subject is viewed from the current viewpoint position is repeatedly executed on the computer. To achieve this.

  In the above-described configuration, if one of the predetermined criteria is a criterion such as “avoids viewpoint movement that crosses the imaginary line”, that is, a criterion according to the manner of shooting a movie, this criterion is determined as the current viewpoint. By applying the position to a set of one candidate position and repeating the selection of the viewpoint position based on this criterion, it is possible to perform viewpoint movement as practiced by professional photographers at movie shooting sites. If a moving image can be obtained by such viewpoint movement, a free viewpoint video that does not give the user a sense of incongruity can be obtained without the help of professional shooting staff.

  If know-how of various shooting methods is compiled into a database and applied as a predetermined standard, a free viewpoint video that is inferior to movies can be obtained. Like movies, free viewpoint videos that remain in the impression of the user can be created with a small investment, so that the production of video documentaries can be made familiar.

An embodiment of a 3D free viewpoint video system according to the present invention will be described with reference to the drawings.
The 3D free-viewpoint video system integrates video shot with six fixed cameras installed in the studio, and allows the space (hereinafter referred to as “shooting space”) that is the subject of the fixed camera from a free viewpoint. Generates the viewed video (hereinafter referred to as “three-dimensional free viewpoint video”).

FIG. 1 is a diagram showing the overall configuration of a 3D free viewpoint video system according to the present invention. As shown in FIG. 1, the 3D free viewpoint video system 1 includes a video generation unit 2, an annotation management unit 3, and a video script conversion unit 4.
The video generation unit 2 sends an unedited three-dimensional free viewpoint video generated from an image captured by a fixed camera to the annotation management unit 3 and the video script conversion unit 4. In addition, a camera work control command in which the viewpoint position coordinates at each time are described in XML (eXtensible Mark-up Language) is received from the video script conversion unit 4, and a three-dimensional free viewpoint video (from a designated viewpoint position at each time) ( Hereinafter, it is referred to as “rendered video”) and output. Here, the unedited three-dimensional free viewpoint image is a side image, a front image, and a front image in which the photographing space is viewed in parallel directions with respect to the three axes XYZ set to be orthogonal to each other as shown in FIG. This is video data consisting of three top videos. The X axis and the Y axis are coordinate axes that form a horizontal plane of the imaging space, and the Z axis is a coordinate axis that is orthogonal to the horizontal plane of the imaging space. In addition, the time in the present embodiment means the time in the video captured by the cameras 11a to 11f.

  The annotation management unit 3 creates and manages annotation information related to an unedited 3D free viewpoint video. Also, the annotation information is sent to the video script conversion unit 4. Annotation information is an annotation that indicates the contents of an unedited 3D free viewpoint video by identifying an event that occurred in the shooting space (hereinafter referred to as an “event”) and a person or object related to the event. It is.

  The video script conversion unit 4 receives a designation of a scene for which a rendering video is to be generated from a system user, generates a camera work control command indicating a viewpoint position for rendering the scene, and sends it to the video generation unit 2. Here, the scene is a series of images obtained by shooting the same shooting target with one or more types of shots within a predetermined continuous time. For example, the person A to be shot enters the shooting space and stops at the center of the shooting space. An entrance scene in which a series of operations up to the above are photographed, a conversation scene in which the subject person B talks to the person C, and the like are assumed. In addition, a shot means a series of moving images that are continuously shot from when the camera starts turning until it stops when shooting a movie, and in this embodiment, from a stationary viewpoint position or a continuously moving viewpoint position. It is expressed as a 3D free viewpoint video. For example, the entrance scene can be composed of a shot in which the foot of the person A entering the shooting space is captured from the front, a shot in which the whole body image of the person A who has started walking is captured from the side, and the like.

Hereinafter, each configuration will be described in detail.
The video generation unit 2 includes cameras 11 a to 11 f, a capture unit 12, a spatial model construction server 13, an XML interpretation unit 23, and a video output unit 2425.
The cameras 11a to 11f are six fixed cameras installed in the studio, and shoot images of the shooting space from different positions.

The capture unit 12 converts the video from the camera into a digital format and stores it as an image file.
The spatial model construction server 13 accumulates the captured video files of the cameras 11a to 11f, and generates a 3D free viewpoint video at an arbitrary time point based on the video files.
The XML interpretation unit 23 interprets the camera work control command.

The video output unit 24 generates and outputs a rendered video in accordance with the camera work control command.
The annotation management unit 3 includes an annotation information registration unit 15 and an annotation database 16.
The annotation information registration unit 15 receives input of annotation information from the system user and registers it in the annotation database 16. FIG. 3 shows an example of annotation information. The annotation information includes an event name, a start time, an end time, an object name, an annotation area, and a gaze point.

The event name is an arbitrary name input by the system user.
The start time is a time indicating the start of the event, and indicates the start time of a period in which the annotation area does not change.
The end time is the time indicating the end of the event, and indicates the end time of the period in which the annotation area does not change.

The object name indicates the name of an object such as a person or an object related to the event.
The annotation area is an area where an object named by an object name exists, and is indicated by a rectangular parallelepiped in which two-dimensional three-dimensional coordinates designated by the system user are diagonal lines.
The gaze point is a coordinate of a point at which a line of sight is poured when rendering the shooting space from the viewpoint position.

  Next, an example of an annotation information input interface is shown in FIG. Thereby, it is possible to specify the annotation area in time units by GUI operation. One event is composed of a large number of frames, and adding an annotation to each frame is a very time-consuming work. For this reason, the annotation region and the gazing point are automatically complemented between consecutive frames in one event. At this time, the center coordinate of the annotation area is used as the gaze point. Annotations created with this interface are accumulated in the annotation database 16.

  The input interface includes a top display screen, a front display screen, a side (right) display screen, and an input panel. On each display screen, the top video, front video, and side video at the time specified in the time specification bar on the input panel are displayed. In these images, the system user uses a pointing device such as a mouse to make a box type Specify the head or whole body with the marker. Thereby, the coordinates of two points indicating the annotation area are determined. Note that the box-type marker may be changed by changing the coordinates of two points indicating the annotation area. The gazing point can be determined by the system user by clicking an arbitrary point using a mouse or the like. However, in this embodiment, for ease of explanation, the center coordinates of the annotation area are used as the gazing point. Use.

The event name and object name are entered on the input panel by the system user using a keyboard or the like. The start time and end time are determined by the system user by designating each time using the time designation bar and pressing the Add button.
Here, the time designation bar is a slide bar indicating the time. That is, the system user confirms the period in which the coordinates of the two points indicating the annotation area do not change by operating the time designation bar, and presses the Add button at the start time and end time to start and end time. To decide.

The annotation database 16 accumulates annotation information registered by the annotation information registration unit 15.
The video script conversion unit 4 includes a scene list acquisition unit 17, a scene control unit 18, a shot control unit 19, an actual parameter acquisition unit 20, a shooting knowledge database 21, and a camera work control XML creation unit 22.

  The scene list acquisition unit 17 acquires, from the system user, a scene list in which information for specifying a scene for which a rendered video is to be generated is linked. An example of the scene list is shown in FIG. The scene list includes a scene name, a scene start time, a scene end time, a shooting target, and an imaginary vector end point coordinate. An imaging target may be replaced with an arbitrarily given area and a gazing point when there are a large number of objects or when no one exists.

The scene name is a name that identifies the scene.
The scene start time and the scene end time indicate the section to be cut out as a scene in the video shot by the cameras 11a to 11f.
As the shooting target, the shooting target to be watched in the rendered video is designated by the object name of the annotation information.

The imaginary vector end point coordinates are the three-dimensional coordinates of the imaginary vector end points. The imaginary vector is a vector whose starting point is the center coordinate P _A of the person A who is the subject of photographing and whose end point is the center coordinate P _B of the person B who is the conversation partner. A straight line drawn so as to divide the imaging space when the XY plane (hereinafter referred to as “the ground”) is looked down vertically from a viewpoint located at infinity above is called an imaginary line. In this embodiment, a straight line including an imaginary vector is defined as an imaginary line.

  When the scene control unit 18 receives the scene list from the scene list acquisition unit 17, the scene control unit 18 obtains information on the scene name, scene start time, scene end time, shooting target, and imaginary vector end point coordinates included in the scene list for each scene. To the shot control unit 19. Further, the camera position list generated in the shot control unit 19 according to each information of the scene list is sent to the camera work control XML creating unit 22 for each scene.

The shot control unit 19 identifies a scene based on information obtained from the scene control unit 18, and obtains a camera position list indicating the viewpoint position for rendering the scene at each time from the three-dimensional coordinates of the shooting target and shot information described later. It is generated and sent to the scene control unit 18.
The actual parameter acquisition unit 20 receives the designation of the time and the photographing target from the shot control unit 19, and from the annotation information registered in the annotation database 16, the annotation region including the photographing target at the designated time, and the gazing point Are obtained and sent to the shot control unit 19.

The shooting knowledge database 21 is a database that stores scene description information, shot information, and shooting rules. The scene description information, the shot information, and the shooting rule are information used when the shot control unit 19 generates a camera position list.
FIG. 6A shows the data structure of scene description information. The scene description information is information indicating shot pattern candidates to be used for each scene specified by the scene name, and includes a scene name and shot designations 1 to n. Shot designations 1 to n are information for designating shot information defining candidate shot patterns, and designate shot information using shot names.

Here, a shot pattern is an image of a screen layout that shows how the direction, size, arrangement, etc. of an object to be photographed looks in one shot video. Described.
FIG. 6B shows a data structure of shot information. The shot information is camera control information in which a viewpoint position for clipping a video with a desired shot pattern is uniquely determined according to the three-dimensional coordinates of a subject to be shot, and includes shot name, horizontal angle Dh, vertical angle Dv, framing pattern , Pan angle, and duration. As shown in FIG. 7A, the horizontal angle Dh included in the shot information is an angle formed with the imaginary vector on the horizontal plane including the center coordinate PA of the object to be imaged, and the vertical angle Dv is the vertical angle Dv. As shown in (b), it is an angle formed with a horizontal line including the center coordinate PA, and the horizontal angle Dh and the vertical angle Dv specify in which direction the viewpoint to be rendered is with respect to the center coordinate PA. As shown in FIG. 8, the framing pattern is an image of an image represented by the ratio of the shooting target in the vertical direction of the screen in the rendered video and the position on the screen of the center coordinates PA, “Long”, This is information specified by a pattern name such as “Full”. Hereinafter, the ratio of the shooting target in the vertical direction of the screen is referred to as “framing condition R1”, and the position of the center coordinate PA on the screen is referred to as “framing condition R2”. The pan angle is an angle by which the viewpoint position is rotated around the center coordinate PA in the shot. If the pan angle is a positive value, it indicates that the viewpoint position is rotated clockwise around the center coordinate PA as viewed from above the object to be imaged. If the pan angle is a negative value, the center coordinate PA is the center. Indicates that the viewpoint position is rotated counterclockwise. The duration is a time for which one shot to which the camera effect indicated by the shot information is applied is continued.

The shooting rule is a reference for evaluating viewpoint position candidates (hereinafter referred to as “candidate positions”) determined based on shot information, and restrictions systematized as a movie shooting technique are applied.
For example, it is known as a movie shooting technique that if the viewpoint position is switched across the imaginary line, the positional relationship of the person on the screen is reversed, giving the viewer a sense of incongruity. Applying such a movie shooting technique, `` If the line segment that projects the line connecting the previous viewpoint position and the candidate position vertically to the ground does not intersect the straight line that projects the imaginary line vertically to the ground, If the candidate position is highly evaluated and the line segment that projects the line connecting the previous viewpoint position and the candidate position perpendicularly to the ground intersects with the straight line that projects the imaginary line on the ground, the candidate position is evaluated low. The shooting rule “Yes” is stored in the shooting knowledge database 21. As another example of the shooting rule, “If the shot information corresponding to the candidate position is intended to perform pan control of the viewpoint position, the evaluation of the candidate position is increased, and the shot information corresponding to the candidate position is If the panning control of the viewpoint position is not intended, the evaluation of the candidate position is lowered ”. Still another example of the shooting rule is “if the previous viewpoint position and the candidate position are the same position, the evaluation is lowered”.

  The candidate position may be evaluated based on a plurality of shooting rules. For example, when the shot information corresponding to the candidate position is intended for pan control of the viewpoint position, the evaluation of the candidate position is performed by projecting a line segment connecting the immediately preceding viewpoint position and the candidate position vertically onto the ground. The line segment is evaluated to be higher than the evaluation for the candidate position in the case where the line segment does not intersect the straight line obtained by projecting the imaginary line perpendicularly to the ground.

The camera work control XML creation unit 22 acquires a camera position list from the scene control unit 18 and creates a camera work control command in which the viewpoint position at each time indicated in the camera position list is described in XML.
The above is the configuration of the three-dimensional free viewpoint video system 1.
Next, the operation for determining the viewpoint position of the rendered video at each time in the video script conversion unit 4 will be described in detail. FIG. 9 is a flowchart showing a flow of operations for generating a camera position list.

  When the scene list is acquired by the scene list acquisition unit 17, the scene control unit 18 generates a camera position list having the data structure shown in FIG. 10 and a shot list having the data structure shown in FIG. S11). Camera parameters such as zoom may be generated. Next, the scene control unit 18 selects scenes in order from the earliest start time from the scene list, and each parameter of the scene name, scene start time, scene end time, shooting target, and imaginary vector end point coordinates of the selected scene. Is notified to the shot control unit 19 (S12).

  After initializing the shot list, the shot control unit 19 reads out the scene description information corresponding to the scene name notified from the scene control unit 18 from the shooting knowledge database 21, and the shot included in the shot designation of the read scene description information Names are registered in the shot list in the order included in the scene description information (S13). The initialization of the shot list is to delete all the data registered in the shot list. When the processing of S13 is completed, only the shot name specified in the scene description information is included in the shot list. It is registered, and no data is registered for candidate coordinates and evaluation points.

  Next, the shot control unit 19 selects one shot pattern in the order in which the shot names are registered in the shot list, and reads shot information describing the selected shot pattern from the shooting knowledge database 21 (S14). The shot control unit 19 calculates the coordinates of the viewpoint position for shooting the shooting target with the shot pattern selected in S14 by the camera position determination process, and registers the calculated coordinates in the shot list as candidate positions (S15). The shot control unit 19 further scores the candidate positions determined in S15 and registers the score as an evaluation score in the shot list (S16).

  In scoring the candidate positions, the end viewpoint of the camera position registered last in the camera position list is set as the immediately preceding viewpoint position, and the relationship between the immediately preceding viewpoint position and the candidate position is registered in the imaging rule database 21. The candidate position is evaluated by increasing the score if it is in the preferred positional relationship and decreasing the score if it is in the undesirable positional relationship. However, when no camera position is registered in the camera position list, the same coordinates as the candidate position are used as the immediately preceding viewpoint position.

  The shot control unit 19 repeats the processes of S14 to S16 until candidate coordinates and evaluation points are registered in the shot list for all shot patterns whose shot names are registered in the shot list. When the processing is completed for all the shot patterns (S17: No), the shot control unit 19 uses the candidate position of the shot with the highest evaluation score registered in the shot list to newly add to the camera position list shown in FIG. A camera position is added (S18). However, when a plurality of shot patterns have the same score, the candidate position of the highest evaluation score registered in the shot list is used to add the camera position.

  The addition of the camera position is performed according to the following procedure. First, the imaginary vector end point coordinate notified from the scene control unit 18 is set for the gaze point, and the coordinate of the candidate position with the highest evaluation score is set for the start point of view. Next, when the pan angle is 0 degrees in the shot information corresponding to the shot with the highest evaluation score, the same coordinates as the start viewpoint are set as the end viewpoint, and the pan angle of the shot information is other than 0 degrees. In this case, a coordinate obtained by panning the candidate coordinate having the highest evaluation score by a specified angle is set as the end viewpoint. Furthermore, the same time as the end time of the previously registered camera position is set as the start time, and the duration of the shot information corresponding to the shot with the highest evaluation score is set as the start time. Set the time added to. However, when the first camera position is registered in the camera position list, the scene start time of the scene selected in S12 is set as the camera position start time.

  If the time set here as the end time has not reached the scene end time of the scene selected in S12 (S19: No), the shot control unit 19 reinitializes the shot list in S13, and until S18. Repeat the process. When the time set as the end time of the camera position list reaches the scene end time of the scene selected in S12 (S19: Yes), the scene control unit 18 converts the camera position list into the camera work control XML. Output to the creation unit 22. The camera work control XML creation unit 22 that has acquired the camera position list converts the viewpoint position coordinates for each time into camera work control commands described in XML, and outputs them to the video generation unit 2 (S20). After the conversion to the camera work control command by the camera work control XML creation unit 22 is completed, the scene control unit 18 deletes all the camera position data and initializes the camera position list.

The video script conversion unit 4 repeats the processes of S12 to S20 for all scenes registered in the scene list acquired by the scene list acquisition unit 17.
With the above operation, it is possible to determine the viewpoint position at each time in the period in which the user instructs the generation of the rendered video in the scene list.
Next, the camera position determination process executed in S15 of FIG. 9 will be described with reference to FIGS. Figure 12 is a flowchart showing the flow of processing of the camera position determination process, 13, and 14 are diagrams schematically showing the relationship between the fixation point P _A and the viewpoint position Cpt.

  In order to determine the viewpoint position at which the shooting target is shot with the shot pattern selected in S14, first, the shot control unit 19 is registered in the shooting position notified from the scene control unit 18 and the camera position list. The actual parameter acquisition unit 20 is notified of the end time of the last camera position, and the annotation region and the coordinates of the gazing point are acquired from the actual parameter acquisition unit 20 based on these pieces of information (S31). However, if no camera position is registered in the camera position list at this time, the scene start time notified from the scene control unit 18 is notified instead of the end time of the last camera position.

The shot control unit 19 that has acquired the annotation area and the coordinates of the gazing point calculates the coordinates of the temporary position Cpt _tmp shown in FIG. 13 (S32). The temporary position Cpt _tmp has the horizontal angle D _h and the vertical angle D _v with respect to the imaginary vector P _A P _B on the unit sphere centered on the gazing point P _A and having a radius r _0. It is a position to make.
Shot Control unit 19 is clipped at a predetermined viewing angle toward the fixation point P _A from the temporary position Cpt _tmp in the screen, the annotation area is calculated ratio R0 occupying the longitudinal direction of the screen (S33). R0 is obtained by dividing the height h _A of the annotation area from the temporary position Cpt _tmp by a predetermined angle of view but dividing it by the height h ₀ of the screen as shown in FIG.

Shot Control unit 19 further includes a R0 calculated in S33, and calculates the ratio k between the framing condition R1 framing pattern set in the shot information (S34), the fixation point P distance from _A is k · r ₀ It is a point, and the viewpoint for a point on an extension line of a line segment is extended to the temporary position Cpt _tmp side for photographing the photographic subject in the shot which is selected in S14 connecting the fixation point P _a and the tentative position Cpt _tmp The coordinates are calculated as the position Cpt (S35). The above is the details of the camera position determination process.

As described above, the three-dimensional free viewpoint video system 1 according to the present embodiment is
Just by specifying the scene description information you want to use, the start time and end time of the scene you want to generate the rendered video, the subject, and the coordinates of the subject's action target, you can easily view the rendered video like a movie Can be generated.
<Other variations>
As mentioned above, although this invention has been demonstrated based on said embodiment, of course, this invention is not limited to said embodiment. The following cases are also included in the present invention.

(1) The present invention may be a method showing a procedure in the above flowchart. Further, the present invention may be a program that realizes these methods by a computer, or may be a digital signal composed of the program.
Further, the present invention may be a program in which the program or the digital signal is recorded on a computer-readable recording medium, such as a magnetic disk, a magneto-optical disk, an optical disk, or a semiconductor memory.

In the present invention, the program or the digital signal may be transmitted via an electric communication line, a wireless or wired communication line, a network represented by the Internet, or the like.
In addition, the program or the digital signal is recorded on the recording medium and transferred, or the program or the digital signal is transferred via the network or the like, and is executed by another independent computer system. It is good.

(2) In the above embodiment, a configuration has been described in which a system user receives designation of an imaginary vector and uses a straight line including the imaginary vector as an imaginary line. However, the video generation apparatus according to the present invention is separate from the imaginary vector. It may be configured to receive designation of an imaginary line from the outside.
For example, an imaginary line specification may be received together with a scene list from a system user. Specifically, when the system user inputs the coordinates of two arbitrary points in the imaging space, a straight line passing through the two points is obtained as an imaginary line.

  In such a modification, separately from the imaginary vector, an arbitrary straight line drawn so as to divide the photographing space as viewed from the upper infinity can be designated as the imaginary line. Therefore, even in a scene where there are three or more persons in the shooting space, such as when the person A to be photographed is talking to the person C while facing the person B, a three-dimensional free viewpoint image without a sense of incongruity is obtained. Can do.

(3) In the above embodiment, the configuration for determining the viewpoint position using the scene description information stored in the shooting knowledge database has been described. However, the scene description information does not necessarily have to be stored in advance in the apparatus. Absent.
For example, the system user may input scene description information and shot information together with the scene list.

  (4) The above embodiment and the above modifications may be combined.

  The present invention captures the behavior of attendees at internal meetings, photographs the medical practices of doctors and nurses at medical sites, and photographs the behavior of users at amusement facilities, theme parks, nursing homes and nursing homes, etc. It can be applied to uses such as a video generation device that generates documentary used for minutes, improvement materials, and video service providing media from the recorded video.

The figure which shows the whole structure of the three-dimensional free viewpoint image | video system which concerns on this invention. The figure which shows an unedited three-dimensional free viewpoint image | video typically. The figure which shows an example of annotation information. The figure which shows an example of the input interface of annotation information. The figure which shows an example of a scene list. (A) is a figure which shows the data structure of scene description information. FIG. 6B is a diagram illustrating a data structure of shot information. (A) is a schematic diagram showing the relationship between the imaginary vector and the horizontal angle D _h. (B) is a schematic diagram showing the relationship between the vertical angle D _v and imaginary vector. The figure which shows the example of a framing pattern. The flowchart which shows the flow of the operation | movement which produces | generates a camera position list. The figure which shows an example of a camera position list. The figure which shows an example of a shot list. The flowchart which shows the flow of a process of a camera position determination process. Perspective view showing the relationship between the fixation point P _A and the viewpoint position Cpt. Schematic view showing the relationship between the fixation point P _A and the viewpoint position Cpt.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 3D free viewpoint video system 2 Video generation part 3 Annotation management part 4 Video script conversion part 11a-11f Camera 12 Capture unit 13 Spatial model construction server 15 Annotation information registration part 16 Annotation database 17 Scene list acquisition part 18 Scene control part 19 shot control unit 20 actual parameter acquisition unit 21 imaging knowledge database 22 camera work control XML creation unit 23 XML interpretation unit 24 video output unit

Claims (8)

A video generation device that obtains a moving image by changing a viewpoint position based on scene description information,
The scene description information consists of camera control information corresponding to each of a plurality of candidate positions.
An evaluation unit that evaluates how much each candidate position corresponds to the current viewpoint position by applying a predetermined rule to the set of the immediately preceding viewpoint position and one candidate position;
A generation unit that selects a candidate position with the highest evaluation result as a current viewpoint position and generates a moving image when the subject is viewed from the current viewpoint position using camera control information corresponding to the current viewpoint position;
An image generation apparatus comprising: an evaluation unit; and a control unit that causes the generation unit to repeatedly execute a process. The predetermined rule includes a restriction regarding a boundary that divides the subject space,
Based on these constraints,
When a straight line connecting the previous viewpoint position and the candidate position intersects the boundary, the evaluation for the candidate position is lowered,
The video generation device according to claim 1, wherein when a straight line connecting the immediately preceding viewpoint position and the candidate position does not intersect the boundary, the evaluation for the candidate position is increased. The subject space includes two or more persons,
The video generation apparatus according to claim 2, wherein the boundary includes a straight line connecting two persons included in the subject space or an arbitrary straight line, and divides the subject space into two. Some camera control information is intended for camera pan control,
The predetermined rule includes:
When the camera control information associated with any candidate position indicates pan control from the previous viewpoint position to the candidate position, the evaluation value of the candidate position is increased,
The camera control information associated with any candidate position does not indicate pan control from the previous viewpoint position to the candidate position, and the evaluation value of the candidate position is lowered. The video generation device described The predetermined rule includes a restriction regarding a boundary that divides the subject space,
Based on these constraints,
When a straight line connecting the previous viewpoint position and the candidate position intersects the boundary, the evaluation for the candidate position is lowered,
The video generation device according to claim 4, wherein when a straight line connecting a previous viewpoint position and a candidate position does not intersect the boundary, the evaluation for the candidate position is increased. When the camera control information associated with any candidate position indicates pan control from the previous viewpoint position to the candidate position, the evaluation value for the candidate position is:
The video generation device according to claim 5, wherein a straight line connecting the immediately preceding viewpoint position and the candidate position is higher than an evaluation for the candidate position when the boundary does not intersect the boundary. The camera control information includes a viewing angle θ indicating in which direction the candidate position is with respect to the subject, and framing condition information indicating a ratio that the subject should occupy on the screen ,
The procedure for obtaining the moving image is as follows:
The method according to claim 1, further comprising: determining a distance r between the subject and the viewpoint position based on the framing condition information , and obtaining a three-dimensional coordinate of the candidate position from the viewpoint angle θ and the distance r. Video generation device. A program for causing a computer to generate a moving image whose viewpoint position is changed based on scene description information,
The scene description information consists of camera control information corresponding to each of a plurality of candidate positions.
Evaluating how each candidate position corresponds to the current viewpoint position by applying a predetermined rule to the set of the previous viewpoint position and one candidate position;
Selecting the candidate position with the highest evaluation result as the current viewpoint position and using the camera control information corresponding to the current viewpoint position to obtain a moving image when the subject is viewed from the current viewpoint position. A program characterized by being executed.

Images