JP7277184B2 - Information processing device, information processing method and program. - Google Patents

Description

The present invention relates to technology for setting a virtual viewpoint for a virtual viewpoint image.

In recent years, there has been a technology in which a plurality of real cameras are installed at different positions and synchronously photographed from multiple viewpoints, and a virtual viewpoint image observed from an arbitrary viewpoint is generated using the multi-viewpoint images obtained by the synchronous photography. Attention has been paid. With services using virtual viewpoint images, for example, video creators can create powerful content from video footage of soccer or rugby matches. In addition, the user viewing the content can watch the game while freely moving his/her viewpoint. Therefore, according to the service using the virtual viewpoint image, it is possible to give the user a high sense of realism as compared with a general photographed image.

Patent Literature 1 describes using a keyframe method as a method for displaying animation by three-dimensional computer graphics. In the keyframe method, the intermediate frames between the keyframes specifying the key display state are automatically interpolated by the spline function.

JP 2007-025979 A

In setting the camera path representing the movement of the virtual viewpoint related to the virtual viewpoint image, the user can easily set the camera path by specifying the time and viewpoint of the key frame. For example, the speed of movement of the viewpoint in the camera path is determined according to the time interval between designated key frames and the amount of movement of the viewpoint. However, depending on the relationship between a plurality of keyframes designated by the user, the moving speed of the viewpoint may differ significantly before and after the keyframe, and the virtual viewpoint image obtained according to the camera path may become unnatural.

SUMMARY OF THE INVENTION The present invention provides a technique that enables easy setting of the movement of a virtual viewpoint for generating a virtual viewpoint image with little discomfort.

An information processing apparatus according to one aspect of the present invention has the following configuration. i.e.
An information processing device that edits a camera path that defines the movement of a virtual viewpoint,
designation means for receiving designation of a key frame at an arbitrary time and designation of a state of a virtual viewpoint corresponding to the designated key frame for configuring the camera path;
acquisition means for acquiring an amount of change in the state of the virtual viewpoint before and after the key frame specified by the specifying means;
changing means for changing the position of the specified keyframe in the sequence of frames forming the camera path so that the difference between the amount of change acquired by the acquiring means before and after the specified keyframe becomes small; Prepare.

According to the present invention, it is possible to easily set the movement of a virtual viewpoint for generating a virtual viewpoint image that does not give a sense of discomfort.

FIG. 2 is a block diagram showing a configuration of an imaging system and a hardware configuration example of an information processing apparatus; FIG. 2 is a block diagram showing a functional configuration example of an information processing apparatus; FIG. 4 is a diagram for explaining the arrangement of real cameras and the setting of virtual cameras in an imaging system; FIG. 4 is a diagram for explaining a user interface for camera path editing; FIG. 10 is a diagram for explaining a display example of a user interface in camera path editing; 4 is a flowchart for explaining camera path editing processing. 5 is a flowchart for explaining camera path editing processing in a time change mode; FIG. 11 is a diagram for explaining an example of editing a camera path by changing the times of keyframes; 6 is a flowchart for explaining camera path editing processing in a frame number change mode; 4A and 4B are diagrams for explaining an example of camera path editing by changing the number of frames; FIG. 4A and 4B are diagrams for explaining calculation of the amount of change in the state of a virtual camera; FIG.

Hereinafter, embodiments will be described in detail with reference to the accompanying drawings. In addition, the following embodiments do not limit the invention according to the scope of claims. Although multiple features are described in the embodiments, not all of these multiple features are essential to the invention, and multiple features may be combined arbitrarily. Furthermore, in the accompanying drawings, the same or similar configurations are denoted by the same reference numerals, and redundant description is omitted.

FIG. 1 is a diagram illustrating a configuration example of an imaging system according to an embodiment; The imaging system has an information processing device 100 , a server device 120 , a plurality of real cameras 301 and a UI section 400 . The server device 120 is connected to the plurality of real cameras 301 and the information processing device 100 , collects and stores captured images obtained by synchronous photography of the plurality of real cameras 301 , and provides the captured images to the information processing device 100 . Further, FIG. 1 shows a hardware configuration example of the information processing apparatus. In information processing apparatus 100 , CPU 101 controls the entire information processing apparatus 100 using computer programs and data stored in RAM 102 and ROM 103 . Note that the information processing apparatus 100 may include one or more pieces of dedicated hardware or a GPU (Graphics Processing Unit) different from the CPU 101, and at least part of the processing by the CPU 101 may be performed by the GPU or dedicated hardware. Examples of dedicated hardware include ASICs (Application Specific Integrated Circuits) and DSPs (Digital Signal Processors).

The RAM 102 temporarily stores computer programs and data read from the ROM 103, data externally supplied via the input/output unit 104, and the like. ROM 103 holds computer programs and data that do not require modification. The input/output unit 104 inputs and outputs data to and from a controller for editing camera paths, a display for displaying a graphical user interface (hereinafter referred to as GUI), and the like. In this embodiment, the input/output unit 104 is connected to the server device 120 and the UI unit 400 .

FIG. 2 is a block diagram illustrating a functional configuration example implemented by the information processing apparatus 100 according to the embodiment. As described above, each functional unit may be realized by the CPU 101 executing a predetermined computer program, or at least a part thereof may be realized by dedicated hardware.

The image generation unit 201 generates a virtual viewpoint image seen from the viewpoint of the virtual camera based on a plurality of images captured synchronously from multiple viewpoints. A virtual camera is a virtual camera that can move freely within a shooting space. FIG. 3 shows the arrangement of real cameras and an example of virtual cameras. A plurality of real cameras 301 are arranged so as to surround the shooting space. These multiple real cameras 301 are set to perform synchronous photography. A virtual camera 302 represents an arbitrary viewpoint (virtual viewpoint) different from any real camera 301 . Note that the image generator 201 may be provided in the server device 120 .

Next, an example of a method for the image generation unit 201 to generate a virtual viewpoint image will be described. Hereinafter, in this embodiment, a subject whose position changes, such as a player or a ball, will be referred to as the foreground. A subject other than the foreground, such as a field (lawn), is called a background. First, the image generator 201 separates a plurality of captured images captured by a plurality of real cameras 301 into a foreground and a background. The image generation unit 201 calculates the three-dimensional shape (hereinafter referred to as 3D shape) and the position of the foreground such as a player and a ball from a plurality of separated foregrounds. Next, the image generator 201 reconstructs the foreground from the viewpoint of the virtual camera 302 using the calculated 3D shape and position of the foreground. Next, the image generator 201 generates a background corresponding to the viewpoint of the virtual camera 302 from the separated backgrounds. Note that a background corresponding to the viewpoint of the virtual camera 302 may be generated using the 3D shape of the background. The image generation unit 201 combines the reconstructed foreground with the generated background to generate a virtual viewpoint image.

Note that the information processing apparatus 100 can also be used to edit camera paths of computer graphics images (CG images). In that case, a CG renderer is used instead of the image generator 201, and a virtual camera 302 as a virtual viewpoint represents the viewpoint of the CG image and can move to any position in the CG space.

A camera path editing unit 202 edits a camera path that defines the movement of the virtual viewpoint according to user's operation. In other words, the camera path defines the movement of the virtual camera 302 in a moving image created by sequentially reproducing a plurality of virtual viewpoint images and CG images. Camera paths are managed by frames and timelines. A frame holds information necessary for generating each image that constitutes a moving image, more specifically, information such as the time of a scene and the position/orientation of a virtual camera. The time of the scene is represented by a time code, for example, with the time at which the game to be photographed started at 00:00:00:00 frame. The position of the virtual camera is represented by three coordinates of X, Y, and Z, for example, by setting the origin in the shooting space. The posture of the virtual camera is represented by three angles of pan, tilt, and roll, for example. The number of frames included in the timeline is determined by the number of images reproduced per second (frame rate). For example, if the frame rate is 60 frames/second, 60 frames per second are included in the timeline. Zoom (angle of view) may be included as the state of the virtual viewpoint (virtual camera) represented by the camera path.

In camera path editing using keyframes, frames are divided into two types: keyframes and intermediate frames. Keyframes are frames where information is explicitly specified by the user editing the camera path. Intermediate frames, on the other hand, are frames between key frames. The camera path editing unit 202 calculates the position/orientation of the virtual camera between key frames by interpolation using the position/orientation of the virtual camera specified in the keyframes, and determines the information of each frame. Further, keyframes are divided into three types: start point, end point, and middle point. A camera path includes one key frame (start point), one key frame (end point), and zero or more key frames (middle point). The information processing apparatus 100 of the present embodiment particularly targets a camera path including one or more key frames (midpoints), that is, a camera path including three or more key frames. The camera path to be edited is between the start point keyframe and the end point keyframe.

FIG. 4 shows an example of a user interface (hereinafter referred to as UI) (UI unit 400) for editing camera paths. The image display unit 401 displays the virtual viewpoint image generated by the image generation unit 201 , that is, the image viewed from the virtual camera 302 . A GUI display unit 402 displays information about camera paths and key frames. Edit controller 403 is the controller used by the user to edit the camera path.

A display example of the GUI display unit 402 is shown in FIG. A timeline 501 displays each frame on a single time axis. That is, the timeline 501 schematically displays a sequence of frames of virtual viewpoint images or computer graphics images and positions of key frames in the sequence of frames. In FIG. 5, three key frames of key frame 502, key frame 503, and key frame 504 are shown. A keyframe 502 is a keyframe corresponding to the starting point of the camera path 505 . A keyframe 503 is a keyframe corresponding to the end point of the camera path 505 . Keyframe 504 is the midpoint keyframe. On the other hand, a camera path 505 represents the movement of the position/orientation of the virtual camera corresponding to each frame. A state 506 includes the position and orientation of the virtual camera corresponding to the key frame 502 . A state 507 includes the position and orientation of the virtual camera corresponding to the key frame 503 . A state 508 includes the position and orientation of the virtual camera corresponding to the key frame 504 . A mode display section 509 indicates the camera path change mode being executed. In this embodiment, a time change mode and a frame number change mode are used as camera path change modes, and these modes will be described later.

Note that the camera path editing unit 202 transmits to the image generating unit 201 the information of the frame that is being edited by the user. The image generation unit 201 generates a virtual viewpoint image based on the received frame information, and causes the image display unit 401 to display the generated virtual viewpoint image. This allows the user to edit the camera path while checking the image of the frame to be edited viewed from the virtual camera.

The keyframe adding unit 203 adds a keyframe to the time specified by the user on the timeline 501 . The user uses the edit controller 403 to operate (advance or return) the time to designate a desired time. When the user operates the time, the virtual viewpoint image at the corresponding time is displayed on the image display unit 401 . The user can easily specify the time of a desired scene, such as when a player passes, by performing operations while viewing the virtual viewpoint image displayed on the image display unit 401 . The keyframes 502, 503, 504 added by the user are displayed on the timeline 501 as shown in FIG. By specifying a desired position on the timeline 501 using a pointing device such as a mouse, the user can also add a keyframe to that position.

A position/orientation designation unit 204 designates the state (position/orientation) of the virtual camera of the key frame according to the user's operation. The user uses the edit controller 403 to move the virtual camera back and forth, left and right, and up and down to designate a desired position. Also, using the edit controller 403, the virtual camera is rotated in the pan, tilt, and roll directions to designate a desired posture. When the user operates the position/orientation of the virtual camera, a virtual viewpoint image of the corresponding position/orientation is displayed on the image display unit 401 . The user can easily specify the desired position and orientation of the virtual camera by performing operations while viewing the image display unit 401 . The positions and orientations of the virtual camera corresponding to the keyframes specified by the user are displayed as states 506 , 507 , and 508 . The virtual camera state 506 is the position/orientation of the virtual camera in the key frame 502 . Similarly, the state 508 is the position/orientation of the virtual camera in the keyframe 504 , and the state 507 is the position/orientation of the virtual camera in the keyframe 503 .

A position/posture interpolation unit 205 interpolates the position/posture of the camera between key frames. First, the area between the start point keyframe and the end point keyframe is divided by an intermediate keyframe. In the example of FIG. 5, it is divided into two sections. The first interval is between the starting keyframe 502 and the middle keyframe 504 . The second section is between the middle keyframe 504 and the end keyframe 503 . For each divided section, the position and orientation of the camera are interpolated with curves. As a method of interpolating the camera position, for example, linear interpolation, Bezier interpolation, or the like is used. As a method for interpolating the orientation of the camera, for example, linear interpolation, spherical linear interpolation, or the like is used. The interpolated curve is camera path 505 .

The change amount calculation unit 206 calculates and acquires the amount of change in the state of the virtual viewpoint (virtual camera) before and after the specified keyframe (added keyframe). In this embodiment, the change amount calculation unit 206 calculates the amount of change in the position/orientation of the camera between key frames. The amount of change in camera position is the length of the curve/straight line interpolated between keyframes. The length of the curve/straight line can be calculated, for example, by integrating the function representing the curve/straight line. The amount of change in camera posture is the amount of change in pan, tilt, and roll angles. The amount of change in pan, tilt, and roll angles can be calculated as the difference between the pan, tilt, and roll angles in adjacent keyframes. Alternatively, the average amount of change in the pan, tilt, and roll angles between key frames may be used as the amount of change in the orientation of the camera. In addition, in the present embodiment, the amount of change in the position and the orientation is synthesized (for example, synthesized by linear summation) to obtain the amount of change in the state.

A change mode management unit 207 manages a mode for changing the camera path. One of the camera path change modes is a time change mode that changes the time of keyframes. Another mode is a frame number change mode that changes the number of intermediate frames inserted between key frames. The user can use the edit controller 403 to switch between the time change mode and the frame number change mode as desired. In either mode, the position of the designated key frame in the frame sequence of the virtual viewpoint image or the CG image is adjusted so that the difference in the amount of change acquired by the change amount calculation unit 206 before and after the designated key frame is small. is changed. The time change unit 208 operates in the time change mode, and changes the time of the added keyframe so that the amount of change in the virtual viewpoint before and after the added keyframe becomes small. The frame number change unit 209 operates in a frame number change mode, and adjusts the amount of change between the added keyframe and its adjacent keyframes so that the amount of change in the virtual viewpoint before and after the added keyframe is small. Change the number of intermediate frames. The operations of the time changing unit 208 and the frame number changing unit 209 will be described below.

[Time change mode]
The time changing unit 208 adjusts the amount of change in the state of the virtual viewpoint between the keyframes (for example, the amount of change in the position/orientation) so that the difference in the amount of change before and after the added keyframe becomes small. Change the times of keyframes. More specifically, the time changing unit 208 changes the time of the keyframe at the intermediate point so that, for example, before and after the keyframe at the intermediate point, the amount of change per hour in the position/orientation of the camera becomes equal. do. Note that the time change unit 208 operates in the time change mode for changing the time of the keyframe.

Hereinafter, a method of changing the time of the keyframe at the intermediate point in the time change mode will be described using the example of FIG. 11(a). FIG. 11A is a diagram illustrating calculation of the amount of change in the state of the virtual viewpoint. In FIG. 11A, the amount of change in the state of the virtual viewpoint (the position and orientation of the virtual camera) between the keyframes 502 and 504 is L1. That is, L1 is the amount of change from state 506 to state 508 of the virtual camera. Let L2 be the amount of change in the state of the virtual viewpoint (the position and orientation of the virtual camera) between the key frames 504 and 503 . That is, L2 is the amount of change from state 508 to state 507 of the virtual camera. Also, the time of the key frame 502 is T1, the time of the key frame 503 is T2, and the changed time of the key frame 504 is T3a.

Before the key frame 504, the change amount ΔL1 of the virtual camera position/orientation per time is L1/(T3a-T1). After the key frame 504, the change amount ΔL2 of the virtual camera position/orientation per time is L2/(T2−T3a). Therefore, in order to equalize the amounts of change ΔL1 and ΔL2 per time in the position/orientation of the virtual camera before and after the key frame 504, L1/(T3a−T1)=L2/(T2−T3a) should be satisfied. Time T3a of key frame 504 may be set. Therefore, time T3a of key frame 504=(L1*T2+L2*T1)/(L1+L2). When the time of the keyframe 504 is changed, the keyframe 504 moves on the timeline 501 . Note that the state 508 (position/orientation) of the virtual camera in the key frame 504 does not change. Note that ΔL1 and ΔL2 do not necessarily have to match, and it is sufficient if the difference between ΔL1 and ΔL2 after the change is smaller than the difference between ΔL1 and ΔL2 before the time of the key frame 504 is changed.

The amount of change in the position of the virtual camera is equal to the moving speed of the virtual camera. According to the above processing, the difference in the moving speed of the virtual camera before and after the key frame 504 becomes smaller or equal. As a result, it is possible to reduce the unnaturalness caused by the change in the moving speed of the virtual camera before and after the key frame without the user having to separately adjust the moving speed of the virtual camera.

Note that the state of the virtual viewpoint (virtual camera) is at least one of the position and orientation of the virtual viewpoint (virtual camera). That is, as the amounts of change L1 and L2 in the state of the virtual camera, only the amount of change in the position of the virtual camera may be used, or only the amount of change in the posture of the virtual camera may be used. Alternatively, only one change amount of pan, tilt, and roll of the posture of the virtual camera may be used as the change amounts L1 and L2. When the position and orientation of the virtual camera are used together as the amounts of change L1 and L2, for example, values obtained by linearly combining the amounts of change may be used.

Also, if the difference in the amount of change per time in the state of the virtual camera (for example, position/orientation) before and after the keyframe at the intermediate point is greater than or equal to a predetermined value, the time at the keyframe at the intermediate point is changed. can be For example, if the rate of change per time in the position/orientation of the camera before and after the midpoint keyframe is greater than 0.8, the keyframe (midpoint) time is changed to T3a. In this case, the following (formula 1)
0.8≦{L1(T2−T3)}/{L2(T3−T1)}≦1/0.8 (Formula 1)
is used to determine whether to change the time of the keyframe. Here, T3 is the time before the key frame 504 is changed. If (Formula 1) is not satisfied, the time changing unit 208 changes the time of the key frame at the intermediate point added according to the method described above to T3a.

Also, the change amount calculation unit 206 calculates the change amount L1 of the position/orientation of the virtual camera at the keyframe at the intermediate point using frames (intermediate frames or keyframes) within a predetermined period of time before and after the keyframe at the intermediate point. , L2 may be calculated. For example, as shown in FIG. 11B, the change amount calculation unit 206 calculates the state of the virtual viewpoint at the frame 1101 a predetermined time (T3a-Tx) before the key frame 504 at the midpoint and the virtual viewpoint at the key frame 504. , and the amount of change L1 is calculated. Further, the change amount calculation unit 206 calculates the state change amount L2 from the state of the virtual viewpoint in the frame 1102 after a predetermined time (T3a+Tx) after the key frame 504 and the state of the virtual viewpoint in the key frame. Note that the state of the virtual viewpoint (the position/orientation of the virtual camera) corresponding to the frames 1101 and 1102 can be acquired by the position/orientation interpolation unit 205 . The time changing unit 208 determines the time T3a so that L1 and L2 thus calculated are equal. In the above example, the same predetermined time Tx is used before and after the keyframe 504 at the specified intermediate point, but different times before and after the keyframe 504 may be used. Also, the keyframes 02 and 503 at times T1 and T2 are defined as the start point keyframe and the end point keyframe, respectively, but the present invention is not limited to this. That is, the time change mode process described above may be executed using the key frames adjacent to the added key frame 504 .

[Frame number change mode]
The number-of-frames changing unit 209 changes the number of intermediate frames between keyframes so that the difference in the amount of change per frame in the state (position/orientation) of the virtual camera between keyframes becomes smaller. Specifically, in this embodiment, the number of intermediate frames is changed so that the amount of change per frame in the state (position/orientation) of the virtual camera before and after the intermediate point keyframe is the same. Note that the frame number changing unit 209 operates in a frame number changing mode for changing the number of intermediate frames between key frames. The state of the virtual camera (the state of the virtual viewpoint) is as described above.

The operation of the frame number change mode will be described below using the example of FIG. 11(a). In FIG. 11A, the number of pre-change intermediate frames between the keyframes 502 and 504 is N1, and the number of pre-change intermediate frames between the keyframes 504 and 503 is N2. Also, let the time of the key frame 504 be T3. The number of intermediate frames before change is determined by the time difference between key frames and the frame rate. For example, if the frame rate is 60 frames/second, N1=(T3-T1)*60-1 and N2=(T2-T3)*60-1.

Here, the number of intermediate frames after change between the key frame 502 and the key frame 504 is assumed to be N1a. Also, let N2a be the number of post-change intermediate frames between the key frame 504 and the key frame 503 . Then, the amount of change per frame of the state (position/orientation) of the virtual camera before the key frame 504 is L1/N1a. Also, the amount of change per frame in the state (position/orientation) of the virtual camera after the key frame 504 is L2/N2a. Therefore, before and after the key frame 504, the condition for equalizing the amount of change per frame in the state (position/orientation) of the virtual camera is L1/N1a=L2/N2a.

Next, the frame number changing unit 209 determines on which side before or after the key frame 504 the number of intermediate frames is to be changed. In the present embodiment, the frame is added to the one with the greater amount of change, either before or after the designated frame. More specifically, for example, the frame number changing unit 209 compares the amount of change per time before and after the key frame 504, changes the number of intermediate frames on the side with the larger amount of change, and changes the number of intermediate frames on the side with the smaller amount of change. The number of intermediate frames is not changed. Note that the change in the number of frames is executed when the difference in the amount of change in the virtual camera state (for example, position/orientation) per frame before and after the keyframe at the intermediate point is greater than or equal to a predetermined value. good too.

A specific description will be given below. The amount of change per time ΔL1 in the state of the virtual camera before keyframe 504 is L1/(T3−T1). Also, the amount of change per time ΔL2 of the state of the virtual camera after the key frame 504 is L2/(T2−T3). Here, for example, it is assumed that the amount of change ΔL1 per time before the keyframe 504 , that is, between the keyframes 502 and 504 is greater than the amount of change ΔL2 after the keyframe 504 . In this case, the number-of-frames changing unit 209 changes the number of intermediate frames between the keyframes 502 and 504 and does not change the number of intermediate frames between the keyframes 504 and 503 . Therefore, N2a=N2=(T2-T3)*60-1 and N1a=L1/L2*N2a=L1/L2*{(T2-T3)*60-1}. Here, N1a>N1, and the frame number changing unit 209 adds “N1a−N1” intermediate frames between the key frames 502 and 504. FIG. Note that .DELTA.L1 and .DELTA.L2 do not necessarily have to match, as long as the difference between .DELTA.L1 and .DELTA.L2 after the change is smaller than the difference between .DELTA.L1 and .DELTA.L2 before the number of intermediate frames is changed.

According to the above processing, after the keyframe 504 in which the number of intermediate frames is not changed, the moving image is reproduced at the same size, while before the keyframe 504 to which the intermediate frame is added, the moving image is reproduced in slow motion. will be However, the amount of change in the state of the virtual camera per frame (for example, moving speed) is the same before and after key frame 504, or the difference is small. As a result, it is possible to reduce the unnaturalness caused by the camera moving speed changing before and after the added keyframe without the user having to separately adjust the moving speed of the virtual camera. Note that the number of intermediate frames may be changed by reducing frames instead of adding frames. For example, if the amount of change ΔL1 before the keyframe 504 is greater than the amount of change ΔL2 after the keyframe 504, the frame number changing unit 209 may reduce intermediate frames between the keyframes 504 and 503. good.

Note that the method of adding intermediate frames can be realized, for example, by repeating arbitrary intermediate frames between key frames. Also, in the same way as in the time change mode, in the frame number change mode as well, frames corresponding to a predetermined time period before and after the added keyframe are used to reduce the difference in the amount of change in the state before and after the keyframe. may That is, in FIG. 11B, the time (T3-Tx) a predetermined time before the specified key frame 504 is set to T1, and the time (T3+Tx) after the key frame 504 by a predetermined time is used as T2. obtain. In this case, the frame number changing unit 209 changes the number of frames between the key frame 504 and the frame 1101 a predetermined time before, or the number of frames between the key frame 504 and the frame 1102 after the predetermined time. By increasing or decreasing, the difference in the amount of change is reduced. Moreover, although the same predetermined time Tx is used before and after the key frame 504 at the specified intermediate point in the above description, different times may be used before and after the key frame 504 . Also, the keyframes 02 and 503 at times T1 and T2 are defined as the start point keyframe and the end point keyframe, respectively, but the present invention is not limited to this. That is, the time change mode process described above may be executed using the key frames adjacent to the added key frame 504 .

[Position/Posture Allocation]
The allocation unit 210 allocates the position/orientation of the camera to the intermediate frames determined through the time change unit 208 or the frame number change unit 209 . For example, assignment of the state (position/orientation) of the virtual camera to the N1 intermediate frames is performed as follows. First, the allocation unit 210 equally divides the camera path 505 between the virtual camera state 506 in the key frame 502 and the virtual camera state 508 in the key frame 504 into N1+1 pieces, and divides the virtual camera state at the division point into N1 pieces. sequentially assigned to intermediate frames. The state (position/orientation) of the virtual camera can be similarly assigned to intermediate frames between the keyframes 504 and 503 .

Moreover, when the number of intermediate frames between key frames is changed, the allocation unit 210 also reassigns times to the intermediate frames. For example, when the number of intermediate frames is changed from N1 to N1a, the allocation unit 210 allocates the position/orientation of the camera to the N1a intermediate frames, and reassigns the time of each intermediate frame as follows. . First, the allocation unit 210 equally divides the time between the key frame 502 and the key frame 504 into N1a+1 pieces. Then, the allocation unit 210 sequentially allocates the division position times to the N1a intermediate frames. On the other hand, it is not necessary to reassign times to sections in which the number of intermediate frames has not been changed.

A camera path output unit 211 outputs a camera path made up of key frames and intermediate frames. The image generation unit 201 moves the virtual viewpoint according to the camera path output from the camera path output unit 211 and generates a virtual viewpoint image.

[Time change mode operation]
FIG. 6 is a flowchart for explaining the processing procedure of the information processing apparatus 100 according to this embodiment. Processing when the time change mode is selected by the change mode management unit 207 will be described.

In step S601, the keyframe addition unit 203 adds a start point keyframe according to the user's operation. For example, when the user selects a desired time on the timeline 501 using the edit controller 403, a starting keyframe is added at that time. Also, the camera path editing unit 202 displays the added starting point keyframe (for example, the keyframe 502 ) on the timeline 501 .

In step S602, the position/posture designation unit 204 designates the position/posture as the state of the virtual camera at the starting keyframe in accordance with the user's operation. For example, the user uses the edit controller 403 to move the virtual camera back and forth, left and right, and up and down, or to rotate the virtual camera in pan, tilt, and roll directions, thereby specifying the position and orientation of the virtual camera. . In step S<b>603 , the camera path editing unit 202 transmits to the image generating unit 201 the time of the starting key frame and the position/orientation information of the camera. The image generation unit 201 generates a virtual viewpoint image from the time of the received key frame and the position/orientation of the virtual camera. The generated virtual viewpoint image is transmitted to the image display unit 401 and displayed.

The user confirms the virtual viewpoint image displayed on the image display unit 401 and confirms whether there is any problem with the position/orientation of the camera of the starting key frame. If there is no problem, the user uses the editing controller 403 to instruct the camera path editing unit 202 that there is no problem. When instructed that there is no problem, the camera path editing unit 202 determines that there is no problem (YES in step S604), and advances the process to step S605. If there is a problem with the position/orientation of the virtual camera in the starting keyframe, the user uses the edit controller 403 to specify the position/orientation of the virtual camera again. When the editing controller 403 designates the position and orientation of the virtual camera again, the camera path editing unit 202 determines that there is a problem (NO in step S604), and returns the process to step S602.

In step S605, the keyframe addition unit 203 adds the end point keyframe according to the user's operation. A specific procedure is the same as that of step S601. In step S<b>606 , the position/orientation designation unit 204 designates the position/orientation of the virtual camera at the end point keyframe in accordance with the user's operation. A specific procedure is the same as that of step S602. In step S<b>607 , a virtual viewpoint image is generated according to the time of the end point keyframe and the position/orientation of the virtual camera and displayed on the image display unit 401 in the same procedure as in step S<b>603 .

The user confirms the virtual viewpoint image displayed on the image display unit 401, confirms whether there is any problem with the position and orientation of the virtual camera at the end point key frame, and uses the editing controller 403 to indicate the confirmation result. . As in step S604, if the camera path editing unit 202 determines that there is no problem in response to the user operation (YES in step S608), the process proceeds to step S609. NO), the process returns to step S606.

In step S<b>609 , the keyframe adding unit 203 adds a keyframe at the intermediate point according to the user's operation using the editing controller 403 . The details of the addition process will be described later with reference to the flowchart of FIG. In step S610, the allocation unit 210 allocates the interpolated camera path 505 between keyframes to the position/orientation of the virtual camera in the intermediate frame. In step S611, the camera path output unit 211 outputs a camera path made up of key frames and intermediate frames.

FIG. 7 is a flowchart for explaining the processing procedure in the time change mode in step S610 of FIG. First, in step S701, the keyframe addition unit 203 determines whether or not a keyframe at the intermediate point has been added. If a midpoint keyframe has been added (YES in step S701), the process proceeds to step S702. If no intermediate point keyframe has been added (NO in step S701), this process ends. After step S702, the camera path editing unit 202 performs key frame (midpoint) addition processing.

In step S702, the keyframe addition unit 203 adds a keyframe at the intermediate point according to the user's operation. A specific procedure is the same as that of step S601. In step S703, the position/orientation designation unit 204 designates the position/orientation as the state of the virtual camera in the key frame at the intermediate point in accordance with the user's operation. A specific procedure is the same as that of step S602. In step S704, the position/posture interpolation unit 205 interpolates the position/posture of the camera between key frames. The camera path editing unit 202 also displays the interpolated camera position/orientation as a camera path 505 on the GUI display unit 402 .

In step S705, the change amount calculation unit 206 calculates the amount of change in the position/orientation of the virtual camera between the keyframes before and after the keyframe at the added intermediate point. In step S706, the time changing unit 208 changes the time of the keyframe (midpoint) according to the amount of change in the position/orientation of the camera between the keyframes. As described above, the time changing unit 208 changes the time of the keyframe at the intermediate point so that the amount of change per time in the position/orientation of the virtual camera before and after the keyframe at the intermediate point is equal or the difference is small. to change Also, the camera path editing unit 202 updates the display of the timeline 501 according to the time change in the keyframe at the intermediate point.

In step S707, in the same procedure as in step S603, the image generation unit 201 generates a virtual viewpoint image according to the time of the intermediate point keyframe added in step S701 and the position/orientation of the virtual camera. 401 to display. The user confirms the virtual viewpoint image displayed on the image display unit 401 and confirms whether there is any problem. In step S708, it is determined whether or not there is a problem in the same procedure as in step S604, and if it is determined that there is no problem (YES in step S708), this process ends. If it is determined that there is a problem (NO in step S708), the process returns to step S702.

With reference to FIG. 8, an operation example when the time change mode for changing the time of the keyframe is set as the camera path change mode will be described. It is assumed that the position and orientation of the camera have been specified for the start point keyframe and the end point keyframe. In a timeline 501, a key frame 502 is the time of the starting key frame, and a key frame 503 is the time of the ending key frame. A state 506 represents the position and orientation of the virtual camera in the starting keyframe, and a state 507 represents the position and orientation of the virtual camera in the ending keyframe. Also, the mode display portion 509 displays "time" representing the time change mode.

FIG. 8(a) shows a state in which a key frame 504 at the intermediate point is newly added (step S702). State 508 is the position and pose of the virtual camera at midpoint keyframe 504 . At this point, the position/orientation of the virtual camera in the keyframe 504 is not specified, so for example, the value of the starting keyframe 502 is used as the initial value of the position/orientation of the virtual camera. As the position/orientation of the virtual camera in the added keyframe, the position/orientation of the virtual camera in the keyframe to the left may be used, or the position/orientation of the virtual camera in the keyframe closest to the added keyframe may be used. A position/orientation may be used.

FIG. 8B shows display contents when the state 508 of the virtual camera is specified for the key frame 504 (step S703). FIG. 8C shows display contents when the time of the key frame 504 is changed in steps S704 to S706. When the state 508 (position/orientation) of the virtual camera for the keyframe 504 is specified (step S703), the position/orientation interpolation unit 205 interpolates the position/orientation of the virtual camera between the keyframes (step S704). Display the result as camera path 505 . Next, the time changing unit 208 changes the time of the key frame 504 from T3 to T3a so that the amount of change per time in the position/orientation of the virtual camera between key frames becomes equal (steps S705 and S706). By changing the time, the keyframe 504 moves on the timeline 501 as indicated by an arrow 801 . That is, the position of the key frame 504 in the frame sequence of the virtual viewpoint image is changed. However, the state 508 (position/orientation) of the virtual camera corresponding to the key frame 504 is not changed.

Note that when the time of the key frame 504 is changed, the state in the shooting space (such as the position of the subject) changes, and the position/orientation of the virtual camera may deviate from the user's initial intention. However, by repeating the change of the time of the key frame 504 at short intervals (for example, 1/60th of a second), the user can confirm the change of the state in real time. As a result, the user can appropriately update his/her intention under the constraint that the amount of change in the state of the virtual camera before and after the key frame 504 is within a certain range (the movement speed is within a certain range), and the virtual The camera state 508 can be specified efficiently and easily.

[Operation of frame number change mode]
FIG. 9 is a flowchart for explaining the processing procedure in the frame number change mode in step S610 of FIG. The operation when the camera path change mode is the frame number change mode for changing the number of intermediate frames to be inserted between key frames will be described below with reference to FIG. Note that the process of setting the start and end point keyframes, the process of assigning the position and orientation of the camera to the intermediate frames after setting the intermediate point keyframes, and the process of outputting the camera path are in the time change mode (Fig. 6). is similar to

When the keyframe addition unit 203 adds the intermediate point keyframe according to the user operation (YES in step S901), in step S902, the keyframe addition unit 203 adds the intermediate point keyframe according to the user operation. do. In step S903, the position/posture designation unit 204 designates the position/posture as the state of the virtual camera with respect to the key frame at the intermediate point in accordance with the user's operation. In step S904, the position/posture interpolation unit 205 interpolates the position/posture of the virtual camera between key frames. The camera path editing unit 202 also displays the interpolated camera position/orientation as a camera path 505 on the GUI display unit 402 . The processing of steps S901 to S904 is the same as steps S701 to S704 described with reference to FIG.

In step S<b>905 , the image generation unit 201 generates a virtual viewpoint image according to the time of the intermediate point key frame and the position/orientation of the virtual camera, and displays the generated virtual viewpoint image on the image display unit 401 . The processing of step S905 is similar to that of step S603. In step S906, the user confirms the virtual viewpoint image displayed on the image display unit 401 and confirms whether there is any problem. Whether or not there is a problem is determined by the same procedure as in step S604. If there is no problem (YES in step S906), the process proceeds to step S907, and if there is a problem (NO in step S906), the process returns to step S903. .

In step S907, the change amount calculation unit 206 calculates the amount of change in the position/orientation of the virtual camera between key frames. In step S908, the number-of-frames changing unit 209 changes the number of intermediate frames to be inserted according to the amount of change in the position/orientation of the virtual camera between keyframes. As described above, the number-of-frames changing unit 209 of this embodiment changes the number of intermediate frames so that the amount of change per frame in the position/orientation of the virtual camera before and after the keyframe at the intermediate point is the same.

A specific example will be described with reference to FIG. 10 when the number-of-frames change mode is set to change the number of intermediate frames to be inserted between key frames. Note that FIG. 10A shows the state before specifying the position/orientation of the virtual camera corresponding to the added keyframe, and shows the same display state as FIG. 8A. FIG. 10(b) shows the state immediately after specifying the position/orientation of the virtual camera corresponding to the added keyframe, which is the same display state as in FIG. 8(b). In FIG. 10, the mode display section 509 displays "frame" representing the frame number change mode.

FIG. 10C shows a state in which the number of intermediate frames between keyframes has been changed. First, when the state 508 (position/orientation) of the virtual camera in the keyframe 504 is specified, the position/orientation interpolating unit 205 interpolates the position/orientation of the virtual camera between the keyframes, and calculates the camera path obtained by the interpolation. 505 is displayed. Next, the number-of-frames changing unit 209 changes the number of intermediate frames so that the amount of change per frame in the position/orientation of the virtual camera between key frames is equal or the difference is small. In this example, an intermediate frame is added between keyframe 502 and keyframe 504 . With the intermediate frame added, the keyframe 504 moves on the timeline 501 as indicated by an arrow 1001 . That is, the position of the key frame 504 in the frame sequence of the virtual viewpoint image is changed. As a result, the keyframe 503 moves on the timeline 501 as indicated by an arrow 1002 . Note that the times of the start point key frame 502, intermediate point key frame 504, and end point key frame 503 are not changed. Also, the number of intermediate frames between the key frame 504 at the intermediate point and the key frame 503 at the end point is not changed.

(Other embodiments)
The above-described time change mode and frame number change mode can be used in combination. That is, both the addition of the intermediate frame and the change of the time of the keyframe may be executed according to the designation of the position and orientation of the virtual viewpoint in the keyframe.
The present invention supplies a program that implements one or more functions of the above-described embodiments to a system or device via a network or a storage medium, and one or more processors in the computer of the system or device reads and executes the program. It can also be realized by processing to It can also be implemented by a circuit (for example, ASIC) that implements one or more functions.

The invention is not limited to the embodiments described above, and various modifications and variations are possible without departing from the spirit and scope of the invention.

201: Image generation unit 202: Camera path editing unit 203: Key frame addition unit 204: Position/posture specification unit 205: Position/posture interpolation unit 206: Change amount calculation unit 207: Change mode management unit 208: time changing unit, 209: frame number changing unit, 210: allocation unit, 211: camera path output unit

Claims (17)

An information processing device that edits a camera path that defines the movement of a virtual viewpoint,
designation means for receiving designation of a key frame at an arbitrary time and designation of a state of a virtual viewpoint corresponding to the designated key frame for configuring the camera path;
acquisition means for acquiring an amount of change in the state of the virtual viewpoint before and after the key frame specified by the specifying means;
changing means for changing the position of the specified keyframe in the sequence of frames forming the camera path so that the difference between the amount of change acquired by the acquiring means before and after the specified keyframe becomes small; An information processing device comprising: 2. The information processing apparatus according to claim 1, wherein said acquisition means acquires the amount of change per time in the state of the virtual viewpoint before and after the designated key frame. The acquisition means acquires the amount of change per time in the state of the virtual viewpoint based on the time and state of the designated keyframe and the time and state of keyframes adjacent before and after the designated keyframe. 3. The information processing apparatus according to claim 2, characterized by: 2. The acquiring means acquires the amount of change in the state from a predetermined time before the specified key frame and the amount of change in the state from the specified key frame to the time after the specified time. The information processing device according to . 5. The information processing apparatus according to claim 2, wherein said changing means changes the time of said designated key frame. 6. The method according to claim 5, wherein said changing means changes the time of said specified key frame so that the amount of change per time in the state of the virtual viewpoint is equal before and after said specified key frame. The information processing device described. 2. The information processing apparatus according to claim 1, wherein said acquisition means acquires the amount of change per frame of the virtual viewpoint state before and after said specified key frame. The acquisition means acquires a virtual viewpoint based on the time and state of the designated key frame, the number of frames between key frames adjacent before and after the designated key frame, and the states of the adjacent key frames. 8. The information processing apparatus according to claim 7, wherein the amount of change in state per frame is acquired. 9. An information processing apparatus according to claim 8, wherein said changing means changes the number of frames between said designated key frame and said adjacent key frame. 10. The information processing apparatus according to claim 9, wherein the changing unit changes the number of frames so that the amount of change per frame of the virtual viewpoint state before and after the designated key frame is equal. . 11. The information processing apparatus according to claim 10, wherein the frame is added to the one before and after the designated frame, which has the greater amount of change. 12. The information processing apparatus according to any one of claims 1 to 11, further comprising interpolation means for interpolating the position and/or orientation of the camera between key frames. 13. The information processing apparatus according to any one of claims 1 to 12, further comprising display means for displaying a virtual viewpoint image corresponding to the time of the specified key frame and the state of the virtual viewpoint. 14. The information processing apparatus according to any one of claims 1 to 13, further comprising display means for schematically displaying the sequence of frames and positions of key frames in the sequence of frames. 15. The information processing apparatus according to any one of claims 1 to 14, wherein said changing means executes said change when the difference between said change amounts is equal to or greater than a predetermined value. An information processing method for editing a camera path that defines movement of a virtual viewpoint, comprising:
a designation step of receiving designation of a key frame at an arbitrary time and designation of a state of a virtual viewpoint corresponding to the designated key frame for configuring the camera path;
an acquiring step of acquiring a change amount of the state of the virtual viewpoint before and after the key frame specified by the specifying step;
a changing step of changing the position of the specified keyframe in the sequence of frames forming the camera path so that the difference between the amounts of change acquired in the acquiring step before and after the specified keyframe becomes small; An information processing method comprising: A program for causing a computer to function as each means of the information processing apparatus according to any one of claims 1 to 15.

Images