JP7466730B2 - Program, electronic device and data recording method - Google Patents

Description

The present invention relates to a program for enabling a computer to generate images of a virtual three-dimensional space.

Augmented reality (AR) technology is used in a variety of fields. For example, Patent Document 1 discloses a technology that associates the position of a user terminal in real space with the position of a virtual camera in a virtual three-dimensional space, superimposes an image of an object (such as a character appearing in a game) located in the virtual three-dimensional space captured by the virtual camera on an image of the real space captured by a camera mounted on the user terminal, and then records the superimposed image based on the user's operational input.

JP 2012-69126 A

Using the technology in Patent Document 1, users can have fun creating still images that make it seem as if virtual characters are actually present in real space.

It is possible to create augmented reality videos using 3DCG (3-Dimensional Computer Graphics) software with a high-performance personal computer, but in that case, the movement trajectory of the virtual camera created with the 3DCG software is created using mathematical functions that have no jitter. Therefore, the camerawork in the completed augmented reality video is very smooth, as if a professional videographer had taken the video using a high-performance gimbal. There is no realistic sense of camera shake that occurs when the user himself holds a video camera in his hand. Therefore, the handmade feeling of "made by the user himself" and the reality of "filmed by the user" are weak, and it is difficult to wipe away the artificial feeling of "made with CG software". If you look at the same thing from the standpoint of a professional videographer, in a case where you have to create an augmented reality video that pursues the reality of "filmed by the user himself holding a video camera in his hand" in terms of video expression, a camerawork that is too smooth and without a sense of camera shake will not be able to achieve the required effect.

These problems are not limited to creating videos of augmented reality images, but also apply to creating videos of virtual reality (VR) images.

This invention was conceived with the objective of providing new technology for creating augmented reality or virtual reality videos with the same realism as if the user had taken the video with a handheld video camera.

The first invention for solving the above-mentioned problems is a program for causing a portable computer to function as a measurement data acquisition control means for controlling the acquisition of measurement data obtained by measuring the position and attitude of the computer, a virtual space setting means for setting a virtual three-dimensional space in which a subject object is placed, a virtual camera control means for controlling the position and attitude of a virtual camera that captures the virtual three-dimensional space in conjunction with the measurement data, a virtual space image generation means for generating an image of the virtual three-dimensional space captured by the virtual camera, and a recording control means for recording camerawork data capable of reproducing the position and attitude of the virtual camera in the virtual three-dimensional space continuously controlled by the virtual camera control means.

The "computer" referred to here can refer to a single computer, or it can refer to multiple computers working together.

According to the first invention, the computer can generate a virtual space image taken by a virtual camera linked to the measured position and attitude, while recording camerawork data that can reproduce the position and attitude of the virtual camera. In other words, if a user performs camerawork for taking a video by changing the position and attitude of the computer as if it were a portable video camera while looking at the virtual space image as if it were a viewfinder image, the camerawork data is recorded in the computer.

Naturally, the trajectory of the virtual camera's movement, which is reproduced based on this camerawork data, contains the realistic "wobble" that occurs when a user actually holds the camera in their hands while filming. Therefore, by using the camerawork data recorded on a computer with 3DCG software, it is possible to create videos of augmented reality or virtual reality images that have the realism of "as if the user were filming with a video camera in their hands."

The second invention is the program of the first invention, which causes the computer to function as a subject control means that controls the subject object to perform a given action along a given time axis, and the recording control means records the position and orientation of the virtual camera in association with the time axis of the measurement data, thereby recording camerawork data that can reproduce camerawork synchronized with the time axis.

The second invention makes it possible to reproduce camerawork synchronized with the movement of the subject object.

The third invention is the program of the second invention, which causes the computer to function as a sound output control means that controls the output of a given piece of music and/or a given sound (hereinafter collectively referred to as "sound") along the time axis, and the subject control means causes the subject object to perform an action in accordance with the sound.

The third invention makes it possible to reproduce camera work synchronized with sound playback.

The fourth invention is the program of the second or third invention, in which the recording control means starts recording in response to a given instruction to start recording and stops recording in response to a given instruction to stop recording, and the subject control means performs simultaneous control to start the time axis in response to the instruction to start recording, to start the movement of the subject object, and stops the time axis in response to the instruction to stop recording, to stop the movement of the subject object.

According to the fourth invention, the user himself/herself can start and stop recording of camerawork data, and in addition, when recording starts, the movement of the subject object can be started, and when recording stops, the movement of the subject object can be stopped.

In a fifth aspect of the present invention, the simultaneous control can be switched on/off by a switching operation,
The subject control means is a program of a fourth invention in which the subject control means performs the simultaneous control when the simultaneous control is on, and continues to control the movement of the subject object when the simultaneous control is off regardless of the recording start instruction and the recording stop instruction.

According to the fifth invention, the user can switch the simultaneous control of the subject objects on and off by himself/herself.

The sixth invention is a program according to any one of the first to fifth inventions for causing the computer to function as an automatic stop means upon inappropriate change, in which the recording control means stops recording in response to a given instruction to stop recording, resumes recording in response to a given instruction to resume recording, detects that the measurement data meets a given inappropriate change condition indicating that an inappropriate change has occurred when photographing the virtual three-dimensional space, and issues the instruction to stop recording in response to the detection.

According to the sixth invention, recording can be stopped in response to a given instruction to stop recording, and can be resumed in response to a given instruction to resume recording. In addition, if the position or attitude of the computer used by the user changes in a way that is inappropriate for camerawork in video shooting, this can be automatically detected, and recording of camerawork data can be automatically stopped.

The seventh invention is a program according to any one of the first to sixth inventions for causing the computer to function as a playback image generating means for generating a playback image, which is an image of the virtual three-dimensional space from a virtual camera at a given playback timing of the camerawork data recorded by the recording control means, and a comparison display control means for controlling the display of a live image, which is an image generated by the virtual space image generating means, and the playback image in a comparative manner.

According to the seventh invention, it becomes possible to provide the user with a playback image at a given playback timing and a live image that can be compared based on already recorded camerawork data.

The eighth invention is a program according to any one of the first to sixth inventions for causing the computer to function as a comparison display control means for controlling the display of a playback image, which is an image of real space at a given playback timing of the camerawork data recorded by the recording control means, and a live image, which is an image of real space at the current time, in a comparative manner.

According to the eighth aspect of the invention, it is possible to display a comparison between an image in real space at the playback timing and an image in real space at the current time.

The ninth aspect of the present invention is a program according to the seventh or eighth aspect of the present invention for causing the computer to function as a playback timing manual designation means for designating the playback timing based on a user operation.

According to the ninth invention, the user can specify the playback timing.

The tenth invention is a program according to any one of the seventh to ninth inventions for causing the computer to function as a playback timing automatic designation means, in which the recording control means stops recording in response to a given instruction to stop recording, resumes recording in response to a given instruction to resume recording, and, when the recording stop instruction is given, sets the playback timing to a given playback timing based on the timing at which the recording stop instruction was given.

An eleventh aspect of the present invention is the program of the tenth aspect of the present invention, in which the playback timing automatic designation means sets the playback timing to a point on the time axis that is a given time from the timing when the recording stop instruction is given.

According to the tenth or eleventh invention, the playback timing can be automatically specified.

The twelfth invention is a program according to any one of the seventh to eleventh inventions, in which the comparative display control means controls the display of the live image and the playback image in a superimposed manner.

According to the twelfth invention, it becomes possible to display a live image and a playback image in a superimposed manner.

The thirteenth invention is a program according to any one of the seventh to twelfth inventions for causing the computer to function as a match determination means for determining the degree of match between the live image and the reproduced image and determining whether or not a predetermined match condition is met, and a match notification control means for notifying a user of the match when the match determination means determines that the match condition is met.

According to the thirteenth invention, it becomes possible to automatically determine whether a live image and a reproduced image match to such an extent that a predetermined matching condition is satisfied, and to notify the user of this fact.

The fourteenth invention is a program according to any one of the seventh to thirteenth inventions for causing the computer to function as a matching determination means for stopping recording in response to a given instruction to stop recording, restarting recording in response to a given instruction to restart recording, determining the degree of matching between the live image and the played-back image, and determining whether a predetermined matching condition is met, and an automatic recording start means for issuing an instruction to restart recording when the matching determination means determines that the matching condition is met.

According to the fourteenth invention, when the recording of camerawork data is stopped, if the result of the match determination satisfies the compatibility condition, it becomes possible to automatically start (resume) recording. Then, by appropriately setting the compatibility condition, it becomes possible to reproduce a series of visually connected camerawork by linking the newly started camerawork data to the playback timing of the recorded camerawork data.

The fifteenth invention is a program according to any one of the first to fourteenth inventions, in which the computer includes a shooting unit capable of changing the zoom magnification, the subject object is an AR (Augmented Reality) character based on an image captured by the shooting unit, the virtual camera control means controls the shooting angle of view of the virtual camera in conjunction with the zoom magnification, and the recording control means records data capable of reproducing the shooting angle of view of the virtual camera by including it in the camerawork data.

According to the fifteenth aspect of the invention, camerawork data can be recorded that captures an AR character using zoom.

The sixteenth invention is an electronic device comprising: a measurement data acquisition control means for controlling the acquisition of measurement data obtained by measuring a position and an orientation; a virtual space setting means for setting a virtual three-dimensional space in which a subject object is placed; a virtual camera control means for controlling the position and orientation of a virtual camera that captures the virtual three-dimensional space in conjunction with the measurement data; a virtual space image generation means for generating an image of the virtual three-dimensional space captured by the virtual camera; and a recording control means for recording camerawork data capable of reproducing the position and orientation of the virtual camera in the virtual three-dimensional space continuously controlled by the virtual camera control means.

According to the sixteenth invention, an electronic device that can achieve the same effect as the first invention can be realized.

A seventeenth aspect of the present invention is a data recording method executed by a portable computer, comprising:
a measurement data acquisition control step of controlling acquisition of measurement data obtained by measuring the position and attitude of the computer;
a virtual space setting step of setting a virtual three-dimensional space in which a subject object is placed;
a virtual camera control step of controlling a position and an attitude of a virtual camera that captures the virtual three-dimensional space in conjunction with the measurement data;
a virtual space image generating step of generating an image of the virtual three-dimensional space captured by the virtual camera;
a recording and control step of recording camerawork data capable of reproducing the position and attitude of the virtual camera in the virtual three-dimensional space continuously controlled by the virtual camera control step;
A data recording method comprising the steps of:

According to the seventeenth aspect, a data recording method can be realized that provides the same effect as the first aspect.

FIG. 1 is a diagram showing an example of the configuration of an augmented reality video generation system. FIG. 4 is a front view showing a configuration example of a user terminal. 1A and 1B are diagrams for explaining display control of a subject using augmented reality in a user terminal. 5A to 5C are diagrams for explaining a movement trajectory of camera work that can be acquired in this embodiment. FIG. 4 is a diagram showing an example of the data structure of camera work data. 10 is a diagram for explaining uploading of information and data required for generating an ambitious video to a server system. FIG. FIG. 11 is a diagram for explaining generation of video data of an augmented reality image by the server system. 11 is a diagram for explaining a manual pause of recording of camerawork data. FIG. 11 is a diagram for explaining automatic pausing of recording of camerawork data. FIG. FIG. 13 is a diagram for explaining resumption of recording of camerawork data that has been temporarily stopped. FIG. 13 is a diagram for explaining automatic time retrogression. FIG. 11 is a diagram for explaining restart of recording of camerawork data. FIG. 4 is a functional block diagram showing an example of a functional configuration of a user terminal. FIG. 4 is a diagram showing examples of programs and data stored in a terminal storage unit. 10 is a flowchart illustrating a process flow in a user terminal. 11 is a flowchart for explaining the flow of a recording interruption/resumption process. 17 is a flowchart continuing from FIG. 16 .

An example of an embodiment of the present invention will be described below, but it goes without saying that the forms to which the present invention can be applied are not limited to the following embodiment.

FIG. 1 is a diagram illustrating an example of the configuration of an augmented reality video generation system.
The augmented reality video generation system 1000 is a computer system that provides a service for creating augmented reality videos, and includes a server system 1100 and a user terminal 1500 that are connected to each other via a network 9 so that data communication is possible.

The network 9 refers to a communication path that allows data communication. In other words, the network 9 includes a dedicated line (dedicated cable) for direct connection, a LAN (Local Area Network) such as Ethernet (registered trademark), a telephone communication network, a cable network, the Internet, and other communication networks, and the communication method can be either wired or wireless.

The server system 1100 has, for example, a keyboard 1106, a touch panel 1108, and storage 1140, and the main unit is equipped with a control board 1150.

The control board 1150 is equipped with various microprocessors such as a CPU (Central Processing Unit) 1151, a GPU (Graphics Processing Unit), and a DSP (Digital Signal Processor), various IC memories 1152 such as a VRAM, RAM, and ROM, and a communication device 1153. Note that a part or the whole of the control board 1150 may be realized by an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a SoC (System on a Chip).

The server system 1100 then realizes the function of generating an augmented reality image (augmented reality image) based on camerawork data (described in detail below) created and recorded by the user terminal 1500, and providing the user with video data of the augmented reality image, by the control board 1150 performing calculations based on a specified program and data.

Note that while Figure 1 shows only one user terminal 1500, in actual system operation, multiple user terminals 1500 may be configured to access the server system 1100 at the same time.

Although the server system 1100 is depicted as if it were a single server device, it may be realized by a configuration of multiple devices. For example, the server system 1100 may be configured to include multiple blade servers each sharing a function, connected to each other via an internal bus so that data communication is possible. Furthermore, the location of the hardware that constitutes the server system 1100 does not matter. It may be configured so that multiple independent servers installed in remote locations communicate data via the network 9, and function as the server system 1100 as a whole.

The user terminal 1500 is a computer system used by the user 2 to acquire data for generating an augmented reality image, and is an electronic device that can access the server system 1100 via the network 9. The user terminal 1500 in this embodiment is a device known as a smartphone, but it may also be a handheld computer system such as a portable game device or a tablet computer.

FIG. 2 is a front view showing an example of the configuration of a user terminal 1500 in this embodiment.
The user terminal 1500 includes a directional input key 1502, a button switch 1504, a touch panel 1506 that functions as an image display device and a touch position input device, a speaker 1510, an internal battery 1509, a microphone 1512, a camera 1520, a control board 1550, and a memory card reader 1542 that can read and write data from a memory card 1540 that is a computer-readable storage medium. In addition, a power button, a volume control button, etc. that are not shown are provided. In addition, an IC card reader that can read and write data contactlessly from an IC card type credit card or prepaid card that can be used to pay for system usage may be provided.

Camera 1520 is an image sensor unit with a variable zoom magnification. That is, camera 1520 includes various components that enable digital video capture, such as an image sensor such as a CCD (Charge Coupled Device), a zoom mechanism, an autofocus mechanism, and an LSI that generates video data from a signal captured by the image sensor.

The control board 1550 is equipped with: 1) a CPU 1551 and various microprocessors such as a GPU and DSP; 2) various IC memories 1552 such as a VRAM, RAM and ROM; 3) a wireless communication module 1553 for wireless communication with mobile phone base stations and wireless LAN base stations connected to the network 9; 4) a positioning module 1555; 5) a three-axis acceleration sensor 1556; 6) a three-axis gyro 1557; 7) an interface circuit 1558; and the like.

The positioning module 1555 acquires the position of the user terminal 1500. In this embodiment, the position coordinates in the real space can be acquired using an existing positioning system. That is, the positioning module 1555 provides a positioning function in the user terminal 1500 by receiving a signal provided by the positioning system and outputting positioning information at a predetermined period (for example, every second). In this embodiment, the positioning module 1555 uses GPS (Global Positioning System) as the positioning system. Therefore, the positioning module 1555 can use an existing "GPS module" or "GPS receiver". The "positioning information" includes the positioning date and time (for example, UTC: Coordinated Universal Time), position coordinates (latitude, longitude, altitude), direction, etc. The positioning system to be used is not limited to GPS and can be appropriately selected. For example, the positioning module 1555 can be replaced with a configuration of a communication device that wirelessly connects to a wireless base station of a mobile phone or a wireless base station of a Wi-Fi network, and a direction sensor. That is, the location information that has been previously determined for the connected wireless base station may be acquired and used as the location coordinates of the user terminal 1500 in real space.

The interface circuit 1558 includes a driver circuit for the touch panel 1506, a circuit for receiving signals from the directional input keys 1502 and button switches 1504, an output amplifier circuit for outputting audio signals to the speaker 1510, an input signal generation circuit for generating audio signals collected by the microphone 1512, a circuit for inputting image data of an image captured by the camera 1520, and a signal input/output circuit for the memory card reader 1542.

These elements mounted on the control board 1550 are electrically connected via a bus circuit or the like, and are connected so that data can be read and written and signals can be sent and received. Note that part or all of the control board 1550 may be configured using an ASIC, FPGA, or SoC. The control board 1550 stores programs and various data for realizing the functions of the user terminal of this embodiment in the IC memory 1552.

In this embodiment, the user terminal 1500 is configured to download programs and various setting data from the server system 1100, but it may also be configured to read them from a storage medium such as a memory card 1540 obtained separately.

FIG. 3 is a diagram for explaining display control of a subject using augmented reality in the user terminal 1500. In FIG.
The user terminal 1500 can use augmented reality to display a subject on the touch panel 1506. That is, an AR marker 5 is placed in real space, and the user 2 adjusts the position and orientation of the user terminal 1500 and the zoom of the camera 1520 to capture the AR marker 5. The image of the AR marker 5 in the image captured by the camera 1520 is subjected to image recognition processing and used in calculation processing of the relative position and relative orientation between the user terminal 1500 and the AR marker 5. The relative position and relative orientation with respect to the AR marker 5 become measurement data of the position and orientation of the user terminal 1500.

The user terminal 1500 can constantly acquire the position and attitude changes of the camera 1520 from the calculated relative position and relative attitude, and the measurement data of the acceleration and attitude of the user terminal 1500 obtained using the three-axis acceleration sensor 1556 and three-axis gyro 1557. In addition, information on the shooting angle of view (also called the zoom magnification) can be periodically acquired by zooming the camera 1520. Hereinafter, parameter values that describe the so-called shooting conditions, such as the relative position, relative attitude, position change, attitude change, and shooting angle of view, as well as the size and orientation of the AR marker 5, attitude data, acceleration data, zoom control values, sensor measurement values, and the like that are the basis for calculating these, will be collectively referred to as the "actual shooting conditions".

While acquiring the actual shooting conditions, the user terminal 1500 can place the subject object 4 in a virtual three-dimensional space and control its movement. In this embodiment, the subject object 4 is a virtual character in the form of a human, and is an AR (Augmented Reality) character based on an image captured by the camera 1520. The user terminal 1500 controls the movement of the subject object 4 so that it dances while singing the music, using given music data prepared in advance and predetermined motion data for realizing motion synchronized with the music of the music data.

In the virtual three-dimensional space, a virtual camera C1 is placed, whose shooting position, shooting attitude, and shooting angle of view are controlled in conjunction with parameter values of the actual shooting conditions of the camera 1520. The user terminal 1500 generates an image of the virtual three-dimensional space captured by the virtual camera C1, i.e., a virtual space image 90, and displays this image superimposed on an image of real space (real space image 92) on the touch panel 1506. In other words, an augmented reality image (augmented reality image 94) is displayed.

After inputting a predetermined recording start operation into the user terminal 1500, the user 2 treats the subject object 4 displayed on the touch panel 1506 singing and dancing to music as a viewfinder image, and performs shooting operations by moving the user terminal 1500 in real space, sometimes changing its position, and sometimes changing the zoom magnification, in a manner similar to filming with a portable video camera. In other words, the user performs camerawork.

As shown in FIG. 4, the movement trajectory L1 caused by the shooting operation includes "wobble" that corresponds to camera shake that occurs when a person operates a portable video camera.

Then, when the user terminal 1500 detects input of a predetermined recording start operation by the user 2, it starts recording camerawork data 550 as shown in FIG. 5, for example.
The camerawork data 550 includes a unique shooting ID 551, a shooting start date and time 553, a shooting end date and time 555, actual shooting condition data 560, virtual shooting condition data 570, real image recording data 580 in which the real space image 92 captured by the camera 1520 is recorded, and virtual space image recording data 582 in which the virtual space image 90 is recorded. Of course, data other than these may be included as appropriate, or may be omitted as appropriate.

The actual shooting condition data 560 is, for example,
1) Actual camera attitude data 561 in which a relative attitude calculated based on the AR marker 5 and/or measurement data obtained by the three-axis gyro 1557 are stored in chronological order;
2) real camera acceleration data 563 in which acceleration calculated from a change in shooting position based on the AR marker 5 and/or measurement data by the three-axis acceleration sensor 1556 are stored in chronological order;
3) Actual zoom control data 565 in which the zoom magnification (photographing angle of view) of the camera 1520 or the control parameter value of the zoom magnification is stored in chronological order;
4) actual movement trajectory data 567 describing a movement trajectory L1 obtained by tracing the shooting position of the camera 1520 in time series based on the AR marker 5; and
including.

The virtual shooting condition data 570 is data about the shooting conditions in the virtual three-dimensional space of the virtual camera C1, and is a parameter value calculated for control in conjunction with the measurement data of the position and orientation of the user terminal 1500.

The virtual shooting condition data 570 is, for example,
1) Virtual camera attitude data 571 in which the shooting attitude of the virtual camera C1 obtained from the real camera attitude data 561 is stored in chronological order;
2) Virtual camera acceleration data 573 in which the acceleration occurring in the virtual camera C1 obtained from the real camera acceleration data 563 is stored in chronological order; and
3) Virtual camera shooting angle of view data 575 in which the shooting angle of view (or zoom magnification) of the virtual camera C1 calculated from the actual zoom control data 565 is stored in chronological order; and
4) virtual camera trajectory data 577 describing a trajectory Lc of a virtual camera C1 that reproduces the trajectory L1 in a virtual three-dimensional space based on the actual trajectory data 567;
including.

The time code, i.e., the time axis, of the actual shooting condition data 560 and the time code of the virtual shooting condition data 570 are recorded in synchronization with the time code of the music playback and the motion control of the subject object 4. In addition, either the actual shooting condition data 560 or the virtual shooting condition data 570 may be omitted, and the shooting condition data of the omitted side may be obtained from the remaining side as necessary.

When user 2 inputs a predetermined recording stop operation, user terminal 1500 stops recording camerawork data 550 and uploads the information and data required to generate the augmented reality video to server system 1100, as shown in FIG. 6. Specifically, for example, model data 510, motion data 512, music data 514, and camerawork data 550 of subject object 4 are uploaded. Of course, other data may also be uploaded as appropriate. For example, if a background object such as a stage set on which subject object 4 stands is prepared in the virtual three-dimensional space, data for realizing the display and operation, such as its model data and motion data, is also uploaded.

Note that if model data 510, motion data 512, and music data 514 that have been recorded in advance in server system 1100 are downloaded to user terminal 1500 in advance and used when recording camerawork data 550, uploading of these data can be omitted as appropriate. Also, it does not matter whether model data 510, motion data 512, and music data 514 have been created by user 2 or by others.

Figure 7 is a diagram for explaining the generation of augmented reality video by the server system 1100. When the model data 510 of the subject object 4, the motion data 512, the music data 514, and the camerawork data 550 are uploaded, the server system 1100 places the subject object 4 and background objects in a virtual three-dimensional space based on the data, and controls their movements in synchronization with the playback of the music. Furthermore, the server system 1100 controls the virtual camera C2 based on the uploaded camerawork data 550 in synchronization with the playback of the music.

Then, the server system 1100 generates an image (virtual reality image) of the virtual three-dimensional space captured by the virtual camera C2, and superimposes it on the real image recording data 580 included in the uploaded camerawork data 550 to create augmented reality video data (augmented reality video data 584).

The augmented reality videos created in this way reproduce the screen shaking caused by the "wobble" that is seen when a person operates a portable video camera, making them realistic augmented reality videos that "seem as if the user were holding the video camera in their hands and filming it."

In this embodiment, the recording of the camerawork data 550 can be temporarily stopped or resumed.
8 is a diagram for explaining a manual operation for pausing the recording of the camerawork data 550. The user terminal 1500 displays a shooting screen W8 on the touch panel 1506 while the camerawork data 550 is being recorded.

The shooting screen W8 includes: 1) a monitor display 10 that displays the image being shot; 2) a time code 11; 3) a status notification display 12 that shows the current shooting or recording status; 4) a start recording operation icon 13 for inputting a new start recording operation; 5) an interruption operation icon 14 for inputting a pause operation that temporarily stops the recording of the camerawork data 550; 6) a resume operation icon 15 for inputting a resume recording operation that is a start recording operation that restarts the recording of the camerawork data 550 that was paused; and 7) an end recording operation icon 16 for inputting an end recording operation that corresponds to the end of recording. The pause operation and the end recording operation can both be said to be operations that stop recording, and the start recording operation and the resume recording operation can both be said to be operations that start recording.

Then, user 2 can pause or resume recording of camerawork data 550 at any time by appropriately selecting interrupt operation icon 14 or resume operation icon 15 by touch operation.

Figure 9 is a diagram for explaining the automatic pausing of recording of camera work data 550. During recording of camera work data 550, user terminal 1500 constantly monitors the acceleration measured by triaxial acceleration sensor 1556 and the posture measured by triaxial gyro 1557. Then, when either the acceleration satisfies a predetermined inappropriate position change condition or the posture satisfies a predetermined inappropriate posture change condition, user terminal 1500 automatically pauses recording and notifies user 2 by displaying automatic stop notification 19 on touch panel 1506.

The “inappropriate position change condition” referred to here is a description, as a condition of acceleration change measured by the three-axis acceleration sensor 1556, of a position change of the camera 1520 that causes excessive screen shaking that goes beyond the limits of video expression.
Similarly, the "conditions for unsuitable posture change" referred to here are described as conditions for posture change of the camera 1520 that result in excessive screen shaking that goes beyond the limits of video presentation, as measured by the three-axis gyro 1557.

In other words, these conditions define a state in which the camera has been moved too suddenly. The user terminal 1500 performs automatic pause control based on these conditions. As a result, the recording of the camerawork data 550 can be paused simply by the user 2 intentionally performing an action that abruptly changes the position of the user terminal 1500 or by suddenly tipping the user terminal 1500, without the user 2 operating the interrupt operation icon 14, improving operability. Also, if an unexpected event occurs in which the user 2 accidentally drops the user terminal 1500 or trips while engrossed in filming, the automatic pause prevents unnecessary recording from continuing after the unexpected event.

Figure 10 is a diagram for explaining the resumption of recording of the paused camerawork data 550. User 2 can resume recording from any desired resumption timing of the previously recorded camerawork data 550.

After pausing, a seek bar 20 is additionally displayed on the shooting screen W10 to receive input for specifying the timing to resume. User 2 can specify the timing to resume by shifting the position of the slider on the seek bar 20.

When the restart timing is specified, the monitor display 10 displays a virtual space image 32 at the restart timing, which was taken at the restart timing corresponding to the slider position of the seek bar 20, superimposed on a live image 30 taken by the virtual camera C1 based on the current actual shooting conditions.

The virtual space image 32 at the restart timing is a playback image that is played back from the virtual space image recording data 582 (see FIG. 5) of the camerawork data 550 recorded most recently, with the specified restart timing regarded as the playback timing. Alternatively, the image at the restart timing may be regenerated from the motion data 512 and the virtual shooting condition data 570.

The "composite display" referred to here can be set as appropriate as long as the live image 30 and the virtual space image 32 at the time of restart can be displayed in a comparable manner on the shooting screen W10 after the pause. For example, the composite display may be a display in which one of the live image 30 and the virtual space image 32 at the time of restart is semi-transparently processed and superimposed on the other. Alternatively, a contour line extraction process may be performed from the virtual space image 32 at the time of restart, and the extracted contour line may be superimposed on the live image 30. Alternatively, the shooting screen W10 after the pause may be divided to display the live image 30 and the virtual space image 32 at the time of restart side by side. After displaying in a comparable manner, when recording is resumed, the display of the comparable manner may be canceled and only the live image 30 may be displayed. In addition, focusing on the material of the image to be composited, an augmented reality image is usually displayed on the live image 30 on the shooting screen, but only the composite display referred to here may be displayed in a comparable manner without superimposing it on the real space image. Similarly, for composite display only, the current real-space image and a real-space image at a past point in time that is set as the resume timing may be displayed for comparison.

The initial position of the restart timing is automatically specified by the user terminal 1500 and presented to the user when the seek bar 20 is displayed. For example, as shown in FIG. 11, the user terminal 1500 specifies and presents as the initial position of the restart timing a point in time obtained by going back in time on the time code (time axis) a predetermined time (e.g., one second) from the end timing of the most recently recorded camerawork data 550.

The specified time for "time retroactive" here is the time when it is assumed that the captured image will be too blurred to be unusable if an event occurs that satisfies the improper position change condition or the improper posture change condition. For example, it can be set to within a few seconds. Automatic time retroactively sets the initial position of the seek bar 20 to the assumed position just before the captured image becomes too blurred to be unusable, saving the user the trouble of searching for the right timing to resume.

FIG. 12 is a diagram for explaining resumption of recording of camera work data 550. In FIG.
After the restart timing is specified, the user terminal 1500 executes the process for restarting the recording of the camerawork data 550 according to the user's preference from two options, "immediate restart" and "match detection restart."

When a standard touch operation, which is a touch operation performed in a relatively short time, is performed on the resume operation icon 15, the user terminal 1500 executes an "immediate resume" process and immediately resumes recording of the camerawork data 550. Alternatively, the recording is resumed after a countdown of a predetermined number of seconds.

On the other hand, when a so-called "long press touch operation" is performed in which the resume operation icon 15 is touched and maintained in a touched state for a comparatively long time before being released, the user terminal 1500 executes a "match detection resume" process, compares the live image 30 with the virtual space image 32 at the time of the resume, and determines the degree of match. If the determined degree of match satisfies a predetermined match condition, a match notification 34 is displayed informing the user 2 that the two images are recognized as a match, and recording is resumed. The "match condition" referred to here can be set as appropriate, but it is preferable to extract the contours of each image and set the condition that the two match.

In addition, the recording resume process may be specified by the length of touch as described above, or an icon may be operated to determine which resume process to perform. Furthermore, the user terminal 1500 may automatically execute one of the resume processes, regardless of the selection operation of the user 2. For example, one of the resume processes may be determined in advance as a default, or if the position or orientation of the user terminal 1500 at the time of the pause and the current position or orientation of the user terminal 1500 satisfy a given approximation condition, the process may be executed as "immediate resume," and if the approximation condition is not satisfied, the process may be executed as "match detection resume."

In addition, even when an augmented reality image is displayed on the live image 30, the determination of whether the compatibility conditions are met may be made by comparing the virtual space images superimposed on the real space image to determine whether the compatibility conditions are met.

FIG. 13 is a functional block diagram showing an example of the functional configuration of a user terminal 1500 in this embodiment. The user terminal 1500 includes an operation input unit 100, a positioning unit 102, an acceleration measurement unit 106, an attitude measurement unit 108, an audio input unit 110, an image capture unit 120, a device processing unit 200, a sound output unit 390, an image display unit 392, a communication unit 394, and a device storage unit 500.

The operation input unit 100 outputs operation input signals corresponding to various operation inputs made by the player to the device processing unit 200. For example, this can be realized by a push switch, joystick, touch pad, track ball, acceleration sensor, gyro, CCD module, etc. Examples of this include the directional input keys 1502, button switches 1504, and touch panel 1506 in Figure 2.

The positioning unit 102 acquires the location information of the user terminal 1500 and outputs it to the terminal processing unit 200. This corresponds to the positioning module 1555 in Figure 2.

The acceleration measurement unit 106 measures the acceleration occurring in the user terminal 1500 and outputs it to the device processing unit 200. This corresponds to the three-axis acceleration sensor 1556 in Figure 2.

The posture measurement unit 108 measures the posture change of the user terminal 1500 and outputs it to the device processing unit 200. This corresponds to the three-axis gyro 1557 in Figure 2.

The audio input unit 110 collects the voice generated by the user (user voice) and environmental sounds, and outputs audio signals to the device processing unit 200. This corresponds to the microphone 1210 in FIG. 2.

The image capturing unit 120 captures the outside world, generates image data, and outputs the data to the device processing unit 200. This corresponds to the camera 1520 in FIG. 2.

The device processing unit 200 is realized by electronic components such as a microprocessor such as a CPU or a GPU, and an IC memory, and controls input and output of data between each functional unit including the operation input unit 100 and the terminal storage unit 500. Then, it executes various arithmetic processes based on predetermined programs and data, operation input signals from the operation input unit 100, and various data received from the server system 1100, to control the operation of the user terminal 1500. This corresponds to the control board 1550 in FIG. 2.
The device processing unit 200 in this embodiment has a camerawork record management unit 202 , a timing unit 280 , a sound generation unit 290 , an image generation unit 292 , and a communication control unit 294 .

The camerawork record management unit 202 executes various processes related to the recording of the camerawork data 550. Specifically, the camerawork record management unit 202 has a measurement data acquisition control unit 210, a virtual space setting unit 212, a subject control unit 214, a virtual camera control unit 216, a virtual space image generation control unit 218, an augmented reality image generation control unit 220, a sound output control unit 222, a recording control unit 224, an operation instruction control unit 226, an inappropriate change automatic stop control unit 228, a recording stop notification control unit 230, a playback image generation control unit 232, a comparison display control unit 234, a playback timing manual designation control unit 236, an automatic playback timing designation control unit 238, a match determination unit 240, a conformity notification control unit 242, and an automatic recording start control unit 244. Of course, other functional units may also be included as appropriate.

The measurement data acquisition control unit 210 controls the acquisition of measurement data obtained by measuring the position and orientation of the user terminal 1500. In this embodiment, the measurement data is acquired using the camera 1520 provided in the user terminal 1500. Specifically, control is performed using augmented reality technology. That is, the AR marker 5 is recognized from the image captured by the camera 1520, and the relative position and relative orientation are calculated to obtain the measurement data (see FIG. 3). This also applies to the control of determining the position and orientation, position change, orientation change, and movement trajectory L1 of the user terminal 1500 based on the measurement data of the positioning unit 102, acceleration measurement unit 106, and orientation measurement unit 108. If we focus on the data, this also applies to the control of generating the actual shooting condition data 560.

Note that the acquisition of measurement data is not limited to the case where it is realized by measurement using the camera 1520 or sensors equipped in the user terminal 1500. For example, it is also possible to configure a configuration in which measurement data of the position and orientation of the user terminal 1500 is acquired from an external device with which communication is possible. Specifically, a position and orientation measurement device separate from the user terminal 1500 may be prepared, and the device may measure the relative position and relative orientation between the user terminal 1500 and an object that serves as a reference for the position and orientation (such as the AR marker 5), and the measurement data may be transmitted to the user terminal 1500 one by one via data communication. In this case, the measurement method by the position and orientation measurement device may be appropriately set. For example, it may be a method of measuring the relative distance between each part by irradiating an infrared beam, or a method of capturing an image and analyzing the image to obtain the measurement.

The virtual space setting unit 212 sets a virtual three-dimensional space in which the subject object is placed. If a background object is set, this is also placed and set in the virtual three-dimensional space.
In this embodiment, the virtual space setting unit 212 places the subject object 4 and background objects in predetermined initial positions based on setting data for the virtual three-dimensional space prepared in advance by a system administrator or an operating company. Of course, the configuration may be such that the user 2 can set the initial positions and initial states of the objects to be placed in the virtual three-dimensional space. In that case, setting data created by another user may be used, not limited to setting data created by the user himself.

The subject control unit 214 controls the subject object to perform a given action along a given time axis. In this embodiment, the subject control unit 214 controls the action in synchronization with the playback of music. That is, the subject control unit 214 causes the subject object to perform an action in accordance with the sound (a given music and/or a given voice). In this case, the subject control unit 214 simultaneously controls to start the time axis in response to a recording start instruction to start the action of the subject object, and to stop the time axis in response to a recording stop instruction to stop the action of the subject object.

The simultaneous control can be switched on/off by a switching operation, and the subject control unit 214 performs simultaneous control when the simultaneous control is on, and continues to control the operation of the subject object when the simultaneous control is off regardless of the recording start command or recording stop command.

The virtual camera control unit 216 controls the position and orientation of the virtual camera that captures the virtual three-dimensional space in conjunction with the measurement data of the position and orientation. Focusing on the data, the generation control of the virtual shooting condition data 570 also falls under this category. During the linked control, the virtual camera control unit 216 controls the shooting angle of view of the virtual camera C1 in conjunction with the zoom magnification of the shooting unit 120. When the virtual camera control unit 216 resumes recording from a pause, the zoom magnification at the time of resumption (i.e. the zoom magnification in the live image) may be automatically changed to match the zoom magnification at the time of resumption. In this case, it is preferable because the user 2 can concentrate on adjusting the position and orientation of the user terminal 1500 to the state at the time of resumption when resuming recording.

The virtual space image generation control unit 218 is one of the virtual space image generation means that generates an image of a virtual three-dimensional space captured by the virtual camera C1, and controls the generation.

The augmented reality image generation control unit 220 controls the generation of an augmented reality image. In this embodiment, this corresponds to the generation control of the image displayed on the monitor display 10 of the shooting screen W8.

The sound output control unit 222 controls the output of a given piece of music and/or a given sound along a time axis. In this embodiment, this corresponds to the control of the playback of a piece of music based on the music data 514.

The recording control unit 224 performs recording control to record camerawork data 550 (see FIG. 5) that can reproduce the position and attitude of the virtual camera C1 in the virtual three-dimensional space continuously controlled by the virtual camera control unit 216.

Specifically, the recording control unit 224 controls the recording of the camerawork data 550 by recording the position and orientation of the virtual camera C1 in association with the time axis, so that camerawork synchronized with the time axis can be reproduced.

Furthermore, the recording control unit 224 starts recording in response to a given instruction to start recording, stops recording in response to a given instruction to stop recording, and resumes recording in response to a given instruction to resume recording.
Furthermore, the recording control unit 224 records data that can reproduce the shooting angle of view of the virtual camera by including it in the camerawork data.

The operation instruction control unit 226 issues a recording start instruction by a specified recording start operation, and issues a recording stop instruction by a specified recording stop operation.

The inappropriate change automatic stop control unit 228 detects when the position and orientation measurement data meets a given inappropriate change condition that indicates that an inappropriate change has occurred when capturing an image of the virtual three-dimensional space, and issues an instruction to stop recording in response to this detection.

The recording stop notification control unit 230 controls the notification to the user that recording has been stopped by the inappropriate change automatic stop control unit 228. In this embodiment, this corresponds to the display control of the automatic stop notification 19 (see FIG. 9).

The playback image generation control unit 232 performs control to generate a playback image, which is an image of the virtual three-dimensional space from the virtual camera at a given playback timing of the camerawork, based on the camerawork data 550 recorded by the recording control unit 224. In this embodiment, the playback timing is specified as the resume timing. This corresponds to reading out the data of the virtual space image 32 at the resume timing from the virtual space image recording data 582, or reproducing the movement of the subject object 4 and the camerawork of the virtual camera C1 at the playback timing.

The comparative display control unit 234 controls the display of the live image at that time and the playback image at the playback timing in a comparative manner (see Fig. 10). Specifically, the comparative display control unit 234 controls the display of the live image of the virtual space being shot at that time and the playback image in a superimposed manner.
In addition, in a configuration in which a real space image rather than a virtual space image is used as a comparison target as a live image, the comparison display control unit 234 displays the real space image being captured at that time and the real space image at the playback timing in a superimposed manner.

The playback timing manual designation control unit 236 controls the designation of playback timing based on user operations.

When a recording stop instruction is given, the playback timing automatic designation control unit 238 sets the playback timing to a given playback timing based on the timing at which the recording stop instruction was given. Specifically, the playback timing automatic designation control unit 238 sets the playback timing to a point going back on the time axis by a given amount of time from the timing at which the recording stop instruction was given (effectively, the end timing of the camerawork data 550 whose recording was stopped by the recording stop instruction).

The match determination unit 240 determines the degree of match between the live image and the playback image and determines whether or not a specified match condition is met.

The conformance notification control unit 242 controls the display of a conformance notification to the user when the match determination unit 240 determines that the conformance conditions are met. In this embodiment, this corresponds to the display control of the conformance notification 34 (see FIG. 12).

The automatic recording start control unit 244 issues a recording start instruction when the match determination unit 240 determines that the compatibility conditions are met.

The timing unit 280 uses the system clock to measure the current date and time, time limits, etc.

The sound generation unit 290 is realized by, for example, a digital signal processor (DSP), a processor such as a voice synthesis IC, an audio codec capable of playing audio files, etc., and generates sound signals for game-related sound effects, background music, and various operation sounds, and outputs them to the sound output unit 390.

The sound output unit 390 is realized by a device that outputs sound effects, background music, etc. based on the sound signal input from the sound generation unit 290. This corresponds to the speaker 1510 in FIG. 2.

The image generation unit 292 controls the generation of various image data and the generation and output of image signals for displaying those images on the image display unit 392. In this embodiment, it generates virtual space images and augmented reality images.

The image display unit 392 displays various images based on the image signal input from the image generation unit 292. For example, this can be realized by an image display device such as a flat panel display, a projector, or a head-mounted display. In this embodiment, this corresponds to the touch panel 1506 in FIG. 2.

The communication control unit 294 executes data processing related to data communication and realizes data exchange with external devices via the communication unit 394.

The communication unit 394 connects to the network 9 to realize communication. For example, this is realized by a wireless communication device, a modem, a TA (terminal adapter), a jack for a wired communication cable, a control circuit, etc., and corresponds to the wireless communication module 1553 in FIG. 2.

The terminal storage unit 500 stores various data and programs for implementing various functions that allow the terminal processing unit 200 to comprehensively control the user terminal 1500. It is also used as a working area for the terminal processing unit 200, and temporarily stores the results of calculations executed by the terminal processing unit 200 according to various programs and input data input from the operation input unit 100. These functions are implemented, for example, by IC memory such as RAM or ROM, magnetic disks such as hard disks, and optical disks such as CD-ROMs and DVDs. This corresponds to the IC memory 1552 and memory card 1540 mounted on the control board 1550 in Figure 2. A configuration using online storage is also possible.

FIG. 14 is a diagram showing an example of programs and data stored in the terminal storage unit 500. As shown in FIG.
The terminal storage unit 500 stores a camerawork record management program 502, model data 510, motion data 512, music data 514, camerawork data 550 (see FIG. 5 ), and a current date and time 800. Of course, other programs and data can also be stored as appropriate.

The camerawork record management program 502 realizes the functions of the camerawork record management unit 202 in the device processing unit 200.

Next, the operation of the augmented reality video generation system 1000 of this embodiment will be described.
15 is a flowchart for explaining the flow of processing in the user terminal 1500. The flow of processing explained here is implemented by the user terminal 1500 by executing the camerawork record management program 502 in the device processing unit 200.

The user terminal 1500 first starts measuring the position and orientation of the user terminal 1500 (position and orientation measurement) and capturing images with the camera 1520 (step S10), displays the capture screen W8 (see FIG. 8) on the touch panel 1506, and starts displaying an augmented reality image (step S12).

Next, when the user terminal 1500 detects a touch operation on the record start operation icon 13 (see FIG. 8), it determines that a new record start operation (record start instruction) has been input (step S14), and starts new recording of the camerawork data 550 (step S16). Then, until a touch operation input on the record stop operation icon 16 is detected, that is, until a record stop operation input is detected (NO in step S18), the user terminal 1500 periodically executes the recording interruption/resume process (step S20).

16 and 17 are flowcharts for explaining the flow of the recording interruption and resumption process.
As shown in FIG. 16, in this process, if the measurement results of the position or posture of the user terminal 1500 satisfy an inappropriate position change condition (YES in step S30), if the measurement results satisfy an inappropriate posture change condition (YES in step S32), or if an input of an interrupt operation is detected (YES in step S34), the user terminal 1500 determines that an instruction to stop recording has been made and temporarily suspends recording of the camerawork data 550 (step S40).

The user terminal 1500 further pauses the playback of the music and the movement control of the subject object 4 (step S42), and notifies the user of the pause by controlling a display to that effect (step S44). However, the ON/OFF setting of the process of pausing the movement control of the subject object 4 accompanying the stopping of recording of the camerawork data 550 is subject to an ON/OFF setting operation performed in advance by the user, and the result of the ON/OFF setting operation is stored in the terminal storage unit 500. If this is ON, the user terminal 1500 executes steps S42 and S44, and if it is OFF, it skips.

Next, the user terminal 1500 automatically selects and automatically specifies the initial value of the resume timing (step S46). If the pause is due to an interrupt operation input by the user, the resume timing is set to the end timing of the paused camerawork data 550. If the pause is due to a condition for an inappropriate change in position or a condition for an inappropriate change in posture being satisfied, automatic time retrogression is performed to automatically select the resume timing (see FIG. 11). Of course, regardless of whether the reason for the pause is a user operation input or the detection of an inappropriate change, automatic time retrogression may be performed in all cases, and a configuration is also possible in which the end timing is set as the resume timing in all cases.

Next, the user terminal 1500 starts a simultaneous display on the shooting screen W10 after the pause (see FIG. 10) that allows comparison of the virtual space image 32 at the time of the restart and the live image 30 (step S48), and starts accepting input of a user-specified operation for the restart timing by displaying the seek bar 20 (step S50).

Moving on to FIG. 17, if input of a restart timing designation operation is detected (YES in step S60), the user terminal 1500 replaces the virtual space image 32 at the restart timing with an image at the time of the designated restart timing (step S62). Then, the playback position of the music and the playback position of the subject object 4 are aligned with the time of the designated restart timing (step S64).

Next, when a short, single touch on the resume operation icon 15 is detected, the user terminal 1500 determines that an immediate resume operation has been input (recording resume instruction) (YES in step S70; see FIG. 12), and resumes recording of the camerawork data 550 (step S82), and resumes playing the music and controlling the movement of the subject object 4 (step S84). Note that if the resume timing is not the end timing of the paused camerawork data 550, the data from the specified resume timing to the end timing is overwritten with the resumed recording data.

Next, the user terminal 1500 performs control to notify the user that recording is to be resumed (step S86), and ends the recording interruption/resume process.

On the other hand, when a long touch on the resume operation icon 15 is detected, the user terminal 1500 determines that an automatic resume operation has been input (recording resume instruction) (YES in step S72; see FIG. 12), and starts judging the degree of match between the virtual space image 32 and the live image 30 at the time of the resume, and checking against the compatibility conditions (step S74). Furthermore, if the degree of match satisfies the compatibility conditions, the images are deemed to match, a compatibility notification 34 is displayed, and recording is automatically resumed, judging that the automatic resume conditions based on the recording resume instruction have been met (step S82). In other words, the user terminal 1500 itself gives a recording resume instruction when it is determined that the compatibility conditions for match judgment are met.

After automatic restart, the user terminal 1500 restarts the playback of the music and the control of the movement of the subject object 4 (step S84), in the same way as in the case of immediate restart, and notifies the user that recording has been restarted (step S86), and then ends the recording interruption and restart process.

If an input to stop recording is detected (YES in step S90), the user terminal 1500 can end the recording interruption and resumption process even if recording remains paused, and can also end recording of the camerawork data 550.

Returning to FIG. 15, when input of a specified recording stop operation is detected (YES in step S18), the user terminal 1500 ends recording of the camerawork data 550 (step S100), ends playback of the music and control of the subject object 4 (step S102), and ends the series of processes.

The user terminal 1500 executes uploading of the recorded camerawork data 550, etc. to the server system 1100 when it detects input of a specified upload operation (see FIG. 6). Then, when it detects input of an operation to request generation of a specified augmented reality image, the user terminal 1500 transmits the specified generation request to the server system 1100.

When the server system 1100 receives a generation request, it creates video data of the augmented reality image using the camerawork data 550, etc., uploaded from the user terminal 1500 that sent the request, and makes this available to the user terminal 1500 (see Figure 7).

In this way, this embodiment makes it easy to create augmented reality videos that have the same realism as if the user were "taking the video with a handheld video camera."

[Modifications]
Although the embodiment to which the present invention is applied has been described above, the forms to which the present invention can be applied are not limited to the above-described forms, and constituent elements can be added, omitted, or modified as appropriate.

[Modification 1]
For example, in the above embodiment, the number of subject objects 4 is illustrated as one, but a plurality of objects may be used at the same time.

[Modification 2]
In addition, although the above embodiment shows an example of creating a video of an augmented reality (AR) image, the above embodiment is similar to the case of creating a video of a virtual reality (VR) image, and "augmented reality" in the explanation of the above embodiment may be appropriately replaced with "virtual reality."

[Modification 3]
In addition, in the above embodiment, a configuration in which the AR marker 5 is used to measure the position and orientation of the user terminal 1500 is illustrated, but the position and orientation may be measured using a markerless augmented reality technology.

[Modification 4]
In the above embodiment, an example was shown in which a virtual character having a human form was photographed singing and dancing to music, but the subject object 4 may be an animal, a plant, a landscape, a vehicle, etc. Depending on the subject object, it may be possible to omit playing music or omit control of actions synchronized with music as appropriate.

[Modification 5]
In the above embodiment, the resumption of recording of the camerawork data 550 targets the most recently recorded camerawork data 550, but this is not limited to the above. For example, steps S60 to S82 in the above embodiment can be configured to be separately executable as a designated recording resume process, and the recorded camerawork data 550 can be presented to the user in a selectable manner, and the designated recording resume process can be executed for the selected camerawork data 550.

[Modification 6]
In addition, in the above embodiment, an example was shown in which the generation of the video of the augmented reality image was performed by the server system 1100, but a configuration in which this is performed by the user terminal 1500 is also possible.

[Modification 7]
In the above embodiment, the operation icons displayed on the shooting screen can be selected according to the situation as appropriate (see Figs. 8, 10, 11, and 12). For example, depending on the execution status of the recording process, only icons that the user can operate at that time may be displayed. Specifically, during recording, only the interrupt operation icon 14 and the recording end icon 16 are displayed, and during pause, the resume operation icon 15 is displayed instead of the interrupt operation icon 14.

In addition, various operation inputs may be accepted without using operation icons on the shooting screen. For example, various operations may be instructed by input using gestures that use touch trajectories on a touch panel, or by operation input using physical buttons on the user terminal 1500, or by touching the screen to instruct pausing (touching the screen while recording to instruct pausing) and resuming recording (touching the screen while paused to instruct resuming).

[Variation 8]
Although the example of automatically stopping recording has been described as a case where a sudden change in the position or attitude of the user terminal 1500 is detected, it is also possible to configure the recording to be automatically stopped as appropriate in other cases.
For example, recording may be paused when there is a notification from the system of the user terminal 1500 (such as a power drop, a notification from another application, or an incoming call). Also, recording may be paused when the user terminal 1500 transitions to a sleep mode based on an operation input by the user 2. Also, recording may be paused when the user terminal 1500 receives an operation to switch to another application.

[Modification 9]
In addition, in the above embodiment, the restart timing is set by receiving the operation input for designating the restart timing through the seek bar 20, but the present invention is not limited to this.
For example, the user terminal 1500 may automatically select any of a plurality of resume timing candidates, display images corresponding to the resume timing candidates as thumbnail images, and allow the user to select one. In this case, if the user terminal 1500 detects an inappropriate change in the position or posture of the user terminal 1500 and automatically stops, it is preferable to include timings going back a predetermined time from the time of the automatic stop as resume timing candidates. In addition, going back from the time of the automatic stop, timings at which the position change or posture change of the user terminal 1500 is stable in terms of data (parameters such as acceleration are within a predetermined range) may be set as playback timing candidates. Furthermore, the user terminal 1500 may automatically determine the resume timing without accepting manual selection of the resume timing by the user 2.

2...User 4...Subject object 5...AR marker 11...Time code 13...Record start operation icon 14...Interruption operation icon 15...Resume operation icon 16...Record end operation icon 19...Automatic stop notification 20...Seek bar 30...Live image 32...Virtual space image at time of resume timing 34...Compatibility notification 100...Operation input unit 106...Acceleration measurement unit 108...Posture measurement unit 120...Photographing unit 200...Device processing unit 202...Camerawork record management unit 210...Measurement data acquisition control unit 212...Virtual space setting unit 214...Subject control unit 216...Virtual camera control unit 218...Virtual space image generation control unit 220...Augmented reality image generation control unit 222...Sound output control unit 224...Recording control unit 226...Operation instruction control unit 228...Automatic stop control unit at inappropriate change 230...Recording stop notification control unit 232...Playback image generation control unit 234...Comparative display control unit 236...Manual playback timing designation control unit 238...Automatic playback timing designation control unit 240...Match determination unit 242...Compliance notification control unit 244...Automatic recording start control unit 292...Image generation unit 500...Terminal memory unit 502...Camerawork record management program 550...Camerawork data 560...Actual shooting condition data 567...Actual movement trajectory data 570...Virtual shooting condition data 577...Virtual camera movement trajectory data 580...Real image recording data 582...Virtual space image recording data 584...Augmented reality video data 800...Current date and time 1000...Augmented reality video generation system 1100...Server system 1500...User terminal 1520...Camera 1550...Control board 1556...3-axis acceleration sensor 1557...3-axis gyro C1...Virtual camera

Claims (21)

A program for controlling a portable computer having a camera and a display unit to display on the display unit an image of a virtual three-dimensional space captured by a virtual camera that is controlled in conjunction with the shooting direction of the camera,
a virtual space setting means for setting the virtual three-dimensional space in which a subject object is placed ;
a subject control means for controlling the subject object to perform a given action along a given time axis;
a virtual space image generating means for generating an image of the virtual three-dimensional space captured by the virtual camera;
a recording control means for recording camerawork data capable of reproducing a position and an attitude of the virtual camera in the virtual three-dimensional space during the interlocking control ;
and causing the computer to function as
the recording control means starts recording the camerawork data in synchronization with the time axis in response to a given recording start instruction, and stops recording in response to a given recording stop instruction;
the subject control means performs simultaneous control to start the operation of the time axis in response to the recording start instruction to start the motion of the subject object, and to stop the operation of the time axis in response to the recording stop instruction to stop the motion of the subject object.
program. a sound output control means for controlling the output of a given piece of music and/or a given sound (hereinafter collectively referred to as "sound") along the time axis;
and causing the computer to function as
The subject control means causes the subject object to perform an action in accordance with the sound.
The program according to claim 1 . The simultaneous control can be switched on/off by a switching operation,
the object control means performs the simultaneous control when the simultaneous control is on, and continues to control the motion of the object when the simultaneous control is off regardless of the recording start instruction and the recording stop instruction;
The program according to claim 1 or 2 . A program for controlling a portable computer having a camera and a display unit to display on the display unit an image of a virtual three-dimensional space captured by a virtual camera that is controlled in conjunction with the shooting direction of the camera,
a virtual space setting means for setting the virtual three-dimensional space in which a subject object is placed ;
a virtual space image generating means for generating an image of the virtual three-dimensional space captured by the virtual camera;
a recording control means for recording camerawork data capable of reproducing a position and an attitude of the virtual camera in the virtual three-dimensional space in the interlocking control , the recording control means stopping recording in response to a given recording stop instruction and resuming recording in response to a given recording resume instruction;
an automatic stop means for detecting that a given inappropriate change condition based on the shooting direction, which indicates that an inappropriate change has occurred during shooting of the virtual three-dimensional space by the interlocking control, has been satisfied, and giving the recording stop instruction in response to the detection;
A program for causing the computer to function as a A program for controlling a portable computer having a camera and a display unit to display on the display unit an image of a virtual three-dimensional space captured by a virtual camera that is controlled in conjunction with the shooting direction of the camera,
a virtual space setting means for setting the virtual three-dimensional space in which a subject object is placed ;
a virtual space image generating means for generating an image of the virtual three-dimensional space captured by the virtual camera;
a recording control means for recording camerawork data capable of reproducing a position and an attitude of the virtual camera in the virtual three-dimensional space during the interlocking control ;
a playback image generating means for generating a playback image which is an image of the virtual three-dimensional space from the virtual camera at a given playback timing of the camerawork data recorded by the recording control means;
a comparison display control means for controlling display of a live image, which is an image generated by the virtual space image generating means, and the reproduced image in a predetermined display mode for comparison;
A program for causing the computer to function as a A program for controlling a portable computer having a camera and a display unit to display on the display unit an image of a virtual three-dimensional space captured by a virtual camera that is controlled in conjunction with the shooting direction of the camera,
a virtual space setting means for setting the virtual three-dimensional space in which a subject object is placed ;
a virtual space image generating means for generating an image of the virtual three-dimensional space captured by the virtual camera;
a recording control means for recording camerawork data capable of reproducing a position and an attitude of the virtual camera in the virtual three-dimensional space during the interlocking control ;
a comparison display control means for controlling displaying, in a predetermined display mode for comparing a playback image, which is an image of real space at a given playback timing of the camerawork data recorded by the recording control means, and a live image, which is an image of real space at the current time;
A program for causing the computer to function as a a reproduction timing manual designation means for designating the reproduction timing based on a user's operation;
7. The program according to claim 5 or 6 , for causing the computer to function as a the recording control means stops recording in response to a given recording stop instruction and resumes recording in response to a given recording resume instruction;
a playback timing automatic designation means for, when the recording stop instruction is given, setting a given playback timing based on the timing at which the recording stop instruction is given as the playback timing;
The program according to any one of claims 5 to 7 , for causing the computer to function as the automatic playback timing designation means designates a time point preceding the timing at which the recording stop instruction was given as the playback timing;
The program according to claim 8 . the comparative display control means performs control to display the live image and the reproduced image in a superimposed manner;
The program according to any one of claims 5 to 9 . a matching determination means for determining a degree of matching between the live image and the reproduced image and determining whether or not a predetermined matching condition is satisfied;
a conformity notification control means for notifying a user of conformity when the conformity determination means determines that the conformity condition is satisfied;
The program according to any one of claims 5 to 10, for causing the computer to function as a the recording control means stops recording in response to a given recording stop instruction and resumes recording in response to a given recording resume instruction;
a matching determination means for determining a degree of matching between the live image and the reproduced image and determining whether or not a predetermined matching condition is satisfied;
an automatic recording start means for giving the recording restart instruction when the matching determination means determines that the matching condition is satisfied;
The program according to any one of claims 5 to 11, for causing the computer to function as The camera has a variable zoom ratio ,
the subject object is an AR (Augmented Reality) character based on an image captured by the camera ,
the linked control includes controlling a shooting angle of view of the virtual camera in linkage with the zoom magnification;
The recording control means records data capable of reproducing a shooting angle of view of the virtual camera in the camerawork data.
The program according to any one of claims 1 to 12 . A portable electronic device that includes a camera and a display unit, and controls the display unit to display an image of a virtual three-dimensional space captured by a virtual camera that is controlled in conjunction with the shooting direction of the camera,
a virtual space setting means for setting the virtual three-dimensional space in which a subject object is arranged ;
a subject control means for controlling the subject object to perform a given action along a given time axis;
a virtual space image generating means for generating an image of the virtual three-dimensional space captured by the virtual camera;
a recording control means for recording camerawork data capable of reproducing a position and an attitude of the virtual camera in the virtual three-dimensional space during the interlocking control ;
Equipped with
the recording control means starts recording the camerawork data in synchronization with the time axis in response to a given recording start instruction, and stops recording in response to a given recording stop instruction;
the subject control means performs simultaneous control to start the operation of the time axis in response to the recording start instruction to start the motion of the subject object, and to stop the operation of the time axis in response to the recording stop instruction to stop the motion of the subject object.
Electronics. A portable electronic device that includes a camera and a display unit, and controls the display unit to display an image of a virtual three-dimensional space captured by a virtual camera that is controlled in conjunction with the shooting direction of the camera,
a virtual space setting means for setting the virtual three-dimensional space in which a subject object is arranged ;
a virtual space image generating means for generating an image of the virtual three-dimensional space captured by the virtual camera;
a recording control means for recording camerawork data capable of reproducing a position and an attitude of the virtual camera in the virtual three-dimensional space in the interlocking control , the recording control means stopping recording in response to a given recording stop instruction and resuming recording in response to a given recording resume instruction;
an automatic stop means for detecting that a given inappropriate change condition based on the shooting direction, which indicates that an inappropriate change has occurred during shooting of the virtual three-dimensional space by the interlocking control, has been met, and giving the recording stop instruction in response to the detection;
Electronic equipment equipped with A portable electronic device that includes a camera and a display unit, and controls the display unit to display an image of a virtual three-dimensional space captured by a virtual camera that is controlled in conjunction with the shooting direction of the camera,
a virtual space setting means for setting the virtual three-dimensional space in which a subject object is arranged;
a virtual space image generating means for generating an image of the virtual three-dimensional space captured by the virtual camera;
a recording control means for recording camerawork data capable of reproducing a position and an attitude of the virtual camera in the virtual three-dimensional space during the interlocking control ;
a playback image generating means for generating a playback image which is an image of the virtual three-dimensional space from the virtual camera at a given playback timing of the camerawork data recorded by the recording control means;
a comparison display control means for controlling display of a live image, which is an image generated by the virtual space image generating means, and the reproduced image in a predetermined display mode for comparison;
Electronic equipment equipped with A portable electronic device that includes a camera and a display unit, and controls the display unit to display an image of a virtual three-dimensional space captured by a virtual camera that is controlled in conjunction with the shooting direction of the camera,
a virtual space setting means for setting the virtual three-dimensional space in which a subject object is arranged;
a virtual space image generating means for generating an image of the virtual three-dimensional space captured by the virtual camera;
a recording control means for recording camerawork data capable of reproducing a position and an attitude of the virtual camera in the virtual three-dimensional space during the interlocking control ;
a comparison display control means for controlling displaying, in a predetermined display mode for comparing a playback image, which is an image of real space at a given playback timing of the camerawork data recorded by the recording control means, and a live image, which is an image of real space at the current time;
Electronic equipment equipped with A data recording method in which a portable computer equipped with a camera and a display unit controls the display unit to display an image of a virtual three-dimensional space captured by a virtual camera that is controlled in conjunction with the shooting direction of the camera, and records camera work data ,
a virtual space setting step of setting the virtual three-dimensional space in which a subject object is placed ;
a subject control step of controlling the subject object to perform a given action along a given time axis;
a virtual space image generating step of generating an image of the virtual three-dimensional space captured by the virtual camera;
a recording control step of recording camerawork data capable of reproducing a position and an attitude of the virtual camera in the virtual three-dimensional space in the interlocking control ;
Including,
the recording control step includes starting recording of the camerawork data in synchronization with the time axis in response to a given recording start instruction and stopping recording in response to a given recording stop instruction;
the subject control step includes performing simultaneous control to start the operation of the time axis in response to the recording start instruction to start the motion of the subject object, and to stop the operation of the time axis in response to the recording stop instruction to stop the motion of the subject object.
Data recording methods. A data recording method in which a portable computer equipped with a camera and a display unit controls the display unit to display an image of a virtual three-dimensional space captured by a virtual camera that is controlled in conjunction with the shooting direction of the camera, and records camera work data ,
a virtual space setting step of setting the virtual three-dimensional space in which a subject object is placed ;
a virtual space image generating step of generating an image of the virtual three-dimensional space captured by the virtual camera;
a recording control step of recording camerawork data capable of reproducing a position and an attitude of the virtual camera in the virtual three-dimensional space in the interlocking control , the recording control step being for stopping the recording in response to a given recording stop instruction and for resuming the recording in response to a given recording resume instruction;
An automatic stop step at the time of inappropriate change, which detects that a given inappropriate change condition based on the shooting direction indicating that an inappropriate change has occurred during shooting of the virtual three-dimensional space by the interlocking control is satisfied, and gives the recording stop instruction in response to the detection;
A data recording method comprising: A data recording method in which a portable computer equipped with a camera and a display unit controls the display unit to display an image of a virtual three-dimensional space captured by a virtual camera that is controlled in conjunction with the shooting direction of the camera, and records camera work data ,
a virtual space setting step of setting the virtual three-dimensional space in which a subject object is placed ;
a virtual space image generating step of generating an image of the virtual three-dimensional space captured by the virtual camera;
a recording control step of recording camerawork data capable of reproducing a position and an attitude of the virtual camera in the virtual three-dimensional space in the interlocking control ;
a playback image generating step of generating a playback image which is an image of the virtual three-dimensional space from the virtual camera at a given playback timing of the camerawork data recorded in the recording control step;
a comparison display control step of controlling display of a live image, which is an image generated in the virtual space image generating step, and the reproduced image in a predetermined display mode for comparison;
A data recording method comprising: A data recording method in which a portable computer equipped with a camera and a display unit controls the display unit to display an image of a virtual three-dimensional space captured by a virtual camera that is controlled in conjunction with the shooting direction of the camera, and records camera work data ,
a virtual space setting step of setting the virtual three-dimensional space in which a subject object is placed ;
a virtual space image generating step of generating an image of the virtual three-dimensional space captured by the virtual camera;
a recording control step of recording camerawork data capable of reproducing a position and an attitude of the virtual camera in the virtual three-dimensional space in the interlocking control ;
a comparison display control step of controlling displaying, in a predetermined display mode for comparing a playback image, which is an image of real space at a given playback timing of the camerawork data recorded in the recording control step, and a live image, which is an image of real space at the current time;
A data recording method comprising:

Images