JP2020119335A - Program, camera work data generation method, and electronic apparatus - Google Patents

Abstract

To provide a new technique for creating augmented reality or virtual reality moving images with reality that gives an impression as if users themselves held a video camera in a hand and shot it.SOLUTION: A user terminal 1500 controls the position and attitude of a virtual camera C1 that photographs a virtual three-dimensional space in conjunction with measurement data of the position and attitude, and displays a virtual space image as a finder image. On the other hand, it records camera work data capable of reproducing the position and attitude of the virtual camera. When recording, the user terminal records camera work data after a recording resumption instruction as camera work data that continues after interruption timing of the camera work data before stop by adjusting the position and attitude of the virtual camera when starting recording in response to the recording resumption instruction to the position and attitude of the virtual camera at the given interruption timing of camera work data before stop by a recording stop instruction.SELECTED DRAWING: Figure 10

Description

The present invention relates to a program and the like for causing a computer to control a virtual camera that captures a virtual three-dimensional space.

Augmented Reality (AR) technology is used in various fields. For example, in Patent Document 1, the position of the user terminal in the physical space and the position of the virtual camera in the virtual three-dimensional space are associated with each other, and an object (for example, a game) arranged in the virtual three-dimensional space by the virtual camera is associated. The technique of superimposing a captured image of a character that appears on the camera with the image of the physical space captured by the camera installed in the user terminal, and recording the superimposed image based on the user's operation input Techniques that can be done are disclosed.

JP 2012-69126 A

By using the technique of Patent Document 1, the user can enjoy making a still image as if a virtual character exists in the real space. However, the user could not obtain an augmented reality moving image as if he/she held the video camera in his hand.

By using 3DCG (3-Dimensional Computer Graphics) software using a high-performance personal computer, it is possible to create an augmented reality moving image. In that case, however, the resulting augmented reality video camera work is very smooth, as if the video professional was shooting with a high-performance gimbal. There is no real camera shake that occurs when the user himself holds the video camera and shoots. Therefore, the handmade feeling of "made by the user" and the reality of "photographed by the user" were thin, and I couldn't wipe out the feeling of something made of "CG software". From the perspective of a professional filmmaker, the same thing must be done to create an augmented reality movie that pursues the reality that gives the impression that the user himself/herself has taken a video camera in order to produce a video expression. In the case, if the camerawork is too smooth without camera shake, the required performance cannot be realized.

Such a problem is not limited to the creation of a moving image of augmented reality images, and can be similarly applied to the creation of a virtual reality (VR) moving image.

The present invention has been devised as a subject to provide a new technique for creating a realistic augmented reality or virtual reality moving image that gives the impression that the user himself/herself has taken a video camera and photographed it. Is.

A first aspect of the present invention for solving the above-mentioned problem is to provide a portable computer with a virtual space setting means for setting a virtual three-dimensional space (for example, control board 1550 in FIG. 2, terminal processing unit 200 in FIG. 14, camera). The work record management unit 210, the virtual space setting control unit 212, step S12 in FIG. 16),
Virtual camera control means for controlling the position and orientation of the virtual camera that captures the virtual three-dimensional space based on the position and orientation of the computer in the physical space (for example, control board 1550 in FIG. 2, terminal processing unit in FIG. 14). 200, camera work record management unit 210, virtual camera control unit 214, step S12 in FIG. 16,),
A camera work data generation unit that records camera work data capable of reproducing the position and orientation of the virtual camera in the virtual three-dimensional space continuously controlled by the virtual camera control unit (for example, the control board 1550 in FIG. 2). , Camera work data 550 in FIG. 5, terminal processing unit 200 in FIG. 14, camera work record management unit 210, camera work data generation unit 220, terminal storage unit 500, step S18 in FIG. 16,).
The camera work data generation means,
The position and orientation of the virtual camera at the time of starting recording in response to a given recording restart instruction are defined as the position of the virtual camera at a given interruption timing of the camera work data before the stop by the given recording stop instruction and An interruption/resumption control unit (for example, the control board in FIG. 2) that records the camera work data after the recording resumption instruction as camera work data that continues after the interruption timing of the camera work data before the stop by adapting to the posture. 1550, the terminal processing unit 200 of FIG. 14, the camera work recording management unit 210, the interruption/resumption control unit 222, step S42 of FIG. 17, step S80 of FIG. 18).

According to the first aspect of the present invention, the computer generates and records camera work data capable of reproducing the position and orientation of the virtual camera while generating the virtual space image captured by the virtual camera linked to the position and orientation of the computer. .. In other words, if a user looks at a virtual space image like a viewfinder image and uses the computer as if it were a portable video camera, he or she changes the position and changes the posture. Recorded on the computer.

Naturally, the movement locus of the virtual camera reproduced based on the camera work data includes a real “fluctuation” caused by the user actually holding and photographing. Therefore, if the camera work data recorded in the computer is used by the 3DCG software, it is possible to create a realistic moving image of an augmented reality image or a virtual reality image "a user himself/herself took a video camera and took a picture".

In addition, recording camera work data can be interrupted and resumed. Therefore, the user can arbitrarily pause and resume the recording of the camera work, and it becomes easy to record the desired camera work data. For example, even if the user's posture becomes uncomfortable during the recording of camerawork data and the desired camerawork cannot be continued, the recording is temporarily stopped, the posture is corrected, and then the recording of the camerawork data is resumed. can do.

Then, when the recording is restarted, the computer determines the position and the posture of the virtual camera when the recording is started in response to the recording restart instruction at the given interruption timing of the camera work data before the stop by the stop instruction. It is possible to automatically perform the correction adapted to the position and orientation of the virtual camera, and restart the recording of the camera work data after the recording restart instruction as the camera work data that continues after the interruption timing of the camera work data before the stop. In other words, even if there is a difference between the position of the computer at the time of interruption and the position of the computer at the time of restart, it is possible to record a series of camera work data that does not cause a position jump before and after the interruption. It is possible to provide a new technology for creating a video of augmented reality or virtual reality.

A second aspect of the present invention is data in which the camera work data generation means records in time series the relative position and relative attitude of the computer with respect to a given position set in the physical space, or the virtual camera It is a first invention program for controlling recording of data including at least data in which position and orientation are recorded in time series.

According to the second invention, as the camera work data, data in which the relative position and the relative attitude of the computer with respect to the given position set in the physical space are recorded in time series, or the position and the attitude of the virtual camera are recorded. It is possible to record data including at least data recorded in time series.

In a third aspect of the invention, the position and orientation of the virtual camera when the interruption/resumption control unit starts recording in response to the recording restart instruction is the virtual camera at the interruption timing of the camera work data before the stop. Conversion function generating means for generating a conversion function for adapting to the position and orientation of the camera (for example, the control board 1550 in FIG. 2, the terminal processing unit 200 in FIG. 14, the camera work record management unit 210, the conversion function generation unit 224, FIG. 15 conversion function data 590, steps S56 to S58 in FIG. 17, and uses the conversion function to record the camera work data after the recording restart instruction, or to store the conversion function in time series. The program according to the first or second invention.

According to the third aspect, the camera work data after the recording restart instruction can be recorded using the conversion function, and the camera work data before the recording is stopped and the camera work data after the recording is restarted are a series of data. Since it is recorded, data management becomes easy. In addition, by storing the conversion function in time series in association with the camera work data, the camera work data is corrected after recording, and a correction process regarding conformity is performed at the stage of generating a virtual space image based on the camera work data. It will be possible.

A fourth invention is the program according to the third invention, wherein the interruption/resumption control means gradually reduces the application effect of the conversion function to the camera work data after the recording restart instruction, as time elapses.

According to the fourth aspect, the effect of the conversion function can be gradually reduced.

5th invention detects that the said measurement data satisfy|filled the given improper change conditions which show that the improper change at the time of image|photographing the said virtual three-dimensional space, and recorded the said recording according to the said detection. Improper change automatic stop means for giving a stop instruction (for example, control board 1550 in FIG. 2, terminal processing unit 200 in FIG. 14, camera work record management unit 210, improper change automatic stop unit 234, step S32 in FIG. 17),
Notification control means for notifying the user of the stop of the recording by the automatic stop means at the time of improper change (for example, the control board 1550 in FIG. 2, the terminal processing unit 200 in FIG. 14, the camera work record management unit 210, the notification control unit 236, FIG. 17 step S41),
Is a program of any one of the first to fourth inventions for causing the computer to function as.

According to the fifth aspect, when the position or orientation of the computer changes unsuitable for photographing, recording can be automatically stopped and the user can be notified of that fact. Therefore, for example, if a user carrying a computer stumbles and a large blur occurs, or if the computer hits something in the vicinity to cause a large blur, the user does not have to stop the recording. This is convenient because recording is automatically stopped.

A sixth invention is to present a candidate timing for performing control to automatically select a candidate timing of the interruption timing and present a user with an image of the virtual three-dimensional space photographed at the position and orientation of the virtual camera at the candidate timing. For operating the computer as means (for example, control board 1550 in FIG. 2, terminal processing unit 200 in FIG. 14, camera work record management unit 210, candidate timing presentation unit 242, step S46 to step S50 in FIG. 17). It is a program according to any one of the first to fifth inventions.

According to the sixth aspect, it is possible to present the user with an appropriate candidate as the interruption timing. Therefore, the user can save the trouble of searching for the interruption timing by himself.

A seventh aspect of the present invention, when recording is started in response to the recording restart instruction, the position and orientation of the virtual camera based on the measurement data and the interruption timing based on the camera work data before the stop. The program according to any one of the first to sixth inventions for causing the computer to function as warning control means for warning the user when the difference between the position and the posture of the virtual camera satisfies a given deviation condition. Is.

According to the seventh aspect of the invention, it is possible to determine the deviation of the camera work and issue a warning to the user with the interruption and the resumption of the recording sandwiched.

An eighth invention is a camera work data generation method executed by a portable computer,
A virtual space setting step of setting a virtual three-dimensional space (for example, step S12 of FIG. 16),
A measurement data acquisition step of acquiring measurement data obtained by measuring the position and orientation of the portable computer in the real space (for example, step S10 of FIG. 16),
A virtual camera control step (for example, step S12 in FIG. 16) that continuously controls the position and orientation of the virtual camera that captures the virtual three-dimensional space based on the acquired measurement data,
A camera work data generation step of recording camera work data capable of reproducing the position and orientation of the virtual camera in the continuously controlled virtual three-dimensional space (for example, step S18 to step S22 in FIG. 16),
Including
The camerawork data generation step,
The position and orientation of the virtual camera at the time of starting recording in response to a given recording restart instruction are defined as the position of the virtual camera at a given interruption timing of the camera work data before the stop by the given recording stop instruction and By adapting to the posture, the step of recording the camera work data after the recording restart instruction is recorded as camera work data that continues after the interruption timing of the camera work data before the stop,
This is a camera work data generation method.

According to the eighth invention, the same effect as that of the first invention can be obtained.

A ninth aspect of the present invention is a portable electronic device that captures the virtual three-dimensional space based on a virtual space setting unit that sets a virtual three-dimensional space and the position and orientation of the electronic device in the real space. Virtual camera control means for controlling the position and orientation of the virtual camera, and camera work data capable of reproducing the position and orientation of the virtual camera in the virtual three-dimensional space continuously controlled by the virtual camera control means are recorded. And a camera work data generation unit that performs the camera work data generation unit that determines the position and orientation of the virtual camera when recording is started in response to a given recording restart instruction according to a given recording stop instruction. By adapting the position and the posture of the virtual camera at a given interruption timing of the camera work data before the stop, the camera work data after the recording restart instruction is made after the interruption timing of the camera work data before the stop. An electronic device having means for recording as subsequent camera work data.

According to the ninth aspect, it is possible to realize an electronic device that achieves the same effects as the first aspect.

The figure which shows the structural example of a moving image production system. The front view which shows the structural example of a user terminal. FIG. 6 is a diagram for explaining display control of a subject using augmented reality in a user terminal. The figure for demonstrating the movement track of the camerawork which can be acquired in this embodiment. The figure which shows the data structural example of camerawork data. The figure for demonstrating uploading to the server system of the information and data required for generation of a Zhang reality video. The figure for demonstrating generation of the moving image data of an augmented reality image by a server system. The figure which shows the example of the case forced to suspend. FIG. 6 is a diagram for explaining a basic concept of suspending/pausing and resuming recording of camera work data. The figure for demonstrating the recording of the camera work data which follows. FIG. 6 is a diagram for explaining interruption of recording of camera work data by a manual operation (recording stop operation). The figure for demonstrating the automatic pause of recording of camera work data. The figure for demonstrating the resumption of the recording of the camera work data interrupted by the temporary stop. The functional block diagram which shows the function structural example of a user terminal. The figure which shows the example of the program and data which a terminal memory|storage part memorize|stores. The flowchart for demonstrating the flow of a process in a user terminal. 6 is a flowchart for explaining the flow of a recording interruption restart process. The flowchart continued from FIG.

Hereinafter, an example of the embodiment of the present invention will be described, but it goes without saying that the mode to which the present invention is applicable is not limited to the following embodiment.

[First Embodiment]
FIG. 1 is a diagram illustrating a configuration example of a moving image generation system. The moving image generation system 1000 is a computer system that provides a service for creating an augmented reality moving image, and includes a server system 1100 and a user terminal 1500 that are connected to each other via a network 9 so as to be capable of data communication with each other.

The network 9 means a communication path capable of data communication. That is, the network 9 includes a dedicated line (dedicated cable) for direct connection, a LAN (Local Area Network) such as Ethernet (registered trademark), a communication network such as a telephone communication network, a cable network, and the Internet. The communication method may be wired or wireless.

The server system 1100 has, for example, a keyboard 1106, a touch panel 1108, and a storage 1140, and a control board 1150 is mounted on the main body device.

On the control board 1150, various microprocessors such as a CPU (Central Processing Unit) 1151, a GPU (Graphics Processing Unit) and a DSP (Digital Signal Processor), various IC memories 1152 such as VRAM, RAM and ROM, and a communication device 1153 are mounted. Has been done. Note that part or all of the control board 1150 may be realized by an ASIC (Application Specific Integrated Circuit), an FPGA (Field-Programmable Gate Array), or a SoC (System on a Chip).

Then, in the server system 1100, the control board 1150 performs an arithmetic processing based on a predetermined program and data, so that an image of augmented reality (expanded reality) is generated based on camera work data (details will be described later) created and recorded in the user terminal 1500. Realization image) and provides the user with moving image data of the augmented reality image.

Although only one user terminal 1500 is shown in FIG. 1, a plurality of user terminals 1500 may simultaneously access the server system 1100 in actual system operation.

Further, although the server system 1100 is drawn as if it is one server device, it may be configured by a plurality of devices. For example, the server system 1100 may have a configuration in which a plurality of blade servers that share respective functions are mounted and mutually connected via an internal bus so that data communication is possible. Further, the installation location of the hardware configuring the server system 1100 does not matter. A configuration may be adopted in which a plurality of independent servers installed at distant locations are caused to perform data communication via the network 9 and function as the server system 1100 as a whole.

The user terminal 1500 is a computer system used by the user 2 to acquire data for generating an augmented reality image, and is an electronic device (electronic device) that can access the server system 1100 via the network 9. The user terminal 1500 of the present embodiment is a so-called smartphone device, but may be a portable game device, a tablet computer, or the like as long as it is a handheld portable computer system.

FIG. 2 is a front view showing a configuration example of the user terminal 1500 according to this embodiment.
The user terminal 1500 includes a direction input key 1502, a button switch 1504, a touch panel 1506 that functions as an image display device and a contact position input device, a speaker 1510, a built-in battery 1509, a microphone 1512, a camera 1520, and a control board. 1550 and a memory card reading device 1542 capable of reading and writing data from a memory card 1540 which is a computer-readable storage medium. In addition, a power button, a volume control button, etc., which are not shown, are provided. Further, an IC card reader or the like that can read and write data in a contactless manner with respect to an IC card type credit card or a prepaid card that can pay the system usage fee may be provided.

The camera 1520 is an image sensor unit whose zoom magnification can be changed. That is, the camera 1520 is capable of capturing various digital moving images such as an image sensor such as a CCD (Charge Coupled Device), a zoom mechanism, an autofocus mechanism, and an LSI that generates image data from a signal captured by the image sensor. Contains elements.

The control board 1550 is a radio for wirelessly communicating with a CPU 1551, various microprocessors such as GPU and DSP, various IC memories 1552 such as VRAM, RAM and ROM, mobile phone base stations and wireless LAN base stations connected to the network 9. A communication module 1553, a positioning module 1555, a triaxial acceleration sensor 1556, a triaxial gyro 1557, an interface circuit 1558, etc. are mounted.

The positioning module 1555 acquires the position of the user terminal 1500. In the present embodiment, the position coordinates in the physical space can be acquired by using the existing positioning system. That is, the positioning module 1555 provides the positioning function in the user terminal 1500 by receiving the signal provided from the positioning system and outputting the positioning information at a predetermined cycle (for example, every one second). In this embodiment, a GPS (Global Positioning System) is used as a positioning system. Therefore, the positioning module 1555 can use an existing “GPS module”, “GPS receiver”, or the like. “Positioning information” includes positioning date and time (for example, UTC: Coordinated Universal Time), position coordinates (latitude/longitude/altitude), direction, and the like. The positioning system to be used is not limited to GPS and can be selected as appropriate. For example, the positioning module 1555 can be replaced with a configuration of a direction sensor and a communication device that wirelessly connects to a wireless base station of a mobile phone, a wireless base station of a Wi-Fi network, or the like. That is, it is also possible to acquire the position information that has been measured in advance for the connected wireless base station and use this as the position coordinates of the user terminal 1500 in the physical space.

The interface circuit 1558 includes a driver circuit of the touch panel 1506, a circuit for receiving signals from the direction input keys 1502 and the button switches 1504, an output amplifier circuit for outputting a sound signal to the speaker 1510, and a sound signal collected by the microphone 1512. An input signal generation circuit for generating, a circuit for inputting image data of an image captured by the camera 1520, a signal input/output circuit for the memory card reader 1542, and the like are included.

These elements mounted on the control board 1550 are electrically connected to each other via a bus circuit or the like, and are connected so that reading/writing of data and transmission/reception of signals can be performed. Note that part or all of the control board 1550 may be configured with an ASIC, FPGA, or SoC. Then, the control board 1550 stores a program and various data for realizing the function as the user terminal in the IC memory 1552.

In the present embodiment, the user terminal 1500 is configured to download the program and various setting data from the server system 1100, but the user terminal 1500 may be configured to read from a storage medium such as a memory card 1540 that is separately obtained.

FIG. 3 is a diagram for explaining display control of a subject using augmented reality in the user terminal 1500.
The user terminal 1500 can display an object on the touch panel 1506 by using augmented reality. That is, the AR marker 5 is installed in the physical space, and the user 2 adjusts the position/orientation of the user terminal 1500 and the zoom of the camera 1520 so as to capture the AR marker 5. The AR marker 5 in the image captured by the camera 1520 is subjected to image recognition processing and is used for calculation processing of the relative position and relative attitude between the user terminal 1500 and the AR marker 5. The relative position and relative attitude with respect to the AR marker 5 become measurement data of the position and attitude of the user terminal 1500.

The user terminal 1500 uses the calculated relative position and relative attitude and the measurement data of the acceleration and attitude of the user terminal 1500 obtained by using the triaxial acceleration sensor 1556 and the triaxial gyro 1557 to determine the position change and attitude of the camera 1520. Changes and can always be obtained. In addition, information on a shooting angle of view (also referred to as zoom magnification) can be periodically acquired by a zoom operation of the camera 1520. In the following, the relative position, the relative attitude, the position change, the attitude change, the parameter values that describe so-called shooting conditions such as the shooting angle of view, and the size and orientation of the AR marker 5 that is the basis for obtaining these, and the attitude data. The acceleration data, the zoom control value, the measurement value of the sensor, and the like are collectively referred to as “actual shooting conditions”.

While acquiring the actual shooting condition, the user terminal 1500 can arrange the subject object 4 in the virtual three-dimensional space and control the operation thereof. In the present embodiment, the subject object 4 is a virtual character that imitates a human form, and is an AR (Augmented Reality) character based on a captured image captured by the camera 1520. The user terminal 1500 uses the given music data prepared in advance and the predetermined motion data for realizing the motion synchronized with the music of the music data so that the subject object 4 dances while singing the music. Motion control.

In the virtual three-dimensional space, a virtual camera C1 that is linked to the parameter value of the actual shooting condition of the camera 1520 and whose shooting position, shooting posture, and shooting angle of view are controlled is arranged. The user terminal 1500 generates an image of a virtual three-dimensional space captured by the virtual camera C1, that is, a virtual space image 90, and superimposes the generated image on the image of the real space (real space image 92). The touch panel 1506 displays the augmented reality image (augmented reality image 94) as the live image 30 (corresponding to the finder image of the camera 1520).

The user 2 inputs a predetermined recording start operation to the user terminal 1500, then uses the image displayed on the touch panel 1506 as a finder image, watching the subject object 4 dancing while singing a song, while watching the portable video. Operate the camera as you would with a camera. The shooting operation includes an operation of moving the user terminal 1500 in the physical space, sometimes changing the posture, and sometimes changing the zoom magnification. In other words, it is an operation to attach camera work.

As shown in FIG. 4, when camera work is applied using the user terminal 1500, a movement locus L1 caused by the camera work corresponds to camera shake or the like that occurs when a person holds a portable video camera in his/her hand and takes a picture. Including "fluctuation".

Then, when the user terminal 1500 detects the input of the predetermined recording start operation by the user 2, the user terminal 1500 starts recording the camera work data 550 as shown in FIG. 5, for example.
The camera work data 550 includes a unique shooting ID 551, a shooting start date/time 553, a shooting end date/time 555, real shooting condition data 560, virtual shooting condition data 570, and a physical space image 92 shot by the camera 1520. The recorded real image recording data 580 and the virtual space image recording data 582 recording the virtual space image 90 are included. Of course, data other than these may be included as appropriate, and some of these may be omitted as appropriate.

The actual shooting condition data 560 is, for example,
1) Real camera posture data 561 in which the relative posture calculated based on the AR marker 5 and/or the measurement data by the 3-axis gyro 1557 are stored in time series,
2) Real camera acceleration data 563 in which the acceleration obtained from the change in the shooting position calculated based on the AR marker 5 and/or the measurement data by the triaxial acceleration sensor 1556 are stored in time series,
3) Actual zoom control data 565 in which the zoom magnification (shooting angle of view) of the camera 1520 or the control parameter value of the zoom magnification is stored in time series,
4) Actual movement describing a movement locus L1 in which a relative position of the camera 1520 calculated based on the AR marker 5 and/or a photographing position obtained by integrating measurement data by the triaxial acceleration sensor 1556 are traced in time series. Trajectory data 567,
including.

The virtual shooting condition data 570 is, for example,
1) Virtual camera attitude data 571 that stores the shooting attitude of the virtual camera C1 obtained from the actual camera attitude data 561 in time series,
2) Virtual camera acceleration data 573 in which the acceleration occurring in the virtual camera C1 obtained from the actual camera acceleration data 563 is stored in time series,
3) Virtual camera shooting angle-of-view data 575 in which the shooting angle of view (or zoom magnification) of the virtual camera C1 obtained from the actual zoom control data 565 is stored in time series,
4) Virtual camera movement locus data 577 describing the movement locus Lc of the virtual camera C1 that reproduces the movement locus L1 in the virtual tertiary space based on the actual movement locus data 567,
including.

The time code, that is, the time axis of the actual shooting condition data 560 and the time code, that is, the time axis of the virtual shooting condition data 570 are associated with each other so as to be synchronized with the time code for playing the music or controlling the motion of the subject object 4. It shall be recorded. Further, one of the actual shooting condition data 560 and the virtual shooting condition data 570 may be omitted, and the shooting condition data from the remaining side to the omitted side may be obtained by back calculation as necessary.

When the user 2 inputs a predetermined recording stop operation, the user terminal 1500 stops the recording of the camera work data 550, and as shown in FIG. 6, the server system 1100 stores the information and data necessary for generating the augmented reality moving image. Upload to. Specifically, for example, the model data 510 of the subject object 4, the motion data 512, the music data 514, and the camera work data 550 are uploaded. Of course, data other than these may be uploaded as appropriate. For example, when a background object such as a stage device for the subject object 4 to perform a song or dance is prepared in the virtual three-dimensional space, the model data and motion data thereof are used to realize display and operation. Data is also uploaded. Of course, data other than these may be uploaded as appropriate.

When the model data 510, the motion data 512, and the music data 514 previously recorded in the server system 1100 are downloaded to the user terminal 1500 and used at the time of recording the camera work data 550, these data are recorded. Data upload can be omitted. The model data 510, the motion data 512, and the music data 514 may be the user 2's own work or another work.

FIG. 7 is a diagram for explaining generation of an augmented reality moving image by the server system 1100. When the server system 1100 uploads the model data 510, the motion data 512, the music data 514, and the camera work data 550 of the subject object 4, the subject object 4 and the background object are placed in the virtual three-dimensional space based on those data. Are arranged and their operations are controlled in synchronization with the reproduction of music. Further, the virtual camera C2 is controlled based on the uploaded camera work data 550 in synchronization with the reproduction of the music.

Then, the server system 1100 generates an image (virtual reality image) in the virtual three-dimensional space captured by the virtual camera C2, and superimposes it on the real image recording data 580 included in the uploaded camera work data 550. Then, the data of the augmented reality moving image (augmented reality moving image data 584) is created.

In the augmented reality video created in this way, screen fluctuations due to "fluctuations" that occur when a person operates a portable video camera are reproduced, and "the user himself took the video camera in his hand to shoot" It is a video of augmented reality with reality.

There are cases in which interruption is inevitable depending on how the camera work is attached, but this embodiment also accommodates such cases.
For example, FIG. 8 is a diagram illustrating an example of a case in which interruption is forced. When the user 2 performs a shooting operation, depending on the height of the user 2 and the size of the subject object 4, the desired camera angle cannot be continuously secured. In the example of FIG. 8, the user 2 starts to photograph the subject object 4 from the front, and tries to perform leaning photography such as looking up while turning around to the left of the subject object 4. However, since there is the ground (floor surface) in the real world, there is a limit to the position where the user terminal 1500 is lowered. As a matter of course, the user 2 is also constrained from a cramped posture, and it becomes difficult to stably obtain a desired angle due to physical limitations. In such a case, recording of the camera work data 550 is forced to end there. A movement locus L1 in the figure is a locus of camera work that can be realized within such limits. Even if the user 2 wants to take a picture on the movement locus L2, the broken line portion just dives below the floor surface and cannot realize it.

Therefore, in the present embodiment, the recording of the camera work data 550 can be suspended (temporarily stopped) and restarted.
FIG. 9 is a diagram for explaining the basic idea regarding the interruption (temporary stop) and the restart of the recording of the camera work data 550.

The user terminal 1500 is involved in recording control of the camera work data 550, and stops the recording in response to a given recording stop instruction and starts recording in response to a given recording restart instruction. Control the suspension and resumption of the. From the standpoint of the user 2, when the camerawork is forced to be interrupted, the user 2 interrupts the recording of the camerawork data 550, and the position and the posture of the user terminal 1500 are changed to those of the camerawork assumed at the time of restart. It is possible to resume the recording of the camera work data 550 by adjusting the position and the posture as appropriate as the starting point.

FIG. 10 is a diagram for explaining the recording of the camera work data that continues after the interruption timing set in the camera work data 550 before the stop.
The user terminal 1500 controls interruption and resumption of camera work data by stopping recording according to a recording stop instruction and starting recording according to a recording restart instruction. Then, in performing the interruption and the resumption, the user terminal 1500 starts recording in response to the recording resumption instruction with respect to the virtual photographing condition of the virtual camera C1 at the interruption timing (see the virtual photographing condition data 570 in FIG. 5). Correction processing is performed so as to match the virtual imaging conditions of the virtual camera C1 when (restarted). Then, the camera work data continuing after the interruption timing of the camera work data before the stop is recorded.

More specifically, when recording the camerawork data 550 after resumption, the user terminal 1500 starts recording in response to the virtual photographing condition at the interruption timing of the movement locus Lc1 of the camerawork before interruption and the recording restart instruction. Subsequent camera work data is recorded while performing a correction process so as to match the virtual shooting condition at the start timing of the movement locus Lc2 of the camera work after that. Then, the camera work data 550 before the stop, in which the last part of the camera work data 550 before the interruption is partially changed, deleted, or added, and the camera work data 550 after the recording is started according to the recording restart instruction. (Continued camera work data) is connected to perform control to record a series of camera work data 550.

The method of correction processing can be set as appropriate.
For example, focusing on the position, in the present embodiment, the start point position Ps of the camera work at the start timing when recording is started in response to the recording restart instruction is defined as the interruption timing position at the interruption timing of the movement locus L1 of the camera work before the interruption. A coordinate function (for example, a conversion matrix) that changes to Pe is generated, and this conversion matrix is applied to perform conversion. As another correction processing method, a vector of the positional deviation from the interruption timing position Pe to the starting point position Ps of the movement locus L2 is obtained, and a vector in which the direction of this vector is reversed is applied to the position coordinates for correction. Good.

Such correction processing is similarly performed for the posture and zoom included in the shooting conditions. That is, the recording of the camera work data 550 after the restart is controlled while performing the correction processing so that the attitude at the interruption timing matches the attitude at the start timing when the recording is started in response to the recording restart instruction. Further, the camera work data 550 after the restart is corrected while performing the correction processing so that the zoom ratio at the interruption timing (or the shooting angle of view of the virtual camera) matches the zoom ratio at the start timing when recording is started in response to the recording restart instruction. Control the recording of.

By performing such a correction, the user 2 is forced to have a cramped posture, and it is difficult to stably obtain a desired angle due to physical limitations (see FIG. 8 ). It is possible to suspend and resume shooting without being aware of it. Then, even if there is a difference between the position of the user terminal 1500 at the time of interruption and the position of the user terminal 1500 at the time of resumption, it is possible to record a series of camera work data 550 that does not cause a position jump before and after the interruption. .. Therefore, the user 2 can continue the shooting operation without taking an uncomfortable posture after restarting.

Next, in the present embodiment, a procedure for suspending and resuming recording of the camera work data 550 will be described in detail.
FIG. 11 is a diagram for explaining interruption of recording of the camera work data 550 by a manual operation (recording stop operation). The user terminal 1500 displays the shooting screen W11 on the touch panel 1506 while recording the camera work data 550.

On the shooting screen W11,
1) A monitor display 10 for displaying an image being taken,
2) Time code 11,
3) Status notification display 12 showing the current status of shooting and recording,
4) Recording start operation icon 13 for inputting a new recording start operation,
5) A recording stop operation icon 14 for inputting a recording stop operation for temporarily suspending/stopping the recording of the camera work data 550,
6) A recording restart operation icon 15 for inputting a recording restart operation for restarting the recording of the temporarily stopped camera work data 550,
7) A recording end operation icon 16 for inputting a recording end operation corresponding to the end of recording,
including. The recording end operation can be said to be a second recording stop operation for stopping recording.

That is, the user 2 can interrupt or restart the recording of the camera work data 550 at an arbitrary timing by appropriately selecting the recording stop operation icon 14 and the recording restart operation icon 15 by a touch operation. ..

FIG. 12 is a diagram for explaining the automatic pause of the recording of the camera work data 550. During recording of the camera work data 550, the user terminal 1500 constantly monitors the acceleration measured by the triaxial acceleration sensor 1556 and the posture measured by the triaxial gyro 1557. Then, the user terminal 1500 automatically pauses recording when either the acceleration satisfies a predetermined position inappropriate change condition or the posture satisfies a predetermined posture inappropriate change condition. Then, the user 2 is notified by displaying the automatic recording stop notification 19 on the touch panel 1506.

The “positional inappropriate change condition” mentioned here means a change in position of the camera 1520 that causes excessive screen blur even when it is used as an effect of image expression, and a change in acceleration measured by the triaxial acceleration sensor 1556. Is described as a condition of.

Similarly, the “posture inappropriate change condition” is a posture change of the camera 1520, which causes excessive screen blur even when it is used as a visual representation, and is measured by the 3-axis gyro 1557. It is described as a condition of posture change.

In other words, these conditions define a state in which the camera has moved too rapidly or a state in which the camera has moved abruptly. The user terminal 1500 automatically performs temporary stop control based on these conditions. As a result, the user 2 intentionally changes the position of the user terminal 1500 abruptly or operates the user terminal 1500 suddenly without the user 2 operating the recording stop operation icon 14. Since the recording of the camera work data 550 can be temporarily stopped only by performing, the operability is improved. In addition, if the user 2 accidentally drops the user terminal 1500 or if a happening occurs such that the user 2 is overwhelmed by shooting and the user 2 stumbles, it is automatically paused, so that useless recording continues even after the happening. It can be prevented.

FIG. 13 is a diagram for explaining the setting of the restart timing for restarting the recording regarding the restart of the recording of the camera work data 550 which is interrupted by the temporary stop. A seek bar 20 for additionally accepting an operation input of a restart timing is additionally displayed on the shooting screen W13 after the recording is interrupted due to the temporary stop. As a result, the user 2 can perform an operation input for shifting the position of the slider 21 of the seek bar 20 and designating the restart timing. Here, if the process before suspension is temporarily divided into the process of suspending and the process after the process of suspending is resumed, the restart timing can be said to be the suspend timing for the restart process. Therefore, the resumption timing in the process of resuming after the temporary suspension is also referred to as interruption timing.

Then, in response to the display of the seek bar 20 and the reception of the designated operation input, the user terminal 1500 automatically selects the candidate timing of the restart timing (interruption timing), and the virtual photographing condition (the position and the posture of the virtual camera C1 etc.) at the candidate timing. The control of presenting the image of the virtual three-dimensional space photographed in () to the user 2 is performed.

Specifically, the user terminal 1500 automatically selects a plurality of candidate timings, and displays the thumbnails 22 (22a, 22b,...) Which serve as a guide for the operation of designating the restart position at each candidate timing. The thumbnail 22 is a reduced image of an image in the virtual three-dimensional space captured under the virtual imaging condition (position and orientation of the virtual camera C1) at the timing of the seek bar 20 pointed to by the thumbnail 22. A plurality of thumbnails 22 are prepared. One of them is a thumbnail 22a that automatically determines the timing when the user terminal 1500 goes back a predetermined time from the stop timing as the initial setting of the restart timing (interruption timing). The one or more thumbnails 22b and 22c other than the thumbnails are augmented reality images at a timing determined at a predetermined time interval (N frame before, M frame before) from the end timing of the camera work data 550 before the stop due to the recording stop instruction. Is a reduced image of.

It is assumed that the predetermined time of "retrogression" referred to here is that the captured image is excessively blurred and unusable when an event that satisfies the position change inadequacy condition or the posture change inadequacy condition occurs. It is time to be done. In the present embodiment, the timing immediately before the detection that the inappropriate change condition is satisfied (either the position change inappropriate condition or the posture change inappropriate condition is satisfied) is detected. In other words, the user terminal 1500 automatically determines and determines the timing when the change in the position or the change in the posture of the user terminal 1500 does not satisfy a given stability condition. Alternatively, the “retroactive” predetermined time may be a timing (for example, 1 second) that is a predetermined time before the end timing of the camera work data 550 recorded in the latest past. Due to this automatic retroactive movement, the initial position of the seek bar 20 becomes an assumed position immediately before the captured image becomes an image that is excessively blurred and cannot be used, so that the user can select an appropriate timing to restart recording. You can save the trouble of searching.

In the present embodiment, the user terminal 1500 records the camera work data 550 and also records the augmented reality image displayed on the monitor display 10 (see FIG. 11 ). The thumbnail 22 is created by extracting and reducing the timing corresponding to each thumbnail 22 (22a, 22b,...) From the recorded augmented reality image. Of course, it is also possible to read the virtual imaging conditions at the timings corresponding to the thumbnails 22 (22a, 22b,...) From the camera work data 550, generate the virtual space image again, and reduce and use it.

FIG. 14 is a functional block diagram showing a functional configuration example of the user terminal 1500 in this embodiment. The user terminal 1500 includes an operation input unit 100, a positioning unit 102, an acceleration measuring unit 106, a posture measuring unit 108, a voice input unit 110, an image capturing unit 120, a terminal processing unit 200, and a sound output unit 390. The image display unit 392, the communication unit 394, and the terminal storage unit 500.

The operation input unit 100 outputs an operation input signal corresponding to various operation inputs made by the player to the terminal processing unit 200. For example, it can be realized by a push switch, a joystick, a touch pad, a trackball, an acceleration sensor, a gyro, a CCD module, or the like. The direction input key 1502, the button switch 1504, and the touch panel 1506 in FIG. 2 correspond to this.

The positioning unit 102 acquires the position information of the user terminal 1500 and outputs it to the terminal processing unit 200. The positioning module 1555 of FIG. 2 corresponds to this.

The acceleration measuring unit 106 measures the acceleration generated in the user terminal 1500 and outputs it to the terminal processing unit 200. The triaxial acceleration sensor 1556 in FIG. 2 corresponds to this.

The posture measurement unit 108 measures the posture change of the user terminal 1500 and outputs it to the terminal processing unit 200. The 3-axis gyro 1557 of FIG. 2 corresponds to this.

The voice input unit 110 collects a voice (user voice) emitted by the user and environmental sounds, and outputs a voice signal to the terminal processing unit 200. The microphone 1210 in FIG. 2 corresponds to this.

The image capturing unit 120 captures an image of the outside, generates image data of the image, and outputs the image data to the terminal processing unit 200. The camera 1520 of FIG. 2 corresponds to this.

The terminal processing unit 200 is realized by, for example, a microprocessor such as a CPU or a GPU, or an electronic component such as an IC memory, and controls input/output of data with each functional unit including the operation input unit 100 and the terminal storage unit 500. I do. Then, various arithmetic processes are executed based on a predetermined program and data, an operation input signal from the operation input unit 100, and various data received from the server system 1100 to control the operation of the user terminal 1500. The control board 1550 of FIG. 2 corresponds to this.

Then, the terminal processing unit 200 in the present embodiment includes a camera control unit 202, a camera work record management unit 210, a clock unit 280, a sound generation unit 290, an image generation unit 292, and a communication control unit 294. Have.

The camera control unit 202 detects a user's input of a zoom operation to the operation input unit 100, and controls to change the zoom magnification of the imaging unit 120 according to the zoom operation.

The camerawork record management unit 210 executes various processes related to recording the camerawork data 550. Specifically, the camera work record management unit 210 includes a virtual space setting control unit 212, a measurement data acquisition control unit 213, a virtual camera control unit 214, an operation instruction control unit 216, and a camera work data generation unit 220. , Virtual space image generation control unit 230, virtual space image recording control unit 232, improper change automatic stop unit 234, notification control unit 236, interruption timing setting unit 240, candidate timing presentation unit 242, and warning control unit. 244, and. Of course, functional units other than these may be included as appropriate.

The virtual space setting control unit 212 sets a virtual three-dimensional space. In the present embodiment, provision of a virtual three-dimensional space, placement of the subject object 4 and the background object, and control of their operations in synchronization with the reproduction of the music correspond to this.

The measurement data acquisition control unit 213 performs control to acquire measurement data obtained by measuring the position and orientation of the user terminal 1500. In this embodiment, measurement data is acquired using the camera 1520 provided in the user terminal 1500. Specifically, the control by the augmented reality technology is performed. That is, the AR marker 5 is recognized from the image captured by the camera 1520, and the relative position/orientation is calculated and used as the measurement data (see FIG. 3). Further, the control for obtaining the position and orientation of the user terminal 1500, the position change and orientation change, and the movement locus L1 based on the measurement data of the positioning unit 102, the acceleration measurement unit 106, and the orientation measurement unit 108 also corresponds to this. Focusing on the data, the generation control of the actual photographing condition data 560 also applies.

It should be noted that the acquisition of the measurement data is not limited to being realized by the measurement using the camera 1520 and the sensors provided in the user terminal 1500. For example, a configuration is possible in which measurement data of the position and orientation of the user terminal 1500 is acquired from an external device that can communicate. Specifically, a position/orientation measuring device that is separate from the user terminal 1500 is prepared, and the relative position and relative attitude of the user terminal 1500 with respect to a device that serves as a position or orientation reference (such as the AR marker 5) are provided in the device. May be measured and the measurement data may be transmitted to the user terminal 1500 by data communication one by one. In that case, the measuring method by the position/orientation measuring apparatus can be set appropriately. For example, a method of irradiating an infrared beam to measure the relative distance of each part, or a method of taking an image and analyzing the image to obtain the image may be used.

The virtual camera control unit 214 controls the position and orientation of the virtual camera that captures the virtual three-dimensional space in association with the measurement data of the position and orientation of the user terminal 1500 (see FIG. 3). At that time, the virtual camera control unit 214 controls the shooting angle of view of the virtual camera in association with the zoom magnification of the shooting unit 120.

The operation instruction control unit 216 performs control to give a recording stop instruction by a predetermined recording stop operation and give a recording restart instruction by a predetermined recording restart operation. In the present embodiment, the operation instruction control unit 216 outputs a recording stop instruction when detecting a touch operation on the recording stop operation icon 14 or the recording end operation icon 16. When a touch operation on the recording restart operation icon 15 is detected, a recording restart instruction is output.

The camera work data generation unit 220 controls to generate and record camera work data capable of reproducing the position and orientation of the virtual camera in the virtual three-dimensional space continuously controlled by the virtual camera control unit 214 (FIG. 5). reference). At that time, the camerawork data generation unit 220 records, as the camerawork data, time-series data of a relative position and a relative attitude of the user terminal 1500 based on a given position set in the physical space, or a virtual image. The recording of data including at least the data in which the position and orientation of the camera C1 are recorded in time series is controlled. Further, the camera work data generation unit 220 records the data that can reproduce the shooting angle of view of the virtual camera in the camera work data.

Then, the camera work data generation unit 220 has an interruption/resumption control unit 222.
The interruption/resumption control unit 222 controls interruption and resumption of camera work data by stopping recording in response to a given recording stop instruction and starting recording in response to a given recording restart instruction. At that time, the interruption/resumption control unit 222 determines the position and orientation of the virtual camera at the time of starting recording in response to the recording restart instruction by using the virtual camera at the given interruption timing of the camera work data before the stop by the recording stop instruction. By adapting to the position and orientation, the camera work data after the recording restart instruction is recorded as the camera work data that continues after the interruption timing of the camera work data before the stop.
Specifically, the suspension/resumption control unit 222 records the camera work data after the recording restart instruction by using the conversion function.

In the present embodiment, the recording stop instruction corresponds to a touch operation on the recording stop operation icon 14 or the recording end operation icon 16. The recording restart instruction corresponds to a touch operation on the recording restart operation icon 15. The restart timing (interruption timing) for restarting the interrupted recording is the timing at which the user 2 inputs an instruction operation via the seek bar 20.

Further, the interruption/resumption control unit 222 controls the zoom magnification of the virtual camera at the time of starting recording in response to the recording restart instruction so as to be the zoom magnification of the virtual camera at the interruption timing to continuously record the camera work data. To do.

Then, the suspension/resumption control unit 222 connects the data up to the suspension timing of the camera work data before the suspension and the subsequent camera work data, and records them as a series of camera work data. Specifically, the interruption/resumption control unit 222 adapts the position and orientation of the virtual camera when recording is started in response to the recording restart instruction to the position and orientation of the virtual camera at the interruption timing of the camera work data before the stop. A conversion function for performing the above is generated, and the conversion function is applied to the subsequent camera work data to perform recording as a series of camera work data (see FIG. 10).

When recording the camera work data by the camera work data generation unit 220, the virtual space image generation control unit 230 generates an image of the virtual three-dimensional space photographed at the position and orientation of the virtual camera according to the camera work data being recorded. To do. The generated virtual space image is superimposed on the image of the real space captured by the camera 1520 and displayed as the live image 30 on the capturing screen W11 (see FIGS. 3 and 11).

The virtual space image recording control unit 232 controls the recording of the image generated by the virtual space image generation control unit 230 in association with the camera work data recorded by the camera work data generation unit 220. In this embodiment, the virtual space image recording data 582 is recorded in the camera work data 550 (see FIG. 5).

The improper change automatic stop unit 234 detects that the measurement data satisfies a given improper change condition indicating that the measurement data has made an improper change when capturing an image in the virtual three-dimensional space, and stops recording according to the detection. Give instructions (see Figure 12).

The notification control unit 236 notifies the user of the stop of the recording by the improper change automatic stop unit 234. In the present embodiment, the display control of the automatic recording stop notification 19 corresponds to this (see FIG. 12).

The suspension timing setting unit 240 is a functional unit for setting a resumption timing (suspension timing) at the time of resuming after the suspension, and sets the suspension timing based on a user's operation input. In the present embodiment, the display of the seek bar 20 and the setting of the interruption timing (restart timing) according to the instruction operation input to the seek bar 20 correspond to this (see FIG. 13 ).

The candidate timing presentation unit 242 automatically selects the candidate timing of the interruption timing (restart timing), and presents the user with the image of the virtual three-dimensional space photographed at the position and orientation of the virtual camera at the candidate timing. In the present embodiment, the control relating to the display of the thumbnail 22 that is attached and displayed on the seek bar 20 corresponds to this (see FIG. 13 ).

In addition, the candidate timing presentation unit 242 selects the timing immediately before the unsuitable change automatic stop unit 234 detects that the unsuitable change condition is satisfied, as the candidate timing of the interruption timing, and the position of the virtual camera at the candidate timing. The control for presenting the image of the virtual three-dimensional space captured in the posture to the user is performed. In the present embodiment, automatic retrospective selection of candidate timings and display control of the thumbnails 22a correspond to this.

The warning control unit 244, when starting recording in response to the recording restart instruction, the position and orientation of the virtual camera based on the measurement data and the position and orientation of the virtual camera at the interruption timing based on the camera work data before the stop. If the difference between and the condition satisfies a given deviation condition, control is performed to warn the user. Specifically, when the shift condition is satisfied, a predetermined warning is displayed on the shooting screen W13 after the recording is interrupted.

The shift condition can be set as appropriate. For example, “positional deviation equal to or more than the expected average height of the user image” and “posture deviation of 180°±5°” are defined individually or in combination. In the former case, the position of the user terminal 1500 at the time of resumption is too far from the position at the resumption timing, which may make it difficult to physically move in the operation of attaching the camerawork after the resumption. Can be reported to the user. More specifically, for example, when the position of the user terminal 1500 at the reproduction timing is a position close to the floor surface and the user terminal 1500 is lifted and restarted from above the head of the user 2, after that, the user terminal 1500 Moving 1500 further upwards can be physically difficult. For example, it will not reach unless you use a springboard or ladder. Therefore, by giving a warning to the user 2, it is possible to prevent such a situation from occurring. In the latter case, since the recognition of the posture of the user terminal 1500 is reversed between the interruption and the restart of the recording, it is necessary to hold the user terminal 1500 so that it can be reversed during the camera work after the restart. It is possible to provide the user 2 with an opportunity to reconfirm camerawork after resumption.

The clock unit 280 uses the system clock to clock the current date and time, time limit, and the like.

The sound generation unit 290 is realized by, for example, a digital signal processor (DSP), a processor such as a voice synthesis IC, an audio codec capable of reproducing a voice file, or the like, and generates a sound signal of a music piece, a sound effect, or various operation sounds. , To the sound output unit 390.

The sound output unit 390 is realized by a device that outputs (sounds) sound based on the sound signal input from the sound generation unit 290. The speaker 1510 of FIG. 2 corresponds to this.

The image generation unit 292 controls generation of various image data and generation/output of image signals for displaying those images on the image display unit 392. In this embodiment, a virtual space image and an augmented reality image are generated.

The image display unit 392 displays various images based on the image signal input from the image generation unit 292. For example, it can be realized by an image display device such as a flat panel display, a projector, or a head mounted display. In the present embodiment, the touch panel 1506 of FIG. 2 corresponds to this.

The communication control unit 294 executes data processing related to data communication and realizes data exchange with an external device via the communication unit 394.

The communication unit 394 connects to the network 9 to realize communication. For example, it is realized by a wireless communication device, a modem, a TA (terminal adapter), a jack of a wired communication cable, a control circuit, and the like, and the wireless communication module 1553 of FIG. 2 corresponds to this.

The terminal storage unit 500 stores a program for realizing various functions for causing the terminal processing unit 200 to integrally control the user terminal 1500, various data, and the like. Further, it is used as a work area of the terminal processing unit 200, and temporarily stores a calculation result executed by the terminal processing unit 200 according to various programs, input data input from the operation input unit 100, and the like. Such a function is realized by, for example, an IC memory such as a RAM or a ROM, a magnetic disk such as a hard disk, an optical disk such as a CD-ROM or a DVD. The IC memory 1552 and the memory card 1540 mounted on the control board 1550 of FIG. 2 correspond to this. A configuration that uses online storage is also possible.

FIG. 15 is a diagram showing an example of programs and data stored in the terminal storage unit 500. The terminal storage unit 500 includes a terminal program 501, model data 510, motion data 512, music data 514, measurement data 548, camera work data 550 (see FIG. 5), conversion function data 590, and current date and time. 800 is stored. Of course, programs and data other than these can be stored as appropriate.

The terminal program 501 is a program for causing the terminal processing unit 200 to realize the function as the user terminal 1500 and can realize the function as the camera control unit 202. The terminal program 501 includes a camera work record management program 502, and the function of the camera work record management unit 210 is implemented by the terminal processing unit 200 executing the program.

The model data 510 is model data of the subject object 4. The motion data 512 is data for realizing motion control of the subject object 4. The music data 514 is sound and/or voice data of music. These data may be data created or prepared separately by the user himself, or may be appropriately downloaded by the server system 1100 having an original.

The conversion function data 590 is used for correction processing for adapting the position and orientation of the virtual camera at the time of starting recording according to the recording restart instruction to the position and orientation of the virtual camera at the interruption timing of the camera work data before the stop. This is the data that defines the conversion function to be used. In this embodiment, the data of the conversion matrix is stored (see FIG. 10).

Next, the operation of the moving image generation system 1000 of this embodiment will be described.
FIG. 16 is a flowchart for explaining the flow of processing in the user terminal 1500. The process flow described here is realized by the user terminal 1500 executing the terminal program 501 in the terminal processing unit 200.

The user terminal 1500 first starts acquisition of measurement data of the position and orientation of the user terminal 1500 and shooting with the camera 1520 (step S10), sets a virtual three-dimensional space, and controls an object such as the subject object 4. And the interlocking control with the virtual camera C1 based on the measurement data of the position and orientation, the zoom operation, etc. are started (step S12). Then, the user terminal 1500 starts displaying the shooting screen W11 (see FIG. 11) (step S14).

Next, when the user terminal 1500 detects the input of the recording start operation, the user terminal 1500 determines that a recording start instruction has been given (step S16), and starts generation/recording of the camera work data 550 (step S18). Then, until the input of the recording end operation is detected, that is, until the recording end instruction is given (NO in step S20), the user terminal 1500 periodically executes the recording interruption restart processing (step S22).

17 to 18 are flowcharts for explaining the flow of the recording interruption restart processing. As shown in FIG. 17, in the same process, when the user terminal 1500 detects a touch operation on the recording stop operation icon 14 (see FIG. 11), that is, an input of a recording stop operation (YES in step S30), the recording is performed. When it is determined that a stop instruction has been issued, display control to the effect that recording has been stopped/interrupted is performed based on the operation input of the user 2 (step S40). Then, the user terminal 1500 stops/interrupts the recording of the camera work data 550 (step S42), and temporarily stops the reproduction of the music and the operation control of the subject object 4 (step S44).

In addition, when the measurement result of the position or orientation of the user terminal 1500 satisfies the position inappropriate change condition or the posture inappropriate change condition (YES in step S32), the position or posture is automatically changed because the position or posture is inappropriately changed. Then, display control is performed to the effect that recording has been stopped/interrupted (step S41). Then, it is determined that the recording stop instruction is given, the recording of the camera work data 550 is stopped (step S42), and the reproduction of the music and the operation control of the subject object 4 are temporarily stopped (step S44).

Next, the user terminal 1500 automatically selects the candidate timing of the interruption timing (corresponding to the resumption timing for resuming recording) (step S46), displays the seek bar 20 in the shooting screen displayed on the touch panel 1506, and interrupts. Reception of a timing instruction operation input is started (step S48). Then, the user terminal 1500 displays the thumbnail 22 in association with the position of the seek bar 20 indicated by the automatically selected candidate timing (step S50). At this stage, the position of the slider 21 is displayed in alignment with the first candidate timing that is automatically retroactively searched.

Next, the user terminal 1500 starts warning control (step S52).
Specifically, from the camera work data 550, the position and orientation of the user terminal 1500 at the past time, which is now the restart timing, is read out and compared with the current measured value of the position and orientation of the user terminal 1500. Then, it is determined whether or not a predetermined shift condition is satisfied. If it is satisfied, the user terminal 1500 displays a predetermined warning display in the operation screen.

Next, the user terminal 1500 generates a conversion function based on the latest instruction timing (step S56), and starts the sequential update of the conversion function (step S58). Further, the user terminal 1500 starts control for rewinding the music reproduction control and the motion control of the subject object 4 to the timing indicated by the latest instruction timing (step S60).

Further, after step S56 to step S58, when the position/orientation of the user terminal 1500 is changed or the zoom magnification of the camera 1520 is changed, the conversion function is automatically updated.

Further, by the step S60, thereafter, the reproduction of the music and the operation control of the virtual three-dimensional space including the subject object 4 are prepared so that they can be restarted at any time from the latest instruction timing.

18, when the user terminal 1500 detects a touch on the recording restart operation icon 15, the user terminal 1500 determines that the input of the recording restart operation is detected (YES in step S70), and the warning control ends. Yes (step S78). Then, the user terminal 1500 restarts the recording of the camera work data 550 by applying the latest conversion function indicated by the conversion function data 590 (see FIG. 15) (step S80; see FIG. 10), the reproduction of the music and the subject object 4 The operation control of is restarted (step S82). It should be noted that the data after the interruption timing of the camera work data 550 is overwritten by the data whose recording has been restarted. Then, the user terminal 1500 performs control for notifying the user of the resumption of recording (step S84), and ends the recording interruption resuming process.

It should be noted that, even when the recording suspension/resumption processing is being executed, when the input of the end operation is detected (YES in step S90), the user terminal 1500 ends the processing and ends the recording of the camera work data 550. ..

Returning to FIG. 16, when the input of a predetermined end operation is detected (YES in step S18), the user terminal 1500 ends the recording of the camera work data 550 (step S100), and the music reproduction and the object object 4 are performed. The control is finished (step S102), and the series of processes is finished.

The user terminal 1500 uploads the recorded camera work data 550 and the like to the server system 1100 when it detects the input of a predetermined upload operation (see FIG. 6 ). Then, upon detecting the input of the predetermined augmented reality image generation request operation, the user terminal 1500 transmits the predetermined generation request to the server system 1100.

Upon receiving the generation request, the server system 1100 can create moving image data of the augmented reality image using the camera work data 550 uploaded from the user terminal 1500, which is the transmission source, and can provide the moving image data to the user terminal 1500. (See FIG. 7).

As described above, according to the present embodiment, it is possible to easily create a realistic augmented reality moving image having an impression that the user himself/herself has taken a video camera and photographed it.
In addition, recording camera work data can be interrupted and resumed. Therefore, the user can arbitrarily pause and resume the recording of the camera work, and it becomes easy to record the desired camera work data. For example, even if the posture becomes uncomfortable during the recording of the camera work and the intended camera work cannot be continued, the recording can be temporarily stopped. Then, after the posture is corrected, the recording of the camera work can be resumed. You don't have to record again from the beginning. Since the recording can be paused, it is possible to advance the recording one by one.

Then, when the recording is restarted, the computer determines the position and the posture of the virtual camera when the recording is started in response to the recording restart instruction at the given interruption timing of the camera work data before the stop by the stop instruction. It is possible to automatically perform the correction adapted to the position and orientation of the virtual camera, and restart the recording of the camera work data after the recording restart instruction as the camera work data that continues after the interruption timing of the camera work data before the stop. In other words, even if there is a difference between the position of the computer at the time of interruption and the position of the computer at the time of restart, it is possible to record a series of camera work data that does not cause a position jump before and after the interruption.
According to this embodiment, it is possible to provide a new technique for creating a video of augmented reality or virtual reality.

[Modification]
Although the embodiments to which the present invention is applied have been described above, the embodiments to which the present invention can be applied are not limited to the above-described embodiments, and constituent elements can be appropriately added, omitted, or changed.

(Modification 1)
For example, a function of changing the conversion function or a function of changing the application effect of the conversion function according to the passage of time can be added to the above embodiment.
Specifically, the suspension/resumption control unit 222 may be configured to change the conversion function that gradually reduces the application effect of the conversion function to the subsequent camera work data as time passes. More specifically, for example, focusing on the position of the virtual camera, a vector from the position of the virtual camera at the time of restart to the position of the virtual camera at the interruption timing is obtained and used as a conversion function. After the recording is resumed, the virtual camera movement trajectory data 577 (see FIG. 5) is recorded by applying this vector to the virtual space of the virtual camera obtained from the relative position of the user terminal 1500 with respect to the AR marker 5. I will. On the other hand, the interruption/resumption control unit 222 may change the magnitude of the vector so as to gradually decrease toward “0” according to the lapse of time from the resumption of recording.

Also, as an example of a function that changes the application effect of a conversion function over time, the conversion function is fixed while the coefficient is multiplied when the conversion function is applied, and this coefficient is restarted for recording. It is also possible to adopt a configuration in which it is reduced (for example, reduced from coefficient=1 to coefficient=0) in accordance with the passage of time from.

(Modification 2)
Further, in the above-described embodiment, the restart of the recording of the camera work data 550 is configured such that the interruption timing is set in the camera work data 550 recorded in the latest past, and the camera work data that continues after the interruption timing is recorded. The target for setting the interruption timing is not limited to the camera work data 550 recorded in the latest past.

Specifically, for example, between step S44 and step S46 in the above-described embodiment, a selection operation input of data for which interruption timing is to be set is received from the camera work data 550 recorded in the past. Add steps. Then, after the step, the interruption timing may be set for the camera work data 550 selected in the step, and the subsequent camera work data may be recorded.

(Modification 3)
Further, regarding the subject object, although the number of subject objects 4 is illustrated as one in the above embodiment, a plurality of subject objects may be used at the same time. Further, although the subject object 4 is a virtual character imitating the form of a person, the present invention is not limited to this. The subject object 4 may be an animal, a plant, a landscape, or the like. Depending on the subject object, reproduction of music and operation control synchronized with music can be omitted as appropriate.

(Modification 4)
Further, in the above-described embodiment, an example of creating a moving image of augmented reality (AR) images has been described, but the same embodiment is also applied to creating a moving image of virtual reality (VR) images. Can be applied. The “augmented reality” in the description of the above embodiment may be appropriately read as “virtual reality”.

(Modification 5)
In the above embodiment, the AR marker 5 is used to measure the position and orientation of the user terminal 1500 of the user terminal 1500, but the position and orientation may be measured using markerless AR technology.

(Modification 6)
Further, in the above-described embodiment, the example in which the moving image of the augmented reality image is generated by the server system 1100 is shown, but a configuration in which this is performed by the user terminal 1500 is also possible.

(Modification 7)
Further, in the above embodiment, the operation icon displayed on the shooting screen can be appropriately selected according to the situation (see FIGS. 8, 10, 11, and 12). For example, only icons that can be operated by the user at that time may be displayed according to the execution status of the recording process. Specifically, the recording stop operation icon 14 and the recording end operation icon 16 are displayed during recording, and the recording restart operation icon 15 is displayed instead of the recording stop operation icon 14 during pause.

Further, various operation inputs may be accepted without using the operation icons on the shooting screen. For example, various operations may be instructed by an input by a gesture using a touch track on the touch panel, an operation may be instructed by an operation input by a physical button of the user terminal 1500, or a temporary stop by touching the screen ( It is also possible to instruct a pause instruction by touching the screen during recording) and a restart of recording (a restart instruction by touching the screen during the pause).

(Modification 8)
Further, in the above embodiment, the camera work day 550 after the recording restart instruction is recorded using the conversion function indicated by the conversion function data 590. That is, a correction process for adjusting the position and orientation of the virtual camera when recording is started according to the recording restart instruction to the position and orientation of the virtual camera at a given interruption timing of the camera work data before the stop by the recording stop instruction. Is performed when the camera work data 550 is recorded. However, the correction process may be performed without performing the correction process when recording the camera work data 550, and when performing the control of the virtual camera C2 as the augmented reality moving image data 584 is generated by the server system 1100.

In this case, the interruption/resumption control unit 222 stores the conversion function data 590 in time series in association with the camera work data 550, and stores the conversion function data 590 in chronological order. It may be uploaded together with 590 (see FIG. 6). That is, the conversion function may be recorded in time series in association with the time axis of the camera work data so that the conversion function can be referred to when correction processing is necessary.

(Modification 9)
Further, as an example of the automatic stop of recording, a case has been exemplified in which a rapid change in the position or posture of the user terminal 1500 is detected, but other automatic stop may be realized.

For example, the recording may be temporarily stopped when there is a notification from the OS (operating system) of the user terminal 1500 (power down, notification from another application, incoming call, etc.). Further, when the user terminal 1500 shifts to the sleep mode based on the operation input of the user 2, the user terminal 1500 may be temporarily stopped. Also, the user terminal 1500 may suspend recording when a switching operation is input to another application.

2... User 4... Subject object 5... AR marker 10... Monitor display 13... Recording start operation icon 14... Recording stop operation icon 15... Recording restart operation icon 16... Recording end operation icon 19... Automatic recording stop notification 20... Seek bar 21... Slider 22... Thumbnail 30... Live image 32... Virtual space image at instruction timing 34... Judgment result notification 100... Operation input unit 102... Positioning unit 106... Acceleration measuring unit 108... Posture measuring unit 120... Imaging unit 200... Terminal processing unit 202 ... camera control unit 210... camera work record management unit 212... virtual space setting control unit 213... measurement data acquisition control unit 214... virtual camera control unit 216... operation instruction control unit 220... camera work data generation unit 222... suspend/resume control unit 224... Transformation function generation unit 230... Virtual space image generation control unit 232... Virtual space image recording control unit 234... Improper change automatic stop unit 236... Notification control unit 240... Interruption timing setting unit 242... Candidate timing presenting unit 500... Terminal storage Part 501... Terminal program 502... Camera work record management program 550... Camera work data 551... Shooting ID
560... Real shooting condition data 561... Real camera attitude data 563... Real camera acceleration data 565... Real zoom control data 567... Real movement trajectory data 570... Virtual shooting condition data 571... Virtual camera attitude data 573... Virtual camera acceleration data 575... Virtual camera shooting angle of view data 577... Virtual camera movement trajectory data 580... Real image recording data 582... Virtual space image recording data 584... Augmented reality moving image data 590... Transformation function data 1000... Moving image generating system 1100... Server system 1500... User terminal 1550... Control board 1555... Positioning module 1556... 3-axis acceleration sensor 1557... 3-axis gyro C1... Virtual camera C2... Virtual camera Pe... Interruption timing position Ps... Start point position W11... Shooting screen

Claims (9)

A portable computer,
Virtual space setting means for setting a virtual three-dimensional space,
Virtual camera control means for controlling the position and orientation of the virtual camera that captures the virtual three-dimensional space based on the position and orientation of the computer in the physical space,
Camera work data generation means for recording camera work data capable of reproducing the position and orientation of the virtual camera in the virtual three-dimensional space continuously controlled by the virtual camera control means,
Function as
The camera work data generation means,
The position and orientation of the virtual camera at the time of starting recording in response to a given recording restart instruction are defined as the position of the virtual camera at a given interruption timing of the camera work data before the stop by the given recording stop instruction and An interruption/resumption control unit that records the camera work data after the recording restart instruction by adapting to the posture as camera work data that continues after the interruption timing of the camera work data before the stop,
Has,
program. The camera work data generation means is data in which the relative position and relative attitude of the computer are recorded in time series with reference to a given position set in the real space, or the position and attitude of the virtual camera are time series. Control the recording of data including at least the data recorded in
The program according to claim 1. The suspension/resumption control means is
A conversion function is generated for adapting the position and orientation of the virtual camera when recording is started in response to the recording restart instruction to the position and orientation of the virtual camera at the interruption timing of the camera work data before the stop. Conversion function generating means,
And recording the camera work data after the recording restart instruction using the conversion function, or storing the conversion function in time series,
The program according to claim 1. The suspension/resumption control means gradually reduces the application effect of the conversion function to the camera work data after the recording restart instruction, as time passes.
The program according to claim 3. An improper change that detects that the measurement data satisfies a given improper change condition that indicates an improper change when capturing the virtual three-dimensional space, and gives the recording stop instruction in response to the detection. When automatic stop means,
Notification control means for notifying the user of the stop of recording by the automatic stop means at the time of inappropriate change,
The program according to claim 1, which causes the computer to function as a computer. Candidate timing presenting means for automatically selecting a candidate timing of the interruption timing and performing control for presenting to the user an image of the virtual three-dimensional space photographed at the position and orientation of the virtual camera at the candidate timing.
The program according to claim 1, which causes the computer to function as a computer. When recording is started in response to the recording restart instruction, the position and orientation of the virtual camera based on the measurement data, and the position and orientation of the virtual camera at the interruption timing based on the camera work data before the stop A warning control means for warning the user when the difference between
The program according to claim 1, which causes the computer to function as a computer. A camera work data generation method executed by a portable computer,
A virtual space setting step for setting a virtual three-dimensional space,
A measurement data acquisition step of acquiring measurement data obtained by measuring the position and orientation of the portable computer in the real space;
A virtual camera control step for continuously controlling the position and orientation of the virtual camera that captures the virtual three-dimensional space based on the acquired measurement data;
A camera work data generation step of recording camera work data capable of reproducing the position and orientation of the virtual camera in the continuously controlled virtual three-dimensional space;
Including
The camerawork data generation step,
The position and orientation of the virtual camera at the time of starting recording in response to a given recording restart instruction are defined as the position of the virtual camera at a given interruption timing of the camera work data before the stop by the given recording stop instruction and By adapting to the posture, the step of recording the camera work data after the recording restart instruction is recorded as camera work data that continues after the interruption timing of the camera work data before the stop,
Camera work data generation method. A portable electronic device,
Virtual space setting means for setting a virtual three-dimensional space,
Virtual camera control means for controlling the position and orientation of the virtual camera that captures the virtual three-dimensional space based on the position and orientation of the electronic device in the real space;
Camera work data generation means for recording camera work data capable of reproducing the position and orientation of the virtual camera in the virtual three-dimensional space continuously controlled by the virtual camera control means;
Equipped with
The camera work data generation means,
The position and orientation of the virtual camera at the time of starting recording in response to a given recording restart instruction are defined as the position of the virtual camera at a given interruption timing of the camera work data before the stop by the given recording stop instruction and A means for recording the camera work data after the recording restart instruction by adapting the posture as camera work data that continues after the interruption timing of the camera work data before the stop,
Has,
Electronics.

Images