JP5568610B2 - Augmented reality system, video composition device, video composition method, and program - Google Patents

Description

Augmented reality system for providing a combined image of the video and computer graphics objects, the augmented reality system is Ru with movies image synthesis apparatus, as well as movies image synthesis method and a program that by the the augmented reality system.

  In recent years, a technique called “augmented reality (AR)” that synthesizes an image of a real space captured by an imaging device and an object of a virtual space expressed by computer graphics has been studied (for example, non-patented). Reference 1).

  Specifically, it is realized by generating a composite video that realizes augmented reality by the video composition device combining the video captured by the imaging device and the image obtained by rendering the object in the virtual space. At this time, the video composition device changes the camera information in the virtual space based on information acquired from a posture sensor such as a rotary encoder attached to the image capture device, thereby causing the object to follow the movement of the image capture device. Can be generated. In addition, the imaging device captures an image including a predetermined marker, and the video composition device identifies the position of the marker from the video, whereby a predetermined object can be displayed at a place where the marker exists. The marker is a mark for attaching to a subject and specifying its three-dimensional coordinates.

  As a method for controlling the movement of an object in such an augmented reality system, a method for determining the movement of the object in advance, a method for causing the object to follow the marker included in the video, and a method for applying the movement obtained by motion capture to the object Etc.

Kazunari Kojima, 4 others, construction of character display system using real-time transmission technology, "Information Processing Society Research Report. Graphics and CAD Research Report", Information Processing Society of Japan, June 20, 2011

  In recent years, there is a desire to interactively control an augmented reality system. That is, there is a demand for the viewer of the composite video to control the object included in the composite video. However, the above-described method for controlling the movement of an object does not accept an input from a viewer, and there is a problem that interactive control cannot be performed. In addition, although it is conceivable to apply motion capture to the viewer of the composite video, there is a problem that the system becomes large.

The present invention has been made in order to solve the above-described problems, and includes a video composition device that generates a composite video image of a video image captured by an imaging device and a computer graphics object, and a plurality of terminals that display the composite video image. An augmented reality system comprising: a video reception unit that receives a composite video from the video synthesis device; a video display unit that displays a composite video received by the video reception unit; and the video An operation unit that receives an input of operation information indicating an operation on the composite video displayed on the display unit; and an operation information transmission unit that transmits the operation information received by the operation unit to the video composition device. It is based on the operation information receiving unit from a plurality of terminal devices to receive the operation information, the operation information the operation information receiving unit has received from the plurality of terminal devices An object determination unit for changing the position and shape in the virtual space of the objects have the video combining unit, wherein the imaging device arranged in the virtual space object is synthesized video imaged, synthesis the image synthesizing unit has generated A video transmission unit configured to transmit the video to the plurality of terminal devices.

  In the present invention, when the marker is present within a predetermined range around the two-dimensional coordinate indicated by the operation information in the image captured by the imaging device, the coordinate specifying unit displays the three-dimensional coordinate where the marker exists. In particular, the object determination unit places the object at a position indicated by the three-dimensional coordinates specified by the coordinate specification unit.

  Further, in the present invention, the coordinate specifying unit is configured to generate the virtual corresponding to the two-dimensional coordinate indicated by the operation information based on the two-dimensional coordinate indicated by the operation information and a display resolution of a terminal device that is a transmission source of the operation information. It is characterized by specifying three-dimensional coordinates in space.

Further, in the present invention, the video composition device includes a resolution acquisition unit that acquires the display resolution of the terminal device from the plurality of terminal devices, and the video transmission unit generates the composite video generated by the video composition unit, The resolution is acquired by the resolution acquisition unit and transmitted to the terminal device.

  In the present invention, the operation unit receives an input of a touch operation on the touch panel and specifies operation information corresponding to a locus of the touch operation.

In addition, the present invention is a video composition device that generates a composite image of a video image captured by an imaging device and a computer graphics object, and an operation for receiving operation information indicating an operation on the composite image from a plurality of terminal devices. An information receiving unit, an object determining unit that changes the position and shape of the object in the virtual space based on the operation information received from the plurality of terminal devices by the operation information receiving unit, and an object arranged in the virtual space A video composition unit that synthesizes the video imaged by the imaging device, and a video transmission unit that transmits the composite video generated by the video composition unit to the plurality of terminal devices.

Further, according to the present invention, a video composition device that generates a composite video of a video imaged by an imaging device and a computer graphics object receives operation information indicating an operation on the composite video from a plurality of terminal devices, and the video The composition device changes the position and shape of the object in the virtual space based on the operation information received from the plurality of terminal devices, and the image composition device captures the object arranged in the virtual space. The video synthesizing method is characterized in that the video synthesizing device transmits the generated synthesized video to the plurality of terminal devices.

According to another aspect of the present invention, there is provided an operation information receiving unit that receives, from a plurality of terminal devices, operation information indicating operation on a composite video of a video image captured by an imaging device and a computer graphics object, and the operation information receiving unit. An object determination unit that changes the position and shape of the object in the virtual space based on the operation information received from the plurality of terminal devices, and a video that synthesizes the object arranged in the virtual space with the video captured by the imaging device It is a program for causing a synthesizing unit and a synthesized video generated by the video synthesizing unit to function as a video transmitting unit that transmits to the plurality of terminal devices.

  According to the present invention, a synthesized video is displayed on a terminal device, and the video synthesizer receives an operation from the terminal device to control an object. As a result, the viewer of the composite video can control the objects included in the composite video.

It is the schematic which shows the structure of the augmented reality system by one Embodiment of this invention. It is a schematic block diagram which shows the software structure of a terminal device and a video synthesizing | combining apparatus. It is a figure which shows the example of the user interface of a terminal device. It is a figure which shows the example of the information which an operation information storage part memorize | stores. It is a sequence diagram which shows operation | movement of the augmented reality system by this embodiment. It is the schematic of the 1st example of control by an augmented reality system. It is a flowchart which shows operation | movement of the 1st control example by a video synthesizing | combining apparatus. It is the schematic of the 2nd control example by an augmented reality system. It is a flowchart which shows operation | movement of the 2nd control example by a video synthesizing | combining apparatus. It is the schematic of the 3rd control example by an augmented reality system. It is a flowchart which shows operation | movement of the 3rd control example by a video synthesizing | combining apparatus. It is the schematic of the 4th control example by an augmented reality system. It is a flowchart which shows operation | movement of the 4th control example by a video synthesizing | combining apparatus. It is the schematic of the 5th control example by an augmented reality system. It is a flowchart which shows operation | movement of the 5th control example by a video synthesizing | combining apparatus. It is the schematic of the 6th control example by an augmented reality system. It is a flowchart which shows operation | movement of the 6th control example by a video synthesizing | combining apparatus. It is the schematic of the 7th control example by an augmented reality system. It is a flowchart which shows operation | movement of the 7th control example by a video synthesizing | combining apparatus. It is the schematic of the 8th control example by an augmented reality system.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
FIG. 1 is a schematic diagram showing the configuration of an augmented reality system according to an embodiment of the present invention.
The augmented reality system includes a plurality of terminal devices 100, an imaging device 200, and a video composition device 300. The video composition device 300 and the terminal device 100 are connected via a network.

  The terminal device 100 receives the composite video from the augmented reality system, displays the video, accepts an operation from the user, and transmits operation information indicating the operation to the video composition device 300 by TCP (Transmission Control Protocol). To do. The terminal device 100 can be realized by a PC (Personal Computer), a tablet terminal, a mobile phone, a smartphone, a game device, or the like. However, in the present embodiment, the terminal device 100 is a tablet terminal including a touch panel. Explained as an example.

  The imaging device 200 captures an image of real space and inputs the captured image to the image composition device 300. In addition, the imaging apparatus 200 includes an attitude sensor that detects the attitude of the imaging apparatus 200 and a depth sensor that detects the depth of the captured video.

  The video synthesizing device 300 generates a synthesized video obtained by synthesizing a computer graphics object with the video imaged by the imaging device 200, and transmits the synthesized video to the terminal device 100 by UDP (User Datagram Protocol).

  By providing the augmented reality system having such a configuration, each viewer of the composite video can operate an object included in the composite video.

FIG. 2 is a schematic block diagram illustrating software configurations of the terminal device 100 and the video composition device 300.
The terminal device 100 includes a video reception unit 101, a video display unit 102, an object display unit 103, an operation unit 104, and an operation information transmission unit 105.
The video receiving unit 101 receives the composite video from the video composition device 300.
The video display unit 102 displays the composite video received by the video receiving unit 101 on the touch panel.
The object display unit 103 displays an operation pane for operating an object included in the composite video on the touch panel in accordance with the operation of the operation unit 104.
The operation unit 104 receives an operation by touch input on the touch panel from a user.
The operation information transmission unit 105 transmits operation information indicating the operation accepted by the operation unit 104 to the video composition device 300. The operation information includes an operation type and option information. The operation type is information indicating the type of operation (for example, creation, movement, deformation, deletion, etc.) of an object of a composite video. The option information is information required according to the operation type.

FIG. 3 is a diagram illustrating an example of a user interface of the terminal device 100.
On the touch panel of the terminal device 100, a composite video received by the video receiving unit 101 and a plurality of buttons for specifying the operation type are displayed. The operation unit 104 determines the operation type according to the button touched by the user. The operation unit 104 generates operation information when the user touches a button or when a predetermined touch operation is performed after the button is touched.

Returning to FIG. 2, the configuration of the video composition device 300 will be described next.
The video composition device 300 includes an operation information reception unit 301, an operation information storage unit 302, a coordinate specification unit 303, a counting unit 304, an object determination unit 305, a virtual space calculation unit 306, a video acquisition unit 307, a video composition unit 308, and resolution acquisition. A unit 309, a video transmission unit 310, a change command input unit 311, and an operation information change unit 312.

The operation information receiving unit 301 receives operation information from the terminal device 100.
The operation information storage unit 302 stores operation types, option information used for operations, and details of operations on objects in association with each other.

FIG. 4 is a diagram illustrating an example of information stored in the operation information storage unit 302.
The operation information storage unit 302 stores option information and operation contents in association with the operation type.
In the present embodiment, the operation information indicating the operation type “1” includes touch coordinates as option information. When the operation information is received, an operation of “display a firework object at a position corresponding to the touch coordinates” is executed.
The operation information indicating the operation type “2” includes touch coordinates as option information. When the operation information is received, an operation “execute an action associated with the touched object” is executed.
The operation information indicating the operation type “3” includes touch coordinates as option information. When the operation information is received, an operation of “display effect object on touched marker” is executed. The marker is a mark for attaching to a subject and specifying its three-dimensional coordinates.
The operation information indicating the operation type “4” does not include option information. And when the said operation information is received, operation which "inflates a balloon according to the frequency | count of reception of operation information" is performed.

The operation information indicating the operation type “5” does not include option information. When the operation information is received, an operation of “display a support message when the number of operation information received during a predetermined time is 100 or more” is executed.
The operation information indicating the operation type “6” includes gesture information as option information. Then, when the operation information is received, an operation of “accepting a gesture“ ◯ ”and a gesture“ × ”for a predetermined time and displaying an object associated with a large number of objects” is executed.
The operation information indicating the operation type “7” does not include option information. Then, when the operation information is received, an operation “change the displayed effect object” is executed.
The operation information indicating the operation type “8” includes joint ID and angle information as option information. When the operation information is received, an operation of “deform the joint angle of the target object to the angle indicated by the angle information” is executed.
In the example of FIG. 4, the contents of the operation are written in text, but the operation information storage unit 302 actually stores a command script to be executed by the object determination unit 305.

Returning to FIG. 2, when the operation information includes touch coordinates, the coordinate specifying unit 303 specifies coordinates in the virtual space indicated by the touch coordinates.
The counting unit 304 counts the number of times operation information indicating a predetermined operation type is received.

The object determination unit 305 specifies the content of the operation to be performed on the operation information received by the operation information reception unit 301 from the operation information storage unit 302, and the operation information, the coordinates and counts specified by the coordinate specification unit 303 Based on the number of receptions counted by the unit 304, the position and shape of the object in the virtual space are determined.
The virtual space calculation unit 306 controls objects in the virtual space based on the position and shape determined by the object determination unit 305.

The video acquisition unit 307 acquires the video captured by the imaging device 200, the posture information of the imaging device 200, and the depth information of the video from the imaging device 200.
The video composition unit 308 synthesizes the video imaged by the imaging apparatus 200 and the object in the virtual space calculated by the virtual space calculation unit 306 to generate a composite video.

The resolution acquisition unit 309 acquires the resolution of the touch panel of the terminal device 100.
The video transmission unit 310 converts the synthesized video synthesized by the video synthesis unit 308 into a video having the resolution acquired by the resolution acquisition unit 309, and transmits the synthesized video to the terminal device 100.

The change command input unit 311 receives an input of a change command for changing the content of the operation associated with the operation type from the administrator. The change command includes the operation type to be changed and the content of the operation after the change.
The operation information changing unit 312 rewrites the operation content associated with the operation type of the change command in the operation information storage unit 302 to the operation content of the change command.
That is, when the administrator inputs a change command to the video composition device 300 at the timing of switching the scene of the video captured by the imaging device 200, the user can execute an operation of the object that matches the scene. .

Next, the overall operation of the augmented reality system according to the present embodiment will be described.
FIG. 5 is a sequence diagram showing the operation of the augmented reality system according to the present embodiment.
First, when the terminal device 100 accesses the video composition device 300, the resolution acquisition unit 309 of the video composition device 300 acquires the resolution of the touch panel from the terminal device 100 (step S1).
Thereafter, the following processes of steps S2 to S13 are executed until the video imaged by the imaging device 200 is completed.

  The video synthesizing unit 308 of the video synthesizing device 300 generates a synthesized video obtained by synthesizing the video acquired by the video acquiring unit 307 and the object calculated by the virtual space calculating unit 306 (step S2). The following is an example of a procedure in which the video composition unit 308 generates a composite video. First, the video composition unit 308 determines the parameters of the rendering camera that renders the object calculated by the virtual space calculation unit 306 based on the attitude information of the imaging device 200 acquired by the video acquisition unit 307. The rendering camera parameters include, for example, the position of the rendering camera, the line-of-sight direction, and the angle of view.

  Next, the video composition unit 308 applies the depth information detected by the depth sensor of the imaging device 200 to each pixel of the video acquired by the video acquisition unit 307. Next, the video composition unit 308 renders the objects calculated by the virtual space calculation unit 306 whose depth from the determined camera position is lower than the depth of the corresponding pixel of the video (present on the near side). To do. Thereby, the video composition unit 308 can generate a composite video.

  Next, the video transmission unit 310 changes the size of the synthesized video generated by the video synthesis unit 308 according to the resolution acquired by the resolution acquisition unit 309 (step S3). Next, the video transmission unit 310 transmits the composite video whose size has been changed to the terminal device 100 (step S4).

When the video synthesizing device 300 transmits the synthesized video, the video receiving unit 101 of the terminal device 100 receives the synthesized video from the video synthesizing device 300 (step S5). Next, the video display unit 102 displays the video received by the video receiving unit 101 on the touch panel (step S6).
Next, the operation unit 104 determines whether or not there is a touch operation input from the user (step S7). When the operation unit 104 determines that there is no touch operation input (step S7: NO), the terminal device 100 waits for reception of the next composite video.

  On the other hand, when it is determined that the touch operation has been input (step S7: YES), the operation unit 104 receives the touch operation input and generates operation information (step S8). Note that what touch operation input the operation unit 104 receives and what operation information is generated differ depending on the operation type. When the operation unit 104 generates operation information, the operation information transmission unit 105 transmits the generated operation information to the video composition device 300 (step S9).

  The operation information receiving unit 301 of the video composition device 300 determines whether operation information has been received from the terminal device 100 (step S10). When the terminal device 100 accepts the operation in step S7 and transmits the operation information, the operation information receiving unit 301 of the video composition device 300 receives the operation information (step S10: YES). In this case, the object determining unit 305 reads out the operation content associated with the operation type of the operation information received by the operation information receiving unit 301 from the operation information storage unit 302 (step S11). Then, the object determination unit 305 determines the position and shape of the object according to the content of the read operation (step S12).

  When the object determination unit 305 executes the content of the operation or when the operation information receiving unit 301 does not receive the operation information because the terminal device 100 did not accept the operation in step S7 (step S10: NO), the virtual space The calculation unit 306 updates the position and shape of the object in the virtual space (step S13). Then, returning to step S2, the video composition unit 308 generates a composite video.

  As described above, according to the present embodiment, the video composition device 300 displays the composite video on the terminal device 100, and the video composition device 300 receives an operation from the terminal device 100, thereby controlling the object. As a result, the viewer of the composite video can control the objects included in the composite video.

  Further, according to the present embodiment, the video composition device 300 acquires the resolution of the touch panel from the terminal device 100 and transmits the composite video that matches the resolution to the terminal device 100. Thereby, a suitable synthetic | combination image | video can be provided with respect to the some terminal from which the resolution of a touchscreen differs.

  Next, some examples of specific object control will be described.

《Display fireworks object》
FIG. 6 is a schematic diagram of a first control example by the augmented reality system.
When the user interface shown in FIG. 3 is displayed on the touch panel of the terminal device 100, when the user touches the “fireworks” button as shown in FIG. 6A, the operation unit 104 of the terminal device 100 The operation is identified as an operation of the operation type (here, operation type “1”) associated with the “fireworks” button. As shown in FIG. 4, since the option information of the operation type “1” includes touch coordinates, the operation unit 104 receives input of coordinates for displaying the firework object next.

As shown in FIG. 6B, when the user touches within the range of the touch panel where the synthesized video is displayed, the operation unit 104 specifies the touched coordinates (two-dimensional coordinates). Thereby, the operation unit 104 sets the operation type to “1”, and generates operation information including the touch coordinates specified in the option information. Then, the operation information transmission unit 105 transmits the operation information generated by the operation unit 104 to the video composition device 300.
As a result, the video composition device 300 generates a composite video in which the firework objects are arranged at the touched coordinates, as shown in FIG.

Here, the operation of the video composition device 300 when the operation information of the operation type “1” is received will be described.
FIG. 7 is a flowchart showing the operation of the first control example by the video composition device 300.
When the object determination unit 305 of the video composition device 300 reads the content of the operation type “1” in step S <b> 11 described above, the coordinate specification unit 303 displays the touch coordinates included in the operation information received by the operation information reception unit 301. Based on the resolution acquired by the resolution acquisition unit 309, the relative position of the touched coordinate is specified (step S101). Specifically, the calculation is performed by dividing the X coordinate and Y coordinate of the touch coordinates by the resolution in the X direction and the resolution in the Y direction, respectively.

  Next, the coordinate specifying unit 303 sets a half-line equation that is shifted from the line-of-sight direction by the angle obtained by multiplying the angle of view of the rendering camera by the relative position, with the position of the rendering camera in the virtual space calculation unit 306 as the origin. calculate. The half line corresponds to a set of points corresponding to the touch coordinates included in the operation information (step S102). Next, the coordinate specifying unit 303 specifies three-dimensional coordinates having a predetermined length from the position of the rendering camera among the calculated half lines as coordinates for arranging the firework object (step S103). Here, examples of the predetermined length include an average value of the depth of the entire imaging range, a depth of a position where a specific object exists, and the like from the depth information acquired by the video acquisition unit 307.

  Next, the object determination unit 305 generates a firework object and places the firework object at the three-dimensional coordinates specified by the coordinate specification unit 303 in the virtual space (step S104). Then, when the virtual space calculation unit 306 updates the object in the virtual space in step S13 described above, the video composition unit 308 may generate a composite video including the firework object as illustrated in FIG. it can.

<< Action instructions for objects >>
FIG. 8 is a schematic diagram of a second control example by the augmented reality system.
When the user interface shown in FIG. 3 is displayed on the touch panel of the terminal device 100, when the user touches the “action” button as shown in FIG. 8A, the operation unit 104 of the terminal device 100 The operation is identified as an operation of the operation type (here, operation type “2”) associated with the “action” button. As shown in FIG. 4, since the option information of the operation type “2” includes touch coordinates, the operation unit 104 receives input of coordinates where an object to be executed next is present.

As shown in FIG. 8B, when the user touches the touch panel within the range where the synthesized video is displayed, the operation unit 104 specifies the touched coordinates. Thereby, the operation unit 104 sets the operation type to “2” and generates operation information including the touch coordinates specified in the option information. Then, the operation information transmission unit 105 transmits the operation information generated by the operation unit 104 to the video composition device 300.
As a result, as shown in FIG. 8C, the video composition device 300 generates a composite video in which an action is performed on an object existing at the touched coordinates.

Here, the operation of the video composition device 300 when the operation information of the operation type “2” is received will be described.
FIG. 9 is a flowchart showing the operation of the second control example by the video composition device 300.
When the object determination unit 305 of the video composition device 300 reads the content of the operation type “2” in step S <b> 11 described above, the coordinate specification unit 303 displays the touch coordinates included in the operation information received by the operation information reception unit 301. Based on the resolution acquired by the resolution acquisition unit 309, the relative position of the touched coordinate is specified (step S201).

  Next, the coordinate specifying unit 303 sets a half-line equation that is shifted from the line-of-sight direction by the angle obtained by multiplying the angle of view of the rendering camera by the relative position, with the position of the rendering camera in the virtual space calculation unit 306 as the origin. Calculate (step S202). Next, the coordinate specification unit 303 determines whether there is an object that intersects the calculated half line in the virtual space (step S303).

  When the coordinate specifying unit 303 determines that there is no object that intersects the half line (step S303: NO), the object determining unit 305 does not change the position and shape of any object, and ends the process. On the other hand, when it is determined that there is an object that intersects the half line (step S303: YES), the coordinate specifying unit 303 specifies the three-dimensional coordinates where the object exists. Then, the object determining unit 305 determines the position and shape of the object existing on the three-dimensional coordinates specified by the coordinate specifying unit 303 as the position and shape when the object is caused to execute a predetermined action (step S304). ).

  Then, when the virtual space calculation unit 306 updates the object in the virtual space in step S13 described above, the video composition unit 308 is in a state where the object is executing a predetermined action as shown in FIG. 8C. Can be generated.

《Display effect object on marker》
FIG. 10 is a schematic diagram of a third control example by the augmented reality system.
When the user interface shown in FIG. 3 is displayed on the touch panel of the terminal device 100, when the user touches the “effect display” button as shown in FIG. 10A, the operation unit 104 of the terminal device 100 is displayed. The operation is identified as an operation of the operation type (here, operation type “3”) associated with the “effect display” button. As shown in FIG. 4, since the touch information is included in the option information of the operation type “3”, the operation unit 104 receives an input of a coordinate where a marker for displaying an effect next exists. The marker is a mark for attaching to a subject and specifying its three-dimensional coordinates.

As shown in FIG. 10B, when the user touches within the range where the synthesized video is displayed on the touch panel, the operation unit 104 specifies the touched coordinates. Thereby, the operation unit 104 sets the operation type to “3” and generates operation information including the touch coordinates specified in the option information. Then, the operation information transmission unit 105 transmits the operation information generated by the operation unit 104 to the video composition device 300.
As a result, as shown in FIG. 10C, the video composition device 300 generates a composite video in which a predetermined effect object is arranged at the position of a marker existing around the touched coordinates.

Here, the operation of the video composition device 300 when the operation information of the operation type “3” is received will be described.
FIG. 11 is a flowchart showing the operation of the third control example by the video composition device 300.
When the object determination unit 305 of the video composition device 300 reads the content of the operation type “3” in step S <b> 11 described above, the coordinate specification unit 303 displays the touch coordinates included in the operation information received by the operation information reception unit 301. Based on the resolution acquired by the resolution acquisition unit 309, the relative position of the touched coordinate is specified (step S301).

  Next, the coordinate specifying unit 303 specifies the coordinates on the video acquired by the video acquisition unit 307 corresponding to the specified relative position (step S302). Here, the coordinate specifying unit 303 specifies the coordinates of the marker included in the video acquired by the video acquisition unit 307, and whether the marker exists around the coordinate specified in step S302 (for example, within a range of 100 pixels). It is determined whether or not (step S303).

  When the coordinate specifying unit 303 determines that there is no marker around the specified coordinates (step S303: NO), the object determining unit 305 ends the process without changing the position and shape of any object. On the other hand, if the coordinate specifying unit 303 determines that there is a marker around the specified coordinate (step S303: YES), the coordinate specifying unit 303 acquires the depth of the coordinate where the marker exists from the depth information acquired by the video acquisition unit 307 (step S303). S304).

  Next, the coordinate specifying unit 303 sets a half-line equation that is shifted from the line-of-sight direction by the angle obtained by multiplying the angle of view of the rendering camera by the relative position, with the position of the rendering camera in the virtual space calculation unit 306 as the origin. Calculate (step S305). Next, the coordinate specifying unit 303 specifies the three-dimensional coordinates corresponding to the position of the depth specified in step S304 from the position of the rendering camera among the calculated half lines as coordinates for arranging the effect object (step S306).

  Next, the object determination unit 305 generates an effect object and places the effect object in the three-dimensional coordinates specified by the coordinate specification unit 303 in the virtual space (step S307). Then, when the virtual space calculation unit 306 updates the object in the virtual space in step S13 described above, the video composition unit 308 may generate a composite video including the effect object as illustrated in FIG. it can. Note that after the effect object is added by this processing, the virtual space calculation unit 306 calculates the position of the effect so as to follow the position of the marker included in the video acquired by the video acquisition unit 307.

  Here, although a method has been described in which the effect is made to follow the subject included in the video acquired by the video acquisition unit 307 by using a marker that is attached to the subject and identifies the three-dimensional coordinates, the present invention is not limited thereto. For example, when the subject to be effected is a person, instead of attaching a marker to the subject, the bone structure of the subject may be estimated, and the position of the effect may be calculated based on the bone structure. As a method for estimating the bone structure of the subject, for example, a method of applying pattern matching to an image acquired by the image acquisition unit 307, or a three-dimensional coordinate of an obstacle existing in a space imaged by the imaging device 200 using an infrared sensor or the like. A method of calculating and extracting a subject from an obstacle based on the three-dimensional coordinates is exemplified.

《Action against balloon object》
FIG. 12 is a schematic diagram of a fourth control example by the augmented reality system.
When the user interface shown in FIG. 3 is displayed on the touch panel of the terminal device 100, when the user touches the “balloon” button as shown in FIG. 12A, the operation unit 104 of the terminal device 100 The operation is identified as an operation of the operation type (here, operation type “4”) associated with the “balloon” button, and operation information with the operation type “4” is generated. Then, the operation information transmission unit 105 transmits the operation information generated by the operation unit 104 to the video composition device 300.

  When the total number of touches of the “balloon” button in all the terminal devices 100 reaches the first threshold (for example, 10 times), the video composition device 300 displays the balloon object as shown in FIG. Generate inflated composite video. Further, when the total number of touches of the “balloon” button reaches a second threshold (for example, 100 times), the video composition device 300 displays a composite video obtained by rupturing the balloon object as shown in FIG. Generate.

Here, the operation of the video composition device 300 when the operation information of the operation type “4” is received will be described.
FIG. 13 is a flowchart showing the operation of the fourth control example by the video composition device 300.
When the object determination unit 305 of the video composition device 300 reads the operation type “4” in step S <b> 11 described above, the object determination unit 305 first has a balloon object in the virtual space calculated by the virtual space calculation unit 306. It is determined whether or not to perform (step S401). If the object determination unit 305 determines that there is no balloon object in the virtual space (step S401: NO), the process ends.

  On the other hand, when the object determination unit 305 determines that a balloon object is present in the virtual space (step S401: YES), the counting unit 304 sets the reception count of the operation type “4” stored in the internal memory to “1”. Add (step S402).

  Next, the object determination unit 305 determines whether or not the reception count of the operation information of the operation type “4” counted by the counting unit 304 has reached the first threshold (step S403). If the object determination unit 305 determines that the number of receptions has not reached the first threshold (step S403: NO), the object determination unit 305 ends the process without changing the position and shape of the balloon object. On the other hand, when the object determination unit 305 determines that the number of receptions has reached the first threshold (step S403: YES), the object determination unit 305 determines whether the number of receptions has reached the second threshold (step S404).

When the object determination unit 305 determines that the number of receptions has not reached the second threshold (step S404: NO), the position and shape of the balloon object are determined in advance as the state when the balloon object is inflated. Then, the shape is determined (step S405).
On the other hand, when the object determination unit 305 determines that the number of receptions has reached the second threshold (step S404: YES), the position and shape of the balloon object are determined in advance as the state when the balloon object bursts. The position and shape are determined (step S406).

Then, the virtual space calculation unit 306 updates the object in the virtual space in the above-described step S13, so that the video composition unit 308 has a balloon object state as shown in FIGS. 12B and 12C. A changed composite image can be generated.
In this way, by determining the position and shape of the object based on the value counted by the counting unit 304, the object to be displayed in the synthesized video even when the number of terminal devices 100 communicating with the video synthesizing device 300 is large. Can be avoided.

《Support action》
FIG. 14 is a schematic diagram of a fifth control example by the augmented reality system.
When the user interface shown in FIG. 3 is displayed on the touch panel of the terminal device 100, when the user touches the “support” button as shown in FIG. 14B, the operation unit 104 of the terminal device 100 The operation is identified as an operation of the operation type (here, operation type “5”) associated with the “support” button, and operation information with the operation type “5” is generated. Then, the operation information transmission unit 105 transmits the operation information generated by the operation unit 104 to the video composition device 300.

Here, a scene where the “support” button is touched will be described.
The video composition unit 308 of the video composition device 300 displays the characters “READY...” As shown in FIG. 14 (A), and after a predetermined time, the characters “CHEER!” As shown in FIG. 14 (B). Generate a composite image that displays. Then, when the number of touches of the “cheer” button reaches the predetermined threshold (for example, 100 times) at the timing when the characters “CHEER!” Are displayed on the composite video, the video composition device 300 is displayed in FIG. As shown in (C), a composite video in which predetermined effect objects are arranged is generated.

Here, the operation of the video composition device 300 when the operation information of the operation type “5” is received will be described.
FIG. 15 is a flowchart showing the operation of the fifth control example by the video composition device 300.
When the object determination unit 305 of the video composition device 300 reads the content of the operation type “5” in step S11 described above, the counting unit 304 includes the current time and the video composition unit 308 includes the characters “CHEER!”. It is determined whether the time is within a predetermined time limit (for example, 1 second) after the composite video is generated (step S501). If the counting unit 304 determines that the current time is not within the predetermined time limit (step S501: NO), the process ends.

  On the other hand, when the counting unit 304 determines that the current time is within the predetermined time limit (step S501: YES), the counting unit 304 adds 1 to the number of times of reception of the operation information of the operation type “5” stored in the internal memory. (Step S502).

  Next, the object determining unit 305 determines whether or not the number of times of receiving operation information of the operation type “5” counted by the counting unit 304 has reached a predetermined threshold (step S503). If the object determination unit 305 determines that the number of receptions has not reached the predetermined threshold (step S503: NO), the object determination unit 305 ends the process without changing the position and shape of any object.

  On the other hand, when the object determination unit 305 determines that the number of receptions has reached a predetermined threshold (step S503: YES), the object determination unit 305 generates an effect object and places the effect object at a predetermined position (step S504). Then, the virtual space calculation unit 306 updates the object in the virtual space in step S13 described above, so that the video composition unit 308 touches the “support” button at a predetermined timing as shown in FIG. When the predetermined number is reached, a composite video including an effect object can be generated.

《○ × gesture input》
FIG. 16 is a schematic diagram of a sixth control example by the augmented reality system.
When the user interface shown in FIG. 3 is displayed on the touch panel of the terminal device 100, when the user touches the “XX” button as shown in FIG. 16A, the operation unit 104 of the terminal device 100 is displayed. The operation is identified as an operation of the operation type (here, operation type “6”) associated with the “Ox” button. As shown in FIG. 4, since the option information of the operation type “6” includes gesture information, the operation unit 104 next receives an input of a gesture by a touch operation.

As shown in FIG. 16B, when the user performs a predetermined touch operation on the touch panel, the operation unit 104 identifies a trajectory of the touch operation, and a gesture in which the trajectory is predetermined (for example, ◯, ×). Of which gesture is applicable. Then, the operation unit 104 sets the operation type to “6” and generates operation information including gesture information indicating the gesture specified in the option information. Then, the operation information transmission unit 105 transmits the operation information generated by the operation unit 104 to the video composition device 300.
As a result, the video composition device 300 generates a composite video including an object indicating the result of the majority of inputs indicated by the gesture information, as shown in FIG.

Here, the operation of the video composition device 300 when the operation information of the operation type “6” is received will be described.
FIG. 17 is a flowchart showing the operation of the sixth control example by the video composition apparatus 300.
When the object determination unit 305 of the video composition device 300 reads the content of the operation type “6” in step S11 described above, the counting unit 304 reads the gesture information of the received operation information, and the corresponding gesture information (◯ or 1 is added to the number of receptions of (x) (step S601).
Next, the counting unit 304 determines whether or not the current time has reached the deadline for accepting operation information of the operation type “6” (step S602). When the counting unit 304 determines that the current time has not reached the deadline for accepting operation information of the operation type “6” (step S602: NO), the processing ends.

  On the other hand, when the counting unit 304 determines that the current time has reached the deadline for accepting operation information of the operation type “6” (step S602: YES), the object determining unit 305 indicates that the counting unit 304 indicates the number of acceptances. The gesture information having the largest number of received receptions is identified (step S603). Then, the object determining unit 305 generates an object (for example, a character object “YES” in FIG. 16C) corresponding to the specified gesture information, and places the effect object at a predetermined position (step S604). .

  Then, when the virtual space calculation unit 306 updates the object in the virtual space in step S13 described above, the video composition unit 308 displays a composite video including the object corresponding to the gesture information as illustrated in FIG. Can be generated.

《Change effect object》
FIG. 18 is a schematic diagram of a seventh control example by the augmented reality system.
When the user interface shown in FIG. 3 is displayed on the touch panel of the terminal device 100, when the user touches the “change effect” button as shown in FIG. 18A, the operation unit 104 of the terminal device 100 is displayed. The operation is identified as an operation of the operation type (here, operation type “7”) associated with the “change effect” button, and operation information with the operation type “7” is generated. Then, the operation information transmission unit 105 transmits the operation information generated by the operation unit 104 to the video composition device 300.
As a result, the video composition device 300 generates a composite video in which the effect object is changed, as shown in FIG.

Here, the operation of the video composition device 300 when the operation information of the operation type “7” is received will be described.
FIG. 19 is a flowchart showing the operation of the seventh control example by the video composition apparatus 300.
When the object determination unit 305 of the video composition device 300 reads the operation type “7” in step S11 described above, the object determination unit 305 first has an effect object in the virtual space calculated by the virtual space calculation unit 306. It is determined whether or not to perform (step S701). If the object determination unit 305 determines that there is no effect object in the virtual space (step S701: NO), the process ends.

  On the other hand, when the object determination unit 305 determines that an effect object exists in the virtual space (step S701: YES), the shape of the effect object is determined to be another predetermined shape (step S702).

  Then, when the virtual space calculation unit 306 updates the object in the virtual space in step S13 described above, the video composition unit 308 generates a composite video in which the effect object is changed as illustrated in FIG. Can do.

<Manipulating character objects>
FIG. 20 is a schematic diagram of an eighth control example by the augmented reality system.
When the user interface shown in FIG. 3 is displayed on the touch panel of the terminal device 100, when the user touches the “character operation” button as shown in FIG. The operation is specified as an operation of the operation type (here, operation type “8”) associated with the “character operation” button.

  Next, the object display unit 103 displays an operation pane including a character object included in the video received by the video receiving unit 101 on the touch panel. In the operation pane, a link line indicating the connection between the joints of the character object is displayed in a manner superimposed on the character object. The operation pane also includes a rotation button for rotating the character object.

  Here, as shown in FIG. 6B, when the user touches the character object on the operation pane displayed on the touch panel, the operation unit 104 displays the joint ID of the joint serving as a reference for the part of the touched character object. Is identified. When the user performs a drag operation on the character object, the operation unit 104 calculates a rotation angle of the joint corresponding to the movement amount of the drag operation. Then, the operation unit 104 sets the operation type to “8”, and generates operation information including the joint ID and the rotation angle specified in the option information. Then, the operation information transmission unit 105 transmits the operation information generated by the operation unit 104 to the video composition device 300.

Here, the operation of the video composition device 300 when the operation information of the operation type “8” is received will be described.
When the object determination unit 305 of the video composition device 300 reads the content of the operation of the operation type “8” in step S11 described above, the object determination unit 305 includes the joint ID included in the operation information among the joints of the corresponding character object. Is rotated by the angle indicated by the angle information included in the operation information. That is, the object determination unit 305 changes the position and shape of a part of the object dragged by the drag operation by the user to the position and shape after dragging. Then, the virtual space calculation unit 306 updates the object in the virtual space in step S13 described above, thereby generating a composite image in which the position and shape of the object are changed in synchronization with the user's operation.

  When the user touches the rotation button on the operation pane, the object display unit 103 increases or decreases the rotation angle of the rendering camera that draws the object in the operation pane in response to the touch, so that the character in the operation pane is displayed. Rotate the object. Accordingly, the user can perform an operation of changing the shape of the character object in the depth direction in addition to the height direction and the width direction. Here, an example in which the character object in the operation pane is rotated by touching the rotation button has been described. However, the present invention is not limited to this, and the configuration may include a button for changing other parameters such as the position and angle of view of the rendering camera. good.

As described above, the embodiment of the present invention has been described in detail with reference to the drawings. However, the specific configuration is not limited to the above, and various design changes and the like can be made without departing from the scope of the present invention. It is possible to
For example, the operation of the composite video is not limited to the eight control examples described above, and other controls may be performed.

  In this embodiment, the case where there are a plurality of terminal devices 100 that communicate with the video composition device 300 has been described. However, the present invention is not limited to this, and the video composition device 300 communicates with one terminal device 100. May be.

  Further, in the present embodiment, the case where the administrator directly inputs a change command to the video composition device 300 has been described. However, the present invention is not limited to this, and is input via an external device connected via a network. Also good.

  In the present embodiment, the case has been described in which the terminal device 100 transmits the operation information to the video composition device 300 using TCP, and the video composition device 300 transmits the composite video to the terminal device 100 using UDP. Absent. For example, the terminal device 100 may transmit operation information according to UDP or another protocol, and the video composition device 300 may transmit composite video according to TCP or another protocol.

  Note that the terminal device 100 and the video composition device 300 described above each have a computer system. The operation of each processing unit described above is stored in a computer-readable recording medium in the form of a program, and the above processing is performed by the computer reading and executing this program. Here, the computer-readable recording medium means a magnetic disk, a magneto-optical disk, a CD-ROM, a DVD-ROM, a semiconductor memory, or the like. Alternatively, the computer program may be distributed to the computer via a communication line, and the computer that has received the distribution may execute the program.

  The program may be for realizing a part of the functions described above. Furthermore, what can implement | achieve the function mentioned above in combination with the program already recorded on the computer system, what is called a difference file (difference program) may be sufficient.

  DESCRIPTION OF SYMBOLS 100 ... Terminal device 101 ... Image | video receiving part 102 ... Image | video display part 103 ... Object display part 104 ... Operation part 105 ... Operation information transmission part 200 ... Imaging device 300 ... Image composition apparatus 301 ... Operation information receiving part 302 ... Operation information storage part 303 ... Coordinate specifying unit 304 ... Counting unit 305 ... Object determining unit 306 ... Virtual space calculation unit 307 ... Video acquisition unit 308 ... Video composition unit 309 ... Resolution acquisition unit 310 ... Video transmission unit 311 ... Change command input unit 312 ... Operation information Change part

Claims (7)

An augmented reality system comprising: a video composition device that generates a composite video of a video captured by an imaging device and a computer graphics object; and a plurality of terminal devices that display the composite video,
The terminal device
A video receiver for receiving a composite video from the video synthesizer;
A video display unit for displaying the composite video received by the video receiving unit;
An operation unit that accepts designation of two-dimensional coordinates on the composite video displayed by the video display unit that performs a predetermined operation as operation information indicating an operation on the composite video displayed on the video display unit;
An operation information transmission unit that transmits operation information received by the operation unit to the video composition device;
The video composition device
An operation information receiving unit for receiving the operation information from the plurality of terminal devices;
When a marker is present within a predetermined range around the two-dimensional coordinate indicated by the operation information in the image captured by the imaging device, a coordinate specifying unit that specifies the three-dimensional coordinate where the marker exists;
An object determining unit that arranges an object at a position indicated by the three-dimensional coordinates specified by the coordinate specifying unit;
A video synthesis unit that synthesizes an object placed in the virtual space with a video captured by the imaging device;
An augmented reality system comprising: a video transmission unit configured to transmit the composite video generated by the video synthesis unit to the plurality of terminal devices. The coordinate specifying unit is configured to generate a three-dimensional map in the virtual space corresponding to the two-dimensional coordinates indicated by the operation information based on the two-dimensional coordinates indicated by the operation information and the display resolution of the terminal device that is the transmission source of the operation information. The augmented reality system according to claim 1, wherein coordinates are specified. The video composition device includes a resolution acquisition unit that acquires the display resolution of the terminal device from the plurality of terminal devices,
The video transmitting unit, a combined image in which the image combining unit has generated, by converting the resolution the resolution acquiring unit has acquired as claimed in claim 1 or claim 2, characterized in that transmitting to the terminal device Augmented reality system. The augmented reality according to any one of claims 1 to 3 , wherein the operation unit receives an input of a touch operation on a touch panel and specifies operation information corresponding to a locus of the touch operation. system. A video compositing device that generates a composite video of an image captured by an imaging device and a computer graphics object,
An operation information receiving unit for receiving operation information indicating designation of two-dimensional coordinates on the composite video from a plurality of terminal devices;
When a marker is present within a predetermined range around the two-dimensional coordinate indicated by the operation information in the image captured by the imaging device, a coordinate specifying unit that specifies the three-dimensional coordinate where the marker exists;
An object determining unit that arranges an object at a position indicated by the three-dimensional coordinates specified by the coordinate specifying unit;
A video synthesis unit that synthesizes an object placed in the virtual space with a video captured by the imaging device;
A video synthesizing apparatus comprising: a video transmitting unit configured to transmit the synthesized video generated by the video synthesizing unit to the plurality of terminal devices. A video composition device that generates a composite image of the image captured by the imaging device and a computer graphics object, receives operation information indicating designation of two-dimensional coordinates on the composite image from a plurality of terminal devices;
When the video composition device has a marker within a predetermined range around the two-dimensional coordinate indicated by the operation information in the video imaged by the imaging device, the three-dimensional coordinate where the marker exists is specified,
The video composition device places an object at a position indicated by the identified three-dimensional coordinates,
The video composition device synthesizes an object arranged in the virtual space with a video imaged by the imaging device,
The video composition method, wherein the video composition device transmits the generated composite video to the plurality of terminal devices. Computer
An operation information receiving unit that receives operation information indicating designation of two-dimensional coordinates on the composite video from a plurality of terminal devices,
When a marker is present within a predetermined range around the two-dimensional coordinate indicated by the operation information in the image captured by the imaging device, a coordinate specifying unit that specifies the three-dimensional coordinate where the marker exists,
An object determining unit that arranges an object at a position indicated by the three-dimensional coordinates specified by the coordinate specifying unit;
A video synthesis unit that synthesizes an object placed in the virtual space with a video captured by the imaging device;
A program for causing a synthesized video generated by the video synthesizing unit to function as a video transmitting unit that transmits the synthesized video to the plurality of terminal devices.

Images