KR20170090186A

KR20170090186A - Apparatus for capturing stick-type object in augmented reality environment and method thereof

Info

Publication number: KR20170090186A
Application number: KR1020160010738A
Authority: KR
Inventors: 허상훈
Original assignee: 허상훈
Priority date: 2016-01-28
Filing date: 2016-01-28
Publication date: 2017-08-07
Also published as: KR101895281B1

Abstract

The present invention relates to an apparatus and a method for capturing a stick-type object in an augmented reality environment, the apparatus comprising: a three-dimensional model; a table storing the three-dimensional model; a stick-type object which is vertically fixated to the three-dimensional model stored on the table and emits an infrared signal; a three-dimensional camera which captures an image of the three-dimensional model stored on the table; an infrared ray camera which captures an infrared signal emitted from the stick-type object fixated to the three-dimensional model; a projector which projects a predetermined image to the three-dimensional model; and a control computer which identifies a location of the stick-type object when data, obtained by capturing the image of the stick-type object, are input from the infrared ray camera, controls image content, being currently played, to be continuously played in an area where the stick-type object is located in accordance with progression of a story or controls another predetermined image content to be redundantly played. Accordingly, users interest and concentration can be significantly increased as play in accordance with progression of the story of the image is controlled or redundant play of another image is performed based on interaction between an image, displayed as augmented reality, and users action.

Description

[0001] The present invention relates to an apparatus and method for capturing a bar-shaped object in an augmented reality environment,

The present invention relates to an apparatus for capturing a bar-shaped object fixed in a three-dimensional model by a user and performing playback control of image contents implemented as an augmented reality based on the captured bar-like object, &Lt; / RTI >

In recent years, due to the rapid development of the video industry, the development of technologies using 3D images has been actively performed. In other words, it is possible to check the image in three dimensions rather than two-dimensional plane through television or screen.

Examples of devices for three-dimensional imaging include holography, a three-dimensional display device (for example, 3D glasses), and augmented reality.

Holography is a technique of photographing and reproducing a three-dimensional image with a single photograph using a laser beam having a phase, without using a lens, or optical technology using the same. The three-dimensional display device is a three- The augmented reality is a technique of superimposing a virtual object on the real world of a user by mixing a real environment and a virtual object, and provides a better sense of reality and additional information than a virtual reality technique.

Among them, the holography or the three-dimensional display device is not a technique for displaying the image in three dimensions by referring to the topographic information such as the depth and the shape of the area in which the image is displayed, so that the user can freely create a three- Dimensional image realizing method of projecting and displaying a desired image in the generated three-dimensional region can not be obtained.

In addition, the augmented reality technology in which a real environment and a virtual object are mixed can provide a reality that can not be obtained in the holography or the three-dimensional display device by allowing the user to see the real environment. However, There was a limitation in reproducing in connection with the operation.

That is, when an image is reproduced into an augmented reality, when a user's action is required for continuous progression of the image, a technique capable of capturing user's action and processing the progress of the image is not applied in the conventional method Because.

Korean Patent Publication No. 10-1299191 Korean Patent Publication No. 10-1998-0050467

The present invention provides an apparatus and method for capturing a bar-shaped object in an augmented reality environment in which image reproduction is performed through interaction between an image implemented as an augmented reality and a user's behavior in a three-dimensional model.

According to the present invention, a bar-shaped object fixed to a three-dimensional model is captured by an action of a user who confirms an image being implemented as an augmented reality, and control of reproduction according to the story progress of the image content is performed based on the capture information, And a method for capturing a bar-shaped object in an augmented reality environment.

An apparatus for capturing a bar-shaped object in an augmented reality environment according to an embodiment of the present invention includes a three-dimensional model, a table in which a three-dimensional model is stored, a vertically fixed three- An infrared camera for capturing an infrared signal emitted from a rod-shaped object fixed to the 3D model, and an infrared camera for capturing a 3D model of the 3D model, A 3D camera and an infrared camera are initialized at the time of driving the apparatus, and an OpenGL (Open Graphics Library) program, which is a pre-stored three-dimensional engine driving program, is initialized and synchronized with the projector, Dimensional vertex data based on photographed data input from the three-dimensional vertex data and the distance between the three-dimensional vertex data and the three-dimensional camera The color and brightness to be rendered are determined, and the previously stored image content is rendered according to the determination of the color and brightness to be rendered, and is controlled to be projected to the 3D model through the projector, and the process of projecting the image content to the 3D model , The position of the bar-shaped object is confirmed when the data captured from the infrared camera is input, and the image content currently being reproduced is controlled to be continuously reproduced according to the progress of the story in the area where the bar- And a control computer for controlling the other video contents to be reproduced redundantly.

At this time, the control computer processes the 3D camera, the infrared camera, and the OpenGL initialization, and generates three-dimensional vertex data based on the imaging parameters input from the 3D camera and the internal parameters including the viewing angle and the distortion coefficient of the 3D camera Determines the color and brightness to be rendered based on the generated three-dimensional vertex data and the distance information from the three-dimensional camera, and renders the image to be projected on the three-dimensional model based on the color and brightness information to be rendered, A database for storing a plurality of image contents projected on the 3D model through the projector, and a database for storing the captured image data based on the capture data when the capture data of the bar-shaped object is input from the infrared camera Check the position of the rod-shaped object, and check the position of the rod- An infrared ray capturing and analyzing unit for causing the image content currently being reproduced to be reproduced according to the story progression or for allowing other image contents previously set in the database to be reproduced redundantly, and a three-dimensional camera, an infrared camera and OpenGL initialization processing , 3-D vertex data generation based on photographed data from a 3-D camera, determination of color and brightness to be rendered based on 3-D vertex data, control of rendering of an image to be projected on a 3-D model, And a control unit for controlling the position of the bar-shaped object in the bar-shaped object area, the reproduction according to the story progress of the currently reproduced image content to the bar-shaped object area, or the overlapping reproduction of other predetermined image contents.

In addition, the three-dimensional model is preferably a granular object containing sand, and it is preferable that real-time movement or change operation is possible, and an image which is changed through the projector according to real-time movement or change is projected.

The rod-shaped object may include a main body and an infrared LED provided at an upper end of the main body and emitting an infrared signal at an upper portion.

The OpenGL initialization performed by the control computer is synchronized with the projector based on the initialization of the external parameters including the relative position from the 3D camera, the rotation, the internal parameters including the field of view, the position of the optical axis, and the rotation.

In addition, a method for capturing a rod-shaped object in an augmented reality environment according to an embodiment of the present invention includes: (1) a control computer, when the device is driven, (2) a step of initializing an infrared camera for capturing a 3D model of the 3D engine and initializing an OpenGL (Open Graphics Library) which is a pre-stored 3D engine driving program, and Dimensional vertex data based on the photographic data input from the 3D camera, and determining the color and brightness to be rendered based on the distance between the three-dimensional vertex data and the three-dimensional camera; and (3) Rendering the previously stored image content according to the determination of the color and brightness to be rendered and projecting the image content to the 3D model via the projector; (4) In the process of projecting the image contents to the 3D model through the projector, the control computer determines whether the captured data of the rod-shaped object which is fixed vertically to the 3D model from the infrared camera and emits the infrared signal is input (5) If data obtained by capturing a bar-shaped object from the infrared camera is input as a result of the determination in step (4), the control computer confirms the position of the bar-shaped object based on the capture data input from the infrared camera, Controlling the image content currently being reproduced to be continuously reproduced in accordance with the story progression or controlling other preset image contents to be reproduced in an overlapping area in an area where the object is located, and (6) And repeating the step (2) and thereafter until the video content is terminated.

And (1) the step (1) comprises the steps of: (1-1) when the apparatus is driven, the control computer initializes a three-dimensional camera for photographing a three-dimensional model; and (1-2) (1-3) initializing an OpenGL which is a program for driving a three-dimensional engine; and (1-4) a step of initializing an infrared camera for capturing an infrared signal, , And (1-5) the control computer may include initializing the external parameters of OpenGL, including the relative position and rotation from the 3D camera.

(2-1) receiving the photographed data of the three-dimensional model from the three-dimensional camera; and (2-2) the control computer performing the step (2-1) (3-2) generating three-dimensional vertex data based on imaging parameters of the three-dimensional camera received through the three-dimensional camera and internal parameters including a viewing angle and a distortion coefficient of the three-dimensional camera, and (2-3) And determining the color and brightness to be rendered based on the three-dimensional vertex data and the distance information from the three-dimensional camera.

In step (5), when data obtained by capturing a bar-shaped object from the infrared camera is input as a result of the determination in step (5-1) (4), the control computer confirms the position of the bar-shaped object based on the captured data (5-2) The control computer checks whether the image content needs to be changed by confirming the reproduction rule of the image content being projected on the three-dimensional model preset when capturing of the bar-shaped object is performed, and 5-3) If it is determined in step (5-2) that the change of the video content is not necessary, the control computer stores the video content currently being played in the area where the bar-shaped object identified in step (5-1) If it is determined in step (5-4) (5-2) that the image content should be changed, the control computer performs the following steps (5-1) to Other images that have been set And causing the contents to be reproduced redundantly.

As described above, according to the apparatus and method for capturing a bar-shaped object in the augmented reality environment of the present invention, It is possible to increase the degree of concentration of the image contents by promoting the fun and interest of the user who uses the augmented reality.

In addition, it can be used in various ways such as education, game, etc. according to the types of images and planning intention, which are implemented as augmented reality on three-dimensional images.

1 is a diagram schematically illustrating the configuration of an apparatus for capturing a bar-shaped object in an augmented reality environment according to an embodiment of the present invention.
2 is a detailed view of the configuration of the control computer of FIG.
3 is a flowchart illustrating an operation of a method for capturing a bar-shaped object in an augmented reality environment according to an exemplary embodiment of the present invention.
FIGS. 4 to 6 are flow charts showing the operation of each subroutine of FIG. 3 in detail.

Hereinafter, an apparatus and method for capturing a rod-shaped object in an augmented reality environment according to the present invention will be described in detail with reference to the accompanying drawings. The present invention may be embodied in many different forms and is not limited to the embodiments described herein. Like parts are designated with like reference numerals throughout the specification.

1 is a diagram schematically illustrating the configuration of an apparatus for capturing a bar-shaped object in an augmented reality environment according to an embodiment of the present invention.

As shown, the apparatus of the present invention includes a 3D model 100, a table 200, a bar-shaped object 300, a three-dimensional camera 400, an infrared camera 500, a projector 600, 700).

The three-dimensional model 100 is a granular object including sand. The three-dimensional model 100 can be moved or changed in real time according to the user's manipulation. The changed image is projected through the projector 600 according to real-time movement or change.

The table 200 is opened at the top, and the 3D model 100 is stored in the internal space.

The bar-shaped object 300 is vertically fixed to the 3D model 100 housed in the table 200 based on the user's operation, and emits an infrared signal.

In this case, the bar-shaped object 300 is a part that can be gripped by the user by hand, and includes a main body 310 having a printed circuit board, a battery, or the like for driving the infrared LED 320 therein, And an infrared LED 320 that emits an infrared signal at an upper portion where the infrared camera 500 is located.

The three-dimensional camera 400 photographs the three-dimensional model 100 stored in the table 200, and outputs the photographed data to the control computer 700.

The infrared camera 500 captures an infrared signal emitted from the bar-shaped object 300 fixed to the 3D model 100, and outputs the captured data to the control computer 700.

The projector 600 projects an image preset in the three-dimensional model 100 based on the control of the control computer 700.

The control computer 700 initializes the three-dimensional camera 400 and the infrared camera 500 when the apparatus is driven, initializes the pre-stored three-dimensional engine driving program OpenGL, and synchronizes with the projector 600. Dimensional vertex data based on the photographic data input from the three-dimensional camera 400, and determines the color and brightness to be rendered based on the distance between the three-dimensional vertex data and the three-dimensional camera 400 (for example, And replacing the data with depth, shape, color, and the like according to predetermined image data for each pixel), rendering the pre-stored image contents according to the determination of the color and brightness to be rendered, Dimensional model 100 as shown in FIG.

When the data captured from the infrared camera 500 is input to the control computer 700 in the course of projecting the image contents onto the 3D model 100, Controls the image content currently being reproduced in the area where the bar-shaped object 300 is located to be continuously reproduced according to the story progression, or controls other preset image contents to be reproduced redundantly.

For example, when the user performs an operation of plugging the bar-shaped object 300 into a specific position of the 3D model 100 made of sand or the like, the control computer 700 confirms the interaction and selects an interaction ). These interactions can be used in future games.

At this time, the OpenGL initialization performed by the control computer 700 is synchronization with the projector 600 based on the viewing angle, the internal parameters including the positions of the optical axis, and the initialization of the external parameters including the relative position and rotation from the three-dimensional camera. The reason for performing the OpenGL initialization in this way is that the optical axis of the projector 600 is distorted relative to the optical axis of the three-dimensional camera 400. In other words, the projector 600 projects the image biased toward the upper direction with respect to the optical axis in the horizontal direction, whereas the three-dimensional camera 400 photographs the image with a predetermined width up and down with respect to the optical axis in the horizontal direction, This is to make a difference and adjust it.

Fig. 2 is a diagram showing the configuration of the control computer 700 of Fig. 1 described above in more detail.

The control computer 700 of the present invention includes an image processing unit 710, an infrared capture and analysis unit 720, a database 730, and a control unit 740.

The image processing unit 710 processes the 3D camera 400, the infrared camera 500, and the OpenGL initialization. The image processing unit 710 processes the image data input from the 3D camera 400, the viewing angle of the 3D camera 400, Dimensional vertex data based on the internal parameters, determines the color and brightness to be rendered based on the generated three-dimensional vertex data and the distance information from the three-dimensional camera, and generates a three-dimensional model 100, and outputs the rendered image to the projector 600. [0050]

When the capture data of the bar-shaped object 300 is inputted from the infrared camera 500, the infrared capture and analysis unit 720 confirms an area where the bar-shaped object 300 is located based on the capture data, The image content currently being reproduced in the vicinity of the bar-shaped object 300 is reproduced in accordance with the progress of the story, or the other image contents previously set in the database 730 are reproduced in a redundant manner.

The database 730 stores information on the 3D model 100 stored in the table 200 and a plurality of image contents projected on the 3D model 100 through the projector 600. [

The control unit 740 generates three-dimensional vertex data based on photographed data photographed by the three-dimensional camera 400, the infrared camera 500, and the OpenGL initialization process and the three-dimensional camera 400 in the image processing unit 710, Determines the color and brightness to be rendered on the basis of the dimensional vertex data, and controls the rendering of the image to be projected on the 3D model 100. The position of the bar-shaped object 300 in the infrared capturing and analyzing unit 720, the reproduction according to the story progress of the currently playing image content to the area of the bar- And controls duplication of content.

Next, an embodiment of a method for capturing a rod-shaped object in the augmented reality environment according to the present invention constructed as above will be described in detail with reference to Figs. 3 to 6. Fig. At this time, each step according to the method of the present invention may be changed in the use environment or the order by a person skilled in the art.

FIGS. 3-6 are flowcharts illustrating in detail an operation of a method for capturing a bar-shaped object in an augmented reality environment according to an embodiment of the present invention.

First, when the apparatus is driven, the control computer 700 initializes a three-dimensional camera 400 for photographing the three-dimensional model 100 and an infrared camera 500 for capturing an infrared signal, And initializes the OpenGL program to synchronize the projector 600 projecting the image to the 3D model 100 (S100).

4, when the apparatus starts driving based on power supply or the like, the control computer 700 initializes a three-dimensional camera 400 for photographing the three-dimensional model 100 (S110).

Then, the control computer 700 initializes the infrared camera 500 capturing an infrared signal emitted from the rod-shaped object 300 (S120).

Then, the control computer 700 initializes OpenGL, which is a three-dimensional engine driving program (S130).

The control computer 700 initializes the internal parameters of the OpenGL including the viewing angle and the position of the optical axis at step S140 and initializes the external parameters of the OpenGL including the relative position and rotation from the 3D camera 400 (S150).

After initializing the 3D camera 400, the infrared camera 500 and OpenGL through step S100, the control computer 700 generates three-dimensional vertex data based on the photographic data input from the three-dimensional camera 400, The color and brightness to be rendered are determined based on the distance between the three-dimensional vertex data and the three-dimensional camera 400 (S200).

5, the control computer 700 receives photographed data of the 3D model 100 from the 3D camera 400 (S210). The control computer 700 receives the photographed data of the 3D model 100 Dimensional vertex data based on photographic data of the camera 400 and internal parameters including a viewing angle and a distortion coefficient of the three-dimensional camera 400 (S220).

Thereafter, the control computer 700 determines the color and brightness to be rendered based on the three-dimensional vertex data generated in step S220 and the distance information from the three-dimensional camera 400 (S230).

After determining the color / brightness to be generated and rendered by the three-dimensional vertex data through step S200, the control computer 700 renders the previously stored image content and projects it on the 3D model 100 through the projector 600 (S300).

Meanwhile, the control computer 700 is vertically fixed to the 3D model 100 from the infrared camera 500 in the course of projecting the image contents to the 3D model 100 through the projector 600, and emits the infrared signal It is determined whether the captured data of the bar-shaped object 300 is inputted (S400).

If the captured data of the bar-shaped object 300 is input from the infrared camera 500 as a result of the determination in step S400, the control computer 700 transmits the capture data of the bar-shaped object 300 And controls the image content currently being reproduced in the area where the bar-shaped object 300 is located to be continuously reproduced according to the story progression or controls the other preset image contents to be repeatedly reproduced (S500).

6, if the captured data of the bar-shaped object 300 is input from the infrared camera 500 as a result of the determination in step S400, the control computer 700 displays the bar-shaped object 300 300) (S510).

After the capturing of the bar-shaped object 300 is performed, the control computer 700 checks the reproduction rule of the image content being projected on the three-dimensional model 100, which is set in advance, and determines whether the image content needs to be changed S520).

As a result of the determination in step S520, if it is not necessary to change the image content, the control computer 700 causes the image content currently being reproduced to be continuously reproduced according to the story progress in the area where the bar-shaped object 300 is located (step S510) S530).

However, if it is determined in step S520 that the image content needs to be changed, the control computer 700 causes the other image content preset in the area where the bar-shaped object 300 is located to be reproduced (S540) .

Finally, the control computer 700 repeats step S200 and subsequent steps until the image content being projected on the three-dimensional model 100 is terminated (S700).

As described above, the present invention controls the reproduction of the story progress of the image contents implemented in the augmented reality or the overlap reproduction of the other images based on the capturing of the bar-shaped object fixed to the three-dimensional model by the user, The concentration can be greatly increased.

In addition, the present invention can be utilized in various ways such as education, game, and the like depending on the types of images and planning intentions, which are implemented as augmented reality on three-dimensional images.

It will be apparent to those skilled in the art that various modifications may be made to the invention without departing from the spirit and scope of the invention as defined in the following claims And changes may be made without departing from the spirit and scope of the invention.

100: 3D model
200: Table
300: rod-shaped object
310:
320: Infrared LED
400: Three-dimensional camera
500: Infrared camera
600: Projector
700: Control computer
710:
720: Infrared Capture and Analysis Unit
730: Database
740:

Claims

3D model,
A table in which the 3D model is stored,
A bar-like object vertically fixed to the three-dimensional model accommodated in the table and emitting an infrared signal,
A three-dimensional camera for photographing the three-dimensional model stored in the table,
An infrared camera for capturing an infrared signal emitted from the rod-shaped object fixed to the 3D model,
A projector for projecting an image preset in the three-dimensional model, and
Dimensional camera and an infrared camera, initializes an OpenGL (Open Graphics Library) which is a pre-stored three-dimensional engine driving program and synchronizes with the projector, and based on the shooting data input from the three-dimensional camera Dimensional vertex data, determines the color and brightness to be rendered on the basis of the distance between the three-dimensional vertex data and the three-dimensional camera, and determines the color and brightness to be rendered, Dimensional model, and when the captured data of the rod-shaped object is input from the infrared camera in the process of projecting the image content to the 3D model, the rod- And the current image being reproduced in the area where the bar-shaped object is located A control computer for controlling the content to be continuously reproduced in accordance with the progress of the story or for controlling other previously set video contents to be reproduced in duplicate
And an apparatus for capturing a bar-like object in an augmented reality environment.

The method according to claim 1,
The control computer includes:
The 3D camera, the infrared camera, and OpenGL initialization, and generates three-dimensional vertex data based on internal parameters including photographing data input from the three-dimensional camera and a viewing angle and a distortion coefficient of the three-dimensional camera, Determines the color and brightness to be rendered on the basis of the three-dimensional vertex data and the distance information from the three-dimensional camera, and renders the image to be projected on the three-dimensional model based on the color and brightness information to be rendered and outputs the rendered image to the projector Image processor,
A database storing information on the 3D model stored in the table, a plurality of image contents projected on the 3D model through the projector,
When the capturing data of the bar-shaped object is inputted from the infrared camera, the position of the bar-shaped object is confirmed based on the captured data, and the image content currently being reproduced in the bar- An infrared ray capturing and analyzing unit for allowing other image contents previously set in the database to be reproduced, and
The three-dimensional camera, the infrared camera, and the OpenGL initialization process in the image processing unit, the three-dimensional vertex data generation based on the photographing data photographed by the three-dimensional camera, the determination of the color and brightness to be rendered based on the three- , Controlling the rendering of the image to be projected on the 3D model, checking the position of the bar-shaped object in the infrared capture and analysis unit, and proceeding to the story of the currently playing image content to the bar- And a control unit for controlling reproduction of the video content
And an apparatus for capturing a bar-like object in an augmented reality environment.

The method according to claim 1,
The three-
An apparatus for capturing a bar-shaped object in an augmented reality environment in which a moving object such as sand is capable of real-time movement or change operation and an image that is changed through the projector is projected according to real-time movement or change.

The method according to claim 1,
The rod-
Body, and
An infrared LED provided at an upper end of the main body and emitting an infrared signal to the upper part,
And an apparatus for capturing a bar-like object in an augmented reality environment.

The method according to claim 1,
The OpenGL initialization performed by the control computer includes:
Wherein the synchronization is synchronized with the projector based on initialization of an external parameter including a field of view, an internal parameter including a position of an optical axis, and a relative position from a three-dimensional camera and rotation. / RTI >

(1) When the apparatus is driven, the control computer initializes a three-dimensional camera for capturing a three-dimensional model and an infrared camera for capturing an infrared signal, and then executes an OpenGL (Open Graphics Library) Initializing and synchronizing a projector projecting an image to the 3D model,
(2) The control computer generates three-dimensional vertex data based on photographed data input from the three-dimensional camera, and determines a color and a brightness to be rendered based on the distance between the three-dimensional vertex data and the three- step,
(3) The control computer may further include a step of rendering the previously stored image content according to the determined color and brightness to be rendered, and projecting the image content to the 3D model through the projector,
(4) In the process of projecting the image contents to the 3D model through the projector, the control computer controls the three-dimensional model to project the image contents, ,
(5) If data obtained by capturing the bar-shaped object is input from the infrared camera as a result of the determination in step (4), the control computer checks the position of the bar-shaped object based on the capture data input from the infrared camera , Controlling the image content currently being reproduced in the area where the bar-shaped object is located to be continuously reproduced in accordance with the story progression, or controlling another preset image content to be reproduced redundantly, and
(6) The control computer repeats the step (2) and subsequent steps until the image content being projected on the 3D model is finished
Gt; augmented < / RTI >

The method according to claim 6,
The step (1)
(1-1) When the apparatus is driven, the control computer initializes the 3D camera for photographing the 3D model,
(1-2) The control computer includes a step of initializing an infrared camera capturing an infrared signal emitted from the rod-shaped object,
(1-3) The control computer includes a step of initializing OpenGL which is a program for driving a three-dimensional engine,
(1-4) The control computer initializes the internal parameters of the OpenGL including the field of view, the position of the optical axis, and
(1-5) The control computer initializes the external parameters of the OpenGL including the relative position and rotation from the three-dimensional camera
Gt; augmented < / RTI >

The method according to claim 6,
The step (2)
(2-1) The control computer includes a step of receiving photographing data obtained by photographing the three-dimensional model from the three-dimensional camera,
(2-2) The control computer calculates three-dimensional vertex data based on the internal parameters including the imaging data of the three-dimensional camera and the viewing angle and the distortion coefficient of the three-dimensional camera received through the step (2-1) Creating, and
(2-3) The control computer determines the color and brightness to be rendered based on the three-dimensional vertex data generated in the step (2-2) and the distance information from the three-dimensional camera
Gt; augmented < / RTI >

The method according to claim 6,
The step (5)
(5-1) If data obtained by capturing the bar-shaped object from the infrared camera is inputted as the determination result in the step (4), the control computer confirms the position of the bar-shaped object based on the captured data,
(5-2) The control computer checks whether the image content needs to be changed by checking the reproduction rule of the image content being projected on the three-dimensional model previously set when capturing the bar-shaped object,
(5-3) If it is determined in step (5-2) that the change of the image content is not necessary, the control computer displays the image currently being played back in the area where the bar- Causing the contents to be continuously reproduced according to the story progress, and
(5 - 4) If it is determined in step (5 - 2) that the image contents need to be changed, the control computer determines whether or not the other A step of causing the video contents to be reproduced redundantly
Gt; augmented < / RTI >