JP6535801B1 - Augmented reality display system, augmented reality display method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an augmented reality display system and the like suitable for the user to control display contents using what exists in the real world when displaying by augmented reality technology using a marker. provide. SOLUTION: In addition to an object marker, a new marker called a control marker is introduced. The OM display processing unit 25 controls the control target specified by the object marker with the control content specified by the control marker, and performs display processing as augmented reality. The user can control the control object in the augmented reality technology by using object markers and control markers existing in the real world. [Selected figure] Figure 1

Description

  The present invention relates to an augmented reality display system, an augmented reality display method, and a program, and more particularly to an augmented reality display system and the like that perform display processing by augmented reality in a display unit.

  Augmented Reality (AR) is a technology that adds virtual visual information to an existing landscape and displays it. Patent Document 1 describes that a marker is used to combine and display an image downloaded from a server on a real image.

JP, 2015-124381, A

  As described in Patent Document 1, the conventional augmented reality technology displays a predetermined image downloaded from the server on the real image, corresponding to the marker. The user could change the orientation, position, size, etc. of the image, for example, by an operation such as tilting the terminal. For example, in a service that downloads an image of a dinosaur moving from a server and displays it using augmented reality technology in relation to a dinosaur pictorial book, the user may change the dinosaur to various angles or sizes by tilting the terminal or the like. It was possible to display it with.

  However, the predetermined image downloaded from the server was of the content defined by the server in relation to the marker. For example, the action taken by the dinosaur displayed in relation to the pictorial book is of a content predetermined by the server. Users can not change the behavior of dinosaurs.

  Therefore, the present invention provides an augmented reality display system and the like suitable for controlling display contents by the virtual reality technology in the augmented reality technology using a marker and using the one existing in the real world by the user. The purpose is to

  A first aspect of the present invention is an augmented reality display system that performs display processing by an augmented reality technique in a display unit, and the display unit includes a display control unit that performs display processing by an augmented reality technology that uses markers. When the display control means performs display processing using an object marker without using a control marker, the display processing is performed in which the control target specified by the object marker is controlled according to the control content specified by the object marker. When display processing is performed by the object marker and the control marker, display processing is performed in which the control target specified by the object marker is controlled according to the control content specified by the control marker.

  A second aspect of the present invention is the augmented reality display system according to the first aspect, wherein the control marker includes an indication marker, and the display control means includes the indication marker if the control marker is the indication marker. When the recognition unit does not recognize an instruction from the user, display processing by the object marker is performed, and when the recognition unit recognizes an instruction from the user, display processing by the object marker and the control marker is performed.

  A third aspect of the present invention is the augmented reality display system according to the first or second aspect, wherein the object marker and the control marker are respectively printed on different printed matter of an object card and a control card. The control content for the control target specified by the object card can be changed by combining the object card with one or more of the control cards, and the display control unit specifies the control marker. Among the control contents, if there are a plurality of executable states, display control is performed on the control target by executing a plurality of control contents in executable states in a predetermined order.

  A fourth aspect of the present invention is an augmented reality display method for performing display processing by the augmented reality technology in a display unit, wherein the display control means included in the information processing apparatus uses the marker in the display unit. When the display control means performs the display process with the object marker without using the control marker, the control target specified by the object marker is displayed in the display step; Control processing is performed according to the control content specified by the object marker, and when display processing is performed using the object marker and the control marker, control that the control marker specifies a control target specified by the object marker Control according to the content Perform a display process.

  According to a fifth aspect of the present invention, in the display unit, a computer for performing display processing by the augmented reality technology using a marker, and in the display unit, a control target specified by an object marker by the augmented reality technology is The object marker is a program for functioning as a display control unit that performs display processing controlled according to the control content specified by the control marker different from the object marker. The present invention may be regarded as a computer readable recording medium for recording the program of the fifth aspect.

  The display control means may be regarded as controlling and displaying the control target by the control marker when the recognition means recognizes the control marker. Further, in the present invention, the display control means may specify a part or all of the control target specified by the plurality of object markers to perform control processing by the control marker. Further, in addition to the control content specified by the control marker, the control target may be controlled by an instruction of the user.

  According to each aspect of the present invention, in addition to the object marker, a new marker called a control marker is introduced as a marker for the augmented reality technology. For example, an object marker reproduces a specific moving image by augmented reality technology. And a fast forward function is realized by the control marker. The user can use the control marker to realize the fast-forwarding function to realize fast-forwarding playback while playing a specific moving image by using the object marker.

  In this manner, the object marker can perform display processing by the augmented reality technology alone, similarly to the conventional marker. On the other hand, a control marker is for controlling the control object which an object marker specifies. According to each aspect of the present invention, by introducing a new marker called a control marker, the user utilizes a real-world marker called an object marker and a control marker to identify a control target specified by the object marker. It is possible to realize display processing by the augmented reality technology by performing control according to the control content specified by the control marker.

It is a block diagram which shows an example of (a) structure of the augmented reality display system which concerns on embodiment of this invention, and a flowchart which shows an example of (b) process. It is a figure which shows an example of a process of the augmented reality display system 1 of FIG. FIG. 7 is a diagram showing an example of another process of the augmented reality display system 1 of FIG. 1;

  Hereinafter, embodiments of the present invention will be described with reference to the drawings. The embodiments of the present invention are not limited to the following examples.

  FIG. 1 is a block diagram showing an example of the (a) configuration of the augmented reality display system according to the embodiment of the present invention, and a flow chart showing an example of (b) processing.

  Referring to FIG. 1A, the augmented reality display system 1 includes an information processing device 3, an object card 5, and a control card 7.

  An object marker 11 (an example of “object marker” in the claims of the present application) is printed on the object card 5. The server (not shown) is associated with the object marker 11 and information for controlling the control object and the control object when the control by the control marker 13 is not performed (for example, control to make it stationary). Identify the standby control.

  A control marker 13 (an example of a “control marker” in the claims of the present application) is printed on the control card 7. The server identifies the control content in association with the control marker 13. Also, if display processing is performed in association with the control marker 13, the server specifies the operation information image to be displayed.

  The information processing device 3 includes a photographing unit 15, a recognition unit 17, a display control unit 19, and a display unit 27. The display control unit 19 includes a photographed image display processing unit 21, a CM display processing unit 23, and an OM display processing unit 25. The display unit 27 includes a display screen 29.

  The photographing unit 15 performs photographing processing to obtain a photographed image. The photographed image display processing unit 21 displays the photographed image 31 photographed by the photographing unit 15 on the display screen 29.

  The recognition unit 17 determines whether a marker is present in the captured image. When the captured image includes the object marker 11, the control object and standby control identified by the object marker 11 are identified by communicating with the server. When the control image is included in the captured image, the control content specified by the control marker 13 is communicated by communicating with the server, and the operation information image is specified if display processing is performed in relation to the control marker.

  When the display processing is performed in association with the control marker 13, the CM display processing unit 23 displays the operation information image 39 on the captured image 31 by the augmented reality technology (see reverse rotation image 93 in FIG. 3).

  The OM display processing unit 25 performs display processing of the control target specified by the object marker 11 (see the cube image 63 in FIG. 2 and the triangle image 87 in FIG. 3).

  FIG. 1B is a flowchart showing an example of display processing of the OM display processing unit 25. As shown in FIG. The OM display processing unit 25 determines whether the object marker 11 is recognized and the control target is specified by the object marker 11 (step ST1). If the control target is not specified, the control target image 37 is not displayed (step ST2), and the process returns to step ST1.

  If the control target is specified, it is determined whether or not the control by the control marker 13 is to be performed (step ST3).

  If control by the control marker 13 is performed, display processing of the control target image 37 in which the control target specified by the object marker 11 is controlled according to the control content specified by the control marker 13 is performed by augmented reality technology (step ST4) It returns to step ST1.

  If the control by the control marker 13 is not performed, display processing of the control target image 37 in which the control target specified by the object marker 11 is on standby control is performed by the augmented reality technology (step ST5), and the process returns to step ST1.

  In addition, the data on the motion control by the control marker may be read simultaneously when the object marker is read, or the data on the object marker may be downloaded and displayed and then downloaded, and the individual control markers It may be downloaded separately at the time of performing operation control regarding. When downloading in advance, the display control unit 19 does not need to communicate with the server when recognizing the control marker. Also, display processing by the augmented reality technology is performed when the object marker is in the captured image, and display processing by the augmented reality technology is not performed even if the control marker is recognized when the object marker is not in the captured image. You may

  A plurality of object cards 5 and / or control cards 7 may exist. For example, the upper body and lower body of one robot to be controlled may be specified by two object cards, respectively. The OM display processing unit 25 specifies whether or not control by the control marker 13 is to be performed on the control target corresponding to each object marker 11 if there are a plurality of object cards 5. In the control marker 13, the display control means may specify and control a part of the control targets, or may control all the control targets in common. If a plurality of control markers 13 exist, the control target controlled by each control marker 13 may be individually identified, or the same control target may be controlled commonly to all control markers. Also, for example, when the control target is a robot and the control marker is control of a sword, the robot may display with a sword.

  Moreover, control by a control marker may start control, for example, when the recognition part 17 recognizes a control marker (refer the lower arrow marker 72 of FIG. 2). For example, a plurality of control markers 13 may be arranged and arranged in a horizontal row, and control may be sequentially performed by control markers that have become executable in the order of being arranged from right to left. This can be used, for example, for education such as programming.

  In addition, even when the recognition unit 17 recognizes the control marker 13 and further recognizes that the user has issued an instruction to an object (for example, a control card) existing in the real world, the control is started. Good (see right arrow marker 66 in FIG. 2). For example, when teaching control of a car, in education using a real machine, there was a risk that the real machine would be broken. According to this embodiment, it is possible to avoid a risk such as a car failure in order to use the augmented reality technology by the real-world card. In addition, because it can be realized in printed matter, it can be adopted, for example, in an appendix of a magazine.

  Alternatively, the recognition unit 17 may recognize the control marker 13 to display the operation information image, and start control when it is recognized that the user has instructed the operation information image (see FIG. 3). See reverse rotation marker 91 and reverse rotation image 93). Thereby, for example, even if the photographing of the photographing unit 15 is not stable and the object marker 11 and the control marker 13 are not continuously photographed, the control of the control target can be realized by the operation on the display unit 27 or the like.

  An example of specific processing will be described with reference to FIG. Referring to FIG. 2A, a start zone 51, an intermediate zone 53, and a goal zone 55 exist in the photographed image. There is an intermediate zone 53 to the right of the start zone 51. Below the intermediate zone 53, a goal zone 55 exists. In this example, first, a cube is displayed in the start zone 51 by augmented reality technology, and this cube is moved to the goal zone 55 via the intermediate zone 53.

  Referring to FIG. 2 (b), the user places an object card 61 on which cube markers 62 are printed in the start zone 51. Referring to FIG. 2C, the recognition unit 17 recognizes the cube marker 62, and the OM display processing unit 25 causes the cube image 63 specified by the object marker to stand still by the augmented reality technology. indicate.

  Referring to FIG. 2 (d), the user places a control card 65 on which the right arrow marker 66 is printed in the intermediate zone 53. The right arrow marker 66 is for control of moving the control object in the left zone to the zone in which the control card is placed. The right arrow marker 66 is an instruction marker, and after control of the control marker is performed after the user's instruction. At the time of FIG. 2D, there is no instruction from the user, and the cube image 63 does not move. Referring to FIG. 2E, when recognizing that the user holds control card 65 with a finger, recognition unit 17 recognizes that the user has given an instruction. Referring to FIG. 2F, the OM display processing unit 25 moves the cube image 63 in the left start zone 51 to the right while rotating and displays the cube image 63 in the intermediate zone 53.

  Referring to FIG. 2 (g), the user places a control card 65 on which the left arrow marker 70 is printed in the goal zone 55. The left arrow marker 70 is for moving the control object in the right zone to the zone in which the control card is placed. The left arrow marker 70 is not a pointing marker. However, there is no cube image 63 in the right zone, and the OM display processing unit 25 does not perform processing.

  Referring to FIG. 2 (h), the user places a control card 71 on which a down arrow marker 72 is printed, in place of the control card 65, in the goal zone 55. The down arrow marker 72 is for moving the control object in the upper zone to the zone in which the control card is placed. The down arrow marker 72 is not an indication marker. Referring to FIG. 2I, the OM display processor 25 moves the cube image 63 in the upper middle zone 53 downward while rotating it and displays it in the goal zone 55.

  In this manner, by using the object card 61 and the control cards 65 and 71, the control target image specified by the object marker can be controlled by the control marker to perform display processing.

  In addition, you can learn the basic concept of programming by ordering multiple zones for placing control cards and specifying the order to execute the control markers on the control cards placed in each zone. . For example, it is assumed that the control marker of the control card placed in the second zone, the goal zone 55, is processed following the control marker of the control card placed in the first zone, the middle zone 53. With the photographing unit 15 not photographing the start zone 51, the intermediate zone 53 and the goal zone 55, the user can set the object card 61, the control card 65, and the like in the start zone 51, the intermediate zone 53 and the goal zone 55, respectively. Put on the control card 71. The photographing unit 15 photographs the start zone 51, the intermediate zone 53, and the goal zone 55. In this case, the recognition unit 17 simultaneously recognizes the cube marker 62, the right arrow marker 66, and the down arrow marker 72. The OM display processing unit 25 is capable of cube display by the cube marker 62, but the right arrow marker 66 is an indication marker and can not be executed without any indication. Therefore, the cube display by the cube marker 62 is performed. When the user issues an instruction regarding the right arrow marker 66, the OM display processing unit 25 performs movement control of the cube by the right arrow marker 66 of the first zone, and subsequently, the cube of the second zone is controlled by the lower arrow marker 72. Perform movement control.

  An example of another process will be described with reference to FIG. Referring to FIG. 3A, the recognition unit 17 recognizes the object card 83 on which the rotating triangle marker 85 is printed, using the photographed image 81. The server, in conjunction with the rotating triangle marker 85, identifies the triangle image 87 and the counterclockwise rotation as standby control. Referring to FIG. 3B, the OM display processing unit 25 rotates the triangular image 87 counterclockwise and displays it using the augmented reality technology. With reference to FIG. 3C, in this example, the OM display processing unit 25 continues to rotate the triangle image 87 counterclockwise, even if the rotating triangle marker 85 is not present in the captured image 81. Let it be displayed.

  Referring to FIG. 3D, the recognition unit 17 recognizes the control card 89 on which the reverse rotation marker 91 is printed by using the photographed image 81. The server is associated with the reverse rotation marker 91 to specify the reverse rotation image 93 and the clockwise rotation as the control content. Referring to FIG. 3E, CM display processing unit 23 displays reverse rotation image 93 by the augmented reality technology. The reverse rotation marker 91 is an indication marker, and after the user's instruction, performs control by the control marker. At the time of FIG. 3E, there is no instruction from the user, and the triangular image 87 is not reversely rotated. Referring to FIG. 3E, in this example, the OM display processing unit 25 continuously rotates the triangular image 87 counterclockwise even if the reverse rotation marker 91 is not present in the captured image 81. The CM display processing unit 23 displays the reverse rotation image 93.

  The recognition unit 17 determines whether the user has given an instruction to the reverse rotation image 93 or not. Referring to FIG. 3G, for example, the recognition unit 17 moves the finger to the position where the reverse rotation image 93 is displayed in the photographed image 81 by the user, or the display unit 27 is the touch panel by the user. When it is recognized that the displayed reverse rotation image 93 is pressed with a finger, it is determined that the user has given an instruction to the reverse rotation image 93. When instructed by the user, the OM display processor 25 rotates the triangular image 87 clockwise and displays it.

  Referring to FIG. 3 (h), when the recognition unit 17 recognizes the control card 97 on which the erase marker 99 is printed, the control unit 17 sets the control target and the control content by the rotating triangle marker 85 and the reverse rotation marker 91 as absent. The CM display processing unit 23 does not display the reverse rotation image 93, and the OM display processing unit 25 does not display the triangle image 87.

  Reference Signs List 1 augmented reality display system, 3 information processing device, 5 object card, 7 control card, 11 object marker, 13 control marker, 15 imaging unit, 17 recognition unit, 19 display control unit, 21 photographed image display processing unit, 23 CM display Processing unit, 25 OM display processing unit, 27 display unit, 29 display screen, 31 photographed image, 37 control target image, 39 operation information image

Claims (5)

An augmented reality display system for performing display processing by the augmented reality technology in a display unit, comprising:
The display unit includes display control means for performing display processing by an augmented reality technique using a marker,
The display control means
When display processing is performed using an object marker without using a control marker, display processing is performed in which the control target specified by the object marker is controlled according to the control content specified by the object marker,
When display processing is performed by the object marker and the control marker, display processing is performed in which the control target specified by the object marker is controlled according to the control content specified by the control marker ,
The control marker indicates the moving direction of the control target,
The control marker includes an indication marker
The display control means
The control marker indicates a moving direction according to a predetermined moving path in the real space, and if the control marker is not a pointing marker, the control marker is moved relative to the control target according to the indication of the control marker,
If the control marker indicates the movement direction according to the movement path, and if the control marker is an indication marker, the control marker is directed to the control object when the recognition unit recognizes the indication by the user. Move according to the instructions of
The augmented reality display system , wherein the controlled object is not moved according to the control marker if the control marker does not indicate the moving direction according to the movement path . The movement route is from the start point to the goal point,
The object marker and the control marker are respectively printed on different printed matter of an object card and a control card,
If the recognition unit recognizes the control markers printed on the two control cards, the display control means sets the two control cards as the first control card and the second control card in order of proximity to the start point. ,
If the control marker printed on the first control card does not indicate the movement direction according to the movement path, the control object is not moved according to the control marker,
If the control marker printed on the first control card indicates the movement direction according to the movement path, and the control marker printed on the second control card does not indicate the movement direction according to the movement path, The control object is moved according to the instruction of the control marker printed on the first control card, and the movement according to the control marker printed on the second control card is not performed.
If the control markers printed on the first control card and the second control card indicate movement directions according to the movement path in order of proximity to the start point, the control object may be ordered in order of proximity to the start point. The augmented reality display system according to claim 1, wherein the system is moved according to an instruction of a control marker printed on the first control card and the second control card. If the control marker printed on the first control card indicates the movement direction according to the movement path, and if the indication marker is the indication marker, the recognition point is the start point when the recognition by the user is recognized. Moving the control target according to the instruction of the control marker printed on the nearest first control card,
The control markers printed on the first control card and the second control card indicate the moving direction according to the moving path in order of proximity to the start point, and the control markers printed on the first control card Is not an indication marker, and the control marker printed on the second control card is an indication marker,
Moving the control target according to an instruction of a control marker printed on the first control card,
When the recognition unit recognizes an instruction from the user, the control object is moved according to the instruction of the control marker printed on the second control card,
The control markers printed on the first control card and the second control card indicate the moving direction according to the moving path in order of proximity to the start point, and if not indicated markers, in the order of proximity to the start point The augmented reality display system according to claim 2, wherein the control object is moved according to an instruction of a control marker printed on the first control card and the second control card . An augmented reality display method for performing display processing by an augmented reality technique in a display unit, comprising:
A display control unit included in the information processing apparatus includes a display step of performing display processing by the augmented reality technology using a marker in the display unit;
In the display step, the display control means
When performing a display process by the object marker without a control marker, the object marker that control target be identified, performs a display process in which the control according to the control content which the object marker is identified,
When display processing is performed by the object marker and the control marker, display processing is performed in which the control target specified by the object marker is controlled according to the control content specified by the control marker ,
The control marker indicates the moving direction of the control target,
The control marker includes an indication marker
The display control means
The control marker indicates a moving direction according to a predetermined moving path in the real space, and if the control marker is not a pointing marker, the control marker is moved relative to the control target according to the indication of the control marker,
If the control marker indicates the movement direction according to the movement path, and if the control marker is an indication marker, the control marker is directed to the control object when the recognition unit recognizes the indication by the user. Move according to the instructions of
The augmented reality display method , wherein the controlled object is not moved according to the control marker if the control marker does not indicate the moving direction according to the movement path . In the display unit, a computer for performing display processing by the augmented reality technology using markers,
In the display unit, a program for functioning as display control means for performing O Ri Display processing augmented reality technology,
The display control means
When display processing is performed using an object marker without using a control marker, display processing is performed in which the control target specified by the object marker is controlled according to the control content specified by the object marker,
When display processing is performed by the object marker and the control marker, display processing is performed in which the control target specified by the object marker is controlled according to the control content specified by the control marker,
The control marker indicates the moving direction of the control target,
The control marker includes an indication marker
The display control means
The control marker indicates a moving direction according to a predetermined moving path in the real space, and if the control marker is not a pointing marker, the control marker is moved relative to the control target according to the indication of the control marker,
If the control marker indicates the movement direction according to the movement path, and if the control marker is an indication marker, the control marker is directed to the control object when the recognition unit recognizes the indication by the user. Move according to the instructions of
The program which does not make the control object move according to the control marker if the control marker does not indicate the moving direction according to the movement path.

Images