JP2015198443A - augmented reality system, augmented reality processing method, program and recording medium - Google Patents

Abstract

PROBLEM TO BE SOLVED: To innovate the technic of virtual reality to display a virtual three-dimensional video while superposing on the display of visibility information of reality.SOLUTION: From the captured image 31 of a ticket 3, i.e., an AR marker taken by means of a camera 11, an image recorded in a ticket image storage section 51 is searched, a live video 32 which a live camera section 54, previously associated with the image thus searched, is in shooting is transmitted to a smart phone 1, and then the live video 32 is displayed on a display 10 so as to be superposed on the captured image 31 of a ticket 3. Furthermore, the live video 32 is transmitted to the smart phone 1 so that the live video 32 goes through a relay video storage device 22 once.

Description

  The present invention is an augmented reality system that displays a live camera image associated with a video, such as a building, a T-shirt, a concert ticket, a book, a CD jacket, etc., captured by a camera in an overlapping manner. , Method, program, and recording medium. That is, the present invention relates to a completely new AR technique in which actual view information of different places existing at almost the same time is displayed on top of the display of actual view information.

  A technique for superimposing and displaying a virtual three-dimensional image on the display of actual view information taken by a camera is called augmented reality (AR) or augmented reality. In general, in this AR technology, an identifier such as a one-dimensional code or a two-dimensional code called an AR marker is used, and the camera captures a real field of view including the AR marker, and is associated with the AR marker. A three-dimensional image is searched for, and this virtual three-dimensional image is superimposed on the photographed real field of view information and displayed. In Japanese Patent Laid-Open No. 2011-242816, GPS position information and azimuth information are used for the AR marker.

  Japanese Patent Laid-Open No. 2013-198520 detects a figure from a plurality of real cards photographed by a camera in order to reflect information recorded on a real-world card in processing executed by a game device. It is said that the user selects any one or more graphics and executes the processing associated with the selected graphics. Therefore, the user can enjoy the game by taking the information of the card possessed by the user into the virtual world and changing the progress of the game according to the figure selected by the user among the detected figures.

  On the other hand, Japanese Patent Application No. 2014-031091 is an invention according to the applicant's application, but this is intended to solve the problems of the inventions of the above-mentioned JP2011-242816A and JP2013-198520A. It was. That is, according to the technology of Japanese Patent Laid-Open No. 2013-198520 using an identifier such as a QR code (registered trademark, the same applies hereinafter) for the AR marker, if the card is printed with a beautiful illustration or photograph, There was a problem that it was uncomfortable to see the identifier in the aesthetic point. In Japanese Patent Laid-Open No. 2013-198520, in order to execute a predetermined process, the user is required to select an arbitrary form from among figures detected by photographing with a camera. Therefore, this operation is troublesome.

  Therefore, Japanese Patent Application No. 2014-031091 searches for an image recorded in an image storage unit with a photographed image of an AR marker photographed by a camera, and a virtual tertiary previously associated with the retrieved image. The original video is read from the virtual 3D video storage unit 53, and is superimposed on the photograph 30 and displayed on the display. Further, a plurality of cards each having an arbitrary AR marker printed thereon are photographed with a camera, and the AR marker recorded in the AR marker storage unit is searched with the AR marker, and is associated with the searched AR marker in advance. The virtual three-dimensional video that is present is read from the virtual three-dimensional video storage unit and displayed on the display in a superimposed manner on the card.

JP 2011-242816 A JP2013-198520A Japanese Patent Application No. 2014-031091

  The above-mentioned problem has been successfully solved by Japanese Patent Application No. 2014-031091. However, the conventional AR technology including Japanese Patent Application No. 2014-031091 should be more interesting and more useful. Accordingly, an object of the present invention is to solve such a new problem.

  Accordingly, as a result of earnest research, the present inventors have come up with the idea that live video taken with a live camera is displayed on top of the actual view information display. For example, when a music concert ticket is photographed with a camera of a smartphone, an image of the dressing room at the present time of the concert performer is displayed on top of the ticket image. This is really interesting for users with tickets, and the power to raise their mood is outstanding.

  That is, the above problem is that the AR marker of the marker presenting medium with the AR marker photographed by the camera is received and processed via a network such as the Internet, and the live video of the live camera corresponding to the AR marker is processed by the live video processing device. This is achieved by performing transmission processing on the display device on the camera side so that the live image is displayed on the display device on the camera side so as to be superimposed on the video on the marker presentation medium. In augmented reality or augmented reality, virtual 3D video is superimposed on the display of real field of view information captured by a camera. Instead of this virtual 3D video, live video captured by a live camera is used. The technology is completely new to this invention. In addition, although the live video is remote visual field information that actually exists, it can be said that the present invention also relates to augmented reality.

  The AR marker transmitted and received from the camera side may be used as necessary for authentication confirmation, or may be used to identify the live video of the live camera transmitted to the camera side. In any case, when the live video processing device determines the live video of the live camera corresponding to the AR marker, the live video is transmitted to the display device on the camera side. On the camera side, the live video is received, and the live video is displayed superimposed on the video of the marker presentation medium. The live video processing apparatus may operate as a live video distribution server of the live camera, or may be provided with a distribution server on the live camera side so that the distribution server transmits the live video toward the camera side. It is possible to design as well. Note that the live video may or may not include audio.

  The AR marker can be used as long as it can be a marker such as a character string, an image, a QR code, or a chameleon code (all are registered trademarks), and the marker presenting medium can be a card, a ticket, a T-shirt, or a book magazine. As long as the marker such as a CD jacket, a flyer or a fan is printable. Or you can even add an AR marker to a building called a stadium.

  For example, when a concert ticket is used as a marker presentation medium, in addition to the dressing room scenery described above, the practice scene, the situation of the stage sleeves and the costume change, the waiting queue for admission and the congestion situation in the venue, Alternatively, it is possible to freely set the live video of the stage after the program starts. Of course, in addition to concerts, sports events and gourmet festivals are used in a wide range. In addition, the breeding situation of the breeding room that cannot be seen from the outside at the zoo, the kitchen cooking scene at the restaurant, the scene inside and outside the racing car at the circuit, the class scene at the study course, the lecture at the international conference It is possible to freely set the state of the sky, the sky pattern at the travel destination, the bus window view of the preceding bus or train, the progress of the ceremony at the wedding or funeral, and so on. In this case, the number of live cameras may be one or a plurality, or may be a fixed type, a movable type, or a system in which these are mixed. In the case of the mobile type, it is also possible to set a camera provided on a wearable computer glass (a kind of head-mounted display) as a live camera, and these are optional design matters.

  In addition, the above problem is that the AR marker of the marker presentation medium with the AR marker photographed by the camera is received and processed via a network such as the Internet, and the live video relay processing device performs live processing of the live camera corresponding to the AR marker. By determining the video and transmitting the live video to the display device on the camera side so as to be displayed on the camera side display device so as to be superimposed on the video of the marker presentation medium via the relay device. Achieved.

  This configuration substantially follows the configuration described above, but differs in that it includes a live video relay device. This relay device is provided in the live video relay processing device so that the live video of the live camera is once relayed here, or a relay device is set outside and the live video relay processing device is connected to the live device. It is also possible to design so that the video is relayed and transmitted toward the camera side.

  This relay device can be provided with a storage device for live video. The camera side receives live video from the storage device. Then, if it is based on the original live video, there will be a delay, but there is often a delay in television broadcast satellite relay, etc. In this sense, time shift playback and recording playback can be included in live relay . From another point of view, access may concentrate on the live camera and the transmission process may not be able to catch up. However, according to the transmission process via the storage device, the problem of access concentration can be avoided relatively easily. There is an advantage that you can.

  The live video processing apparatus or the live video relay processing apparatus includes a plurality of the live cameras, and presents an option on the display device side so that the user can select which live video the user wants to view. In addition, the live video of the live camera selected on the display device side can be specified as the live video to be transmitted to the display device. For example, when a marker presentation medium is set for a concert ticket, the user presents three options of the dressing room scene, the stage scene, and the waiting queue as the live video, and the user switches the live video and enjoys it. There is a usage that says to make it possible.

  Next, the above-described problem is that the AR marker of the marker presentation medium with the AR marker photographed by the camera is received and processed via a network such as the Internet, and a live video processing device performs live processing of the live camera corresponding to the AR marker. Determining a video, transmitting the live video to the display device on the camera side so that the live video is superimposed on the video of the marker presenting medium on the display device on the camera side, and a plurality of the live cameras, Alternatively, the user wants to watch using a chat system that includes the live camera movable with the photographer and communicates between the operator of the live video processing device and the user who views the live video on the display device. Achieved by providing an augmented reality processing method that includes operations to accept live scenes by chat It is.

  Using this chat system, the user can request a live scene to be viewed from the operator. When a plurality of live cameras are provided, the user requests by chat which live camera the user wants to view, and the operator switches the live video to be transmitted in response to the request. If the live camera is fixed but the direction can be changed or the arrangement can be changed, such an operation can be requested by chat. Alternatively, if the live camera is movable with the photographer, the user can request a chat to follow the performer's movement. It is possible to respond to a plurality of requests in the order in which the requests are large, or to respond based on the thought of the operator. Anyway, according to this invention, it becomes possible to operate according to the user's request regarding the live video through chat.

  According to the first to third aspects (8 to 10) of the present invention, the live video processing apparatus or the live image processing apparatus performs a process of receiving a plurality of AR markers obtained from a plurality of marker presenting media simultaneously captured by the camera. Markers that match these markers are searched by the video relay processing device from the markers recorded in the marker storage device, and live images corresponding to the search markers are determined from the live images of the plurality of live cameras. The live video can be transmitted to the display device on the camera side. This is a further improvement of the above-mentioned Japanese Patent Application No. 2014-031091.

  When a plurality of markers are received from a plurality of marker presenting media each having an AR marker and matched with the same set of markers recorded in the marker storage device attached to the set of markers, the same set is handled. The live video is transmitted to the camera side and displayed on the display device so as to be superimposed on the video of the marker presenting medium. For the AR marker, a beautiful image itself, a character string, a QR code, a chameleon code (both are registered trademarks), and the like can be used. In the case of an image, a compressed image may be used to reduce the communication burden.

  The live video processing device or the live video relay processing device takes a matching for each of a plurality of AR markers, and determines a live video corresponding to the matching only when all the markers are matched. However, when matching is obtained with that number of markers, the live video corresponding to this subset is determined, or matching is performed as a whole instead of matching individual AR markers. However, it is free to design with which configuration. As for the combination of a plurality of AR markers, the above-described thing is realized, and a specific live video is assigned to each of the marker presentation media, and even one AR can be enjoyed. May be designed to Also, the back side of the marker presentation medium may be used. In this case, the number of live images that can be displayed in AR is greatly increased or the display quality itself is changed by combining the front and back of the marker presentation medium. it can.

  Here, a description will be given of a plurality of marker presentation media. When the marker presenting medium is a trading card, for example, four trading cards each printed with an AR marker are arranged two by two in the vertical and horizontal directions, and four are arranged in a horizontal row. In the present invention, a single card divided into four sections and an AR marker added to each section is defined as a plurality of marker presenting media. This is because if one card after the AR marker is added is separated without breaking each section, a plurality of cards, that is, marker presenting media are obtained.

  For example, the AR marker image is used as a stamp image, and a printed material that can be substantially regarded as a plurality of marker presenting media is provided by providing a partitioned space for printing a plurality of stamps so as not to overlap each other. Can be. A stamp rally card can be assumed as an example of this application. In a game that goes around checkpoints and stamps stamped spaces, the live video processing device or live video relay processing device determines the live video that corresponds to the stamp, and this particular live video is the stamp rally. Overlaid on the card. That is, the stamp rally card with the stamp is provided with a plurality of AR markers.

  In the above description, the stamp is described. However, it is possible to provide a type in which a sticker image is attached instead of the stamp. It should be noted that collecting a plurality of stamps and sticker images is an addition, but it is also possible to design a subtraction. For example, if a stub is cut off at the time of admission, such as an exhibition ticket, different matching results can be obtained for the AR marker before it is cut (before admission) and after it is cut (after admission). It is possible to design. Therefore, depending on the case, different live images can be displayed over the exhibition ticket.

  Moreover, a different kind of marker presentation medium can be used for at least one of the plurality of marker presentation media. For example, a combination of a card and a CD jacket, a pocket tissue wrapping paper, a magazine cover and a promotional fan, or even a combination of a card and a plastic bottle shrink film. Thus, the present invention can be utilized.

  The above augmented reality processing method can be provided as a program for causing a computer to execute the method. Moreover, it can provide as a computer-readable recording medium which recorded this program.

  According to the first invention, the AR marker photographed by the camera is received and processed via the network, the live video processing device determines the live video of the live camera corresponding to the AR marker, and the live video is displayed on the camera side. By performing transmission processing on the device, the display device on the camera side can display the live image superimposed on the image of the marker presentation medium. This makes augmented reality technology even more interesting and its use is more useful and widespread.

  According to the second invention, the AR marker photographed by the camera is received and processed via the network, the live video relay processing device determines the live video of the live camera corresponding to the AR marker, and the live video is transmitted to the relay device. Accordingly, the transmission processing is performed on the camera-side display device, and the camera-side display device can display the live image superimposed on the image of the marker presentation medium. Augmented reality technology has become even more interesting, and its applications are becoming more useful and widespread. In the case where the relay device includes a live video storage device, there is an advantage that the access concentration problem can be relatively easily avoided by the transmission processing via the storage device.

  According to the third invention, the AR marker photographed by the camera is received and processed via the network, the live video processing device determines the live video of the live camera corresponding to the AR marker, and the live video is displayed on the camera side. By performing transmission processing to the device, the display device on the camera side can display the live image superimposed on the image of the marker presentation medium. A live camera that can move with the user, and a chat system for communicating between the live video processing device operator and the user viewing the live video on the display device. According to this, there is an effect that it is possible to accept and respond to the live scene that the user wants to see by chat.

  1 is an explanatory diagram of a system according to Embodiment 1. FIG.   It is explanatory drawing which represented the AR display part of Example 1 with the block diagram.   FIG. 3 is an explanatory diagram illustrating a live video processing unit according to the first embodiment in a block diagram.   It is explanatory drawing showing the process step of AR display part of Example 1. FIG.   6 is an explanatory diagram illustrating processing steps of a live video processing unit according to Embodiment 1. FIG.   It is explanatory drawing showing the process step of AR display part of Example 1. FIG.   It is explanatory drawing which represents AR display of Example 1 typically.   It is explanatory drawing of the system of Example 2. FIG.   It is explanatory drawing which represented the live video relay process part of Example 2 with the block diagram.   It is explanatory drawing of the operation principle of Example 2. FIG.   FIG. 10 is an explanatory diagram illustrating processing steps of an AR display unit according to the second embodiment.   12 is an explanatory diagram illustrating processing steps of a live video relay processing unit according to Embodiment 2. FIG.   It is explanatory drawing which represents AR display of Example 3 typically.   10 is an explanatory diagram illustrating processing steps of a live video processing unit according to Embodiment 3. FIG.   It is explanatory drawing which represented the AR display part of Example 4 with the block diagram.   It is explanatory drawing which represented the live video processing part of Example 4 with the block diagram.   It is explanatory drawing showing the process step of AR display part of Example 4. FIG.   10 is an explanatory diagram illustrating processing steps of a live video processing unit according to Embodiment 4. FIG.   It is explanatory drawing of the operation principle of Example 4.   It is explanatory drawing which represents AR display of Example 4 typically.   FIG. 10 is an explanatory diagram illustrating, in a block diagram, a live video processing unit according to a fifth embodiment.   FIG. 10 is an explanatory diagram illustrating processing steps of a live video processing unit according to a fifth embodiment.   It is explanatory drawing of the marker presentation medium of Example 6. FIG.   It is explanatory drawing of the marker presentation medium of Example 7. FIG.

  This embodiment will be described with reference to FIGS. A smartphone 1 having a display 10 on the front side and a camera 11 on the back side is connected to the Internet wirelessly. A ticket authentication server 2 and a live camera 20 belonging to the ticket authentication server 2 are also connected to the Internet (FIG. 1).

  When the AR display unit 4 of the smartphone 1 is represented by the block diagram of FIG. 2, the AR display unit 4 includes an imaging unit 40, a display unit 41, and a transmission / reception unit 42. On the other hand, when the live video processing unit 5 of the ticket authentication server 2 is represented by the block diagram of FIG. 3, the live video processing unit 5 includes a search unit 50, a ticket image storage unit 51, a video processing unit 52, a transmission / reception unit 53, and the Internet. The remote live camera unit 54 is connected to the remote live camera unit 54.

  The operation of each block is as follows. That is, as shown in FIG. 4, when the image of the ticket 3 is captured by the imaging unit 40 of the camera 11 of the smartphone 1 (step S <b> 1), the captured image is processed by the display unit 41 of the AR display unit 4 and displayed on the display 10. At the same time, a portion of the photograph 30 printed on the ticket 3 is cut out from the photographed image (step S2), and the photograph 30 is sent to the ticket authentication server 2 via the Internet as a AR marker via the Internet (step S2). S3).

  As shown in FIG. 3, the ticket image storage unit 51 and the live camera 54 should be noted on the ticket authentication server 2 side. In the ticket authentication server 2, when the transmission / reception unit 53 receives the photographed image (step S4), the video processing unit 52 searches the ticket image storage unit 51 for an image that matches the photographed image using the search unit 50 ( Step S5) If the ticket 3 is found, it authenticates the ticket 3 as authentic, and the live camera unit 54 transmits the live image currently being shot to the smartphone 1 of the user who is the owner of the ticket 3. Thus, the video processing unit 52 issues a command (step S6), and thus the live camera unit 54 transmits the live image being shot to the smartphone 1 (step S7). As the image data in the ticket image storage unit 51, the same image data as that of the photograph 30 printed on the card 3 is prepared. Therefore, as long as this card 3 is used, a considerable live image can be obtained. The ticket authentication server 2 and the live camera 2 may be installed at the same place. Note that the second live camera 20 ′ is indicated by a chain line in preparation for arranging a plurality of live cameras and displaying live images side by side or switching them.

  Although the smartphone 1 is displaying the captured image 31 of the previously captured ticket 3 on the display 10, the live image transmitted from the live camera unit 54 is received by the transmitting / receiving unit 42. Then, the live video 32 is displayed on the display unit 41 so as to be superimposed on the captured image 31 (step S9). Therefore, for example, it is possible to display a dressing room scene at this time on a concert ticket so as to emerge, and the effect of further enhancing the exhilaration of the user before the concert is outstanding.

  This embodiment will be described with reference to FIGS. What was the ticket authentication server 2 in the first embodiment is a live video relay processing server 21 in this second embodiment. In addition to the live camera 20, a relay video storage device 22 is connected to the Internet as belonging to this side. ing. On the other hand, the smartphone 1 is characterized in that it compresses the image captured by the camera 11 as described later.

  As shown in FIG. 11, when an arbitrary ticket 3 is photographed by the camera 11 of the smartphone 1 (step S10), the photographed image is processed by a display unit (not shown) and displayed on the display 10, and the ticket 3 is obtained from the photographed image. A portion of the photo 30 printed as an AR marker is cut out and compressed (step S11), and the compressed image data is transmitted to the live video relay processing server 21 via the Internet via the transmission / reception unit (step S11). S12).

  The live video relay processing server 21 is connected to the live camera unit 54 of the live camera 20 and the live video relay unit 56 of the relay video storage device 22 via the Internet. That is, the live video relay processing server 21 is substantially composed of a live video relay processing unit 55, a live camera unit 54, and a live video relay unit 56.

  In the live video relay processing unit 55, when the transmission / reception unit 53 receives the compressed image data (step S13), the video processing unit 52 uses the search unit 50 to display a compressed image that matches the received compressed image as a ticket image storage unit. 51 is searched out (step S14), the live camera 20 associated with the searched compressed image is determined (step S15), and the live video of the live camera 20 determined in this way is displayed on the smartphone 1. Is directed to the relay processing unit 57 of the live video relay unit 56 (step S16). When the video processing unit 52 receives the data of the photographed image (c), the image (a, b, c, d, e, f, g,...) Recorded in the ticket image storage unit 51 is hit and the image is hit. (C) is searched, but since the live camera (C) is linked to the image (c), it is determined that this live camera 20 (C) is used, and the relay processing unit of the live video relay unit 56 57 is taken over as described above (FIG. 12). The search unit 50 compares the images (FIG. 10). At this time, the compressed image in the ticket image storage unit 51 is processed by the same compression algorithm used when compressing the photograph 30. Therefore, both of them after compression are roughly matched. This matching ratio may be set arbitrarily. In this embodiment, the search unit 50 compares compressed images with each other. However, it is possible to configure such that comparison is performed without compression or comparison is performed after decompression. is there.

  In the relay video storage device 22, the live video relay unit 52 receives the live video from the live camera unit 54 determined by the relay processing unit 57 through the transmission / reception unit 58 and temporarily records it. In this embodiment, it is a very short time of several minutes. Since the use of the live camera 20 (C) is designated by the video processing unit 52, the relay processing unit 57 transmits the live video to the smartphone 1 in a so-called chasing reproduction. Therefore, the access from the smartphone 1 is concentrated on the live video relay unit 56 without concentrating the access on the live camera unit 54, and the live video distribution process is smoothly performed. .

  This embodiment will be described with reference to FIGS. In the second embodiment described above, the search unit 50 compares the images and determines that the live camera 20 (C) is used from among a plurality of live cameras. On the other hand, in this third embodiment, if the user authenticates that he / she has a genuine ticket as in the first embodiment, the live video to be viewed is first photographed on the display 10 of the smartphone 1. It is characterized in that AR display is performed so that the user can freely select the option display 34 of the live camera so as to be superimposed on the photographed image 31 of the ticket 3 that has been made. Hereinafter, a description will be given with reference to the first embodiment.

  When the image of the ticket 3 is captured by the imaging unit 40 of the camera 11 of the smartphone 1, this is processed by the display unit 41 of the AR display unit 4 and displayed on the display 10, and is printed on the ticket 3 from this captured image. A portion of the photograph 30 is cut out, and the photograph 30 is sent as an AR marker to the ticket authentication server 2 via the Internet via the transmission / reception unit 42. Then, in the ticket authentication server 2, when the transmission / reception unit 53 receives the captured image (step S17), the video processing unit 52 searches the ticket image storage unit 51 for an image that matches the captured image using the search unit 50. (Step S18) From now on, the feature of the third embodiment is described. When the ticket is searched, the ticket 3 is authenticated as authentic, and the smartphone 1 of the user who is the owner of the ticket 3 is authenticated. A plurality of live camera options prepared, i.e., live video options are transmitted, the user selection result is awaited (step S19), and when the selection result information is received, the selected live camera unit 54 is currently shooting. The video processing unit 52 issues a command so as to send the live image to the smartphone 1 (step S20). The live camera unit 54 transmits the control information a live image being captured in the smartphone 1 (step S21). The user can enjoy the convenience of freely selecting a live image to be viewed.

  This embodiment will be described with reference to FIGS. The AR display unit 6 of the smartphone is represented by the block diagram of FIG. The AR display unit 6 includes an imaging unit 60, a display unit 61, an AR marker reading unit 62, and a transmission / reception unit 63. On the other hand, when the live video processing unit 7 is represented by the block diagram of FIG. 16, the live video processing unit 7 includes a search unit 70, an AR marker storage unit 71, a video processing unit 72, and a transmission / reception unit 73. It also has a plurality of live camera units 74 connected to the Internet.

  As in the first embodiment, the smartphone used in this embodiment includes a display 10 on the front side and a camera 11 on the back side, and is connected to the Internet wirelessly. In this embodiment, when four trading cards 35 are used and the card 35 on which the AR marker 36 is printed is photographed by the camera 11 of the smartphone (step S22), the photographed image is processed by the imaging unit 60 of the AR display unit 6. The display unit 61 displays the AR marker 36 on the card 35 from the captured image, and the AR marker reading unit 62 reads the AR marker 36 from the captured image (step S23). The live image is transmitted to the server for processing live video via the Internet (step S24). In this embodiment, a QR code is used for the AR marker 36.

  On the live video processing server side, when the transmitting / receiving unit 73 of the live video processing unit 7 receives the AR marker 36 data (step S25), the video processing unit 72 uses the search unit 70 to convert the received AR marker 36 data. The matching marker data is searched from the AR marker storage unit 71 (step S26), and the live camera unit 74 linked to the searched AR marker is determined (step S27). The live camera unit 74 determined in this way is instructed to transmit the live video being shot to the smartphone via the transmission / reception unit 73 (step S28). In response to this, the live camera unit 74 transmits the live video to the smartphone (step S29). As the marker data in the AR marker storage unit 71, the same data as the AR marker 36 data printed on the card 35 is prepared. Therefore, as long as this card 33 is used, a considerable live image can be obtained.

  By the way, each of the four cards 35 (1, 2, 3, 4) is printed with a unique AR marker 36 (i, ii, iii, iv) together with a unique photograph. Therefore, for example, when a single card 35 (1) is attached and photographed with a camera and the data of the marker 36 (i) of this card 35 (1) is transmitted to the live video processing server, the AR marker storage unit 71 stores the data. The marker (i) is searched from the recorded single marker data and combination marker data. Since the live camera (I) is linked to the marker (i), the live video is displayed as the live camera. (I) will be transmitted to the smartphone. Then, on the smartphone side, the live image of the live camera (I) is displayed on the display 10 so as to be superimposed on the captured image 35 displayed earlier. Similar processing is performed for each of the cards 35 (2, 3, 4).

  However, this embodiment is not limited to the processing described above, and the video determination processing is determined even when four cards 35 (1, 2, 3, 4) are arbitrarily combined and photographed by the camera. . That is, if any two of the four cards 35 are combined, any three are combined, or a set of four is photographed, the live video processing server 74 matches the respective combinations on the live video processing server side. It was configured to send live video to the smartphone. Referring to FIGS. 19 and 20, the case of a set of four cards will be described. On the display 10 of the smartphone, four cards 35 (1, 2, 3, 4) arranged in two vertical and horizontal rows as shown in FIG. Although displayed, the live camera (IV) corresponding to this combination from the images of the markers 36 (i, ii, iii, iv) printed on each of the plurality of live cameras 74 (I, II). , III, IV,...), The live video being shot by the live camera (IV) is returned to the smartphone, and as a result, the live video 38 (iv) is the above four cards. 35 (1, 2, 3, 4).

  In addition, it is possible to determine the arrangement order of the AR markers 36 and determine the live camera unit 74 corresponding thereto. For example, it is designed so that a vertical single row arrangement, a horizontal single row arrangement, or the like can be discriminated.

  This embodiment will be described with reference to FIGS. The live video processing server side includes a main live video processing unit 8, a live camera unit 84, and a switch switching unit 85. Although the live camera unit 84 composed of a plurality of live cameras is distributed and installed in a remote place, the live video processing unit 8 and the switch switching unit 85 are constructed on the same server. The feature of the switch switching unit 85 is that it includes a switch unit 87 that switches the live camera 84 as described later, and a chat unit 86 that communicates with the smartphone side by characters. The chat unit 86 and the switch unit 87 are operated by an operator on the live video processing server side. On the other hand, a chat application (application software) is installed on the smartphone side.

  When the video processing unit 82 of the live video processing unit 8 receives the ticket image from the smartphone using the transmission / reception unit 83 (step S30), the search unit 80 displays an image that matches the ticket image in the ticket image storage unit 81. If they match, the ticket is authenticated as authentic (step S31). Therefore, the user of this smartphone has the right to AR display the live video from the live camera unit 84 and can chat with the operator to switch the live video.

  When the authentication process described above is completed, a predetermined live video is transmitted from the live camera unit 84 and is set to be displayed as an AR on the smartphone side. When there is a switching request, the operator switches the live camera of the live camera unit 84 using the switch unit 87, that is, instructs the live camera of the switched destination to transmit the live video to the smartphone ( In step S32), a new live video is transmitted from the switched live camera to the smartphone side (step S33).

  Here, switching between a plurality of fixed live cameras has been described. However, for example, it is possible to perform a chat with a photographer carrying a live camera to move the shooting location. In any case, it is possible to provide a service that enables collaboration with the user and AR displays live video that meets the user's request as much as possible.

  In the above-described fourth embodiment, four cards 35 (1, 2, 3, 4) are collected and arranged side by side to hold the camera. That is, a plurality of AR markers are obtained by photographing a plurality of marker presentation media. It will be described here that this category includes stamp cards such as this embodiment.

  FIG. 23 shows a stamp rally sheet 39 of this embodiment, on which a section 300 of 3 rows and 3 columns is printed. If you hold the camera in this state, nothing will happen. However, by marking the stamp 301 in the section 300, a live image corresponding to the number of the stamps 301 is designed to be displayed over the stamp 301 of the sheet 39. That is, in the sheet 39 of this embodiment, each section 300 can be regarded as an independent print medium.

  Instead of stamping the stamp 301 in the section 300, it is also possible to provide a sticker with a sticker 302 on which an illustration 303 or the like is drawn.

  So far, the example of using a card-like combination in the marker presentation medium has been taken up, but the marker presentation medium is not limited to a card, and a combination of a plurality of different types of marker presentation mediums. It may be used.

  FIG. 24 shows an embodiment in which a combination of a CD jacket 304, a pocket tissue 305, and a trading card 306 is used as a print medium. That is, only when these three types are collected, when the camera is held over, live images from the live camera are displayed superimposed on the images of the three types of marker presentation media.

  The present invention is not limited to a marker presenting medium with an AR marker, and can be widely applied to, for example, a predetermined combination of persons and objects collected in a landscape. An AR marker can be attached to a building, but in the case of a building, GPS position information or camera orientation information located on the map may be used. These are also within the scope of this invention.

1: Smartphone 10: Display 11: Camera 2: Ticket authentication server 20: Live camera 21: Live video relay processing server 22: Relay video storage device 3: Ticket 30: Photo 31: Captured image 32: Live video 33: Photo 34: Choice display 35: Card 36: AR marker 37: Captured image 38: Live video 39: Sheet 300: Section 301: Stamp 302: Sticker 303: Illustration 304: CD jacket 305: Pocket tissue 306: Trading card 4: AR display section 40 : Imaging unit 41: Display unit 42: Transmission / reception unit 5: Live video processing unit 50: Search unit 51: Ticket image storage unit 52: Video processing unit 53: Transmission / reception unit 54: Live camera unit 55: Live video relay processing unit 56: Live video relay unit 57: Relay processing unit 58 : Transmission / reception unit 6: AR display unit 60: Imaging unit 61: Display unit 62: AR marker reading unit 63: Transmission / reception unit 7: Live video processing unit 70: Search unit 71: AR marker storage unit 72: Video processing unit 73: Transmission / reception Unit 74: Live camera unit 8: Live video processing unit 80: Search unit 81: Ticket image storage unit 82: Video processing unit 83: Transmission / reception unit 84: Live camera unit 85: Switch switching unit 86: Chat unit 87: Switch unit

Claims (16)

  A camera for shooting a marker presenting medium with an AR marker, and a live camera for shooting a live video; the live marker corresponding to the AR marker received by the AR marker shot by the camera; A live video processing device that transmits a live video of the camera to the display device on the camera side, a display device that receives the live video and displays the live video superimposed on the marker presentation medium, and communication of the AR marker and the live video. Augmented reality system consisting of a network such as the Internet.   A camera for shooting a marker presenting medium with an AR marker, a live camera for shooting a live video, and a relay device for relaying the live video of the live camera, which is shot by the camera A live video relay processing device that receives the AR marker and transmits the live video of the live camera corresponding to the AR marker to the display device on the camera side via the relay device; An augmented reality system comprising: the display device that displays a marker presentation medium in a superimposed manner; and a network such as the Internet that mediates communication of the AR marker and relay of the live video.   Provided with a plurality of the live cameras, the live video processing device or the live video relay processing device presents selectable options on the display device side so as to select which live video is desired to be viewed, The augmented reality system according to claim 1 or 2, wherein a live video of the live camera selected on the display device side is specified as a live video to be transmitted to the display device.   A camera for shooting a marker presenting medium with an AR marker, and a live camera for shooting a live video; the live marker corresponding to the AR marker received by the AR marker shot by the camera; A live video processing device that transmits a live video of the camera to the display device on the camera side, a display device that receives the live video and displays the live video superimposed on the marker presentation medium, and communication of the AR marker and the live video. An intermediary network such as the Internet, further comprising a plurality of the live cameras or the live camera movable with the photographer, and the live video processing device operator side and the display device The live scene that this user wants to see depends on the operator side by chat with the user who sees the live video. And it includes a chat system for, augmented reality system.   Upon receiving a plurality of AR markers obtained from a plurality of marker presentation media photographed simultaneously by the camera, the live video processing device or the live video relay processing device stores markers matching these markers in a marker storage device. Searching among recorded markers, determining a live video corresponding to the search marker from a plurality of live videos of the live camera, and transmitting the live video to a display device on the camera side. The augmented reality system according to any one of claims 1 to 3.   The augmented reality system according to claim 5, wherein at least one of the plurality of marker presentation media is a different type of marker presentation media.   The AR marker image is a stamp image obtained by imprinting a stamp or a seal image obtained by applying a sticker, and a plurality of stamps are printed so as not to overlap each other. The augmented reality system according to claim 5, wherein a marker presenting medium that can be regarded as a plurality of marker presenting media is provided by providing a partitioned space to be attached.   The AR marker on the marker presentation medium with the AR marker photographed by the camera is received and processed via a network such as the Internet, and the live video of the live camera corresponding to the AR marker is determined by the live video processing device. An augmented reality processing method of performing transmission processing to a display device on a camera side so that a live image is displayed on the display device on the camera side so as to be superimposed on the image of the marker presentation medium.   The AR marker of the marker presentation medium with the AR marker taken by the camera is received and processed via a network such as the Internet, and the live video relay processing device determines the live video of the live camera corresponding to the AR marker, An augmented reality processing method of performing transmission processing on a display device on the camera side so that the live image is displayed on the display device on the camera side so as to be superimposed on the video on the marker presentation medium via a relay device.   The live video processing device or the live video relay processing device presents an option on the display device side so that the user can select which live video of the live video of the plurality of live cameras is to be viewed. The augmented reality processing method according to claim 1, wherein a live video of the live camera selected on the display device side is specified as a live video to be transmitted to the display device.   The AR marker of the marker presentation medium with the AR marker taken by the camera is received and processed via a network such as the Internet, and a live video of the live camera corresponding to the AR marker is determined by a live video processing device. Transmission processing to the display device on the camera side so that a live image is displayed on the display device on the camera side so as to be superimposed on the image of the marker presentation medium, and a plurality of the live cameras are provided or movable with the photographer An operation that includes the live camera and accepts a live scene that the user wants to see by chat using a chat system for communication between an operator of the live video processing device and a user who views the live video on the display device. Performing an augmented reality processing method.   A marker storage device that performs a process of receiving a plurality of AR markers obtained from a plurality of marker presentation media photographed simultaneously by the camera and that matches the markers by the live video processing device or the live video relay processing device A live image corresponding to the search marker is determined from the live video of the plurality of live cameras, and the live video is transmitted to the display device on the camera side. The augmented reality processing method according to any one of claims 8 to 10.   The augmented reality processing method according to claim 12, wherein at least one of the plurality of marker presentation media is a different type of marker presentation media.   The AR marker image is a stamp image obtained by imprinting a stamp or a seal image obtained by applying a sticker, and a plurality of stamps are printed so as not to overlap each other. The augmented reality processing method according to claim 12, wherein a marker presenting medium that can be substantially regarded as a plurality of marker presenting media is provided by providing a partitioned space for being attached.   The program which makes a computer perform the augmented reality processing method of any one of Claims 8 thru | or 14.   A computer-readable recording medium storing a program for causing a computer to execute the augmented reality processing method according to claim 8.

Images