JP3210555U - Annotation system - Google Patents

Abstract

An annotation system for annotating and displaying a virtual 3D image on the display of real field of view information is provided. An image recorded in an image storage unit is searched for a photographed image of a photograph 30 which is an AR marker photographed by a camera 11, and a virtual three-dimensional image previously associated with the retrieved image is virtually retrieved. The annotation system reads out from the 3D video storage unit, annotates the photograph 30 and displays it on the display 10. A plurality of cards 3 each printed with an AR marker are photographed with a camera, and the AR marker recorded in the AR marker storage unit is searched with the AR marker, and is associated with the searched AR marker in advance. The annotation system is such that the virtual 3D video is read from the virtual 3D video storage unit, annotated on the card 3 and displayed on the display 10. [Selection] Figure 1

Description

  The present invention relates to an annotation system that displays a card or various media printed with a photograph or a pattern on a video shot with a camera so that a 3D video associated with the video is superimposed.

  As disclosed in Japanese Patent Application Laid-Open No. 2011-242816, augmented reality (AR) technology for annotating and displaying a virtual 3D image on the display of actual view information captured by a camera is disclosed. In general, in this AR technology, an identifier such as a one-dimensional code or a two-dimensional code called an AR marker is used, a camera captures a real field of view including the AR marker, and a virtual three-dimensional image associated with the AR marker. An image is searched for, and this virtual 3D image is displayed as an annotation on the captured real field of view information. In JP 2011-242816 A, GPS position information and azimuth information are used as an AR marker.

  Japanese Patent Laid-Open No. 2013-198520 detects a figure from a plurality of real cards photographed by a camera in order to reflect information recorded on a real-world card in processing executed by a game device. It is said that the user selects any one or more graphics and executes the processing associated with the selected graphics. Therefore, the user can enjoy the game by taking the information of the card possessed by the user into the virtual world and changing the progress of the game according to the figure selected by the user among the detected figures.

JP 2011-242816 A JP2013-198520A

  The technique of annotating and displaying virtual 3D video on the display of real field information is novel and fun. However, it has been found that a technique using an identifier such as a QR code (registered trademark, the same applies hereinafter) for the AR marker, for example, the technique disclosed in Japanese Patent Laid-Open No. 2013-198520 has the following problems.

  In other words, an identifier such as a QR code is used for the AR marker, but if the card is printed with a beautiful illustration or photo, it would be unpleasant to see the identifier such as a QR code in it. I remember it.

  In AR technology, it is fun to automatically annotate a virtual 3D image on the display of the actual view information over the camera, but it can be solved even if the above aesthetic problems remain. Even in the case of Japanese Patent Application Laid-Open No. 2013-198520, if the user selects a figure from the detected figures and executes processing corresponding to the figure, the operation of selecting the figure is This operation is troublesome because it is required by the user.

  Therefore, the present invention aims to solve one or both of these problems.

  Accordingly, the invention of claim 1 is directed to an AR marker photographed by a camera by a video determination device in order to solve the aesthetic problem when annotating and displaying a virtual 3D video on the display of actual view information. A virtual three-dimensional image that receives the image of the card on which the image is printed and searches for an image recorded in the image storage device by using this image and searches for a matching image, and is previously associated with the searched image The annotation system is configured to determine a video, receive the virtual 3D video read from the video storage device, and annotate and display the image on a display device. The AR marker is a beautiful image itself.

  In order to solve the problem that the above-described figure selection bothers the user when annotating and displaying a virtual three-dimensional image on the display of actual view information, To search for a marker recorded in the image storage device using these markers and search for a matching marker and receive the marker. An annotation system that determines a virtual 3D video that is associated with an outgoing marker in advance, receives the virtual 3D video read from the video storage device, and annotates and displays the AR markers on a display device. . In a series of these processes, the virtual 3D video is fully automatically displayed on the print medium by simply holding the camera over a plurality of print media.

  The feature of the invention of claim 1 is that the image itself is an AR marker. When a card is used as the AR marker medium and an image such as a beautiful illustration or photo printed on the card taken by the camera matches an image recorded in advance in the image storage device, The corresponding virtual 3D image is displayed on the image taken by the camera as an annotation. Therefore, the card has no nuances such as barcodes and color codes, and the beauty of illustrations and photographs can be kept as they are.

  Note that a compressed image can be used as the received image for matching processing. For example, the annotation system of the present invention is transmitted to the server with the video determination device via the network such as the Internet, and the virtual 3D video obtained here is carried by the above-mentioned image taken by a camera of a mobile terminal such as a smartphone. When the network system is configured to transmit to a terminal and display the image with annotations on the display device, the image is preferably compressed regardless of the network load. This compressed image is decompressed before matching processing is performed by the video determination device, but the images recorded in both image storage devices are also subjected to compression processing using the same algorithm, and both images are not decompressed. It is also possible to realize a method in which comparison processing is performed as is.

  The feature of the invention of claim 2 is that the AR marker is not limited to an image as long as it can be a marker such as a QR code or a chameleon code (registered trademark), and the medium of the AR marker is not limited to a card. Any marker that can be printed, such as a flyer or a fan, can be used. When a plurality of markers are received from a plurality of print media each printed with an AR marker and matched with the same set of markers recorded in the image storage device attached to the set of markers, the same set is handled. The virtual 3D video is annotated by overlapping it with the image taken by the camera.

  In order to determine the virtual 3D image corresponding to the number of AR markers only when matching is obtained for each of the plurality of AR markers and matching is obtained for all the markers, When matching is obtained with several markers, the virtual 3D image corresponding to this subset is determined, or the matching is performed as a whole instead of matching individual markers. It is possible to design with any configuration. As for the combination of a plurality of AR markers, the above can be realized, and a specific virtual 3D image can be assigned to each one of each print medium so that even one image can be enjoyed. May be designed to Also, the back side of the print medium may be used. In this case, the number of virtual 3D images that can be annotated can be greatly increased or the display quality itself can be changed by combining the front and back sides of the print medium. it can.

  Here, the term “a plurality of print media” will be described. In an example in which the print medium is a card, for example, four so-called trading cards each printed with an AR marker are arranged two by two in the vertical and horizontal direction, and four are arranged in a horizontal row. In this device, a single card is divided into four sections, and an AR marker added to each section is defined as a plurality of print media. This is because if one card after the AR marker is added is separated without breaking each section, a plurality of cards, that is, print media are obtained.

  For example, the AR marker image is used as a stamp image, and a print medium that can be substantially regarded as a plurality of print media is provided by providing a divided space for printing a plurality of stamps so as not to overlap each other. Can be. A stamp rally card can be assumed as an example of this application. In a game that goes around checkpoints and stamps the above divided spaces, the video determination device determines a virtual 3D video corresponding to the collected stamp, and this specific virtual 3D video is displayed on the stamp rally card. Is displayed as an annotation. The stamp rally card with the stamp is provided with a plurality of AR markers.

  Next, a different type of print medium can be used for at least one of the plurality of print media. For example, a card and CD jacket, a pocket tissue wrapping paper, a magazine cover and a promotional fan, or a card and a plastic bottle shrink film can be used in a totally free way of thinking. It is possible to utilize this idea.

  Next, the AR marker is an image, and the search image can be matched with an image captured without overlapping the plurality of images. Although the AR marker is an image such as an illustration or a photograph, it has already been explained. However, the image determination device can capture a virtual tertiary that matches a combination of these images by shooting so that a plurality of images do not overlap. The original video can be determined more accurately. Note that the video determination device may be designed so that matching can be achieved with overlapping images, but this may impair the beauty of photographs and illustrations that should serve as the basis for annotation display.

  Note that an image can be used as the AR marker, and a compressed image can be used as the received image for matching processing. In the present invention, a plurality of AR markers are processed. If the AR marker is an image, for example, the above-described image obtained by photographing the annotation system of the present invention with a camera of a mobile terminal such as a smartphone is imaged via a network such as the Internet. When the network system is configured to transmit the virtual 3D image obtained here to the server with the determining device, transmit it to the mobile terminal, and display the image with annotation on the display device, the network load should be considered. The image is preferably compressed. This compressed image is decompressed before the matching process is performed by the video determination device, but the image recorded in the image storage device is also subjected to the compression processing with the same algorithm, and both images are not decompressed. It is also possible to realize a system configured to perform comparison processing as it is.

  According to the first aspect of the present invention, the image determination device receives the image of the card on which the image that is the AR marker taken by the camera is printed, and searches and matches the image recorded in the image storage device with this image. An image to be searched is determined, a virtual 3D image previously associated with the searched image is determined, the virtual 3D image read from the image storage device is received, and the image is annotated on the display device Thus, it has succeeded in solving the problem that a conventional AR marker using an identifier such as a QR code damages a beautiful illustration or photo of a card.

  According to the invention of claim 2, the image determination device receives the markers of a plurality of print media each printed with an AR marker taken by the camera, and records the markers recorded in the image storage device with these markers. Search and search for a matching marker, determine a virtual 3D image previously associated with the searched marker, receive the virtual 3D image read from the image storage device, and display a plurality of images on the display device. Since the AR marker is annotated and displayed, the virtual 3D image is automatically displayed on the print medium by simply holding the camera over a plurality of print media. Thus, there is an effect that it is not necessary for the user to select a print medium from among the printed print media.

  1 is an explanatory diagram of a system according to Embodiment 1. FIG.   It is explanatory drawing which represented the AR display part of Example 1 with the block diagram.   It is explanatory drawing which represented the AR process part of Example 1 with the block diagram.   It is explanatory drawing showing the process step of AR display part of Example 1. FIG.   It is explanatory drawing showing the process step of AR process part of Example 1. FIG.   It is explanatory drawing showing the process step of AR display part of Example 1. FIG.   2 is an explanatory diagram of an operation principle of Embodiment 1. FIG.   It is explanatory drawing which represents the annotation display of Example 1 typically.   It is explanatory drawing which represented the AR display part of Example 2 with the block diagram.   It is explanatory drawing which represented the AR process part of Example 2 with the block diagram.   FIG. 10 is an explanatory diagram illustrating processing steps of an AR display unit according to the second embodiment.   It is explanatory drawing showing the process step of the AR process part of Example 2. FIG.   It is explanatory drawing of the operation principle of Example 2. FIG.   It is explanatory drawing which represents the annotation display of Example 2 typically.   FIG. 10 is an explanatory diagram of a print medium according to a third embodiment.   FIG. 10 is an explanatory diagram of a print medium according to a fourth embodiment.

  This device can be implemented with a so-called stand-alone device such as a personal computer, a video game machine, a smart phone, or a television receiver. However, in the following, an example will be given for an example in which processing is distributed over a network. I will explain. However, this device claims that it is within the scope of rights to implement it on a device that operates in a stand-alone manner.

  This embodiment will be described with reference to FIGS. A smartphone 1 having a display 10 on the front side and a camera 11 on the back side is connected to the Internet wirelessly. An annotation server 2 is also connected to the Internet (FIG. 1).

  When the AR display unit 4 of the smartphone 1 is represented by the block diagram of FIG. 2, the AR display unit 4 includes an imaging unit 40, a display unit 41, and a transmission / reception unit 42. On the other hand, when the AR processing unit 5 of the annotation server 2 is represented by the block diagram of FIG. 3, the AR processing unit 5 includes a search unit 50, an image storage unit 51, a video determination unit 52, a virtual 3D video storage unit 53, and a transmission / reception unit 54. It is composed of

  The operation of each block is as follows. That is, as shown in FIG. 1, when the card 3 is photographed by the camera 11 of the smartphone 1 (step S1), this image is processed by the imaging unit 40 of the AR display unit 4 and displayed on the display 10, and the card 3 is also displayed from this image. A portion of the photograph 30 printed on is cut out and compressed (step S2), and the compressed image data of the photograph 30 is transmitted to the annotation server 2 via the Internet via the transmission / reception unit 42 (step S3).

  As shown in FIG. 7, attention should be paid to the image storage unit 51 and the virtual 3D video storage unit 53 in the annotation server 2. In the annotation server 2, when the transmission / reception unit 54 receives the compressed image data (step S 4), the video determination unit 52 uses the search unit 50 to select a compressed image that matches the received compressed image from the image storage unit 51. The virtual 3D video 32 linked to the searched compressed image is retrieved from the virtual 3D video storage unit 53 (step S6), and the virtual tertiary determined in this way is retrieved. The original video 32 is transmitted to the smartphone 1 through the transmission / reception unit 54 (step S7). The compressed image data in the image storage unit 51 is prepared in the same way as the image data of the photograph 30 printed on the card 3. Therefore, as long as this card 3 is used, considerable virtual 3D video data can be obtained. Since the compressed image in the image storage unit 51 is processed by the same compression algorithm as that used when compressing the photograph 30, the two after compression almost coincide. This matching ratio may be set arbitrarily.

  When the data of the captured image (c) is transmitted to the annotation server 2 now, the image c is obtained by hitting the images (a, b, c, d, e, f, g,...) Recorded in the image storage unit 51. However, since the virtual three-dimensional video (C) is linked to the image (c), the virtual three-dimensional video 32 (C) is transmitted to the smartphone 1. Then, the virtual three-dimensional video 32 (C) is displayed on the smartphone 1 side so as to be superimposed on the captured image 31 (c) displayed on the display 10 (FIG. 8).

  In this embodiment, the search unit 50 compares the compressed images. However, it is possible to compare the compressed images without compression, or to compare them after decompressing them even if they are compressed. It is.

  This embodiment will be described with reference to FIGS. When the AR display unit 6 of the smartphone is represented by the block diagram of FIG. 9, the AR display unit 6 includes an imaging unit 60, a display unit 61, an AR marker reading unit 62, and a transmission / reception unit 63. On the other hand, when the AR processing unit 7 of the annotation server is represented by the block diagram of FIG. 10, the AR processing unit 7 includes a search unit 70, an AR marker storage unit 71, a video determination unit 72, a virtual 3D video storage unit 73, and a transmission / reception unit 74. It consists of and.

  The smartphone used in this embodiment includes a display 10 on the front side and a camera (not shown) on the back side, and is connected to the Internet wirelessly. The AR processing unit 7 of the annotation server is also connected to the Internet.

  In this embodiment, four trading cards 33 are used. When the card 33 on which the AR marker 34 is printed is photographed by the camera of the smartphone (step S10), the photographed image is processed by the imaging unit 60 of the AR display unit 6 and displayed on the display 10, and from the photographed image to the card 33. The printed AR marker 34 is read by the AR marker reading unit 62 (step S11), and the data of the AR marker 34 is transmitted from the transmission / reception unit 63 to the annotation server via the Internet (step S12). In this embodiment, a QR code is used for the AR marker 34.

  On the annotation server side, when the transmission / reception unit 74 of the AR processing unit 7 receives the data of the AR marker 34 (step S13), the video determination unit 72 uses the search unit 70 to obtain marker data that matches the received data of the AR marker 34. A search is made from the AR marker storage unit 71 (step S14), and the virtual 3D video 36 linked to the searched AR marker is read from the virtual 3D video storage unit 73 (step S15). ), The virtual three-dimensional image 36 determined in this way is transmitted to the smartphone via the transmission / reception unit 74 (step S16). The same marker data in the AR marker storage unit 71 as the AR marker 34 data printed on the card 33 is prepared. Therefore, as long as the card 33 is used, a considerable amount of virtual 3D video 36 data can be obtained.

  By the way, each of the four cards 33 (1, 2, 3, 4) is printed with a unique photo and a unique AR marker 34 (i, ii, iii, iv). Therefore, for example, when a single card 33 (1) is photographed with a camera and the data of the marker 34 (i) of this card 33 (1) is transmitted to the annotation server, it is recorded in the AR marker storage unit 71. The marker (i) is searched from the single marker data and the combined marker data. Since the virtual three-dimensional image (I) is associated with the marker (i), the marker (i) is transmitted to the smartphone. It will be. Then, on the smartphone side, the virtual three-dimensional video 36 (I) is reproduced and displayed so as to be superimposed on the captured image 35 previously displayed on the display 10. The same processing is performed for each of the cards 33 (2, 3, 4).

  However, this embodiment is not limited to the above-described processing, and the video determination processing is determined even when four cards 33 (1, 2, 3, 4) are arbitrarily combined and photographed by the camera. . That is, when any two of the four cards 33 are combined, any three are combined, or four are captured together, the annotation server side displays virtual 3D images that match each combination on the smartphone side. It was configured to perform processing to return to. Referring to FIG. 13 and FIG. 14, the case where four cards are arranged will be described. Four cards 33 (1, 2, 3, 4) arranged in two vertical and horizontal rows as shown in FIG. 14 are displayed on the smartphone display. However, from the images of the markers 34 (i, ii, iii, iv) printed on the virtual three-dimensional images 36 (IV) corresponding to this combination, virtual three-dimensional images 36 (I, II, III) are obtained. , IV, ..) is selected and returned to the smartphone, and the virtual 3D image 36 (IV) is played back as annotated on the four cards 33 (1, 2, 3, 4). It is displayed.

  In addition, the arrangement order of the markers 34 can be determined and the corresponding virtual 3D video 36 can be reproduced. For example, it is designed so that a vertical single row arrangement, a horizontal single row arrangement, or the like can be discriminated.

  In the above-described second embodiment, four cards 33 (1, 2, 3, 4) are collected and arranged to hold the camera. That is, a plurality of markers are obtained by photographing a plurality of print media. It will be described here that this category includes stamp cards such as this embodiment.

  FIG. 15 shows a stamp rally sheet 37 of this embodiment, on which a section 38 of 3 rows and 3 columns is printed. Even if the camera is held in this state, nothing happens, but by stamping the stamp 39 in the section 38, a virtual three-dimensional image corresponding to the number of stamps 39 is displayed on the stamp 39 of the sheet 37 as an annotation. Designed to be That is, in the sheet 37 of this embodiment, each section 38 can be regarded as an independent print medium.

  So far, the example of using a combination of card-like print media has been taken up, but the print media is not limited to cards, and the types of multiple print media may be different. It is.

  FIG. 16 shows an embodiment using a combination of a CD jacket 300, a pocket tissue 301, and a trading card 302 as a print medium. That is, only when these three types are collected, a virtual 3D image is displayed as an annotation on the images of the three types of print media when the camera is held over.

  The present invention is not limited to a print medium with an AR marker, and can be widely applied to, for example, a predetermined combination of people and objects collected in a landscape.

1: Smartphone 10: Display 11: Camera 2: Annotation Server 3: Card 30: Photo 31: Captured Image 32: Virtual 3D Video 33: Card 34: AR Marker 35: Captured Image 36: Virtual 3D Video 37: Sheet 38 : Section 39: Stamp 300: CD jacket 301: Pocket tissue 302: Trading card 4: AR display unit 40: Imaging unit 41: Display unit 42: Transmission / reception unit 5: AR processing unit 50: Search unit 51: Image storage unit 52: Video determination unit 53: Virtual 3D video storage unit 54: Transmission / reception unit 6: AR display unit 60: Imaging unit 61: Display unit 62: AR marker reading unit 63: Transmission / reception unit 7: AR processing unit 70: Search unit 71: AR Marker storage unit 72: Video determination unit 73: Virtual 3D video storage unit 74: Transmission / reception unit

Claims (5)

  A camera for shooting a card on which an image as an AR marker is printed, and an image that is received by the camera and that matches the image is searched from the images stored in the image storage device, and the search is performed. An annotation system comprising: a video determination device that determines a virtual 3D video corresponding to an output image; and a display device that receives the virtual 3D video read from the video storage device and displays the image by annotating it.   A camera for photographing a plurality of print media each printed with an AR marker, and a plurality of the markers photographed by the camera are received, and markers that match these markers are recorded in the marker storage device. A video determination device that searches from the marker and determines a virtual 3D video corresponding to the search marker, and receives the virtual 3D video read from the video storage device and displays an annotation on the plurality of images An annotation system comprising a display device.   The AR marker image is used as a stamp image, and the print medium can be substantially regarded as a plurality of print media by providing a partitioned space for marking a plurality of stamps so as not to overlap each other. The annotation system according to claim 2, wherein   The annotation system according to claim 2, wherein at least one of the plurality of print media is a different type of print medium.   The annotation system according to claim 2, wherein the AR marker is an image, and the searched image matches an image captured without overlapping the plurality of images.

Images