WO2019189768A1 - Communication method, communication device, transmitter, and program - Google Patents

Abstract

Provided is a communication method that enables communication between various apparatuses. This communication method includes: determining whether or not a terminal can perform visible light communication (step SG11); when it is determined that the terminal can perform visible light communication (Yes in step GS11), acquiring an image for decoding by imaging a luminance-varying subject using an image sensor, and acquiring first identification information transmitted by the subject from a stripe pattern appearing in the image for decoding (step SG12); and, when it is determined in the determination of visible light communication that the terminal cannot perform visible light communication (No in step SG11), acquiring a captured image by imaging the subject using the image sensor, specifying a prescribed specific region by detecting edges of the captured image, and acquiring second identification information transmitted by the subject from a line pattern in the specific region (step SG13).

Description

COMMUNICATION METHOD, COMMUNICATION DEVICE, TRANSMITTER, AND PROGRAM

The present invention relates to a communication method, a communication device, a transmitter, a program, and the like.

In recent home networks, in addition to cooperation of AV home appliances by IP (Internet Protocol) connection in Ethernet (registered trademark) and wireless LAN (Local Area Network), management of power consumption corresponding to environmental problems, and from outside the home Introduction of a home appliance linkage function in which various home appliances are connected to a network by a home energy management system (HEMS) having a function of turning on / off the power of the home appliance. However, there are home appliances that do not have sufficient computing power to have a communication function, and home appliances that are difficult to install a communication function in terms of cost.

In order to solve such a problem, in Patent Literature 1, in an optical space transmission device that transmits information to free space using light, limited transmission is performed by performing communication using a plurality of monochromatic light sources of illumination light. A technology for efficiently realizing communication between devices is described in the apparatus.

JP 2002-290335 A

However, the conventional method is limited to a case where a device to be applied has a three-color light source such as illumination. Moreover, the receiver which receives the transmitted information cannot display an image useful for the user.

The present invention solves such problems and provides a communication method that enables communication between various devices.

A communication method according to an aspect of the present invention is a communication method using a terminal including an image sensor, and determines whether or not the terminal can perform visible light communication, and the terminal performs visible light communication. When the image sensor determines that it is possible to obtain a decoding image by capturing an image of a subject whose luminance changes, the first image is transmitted from the striped pattern that appears in the decoding image. When acquiring identification information and determining that the terminal is not capable of performing visible light communication in the determination of visible light communication, the image sensor acquires a captured image by capturing the subject, By performing edge detection of the captured image, at least one contour is extracted, a predetermined specific region is specified from the at least one contour, and a line of the specific region The subject from the turn acquires the second identification information to be transmitted.

Note that these comprehensive or specific modes may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, and the system, method, integrated circuit, computer program And any combination of recording media. Moreover, the computer program which performs the method concerning one Embodiment is preserve | saved at the recording medium of the server, and may be implement | achieved in the aspect delivered to a terminal from a server according to the request | requirement of a terminal.

According to the present invention, communication between various devices can be realized.

FIG. 1 is a diagram illustrating an example of an observation method of luminance of a light emitting unit in the first embodiment. FIG. 2 is a diagram illustrating an example of a method of observing the luminance of the light emitting unit in the first embodiment. FIG. 3 is a diagram illustrating an example of a method of observing the luminance of the light emitting unit in the first embodiment. FIG. 4 is a diagram illustrating an example of a method of observing the luminance of the light emitting unit in the first embodiment. FIG. 5A is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 5B is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 5C is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 5D is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 5E is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 5F is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 5G is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 5H is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 6A is a flowchart of the information communication method in Embodiment 1. FIG. 6B is a block diagram of the information communication apparatus according to Embodiment 1. FIG. 7 is a diagram illustrating an example of a photographing operation of the receiver in the second embodiment. FIG. 8 is a diagram illustrating another example of the photographing operation of the receiver in the second embodiment. FIG. 9 is a diagram illustrating another example of the photographing operation of the receiver in the second embodiment. FIG. 10 is a diagram illustrating an example of display operation of the receiver in Embodiment 2. FIG. 11 is a diagram illustrating an example of display operation of the receiver in Embodiment 2. FIG. 12 is a diagram illustrating an example of operation of a receiver in Embodiment 2. FIG. 13 is a diagram illustrating another example of operation of a receiver in Embodiment 2. FIG. 14 is a diagram illustrating another example of operation of a receiver in Embodiment 2. FIG. 15 is a diagram illustrating another example of operation of a receiver in Embodiment 2. FIG. 16 is a diagram illustrating another example of operation of a receiver in Embodiment 2. FIG. 17 is a diagram illustrating another example of operation of a receiver in Embodiment 2. FIG. 18A is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 2. FIG. 18B is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 2. FIG. 18C is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 2. FIG. 19 is a diagram for explaining an application example to the route guidance in the second embodiment. FIG. 20 is a diagram for explaining an application example to usage log accumulation and analysis in the second embodiment. FIG. 21 is a diagram illustrating an example of application of the transmitter and the receiver in the second embodiment. FIG. 22 is a diagram illustrating an example of application of a transmitter and a receiver in Embodiment 2. FIG. 23 is a diagram illustrating an example of an application according to the third embodiment. FIG. 24 is a diagram illustrating an example of an application according to the third embodiment. FIG. 25 is a diagram illustrating an example of a transmission signal and an example of a voice synchronization method in the third embodiment. FIG. 26 is a diagram illustrating an example of a transmission signal in the third embodiment. FIG. 27 is a diagram illustrating an example of a process flow of the receiver in Embodiment 3. FIG. 28 is a diagram illustrating an example of a user interface of the receiver in the third embodiment. FIG. 29 is a diagram illustrating an example of a process flow of the receiver in Embodiment 3. FIG. 30 is a diagram illustrating another example of the processing flow of the receiver in Embodiment 3. FIG. 31A is a diagram for explaining a specific method of synchronized playback in the third embodiment. FIG. 31B is a block diagram illustrating a configuration of a playback device (receiver) that performs synchronized playback in the third embodiment. FIG. 31C is a flowchart illustrating a processing operation of a playback device (receiver) that performs synchronized playback in the third embodiment. FIG. 32 is a diagram for explaining preparations for synchronized playback in the third embodiment. FIG. 33 is a diagram illustrating an example of application of a receiver in Embodiment 3. FIG. 34A is a front view of a receiver held by a holder in the third embodiment. FIG. 34B is a rear view of the receiver held by the holder in the third embodiment. FIG. 35 is a diagram for describing a use case of a receiver held by a holder in the third embodiment. FIG. 36 is a flowchart showing the processing operation of the receiver held by the holder in the third embodiment. FIG. 37 is a diagram illustrating an example of an image displayed by the receiver in Embodiment 3. FIG. 38 is a diagram showing another example of the holder in the third embodiment. FIG. 39A is a diagram illustrating an example of a visible light signal in Embodiment 3. FIG. 39B is a diagram illustrating an example of a visible light signal in Embodiment 3. FIG. 39C is a diagram illustrating an example of a visible light signal in Embodiment 3. FIG. 39D is a diagram illustrating an example of a visible light signal in Embodiment 3. FIG. 40 is a diagram illustrating a configuration of a visible light signal in the third embodiment. FIG. 41 is a diagram illustrating an example in which the receiver in Embodiment 4 displays an AR image. FIG. 42 is a diagram illustrating an example of a display system in Embodiment 4. FIG. 43 is a diagram illustrating another example of the display system according to Embodiment 4. FIG. 44 is a diagram illustrating another example of the display system according to Embodiment 4. FIG. 45 is a flowchart illustrating an example of processing operations of a receiver in Embodiment 4. FIG. 46 is a diagram illustrating another example in which the receiver in Embodiment 4 displays an AR image. FIG. 47 is a diagram illustrating another example in which the receiver in Embodiment 4 displays an AR image. FIG. 48 is a diagram illustrating another example in which the receiver in Embodiment 4 displays an AR image. FIG. 49 is a diagram illustrating another example in which the receiver in Embodiment 4 displays an AR image. FIG. 50 is a diagram illustrating another example in which the receiver in Embodiment 4 displays an AR image. FIG. 51 is a diagram illustrating another example in which the receiver in Embodiment 4 displays an AR image. FIG. 52 is a flowchart illustrating another example of the processing operation of the receiver in the fourth embodiment. FIG. 53 is a diagram illustrating another example in which the receiver in Embodiment 4 displays an AR image. FIG. 54 is a diagram illustrating a captured display image Ppre and a decoding image Pdec acquired by capturing by the receiver in the fourth embodiment. FIG. 55 is a diagram illustrating an example of a captured display image Ppre displayed on the receiver in the fourth embodiment. FIG. 56 is a flowchart illustrating another example of processing operations of a receiver in Embodiment 4. FIG. 57 is a diagram illustrating another example in which the receiver in Embodiment 4 displays an AR image. FIG. 58 is a diagram illustrating another example in which the receiver in Embodiment 4 displays an AR image. FIG. 59 is a diagram illustrating another example in which the receiver in Embodiment 4 displays an AR image. FIG. 60 is a diagram illustrating another example in which the receiver in Embodiment 4 displays an AR image. FIG. 61 is a diagram showing an example of recognition information in the fourth embodiment. FIG. 62 is a flowchart illustrating another example of processing operations of a receiver in Embodiment 4. FIG. 63 is a diagram illustrating an example in which the receiver in Embodiment 4 identifies bright line pattern regions. FIG. 64 is a diagram illustrating another example of the receiver in Embodiment 4. FIG. 65 is a flowchart illustrating another example of the processing operation of the receiver in the fourth embodiment. FIG. 66 is a diagram illustrating an example of a transmission system including a plurality of transmitters in Embodiment 4. FIG. 67 is a diagram illustrating an example of a transmission system including a plurality of transmitters and receivers in Embodiment 4. 68A is a flowchart illustrating an example of processing operations of a receiver in Embodiment 4. FIG. FIG. 68B is a flowchart illustrating an example of processing operations of a receiver in Embodiment 4. FIG. 69A is a flowchart illustrating a display method according to Embodiment 4. FIG. 69B is a block diagram illustrating a structure of a display device in Embodiment 4. FIG. 70 is a diagram illustrating an example in which the receiver in the first modification of the fourth embodiment displays an AR image. FIG. 71 is a diagram illustrating another example in which the receiver in the first modification of the fourth embodiment displays an AR image. FIG. 72 is a diagram illustrating another example in which the receiver in the first modification of the fourth embodiment displays an AR image. FIG. 73 is a diagram illustrating another example in which the receiver in the first modification of the fourth embodiment displays an AR image. 74 is a diagram illustrating another example of a receiver in Modification 1 of Embodiment 4. FIG. FIG. 75 is a diagram illustrating another example in which the receiver in the first modification of the fourth embodiment displays an AR image. FIG. 76 is a diagram illustrating another example in which the receiver in the first modification of the fourth embodiment displays an AR image. FIG. 77 is a flowchart illustrating an example of processing operations of the receiver in the first modification of the fourth embodiment. FIG. 78 is a diagram illustrating an example of a problem when an AR image assumed in the receiver in Embodiment 4 or the modification 1 thereof is displayed. FIG. 79 is a diagram illustrating an example in which the receiver in the second modification of the fourth embodiment displays an AR image. FIG. 80 is a flowchart illustrating an example of processing operations of the receiver in the second modification of the fourth embodiment. FIG. 81 is a diagram illustrating another example in which the receiver in the second modification of the fourth embodiment displays an AR image. FIG. 82 is a flowchart illustrating another example of the processing operation of the receiver in the second modification of the fourth embodiment. FIG. 83 is a diagram illustrating another example in which the receiver in the second modification of the fourth embodiment displays an AR image. FIG. 84 is a diagram illustrating another example in which the receiver in the second modification of the fourth embodiment displays an AR image. FIG. 85 is a diagram illustrating another example in which the receiver in the second modification of the fourth embodiment displays an AR image. FIG. 86 is a diagram illustrating another example in which the receiver in the second modification of the fourth embodiment displays an AR image. FIG. 87A is a flowchart illustrating a display method according to one embodiment of the present invention. FIG. 87B is a block diagram illustrating a structure of a display device according to one embodiment of the present invention. FIG. 88 is a diagram illustrating an example of enlargement and movement of an AR image in the third modification of the fourth embodiment. FIG. 89 is a diagram illustrating an example of expansion of an AR image in the third modification of the fourth embodiment. FIG. 90 is a flowchart illustrating an example of a processing operation related to enlargement and movement of an AR image by a receiver according to the third modification of the fourth embodiment. FIG. 91 is a diagram illustrating an example of superimposition of AR images in the third modification of the fourth embodiment. FIG. 92 is a diagram illustrating an example of superimposition of AR images in the third modification of the fourth embodiment. FIG. 93 is a diagram illustrating an example of superimposition of AR images in the third modification of the fourth embodiment. FIG. 94 is a diagram illustrating an example of superimposition of AR images in the third modification of the fourth embodiment. FIG. 95A is a diagram illustrating an example of a captured display image obtained by imaging by the receiver in the third modification of the fourth embodiment. FIG. 95B is a diagram illustrating an example of a menu screen displayed on the display of the receiver in the third modification of the fourth embodiment. FIG. 96 is a flowchart illustrating an example of processing operations of the receiver and the server in the third modification of the fourth embodiment. FIG. 97 is a diagram for explaining sound volume reproduced by the receiver in the third modification of the fourth embodiment. FIG. 98 is a diagram illustrating a relationship between the distance from the receiver to the transmitter and the sound volume in the third modification of the fourth embodiment. FIG. 99 is a diagram illustrating an example of superimposition of AR images by the receiver in the third modification of the fourth embodiment. FIG. 100 is a diagram illustrating an example of superimposition of AR images by a receiver in the third modification of the fourth embodiment. FIG. 101 is a diagram for describing an example of how to obtain a line scan time by a receiver in the third modification of the fourth embodiment. FIG. 102 is a diagram for describing an example of how to obtain the line scan time by the receiver in the third modification of the fourth embodiment. FIG. 103 is a flowchart illustrating an example of how to obtain the line scan time by the receiver in the third modification of the fourth embodiment. FIG. 104 is a diagram illustrating an example of superimposition of AR images by the receiver in the third modification of the fourth embodiment. FIG. 105 is a diagram illustrating an example of superimposition of AR images by the receiver in the third modification of the fourth embodiment. FIG. 106 is a diagram illustrating an example of superimposition of AR images by the receiver in the third modification of the fourth embodiment. FIG. 107 is a diagram illustrating an example of a decoding image acquired in accordance with the attitude of the receiver in the third modification of the fourth embodiment. FIG. 108 is a diagram illustrating another example of the decoding image acquired in accordance with the attitude of the receiver in the third modification of the fourth embodiment. FIG. 109 is a flowchart illustrating an example of processing operations of the receiver in Modification 3 of Embodiment 4. FIG. 110 is a diagram illustrating an example of a camera lens switching process performed by a receiver according to the third modification of the fourth embodiment. FIG. 111 is a diagram illustrating an example of camera switching processing by the receiver in the third modification of the fourth embodiment. FIG. 112 is a flowchart illustrating an example of processing operations of the receiver and the server in the third modification of the fourth embodiment. FIG. 113 is a diagram illustrating an example of superimposition of AR images by the receiver in the third modification of the fourth embodiment. FIG. 114 is a sequence diagram illustrating processing operations of a system including a receiver, a microwave oven, a relay server, and an electronic settlement server in Modification 3 of Embodiment 4. FIG. 115 is a sequence diagram illustrating processing operations of a system including a POS terminal, a server, a receiver, and a microwave oven according to the third modification of the fourth embodiment. 116 is a diagram illustrating an example of indoor use in Modification 3 of Embodiment 4. FIG. FIG. 117 is a diagram illustrating an example of an augmented reality object display according to the third modification of the fourth embodiment. FIG. 118 is a diagram illustrating a configuration of a display system according to the fourth modification of the fourth embodiment. FIG. 119 is a flowchart illustrating the processing operation of the display system in the fourth modification of the fourth embodiment. FIG. 120 is a flowchart illustrating a recognition method according to an aspect of the present invention. FIG. 121 is a diagram illustrating an example of an operation mode of a visible light signal according to Embodiment 5. 122A is a flowchart illustrating a visible light signal generation method according to Embodiment 5. FIG. FIG. 122B is a block diagram illustrating a configuration of the signal generation device according to Embodiment 5. FIG. 123 is a diagram illustrating a format of an MPM MAC frame according to the sixth embodiment. FIG. 124 is a flowchart illustrating a processing operation of the encoding device that generates an MPM MAC frame according to the sixth embodiment. FIG. 125 is a flowchart showing a processing operation of the decoding apparatus for decoding the MPM MAC frame in the sixth embodiment. FIG. 126 shows MAC PIB attributes in the sixth embodiment. FIG. 127 is a diagram for explaining an MPM light control method according to the sixth embodiment. FIG. 128 is a diagram showing attributes of the PHY PIB in the sixth embodiment. FIG. 129 is a diagram for explaining MPM in the sixth embodiment. FIG. 130 is a diagram illustrating a PLCP header subfield according to the sixth embodiment. FIG. 131 is a diagram illustrating a PLCP center subfield according to the sixth embodiment. FIG. 132 is a diagram illustrating a PLCP footer subfield according to the sixth embodiment. FIG. 133 is a diagram illustrating a waveform of the PHY PWM mode in the MPM according to the sixth embodiment. FIG. 134 is a diagram illustrating a PHY PPM mode waveform in the MPM according to the sixth embodiment. FIG. 135 is a flowchart illustrating an example of the decoding method according to the sixth embodiment. FIG. 136 is a flowchart illustrating an example of the encoding method according to the sixth embodiment. FIG. 137 is a diagram illustrating an example in which the receiver in Embodiment 7 displays an AR image. FIG. 138 is a diagram illustrating an example of a captured display image on which an AR image is superimposed, according to the seventh embodiment. FIG. 139 is a diagram illustrating another example in which the receiver in Embodiment 7 displays an AR image. FIG. 140 is a flowchart illustrating the operation of the receiver in the seventh embodiment. 141 is a diagram for explaining operation of a transmitter in Embodiment 7. FIG. 142 is a diagram for explaining another operation of the transmitter in Embodiment 7. FIG. FIG. 143 is a diagram for describing another operation of the transmitter in the seventh embodiment. FIG. 144 is a diagram illustrating a comparative example for describing easiness of receiving an optical ID in the seventh embodiment. FIG. 145A is a flowchart illustrating an operation of the transmitter in the seventh embodiment. FIG. 145B is a block diagram illustrating a configuration of a transmitter in Embodiment 7. FIG. 146 is a diagram illustrating another example in which the receiver in Embodiment 7 displays an AR image. FIG. 147 is a diagram for explaining an operation of the transmitter in the eighth embodiment. FIG. 148A is a flowchart illustrating a transmission method according to the eighth embodiment. 148B is a block diagram illustrating a configuration of a transmitter in Embodiment 8. FIG. FIG. 149 is a diagram illustrating an example of a detailed configuration of a visible light signal in Embodiment 8. FIG. 150 is a diagram illustrating another example of a detailed configuration of a visible light signal according to Embodiment 8. FIG. 151 is a diagram illustrating another example of a detailed configuration of a visible light signal according to the eighth embodiment. FIG. 152 is a diagram illustrating another example of a detailed configuration of a visible light signal according to the eighth embodiment. FIG. 153 is a diagram illustrating a relationship between the sum of the variables y ₀ to y ₃ , the total time length, and the effective time length in the eighth embodiment. FIG. 154A is a flowchart illustrating a transmission method according to Embodiment 8. FIG. 154B is a block diagram illustrating a configuration of a transmitter in Embodiment 8. FIG. 155 is a diagram illustrating the structure of the display system according to the ninth embodiment. FIG. 156 is a sequence diagram illustrating processing operations of the receiver and the server according to Embodiment 9. FIG. 157 is a flowchart showing the processing operation of the server in the ninth embodiment. FIG. 158 is a diagram illustrating an example of communication in the case where the transmitter and the receiver in Embodiment 9 are mounted on a vehicle, respectively. FIG. 159 is a flowchart showing the processing operation of the vehicle in the ninth embodiment. FIG. 160 is a diagram illustrating an example in which the receiver in Embodiment 9 displays an AR image. FIG. 161 is a diagram illustrating another example in which the receiver in Embodiment 9 displays an AR image. FIG. 162 is a diagram illustrating processing operation of a receiver in Embodiment 9. FIG. 163 is a diagram illustrating an example of operation on a receiver in Embodiment 9. FIG. 164 is a diagram illustrating an example of AR image displayed on the receiver in Embodiment 9. FIG. 165 is a diagram illustrating an example of the AR image superimposed on the captured display image in the ninth embodiment. FIG. 166 is a diagram illustrating an example of the AR image superimposed on the captured display image in Embodiment 9. 167 is a diagram illustrating an example of a transmitter in Embodiment 9. FIG. 168 is a diagram illustrating another example of a transmitter in Embodiment 9. FIG. 169 is a diagram illustrating another example of a transmitter in Embodiment 9. FIG. FIG. 170 is a diagram illustrating an example of a system using a receiver compatible with optical communication and a receiver not compatible with optical communication in Embodiment 9. FIG. 171 is a flowchart illustrating processing operations of the receiver in Embodiment 9. FIG. 172 is a diagram illustrating an example of display of an AR image in Embodiment 9. FIG. 173A is a flowchart illustrating a display method according to one embodiment of the present invention. FIG. 173B is a block diagram illustrating a structure of a display device according to one embodiment of the present invention. 174 is a diagram illustrating an example of an image drawn on a transmitter in Embodiment 10. [FIG. 175 is a diagram illustrating another example of an image drawn on a transmitter in Embodiment 10. FIG. 176 is a diagram illustrating an example of a transmitter and a receiver in Embodiment 10. FIG. FIG. 177 is a diagram for describing the fundamental frequency of a line pattern in the tenth embodiment. FIG. 178A is a flowchart showing a processing operation of the encoding apparatus according to the tenth embodiment. FIG. 178B is a diagram for describing a processing operation of the encoding device according to the tenth embodiment. FIG. 179 is a flowchart illustrating processing operations of a receiver which is a decoding device according to Embodiment 10. FIG. 180 is a flowchart illustrating processing operations of a receiver in Embodiment 10. FIG. 181A is a diagram illustrating an example of a system configuration in Embodiment 10. FIG. 181B is a diagram illustrating processing of the camera according to Embodiment 10. FIG. 182 is a diagram illustrating another example of the configuration of the system according to the tenth embodiment. FIG. 183 is a diagram illustrating another example of an image drawn on the transmitter in Embodiment 10. FIG. 184 is a diagram illustrating an example of a format of a MAC frame constituting the frame ID in the tenth embodiment. FIG. 185 is a diagram illustrating an example of a MAC header configuration in the tenth embodiment. FIG. 186 is a diagram illustrating an example of a table for deriving the number of packet divisions according to the tenth embodiment. FIG. 187 is a diagram illustrating PHY coding according to the tenth embodiment. FIG. 188 is a diagram illustrating an example of a transmission image Im3 having a PHY symbol in Embodiment 10. FIG. 189 is a diagram for explaining two PHY versions in the tenth embodiment. FIG. 190 is a diagram for explaining the Gray code in the tenth embodiment. FIG. 191 is a diagram illustrating an example of decoding processing by the receiver in Embodiment 10. FIG. 192 is a diagram for describing a transmission image fraud detection method by the receiver in the tenth embodiment. FIG. 193 is a flowchart illustrating an example of a decoding process including fraud detection of a transmission image by a receiver in the tenth embodiment. FIG. 194A is a flowchart showing a display method according to a modification of the tenth embodiment. FIG. 194B is a block diagram illustrating a structure of the display device according to the modification of the tenth embodiment. FIG. 194C is a flowchart illustrating a communication method according to a modification of the tenth embodiment. FIG. 194D is a block diagram showing a configuration of a communication apparatus according to this variation of the tenth embodiment. FIG. 194E is a block diagram illustrating a configuration of the transmitter according to Embodiment 10 and its modifications. FIG. 195 is a diagram illustrating an example of a configuration of a communication system including a server in the eleventh embodiment. FIG. 196 is a flowchart illustrating a management method by the first server in the eleventh embodiment. FIG. 197 is a diagram illustrating an illumination system in Embodiment 12. FIG. 198 is a diagram illustrating an example of arrangement of illumination devices and a decoding image in Embodiment 12. FIG. 199 is a diagram illustrating another example of arrangement of illumination devices and a decoding image in Embodiment 12. FIG. 200 is a diagram for describing position estimation using the illumination device in Embodiment 12. FIG. 201 is a flowchart illustrating processing operation of a receiver in Embodiment 12. FIG. 202 is a diagram illustrating an example of a communication system in Embodiment 12. FIG. 203 is a diagram for describing self-position estimation processing by a receiver in Embodiment 12. FIG. 204 is a flowchart illustrating self-position estimation processing by a receiver in Embodiment 12. FIG. 205 is a flowchart illustrating an outline of receiver self-position estimation processing according to the twelfth embodiment. FIG. 206 is a diagram illustrating the relationship between radio wave IDs and optical IDs in the twelfth embodiment. 207 is a diagram for describing an example of imaging by a receiver in Embodiment 12. FIG. FIG. 208 is a diagram for describing another example of imaging by a receiver in Embodiment 12. FIG. 209 is a diagram for describing a camera used by a receiver in Embodiment 12. FIG. 210 is a flowchart illustrating an example of processing in which the receiver in Embodiment 12 changes the visible light signal of the transmitter. FIG. 211 is a flowchart illustrating another example of processing in which the receiver in Embodiment 12 changes the visible light signal of the transmitter. FIG. 212 is a diagram for describing navigation by a receiver in Embodiment 13. FIG. 213 is a flowchart illustrating an example of self-position estimation by the receiver in Embodiment 13. FIG. 214 is a diagram for describing a visible light signal received by the receiver in Embodiment 13. FIG. 215 is a flowchart illustrating another example of self-position estimation by the receiver in the thirteenth embodiment. FIG. 216 is a flowchart illustrating an example of determination of reflected light by the receiver in Embodiment 13. FIG. 217 is a flowchart illustrating an example of navigation by a receiver in Embodiment 13. FIG. 218 is a diagram illustrating an example of the transmitter 100 configured as a projector in Embodiment 13. In FIG. FIG. 219 is a flowchart illustrating another example of self-position estimation by the receiver in Embodiment 13. FIG. 220 is a flowchart illustrating an example of processing performed by a transmitter in the thirteenth embodiment. FIG. 221 is a flowchart illustrating another example of navigation by a receiver in Embodiment 13. FIG. 222 is a flowchart illustrating an example of processing by a receiver in Embodiment 13. 223 is a diagram illustrating an example of a screen which is displayed on the display of the receiver in Embodiment 13. FIG. 224 is a diagram illustrating an example of character display by the receiver in Embodiment 13. FIG. 225 is a diagram illustrating another example of a screen displayed on the display of the receiver in Embodiment 13. FIG. FIG. 226 is a diagram showing a system configuration for performing navigation to a meeting place in the thirteenth embodiment. 227 is a diagram illustrating another example of a screen displayed on the display of the receiver in Embodiment 13. FIG. FIG. 228 is a diagram showing the inside of the concert hall. FIG. 229 is a flowchart illustrating an example of a communication method according to the first aspect of the present invention.

A communication method according to one embodiment of the present invention is a communication method using a terminal including an image sensor, and determines whether or not the terminal can perform visible light communication, and the terminal performs visible light communication. When the image sensor is determined to be able to perform the process, the image sensor captures a subject whose luminance changes to obtain a decoding image, and the subject transmits from the striped pattern appearing in the decoding image. In the determination of visible light communication, when it is determined that the terminal cannot perform visible light communication, a captured image is acquired by capturing the subject with the image sensor. Then, by performing edge detection of the captured image, at least one contour is extracted, a predetermined specific area is specified from the at least one outline, and a line of the specific area is identified. The object acquires the second identification information transmitted from the pattern.

Thus, for example, as in the tenth embodiment, a terminal such as a receiver can receive first identification information or second identification information from a subject such as a transmitter regardless of whether or not visible light communication is possible. Can be acquired. That is, when the terminal can perform visible light communication, the terminal acquires, for example, the light ID from the subject as the first identification information. On the other hand, even if the terminal cannot perform visible light communication, the terminal can acquire, for example, an image ID or a frame ID from the subject as the second identification information. Specifically, for example, the transmission image illustrated in FIGS. 183 and 188 is captured as a subject, the area of the transmission image is selected as a specific area (that is, a selection area), and second identification information is obtained from the line pattern of the transmission image. Is acquired. Therefore, even when visible light communication is impossible, the second identification information can be appropriately acquired. The striped pattern is also called a bright line pattern or a bright line pattern region.

In the specification of the specific area, an area having a quadrangular outline of a predetermined size or an area having a rounded quadrangular outline of a predetermined size or more may be specified as the specific area.

Thus, for example, as shown in FIG. 179, a quadrangular or rounded quadrangular region can be appropriately identified as the specific region.

Further, in the determination of the visible light communication, when the terminal is identified as a terminal that can change the exposure time to a predetermined value or less, it is determined that visible light communication can be performed, When the terminal specifies that the exposure time cannot be changed to the predetermined value or less, it may be determined that the visible light communication cannot be performed.

Thereby, for example, as shown in FIG. 180, it is possible to appropriately determine whether or not a visible light signal can be performed.

Further, in the determination of the visible light communication, when the terminal determines that the visible light communication can be performed, when imaging the subject, the exposure time of the image sensor is set to a first exposure time, By capturing the subject with the first exposure time to obtain the decoding image, and in the determination of the visible light communication, when it is determined that the terminal cannot perform visible light communication, When the subject is imaged, the exposure time of the image sensor is set to a second exposure time, the captured image is acquired by capturing the subject with the second exposure time, and the first exposure time is set. May be shorter than the second exposure time.

Thereby, it is possible to acquire a decoding image having a striped pattern by imaging at the first exposure time, and appropriately acquire the first identification information by decoding the striped pattern. Furthermore, a normal captured image can be acquired as a captured image by imaging at the second exposure time, and the second identification information can be appropriately acquired from the line pattern appearing in the normal captured image. Accordingly, the terminal can acquire the first identification information or the second identification information suitable for the terminal by properly using the first exposure time and the second exposure time.

The subject has a rectangular shape as viewed from the image sensor, and the central area of the subject changes in luminance, whereby the first identification information is transmitted, and a barcode-like line pattern is formed around the subject. When the subject is imaged when it is determined that the terminal can perform visible light communication in the determination of the visible light communication, the image sensor corresponds to a plurality of exposure lines of the image sensor. The decoding image including a bright line pattern composed of a plurality of bright lines is acquired, the first identification information is acquired by decoding the bright line pattern, and in the determination of the visible light communication, the terminal When it is determined that communication is not possible, when the subject is imaged, the second pattern is extracted from the line pattern of the captured image. It may acquire the different information.

Thus, the first identification information and the second identification information can be appropriately acquired from the subject whose central region changes in luminance.

Further, the first identification information obtained from the decoding image and the second identification information obtained from the line pattern may be the same information.

Thus, the same information can be acquired from the subject both in a terminal capable of visible light communication and a terminal incapable of visible light communication.

Further, in the determination of the visible light communication, when it is determined that the terminal can perform visible light communication, the first moving image associated with the first identification information is displayed, and the first When the operation of sliding the moving image is received, the second moving image associated with the first identification information may be displayed next to the first moving image.

For example, each of the first moving image and the second moving image is the first AR image P46 and the second AR image P46c shown in FIG. Further, the first identification information is, for example, an optical ID as described above. In the communication method according to the above aspect, when an operation of sliding the first moving image, that is, a swipe is accepted, the second moving image associated with the first identification information is next to the first moving image. Is displayed. Therefore, an image useful for the user can be easily displayed. Further, as shown in FIG. 194A, since it is determined in advance whether or not visible light communication is possible, it is possible to omit useless processing to acquire a visible light signal until impossible, The processing burden can be reduced.

In the display of the second moving image, when an operation of sliding the first moving image in the horizontal direction is accepted, the second moving image is displayed, and the first moving image is slid in the vertical direction. When the operation to be performed is received, a still image associated with the first identification information may be displayed.

Thereby, for example, as shown in FIG. 162, the second moving image is displayed by sliding the first moving image in the horizontal direction, that is, by swiping. Further, for example, as illustrated in FIGS. 163 and 164, a still image associated with the first identification information is displayed by sliding the first moving image in the vertical direction. The still image is, for example, an AR image P47 shown in FIG. Therefore, it is possible to easily display a wide variety of images useful to the user.

Further, in each of the first moving image and the second moving image, the object in the picture displayed first may be in the same position.

Thus, for example, as shown in FIG. 162, when the second moving image is displayed instead of the first moving image, the first displayed object is at the same position, so the user can It is possible to easily grasp that the moving image and the second moving image are related to each other.

Further, when the first identification information is acquired again by imaging by the image sensor, the next moving image associated with the first identification information may be displayed next to the displayed moving image. .

Thus, as shown in FIG. 162, for example, even when an operation such as slide or swipe is not performed, the next moving image is displayed when the light ID that is the first identification information is taken again. Therefore, a moving image useful for the user can be displayed more easily.

Further, in each of the displayed moving image and the next moving image, the object in the picture displayed first may be in the same position.

Thus, for example, as shown in FIG. 162, when the next moving image is displayed instead of the displayed moving image, the first displayed object is at the same position, so the user can It is possible to easily grasp that moving images are related to each other.

Further, at least one of the first moving image and the second moving image has a higher transparency at the position as the position in the moving image is closer to the end of the moving image. It may be formed.

Thus, for example, as shown in FIG. 93 or FIG. 166, when the moving image is displayed superimposed on the normal captured image, an object with an ambiguous outline actually exists in the environment indicated by the normal captured image. As such, the captured display image can be displayed.

Further, an image may be displayed outside an area where at least one of the first moving image and the second moving image is displayed.

Thereby, for example, as the sub image Ps46 shown in FIG. 161, since the image is displayed outside the area where the moving image is displayed, a variety of images more useful to the user can be easily displayed.

In addition, the image sensor is a region that includes a pattern of a plurality of bright lines by acquiring a normal captured image by imaging with a first exposure time by the image sensor and imaging by a second exposure time shorter than the first exposure time. The decoding image including a bright line pattern region is acquired, the first identification information is acquired by decoding the decoding image, and at least one moving image of the first moving image or the second moving image In the image display, a reference area at the same position as the bright line pattern area in the decoding image is specified from the normal captured image, and the moving image is superimposed on the normal captured image based on the reference area. May be recognized as a target area, and the moving image may be superimposed on the target area. For example, in displaying the moving image of at least one of the first moving image and the second moving image, the upper, lower, left, or right region of the reference region in the normal captured image is the target region. You may recognize as.

As a result, for example, as shown in FIGS. 50 to 52 and 172, the target area is recognized based on the reference area, and the moving image is superimposed on the target area. Can be easily increased.

In the display of at least one of the first moving image and the second moving image, the size of the moving image may be increased as the size of the bright line pattern region is increased.

As a result, as shown in FIG. 172, the size of the moving image changes according to the size of the bright line pattern region, so that the object indicated by the moving image is more compared to the case where the size of the moving image is fixed. The moving image can be displayed so that it exists in reality.

A transmitter according to one embodiment of the present invention includes an illumination plate, a light source that emits light from a back side of the illumination plate, and a microcontroller that changes luminance of the light source, and the microcontroller includes the light source. The first identification information is transmitted from the light source through the illuminating plate, and a barcode-like line pattern is arranged around the front side of the illuminating plate. Second identification information is encoded, and the first identification information and the second identification information are the same information. For example, the lighting plate has a rectangular shape.

Thereby, the same information can be transmitted to a terminal capable of performing visible light communication and a terminal capable of performing visible light communication.

Note that these comprehensive or specific modes may be realized by a recording medium such as an apparatus, a system, a method, an integrated circuit, a computer program, or a computer-readable CD-ROM. You may implement | achieve in arbitrary combinations of a circuit, a computer program, or a recording medium.

Hereinafter, embodiments will be specifically described with reference to the drawings.

It should be noted that each of the embodiments described below shows a comprehensive or specific example. The numerical values, shapes, materials, constituent elements, arrangement positions and connecting forms of the constituent elements, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present invention. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements.

(Embodiment 1)
The first embodiment will be described below.

(Observation of luminance of light emitting part)
We propose an imaging method that starts and ends the exposure at different times for each image sensor, rather than exposing all the image sensors at the same timing when capturing one image. FIG. 1 shows an example in which imaging devices arranged in one row are exposed simultaneously, and imaging is performed by shifting the exposure start time in the order of closer rows. Here, the exposure line of the image sensor that exposes simultaneously is called an exposure line, and the pixel line on the image corresponding to the image sensor is called a bright line.

When this image capturing method is used to capture an image of a blinking light source on the entire surface of the image sensor, bright lines (light and dark lines of pixel values) along the exposure line appear on the captured image as shown in FIG. . By recognizing the bright line pattern, it is possible to estimate the light source luminance change at a speed exceeding the imaging frame rate. Thereby, by transmitting a signal as a change in light source luminance, communication at a speed higher than the imaging frame rate can be performed. When a signal is expressed by the light source taking two types of luminance values, the lower luminance value is called low (LO), and the higher luminance value is called high (HI). Low may be in a state where the light source is not shining, or may be shining weaker than high.

∙ By this method, information is transmitted at a speed exceeding the imaging frame rate.

When there are 20 exposure lines where the exposure times do not overlap in one captured image, and the imaging frame rate is 30 fps, a change in luminance with a period of 1.67 milliseconds can be recognized. When there are 1000 exposure lines whose exposure times do not overlap, it is possible to recognize a luminance change with a period of 1 / 30,000 second (about 33 microseconds). The exposure time is set shorter than 10 milliseconds, for example.

FIG. 2 shows a case where the exposure of the next exposure line is started after the exposure of one exposure line is completed.

In this case, when the number of frames per second (frame rate) is f and the number of exposure lines constituting one image is 1, if information is transmitted depending on whether or not each exposure line receives a certain amount of light, the maximum Can transmit information at a rate of fl bits per second.

It should be noted that communication is possible at higher speeds when exposure is performed with a time difference for each pixel, not for each line.

At this time, when the number of pixels per exposure line is m pixels and information is transmitted depending on whether each pixel receives light above a certain level, the transmission speed is a maximum of flm bits per second.

As shown in FIG. 3, if the exposure state of each exposure line by the light emission of the light emitting unit can be recognized at a plurality of levels, the light emission time of the light emitting unit is controlled by a unit time shorter than the exposure time of each exposure line. More information can be transmitted.

If the exposure state can be recognized in the Elv stage, information can be transmitted at a maximum rate of flElv bits per second.

Also, the basic period of transmission can be recognized by causing the light emitting unit to emit light at a timing slightly different from the exposure timing of each exposure line.

FIG. 4 shows a case where the exposure of the next exposure line is started before the exposure of one exposure line is completed. That is, the exposure times of adjacent exposure lines are partially overlapped in time. With such a configuration, (1) it is possible to increase the number of samples within a predetermined time as compared with the case where the exposure of the next exposure line is started after waiting for the end of the exposure time of one exposure line. By increasing the number of samples in a predetermined time, it becomes possible to more appropriately detect the optical signal generated by the optical transmitter that is the subject. That is, it is possible to reduce the error rate when detecting an optical signal. Further, (2) the exposure time of each exposure line can be made longer than when the exposure time of the next exposure line is started after waiting for the end of the exposure time of one exposure line. However, a brighter image can be acquired. That is, the S / N ratio can be improved. In all exposure lines, it is not necessary that the exposure time of adjacent exposure lines has a partial overlap in time, and a configuration in which some exposure lines have no partial overlap. It is also possible. By configuring a part of the exposure lines so as not to partially overlap in time, it is possible to suppress the generation of intermediate colors due to the overlap of exposure times on the imaging screen, and to detect bright lines more appropriately. .

In this case, the exposure time is calculated from the brightness of each exposure line, and the light emission state of the light emitting unit is recognized.

When the brightness of each exposure line is determined by a binary value indicating whether the luminance is equal to or higher than a threshold value, in order to recognize the state where no light is emitted, the state where the light emitting unit does not emit light is indicated for each line. It must last longer than the exposure time.

FIG. 5A shows the influence of the difference in exposure time when the exposure start times of the exposure lines are equal. 7500a is the case where the exposure end time of the previous exposure line is equal to the exposure start time of the next exposure line, and 7500b is the case where the exposure time is longer than that. As in the case of 7500b, the exposure time of adjacent exposure lines is partially overlapped in time, so that the exposure time can be increased. That is, the light incident on the image sensor increases and a bright image can be obtained. In addition, since the imaging sensitivity for capturing images with the same brightness can be suppressed to a low level, an image with less noise can be obtained, so that communication errors are suppressed.

FIG. 5B shows the influence of the difference in the exposure start time of each exposure line when the exposure times are equal. 7501a is the case where the exposure end time of the previous exposure line is equal to the exposure start time of the next exposure line, and 7501b is the case where the exposure of the next exposure line is started earlier than the end of exposure of the previous exposure line. By adopting a configuration in which the exposure times of adjacent exposure lines partially overlap in time as in 7501b, the number of lines that can be exposed per time can be increased. Thereby, the resolution becomes higher and a large amount of information can be obtained. Since the sample interval (= difference in exposure start time) becomes dense, the change in the light source luminance can be estimated more accurately, the error rate can be reduced, and the change in the light source luminance in a shorter time is recognized. be able to. By making the exposure time overlap, it is possible to recognize blinking of the light source that is shorter than the exposure time by using the difference in exposure amount between adjacent exposure lines.

Further, when the number of samples is small, that is, when the sample interval (time difference t _D shown in FIG. 5B) is long, there is a high possibility that a change in light source luminance cannot be detected accurately. In this case, the possibility can be suppressed by shortening the exposure time. That is, it is possible to accurately detect a change in light source luminance. Further, it is desirable that the exposure time satisfy the exposure time> (sample interval−pulse width). The pulse width is a pulse width of light that is a period during which the luminance of the light source is High. Thereby, the High brightness can be detected appropriately.

As described with reference to FIGS. 5A and 5B, in the configuration in which each exposure line is sequentially exposed so that the exposure times of adjacent exposure lines partially overlap in time, the exposure time is set to be longer than that in the normal shooting mode. By using the bright line pattern generated by setting it short for signal transmission, the communication speed can be dramatically improved. Here, it is possible to generate an appropriate bright line pattern by setting the exposure time during visible light communication to 1/480 seconds or less. Here, when the frame frequency = f, the exposure time needs to be set as exposure time <1/8 × f. Blanking that occurs during shooting is at most half the size of one frame. That is, since the blanking time is less than half of the shooting time, the actual shooting time is 1 / 2f at the shortest time. Furthermore, since it is necessary to receive quaternary information within a time of 1 / 2f, at least the exposure time needs to be shorter than 1 / (2f × 4). Since the normal frame rate is 60 frames / second or less, it is possible to generate an appropriate bright line pattern in the image data and perform high-speed signal transmission by setting the exposure time to 1/480 seconds or less. Become.

FIG. 5C shows an advantage when the exposure times are short when the exposure times of the exposure lines do not overlap. When the exposure time is long, even if the light source has a binary luminance change as in 7502a, the captured image has an intermediate color portion as in 7502e, and it becomes difficult to recognize the luminance change of the light source. There is. However, as the 7502D, after completion exposure of one exposure line, by a configuration in which the free time (predetermined waiting time) t _D2 not predetermined exposure start exposure of the next exposure line, the luminance variation of the light source Can be easily recognized. That is, a more appropriate bright line pattern such as 7502f can be detected. As in 7502D, be provided with a free time without predetermined exposure becomes an exposure time t _E can be realized to be smaller than the time difference t _D of the exposure start time of each exposure line. When the normal shooting mode has a configuration in which the exposure times of adjacent exposure lines partially overlap in time, the exposure time is set shorter than the normal shooting mode until a predetermined idle time occurs. This can be realized. Further, even when the normal photographing mode is the case where the exposure end time of the previous exposure line and the exposure start time of the next exposure line are equal, by setting the exposure time short until a predetermined non-exposure time occurs, Can be realized. Further, as 7502G, also by increasing the distance t _D of the exposure start time of each exposure line, after the exposure of one exposure line, following exposure line exposure start until a predetermined exposure was not free time (predetermined Waiting time) t _D2 can be provided. In this configuration, since the exposure time can be extended, a bright image can be taken, and noise is reduced, so that error tolerance is high. On the other hand, in this configuration, since the number of exposure lines that can be exposed within a certain time is reduced, there is a disadvantage that the number of samples is reduced as in 7502h. For example, when the imaging target is bright, the former configuration is used, and when the imaging target is dark, the latter configuration can be used to reduce the estimation error of the light source luminance change.

In all exposure lines, it is not necessary that the exposure time of adjacent exposure lines has a partial overlap in time, and a configuration in which some exposure lines have no partial overlap. It is also possible. Further, in all exposure lines, it is not necessary to provide a configuration in which an idle time (predetermined waiting time) in which a predetermined exposure is not performed is provided after the exposure of one exposure line is completed until the exposure of the next exposure line is started. It is also possible to have a configuration in which the lines partially overlap in time. With such a configuration, it is possible to take advantage of the advantages of each configuration. Further, the same readout method is used in the normal shooting mode in which shooting is performed at a normal frame rate (30 fps, 60 fps) and in the visible light communication mode in which shooting is performed with an exposure time of 1/480 second or less in which visible light communication is performed. Alternatively, a signal may be read by a circuit. By reading out signals with the same reading method or circuit, it is not necessary to use different circuits for the normal imaging mode and the visible light communication mode, and the circuit scale can be reduced.

FIG. 5D shows the relationship between the minimum change time t _S of the light source luminance, the exposure time t _E , the time difference t _D of the exposure start time of each exposure line, and the captured image. When t _E + t _D <t _S , since one or more exposure lines are always imaged in a state where the light source does not change from the start to the end of exposure, an image with clear brightness as in 7503d is obtained. It is easy to recognize the luminance change of the light source. When 2t _E > t _S , a bright line having a pattern different from the luminance change of the light source may be obtained, and it becomes difficult to recognize the luminance change of the light source from the captured image.

Figure 5E, the transition and time t _T of the light source luminance, which shows the relationship between the time difference t _D of the exposure start time of each exposure line. as t _D is larger than the t _T, exposure lines to be neutral is reduced, it is easy to estimate the light source luminance. When t _D > t _T , the exposure line of the intermediate color is continuously 2 or less, which is desirable. t _T, the light source is less than 1 microsecond in the case of LED, light source for an approximately 5 microseconds in the case of organic EL, a t _D by 5 or more microseconds, to facilitate estimation of the light source luminance be able to.

Figure 5F shows a high frequency noise _{t HT} of light source luminance, the relationship between the exposure time _{t E.} As t _E is larger than t _HT , the captured image is less affected by high frequency noise, and light source luminance is easily estimated. When t _E is an integral multiple of t _HT , the influence of high frequency noise is eliminated, and the light source luminance is most easily estimated. For estimation of the light source luminance, it is desirable that t _E > t _HT . The main cause of high frequency noise derived from the switching power supply circuit, since many of the t _HT in the switching power supply for the lamp is less than 20 microseconds, by the t _E and 20 micro-seconds or more, the estimation of the light source luminance It can be done easily.

Figure 5G is the case t _HT is 20 microseconds, which is a graph showing the relationship between the size of the exposure time t _E and the high frequency noise. When t _HT is considered that there is variation by the light source, from the graph, t _E is the value becomes equal to the value when the amount of noise takes a maximum, 15 microseconds or more, or, 35 microseconds or more, or, It can be confirmed that the efficiency is good when it is set to 54 microseconds or more, or 74 microseconds or more. From the viewpoint of reducing high-frequency noise, it is desirable that t _E be large. However, as described above, there is a property that light source luminance can be easily estimated in that the smaller the t _E , the more difficult the intermediate color portion is generated. Therefore, when the light source luminance change period is 15 to 35 microseconds, t _E is 15 microseconds or more, and when the light source luminance change period is 35 to 54 microseconds, t _E is 35 microseconds or more. t _E is 54 microseconds or more when the cycle is 54 to 74 microseconds of change, t _E when the period of the change in light source luminance is 74 microseconds or more may be set as 74 microseconds or more.

Figure 5H shows the relationship between the exposure time t _E and the recognition success rate. Since the exposure time t _E has a relative meaning with respect to the time when the luminance of the light source is constant, the value (relative exposure time) obtained by dividing the period t _{S in} which the light source luminance changes by the exposure time t _E is taken as the horizontal axis. Yes. From the graph, it can be seen that if the recognition success rate is desired to be almost 100%, the relative exposure time should be 1.2 or less. For example, when the transmission signal is 1 kHz, the exposure time may be about 0.83 milliseconds or less. Similarly, when it is desired to set the recognition success rate to 95% or more, the relative exposure time may be set to 1.25 or less, and when the recognition success rate is set to 80% or more, the relative exposure time may be set to 1.4 or less. Recognize. Also, the recognition success rate drops sharply when the relative exposure time is around 1.5, and becomes almost 0% at 1.6, so it can be seen that the relative exposure time should not be set to exceed 1.5. . It can also be seen that after the recognition rate becomes 0 at 7507c, it rises again at 7507d, 7507e, and 7507f. Therefore, when it is desired to take a bright image by extending the exposure time, use an exposure time in which the relative exposure time is 1.9 to 2.2, 2.4 to 2.6, and 2.8 to 3.0. Just do it. For example, these exposure times may be used as the intermediate mode.

FIG. 6A is a flowchart of the information communication method in the present embodiment.

The information communication method in the present embodiment is an information communication method for acquiring information from a subject, and includes steps SK91 to SK93.

That is, in this information communication method, a plurality of bright lines corresponding to a plurality of exposure lines included in the image sensor are generated in an image obtained by photographing the subject by an image sensor according to a change in luminance of the subject. A first exposure time setting step SK91 for setting a first exposure time of the image sensor; and the image sensor shoots the subject whose luminance changes with the set first exposure time, First image acquisition step SK92 for acquiring a bright line image including a plurality of bright lines, and information acquisition for acquiring information by demodulating data specified by the patterns of the plurality of bright lines included in the acquired bright line image Step SK93, and in the first image acquisition step SK92, the plurality of exposure lines Re starts exposure at successively different times, and the exposure of the adjacent exposure line after a predetermined idle time from the end, the exposure is started adjacent to the exposure line.

FIG. 6B is a block diagram of the information communication apparatus according to the present embodiment.

The information communication device K90 in the present embodiment is an information communication device that acquires information from a subject, and includes constituent elements K91 to K93.

That is, the information communication apparatus K90 causes a plurality of bright lines corresponding to a plurality of exposure lines included in the image sensor to be generated in response to a change in luminance of the subject in an image obtained by photographing the subject by an image sensor. And an exposure time setting unit K91 for setting an exposure time of the image sensor, and the image sensor for acquiring a bright line image including the plurality of bright lines by photographing the subject whose luminance changes with the set exposure time. And an information acquisition unit K93 that acquires information by demodulating data specified by the patterns of the plurality of bright lines included in the acquired bright line image, and the plurality of exposure lines. Each of these starts exposure at different times sequentially and is adjacent to the exposure line. From the exposure of the down is completed after a predetermined idle time has elapsed, exposure is started.

In the information communication method and the information communication apparatus K90 shown in FIGS. 6A and 6B as described above, for example, as shown in FIG. 5C, each of the plurality of exposure lines is exposed to the adjacent exposure line adjacent to the exposure line. Since exposure is started after a lapse of a predetermined idle time after the end, it is possible to easily recognize a change in luminance of the subject. As a result, information can be appropriately acquired from the subject.

In the above embodiment, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. For example, the program causes the computer to execute the information communication method shown by the flowchart of FIG. 6A.

(Embodiment 2)
In the present embodiment, each application example using a receiver such as a smartphone that is the information communication device K90 in the first embodiment and a transmitter that transmits information as a blinking pattern of a light source such as an LED or an organic EL. explain.

In the following description, shooting in the normal shooting mode or normal shooting mode is referred to as normal shooting, and shooting in the visible light communication mode or visible light communication mode is referred to as visible light shooting (visible light communication). Further, instead of normal shooting and visible light shooting, shooting in an intermediate mode may be used, and an intermediate image may be used instead of a composite image described later.

FIG. 7 is a diagram illustrating an example of the photographing operation of the receiver in this embodiment.

The receiver 8000 switches the shooting mode to normal shooting, visible light communication, normal shooting, and so on. Then, the receiver 8000 generates a composite image in which the bright line pattern, the subject, and the surrounding area are clearly displayed by combining the normal captured image and the visible light communication image, and displays the composite image on the display. . This composite image is an image generated by superimposing the bright line pattern of the visible light communication image on the portion where the signal in the normal captured image is transmitted. Further, the bright line pattern, the subject, and the surroundings displayed by the composite image are clear and have a sharpness sufficiently recognized by the user. By displaying such a composite image, the user can more clearly know from where or from where the signal is transmitted.

FIG. 8 is a diagram illustrating another example of the photographing operation of the receiver in this embodiment.

The receiver 8000 includes a camera Ca1 and a camera Ca2. In such a receiver 8000, the camera Ca1 performs normal photographing, and the camera Ca2 performs visible light photographing. Thereby, the camera Ca1 acquires the normal captured image as described above, and the camera Ca2 acquires the visible light communication image as described above. Then, the receiver 8000 generates the above-described combined image by combining the normal captured image and the visible light communication image, and displays the combined image on the display.

FIG. 9 is a diagram illustrating another example of the photographing operation of the receiver in this embodiment.

In the receiver 8000 having two cameras, the camera Ca1 switches the shooting mode to normal shooting, visible light communication, normal shooting, and so on. On the other hand, the camera Ca2 continuously performs normal shooting. When the normal shooting is performed simultaneously with the cameras Ca1 and Ca2, the receiver 8000 receives from the normal shooting images acquired by these cameras using stereo vision (the principle of triangulation). The distance from the machine 8000 to the subject (hereinafter referred to as subject distance) is estimated. By using the subject distance estimated in this way, the receiver 8000 can superimpose the bright line pattern of the visible light communication image on an appropriate position of the normal captured image. That is, an appropriate composite image can be generated.

FIG. 10 is a diagram illustrating an example of the display operation of the receiver in this embodiment.

As described above, the receiver 8000 switches the photographing mode to visible light communication, normal photographing, visible light communication, and so on. Here, the receiver 8000 activates an application program when performing visible light communication for the first time. Then, the receiver 8000 estimates its own position based on the signal received by visible light communication. Next, when performing normal shooting, the receiver 8000 displays AR (Augmented Reality) information on the normal shot image acquired by the normal shooting. This AR information is acquired based on the position estimated as described above. Further, the receiver 8000 estimates the movement and direction change of the receiver 8000 based on the detection result of the 9-axis sensor and the motion detection of the normal captured image, and matches the estimated movement and direction change. To move the display position of the AR information. Thereby, the AR information can be made to follow the subject image of the normal captured image.

Further, when the receiver 8000 switches the shooting mode from the normal shooting to the visible light communication, the AR information is superimposed on the latest normal shooting image acquired at the time of the normal shooting immediately before the visible light communication. The receiver 8000 displays a normal captured image on which the AR information is superimposed. Similarly to the normal shooting, the receiver 8000 estimates the movement and direction change of the receiver 8000 on the basis of the detection result by the 9-axis sensor, and AR in accordance with the estimated movement and direction change. Move information and normal captured images. Thereby, AR information can be made to follow the subject image of the normal captured image in accordance with the movement of the receiver 8000 or the like in the case of visible light communication as in the case of normal imaging. Further, the normal image can be enlarged and reduced in accordance with the movement of the receiver 8000 or the like.

FIG. 11 is a diagram showing an example of the display operation of the receiver in this embodiment.

For example, the receiver 8000 may display the composite image on which the bright line pattern is projected, as shown in FIG. In addition, as shown in FIG. 11B, the receiver 8000 normally captures a signal explicit object that is an image having a predetermined color for notifying that a signal is transmitted instead of the bright line pattern. A composite image may be generated by superimposing on the image, and the composite image may be displayed.

Further, as shown in FIG. 11 (c), the receiver 8000 normally has a location where a signal is transmitted indicated by a dotted frame and an identifier (for example, ID: 101, ID: 102, etc.). The captured image may be displayed as a composite image. Further, as shown in FIG. 11D, the receiver 8000 recognizes a signal that is an image having a predetermined color for notifying that a specific type of signal is transmitted instead of the bright line pattern. A composite image may be generated by superimposing an object on a normal captured image, and the composite image may be displayed. In this case, the color of the signal identification object differs depending on the type of signal output from the transmitter. For example, when the signal output from the transmitter is position information, a red signal identification object is superimposed, and when the signal output from the transmitter is a coupon, the green signal identification object is Superimposed.

FIG. 12 is a diagram illustrating an example of the operation of the receiver in this embodiment.

For example, when receiving a signal through visible light communication, the receiver 8000 may display a normal captured image and output a sound for notifying the user that the transmitter has been found. In this case, the receiver 8000 varies the type of output sound, the number of outputs, or the output time depending on the number of transmitters found, the type of received signal, or the type of information specified by the signal. It may be allowed.

FIG. 13 is a diagram illustrating another example of the operation of the receiver in this embodiment.

For example, when the user touches the bright line pattern displayed in the composite image, the receiver 8000 generates an information notification image based on a signal transmitted from the subject corresponding to the touched bright line pattern, and the information notification Display an image. This information notification image indicates, for example, a store coupon or a place. The bright line pattern may be a signal explicit object, a signal identification object, a dotted line frame, or the like shown in FIG. The same applies to the bright line patterns described below.

FIG. 14 is a diagram illustrating another example of the operation of the receiver in this embodiment.

For example, when the user touches the bright line pattern displayed in the composite image, the receiver 8000 generates an information notification image based on a signal transmitted from the subject corresponding to the touched bright line pattern, and the information notification Display an image. For example, the information notification image indicates the current location of the receiver 8000 by a map or the like.

FIG. 15 is a diagram illustrating another example of the operation of the receiver in this embodiment.

For example, when the user performs a swipe on the receiver 8000 on which the composite image is displayed, the receiver 8000 performs normal shooting having a dotted frame and an identifier, similar to the normal shot image illustrated in FIG. An image is displayed and a list of information is displayed so as to follow the swipe operation. In this list, information specified by a signal transmitted from a location (transmitter) indicated by each identifier is shown. The swipe may be, for example, an operation of moving a finger from outside the right side of the display in the receiver 8000. The swipe may be an operation of moving a finger from the upper side, the lower side, or the left side of the display.

In addition, when information included in the list is tapped by the user, the receiver 8000 may display an information notification image (for example, an image showing a coupon) showing the information in more detail.

FIG. 16 is a diagram illustrating another example of the operation of the receiver in this embodiment.

For example, when the user swipes the receiver 8000 on which the composite image is displayed, the receiver 8000 displays the information notification image superimposed on the composite image so as to follow the swipe operation. This information notification image shows the subject distance with an arrow in an easy-to-understand manner for the user. The swipe may be, for example, an operation of moving a finger from outside the lower side of the display in the receiver 8000. The swipe may be an operation of moving a finger from the left side of the display, from the upper side, or from the right side.

FIG. 17 is a diagram illustrating another example of the operation of the receiver in this embodiment.

For example, the receiver 8000 images a transmitter, which is a signage indicating a plurality of stores, as a subject, and displays a normal captured image acquired by the imaging. Here, when the user taps the signage image of one store included in the subject displayed in the normal captured image, the receiver 8000 generates an information notification image based on a signal transmitted from the signage of the store Then, the information notification image 8001 is displayed. This information notification image 8001 is an image showing, for example, a vacant seat situation in a store.

The information communication method according to the present embodiment is an information communication method for acquiring information from a subject, and an bright line corresponding to an exposure line included in the image sensor is included in an image obtained by photographing the subject with an image sensor. A first exposure time setting step for setting an exposure time of the image sensor so as to occur in accordance with a change in luminance of the subject; and the image sensor photographs the subject whose luminance changes with the set exposure time. The bright line image acquisition step of acquiring a bright line image that is an image including the bright line, and the spatial position of the portion where the bright line appears can be identified based on the bright line image, and the subject and the subject An image display step for displaying a display image in which the surroundings of the subject are projected, and before the image is included in the acquired bright line image Including an information acquisition step of acquiring transmission information by demodulating the data identified by the pattern of bright lines.

For example, a composite image or an intermediate image as shown in FIGS. 7, 8, and 11 is displayed as a display image. In the display image in which the subject and the surroundings of the subject are projected, the spatial position of the part where the bright line appears is identified by a bright line pattern, a signal explicit object, a signal identification object, a dotted line frame, or the like. Therefore, the user can easily find a subject that is transmitting a signal due to a change in luminance by viewing such a display image.

The information communication method further includes a second exposure time setting step for setting an exposure time longer than the exposure time, and the image sensor photographs the subject and the surroundings of the subject with the long exposure time. A normal image acquisition step of acquiring a normal photographed image, a part where the bright line appears in the normal photographed image is identified based on the bright line image, and a signal object which is an image indicating the part is designated as the normal image A composite step of generating a composite image by superimposing the captured image, and the composite image may be displayed as the display image in the image display step.

For example, the signal object is a bright line pattern, a signal explicit object, a signal identification object, a dotted line frame, or the like, and a composite image is displayed as a display image as shown in FIGS. Thus, the user can more easily find the subject that is transmitting the signal due to the luminance change.

In the first exposure time setting step, an exposure time is set to 1/3000 sec. In the bright line image acquisition step, the bright line image in which the periphery of the subject is projected is acquired, and in the image display step. The bright line image may be displayed as the display image.

For example, the bright line image is acquired and displayed as an intermediate image. Therefore, it is not necessary to perform processing such as acquiring and synthesizing the normal captured image and the visible light communication image, and the processing can be simplified.

The image sensor includes a first image sensor and a second image sensor. In the normal image acquisition step, the first image sensor captures the normal captured image, and the bright line is acquired. In the image acquisition step, the bright line image may be acquired by capturing the second image sensor simultaneously with the capturing of the first image sensor.

For example, as shown in FIG. 8, a normal photographed image and a visible light communication image that is a bright line image are acquired by each camera. Therefore, compared with the case where a normal captured image and a visible light communication image are acquired with one camera, those images can be acquired earlier, and the processing can be speeded up.

Further, in the information communication method, when the part where the bright line appears in the display image is designated by a user operation, the information communication method further includes the transmission information acquired from the pattern of the bright line of the designated part. An information presentation step of presenting presentation information based on the information may be included. For example, the operation by the user is shown in association with a tap, swipe, an operation in which a fingertip is continuously applied to the part for a predetermined time, an operation in which a line of sight is directed to the part for a predetermined time or more, An operation of moving a part of the user's body to an arrow, an operation of applying a pen tip that changes in luminance to the part, or a touch sensor is touched, and a pointer displayed on the display image is applied to the part It is an operation.

For example, as shown in FIGS. 13 to 17, the presentation information is displayed as an information notification image. Thereby, desired information can be presented to the user.

In addition, in the information communication method for acquiring information from a subject, a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in luminance of the subject. As described above, the first exposure time setting step of setting the exposure time of the image sensor, and the image including the bright line by the image sensor photographing the subject whose luminance changes with the set exposure time. A bright line image acquiring step for acquiring the bright line image, and an information acquiring step for acquiring information by demodulating data specified by the pattern of the bright line included in the acquired bright line image. In the acquisition step, the plurality of subjects are photographed during the period in which the image sensor is moved. By acquiring the bright line image including a plurality of parts where the bright lines appear, and in the information acquisition step, for each part, by demodulating data specified by the pattern of the bright lines of the parts, The information communication method further estimates the position of the image sensor based on the acquired positions of the plurality of subjects and the movement state of the image sensor. A position estimation step may be included.

This makes it possible to accurately estimate the position of the receiver including the image sensor based on luminance changes caused by subjects such as a plurality of lights.

In addition, in the information communication method for acquiring information from a subject, a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in luminance of the subject. As described above, the first exposure time setting step of setting the exposure time of the image sensor, and the image including the bright line by the image sensor photographing the subject whose luminance changes with the set exposure time. A bright line image acquisition step of acquiring a bright line image, an information acquisition step of acquiring information by demodulating data specified by a pattern of the bright line included in the acquired bright line image, and the acquired information Information presenting step for presenting, in the information presenting step, the image sensor Against over The may present an image that prompts the gesture that has been predetermined as the information.

Thus, depending on whether or not the user performs the gesture as prompted, the user can be authenticated and the convenience can be improved.

In addition, in the information communication method for acquiring information from a subject, a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in luminance of the subject. As described above, the exposure time setting step for setting the exposure time of the image sensor, and the image sensor captures the bright line image including the bright line by photographing the subject whose luminance changes at the set exposure time. And an information acquisition step of acquiring information by demodulating data specified by the bright line pattern included in the acquired bright line image. In the image acquisition step, the image is reflected on the reflection surface. The bright line image is acquired by photographing a plurality of the subjects, and the information acquisition step is performed. The bright line is separated into bright lines corresponding to each of the plurality of subjects according to the intensity of the bright lines included in the bright line image, and each subject is specified by a bright line pattern corresponding to the subject. Information may be acquired by demodulating the data.

This makes it possible to acquire appropriate information from each of the subjects even when the subject such as a plurality of illuminations changes in luminance.

In addition, in the information communication method for acquiring information from a subject, a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in luminance of the subject. As described above, the exposure time setting step for setting the exposure time of the image sensor, and the image sensor captures the bright line image including the bright line by photographing the subject whose luminance changes at the set exposure time. And an information acquisition step of acquiring information by demodulating data specified by the bright line pattern included in the acquired bright line image. In the image acquisition step, the image is reflected on the reflection surface. The bright line image is acquired by photographing the subject, and the information communication method includes: , Based on the luminance distribution in the emission line image may include position estimation step for estimating the position of the object.

This makes it possible to estimate an appropriate subject position based on the luminance distribution.

Further, in the transmission step, when the luminance change is switched between the luminance change according to the first pattern and the luminance change according to the second pattern, it may be switched with a buffer time.

Thereby, interference between the first signal and the second signal can be suppressed.

An information communication method for transmitting a signal according to a luminance change, wherein a determination step of determining a luminance change pattern by modulating a signal to be transmitted, and the light emitter changes in luminance according to the determined pattern And transmitting the signal to be transmitted, the signal comprising a plurality of large blocks, each of the plurality of large blocks including first data and a preamble for the first data. And a check signal for the first data, wherein the first data is composed of a plurality of small blocks, and the small blocks include second data, a preamble for the second data, and the first data. And a check signal for the second data may be included.

This makes it possible to acquire data appropriately for both receivers that require a blanking period and receivers that do not require a blanking period.

An information communication method for transmitting a signal by luminance change, wherein a plurality of transmitters each modulate a signal to be transmitted to determine a luminance change pattern, and for each transmitter, And a transmitting step in which a light emitter provided in the transmitter changes the luminance according to the determined pattern and transmits the signal to be transmitted. In the transmitting step, signals having different frequencies or protocols are transmitted. May be.

This makes it possible to suppress signal interference from multiple transmitters.

An information communication method for transmitting a signal by luminance change, wherein a plurality of transmitters each modulate a signal to be transmitted to determine a luminance change pattern, and for each transmitter, A transmitter in which a light emitter provided in the transmitter transmits a signal to be transmitted by changing in luminance according to the determined pattern, wherein in the transmitting step, one of the plurality of transmitters One transmitter may receive a signal transmitted from the other transmitter and transmit another signal in a manner that does not interfere with the received signal.

This makes it possible to suppress signal interference from multiple transmitters.

(Guidance at the station)
FIG. 18A shows an example of a usage form of the present invention in a train platform. The user holds the portable terminal over an electronic bulletin board or lighting, and obtains information displayed on the electronic bulletin board, train information of a station where the electronic bulletin board is installed, information on the premises of the station, or the like by visible light communication. Here, the information itself displayed on the electronic bulletin board may be transmitted to the portable terminal by visible light communication, or ID information corresponding to the electronic bulletin board is transmitted to the portable terminal, and the ID information acquired by the portable terminal The information displayed on the electronic bulletin board may be acquired by inquiring the server. When the ID information is transmitted from the mobile terminal, the server transmits the content displayed on the electronic bulletin board to the mobile terminal based on the ID information. The train ticket information stored in the memory of the mobile terminal is compared with the information displayed on the electronic bulletin board, and the ticket information corresponding to the user's ticket is displayed on the electronic bulletin board. An arrow indicating the destination to the home where the user's scheduled train arrives is displayed on the display. When getting off, the route to the vehicle near the exit or the transfer route may be displayed.

If a seat is specified, the route to that seat may be displayed. When the arrow is displayed, it can be displayed more easily by displaying the arrow using the same color as the color of the train route in the map or the train guide information. In addition to the arrow display, the user's reservation information (home number, vehicle number, departure time, seat number) can also be displayed. By displaying the user reservation information together, it is possible to prevent erroneous recognition. If the ticket information is stored in the server, query the server from the mobile terminal to obtain and compare the ticket information, or compare the ticket information with the information displayed on the electronic bulletin board on the server side. Thus, information related to the ticket information can be acquired. The target route may be estimated from the history of the user performing a transfer search, and the route may be displayed. Further, not only the contents displayed on the electronic bulletin board but also the train information / premises information of the station where the electronic bulletin board is installed may be acquired and compared. Information related to the user may be highlighted with respect to the display of the electronic bulletin board on the display, or may be rewritten and displayed. When the user's boarding schedule is unknown, an arrow for guiding to the boarding place on each route may be displayed. When station premises information is acquired, an arrow for guiding to a store or restroom may be displayed on the display. The user's behavior characteristics may be managed in advance by a server, and an arrow for guiding the user to a store / restroom may be displayed on the display when the user often stops at a store / restaurant in the station. Only users who have behavioral characteristics of stopping at a store / restroom will display an arrow that directs them to a store / restroom, etc., and will not be displayed to other users, so the amount of processing can be reduced. . The color of the arrow leading to a store / restroom may be different from the arrow guiding the destination to the home. When both arrows are displayed simultaneously, it is possible to prevent erroneous recognition by using different colors. Although FIG. 18A shows an example of a train, it is possible to perform display with a similar configuration even on an airplane or a bus.

Specifically, a portable terminal such as a smartphone (that is, a receiver such as a receiver 200 described later) captures a visible light signal from the electronic bulletin board by imaging the electronic bulletin board as illustrated in (1) of FIG. 18A. Is received as an optical ID or optical data. At this time, the mobile terminal performs self-position estimation. That is, the mobile terminal acquires the position on the map of the electronic bulletin board indicated directly or indirectly by the optical data. Then, the mobile terminal, for example, with respect to the electronic bulletin board based on its own posture obtained by the 9-axis sensor and the position, shape, size, etc. in the image of the electronic bulletin board displayed in the image obtained by imaging. Calculate the relative position of the mobile terminal. The portable terminal estimates its own position, which is the position of the portable terminal on the map, based on the position of the electronic bulletin board on the map and its relative position. The mobile terminal searches for a route from the starting point that is the self-location to a destination indicated by, for example, ticket information, and starts navigation for guiding the user to the destination along the route. The mobile terminal may transmit information indicating the starting point and the destination to the server, and obtain the above-described route searched by the server from the server. At this time, the mobile terminal may acquire a map including the route from the server.

In navigation, as shown in (2) to (4) of FIG. 18A, the mobile terminal repeatedly captures images by the camera, and sequentially displays normal captured images obtained by the imaging in real time, while indicating an arrow indicating the user's destination Or the like is superimposed on the normal captured image. The user moves according to the displayed direction instruction image while carrying the portable terminal. Then, the mobile terminal updates the self-position of the mobile terminal based on the movement of the object or the feature point displayed in each of the above normal captured images. For example, the portable terminal detects the movement of the object or feature point displayed in each of the above-described normal captured images, and estimates the movement direction and movement distance of the portable terminal based on the movement. And a portable terminal updates the present self position based on the estimated moving direction and moving distance, and the self position estimated in (1) of FIG. 18A. This self-position update may be performed every frame period of the normal captured image, or may be performed every period longer than the frame period. That is, when the mobile terminal is on the underground floor or route, the mobile terminal cannot acquire GPS data. Therefore, in such a case, the mobile terminal estimates or updates its own position based on the movement of the above-described feature points of each normal captured image without using GPS data.

Here, as shown in (4) of FIG. 18A, the mobile terminal may guide the elevator to the user during the route to the destination. Further, as shown in (5) and (6) of FIG. 18A, when the portable terminal images the transmitter that transmits the optical data or the reflected light of the optical data, the mobile terminal receives the optical data, and FIG. As in the example shown in (1), the self-position is estimated. For example, even when a user gets on an elevator, the mobile terminal transmits optical data transmitted from a transmitter (that is, a transmitter such as a transmitter 100 described later) installed as an illumination device or the like inside the elevator cage. Receive. For example, the light data directly or indirectly indicates the floor on which the elevator car is currently located. Therefore, the mobile terminal can identify the floor on which the mobile terminal is currently located by receiving the optical data. If the current position of the bag is not directly indicated by the optical data, the mobile terminal transmits the information indicated by the optical data to the server, and stores the floor information associated with the information in the server. , Get from that server. Thus, the mobile terminal specifies the floor indicated by the floor information as the floor where the mobile terminal is currently located. The floor specified in this way is treated as a self-position.

As a result, as shown in (7) of FIG. 18A, the terminal device replaces the self-position derived from the movement of the feature points of the normal captured image with the self-position derived using the optical data. , Reset the self-position.

Then, as shown in (8) of FIG. 18A, the mobile terminal performs the same processing as (2) to (4) of FIG. 18A if the user does not reach the destination after getting off the elevator. While navigating. Moreover, the portable terminal repeatedly confirms whether GPS data can be acquired during navigation. Therefore, the portable terminal determines that the GPS data can be acquired when it goes up from the underground floor or route. And a portable terminal switches the estimation method of a self-position from the estimation method based on motion, such as a feature point, to the estimation method based on GPS data. Then, as shown in (9) of FIG. 18A, the mobile terminal continues to execute navigation until the user arrives at the destination while estimating its own position based on the GPS data. Note that since the mobile terminal cannot acquire GPS data, for example, when the user enters the basement again, the self-position estimation method is changed from the estimation method based on GPS data to the estimation method based on the movement of feature points or the like. Switch to.

Hereinafter, the example of FIG. 18A will be described in detail.

In the example shown in FIG. 18A, for example, a receiver mounted as a wearable device such as a smartphone or smart glass receives the visible light signal (optical data) transmitted from the transmitter in (1) of FIG. 18A. The transmitter is implemented, for example, as an illumination signboard, a poster, or illumination that illuminates an image. The receiver starts navigation to the destination according to the received optical data, information preset in the receiver, and a user instruction. The receiver transmits optical data to the server and obtains navigation information associated with the data. The navigation information includes the following first information to sixth information. The first information is information indicating the position and shape of the transmitter. The second information is information indicating a route to the destination. The third information is information on another transmitter on and near the route to the destination. Specifically, the information of another transmitter indicates the optical data transmitted by the transmitter, the position and shape of the transmitter, and the position and shape of the reflected light. The fourth information is position specifying information regarding the route and its vicinity. Specifically, the position specifying information is radio wave information or sound wave information for specifying an image feature amount or position. The fifth information is information indicating the distance to the destination and the estimated arrival time. The sixth information is part or all of the content information for performing the AR display. The navigation information may be stored in advance in the receiver. Note that the above-described shape may include a size.

The receiver uses the relative position between the transmitter and the receiver, which is calculated from the way the transmitter is reflected in the image obtained by imaging and the sensor value of the acceleration sensor, and the position information of the transmitter. The self position is estimated, and the self position is set as the starting point of navigation. The receiver may start navigation by estimating the receiver's own position not by optical data but by image feature values, barcodes or two-dimensional codes, radio waves, or sound waves.

The receiver displays the navigation to the destination as shown in (2) of FIG. 18A. The navigation display may be an AR display in which another image is superimposed on a normal captured image obtained by imaging by a camera, may be a map display, may be an instruction by voice or vibration, A combination of these may also be used. The display method may be selected by a receiver, optical data, or settings on the server. Any setting may be prioritized. If the destination (that is, the destination) is a boarding place for transportation, the receiver acquires the timetable and displays the reserved time or the departure time or boarding time near the expected arrival time. Also good. If the destination is a theater or the like, the receiver may display the start time or the admission deadline.

The receiver advances navigation according to the movement of the receiver as shown in (3) and (4) of FIG. 18A. In a situation where absolute position information cannot be obtained, the receiver determines from the moving distance between the images of the feature points in the plurality of images, to the moving distance of the receiver during the imaging of those images. The direction may be estimated. The receiver may estimate the moving distance and direction of the receiver from the transition of the acceleration sensor, radio waves, or sound waves. Further, the receiver may estimate the moving distance and direction of the receiver by SLAM (Simultaneous Localization and Mapping) or PTAM (Parallel Tracking and Mapping).

In (5) of FIG. 18A, when the receiver receives optical data different from the optical data received in (1) of FIG. 18A, for example, outside the elevator, the receiver sends the optical data to the server. The shape and position of the transmitter associated with the data may be obtained. Then, the receiver may estimate the receiver's own position by the same method as (1) in FIG. 18A. Thereby, the receiver corrects the current position of navigation by eliminating the error of the receiver's self-position estimation that has occurred in the processes (3) and (4) of FIG. 18A. If the receiver receives only part of the visible light signal and complete optical data cannot be obtained, the transmitter that is nearest to the navigation information is the transmitter that is transmitting the visible light signal. It is estimated that there is, and thereafter, the receiver's self-position is estimated in the same manner as described above. This allows transmitters with poor reception conditions, such as small transmitters, remote transmitters, or dark transmitters, to be used for receiver self-position estimation. it can.

The receiver receives optical data by reflected light in (6) of FIG. 18A. The receiver identifies that the medium of the received optical data is reflected light based on the imaging direction, the light intensity, or the clarity of the outline. In the case of reflected light, the receiver identifies the position of the reflected light (that is, the position on the map) from the navigation information, and estimates the center of the reflected light area being imaged as the position of the reflected light. . Then, as in (5) of FIG. 18A, the receiver estimates the receiver's own position and corrects the current navigation position.

When the receiver receives a signal for specifying the position of GPS, GLONASS, Galileo, Hokuto satellite positioning system, IRNSS, etc., the receiver specifies the position of the receiver based on the signal, and the current position of navigation (ie, self-position). ) Is corrected. If the strength of the signal is sufficient, that is, if the strength is higher than a predetermined strength, the receiver estimates the self-position based only on the signal. If the strength is equal to or lower than the predetermined strength, (3) in FIG. 18A and The method used in (4) may be used in combination.

When the receiver receives a visible light signal, [1] a radio signal having a predetermined ID received simultaneously with the visible light signal, [2] a radio signal having a predetermined ID received last, or [3] The information indicating the position of the last estimated receiver is transmitted to the server together with the information indicated by the visible light signal. Thereby, the transmitter which transmits the visible light signal is specified. Alternatively, the receiver receives a visible light signal with an algorithm specified by the above-described radio signal or information indicating the position of the receiver, and is indicated by the visible light signal to a server specified in the same manner as described above. Information may be transmitted.

The receiver may estimate the self-position and display information on products near the self-position. Further, the receiver may navigate to the position of the product designated by the user. In addition, the receiver may present an optimal route to go around all the locations of the plurality of products specified by the user. The optimum route is the route with the shortest distance, the route with the shortest required time, or the route with the least amount of movement effort. Further, the receiver may perform navigation so as to pass through a predetermined place in addition to the product or place designated by the user. Thereby, the advertisement of a predetermined place, or the goods or store in the place can be performed.

FIG. 18B is a diagram for describing navigation by the receiver 200 in the elevator according to the present embodiment.

For example, when a user is on the third basement floor (B3), a receiver configured as a smartphone executes a guide using AR display, that is, AR navigation, as shown in (1) of FIG. 18B. As shown in FIG. 18A, AR navigation is a navigation function that guides a user to a destination by superimposing a direction indicating image such as an arrow on a normal captured image. Hereinafter, AR navigation is also simply referred to as navigation or navigation.

Then, when the user gets on the elevator, as shown in (2) of FIG. 18B, the receiver receives an optical signal (that is, a visible light signal, optical data, or optical ID) from a transmitter disposed in the elevator cage. Receive. Thereby, a receiver acquires elevator ID and floor information based on the optical signal. The elevator ID is identification information for identifying the elevator in which the transmitter is arranged or the cage, and the floor information is information indicating the floor (or the number of floors) where the cage is currently located. For example, the receiver transmits an optical signal (or information indicated by the optical signal) to a server, and obtains an elevator ID and rank information associated with the optical signal at the server from the server. The transmitter may always transmit the same optical signal regardless of the floor of the elevator, and may transmit different optical signals depending on the floor on which the fence is located. The transmitter is configured as a lighting device, for example. The light from this transmitter shines brightly inside the elevator car. Therefore, the receiver can directly receive the optical signal superimposed on such light from the transmitter, and can also indirectly receive the signal through reflection from the inner wall surface or floor of the fence.

The receiver is located at the current position according to the elevator ID and floor number information acquired based on the optical signal transmitted from the transmitter even when the cage containing the receiver is rising. Sequentially identifies the floors. Then, as shown in (3) of FIG. 18B, when the floor on which the receiver is currently located is the target floor, the receiver displays a message or an image prompting to get off the elevator on the display of the receiver. To do. Further, the receiver may output a sound prompting to get off the elevator.

If the target floor is a place where GPS data does not reach, such as the first basement floor, the receiver uses the estimation method using the movement of the feature points of the normal captured image as described above (( As shown in 4), the above-mentioned AR navigation is resumed while estimating the self-position. On the other hand, if the target floor is a place where GPS data arrives, such as the first floor above the ground, the receiver determines its own position by an estimation method using the GPS data as shown in (4) of FIG. 18B. The above-mentioned AR navigation is resumed while estimating.

FIG. 18C is a diagram illustrating an example of a system configuration provided in the elevator according to the present embodiment.

The transmitter 100, which is the above-mentioned transmitter, is installed in the elevator cage 420. The transmitter 100 is disposed on the ceiling of the eaves 420 as an illuminating device for the eaves 420 of the elevator. The transmitter 100 also includes a built-in camera 404 and a microphone 411. The built-in camera 404 takes an image of the inside of the bag 420, and the microphone 411 collects the sound inside the bag 420.

In addition, a monitoring camera system 401, a floor display unit 414, and a sensor 403 are installed in the basket 420. The surveillance camera system 401 is a system having at least one camera that captures an image of the interior of the bag 420. The floor display unit 414 displays the floor on which the bag 420 is currently located. The sensor 403 includes, for example, at least one of an atmospheric pressure sensor and an acceleration sensor.

The elevator also includes an image recognition unit 402, a current floor detection unit 405, a light modulation unit 406, a light emitting circuit 407, a wireless unit 409, and a voice recognition unit 410.

The image recognizing unit 402 recognizes a character (that is, a floor) displayed on the floor display unit 414 from an image obtained by imaging by the monitoring camera system 401 or the built-in camera 404, and obtains current floor data obtained by the recognition. Output. The current floor data indicates the number of floors displayed in the floor number display unit 414.

The voice recognition unit 410 recognizes the floor where the bag 420 is currently located based on the voice data output from the microphone 411, and outputs floor data indicating the floor.

The current floor detection unit 405 detects the floor on which the bag 420 is currently located based on data output from at least one of the sensor 403, the image recognition unit 402, and the voice recognition unit 410. Then, the current floor detection unit 405 outputs information indicating the detected floor to the light modulation unit 406.

The light modulation unit 406 modulates the signal indicating the floor output from the current floor detection unit 405 and the signal indicating the elevator ID, and outputs the modulated signal to the light emitting circuit 407. The light emitting circuit 407 changes the luminance of the transmitter 100 in accordance with the modulated signal. As a result, the visible light signal, the optical signal, the optical data, or the optical ID, which indicates the floor where the fence 420 is currently located and the elevator ID, is transmitted from the transmitter 100.

Similarly to the light modulation unit 406, the radio unit 409 modulates information indicating the floor output from the current floor detection unit 405 and a signal indicating the elevator ID, and transmits the modulated signal by radio. For example, the wireless unit 409 transmits a signal by Wi-Fi or Bluetooth.

Thereby, the receiver 200 can identify the floor and the elevator ID where the receiver 200 is currently located by receiving at least one of the radio signal and the optical signal.

Further, the elevator may include a current floor detection unit 412 having the above-described floor number display unit 414. The current floor detection unit 412 includes an elevator control unit 413 and a floor number display unit 414. The elevator control unit 413 controls the elevating and stopping of the eaves 420. Such an elevator control unit 413 keeps track of the floor on which the fence 420 is currently located. The elevator control unit 413 may output data indicating the grasped floor to the light modulation unit 406 and the radio unit 409 as current floor data.

With such a configuration, the receiver 200 can realize the AR navigation shown in FIGS. 18A and 18B.

(Example of application to directions)
FIG. 19 is a diagram illustrating an example of application of the transmission / reception system in the second embodiment.

The receiver 8955a receives, for example, the transmission ID of the transmitter 8955b configured as a guide plate, acquires the map data displayed on the guide plate from the server, and displays the map data. At this time, the server may transmit an advertisement suitable for the user of the receiver 8955a, and the receiver 8955a may also display this advertisement information. The receiver 8955a displays a route from the current location to a location designated by the user.

(Application log storage and application examples)
FIG. 20 is a diagram illustrating an example of an application example of the transmission and reception system in the second embodiment.

The receiver 8957a receives the ID transmitted from the transmitter 8957b configured as a signboard, for example, acquires coupon information from the server, and displays the coupon information. The receiver 8957a stores subsequent user actions such as saving a coupon, moving to a store displayed on the coupon, shopping at the store, and leaving without saving the coupon. Save to 8957c. As a result, the subsequent behavior of the user who has acquired information from the sign 8957b can be analyzed, and the advertising value of the sign 8957b can be estimated.

The information communication method in the present embodiment is an information communication method for acquiring information from a subject, and each exposure included in the image sensor is included in an image obtained by photographing the first subject that is the subject by an image sensor. A first exposure time setting step for setting a first exposure time of the image sensor so that a plurality of bright lines corresponding to a line are generated according to a change in luminance of the first subject; A first bright line image acquisition step of acquiring a first bright line image that is an image including the plurality of bright lines by photographing the first subject that changes with the set first exposure time; The first transmission information is acquired by demodulating the data specified by the plurality of bright line patterns included in the acquired first bright line image. Comprising a first information obtaining step, after the first transmission information is acquired by sending a control signal, and a door control steps to open the door against the opening and closing devices of the door.

This makes it possible to use a receiver equipped with an image sensor like a door key and eliminate the need for a special electronic lock. As a result, it is possible to perform communication between various devices including a device having a small computing power.

The information communication method may further include a second image in which the image sensor includes a plurality of bright lines by photographing the second subject whose luminance changes with the set first exposure time. Second transmission information is acquired by demodulating data specified by a pattern of the plurality of bright lines included in the acquired second bright line image and a second bright line image acquiring step of acquiring a bright line image A second information acquisition step; and an approach determination step of determining whether or not a receiving device including the image sensor is approaching the door based on the acquired first and second transmission information. In the door control step, the control signal may be transmitted when it is determined that the receiving device is approaching the door.

Thereby, the door can be opened only when the receiving device (receiver) approaches the door, that is, only at an appropriate timing.

Further, in the information communication method, a second exposure time setting step for setting a second exposure time longer than the first exposure time, and the image sensor sets a third subject. A normal image acquisition step of acquiring a normal image on which the third subject is projected by photographing at the second exposure time, and the normal image acquisition step includes an area including optical black of the image sensor For each of the plurality of exposure lines, the charge is read after a predetermined time has elapsed from the time when the charge is read for the exposure line adjacent to the exposure line, and the first bright line image obtaining step is performed. Then, without using the optical black for the charge readout, the optical black in the image sensor is different from the optical black. For each of the plurality of exposure lines in the region, the charge is read after a time longer than the predetermined time from when the charge is read for the exposure line adjacent to the exposure line. Also good.

As a result, when the first bright line image is acquired, the readout (exposure) of charges from the optical black is not performed. Therefore, the readout (exposure) of charges from the effective pixel area, which is an area other than the optical black in the image sensor, is performed. This time can be lengthened. As a result, it is possible to increase the time for receiving a signal in the effective pixel region, and it is possible to acquire many signals.

Further, in the information communication method, the length in the direction perpendicular to each of the plurality of bright lines in the plurality of bright line patterns included in the first bright line image is less than a predetermined length. A length determination step for determining whether or not there is a frame rate of the image sensor when the pattern length is determined to be less than the predetermined length; A frame rate changing step of changing to a second frame rate that is slower than the first frame rate when the image is acquired; and the image sensor detects the first subject whose luminance changes at the second frame rate. And a third bright line image acquisition step of acquiring a third bright line image, which is an image including a plurality of bright lines, by taking an image with the set first exposure time. It may include a third information obtaining step of obtaining the first transmission information by demodulating the data identified by said plurality of emission lines pattern contained in has been the third bright line image.

As a result, when the signal length indicated by the bright line pattern (bright line region) included in the first bright line image is less than, for example, one block of the transmitted signal, the frame rate is reduced and the bright line is renewed. An image is acquired as a third bright line image. As a result, the length of the bright line pattern included in the third bright line image can be increased, and the transmitted signal can be acquired for one block.

The information communication method further includes a ratio setting step for setting a ratio between a vertical width and a horizontal width of an image obtained by the image sensor, and the first bright line image acquisition step includes the set ratio. A clipping determination step for determining whether or not an end in a direction perpendicular to each exposure line in the image is clipped, and when it is determined that the end is clipped, the ratio set in the ratio setting step Changing the ratio to a non-clipping ratio which is a ratio at which the edge is not clipped, and the image sensor captures the first bright line with the non-clipping ratio by photographing the first subject whose luminance changes. An acquisition step of acquiring an image.

Thereby, for example, when the ratio of the horizontal width to the vertical width of the effective pixel area of the image sensor is 4: 3, the ratio of the horizontal width to the vertical width of the image is set to 16: 9, and a bright line along the horizontal direction appears. That is, when the exposure line is along the horizontal direction, it is determined that the upper end and the lower end of the image are clipped. That is, it is determined that the end of the first bright line image is missing. In this case, the ratio of the image is changed to 4: 3, which is a ratio that is not clipped. As a result, it is possible to prevent the end of the first bright line image from being lost, and a large amount of information can be acquired from the first bright line image.

The information communication method further includes a compression step of generating a compressed image by compressing the first bright line image in a direction parallel to each of the plurality of bright lines included in the first bright line image. And a compressed image transmission step of transmitting the compressed image.

Thereby, it is possible to appropriately compress the first bright line image without losing information indicated by a plurality of bright lines.

The information communication method further determines that the receiving device including the image sensor has been moved in a predetermined manner, and a gesture determination step for determining whether or not the receiving device has been moved in a predetermined manner. In some cases, an activation step of activating the image sensor may be included.

This makes it possible to easily start the image sensor only when necessary, and to improve power consumption efficiency.

In this embodiment, application examples using the above-described receiver such as a smartphone and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

FIG. 21 is a diagram illustrating an application example of the transmitter and the receiver in the second embodiment.

The robot 8970 has, for example, a function as a self-propelled cleaner and a function as a receiver in each of the above embodiments. The lighting devices 8971a and 8971b each have a function as a transmitter in each of the above embodiments.

For example, the robot 8970 performs cleaning while moving in the room and photographs the lighting device 8971a that illuminates the room. The lighting device 8971a transmits the ID of the lighting device 8971a by changing the luminance. As a result, the robot 8970 receives the ID from the lighting device 8971a and estimates its own position (self-position) based on the ID as in the above embodiments. That is, the robot 8970 moves itself based on the detection result by the 9-axis sensor, the relative position of the lighting device 8971a reflected in the image obtained by photographing, and the absolute position of the lighting device 8971a specified by the ID. Is estimated.

Furthermore, when the robot 8970 moves away from the lighting device 8971a by moving, the robot 8970 transmits a signal to turn off the lighting device 8971a (turn-off command). For example, when the robot 8970 leaves the lighting device 8971a by a predetermined distance, the robot 8970 transmits a turn-off command. Alternatively, the robot 8970 transmits a turn-off command to the lighting device 8971a when the lighting device 8971a does not appear in the image obtained by shooting or when another lighting device appears in the image. When the lighting device 8971a receives a turn-off command from the robot 8970, the lighting device 8971a turns off according to the turn-off command.

Next, the robot 8970 detects that it has approached the lighting device 8971b based on the estimated self-position while moving and cleaning. That is, the robot 8970 holds information indicating the position of the lighting device 8971b, and when the distance between the self position and the position of the lighting device 8971b is equal to or less than a predetermined distance, the lighting device 8971b. Detecting that you are approaching. Then, the robot 8970 transmits a signal (lighting command) for instructing lighting to the lighting device 8971b. When the lighting device 8971b receives the lighting command, the lighting device 8971b lights up in accordance with the lighting command.

Thereby, the robot 8970 can brighten only the surroundings while moving and can easily perform cleaning.

FIG. 22 is a diagram illustrating an application example of the transmitter and the receiver in the second embodiment.

The lighting device 8974 has a function as a transmitter in each of the above embodiments. The lighting device 8974 illuminates a route bulletin board 8975 at a railway station, for example, while changing in luminance. The receiver 8973 pointed to the route bulletin board 8975 by the user photographs the route bulletin board 8975. As a result, the receiver 8973 acquires the ID of the route bulletin board 8975, and acquires detailed information about each route described in the route bulletin board 8975, which is information associated with the ID. The receiver 8973 displays a guide image 8973a indicating the detailed information. For example, the guidance image 8973a indicates the distance to the route described on the route bulletin board 8975, the direction toward the route, and the time when the next train arrives on the route.

Here, when the guide image 8973a is touched by the user, the receiver 8973 displays a supplementary guide image 8973b. This supplementary guide image 8973b is, for example, a user's selection operation of any one of a railway timetable, information on another route different from the route indicated by the guide image 8973a, and detailed information on the station. It is an image for displaying accordingly.

(Embodiment 3)
Here, an application example of audio synchronized playback will be described below.

FIG. 23 is a diagram illustrating an example of an application according to the third embodiment.

For example, a receiver 1800a configured as a smartphone receives a signal (visible light signal) transmitted from a transmitter 1800b configured as, for example, a street digital signage. That is, the receiver 1800a receives the timing of image reproduction by the transmitter 1800b. The receiver 1800a reproduces sound at the same timing as the image reproduction. In other words, the receiver 1800a performs synchronized reproduction of the sound so that the image and sound reproduced by the transmitter 1800b are synchronized. Note that the receiver 1800a may reproduce the same image as the image (reproduced image) reproduced by the transmitter 1800b or a related image related to the reproduced image together with the sound. Further, the receiver 1800a may cause a device connected to the receiver 1800a to reproduce sound and the like. Further, after receiving the visible light signal, the receiver 1800a may download content such as sound or related images associated with the visible light signal from the server. The receiver 1800a performs synchronous reproduction after the download.

Thereby, even when the sound from the transmitter 1800b cannot be heard or when the sound from the transmitter 1800b is not reproduced because the street sound reproduction is prohibited, the user can select the sound that matches the display of the transmitter 1800b. Can hear. Further, even when there is a distance that takes time to reach the voice, it is possible to listen to the voice that matches the display.

Here, multilingual support by synchronized playback is described below.

FIG. 24 is a diagram illustrating an example of an application according to the third embodiment.

Each of the receiver 1800a and the receiver 1800c obtains and reproduces audio corresponding to a video such as a movie displayed on the transmitter 1800d from the server, in the language set in the receiver. To do. Specifically, the transmitter 1800d transmits a visible light signal indicating an ID for identifying the displayed video to the receiver. When the receiver receives the visible light signal, the receiver transmits a request signal including the ID indicated in the visible light signal and the language set in the receiver to the server. The receiver acquires the audio corresponding to the request signal from the server and reproduces it. Thereby, the user can enjoy the work displayed on the transmitter 1800d in the language set by the user.

Here, the audio synchronization method will be described below.

25 and 26 are diagrams showing an example of a transmission signal and an example of a voice synchronization method in the third embodiment.

Different data (for example, data shown in FIG. 25: 1 to 6 and the like) are associated with a time every fixed time (N seconds). These data may be, for example, an ID for identifying time, may be time, or may be audio data (for example, 64 Kbps data). The following description is based on the assumption that the data is an ID. Different IDs may have different additional information parts attached to the ID.

It is desirable that the packets that make up the ID are different. Therefore, it is desirable that IDs are not continuous. Alternatively, it is desirable to use a packetizing method in which the discontinuous portion is configured as one packet when the ID is packetized. Since error correction signals tend to have different patterns even with consecutive IDs, the error correction signals may be configured to be distributed in a plurality of packets rather than being combined into one packet.

The transmitter 1800d transmits the ID in accordance with the reproduction time of the displayed image, for example. The receiver can recognize the reproduction time (synchronization time) of the image of the transmitter 1800d by detecting the timing when the ID is changed.

In the case of (a), since the change time points of ID: 1 and ID: 2 are received, the synchronization time can be accurately recognized.

When the time N during which the ID is transmitted is long, there are few such opportunities, and the ID may be received as shown in (b). Even in this case, the synchronization time can be recognized by the following method.

(B1) Assume that the midpoint of the receiving section where the ID has changed is the ID changing point. Further, the time after an integer multiple of the time N from the ID change point estimated in the past is also estimated as the ID change point, and the midpoint of the plurality of ID change points is estimated as a more accurate ID change point. With such an estimation algorithm, an accurate ID change point can be gradually estimated.

(B2) In addition to the above, it is possible that the reception section where the ID has not changed and the time after an integer multiple of the time N are gradually included in the ID change point by estimating that the ID change point is not included. There are fewer sections of the ID, and an accurate ID change point can be estimated.

Ｎ By setting N to 0.5 seconds or less, it can be synchronized accurately.

設定 By setting N to 2 seconds or less, it is possible to synchronize without causing the user to feel a delay.

¡By setting N to 10 seconds or less, it is possible to synchronize while suppressing wasted ID.

FIG. 26 is a diagram illustrating an example of a transmission signal in the third embodiment.

In FIG. 26, waste of ID can be avoided by performing synchronization by time packets. A time packet is a packet that holds the time of transmission. When it is necessary to express a long time, the time packet is divided into a time packet 1 representing a fine time and a time packet 2 representing a rough time. For example, time packet 2 indicates the hour and minute of the time, and time packet 1 indicates only the second of the time. A packet indicating the time may be divided into three or more time packets. Since the coarse time is less necessary, the receiver can recognize the synchronization time quickly and accurately by transmitting more fine time packets than coarse time packets.

That is, in the present embodiment, the visible light signal includes the second information (hour packet 2) indicating the hour and minute of the time, and the first information (time packet 1) indicating the second of the time. The time when the visible light signal is transmitted from the transmitter 1800d is indicated. The receiver 1800a receives the second information and receives the first information more times than the number of times of receiving the second information.

Here, the synchronization time adjustment will be described below.

FIG. 27 is a diagram illustrating an example of a processing flow of the receiver 1800a according to the third embodiment.

Since it takes a certain amount of time for the audio or video to be played back after the signal is transmitted and processed by the receiver 1800a, it is possible to accurately reproduce the audio or video in anticipation of this processing time. Synchronous playback can be performed.

First, a processing delay time is designated for the receiver 1800a (step S1801). This may be stored in the processing program or specified by the user. When the user performs correction, it is possible to realize more accurate synchronization according to the individual receiver. This processing delay time can be synchronized more accurately by changing it depending on the receiver model, the temperature of the receiver, and the CPU usage rate.

The receiver 1800a determines whether or not a time packet has been received or whether or not an ID associated for voice synchronization has been received (step S1802). Here, when the receiver 1800a determines that it has been received (Y in step S1802), it further determines whether there is an image waiting for processing (step S1804). If it is determined that there is an image waiting for processing (Y in step S1804), the receiver 1800a discards the image waiting for processing or delays processing of the image waiting for processing to receive from the latest acquired image. Is performed (step S1805). Thereby, it is possible to avoid an unexpected delay due to the amount of waiting for processing.

The receiver 1800a measures the position in the image where the visible light signal (specifically the bright line) is located (step S1806). In other words, by measuring the position in the direction perpendicular to the exposure line from the first exposure line in the image sensor, the time difference (delay time in the image) from the image acquisition start time to the signal reception time is obtained. Can be calculated.

The receiver 1800a can accurately perform synchronized reproduction by reproducing the sound or moving image at the time obtained by adding the processing delay time and the in-image delay time to the recognized synchronization time (step S1807).

On the other hand, if it is determined in step S1802 that the receiver 1800a has not received the time packet or the voice synchronization ID, the receiver 1800a receives a signal from the image obtained by imaging (step S1803).

FIG. 28 is a diagram illustrating an example of a user interface of the receiver 1800a according to the third embodiment.

As shown in FIG. 28A, the user can adjust the processing delay time described above by pressing one of the buttons Bt1 to Bt4 displayed on the receiver 1800a. Alternatively, the processing delay time may be set by a swipe operation as shown in FIG. Thereby, synchronous reproduction can be performed more accurately based on the user's sense.

Here, the earphone-only playback will be described below.

FIG. 29 is a diagram illustrating an example of a processing flow of the receiver 1800a according to the third embodiment.

The earphone-only playback shown by this processing flow enables audio playback without disturbing the surroundings.

The receiver 1800a checks whether or not the setting limited to the earphone is performed (step S1811). When the setting limited to the earphone is performed, for example, the setting limited to the earphone is set in the receiver 1800a. Alternatively, settings that are limited to earphones are made in the received signal (visible light signal). Alternatively, it is recorded in the server or the receiver 1800a in association with the received signal that it is limited to the earphone.

When it is confirmed that the receiver 1800a is limited to the earphone (Y in step S1811), it is determined whether or not the earphone is connected to the receiver 1800a (step S1813).

When the receiver 1800a confirms that the earphone is not limited (N in Step S1811) or determines that the earphone is connected (Y in Step S1813), the receiver 1800a reproduces the sound (Step S1812). When playing back audio, the receiver 1800a adjusts the volume so that the volume is within the set range. This setting range is set similarly to the setting limited to the earphone.

If the receiver 1800a determines that the earphone is not connected (N in step S1813), the receiver 1800a performs a notification prompting the user to connect the earphone (step S1814). This notification is performed by, for example, screen display, audio output, or vibration.

In addition, when the setting for prohibiting forced audio reproduction is not set, the receiver 1800a prepares an interface for forced reproduction and determines whether or not the user has performed an operation of forced reproduction ( Step S1815). If it is determined that the forced playback operation has been performed (Y in step S1815), the receiver 1800a plays back the audio even when the earphone is not connected (step S1812).

On the other hand, if it is determined that the forced regeneration operation is not performed (N in step S1815), the receiver 1800a retains the audio data received in advance and the analyzed synchronization time so that the earphone is connected. Quickly synchronize audio playback.

FIG. 30 is a diagram illustrating another example of the processing flow of the receiver 1800a according to the third embodiment.

First, the receiver 1800a receives an ID from the transmitter 1800d (step S1821). That is, the receiver 1800a receives a visible light signal indicating the ID of the transmitter 1800d or the ID of the content displayed on the transmitter 1800d.

Next, the receiver 1800a downloads information (content) associated with the received ID from the server (step S1822). Alternatively, the receiver 1800a reads out the information from the data holding unit in the receiver 1800a. Hereinafter, this information is referred to as related information.

Next, the receiver 1800a determines whether or not the synchronous reproduction flag included in the related information indicates ON (step S1823). If it is determined that the synchronous reproduction flag does not indicate ON (N in step S1823), the receiver 1800a outputs the content indicated by the related information (step S1824). That is, when the content is an image, the receiver 1800a displays an image, and when the content is audio, the receiver 1800a outputs audio.

On the other hand, when receiver 1800a determines that the synchronous reproduction flag indicates ON (Y in step S1823), is the time adjustment mode included in the related information set to the transmitter reference mode? Then, it is determined whether or not the absolute time mode is set (step S1825). If it is determined that the absolute time mode is set, the receiver 1800a determines whether or not the last time adjustment has been performed within a certain time from the current time (step S1826). The time adjustment at this time is processing for obtaining time information by a predetermined method and using the time information to adjust the time of a clock provided in the receiver 1800a to the absolute time of the reference clock. The predetermined method is, for example, a method using a GPS (Global Positioning System) radio wave or an NTP (Network Time Protocol) radio wave. Note that the current time described above may be a time when the receiver 1800a, which is a terminal device, receives a visible light signal.

If the receiver 1800a determines that the last time adjustment has been performed within a certain time (Y in step S1826), the receiver 1800a outputs the related information based on the time of the clock of the receiver 1800a, and is displayed on the transmitter 1800d. Content and related information are synchronized (step S1827). When the content indicated by the related information is, for example, a moving image, the receiver 1800a displays the moving image so as to be synchronized with the content displayed on the transmitter 1800d. When the content indicated by the related information is, for example, audio, the receiver 1800a outputs the audio so as to be synchronized with the content displayed on the transmitter 1800d. For example, when the related information indicates sound, the related information includes each frame constituting the sound, and these frames are time stamped. The receiver 1800a outputs a sound synchronized with the content of the transmitter 1800d by playing back a frame with a type stamp corresponding to the time of its own clock.

If the receiver 1800a determines that the last time adjustment has not been performed within a certain time (N in step S1826), the receiver 1800a attempts to obtain the time information by a predetermined method, and whether or not the time information has been obtained. Is determined (step S1828). If it is determined that the time information has been obtained (Y in step S1828), the receiver 1800a updates the time of the clock of the receiver 1800a using the time information (step S1829). Then, the receiver 1800a executes the process of step S1827 described above.

If it is determined in step S1825 that the time adjustment mode is the transmitter reference mode, or if it is determined in step S1828 that time information could not be obtained (N in step S1828), the receiver 1800a The time information is acquired from the transmitter 1800d (step S1830). That is, the receiver 1800a acquires time information that is a synchronization signal from the transmitter 1800d through visible light communication. For example, the synchronization signals are time packet 1 and time packet 2 shown in FIG. Alternatively, the receiver 1800a acquires time information from the transmitter 1800d by radio waves such as Bluetooth (registered trademark) or Wi-Fi. Then, the receiver 1800a executes the processes of steps S1829 and S1827 described above.

In the present embodiment, as in steps S1829 and S1830, processing (time adjustment) is performed for synchronization between the clock of the terminal device that is the receiver 1800a and the reference clock by GPS radio waves or NTP radio waves. The time of the terminal device, the time of the terminal device, and the time of the transmitter according to the time indicated by the visible light signal transmitted from the transmitter 1800d. Synchronize between. Accordingly, the terminal device can reproduce the content (moving image or sound) at the timing synchronized with the transmitter-side content reproduced by the transmitter 1800d.

FIG. 31A is a diagram for explaining a specific method of synchronized playback in the third embodiment. There are methods a to e shown in FIG.

(Method a)
In the method a, the transmitter 1800d outputs a visible light signal indicating the content ID and the content playback time by changing the luminance of the display, as in the above embodiments. The content playback time is the playback time of data that is part of the content that is being played back by the transmitter 1800d when the content ID is transmitted from the transmitter 1800d. The data is a picture or a sequence constituting the moving image if the content is a moving image, or a frame constituting the sound if the content is sound. The playback time indicates, for example, the playback time from the beginning of the content as the time. If the content is a moving image, the playback time is included in the content as a PTS (Presentation Time Stamp). That is, the content includes the reproduction time (display time) of the data for each data constituting the content.

The receiver 1800a receives the visible light signal by photographing the transmitter 1800d as in the above embodiments. Then, the receiver 1800a transmits a request signal including the content ID indicated by the visible light signal to the server 1800f. The server 1800f receives the request signal, and transmits the content associated with the content ID included in the request signal to the receiver 1800a.

When the receiver 1800a receives the content, the receiver 1800a plays the content from the time of (content playback time + elapsed time since ID reception). The elapsed time from the reception of the ID is an elapsed time from when the content ID is received by the receiver 1800a.

(Method b)
In the method b, the transmitter 1800d outputs a visible light signal indicating the content ID and the content playback time by changing the luminance of the display, as in the above embodiments. The receiver 1800a receives the visible light signal by photographing the transmitter 1800d as in the above embodiments. Then, the receiver 1800a transmits a request signal including the content ID indicated by the visible light signal and the content playback time to the server 1800f. The server 1800f receives the request signal, and transmits only a part of the content after the content playback time to the receiver 1800a among the content associated with the content ID included in the request signal.

When the receiver 1800a receives the part of the content, the receiver 1800a reproduces the part of the content from the time point (elapsed time since the reception of the ID).

(Method c)
In the method c, the transmitter 1800d outputs a visible light signal indicating the transmitter ID and the content reproduction time by changing the luminance of the display, as in the above embodiments. The transmitter ID is information for identifying the transmitter.

The receiver 1800a receives the visible light signal by photographing the transmitter 1800d as in the above embodiments. Then, the receiver 1800a transmits a request signal including the transmitter ID indicated by the visible light signal to the server 1800f.

The server 1800f holds, for each transmitter ID, a reproduction schedule that is a timetable of content reproduced by the transmitter with the transmitter ID. Further, the server 1800f includes a clock. When such a server 1800f receives the request signal, the content associated with the transmitter ID included in the request signal and the clock time (server time) of the server 1800f is the content being played back. Identify from the playback schedule. Then, the server 1800f transmits the content to the receiver 1800a.

When the receiver 1800a receives the content, the receiver 1800a plays the content from the time of (content playback time + elapsed time since ID reception).

(Method d)
In the method d, the transmitter 1800d outputs a visible light signal indicating the transmitter ID and the transmitter time by changing the luminance of the display as in the above embodiments. The transmitter time is a time indicated by a clock provided in the transmitter 1800d.

The receiver 1800a receives the visible light signal by photographing the transmitter 1800d as in the above embodiments. Then, the receiver 1800a transmits a request signal including the transmitter ID indicated by the visible light signal and the transmitter time to the server 1800f.

The server 1800f holds the above reproduction schedule. When such a server 1800f receives the request signal, the server 1800f identifies the content associated with the transmitter ID and the transmitter time included in the request signal as the content being reproduced from the reproduction schedule. Furthermore, the server 1800f specifies the content playback time from the transmitter time. That is, the server 1800f finds the playback start time of the specified content from the playback schedule, and specifies the time between the transmitter time and the playback start time as the content playback time. Then, the server 1800f transmits the content and the content playback time to the receiver 1800a.

Upon receiving the content and the content playback time, the receiver 1800a plays the content from the time of (content playback time + elapsed time since reception of ID).

Thus, in the present embodiment, the visible light signal indicates the time when the visible light signal is transmitted from the transmitter 1800d. Therefore, the receiver 1800a, which is a terminal device, can receive content associated with the time (transmitter time) at which the visible light signal is transmitted from the transmitter 1800d. For example, if the transmitter time is 5:43, content played back at 5:43 can be received.

In the present embodiment, the server 1800f has a plurality of contents each associated with a time. However, the content associated with the time indicated by the visible light signal may not exist in the server 1800f. In such a case, the receiver 1800a as the terminal device is closest to the time indicated by the visible light signal and is associated with the time after the time indicated by the visible light signal among the plurality of contents. Content may be received. Thereby, even if the content associated with the time indicated by the visible light signal does not exist in the server 1800f, it is possible to receive appropriate content from among the plurality of contents in the server 1800f.

In addition, the reproduction method according to the present embodiment includes a signal receiving step of receiving a visible light signal by a sensor of the receiver 1800a (terminal device) from a transmitter 1800d that transmits a visible light signal according to a change in luminance of the light source, and a receiver. 1800a transmits a request signal for requesting the content associated with the visible light signal to the server 1800f, the receiver 1800a receives the content from the server 1800f, and reproduces the content. A playback step. The visible light signal indicates a transmitter ID and a transmitter time. The transmitter ID is ID information. The transmitter time is the time indicated by the clock of the transmitter 1800d, and the time when the visible light signal is transmitted from the transmitter 1800d. In the content reception step, the receiver 1800a receives the content associated with the transmitter ID and the transmitter time indicated by the visible light signal. As a result, the receiver 1800a can reproduce appropriate content with respect to the transmitter ID and the transmitter time.

(Method e)
In the method e, the transmitter 1800d outputs a visible light signal indicating the transmitter ID by changing the luminance of the display as in the above embodiments.

The receiver 1800a receives the visible light signal by photographing the transmitter 1800d as in the above embodiments. Then, the receiver 1800a transmits a request signal including the transmitter ID indicated by the visible light signal to the server 1800f.

The server 1800f holds the above-described reproduction schedule and further includes a clock. When such a server 1800f receives the request signal, the server 1800f identifies the content associated with the transmitter ID and the server time included in the request signal from the reproduction schedule as content being reproduced. The server time is the time indicated by the clock of the server 1800f. Further, the server 1800f finds the reproduction start time of the specified content from the reproduction schedule table. Then, the server 1800f transmits the content and the content reproduction start time to the receiver 1800a.

When the receiver 1800a receives the content and the content playback start time, the receiver 1800a plays the content from the time of (receiver time-content playback start time). The receiver time is a time indicated by a clock provided in the receiver 1800a.

As described above, the reproduction method according to the present embodiment includes a signal receiving step of receiving a visible light signal by a sensor of the receiver 1800a (terminal device) from a transmitter 1800d that transmits a visible light signal due to a luminance change of the light source; The transmitting step of transmitting a request signal for requesting the content associated with the visible light signal from the receiver 1800a to the server 1800f, and the receiver 1800a include each time and data reproduced at each time A content receiving step of receiving content from the server 1800f and a playback step of playing back data corresponding to the time of the clock provided in the receiver 1800a among the content. Therefore, the receiver 1800a can appropriately reproduce the data in the content at the correct time indicated by the content without reproducing the data at the wrong time. Also, in the transmitter 1800d, if content related to the content (transmitter-side content) is reproduced, the receiver 1800a can reproduce the content in synchronization with the transmitter-side content appropriately. .

Note that even in the above methods c to e, like the method b, the server 1800f may transmit only a part of the content after the content playback time to the receiver 1800a.

In the above methods a to e, the receiver 1800a transmits a request signal to the server 1800f and receives necessary data from the server 1800f. However, the data in the server 1800f is transmitted in advance without performing such transmission / reception. You may keep it.

FIG. 31B is a block diagram showing the configuration of a playback apparatus that performs synchronized playback by the method e described above.

The playback device B10 is a receiver 1800a or a terminal device that performs synchronous playback by the method e described above, and includes a sensor B11, a request signal transmission unit B12, a content reception unit B13, a clock B14, and a playback unit B15. I have.

Sensor B11 is, for example, an image sensor, and receives the visible light signal from a transmitter 1800d that transmits a visible light signal according to a change in luminance of the light source. The request signal transmission unit B12 transmits a request signal for requesting content associated with the visible light signal to the server 1800f. The content receiving unit B13 receives content including each time and data reproduced at each time from the server 1800f. The reproduction unit B15 reproduces data corresponding to the time of the clock B14 in the content.

FIG. 31C is a flowchart showing the processing operation of the terminal device that performs synchronous reproduction by the method e described above.

The playback device B10 is a receiver 1800a or a terminal device that performs synchronized playback by the method e described above, and executes each process of steps SB11 to SB15.

In step SB11, the visible light signal is received from the transmitter 1800d that transmits the visible light signal according to the luminance change of the light source. In step SB12, a request signal for requesting content associated with the visible light signal is transmitted to server 1800f. In step SB13, content including each time and data reproduced at each time is received from server 1800f. In step SB15, data corresponding to the time of the clock B14 is reproduced from the content.

As described above, in the playback device B10 and the playback method in the present embodiment, the data in the content can be appropriately played back at the correct time indicated by the content without being played back at the wrong time.

In the present embodiment, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software that realizes the playback apparatus B10 and the like of the present embodiment is a program that causes a computer to execute each step included in the flowchart shown in FIG. 31C.

FIG. 32 is a diagram for explaining preparations for synchronized playback in the third embodiment.

The receiver 1800a adjusts the time of the clock provided in the receiver 1800a to the time of the reference clock in order to perform synchronized playback. For this time adjustment, the receiver 1800a performs the following processes (1) to (5).

(1) The receiver 1800a receives a signal. This signal may be a visible light signal transmitted by a change in luminance of the display of the transmitter 1800d, or a radio wave signal based on Wi-Fi or Bluetooth (registered trademark) from a wireless device. Alternatively, the receiver 1800a acquires position information indicating the position of the receiver 1800a by, for example, GPS instead of receiving such a signal. Then, the receiver 1800a recognizes that the receiver 1800a has entered a predetermined place or building based on the position information.

(2) When the receiver 1800a receives the above signal or recognizes that it has entered a predetermined location, it receives a request signal for requesting data (related information) associated with the signal or location. It transmits to the server (visible light ID resolution server) 1800f.

(3) The server 1800f transmits the above-described data and a time adjustment request for causing the receiver 1800a to adjust the time to the receiver 1800a.

(4) When receiving the data and the time adjustment request, the receiver 1800a transmits the time adjustment request to the GPS time server, the NTP server, or the base station of the telecommunications carrier (carrier).

(5) Upon receiving the time adjustment request, the server or the base station transmits time data (time information) indicating the current time (reference clock time or absolute time) to the receiver 1800a. The receiver 1800a adjusts the time by adjusting the time of the clock provided to the receiver 1800a to the current time indicated by the time data.

As described above, in this embodiment, a GPS (Global Positioning System) radio wave or an NTP (Network Time Protocol) radio wave is used between the clock provided in the receiver 1800a (terminal device) and the reference clock. Synchronized. Therefore, the receiver 1800a can reproduce the data corresponding to the time at an appropriate time according to the reference clock.

FIG. 33 is a diagram illustrating an example of application of the receiver 1800a in the third embodiment.

The receiver 1800a is configured as a smartphone as described above, and is used by being held by a holder 1810 formed of, for example, a translucent resin or glass member. The holder 1810 includes a back plate portion 1810a and a locking portion 1810b provided upright on the back plate portion 1810a. The receiver 1800a is inserted between the back plate portion 1810a and the locking portion 1810b so as to be along the back plate portion 1810a.

FIG. 34A is a front view of receiver 1800a held by holder 1810 in the third embodiment.

The receiver 1800a is held by the holder 1810 in the inserted state as described above. At this time, the locking portion 1810b locks with the lower portion of the receiver 1800a and sandwiches the lower portion with the back plate portion 1810a. In addition, the back surface of the receiver 1800a faces the back plate portion 1810a, and the display 1801 of the receiver 1800a is exposed.

FIG. 34B is a rear view of receiver 1800a held by holder 1810 in the third embodiment.

Further, a through hole 1811 is formed in the back plate portion 1810a, and a variable filter 1812 is attached in the vicinity of the through hole 1811. When receiver 1800a is held by holder 1810, camera 1802 of receiver 1800a is exposed through back hole 1811 from back plate portion 1810a. The flashlight 1803 of the receiver 1800a faces the variable filter 1812.

The variable filter 1812 is formed in a disk shape, for example, and has three color filters (a red filter, a yellow filter, and a green filter) each having a fan shape and the same size. The variable filter 1812 is attached to the back plate portion 1810a so as to be rotatable about the center of the variable filter 1812. The red filter is a filter having red translucency, the yellow filter is a filter having yellow translucency, and the green filter is a filter having green translucency.

Therefore, the variable filter 1812 is rotated, and, for example, the red filter is disposed at a position facing the flashlight 1803a. In this case, the light emitted from the flashlight 1803a is diffused inside the holder 1810 as red light by passing through the red filter. As a result, substantially the entire holder 1810 emits red light.

Similarly, the variable filter 1812 is rotated and, for example, the yellow filter is disposed at a position facing the flashlight 1803a. In this case, the light emitted from the flashlight 1803a is diffused inside the holder 1810 as yellow light by passing through the yellow filter. As a result, substantially the entire holder 1810 emits yellow light.

Similarly, the variable filter 1812 is rotated so that, for example, the green filter is disposed at a position facing the flashlight 1803a. In this case, the light emitted from the flashlight 1803a is diffused inside the holder 1810 as green light by passing through the green filter. As a result, substantially the entire holder 1810 emits green light.

That is, the holder 1810 lights in red, yellow or green like a penlight.

FIG. 35 is a diagram for describing a use case of the receiver 1800a held by the holder 1810 in the third embodiment.

For example, a receiver with a holder that is a receiver 1800a held by a holder 1810 is used in an amusement park or the like. That is, the plurality of receivers with holders that are directed to the float moving in the amusement park blink in synchronization with the music flowing from the float. That is, the float is configured as a transmitter in each of the above embodiments, and transmits a visible light signal by a change in luminance of a light source attached to the float. For example, the float transmits a visible light signal indicating the ID of the float. And the receiver with a holder receives the visible light signal, ie, ID, by imaging | photography with the camera 1802 of the receiver 1800a similarly to said each embodiment. The receiver 1800a that has received the ID acquires a program associated with the ID from, for example, a server. This program includes instructions for turning on the flashlight 1803 of the receiver 1800a at each predetermined time. Each predetermined time is set in accordance with the music flowing from the float (so as to be synchronized). Then, the receiver 1800a blinks the flashlight 1803a according to the program.

Thereby, the holder 1810 of each receiver 1800a that has received the ID repeats lighting at the same timing according to the music flowing from the float of the ID.

Here, each receiver 1800a blinks the flashlight 1803 in accordance with a set color filter (hereinafter referred to as a setting filter). The setting filter is a color filter that faces the flashlight 1803 of the receiver 1800a. Each receiver 1800a recognizes the current setting filter based on an operation by the user. Alternatively, each receiver 1800a recognizes the current setting filter based on the color of an image obtained by photographing with the camera 1802.

That is, among the plurality of receivers 1800a that have received the ID, only the holders 1810 of the plurality of receivers 1800a that recognize that the setting filter is a red filter are lit simultaneously at a predetermined time. At the next time, only the holders 1810 of the plurality of receivers 1800a that recognize that the setting filter is a green filter are lit simultaneously. Further, at the next time, only the holders 1810 of the plurality of receivers 1800a that recognize that the setting filter is a yellow filter are lit simultaneously.

As described above, the receiver 1800a held in the holder 1810 is synchronized with the float music and the receiver 1800a held in the other holder 1810 in the same manner as the synchronous playback shown in FIGS. Then, the flashlight 1803, that is, the holder 1810 is blinked.

FIG. 36 is a flowchart showing the processing operation of the receiver 1800a held by the holder 1810 in the third embodiment.

The receiver 1800a receives the float ID indicated by the visible light signal from the float (step S1831). Next, the receiver 1800a acquires a program associated with the ID from the server (step S1832). Next, the receiver 1800a executes the program to turn on the flashlight 1803 at each predetermined time according to the setting filter (step S1833).

Here, the receiver 1800a may cause the display 1801 to display an image corresponding to the received ID or the acquired program.

FIG. 37 is a diagram illustrating an example of an image displayed by the receiver 1800a according to the third embodiment.

For example, when the receiver 1800a receives an ID from a Santa Claus float, the receiver 1800a displays a Santa Claus image as shown in FIG. Further, as shown in FIG. 37B, the receiver 1800a may change the background color of the Santa Claus image to the color of the setting filter simultaneously with the lighting of the flashlight 1803. For example, when the color of the setting filter is red, the holder 1810 is lit red by turning on the flashlight 1803, and at the same time, a Santa Claus image having a red background color is displayed on the display 1801. That is, the blinking of the holder 1810 and the display on the display 1801 are synchronized.

FIG. 38 is a diagram showing another example of the holder in the third embodiment.

The holder 1820 is configured in the same manner as the holder 1810 described above, but does not include the through hole 1811 and the variable filter 1812. Such a holder 1820 holds the receiver 1800a in a state where the display 1801 of the receiver 1800a is directed to the back plate portion 1820a. In this case, the receiver 1800a causes the display 1801 to emit light instead of the flashlight 1803. As a result, light from the display 1801 is diffused over substantially the entire holder 1820. Therefore, when the receiver 1800a causes the display 1801 to emit light with red light according to the above-described program, the holder 1820 is lit red. Similarly, when the receiver 1800a causes the display 1801 to emit light with yellow light according to the above-described program, the holder 1820 is lit in yellow. When the receiver 1800a causes the display 1801 to emit light with green light according to the above-described program, the holder 1820 lights up in green. If such a holder 1820 is used, the setting of the variable filter 1812 can be omitted.

(Visible light signal)
FIG. 39A to FIG. 39D are diagrams illustrating examples of visible light signals in the third embodiment.

Similarly to the above, the transmitter generates a 4PPM visible light signal and changes the luminance according to the visible light signal, for example, as shown in FIG. 39A. Specifically, the transmitter allocates 4 slots to one signal unit, and generates a visible light signal composed of a plurality of signal units. The signal unit indicates High (H) or Low (L) for each slot. The transmitter emits light brightly in the H slot and emits light darkly or extinguishes in the L slot. For example, one slot is a period corresponding to a time of 1/9600 seconds.

Further, for example, as shown in FIG. 39B, the transmitter may generate a visible light signal in which the number of slots allocated to one signal unit is variable. In this case, the signal unit includes a signal indicating H in one or more consecutive slots and a signal indicating L in one slot following the H signal. Since the number of slots of H is variable, the total number of slots in the signal unit is variable. For example, as shown in FIG. 39B, the transmitter generates a visible light signal including these signal units in the order of a signal unit of 3 slots, a signal unit of 4 slots, and a signal unit of 6 slots. Also in this case, the transmitter emits light brightly in the H slot and emits light darkly or extinguishes in the L slot.

Further, for example, as shown in FIG. 39C, the transmitter may allocate an arbitrary period (signal unit period) to one signal unit without allocating a plurality of slots to one signal unit. The signal unit period includes an H period and an L period following the H period. The period of H is adjusted according to the signal before modulation. The period L may be fixed and may be a period corresponding to the slot. The H period and the L period are, for example, periods of 100 μs or more. For example, as shown in FIG. 39C, the transmitter transmits a visible light signal including signal units in the order of a signal unit having a signal unit period of 210 μs, a signal unit having a signal unit period of 220 μs, and a signal unit having a signal unit period of 230 μs. Send. In this case as well, the transmitter emits light brightly during the H period and emits light darkly or extinguishes during the L period.

Further, for example, as shown in FIG. 39D, the transmitter may generate a signal indicating L and H alternately as a visible light signal. In this case, the L period and the H period in the visible light signal are adjusted according to the signals before modulation. For example, as shown in FIG. 39D, the transmitter indicates H for a period of 100 μs, then indicates L for a period of 120 μs, then indicates H for a period of 110 μs, and further indicates L for a period of 200 μs. A visible light signal is transmitted. In this case as well, the transmitter emits light brightly during the H period and emits light darkly or extinguishes during the L period.

FIG. 40 is a diagram showing a configuration of a visible light signal in the third embodiment.

The visible light signal includes, for example, a signal 1, a brightness adjustment signal corresponding to the signal 1, a signal 2, and a brightness adjustment signal corresponding to the signal 2. When the transmitter generates the signal 1 and the signal 2 by modulating the signals before modulation, the transmitter generates a brightness adjustment signal for the signals and generates the above-described visible light signal.

The brightness adjustment signal corresponding to signal 1 is a signal that compensates for increase / decrease in brightness due to a luminance change according to signal 1. The brightness adjustment signal corresponding to the signal 2 is a signal that compensates for increase / decrease in brightness due to a luminance change according to the signal 2. Here, the brightness B1 is expressed by the luminance change according to the signal 1 and the brightness adjustment signal of the signal 1, and the brightness change according to the signal 2 and the brightness adjustment signal of the signal 2 Brightness B2 is expressed. The transmitter in the present embodiment generates the brightness adjustment signals of signal 1 and signal 2 as part of the visible light signal so that the brightness B1 and brightness B2 are equal. Thereby, the brightness is kept constant and flicker can be suppressed.

In addition, when the transmitter 1 generates the signal 1, the transmitter 1 generates the signal 1 including the data 1, the preamble (header) following the data 1, and the data 1 following the preamble. Here, the preamble is a signal corresponding to data 1 arranged before and after the preamble. For example, this preamble is a signal serving as an identifier for reading data 1. Thus, since the signal 1 is composed of the two data 1 and the preamble arranged between them, even if the receiver reads the visible light signal from the middle of the preceding data 1, Data 1 (ie, signal 1) can be correctly demodulated.

The reproduction method according to an aspect of the present invention includes a signal receiving step of receiving the visible light signal by a sensor of a terminal device from a transmitter that transmits a visible light signal according to a luminance change of a light source, and the visible light signal from the terminal device. A transmission step of transmitting a request signal for requesting the content associated with the optical signal to the server, and the terminal device transmits content including each time and data reproduced at each time from the server. A content receiving step of receiving, and a playback step of playing back data corresponding to a time of a clock provided in the terminal device among the content.

Thus, as shown in FIG. 31C, content including each time and data reproduced at each time is received by the terminal device, and data corresponding to the clock time of the terminal device is reproduced. Therefore, the terminal device can appropriately reproduce the data in the content at the correct time indicated by the content without reproducing the data at the wrong time. Specifically, as in the method e of FIG. 31A, the receiver as the terminal device reproduces the content from the time of (receiver time−content reproduction start time). The data corresponding to the clock time of the terminal device described above is data at the time of (receiver time−content reproduction start time) in the content. Also, in the transmitter, if content related to the content (transmitter-side content) is being played back, the terminal device can play back the content appropriately synchronized with the transmitter-side content. The content is sound or image.

Further, the clock provided in the terminal device and the reference clock may be synchronized with each other by GPS (Global Positioning System) radio waves or NTP (Network Time Protocol) radio waves.

As a result, as shown in FIGS. 30 and 32, since the clock of the terminal device (receiver) is synchronized with the reference clock, the time corresponds to the appropriate time according to the reference clock. Can be played back.

Further, the visible light signal may indicate a time when the visible light signal is transmitted from the transmitter.

Thereby, as shown in the method d in FIG. 31A, the terminal device (receiver) can receive the content associated with the time (transmitter time) at which the visible light signal is transmitted from the transmitter. For example, if the transmitter time is 5:43, content played back at 5:43 can be received.

Further, in the reproduction method, the time at which the process for synchronizing the clock of the terminal device and the reference clock is performed by the GPS radio wave or the NTP radio wave is determined by the terminal device as the visible signal. If it is before a predetermined time from the time when the optical signal is received, synchronization is performed between the clock of the terminal device and the clock of the transmitter according to the time indicated by the visible light signal transmitted from the transmitter. It may be taken.

For example, if a predetermined time elapses after the processing for synchronizing the clock of the terminal device and the reference clock is performed, the synchronization may not be properly maintained. In such a case, there is a possibility that the terminal device cannot reproduce the content at a time synchronized with the transmitter-side content reproduced by the transmitter. Therefore, in the playback method according to one aspect of the present invention, as shown in steps S1829 and S1830 of FIG. 30, when a predetermined time has elapsed, the clock of the terminal device (receiver) and the clock of the transmitter are set. Synchronized. Therefore, the terminal device can reproduce the content at a time synchronized with the transmitter-side content reproduced by the transmitter.

Further, the server has a plurality of contents each associated with a time, and in the contents receiving step, when the contents associated with the time indicated by the visible light signal does not exist in the server, Among the plurality of contents, content that is closest to the time indicated by the visible light signal and that is associated with a time after the time indicated by the visible light signal may be received.

As a result, as shown in the method d in FIG. 31A, even if the content associated with the time indicated by the visible light signal does not exist in the server, the appropriate content is received from the plurality of contents in the server. be able to.

In addition, a signal receiving step of receiving the visible light signal by a sensor of a terminal device from a transmitter that transmits a visible light signal due to a luminance change of the light source, and a content associated with the visible light signal from the terminal device. A transmission step of transmitting a request signal for requesting to the server, a content reception step in which the terminal device receives the content from the server, and a reproduction step of reproducing the content, wherein the visible light signal is: ID information and a time at which the visible light signal is transmitted from the transmitter are indicated. In the content receiving step, the ID information indicated by the visible light signal and the content associated with the time are received. Good.

Thus, as in method d of FIG. 31A, the visible light signal is associated with the time (transmitter time) at which the visible light signal is transmitted from the transmitter among the plurality of contents associated with the ID information (transmitter ID). The received content is received and played back. Therefore, it is possible to reproduce content appropriate for the transmitter ID and transmitter time.

The visible light signal includes second information indicating the hour and minute of the time and first information indicating the second of the time, so that the visible light signal is transmitted from the transmitter. In the signal receiving step, the second information may be received and the first information may be received more times than the number of times the second information is received.

Thereby, for example, when notifying the terminal device of the time at which each packet included in the visible light signal is transmitted in seconds, the packet indicating the current time expressed using all of the hour, minute, and second Can be saved in every second. That is, as shown in FIG. 26, if the hour and minute of the time when the packet is transmitted is not updated from the time and minute indicated in the previously transmitted packet, the packet indicating only the second (time packet 1 Only the first information is required to be transmitted. Therefore, by reducing the second information that is the packet indicating the hour and the minute (time packet 2) than the first information that is the packet indicating the second (time packet 1) transmitted by the transmitter, Transmission of packets containing redundant contents can be suppressed.

(Embodiment 4)
In the present embodiment, a display method for realizing AR (Augmented Reality) using an optical ID will be described.

FIG. 41 is a diagram illustrating an example in which the receiver according to the present embodiment displays an AR image.

The receiver 200 according to the present embodiment is a receiver including the image sensor and the display 201 according to any one of the first to third embodiments, and is configured as a smartphone, for example. Such a receiver 200 acquires the above-described captured display image Pa, which is the normal captured image, and the above-described decoding image, which is the visible light communication image or the bright line image, by capturing the subject with the image sensor.

Specifically, the image sensor of the receiver 200 images the transmitter 100 configured as a station name sign. The transmitter 100 is the transmitter according to any one of the first to third embodiments, and includes one or a plurality of light emitting elements (for example, LEDs). The transmitter 100 changes in luminance by blinking one or more light emitting elements, and transmits an optical ID (light identification information) by the change in luminance. This light ID is the above-mentioned visible light signal.

The receiver 200 captures the transmitter 100 with the normal exposure time, thereby acquiring the captured display image Pa projected by the transmitter 100, and the transmitter 100 with a communication exposure time shorter than the normal exposure time. A decoding image is acquired by imaging. The normal exposure time is the exposure time in the above-described normal photographing mode, and the communication exposure time is the exposure time in the above-described visible light communication mode.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P1 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pa as a target area. For example, the receiver 200 recognizes an area in which a station name sign that is the transmitter 100 is displayed as a target area. Then, the receiver 200 superimposes the AR image P1 on the target area, and displays the captured display image Pa on which the AR image P1 is superimposed on the display 201. For example, when “Kyoto Station” is written in Japanese as a station name in the station name that is the transmitter 100, the receiver 200 describes the AR image P1 in which the station name is written in English, that is, “Kyoto Station”. Obtained AR image P1. In this case, since the AR image P1 is superimposed on the target area of the captured display image Pa, the captured display image Pa can be displayed so that a station name with a station name written in English actually exists. As a result, a user who can understand English can easily understand the station name described in the station name sign that is the transmitter 100 by looking at the captured display image Pa even if the user cannot read Japanese.

For example, the recognition information may be an image to be recognized (for example, an image of the above-described station name sign), or may be a feature point and a feature amount of the image. The feature points and feature quantities are obtained by, for example, SIFT (Scale-invariant feature transform), SURF (Speed-Uploaded Robust Feature), ORB (Oriented-BREF), AKAZE (Accelerated) image, etc. Alternatively, the recognition information may be a white square image similar to the image to be recognized, and may further indicate the aspect ratio (aspect ratio) of the square. Alternatively, the identification information may be random dots that appear in the recognition target image. Furthermore, the recognition information may indicate a direction based on a predetermined direction, such as the above-described white square or random dot. The predetermined direction is, for example, the direction of gravity.

The receiver 200 recognizes an area corresponding to such recognition information as a target area from the captured display image Pa. Specifically, if the recognition information is an image, the receiver 200 recognizes a region similar to the image that is the recognition information as a target region. If the recognition information is a feature point and a feature amount obtained by image processing, the receiver 200 performs feature point detection and feature amount extraction by performing the image processing on the captured display image Pa. . Then, the receiver 200 recognizes, in the captured display image Pa, a region having feature points and feature amounts that are similar to the feature points and feature amounts that are recognition information as target regions. If the recognition information indicates a white square and its direction, the receiver 200 first detects the direction of gravity using an acceleration sensor provided in the receiver 200. Then, the receiver 200 recognizes, as a target area, an area similar to a white square directed in the direction indicated by the recognition information from the captured display image Pa arranged with reference to the direction of gravity.

Here, the recognition information may include reference information for specifying a reference area in the captured display image Pa and target information indicating a relative position of the target area with respect to the reference area. The reference information is an image to be recognized, a feature point and a feature amount, a white square image, or a random dot as described above. In this case, when recognizing the target area, the receiver 200 first specifies the reference area from the captured display image Pa based on the reference information. And the receiver 200 recognizes the area | region in the relative position shown by object information among the picked-up display images Pa as a reference | standard by using the position of a reference | standard area | region as a reference | standard. The target information may indicate that the target area is in the same position as the reference area. As described above, since the recognition information includes the reference information and the target information, the target region can be recognized in a wide range. Further, the server can freely set the location where the AR image is superimposed and can be instructed to the receiver 200.

Further, the reference information may indicate that the reference area in the captured display image Pa is an area where the display of the captured display image is displayed. In this case, if the transmitter 100 is configured as a display such as a television, for example, the target area can be recognized with reference to the area where the display is displayed.

In other words, the receiver 200 in the present embodiment specifies the reference image and the image recognition method based on the light ID. The image recognition method is a method for recognizing the captured display image Pa, for example, geometric feature extraction, spectral feature extraction, texture feature extraction, or the like. The reference image is data indicating a reference feature amount. The feature amount is, for example, the feature amount of the white outer frame of the image, and specifically, may be data expressing the feature of the image as a vector. The receiver 200 extracts a feature amount from the captured display image Pa in accordance with an image recognition method, and compares the feature amount with the feature amount of the reference image, so that the above-described reference region or target region is extracted from the captured display image Pa. Find out.

Also, the image recognition method may include, for example, a location use method, a marker use method, and a markerless method. The location utilization method is a method utilizing GPS position information (that is, the position of the receiver 200), and the target area is recognized from the captured display image Pa based on the position information. The marker utilization method is a method of using a marker composed of white and black graphics such as a two-dimensional barcode as a target specifying mark. That is, in this marker usage method, the target region is recognized based on the marker displayed in the captured display image Pa. In the markerless method, a feature point or feature amount is extracted from the captured display image Pa by image analysis on the captured display image Pa, and the position and region of the target are specified based on the extracted feature point or feature amount. Is the method. That is, when the image recognition method is a markerless method, the image recognition method is the above-described geometric feature amount extraction, spectral feature amount extraction, texture feature amount extraction, or the like.

Such a receiver 200 receives an optical ID from the transmitter 100 and acquires a reference image and an image recognition method associated with the optical ID (hereinafter referred to as a received optical ID) from the server, thereby obtaining the reference. Images and image recognition methods may be specified. That is, the server stores a plurality of sets including the reference image and the image recognition method, and each of the plurality of sets is associated with a different light ID. Thus, one set associated with the received light ID can be identified from among a plurality of sets stored in the server. Therefore, the speed of image processing for superimposing the AR image can be improved. Further, the receiver 200 may acquire a reference image associated with the received light ID by inquiring of the server, and the received light ID may be obtained from a plurality of reference images held by the receiver 200 in advance. You may acquire the reference | standard image matched with.

Further, the server may hold the relative position information associated with each light ID together with the reference image, the image recognition method, and the AR image for each light ID. The relative position information is information indicating the relative positional relationship between the reference area and the target area, for example. Thereby, when the receiver 200 sends an inquiry to the server by receiving the received light ID, the receiver 200 acquires the reference image, the image recognition method, the AR image, and the relative position information associated with the received light ID. In this case, the receiver 200 specifies the above-described reference region from the captured display image Pa based on the reference image and the image recognition method. Then, the receiver 200 recognizes the region in the direction and the distance indicated by the relative position information from the position of the reference region as the target region, and superimposes the AR image on the target region. Further, if there is no relative position information, the receiver 200 may recognize the above-described reference area as a target area and superimpose an AR image on the reference area. That is, the receiver 200 may store a program for displaying an AR image based on the reference image in advance, instead of acquiring the relative position information, and display the AR image in a white frame that is a reference region, for example. . In this case, relative position information is not necessary.

There are the following four variations (1) to (4) for holding or acquiring the reference image, relative position information, AR image, and image recognition method.

(1) The server holds a plurality of sets including a reference image, relative position information, an AR image, and an image recognition method. The receiver 200 acquires one set associated with the received light ID from these sets.

(2) The server holds a plurality of sets including a reference image and an AR image. The receiver 200 uses a predetermined relative position information and an image recognition method, and acquires one set associated with the received light ID from the set. Alternatively, the receiver 200 may hold a plurality of sets including the relative position information and the image recognition method in advance, and select one set associated with the received light ID from the plurality of sets. In this case, the receiver 200 may inquire by transmitting the received light ID to the server, and acquire relative position information corresponding to the received light ID and information for specifying the image recognition method from the server. Then, the receiver 200 selects one set based on the information acquired from the server from among a plurality of sets each having the relative position information and the image recognition method. Alternatively, the receiver 200 selects one set associated with the received light ID from a plurality of sets each including the relative position information and the image recognition method stored in advance without inquiring of the server. May be.

(3) The receiver 200 holds a plurality of sets including a reference image, relative position information, an AR image, and an image recognition method, and selects one set from these sets. Similarly to (2) above, the receiver 200 may select one set by making an inquiry to the server, or may select one set associated with the receiver optical ID.

(4) The receiver 200 holds a plurality of sets including the reference image and the AR image, and selects one set associated with the received light ID. The receiver 200 uses a predetermined image recognition method and relative position information.

FIG. 42 is a diagram showing an example of the display system in the present embodiment.

The display system in the present embodiment includes, for example, the transmitter 100, the receiver 200, and the server 300, which are the above-described station names.

The receiver 200 first receives an optical ID from the transmitter 100 in order to display the captured display image on which the AR image is superimposed as described above. Next, the receiver 200 transmits the optical ID to the server 300.

The server 300 holds an AR image and recognition information associated with each light ID for each light ID. Therefore, when the server 300 receives the optical ID from the receiver 200, the server 300 selects the AR image and the recognition information associated with the received optical ID, and sends the selected AR image and the recognition information to the receiver 200. Send. Thereby, the receiver 200 receives the AR image and the recognition information transmitted from the server 300, and displays the captured display image on which the AR image is superimposed.

FIG. 43 is a diagram showing another example of the display system in the present embodiment.

The display system according to the present embodiment includes, for example, the transmitter 100 that is the above-described station name sign, the receiver 200, the first server 301, and the second server 302.

The receiver 200 first receives an optical ID from the transmitter 100 in order to display the captured display image on which the AR image is superimposed as described above. Next, the receiver 200 transmits the optical ID to the first server 301.

When receiving the optical ID from the receiver 200, the first server 301 notifies the receiver 200 of a URL (Uniform Resource Locator) and Key associated with the received optical ID. Receiving such notification, the receiver 200 accesses the second server 302 based on the URL, and passes the key to the second server 302.

The second server 302 holds an AR image and recognition information associated with each key for each key. Therefore, when receiving a key from the receiver 200, the second server 302 selects an AR image and recognition information associated with the key, and transmits the selected AR image and recognition information to the receiver 200. . Thereby, the receiver 200 receives the AR image and the recognition information transmitted from the second server 302, and displays a captured display image on which the AR image is superimposed.

FIG. 44 is a diagram showing another example of the display system in the present embodiment.

The display system according to the present embodiment includes, for example, the transmitter 100 that is the above-described station name sign, the receiver 200, the first server 301, and the second server 302.

The receiver 200 first receives an optical ID from the transmitter 100 in order to display the captured display image on which the AR image is superimposed as described above. Next, the receiver 200 transmits the optical ID to the first server 301.

When receiving the optical ID from the receiver 200, the first server 301 notifies the second server 302 of the Key associated with the received optical ID.

The second server 302 holds an AR image and recognition information associated with each key for each key. Therefore, when the second server 302 receives the key from the first server 301, the second server 302 selects the AR image and the recognition information associated with the key, and the selected AR image and the recognition information are used as the first server. To the server 301. When receiving the AR image and the recognition information from the second server 302, the first server 301 transmits the AR image and the recognition information to the receiver 200. Thereby, the receiver 200 receives the AR image and the recognition information transmitted from the first server 301, and displays the captured display image on which the AR image is superimposed.

In the above-described example, the second server 302 transmits the AR image and the recognition information to the first server 301. However, the second server 302 may transmit the AR image and the recognition information to the receiver 200 without transmitting to the first server 301. .

FIG. 45 is a flowchart showing an example of the processing operation of the receiver 200 in the present embodiment.

First, the receiver 200 starts imaging with the above-described normal exposure time and communication exposure time (step S101). Then, the receiver 200 acquires an optical ID by decoding the decoding image obtained by imaging with the communication exposure time (step S102). Next, the receiver 200 transmits the optical ID to the server (step S103).

The receiver 200 acquires the AR image corresponding to the transmitted optical ID and the recognition information from the server (step S104). Next, the receiver 200 recognizes, as a target area, an area corresponding to the recognition information in the captured display image obtained by imaging with the normal exposure time (step S105). Then, the receiver 200 superimposes the AR image on the target area, and displays the captured display image on which the AR image is superimposed (step S106).

Next, the receiver 200 determines whether or not the imaging and the display of the captured display image should be terminated (step S107). Here, when the receiver 200 determines that it should not be terminated (N in step S107), it further determines whether or not the acceleration of the receiver 200 is equal to or greater than a threshold value (step S108). This acceleration is measured by an acceleration sensor provided in the receiver 200. When the receiver 200 determines that the acceleration is less than the threshold value (N in step S108), the receiver 200 executes the processing from step S105. Thereby, even when the captured display image displayed on the display 201 of the receiver 200 is shifted, the AR image can follow the target area of the captured display image. If the receiver 200 determines that the acceleration is equal to or greater than the threshold (Y in step S108), the receiver 200 executes the processing from step S102. Thereby, when the transmitter 100 is no longer displayed in the captured display image, it is possible to suppress erroneously recognizing an area where a subject different from the transmitter 100 is displayed as the target area.

Thus, in the present embodiment, since the AR image is displayed superimposed on the captured display image, an image useful for the user can be displayed. Furthermore, it is possible to superimpose an AR image on an appropriate target area while suppressing the processing load.

That is, in general augmented reality (that is, AR), a huge number of recognition target images stored in advance are compared with the captured display image, and any of the recognition target images is included in the captured display image. It is determined whether or not If it is determined that the recognition target image is included, the AR image corresponding to the recognition target image is superimposed on the captured display image. At this time, the AR image is aligned based on the recognition target image. In this way, in general augmented reality, a huge number of recognition target images are compared with captured display images, and further, position detection of the recognition target images in the captured display image is necessary even in alignment. There is a problem of a large amount of calculation and a high processing load.

However, in the display method according to the present embodiment, the light ID is acquired by decoding the decoding image obtained by imaging the subject. That is, the optical ID transmitted from the transmitter that is the subject is received. Furthermore, the AR image corresponding to this optical ID and the recognition information are acquired from the server. Therefore, the server does not need to compare a huge number of recognition target images and captured display images, and can select and transmit an AR image previously associated with the optical ID to the display device. Thereby, the amount of calculation can be reduced and the processing load can be significantly suppressed. Furthermore, the AR image display process can be speeded up.

In this embodiment, recognition information corresponding to this optical ID is acquired from the server. The recognition information is information for recognizing a target area that is an area in which an AR image is superimposed in a captured display image. This recognition information may be information indicating that a white square is the target area, for example. In this case, the target area can be easily recognized, and the processing load can be further suppressed. That is, the processing load can be further suppressed according to the content of the recognition information. Further, since the server can arbitrarily set the content of the recognition information according to the optical ID, the balance between the processing load and the recognition accuracy can be appropriately maintained.

In this embodiment, after the receiver 200 transmits the optical ID to the server, the receiver 200 acquires the AR image and the recognition information corresponding to the optical ID from the server. Of the AR image and the recognition information, At least one of these may be acquired in advance. That is, the receiver 200 collects a plurality of AR images and a plurality of recognition information corresponding to a plurality of optical IDs that may be received from the server and stores them. Thereafter, when receiving the optical ID, the receiver 200 selects an AR image and recognition information corresponding to the optical ID from a plurality of AR images and a plurality of recognition information stored in the receiver 200. Thereby, the display processing of the AR image can be further accelerated.

FIG. 46 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

For example, as shown in FIG. 46, the transmitter 100 is configured as a lighting device, and transmits a light ID by changing the luminance while illuminating the facility guide plate 101. Since the guide plate 101 is illuminated by the light from the transmitter 100, the luminance changes in the same manner as the transmitter 100, and the light ID is transmitted.

The receiver 200 acquires the captured display image Pb and the decoding image in the same manner as described above by imaging the guide plate 101 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the guide plate 101. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P2 and the recognition information corresponding to the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pb as a target area. For example, the receiver 200 recognizes the area where the frame 102 on the guide plate 101 is projected as the target area. This frame 102 is a frame for indicating the waiting time of the facility. Then, the receiver 200 superimposes the AR image P2 on the target area, and displays the captured display image Pb on which the AR image P2 is superimposed on the display 201. For example, the AR image P2 is an image including the character string “30 minutes”. In this case, since the AR image P2 is superimposed on the target area of the captured display image Pb, the receiver 200 captures the captured display image so that the guide plate 101 on which the waiting time “30 minutes” is described actually exists. Pb can be displayed. Thereby, it is possible to inform the user of the receiver 200 of the waiting time easily and easily without providing a special display device on the guide plate 101.

47 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

The transmitter 100 includes two illumination devices as shown in FIG. 47, for example. The transmitter 100 transmits the light ID by changing the luminance while illuminating the facility guide plate 104. Since the guide plate 104 is illuminated by the light from the transmitter 100, the luminance changes in the same manner as the transmitter 100, and the light ID is transmitted. The guide plate 104 indicates names of a plurality of facilities such as “ABC land” and “adventure land”.

The receiver 200 acquires the captured display image Pc and the decoding image in the same manner as described above by imaging the guide plate 104 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the guide plate 104. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P3 and the recognition information corresponding to the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pc as a target area. For example, the receiver 200 recognizes an area where the guide plate 104 is projected as a target area. Then, the receiver 200 superimposes the AR image P3 on the target area, and displays the captured display image Pc on which the AR image P3 is superimposed on the display 201. For example, the AR image P3 is an image indicating names of a plurality of facilities. In this AR image P3, the longer the waiting time of the facility, the smaller the name of the facility is displayed. Conversely, the shorter the waiting time of the facility, the larger the name of the facility is displayed. In this case, since the AR image P3 is superimposed on the target area of the captured display image Pc, the receiver 200 seems to actually have a guide plate 104 on which each facility name having a size corresponding to the waiting time is described. The captured display image Pc can be displayed. Thereby, without providing a special display device on the guide plate 104, the user of the receiver 200 can be informed easily and easily of the waiting time of each facility.

FIG. 48 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

The transmitter 100 includes two illumination devices as shown in FIG. 48, for example. The transmitter 100 transmits the light ID by changing the luminance while illuminating the castle wall 105. Since the castle wall 105 is illuminated by the light from the transmitter 100, the luminance is changed in the same manner as the transmitter 100, and the light ID is transmitted. Further, on the castle wall 105, for example, a small mark imitating the character's face is engraved as a hidden character 106.

The receiver 200 acquires the captured display image Pd and the decoding image in the same manner as described above by capturing an image of the castle wall 105 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the castle wall 105. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P4 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pd as a target area. For example, the receiver 200 recognizes, as the target area, an area in which a range including the hidden character 106 in the castle wall 105 is projected. Then, the receiver 200 superimposes the AR image P4 on the target area, and displays the captured display image Pd on which the AR image P4 is superimposed on the display 201. For example, the AR image P4 is an image imitating a character's face. The AR image P4 is an image that is sufficiently larger than the hidden character 106 displayed in the captured display image Pd. In this case, since the AR image P4 is superimposed on the target area of the captured display image Pd, the receiver 200 captures the image so that the castle wall 105 engraved with a large mark imitating the character's face actually exists. The display image Pd can be displayed. Thereby, it is possible to inform the user of the receiver 200 of the position of the hidden character 106 in an easy-to-understand manner.

FIG. 49 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

The transmitter 100 includes two illumination devices as shown in FIG. 49, for example. The transmitter 100 transmits the light ID by changing the luminance while illuminating the facility guide plate 107. Since the guide plate 107 is illuminated by the light from the transmitter 100, the luminance changes in the same manner as the transmitter 100, and the light ID is transmitted. In addition, an infrared shielding paint 108 is applied to a plurality of corners of the guide plate 107.

The receiver 200 acquires the captured display image Pe and the decoding image in the same manner as described above by imaging the guide plate 107 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the guide plate 107. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P5 and the recognition information corresponding to the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pe as a target area. For example, the receiver 200 recognizes an area where the guide plate 107 is projected as a target area.

Specifically, the recognition information indicates that a rectangle circumscribing the plurality of infrared shielding paints 108 is the target region. The infrared blocking paint 108 blocks infrared rays included in the light emitted from the transmitter 100. Therefore, the image sensor of the receiver 200 recognizes the infrared shielding paint 108 as an image darker than the surrounding area. The receiver 200 recognizes rectangles circumscribing the plurality of infrared shielding paints 108 that appear as dark images as target regions.

Then, the receiver 200 superimposes the AR image P5 on the target area, and displays the captured display image Pe on which the AR image P5 is superimposed on the display 201. For example, the AR image P5 shows a schedule of events to be performed in the facility of the guide board 107. In this case, since the AR image P5 is superimposed on the target area of the captured display image Pe, the receiver 200 displays the captured display image Pe so that the guide plate 107 on which the event schedule is described actually exists. can do. Thereby, without providing a special display device on the guide plate 107, it is possible to inform the user of the receiver 200 of the schedule of the facility event in an easy-to-understand manner.

Note that an infrared reflecting paint may be applied to the guide plate 107 instead of the infrared shielding paint 108. The infrared reflecting paint reflects infrared rays included in the light irradiated from the transmitter 100. Therefore, the image sensor of the receiver 200 recognizes the infrared reflective paint as an image brighter than the surrounding area. That is, in this case, the receiver 200 recognizes rectangles circumscribing a plurality of infrared reflective paints that appear as bright images as target regions.

FIG. 50 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

The transmitter 100 is configured as a station name sign and is disposed near the station exit guide plate 110. The station exit guide plate 110 includes a light source and emits light, but, unlike the transmitter 100, does not transmit an optical ID.

When the receiver 200 images the transmitter 100 and the station exit guide plate 110, the captured display image Ppre and the decoding image Pdec are acquired. Since the transmitter 100 changes in luminance and the station exit guide plate 110 emits light, the decoding image Pdec includes a bright line pattern region Pdec1 corresponding to the transmitter 100 and a bright region corresponding to the station exit guide plate 110. Pdec2 appears. The bright line pattern region Pdec1 is a region composed of a plurality of bright line patterns that appear by exposure at a communication exposure time of a plurality of exposure lines of the image sensor of the receiver 200.

Here, as described above, the identification information includes reference information for specifying the reference region Pbas in the captured display image Ppre and target information indicating the relative position of the target region Ptar with respect to the reference region Pbas. Yes. For example, the reference information indicates that the position of the reference area Pbas in the captured display image Ppre is the same as the position of the bright line pattern area Pdec1 in the decoding image Pdec. Further, the target information indicates that the position of the target area is the position of the reference area.

Therefore, the receiver 200 specifies the reference region Pbas from the captured display image Ppre based on the reference information. That is, the receiver 200 specifies, in the captured display image Ppre, an area that is at the same position as the bright line pattern area Pdec1 in the decoding image Pdec as the reference area Pbas. Further, the receiver 200 recognizes, as the target region Ptar, a region in the relative position indicated by the target information with reference to the position of the reference region Pbas in the captured display image Ppre. In the above example, since the target information indicates that the position of the target area Ptar is the position of the reference area Pbas, the receiver 200 recognizes the reference area Pbas in the captured display image Ppre as the target area Ptar.

Then, the receiver 200 superimposes the AR image P1 on the target area Ptar in the captured display image Ppre.

Thus, in the above-described example, the bright line pattern region Pdec1 is used to recognize the target region Ptar. On the other hand, when the region where the transmitter 100 is projected is to be recognized as the target region Ptar from only the captured display image Ppre without using the bright line pattern region Pdec1, erroneous recognition may occur. . That is, in the captured display image Ppre, not the area where the transmitter 100 is projected but the area where the station exit guide plate 110 is projected may be erroneously recognized as the target area Ptar. This is because the image of the transmitter 100 and the image of the station exit guide plate 110 in the captured display image Ppre are similar. However, as in the above example, when the bright line pattern region Pdec1 is used, it is possible to accurately recognize the target region Ptar while suppressing the occurrence of erroneous recognition.

FIG. 51 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

In the example shown in FIG. 50, the transmitter 100 transmits a light ID by changing the luminance of the entire station name sign, and the target information indicates that the position of the target area is the position of the reference area. However, in this embodiment, the transmitter 100 transmits the light ID by changing the luminance of the light emitting elements arranged in a part of the outer frame of the station name sign without changing the brightness of the entire station name sign. Also good. The target information only needs to indicate the relative position of the target area Ptar with respect to the reference area Pbas. For example, the position of the target area Ptar is above the reference area Pbas (specifically, vertically upward). May be shown.

In the example shown in FIG. 51, the transmitter 100 transmits the light ID by changing the luminance of a plurality of light emitting elements arranged along the horizontal direction below the outer frame of the station name sign. The target information indicates that the position of the target area Ptar is above the reference area Pbas.

In such a case, the receiver 200 specifies the reference region Pbas from the captured display image Ppre based on the reference information. That is, the receiver 200 specifies, in the captured display image Ppre, an area that is at the same position as the bright line pattern area Pdec1 in the decoding image Pdec as the reference area Pbas. Specifically, the receiver 200 specifies a rectangular reference region Pbas that is long in the horizontal direction and short in the vertical direction. Further, the receiver 200 recognizes, as the target region Ptar, a region in the relative position indicated by the target information with reference to the position of the reference region Pbas in the captured display image Ppre. That is, the receiver 200 recognizes an area above the reference area Pbas in the captured display image Ppre as the target area Ptar. At this time, the receiver 200 specifies the direction above the reference region Pbas based on the direction of gravity measured by the acceleration sensor provided in the receiver 200.

Note that the target information may indicate not only the relative position of the target area Ptar but also the size, shape, and aspect ratio of the target area Ptar. In this case, the receiver 200 recognizes the target area Ptar having the size, shape, and aspect ratio indicated by the target information. In addition, the receiver 200 may determine the size of the target area Ptar based on the size of the reference area Pbas.

FIG. 52 is a flowchart showing another example of the processing operation of the receiver 200 in the present embodiment.

The receiver 200 executes the processing of steps S101 to S104, as in the example shown in FIG.

Next, the receiver 200 identifies the bright line pattern region Pdec1 from the decoding image Pdec (step S111). Next, the receiver 200 specifies a reference area Pbas corresponding to the bright line pattern area Pdec1 from the captured display image Ppre (step S112). The receiver 200 recognizes the target area Ptar from the captured display image Ppre based on the recognition information (specifically, target information) and the reference area Pbas (step S113).

Next, similarly to the example shown in FIG. 45, the receiver 200 superimposes the AR image on the target area Ptar of the captured display image Ppre, and displays the captured display image Ppre on which the AR image is superimposed (step S106). . Then, the receiver 200 determines whether or not the imaging and the display of the captured display image Pre are to be ended (Step S107). Here, if the receiver 200 determines that it should not be ended (N in step S107), it further determines whether or not the acceleration of the receiver 200 is equal to or greater than a threshold value (step S114). This acceleration is measured by an acceleration sensor provided in the receiver 200. When the receiver 200 determines that the acceleration is less than the threshold value (N in step S114), the receiver 200 executes the processing from step S113. Thereby, even when the captured display image Ppre displayed on the display 201 of the receiver 200 is shifted, the AR image can follow the target area Ptar of the captured display image Ppre. If the receiver 200 determines that the acceleration is equal to or greater than the threshold (Y in step S114), the receiver 200 executes the processing from step S111 or step S102. Thereby, it can suppress that the area | region where the to-be-photographed object (for example, station exit guide board 110) different from the transmitter 100 is reflected is mistakenly recognized as the object area Ptar.

FIG. 53 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

When the AR image P1 in the displayed captured display image Ppre is tapped, the receiver 200 enlarges and displays the AR image P1. Alternatively, when tapped, the receiver 200 may display a new AR image showing more detailed content than that shown in the AR image P1 instead of the AR image P1. In addition, when the AR image P1 indicates information for one page of an information magazine including a plurality of pages, the receiver 200 displays a new AR image indicating information on the next page of the page of the AR image P1. You may display instead of AR image P1. Alternatively, when tapped, the receiver 200 may display a moving image related to the AR image P1 as a new AR image instead of the AR image P1. At this time, the receiver 200 may display a moving image in which an object (autumn leaves in the example of FIG. 53) comes out from the target area Ptar as an AR image.

FIG. 54 is a diagram showing a captured display image Ppre and a decoding image Pdec acquired by imaging of the receiver 200 in the present embodiment.

The receiver 200 acquires captured images such as a captured display image Ppre and a decoding image Pdec at a frame rate of 30 fps as shown in FIG. Specifically, the receiver 200 acquires the captured display image Ppre “A” at time t1, acquires the decoding image Pdec at time t2, and acquires the captured display image Ppre “B” at time t3. The captured display image Ppre and the decoding image Pdec are obtained alternately.

In addition, when displaying the captured image, the receiver 200 displays only the captured display image Ppre among the captured images, and does not display the decoding image Pdec. That is, as shown in (a2) of FIG. 54, the receiver 200 displays the captured display image Ppre acquired immediately before, instead of the decoding image Pdec, when acquiring the decoding image Pdec. Specifically, the receiver 200 displays the acquired captured display image Ppre “A” at time t1, and again displays the captured display image Ppre “A” acquired at time t1 at time t2. To do. Thereby, the receiver 200 displays the captured display image Ppre at a frame rate of 15 fps.

Here, in the example shown in (a1) of FIG. 54, the receiver 200 alternately acquires the captured display image Ppre and the decoding image Pdec, but the acquisition form of these images in the present embodiment is It is not restricted to such a form. That is, the receiver 200 continuously obtains N (N is an integer equal to or greater than 1) decoding images Pdec, and then continuously captures M (M is an integer equal to or greater than 1) captured display images Ppre. You may repeat acquiring.

In addition, the receiver 200 needs to switch the acquired captured image to the captured display image Ppre and the decoding image Pdec, and this switching may take time. Therefore, as illustrated in (b1) of FIG. 54, the receiver 200 may provide a switching period when switching between acquisition of the captured display image Ppre and acquisition of the decoding image Pdec. Specifically, when the receiver 200 obtains the decoding image Pdec at time t3, the receiver 200 executes processing for switching the captured image during the switching period from time t3 to t5, and at time t5, the captured display image Prep " A ”is acquired. Thereafter, the receiver 200 executes processing for switching the captured image in the switching period from time t5 to time t7, and acquires the decoding image Pdec at time t7.

When the switching period is provided in this way, the receiver 200 displays the captured display image Ppre acquired immediately before in the switching period, as shown in (b2) of FIG. Therefore, in this case, the display frame rate of the captured display image Ppre in the receiver 200 is low, for example, 3 fps. Thus, when the frame rate is low, even if the user moves the receiver 200, the captured display image Ppre displayed may not move according to the movement of the receiver 200. That is, the captured display image Ppre is not displayed as a live view. Therefore, the receiver 200 may move the captured display image Ppre according to the movement of the receiver 200.

FIG. 55 is a diagram showing an example of the captured display image Ppre displayed on the receiver 200 in the present embodiment.

The receiver 200 displays, on the display 201, a captured display image Ppre obtained by imaging, for example, as illustrated in FIG. Here, the user moves the receiver 200 to the left side. At this time, when a new captured display image Ppre is not acquired by imaging by the receiver 200, the receiver 200 moves the displayed captured display image Ppre to the right as shown in FIG. 55 (b). That is, the receiver 200 includes an acceleration sensor, and moves the displayed captured display image Ppre to match the movement of the receiver 200 according to the acceleration measured by the acceleration sensor. Thereby, the receiver 200 can display the captured display image Ppre as a live view in a pseudo manner.

FIG. 56 is a flowchart showing another example of the processing operation of the receiver 200 in the present embodiment.

First, similarly to the above, the receiver 200 superimposes the AR image on the target area Ptar of the captured display image Ppre and follows the target area Ptar (step S122). That is, an AR image that moves together with the target area Ptar in the captured display image Ppre is displayed. Then, the receiver 200 determines whether or not to maintain the display of the AR image (step S122). If it is determined that the display of the AR image is not maintained (N in step S122), if the receiver 200 acquires a new light ID by imaging, the new AR image corresponding to the light ID is captured and displayed. It is displayed superimposed on Pre (step S123).

On the other hand, if it is determined that the display of the AR image is to be maintained (Y in step S122), the receiver 200 repeatedly executes the processing from step S121. At this time, the receiver 200 does not display another AR image even if another AR image is acquired. Alternatively, even when the receiver 200 has acquired a new decoding image Pdec, the receiver 200 does not acquire an optical ID by decoding the decoding image Pdec. At this time, power consumption for decoding can be suppressed.

As described above, by maintaining the display of the AR image, it is possible to prevent the displayed AR image from being erased or being difficult to see due to the display of another AR image. That is, the displayed AR image can be easily seen by the user.

For example, in step S122, the receiver 200 determines to maintain the display of the AR image until a predetermined period (a certain period) elapses after the AR image is displayed. That is, when displaying the captured display image Ppre, the receiver 200 is determined in advance while suppressing the display of the second AR image different from the first AR image that is the AR image superimposed in step S121. The first AR image is displayed only during the display period. The receiver 200 may prohibit the decoding of the newly acquired decoding image Pdec during this display period.

Thereby, when the user is viewing the first AR image once displayed, it is possible to prevent the first AR image from being immediately replaced with a different second AR image. Furthermore, since decoding of the newly acquired decoding image Pdec is a wasteful process when the display of the second AR image is suppressed, power consumption can be suppressed by prohibiting the decoding. .

Alternatively, in step S122, the receiver 200 may include a face camera, and when detecting that the user's face is approaching based on the imaging result of the face camera, the receiver 200 may determine to maintain the display of the AR image. . That is, when displaying the captured display image Ppre, the receiver 200 further determines whether or not the user's face is approaching the receiver 200 by imaging with a face camera provided in the receiver 200. When the receiver 200 determines that the face is approaching, the first AR image is suppressed while suppressing the display of the second AR image that is different from the first AR image that is the AR image superimposed in step S121. An AR image is displayed.

Alternatively, in step S122, the receiver 200 may include an acceleration sensor, and when detecting that the user's face is approaching based on the measurement result of the acceleration sensor, the receiver 200 may determine to maintain the display of the AR image. . That is, when displaying the captured display image Ppre, the receiver 200 further determines whether or not the user's face is approaching the receiver 200 based on the acceleration of the receiver 200 measured by the acceleration sensor. For example, when the acceleration of the receiver 200 measured by the acceleration sensor shows a positive value in a direction perpendicular to the display 201 of the receiver 200, the receiver 200 is approaching the user's face. judge. When the receiver 200 determines that the face is approaching, the first 200 is performed while suppressing the display of the second AR image that is different from the first augmented reality image that is the AR image superimposed in step S121. The AR image is displayed.

Accordingly, when the user is approaching the receiver 200 to see the first AR image, the first AR image can be prevented from being replaced with a different second AR image. .

Alternatively, in step S122, the receiver 200 may determine that the display of the AR image is maintained when a lock button provided in the receiver 200 is pressed.

In step S122, the receiver 200 determines that the display of the AR image is not maintained when the above-described certain period (that is, the display period) has elapsed. Further, the receiver 200 determines that the display of the AR image is not maintained when acceleration equal to or greater than the threshold value is measured by the acceleration sensor even when the above-described certain period has not elapsed. That is, when displaying the captured display image Ppre, the receiver 200 further measures the acceleration of the receiver 200 with the acceleration sensor during the display period described above, and determines whether or not the measured acceleration is equal to or greater than a threshold value. When the receiver 200 determines that the second AR image is greater than or equal to the threshold, the receiver 200 cancels the suppression of the display of the second AR image, thereby displaying the second AR image instead of the first AR image in step S123.

Thereby, when the acceleration of the display device equal to or greater than the threshold is measured, the suppression of the display of the second AR image is released. Therefore, for example, when the user moves the receiver 200 greatly so as to point the image sensor at another subject, the second AR image can be displayed immediately.

FIG. 57 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

For example, as shown in FIG. 57, the transmitter 100 is configured as a lighting device, and transmits a light ID by changing the luminance while illuminating a stage 111 for a small doll. Since the stage 111 is illuminated by the light from the transmitter 100, the luminance changes similarly to the transmitter 100, and the optical ID is transmitted.

The two receivers 200 image the stage 111 illuminated by the transmitter 100 from the left and right.

The left receiver 200 of the two receivers 200 captures the captured display image Pf and the decoding image in the same manner as described above by capturing the stage 111 illuminated by the transmitter 100 from the left. The receiver 200 on the left side acquires the optical ID by decoding the decoding image. That is, the left receiver 200 receives the optical ID from the stage 111. The left receiver 200 transmits the optical ID to the server. Then, the left-side receiver 200 acquires a three-dimensional AR image and recognition information corresponding to the optical ID from the server. This three-dimensional AR image is an image for displaying a doll three-dimensionally, for example. The left receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pf as a target area. For example, the left receiver 200 recognizes the upper area at the center of the stage 111 as the target area.

Next, the left-side receiver 200 generates a two-dimensional AR image P6a corresponding to the orientation from the three-dimensional AR image based on the orientation of the stage 111 displayed in the captured display image Pf. Then, the receiver 200 on the left side superimposes the two-dimensional AR image P6a on the target area, and displays the captured display image Pf on which the AR image P6a is superimposed on the display 201. In this case, since the two-dimensional AR image P6a is superimposed on the target area of the captured display image Pf, the left receiver 200 displays the captured display image Pf so that the doll actually exists on the stage 111. can do.

Similarly, the right receiver 200 of the two receivers 200 captures the captured display image Pg and the decoding image in the same manner as described above by capturing the stage 111 illuminated by the transmitter 100 from the right. get. The right receiver 200 acquires the optical ID by decoding the decoding image. That is, the right receiver 200 receives the optical ID from the stage 111. The right receiver 200 transmits the optical ID to the server. Then, the right-side receiver 200 acquires a three-dimensional AR image and recognition information corresponding to the optical ID from the server. The right receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pg as a target area. For example, the right receiver 200 recognizes the upper area at the center of the stage 111 as the target area.

Next, the right-side receiver 200 generates a two-dimensional AR image P6b corresponding to the orientation from the three-dimensional AR image based on the orientation of the stage 111 displayed in the captured display image Pg. Then, the receiver 200 on the right side superimposes the two-dimensional AR image P6b on the target region, and displays the captured display image Pg on which the AR image P6b is superimposed on the display 201. In this case, since the two-dimensional AR image P6b is superimposed on the target region of the captured display image Pg, the right-side receiver 200 displays the captured display image Pg so that the doll actually exists on the stage 111. can do.

Thus, the two receivers 200 display the AR images P6a and P6b at the same position on the stage 111. The AR images P6a and P6b are generated according to the orientation of the receiver 200 so that the virtual doll is actually facing a predetermined direction. Therefore, the captured display image can be displayed so that the doll actually exists on the stage 111 no matter what direction the stage 111 is captured.

In the above example, the receiver 200 generates a two-dimensional AR image corresponding to the positional relationship between the receiver 200 and the stage 111 from the three-dimensional AR image. May be obtained from the server. That is, the receiver 200 transmits information indicating the positional relationship together with the optical ID to the server, and acquires the two-dimensional AR image from the server instead of the three-dimensional AR image. Thereby, the burden on the receiver 200 can be reduced.

FIG. 58 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

For example, as shown in FIG. 58, the transmitter 100 is configured as an illumination device, and transmits a light ID by changing the luminance while illuminating a cylindrical structure 112. Since the structure 112 is illuminated by the light from the transmitter 100, the luminance is changed similarly to the transmitter 100, and the light ID is transmitted.

The receiver 200 acquires the captured display image Ph and the decoding image in the same manner as described above by imaging the structure 112 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the structure 112. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P7 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Ph as a target area. For example, the receiver 200 recognizes an area where the central portion of the structure 112 is projected as a target area. Then, the receiver 200 superimposes the AR image P7 on the target area, and displays the captured display image Ph on which the AR image P7 is superimposed on the display 201. For example, the AR image P <b> 7 is an image including a character string “ABCD”, and the character string is distorted in accordance with the curved surface at the center of the structure 112. In this case, since the AR image P2 including the distorted character string is superimposed on the target region of the captured display image Ph, the receiver 200 makes sure that the character string drawn on the structure 112 actually exists. The captured display image Ph can be displayed.

FIG. 59 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

For example, as shown in FIG. 59, the transmitter 100 transmits the light ID by changing the luminance while illuminating the menu 113 of the restaurant. Since the menu 113 is illuminated by the light from the transmitter 100, the luminance changes in the same manner as the transmitter 100, and the light ID is transmitted. The menu 113 indicates names of a plurality of dishes such as “ABC soup”, “XYZ salad”, and “KLM lunch”.

The receiver 200 acquires the captured display image Pi and the decoding image in the same manner as described above by capturing an image of the menu 113 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the menu 113. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P8 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pi as a target area. For example, the receiver 200 recognizes an area where the menu 113 is displayed as the target area. Then, the receiver 200 superimposes the AR image P8 on the target region, and displays the captured display image Pi on which the AR image P8 is superimposed on the display 201. For example, the AR image P8 is an image that shows the ingredients used for each of a plurality of dishes by marks. For example, the AR image P8 shows a mark imitating an egg for a dish “XYZ salad” using eggs, and a pig for a dish “KLM lunch” using pork. The imitated mark is shown. In this case, since the AR image P8 is superimposed on the target area of the captured display image Pi, the receiver 200 displays the captured display image Pi so that the menu 113 with the food mark is actually present. be able to. Thereby, without providing a special display device in the menu 113, the user of the receiver 200 can be easily and easily informed of the ingredients of each dish.

In addition, the receiver 200 acquires a plurality of AR images, selects an AR image suitable for the user from the plurality of AR images based on the user information set by the user, and superimposes the AR images. May be. For example, if the user information indicates that the user shows an allergic reaction to the egg, the receiver 200 selects an AR image that is marked with an egg for a dish in which the egg is used. If the user information indicates that the intake of pork is prohibited, the receiver 200 selects an AR image in which a pork mark is attached to a dish in which pork is used. Alternatively, the receiver 200 may transmit the user information together with the optical ID to the server and acquire an AR image corresponding to the optical ID and the user information from the server. Thereby, for each user, a menu that prompts the user to call can be displayed.

FIG. 60 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

For example, as shown in FIG. 60, the transmitter 100 is configured as a television, and transmits an optical ID by changing luminance while displaying an image on a display. In addition, a normal television 114 is disposed in the vicinity of the transmitter 100. The television 114 displays an image on the display, but does not transmit an optical ID.

The receiver 200 acquires the captured display image Pj and the decoding image in the same manner as described above, for example, by imaging the television 114 together with the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P9 and the recognition information corresponding to the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pj as a target area.

For example, the receiver 200 uses the bright line pattern area of the image for decoding, so that the lower part of the area where the transmitter 100 transmitting the optical ID is displayed in the captured display image Pj is the first target area. Recognize as At this time, the reference information included in the recognition information indicates that the position of the reference area in the captured display image Pj is the same as the position of the bright line pattern area in the decoding image. Furthermore, the target information included in the recognition information indicates that there is a target area below the reference area. The receiver 200 recognizes the first target area described above using such recognition information.

Furthermore, the receiver 200 recognizes an area whose position is fixed in advance below the captured display image Pj as the second target area. The second target area is larger than the first target area. The target information included in the recognition information further indicates not only the position of the first target area but also the position and size of the second target area as described above. The receiver 200 recognizes the second target area described above using such recognition information.

Then, the receiver 200 superimposes the AR image P9 on the first target area and the second target area, and displays the captured display image Pj on which the AR image P8 is superimposed on the display 201. In the superposition of the AR image P9, the receiver 200 matches the size of the AR image P9 with the size of the first target area, and superimposes the AR image P9 whose size has been adjusted on the first target area. Furthermore, the receiver 200 matches the size of the AR image P9 with the size of the second target area, and superimposes the AR image P9 whose size has been adjusted on the second target area.

For example, the AR image P9 indicates a caption for the video of the transmitter 100. Further, the language of the caption of the AR image P9 is a language according to the user information set and registered in the receiver 200. That is, when transmitting the optical ID to the server, the receiver 200 also transmits the user information (for example, information indicating the user's nationality or language used) to the server. Then, the receiver 200 acquires an AR image P9 indicating a language caption corresponding to the user information. Alternatively, the receiver 200 acquires a plurality of AR images P9 indicating subtitles in different languages, and uses the AR images used for superimposition from the plurality of AR images P9 according to the user information registered and registered. P9 may be selected.

In other words, in the example illustrated in FIG. 60, the receiver 200 acquires a captured display image Pj and a decoding image by capturing a plurality of displays each displaying an image as a subject. Then, when the receiver 200 recognizes the target area, an area in which the transmission display (that is, the transmitter 100) that is the display that transmits the light ID among the plurality of displays appears in the captured display image Pj. Is recognized as a target area. Next, the receiver 200 superimposes the first subtitle corresponding to the image displayed on the transmission display as an AR image on the target area. Furthermore, the receiver 200 superimposes a second subtitle, which is a subtitle obtained by enlarging the first subtitle, on a region larger than the target region in the captured display image Pj.

Thereby, the receiver 200 can display the captured display image Pj so that captions actually exist in the video of the transmitter 100. Furthermore, since the receiver 200 also superimposes a large caption on the lower part of the captured display image Pj, even if the caption attached to the video of the transmitter 100 is small, the caption can be easily viewed. In addition, when there is no caption attached to the video of the transmitter 100 and only a large caption is superimposed on the lower part of the captured display image Pj, whether the superimposed caption is a caption for the video of the transmitter 100 or a television It is difficult to determine whether it is a caption for 114 videos. However, in the present embodiment, since captions are attached to the video of the transmitter 100 that transmits the optical ID, the user can easily determine which video the superimposed caption is for. Can do.

In the display of the captured display image Pj, the receiver 200 may further determine whether or not audio information is included in the information acquired from the server. When the receiver 200 determines that the audio information is included, the receiver 200 outputs the audio indicated by the audio information with priority over the first and second subtitles. Thereby, since sound is preferentially output, it is possible to reduce the burden of the user reading subtitles.

In the above example, the subtitle language is changed according to the user information (that is, the user attribute), but the video (that is, the content) itself displayed on the transmitter 100 may be changed. For example, in the case where the video displayed on the transmitter 100 is a news video, if the user information indicates that the user is Japanese, the receiver 200 may update the news video broadcast in Japan. Is acquired as an AR image. Then, the receiver 200 superimposes the news video on an area where the display of the transmitter 100 is displayed (that is, the target area). On the other hand, if the user information indicates that the user is American, the receiver 200 acquires a news video broadcast in the United States as an AR image. Then, the receiver 200 superimposes the news video on an area where the display of the transmitter 100 is displayed (that is, the target area). Thereby, an image suitable for the user can be displayed. Note that the user information indicates, for example, nationality or language used as the user attribute, and the receiver 200 acquires the above-described AR image based on the attribute.

FIG. 61 is a diagram showing an example of recognition information in the present embodiment.

Even if the recognition information is, for example, the above-described feature points and feature amounts, misrecognition may occur. For example, the transmitters 100a and 100b are configured as station names as with the transmitter 100, respectively. Even if these transmitters 100a and 100b are station names different from each other, they may be misrecognized because they are similar if they are located close to each other.

Therefore, the recognition information of each of the transmitters 100a and 100b does not indicate each feature point and each feature amount of the entire image of the transmitter 100a or 100b, and each feature point of only a characteristic part of the image and Each feature amount may be indicated.

For example, the part a1 of the transmitter 100a and the part b1 of the transmitter 100b are greatly different from each other, and the part a2 of the transmitter 100a and the part b2 of the transmitter 100b are greatly different from each other. Therefore, if the transmitters 100a and 100b are installed in a predetermined range (that is, a short distance), the server uses image characteristics of the parts a1 and a2 as the recognition information corresponding to the transmitter 100a. Preserve points and features. Similarly, the server holds the feature points and feature amounts of the images of the parts b1 and b2 as the identification information corresponding to the transmitter 100b.

Thereby, even when the transmitters 100a and 100b similar to each other are close to each other (when they are within the predetermined range described above), the receiver 200 can appropriately use the identification information. The target area can be recognized.

FIG. 62 is a flowchart showing another example of the processing operation of the receiver 200 in the present embodiment.

The receiver 200 first determines whether or not the user has a visual impairment based on the user information set and registered in the receiver 200 (step S131). If the receiver 200 determines that there is a visual impairment (Y in step S131), the receiver 200 outputs the characters of the AR image displayed in a superimposed manner by voice (step S132). On the other hand, when the receiver 200 determines that there is no visual impairment (N in step S131), the receiver 200 further determines whether the user has a hearing impairment based on the user information (step S133). Here, if the receiver 200 determines that there is a hearing impairment (Y in step S133), the receiver 200 stops the sound output (step S134). At this time, the receiver 200 stops outputting sound by all functions.

The receiver 200 may perform the process of step S133 when it is determined in step S131 that there is a visual impairment (Y in step S131). That is, when it is determined that there is a visual impairment and there is no hearing impairment, the receiver 200 may output the AR image characters displayed in a superimposed manner by voice.

FIG. 63 is a diagram illustrating an example in which the receiver 200 according to the present embodiment identifies bright line pattern regions.

First, the receiver 200 obtains a decoding image by imaging two transmitters each transmitting an optical ID, and performs decoding on the decoding image to obtain an optical ID as shown in FIG. To get. At this time, since the decoding image includes two bright line pattern areas X and Y, the receiver 200 transmits the light ID of the transmitter corresponding to the bright line pattern area X and the transmission corresponding to the bright line pattern area Y. Get the machine's optical ID. The light ID of the transmitter corresponding to the bright line pattern region X is made up of numerical values (that is, data) corresponding to addresses 0 to 9, for example, “5, 2, 8, 4, 3, 6, 1, 9, 4,3 ". Similarly, the transmitter optical ID corresponding to the bright line pattern region X is also composed of numerical values corresponding to addresses 0 to 9, for example, “5, 2, 7, 7, 1, 5, 3, 2, 7, 4 ".

Even if the receiver 200 acquires these light IDs once, that is, even if these light IDs are known, from which bright line pattern area each light ID was obtained when taking an image. It may be a situation that you do not understand. In such a case, the receiver 200 can easily determine from which bright line pattern area each known light ID is obtained by performing the processes shown in FIGS. Can be determined quickly.

Specifically, the receiver 200 first acquires the decoding image Pdec11 as shown in (a) of FIG. 63, and decodes each of the bright line pattern regions X and Y by decoding the decoding image Pdec11. The numerical value of the optical ID address 0 is acquired. For example, the numerical value of the light ID address 0 of the bright line pattern region X is “5”, and the numerical value of the light ID address 0 of the bright line pattern region Y is also “5”. Since the numerical value of the address 0 of each light ID is “5”, at this time, it cannot be determined from which bright line pattern area the known light ID is obtained.

Therefore, as shown in FIG. 63B, the receiver 200 acquires the decoding image Pdec12 and decodes the decoding image Pdec12 to obtain the address 1 of each light ID of the bright line pattern regions X and Y. Get the number of. For example, the numerical value of the address 1 of the light ID in the bright line pattern region X is “2”, and the numerical value of the address 1 of the light ID in the bright line pattern region Y is also “2”. Since the numerical value of the address 1 of each light ID is “2”, it is impossible to determine from which bright line pattern region the known light ID is obtained.

Therefore, as shown in FIG. 63 (c), the receiver 200 acquires the decoding image Pdec13, and decodes the decoding image Pdec13 to obtain the respective light IDs of the bright line pattern regions X and Y. Get the numerical value of address 2. For example, the numerical value of the address 2 of the light ID in the bright line pattern region X is “8”, and the numerical value of the address 2 of the light ID in the bright line pattern region Y is “7”. At this time, it can be determined that the known light ID “5, 2, 8, 4, 3, 6, 1, 9, 4, 3” has been obtained from the bright line pattern region X, and the known light ID “5, 2, 7, 7, 1, 5, 3, 2, 7, 4 "can be determined to have been obtained from the bright line pattern region Y.

However, in order to increase the reliability, the receiver 200 may further acquire the numerical value of the address 3 of each optical ID, as shown in FIG. That is, the receiver 200 acquires the decoding image Pdec14, and acquires the numerical value of the address 3 of the light ID of each of the bright line pattern areas X and Y by decoding the decoding image Pdec14. For example, the numerical value of the address 3 of the light ID in the bright line pattern region X is “4”, and the numerical value of the address 3 of the light ID in the bright line pattern region Y is “7”. At this time, it can be determined that the known light ID “5, 2, 8, 4, 3, 6, 1, 9, 4, 3” has been obtained from the bright line pattern region X, and the known light ID “5, 2, 7, 7, 1, 5, 3, 2, 7, 4 "can be determined to have been obtained from the bright line pattern region Y. That is, since the light IDs of the bright line pattern areas X and Y can be identified not only by the address 2 but also by the address 3, the reliability can be increased.

Thus, in this embodiment, the numerical value of at least one address is reacquired without acquiring the numerical values (that is, data) of all addresses of the optical ID again. This makes it possible to easily and quickly determine from which bright line pattern region the known light ID is obtained.

In the example shown in FIGS. 63C and 63D described above, the numerical value acquired for the predetermined address matches the numerical value of the known optical ID, but it does not have to match. Good. For example, in the example shown in (d) of FIG. 63, the receiver 200 acquires “6” as the numerical value of the address 3 of the light ID of the bright line pattern region Y. The numerical value “6” of the address 3 is different from the numerical value “7” of the address 3 of the known light ID “5, 2, 7, 7, 1, 5, 3, 2, 7, 4”. However, since the numerical value “6” is a numerical value close to the numerical value “7”, the receiver 200 has a known light ID “5, 2, 7, 7, 1, 5, 3, 2, 7, 4” with a bright line. It may be determined that the pattern area Y is obtained. In the receiver, the numerical value “6” is a numerical value close to the numerical value “7” depending on whether the numerical value “6” is within the range of the numerical value “7” ± n (n is a number of 1 or more, for example). It may be determined whether or not.

FIG. 64 is a diagram illustrating another example of the receiver 200 in the present embodiment.

The receiver 200 is configured as a smartphone in the above example, but may be configured as a head-mounted display (also referred to as glass) including an image sensor.

Such a receiver 200 detects a predetermined signal because power consumption increases when a processing circuit for displaying an AR image as described above (hereinafter referred to as an AR processing circuit) is always activated. When this occurs, the AR processing circuit may be activated.

For example, the receiver 200 includes a touch sensor 202. The touch sensor 202 outputs a touch signal when it touches a user's finger or the like. The receiver 200 activates the AR processing circuit when detecting the touch signal.

Alternatively, the receiver 200 may activate the AR processing circuit when detecting a radio signal such as Bluetooth (registered trademark) or Wi-Fi (registered trademark).

Alternatively, the receiver 200 may include an acceleration sensor, and may activate the AR processing circuit when the acceleration sensor measures an acceleration equal to or greater than a threshold value in a direction opposite to the direction of gravity. That is, the receiver 200 activates the AR processing circuit when detecting the signal indicating the acceleration. For example, when the user pushes up the nose pad portion of the receiver 200 configured as a glass upward with a fingertip from below, the receiver 200 detects a signal indicating the acceleration and activates the AR processing circuit.

Alternatively, the receiver 200 may activate the AR processing circuit when it is detected by the GPS and the 9-axis sensor that the image sensor is directed to the transmitter 100. That is, the receiver 200 activates the AR processing circuit when detecting a signal indicating that the receiver 200 is directed in a predetermined direction. In this case, if the transmitter 100 is the above-mentioned Japanese station name mark, the receiver 200 displays an AR image indicating the English station name superimposed on the station name mark.

FIG. 65 is a flowchart showing another example of the processing operation of the receiver 200 in the present embodiment.

When the receiver 200 acquires the optical ID from the transmitter 100 (step S141), the receiver 200 switches the mode of noise cancellation by receiving mode designation information corresponding to the optical ID (step S142). Then, the receiver 200 determines whether or not the mode switching process should be terminated (step S143), and when it is determined that the mode switching process should not be terminated (N in step S143), the process from step S141 is repeatedly executed. The switching of the noise canceling mode is, for example, a mode (ON) for canceling noise such as an engine in an airplane and a mode (OFF) for not canceling the noise. Specifically, a user carrying the receiver 200 is listening to a sound such as music output from the receiver 200 by putting an earphone connected to the receiver 200 on the ear. When such a user gets on the airplane, the receiver 200 acquires an optical ID. As a result, the receiver 200 switches the noise cancellation mode from OFF to ON. As a result, the user can hear a voice that does not include noise such as engine noise even in the cabin. The receiver 200 also acquires the light ID when the user leaves the airplane. The receiver 200 that has acquired this optical ID switches the noise cancellation mode from ON to OFF. Note that the noise to be subject to noise cancellation is not limited to engine noise but may be any sound such as a human voice.

FIG. 66 is a diagram illustrating an example of a transmission system including a plurality of transmitters in the present embodiment.

This transmission system includes a plurality of transmitters 120 arranged in a predetermined order. Like the transmitter 100, these transmitters 120 are transmitters in any one of the first to third embodiments, and include one or a plurality of light emitting elements (for example, LEDs). The leading transmitter 120 transmits the optical ID by changing the luminance of one or a plurality of light emitting elements according to a predetermined frequency (carrier frequency). Further, the first transmitter 120 outputs a signal indicating the change in luminance to the subsequent transmitter 120 as a synchronization signal. When the subsequent transmitter 120 receives the synchronization signal, it transmits the optical ID by changing the luminance of one or more light emitting elements in accordance with the synchronization signal. Further, the subsequent transmitter 120 outputs a signal indicating the change in luminance to the subsequent subsequent transmitter 120 as a synchronization signal. Thereby, all the transmitters 120 included in the transmission system transmit the optical ID in synchronization.

Here, the synchronization signal is transferred from the first transmitter 120 to the subsequent transmitter 120, and is further transferred from the subsequent transmitter 120 to the next subsequent transmitter 120 to the last transmitter 120. reach. For example, it takes about 1 μsec to transfer the synchronization signal. Therefore, if N (N is an integer of 2 or more) transmitters 120 are provided in the transmission system, it takes 1 × Nμ seconds for the synchronization signal to reach the last transmitter 120 from the first transmitter 120. become. As a result, the transmission timing of the optical ID is shifted by a maximum of N μ seconds. For example, even if N transmitters 120 transmit optical IDs according to a frequency of 9.6 kHz, and the receiver 200 attempts to receive an optical ID at a frequency of 9.6 kHz, the receiver 200 receives light that is shifted by N μ seconds. Since the ID is received, the optical ID may not be received correctly.

Therefore, in the present embodiment, the head transmitter 120 transmits the optical ID at a higher speed according to the number of transmitters 120 included in the transmission system. For example, the first transmitter 120 transmits an optical ID according to a frequency of 9.605 kHz. On the other hand, the receiver 200 receives the optical ID at a frequency of 9.6 kHz. At this time, even if the receiver 200 receives the optical ID shifted by N μs, the frequency of the leading transmitter 120 is higher than the frequency of the receiver 200 by 0.005 kHz. Occurrence can be suppressed.

Also, the first transmitter 120 may control the frequency adjustment amount by having the synchronization signal fed back from the last transmitter 120. For example, the first transmitter 120 measures the time from when it outputs the synchronization signal itself until it receives the synchronization signal fed back from the last transmitter 120. And the head transmitter 120 transmits optical ID according to a frequency higher than a reference frequency (for example, 9.6 kHz), so that the time is long.

FIG. 67 is a diagram illustrating an example of a transmission system including a plurality of transmitters and receivers in the present embodiment.

This transmission system includes, for example, two transmitters 120 and a receiver 200. One of the two transmitters 120 transmits an optical ID according to a frequency of 9.599 kHz. The other transmitter 120 transmits an optical ID according to a frequency of 9.601 kHz. In such a case, each of the two transmitters 120 notifies the receiver 200 of the frequency of its own optical ID with a radio wave signal.

When the receiver 200 receives the notification of those frequencies, the receiver 200 tries to perform decoding according to each of the notified frequencies. That is, the receiver 200 attempts to decode the decoding image according to the frequency of 9.599 kHz. If the optical ID cannot be received by this, the receiver 200 attempts to decode the decoding image according to the frequency of 9.601 kHz. As described above, the receiver 200 attempts to decode the decoding image according to each of all the notified frequencies. In other words, the receiver 200 performs brute force for each notified frequency. Alternatively, the receiver 200 may attempt decoding according to the average frequency of all the notified frequencies. That is, the receiver 200 attempts decoding according to 9.6 kHz, which is an average frequency of 9.599 kHz and 9.601 kHz.

Thereby, it is possible to reduce the occurrence rate of reception errors due to the difference in frequency between the receiver 200 and the transmitter 120.

FIG. 68A is a flowchart illustrating an example of the processing operation of the receiver 200 in the present embodiment.

First, the receiver 200 starts imaging (step S151) and initializes the parameter N to 1 (step S152). Next, the receiver 200 decodes the decoding image obtained by the imaging according to the frequency corresponding to the parameter N, and calculates an evaluation value for the decoding result (step S153). For example, parameters N = 1, 2, 3, 4, 5 are associated with frequencies such as 9.6 kHz, 9.601 kHz, 9.599 kHz, and 9.602 kHz in advance. The evaluation value indicates a higher numerical value as the decoding result is more similar to the correct optical ID.

Next, the receiver 200 determines whether or not the numerical value of the parameter N is equal to Nmax that is a predetermined integer of 1 or more (step S154). Here, if the receiver 200 determines that it is not equal to Nmax (N in step S154), it increments the parameter N (step S155) and repeats the processing from step S153. On the other hand, if the receiver 200 determines that it is equal to Nmax (Y in step S154), the frequency for which the maximum evaluation value is calculated is registered as the optimum frequency in association with the location information indicating the location of the receiver 200. To do. The optimum frequency and location information registered in this way are used for receiving the optical ID by the receiver 200 that has moved to the location indicated by the location information after registration. The location information may be information indicating a position measured by GPS, for example, or may be identification information (for example, SSID: Service Set Identifier) of an access point in a wireless LAN (Local Area Network).

In addition, the receiver 200 that has registered with the server displays, for example, the AR image as described above according to the optical ID obtained by decoding at the optimum frequency.

FIG. 68B is a flowchart illustrating an example of the processing operation of the receiver 200 in the present embodiment.

68. After registration with the server shown in FIG. 68A is performed, the receiver 200 transmits location information indicating the location where the receiver 200 exists to the server (step S161). Next, the receiver 200 acquires the optimum frequency registered in association with the location information from the server (step S162).

Next, the receiver 200 starts imaging (step S163), and decodes the decoding image obtained by the imaging according to the optimum frequency acquired in step S162 (step S164). The receiver 200 displays an AR image as described above, for example, according to the optical ID obtained by this decoding.

Thus, after registration to the server is performed, the receiver 200 can acquire the optimum frequency and receive the optical ID without executing the processing shown in FIG. 68A. Note that the receiver 200 may acquire the optimum frequency by executing the process shown in FIG. 68A when the optimum frequency cannot be obtained in step S162.

[Summary of Embodiment 4]
FIG. 69A is a flowchart showing a display method in the present embodiment.

The display method in the present embodiment is a display method in which the display device that is the above-described receiver 200 displays an image, and includes steps SL11 to SL16.

In step SL11, the image sensor captures an image of the subject to acquire a captured display image and a decoding image. In step SL12, the optical ID is acquired by decoding the decoding image. In step SL13, the optical ID is transmitted to the server. In step SL14, the AR image corresponding to the optical ID and the recognition information are acquired from the server. In step SL15, an area corresponding to the recognition information in the captured display image is recognized as a target area. In step SL16, the captured display image in which the AR image is superimposed on the target area is displayed.

Thereby, since the AR image is displayed superimposed on the captured display image, an image useful for the user can be displayed. Furthermore, it is possible to superimpose an AR image on an appropriate target area while suppressing the processing load.

That is, in general augmented reality (that is, AR), a huge number of recognition target images stored in advance are compared with the captured display image, and any of the recognition target images is included in the captured display image. It is determined whether or not If it is determined that the recognition target image is included, the AR image corresponding to the recognition target image is superimposed on the captured display image. At this time, the AR image is aligned based on the recognition target image. In this way, in general augmented reality, a huge number of recognition target images are compared with captured display images, and further, position detection of the recognition target images in the captured display image is necessary even in alignment. There is a problem of a large amount of calculation and a high processing load.

However, in the display method according to the present embodiment, as shown in FIGS. 41 to 68B, the light ID is acquired by decoding the decoding image obtained by imaging the subject. That is, the optical ID transmitted from the transmitter that is the subject is received. Furthermore, the AR image corresponding to this optical ID and the recognition information are acquired from the server. Therefore, the server does not need to compare a huge number of recognition target images and captured display images, and can select and transmit an AR image previously associated with the optical ID to the display device. Thereby, the amount of calculation can be reduced and the processing load can be significantly suppressed.

Further, in the display method in the present embodiment, the recognition information corresponding to this light ID is acquired from the server. The recognition information is information for recognizing a target area that is an area in which an AR image is superimposed in a captured display image. This recognition information may be information indicating that a white square is the target area, for example. In this case, the target area can be easily recognized, and the processing load can be further suppressed. That is, the processing load can be further suppressed according to the content of the recognition information. Further, since the server can arbitrarily set the content of the recognition information according to the optical ID, the balance between the processing load and the recognition accuracy can be appropriately maintained.

Here, the recognition information is reference information for specifying a reference area in the captured display image. In recognition of the target area, the reference area is specified from the captured display image based on the reference information, and the captured display image is displayed. Of these, the target area may be recognized based on the position of the reference area.

Alternatively, the recognition information may include reference information for specifying a reference area in the captured display image and target information indicating a relative position of the target area with respect to the reference area. In this case, in the recognition of the target area, the reference area is identified from the captured display image based on the reference information, and the area at the relative position indicated by the target information in the captured display image with the position of the reference area as a reference, Recognize as a target area.

Thereby, as shown in FIGS. 50 and 51, the degree of freedom of the position of the target area recognized in the captured display image can be expanded.

Further, the reference information is such that the position of the reference area in the captured display image is the same as the position of the bright line pattern area composed of a plurality of bright line patterns appearing by exposure of the multiple exposure lines of the image sensor in the decoding image. May be indicated.

Thereby, as shown in FIGS. 50 and 51, the target area can be recognized with reference to the area corresponding to the bright line pattern area in the captured display image.

Further, the reference information may indicate that the reference area in the captured display image is an area in which the display of the captured display image is displayed.

Thus, as shown in FIG. 41, for example, if a station name sign is used as a display, the target area can be recognized with reference to the area where the display is displayed.

Further, in the display of the captured display image, the first AR image is displayed for a predetermined display period while suppressing the display of the second AR image different from the first AR image that is the above-described AR image. May be.

As a result, as shown in FIG. 56, when the user is viewing the first AR image once displayed, the first AR image is immediately replaced with a different second AR image. Can be suppressed.

In the display of the captured display image, decoding of a newly acquired decoding image may be prohibited during the display period.

As a result, as shown in FIG. 56, decoding of the newly acquired decoding image is a wasteful process when the display of the second AR image is suppressed. Power consumption can be reduced.

In the display of the captured display image, the acceleration of the display device may be measured by an acceleration sensor during the display period, and it may be determined whether or not the measured acceleration is equal to or greater than a threshold value. And when it determines with more than a threshold value, you may display a 2nd AR image instead of a 1st AR image by canceling suppression of a display of the 2nd AR image.

Thereby, as shown in FIG. 56, when the acceleration of the display device equal to or higher than the threshold is measured, the suppression of the display of the second AR image is released. Accordingly, for example, when the user moves the display device greatly to direct the image sensor toward another subject, the second AR image can be displayed immediately.

Further, in the display of the captured display image, it may be further determined whether or not the user's face is approaching the display device by imaging with a face camera provided in the display device. And if it determines with the face approaching, you may display a 1st AR image, suppressing the display of the 2nd AR image different from a 1st AR image. Alternatively, in the display of the captured display image, whether or not the user's face is approaching the display device may be further determined based on the acceleration of the display device measured by the acceleration sensor. And if it determines with the face approaching, you may display a 1st AR image, suppressing the display of the 2nd AR image different from a 1st AR image.

As a result, as shown in FIG. 56, when the user is looking at the first AR image and the face is brought close to the display device, the first AR image is replaced with a different second AR image. That can be suppressed.

Further, as shown in FIG. 60, in the acquisition of the captured display image and the decoding image, the captured display image and the decoding image are acquired by capturing a plurality of displays each displaying an image as a subject. Also good. At this time, in the recognition of the target area, an area in which a transmission display that is a display that transmits the light ID among the plurality of displays appears in the captured display image is recognized as the target area. Further, in the display of the captured display image, the first subtitle corresponding to the image displayed on the transmission display is superimposed on the target area as an AR image, and further, in an area larger than the target area of the captured display image, The second subtitle, which is an expanded subtitle of the first subtitle, is superimposed.

Thereby, since the first subtitle is superimposed on the image on the transmission display, the user can easily grasp which display subtitle is the subtitle for the display image of the plurality of displays. it can. In addition, since the second subtitle, which is an enlarged subtitle of the first subtitle, is also displayed, even if the first subtitle is small and difficult to read, the subtitle can be easily read by displaying the second subtitle. can do.

Further, in the display of the captured display image, it is further determined whether or not audio information is included in the information acquired from the server. When it is determined that the information is included, the first and second subtitles are determined. The voice indicated by the voice information may be output with priority.

This allows the user to reduce the burden of reading subtitles because the audio is output preferentially.

FIG. 69B is a block diagram illustrating a configuration of the display device in the present embodiment.

The display device 10 according to the present embodiment is a display device that displays an image, and includes an image sensor 11, a decoding unit 12, a transmission unit 13, an acquisition unit 14, a recognition unit 15, and a display unit 16. Prepare. The display device 10 corresponds to the receiver 200 described above.

The image sensor 11 acquires a captured display image and a decoding image by imaging a subject. The decoding unit 12 acquires the optical ID by decoding the decoding image. The transmission unit 13 transmits the optical ID to the server. The acquisition unit 14 acquires the AR image corresponding to the optical ID and the recognition information from the server. The recognition unit 15 recognizes a region corresponding to the recognition information in the captured display image as a target region. The display unit 16 displays a captured display image in which an AR image is superimposed on the target area.

Thereby, since the AR image is displayed superimposed on the captured display image, an image useful for the user can be displayed. Furthermore, it is possible to superimpose an AR image on an appropriate target area while suppressing the processing load.

In the present embodiment, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, software for realizing the receiver 200 or the display device 10 according to the present embodiment is included in the flowcharts shown in FIGS. 45, 52, 56, 62, 65, and 68A to 69A. A program for causing a computer to execute steps.

[Modification 1 of Embodiment 4]
Hereinafter, a first modification of the fourth embodiment, that is, a first modification of the display method for realizing the AR using the optical ID will be described.

FIG. 70 is a diagram illustrating an example in which the receiver in the first modification of the fourth embodiment displays an AR image.

The receiver 200 acquires the above-described captured display image Pk, which is the normal captured image, and the above-described decoding image, which is the visible light communication image or the bright line image, by capturing the subject with the image sensor.

Specifically, the image sensor of the receiver 200 images the transmitter 100c configured as a robot and the person 21 adjacent to the transmitter 100c. The transmitter 100c is the transmitter according to any one of the first to third embodiments, and includes one or a plurality of light emitting elements (for example, LEDs) 131. The transmitter 100c changes its luminance by blinking the one or more light emitting elements 131, and transmits an optical ID (optical identification information) by the luminance change. This light ID is the above-mentioned visible light signal.

The receiver 200 acquires the captured display image Pk on which the transmitter 100c and the person 21 are imaged by the normal exposure time. Furthermore, the receiver 200 acquires the decoding image by capturing the transmitter 100c and the person 21 with the communication exposure time shorter than the normal exposure time.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100c. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P10 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pk as a target area. For example, the receiver 200 recognizes an area on the right side of the area where the robot that is the transmitter 100c is projected as the target area. Specifically, the receiver 200 specifies the distance between the two markers 132a and 132b of the transmitter 100c displayed in the captured display image Pk. Then, the receiver 200 recognizes an area having a width and a height corresponding to the distance as a target area. That is, the recognition information indicates the shape of the markers 132a and 132b and the position and size of the target region with reference to the markers 132a and 132b.

Then, the receiver 200 superimposes the AR image P10 on the target area, and displays the captured display image Pk on which the AR image P10 is superimposed on the display 201. For example, the receiver 200 acquires an AR image P10 indicating another robot different from the transmitter 100c. In this case, since the AR image P10 is superimposed on the target area of the captured display image Pk, the captured display image Pk can be displayed so that another robot actually exists next to the transmitter 100c. As a result, the person 21 can be photographed together with the other robot together with the transmitter 100c even if no other robot exists.

FIG. 71 is a diagram illustrating another example in which the receiver 200 according to the first modification of the fourth embodiment displays an AR image.

For example, as shown in FIG. 71, the transmitter 100 is configured as an image display device having a display panel, and transmits a light ID by changing the luminance while displaying a still image PS on the display panel. The display panel is, for example, a liquid crystal display or an organic EL (electroluminescence) display.

The receiver 200 acquires the captured display image Pm and the decoding image by imaging the transmitter 100 in the same manner as described above. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P11 and the recognition information corresponding to the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pm as a target area. For example, the receiver 200 recognizes the area where the display panel of the transmitter 100 is displayed as the target area. Then, the receiver 200 superimposes the AR image P11 on the target area, and displays the captured display image Pm on which the AR image P11 is superimposed on the display 201. For example, the AR image P11 is a moving image having the same or substantially the same picture as the still image PS displayed on the display panel of the transmitter 100 as the first picture in the display order. That is, the AR image P11 is a moving image that starts to move from the still image PS.

In this case, since the AR image P11 is superimposed on the target region of the captured display image Pm, the receiver 200 displays the captured display image Pm so that an image display device that displays a moving image actually exists. Can do.

FIG. 72 is a diagram illustrating another example in which the receiver 200 according to the first modification of the fourth embodiment displays an AR image.

For example, as shown in FIG. 72, the transmitter 100 is configured as a station name sign, and transmits a light ID by changing the luminance.

The receiver 200 images the transmitter 100 from a position away from the transmitter 100 as shown in FIG. Thereby, the receiver 200 acquires the captured display image Pn and the decoding image in the same manner as described above. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR images P12 to P14 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes two regions corresponding to the recognition information in the captured display image Pn as first and second target regions. For example, the receiver 200 recognizes the area around the transmitter 100 as the first target area. Then, the receiver 200 superimposes the AR image P12 on the first target area, and displays the captured display image Pn on which the AR image P12 is superimposed on the display 201. For example, the AR image P12 is an arrow that prompts the user of the receiver 200 to approach the transmitter 100.

In this case, since the AR image P12 is displayed superimposed on the first target area of the captured display image Pn, the user approaches the transmitter 100 with the receiver 200 facing the transmitter 100. As the receiver 200 approaches the transmitter 100, the area of the transmitter 100 (corresponding to the reference area described above) displayed in the captured display image Pn becomes larger. When the size of the area becomes equal to or larger than the first threshold, the receiver 200 further moves to a second target area, which is an area where the transmitter 100 is projected, as shown in FIG. 72 (b), for example. The AR image P13 is superimposed. That is, the receiver 200 displays the captured display image Pn on which the AR images P12 and P13 are superimposed on the display 201. For example, the AR image P13 is a message that informs the user of the outline of the vicinity of the station indicated by the station name sign. The AR image P13 is equal to the size of the area of the transmitter 100 displayed in the captured display image Pn.

Also in this case, since the AR image P12 that is an arrow is superimposed and displayed on the first target area of the captured display image Pn, the user transmits the transmitter 200 with the receiver 200 facing the transmitter 100. Approaching 100. As the receiver 200 approaches the transmitter 100, the area of the transmitter 100 (corresponding to the reference area described above) displayed in the captured display image Pn becomes larger. When the size of the area becomes equal to or larger than the second threshold, the receiver 200 changes the AR image P13 superimposed on the second target area to the AR image P14, for example, as illustrated in FIG. To do. Furthermore, the receiver 200 deletes the AR image P12 superimposed on the first target area.

That is, the receiver 200 displays the captured display image Pn on which the AR image P14 is superimposed on the display 201. For example, the AR image P14 is a message that informs the user of the details around the station indicated by the station name sign. The AR image P14 is equal to the size of the area of the transmitter 100 displayed in the captured display image Pn. The area of the transmitter 100 is larger as the receiver 200 is closer to the transmitter 100. Therefore, the AR image P14 is larger than the AR image P13.

Thus, the receiver 200 enlarges the AR image and displays more information as it approaches the transmitter 100. In addition, since an arrow that prompts the user to approach, such as the AR image P12, is displayed, the user can easily grasp that a large amount of information is displayed when approaching the transmitter 100.

FIG. 73 is a diagram illustrating another example in which the receiver 200 in the first modification of the fourth embodiment displays an AR image.

In the example shown in FIG. 72, the receiver 200 displays a lot of information when approaching the transmitter 100, but displays a lot of information in the form of, for example, a balloon regardless of the distance to the transmitter 100. Also good.

Specifically, as shown in FIG. 73, the receiver 200 captures the captured display image Po and the decoding image by capturing an image of the transmitter 100 as described above. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P15 and the recognition information corresponding to the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Po as a target area. For example, the receiver 200 recognizes an area around the transmitter 100 as a target area. Then, the receiver 200 superimposes the AR image P15 on the target area, and displays the captured display image Po on which the AR image P15 is superimposed on the display 201. For example, the AR image P15 is a message that informs the user of the details of the vicinity of the station indicated by the station name in a balloon form.

In this case, since the AR image P15 is superimposed on the target area of the captured display image Po, the user of the receiver 200 can display a lot of information on the receiver 200 without approaching the transmitter 100.

FIG. 74 is a diagram illustrating another example of the receiver 200 in the first modification of the fourth embodiment.

The receiver 200 is configured as a smartphone in the above-described example, but may be configured as a head-mounted display (also referred to as glass) including an image sensor, as in the example illustrated in FIG.

Such a receiver 200 acquires the optical ID by performing decoding only on a part of the decoding target area of the decoding image. For example, the receiver 200 includes a line-of-sight detection camera 203 as shown in FIG. The line-of-sight detection camera 203 images the eyes of the user wearing the head-mounted display that is the receiver 200. The receiver 200 detects the user's line of sight based on the eye image obtained by the imaging by the line-of-sight detection camera 203.

As shown in FIG. 74B, the receiver 200 displays the line-of-sight frame 204 so that the line-of-sight frame 204 appears in an area of the user's field of view where the detected line of sight is directed, for example. . Accordingly, the line-of-sight frame 204 moves according to the movement of the user's line of sight. The receiver 200 treats an area corresponding to the line-of-sight frame 204 in the decoding image as a decoding target area. In other words, the receiver 200 does not decode the bright line pattern area in the decoding image even if there is a bright line pattern area outside the decoding target area, and performs decoding only on the bright line pattern area in the decoding target area. . As a result, even when there are a plurality of bright line pattern areas in the decoding image, decoding is not performed for all of the bright line pattern areas, so that the processing load can be reduced and the display of an extra AR image can be suppressed. Can do.

In addition, when a plurality of bright line pattern areas for outputting sound are included in the decoding image, the receiver 200 decodes only the bright line pattern area in the decoding target area, and the bright line pattern area Only the sound corresponding to may be output. Alternatively, the receiver 200 decodes each of the plurality of bright line pattern regions included in the decoding image, outputs a large amount of sound corresponding to the bright line pattern region in the decoding target region, and outputs to the bright line pattern region outside the decoding target region. The corresponding voice may be output small. In addition, when there are a plurality of bright line pattern areas outside the decoding target area, the receiver 200 may output a larger amount of sound corresponding to the bright line pattern area as the bright line pattern area is closer to the decoding target area.

FIG. 75 is a diagram illustrating another example in which the receiver 200 in the first modification of the fourth embodiment displays an AR image.

For example, as shown in FIG. 75, the transmitter 100 is configured as an image display device having a display panel, and transmits an optical ID by changing luminance while displaying an image on the display panel.

The receiver 200 acquires the captured display image Pp and the decoding image in the same manner as described above by imaging the transmitter 100.

At this time, the receiver 200 specifies, from the captured display image Pp, an area having the same position as the bright line pattern area in the decoding image and the same size as the bright line pattern area. The receiver 200 may display the scanning line P100 that repeatedly moves from one end of the region to the other end.

While the scanning line P100 is displayed, the receiver 200 acquires the optical ID by decoding the decoding image and transmits the optical ID to the server. Then, the receiver 200 acquires the AR image corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pp as a target area.

When recognizing such a target area, the receiver 200 ends the display of the scanning line P100, superimposes the AR image on the target area, and displays the captured display image Pp on which the AR image is superimposed on the display 201. .

Accordingly, since the moving scanning line P100 is displayed from when the transmitter 100 is imaged until the AR image is displayed, processing such as reading of the optical ID and the AR image is performed. Can be notified to the user.

FIG. 76 is a diagram illustrating another example in which the receiver 200 in the first modification of the fourth embodiment displays an AR image.

Each of the two transmitters 100 is configured as an image display device having a display panel, for example, as shown in FIG. 76, and transmits an optical ID by changing the luminance while displaying the same still image PS on the display panel. is doing. Here, the two transmitters 100 transmit different optical IDs (for example, optical IDs “01” and “02”) by changing the luminance in different manners.

The receiver 200 acquires the captured display image Pq and the decoding image by capturing images of the two transmitters 100 as in the example illustrated in FIG. The receiver 200 acquires the optical IDs “01” and “02” by decoding the decoding image. That is, the receiver 200 receives the optical ID “01” from one of the two transmitters 100 and receives the optical ID “02” from the other. The receiver 200 transmits those optical IDs to the server. Then, the receiver 200 acquires the AR image P16 corresponding to the optical ID “01” and the recognition information from the server. Furthermore, the receiver 200 acquires the AR image P17 corresponding to the optical ID “02” and the recognition information from the server.

The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pq as a target area. For example, the receiver 200 recognizes an area where the display panels of the two transmitters 100 are displayed as the target area. Then, the receiver 200 superimposes the AR image P16 on the target area corresponding to the light ID “01”, and superimposes the AR image P17 on the target area corresponding to the light ID “02”. Then, the receiver 200 displays the captured display image Pq on which the AR images P16 and P17 are superimposed on the display 201. For example, the AR image P16 is a moving image having the same or substantially the same picture as the still picture PS displayed on the display panel of the transmitter 100 corresponding to the optical ID “01” as the first picture in the display order. is there. The AR image P17 is a moving image having the same or substantially the same picture as the still picture PS displayed on the display panel of the transmitter 100 corresponding to the optical ID “02” as the first picture in the display order. is there. That is, the leading pictures of the AR image P16 and the AR image P17, which are moving images, are the same. However, the AR image P16 and the AR image P17 are different moving images, and the pictures other than the head of each are different.

Therefore, since the AR image P16 and the AR image P17 are superimposed on the captured display image Pq, the receiver 200 actually has an image display device that displays different moving images reproduced from the same picture. The captured display image Pq can be displayed.

FIG. 77 is a flowchart illustrating an example of a processing operation of the receiver 200 in the first modification of the fourth embodiment. The processing operation shown by the flowchart of FIG. 77 is an example of the processing operation of the receiver 200 that individually images each of the transmitters 100 when there are two transmitters 100 shown in FIG. is there.

First, the receiver 200 acquires the first light ID by imaging the first transmitter 100 as the first subject (step S201). Next, the receiver 200 recognizes the first subject from the captured display image (step S202). That is, the receiver 200 acquires the first AR image and the first recognition information corresponding to the first light ID from the server, and recognizes the first subject based on the first recognition information. Then, the receiver 200 starts reproduction of the first moving image that is the first AR image from the beginning (step S203). That is, the receiver 200 starts reproduction from the first picture of the first moving image.

Here, the receiver 200 determines whether or not the first subject is out of the captured display image (step S204). That is, the receiver 200 determines whether or not the first subject cannot be recognized from the captured display image. Here, if it is determined that the first subject has deviated from the captured display image (Y in step S204), the receiver 200 interrupts the reproduction of the first moving image that is the first AR image (step S205).

Next, the receiver 200 captures a second transmitter 100 different from the first transmitter 100 as a second subject, thereby different from the first light ID acquired in step S201. It is determined whether or not the optical ID of the first one has been acquired (step S206). Here, when the receiver 200 determines that the second optical ID has been acquired (Y in step S206), the receiver 200 performs the same processing as steps S202 to S203 after the first optical ID is acquired. That is, the receiver 200 recognizes the second subject from the captured display image (step S207). Then, the receiver 200 starts reproduction of the second moving image that is the second AR image corresponding to the second optical ID from the beginning (step S208). That is, the receiver 200 starts playback from the first picture of the second moving image.

On the other hand, if the receiver 200 determines in step S206 that the second light ID has not been acquired (N in step S206), it determines whether or not the first subject has entered the captured display image again (step S206). S209). That is, the receiver 200 determines whether or not the first subject is recognized again from the captured display image. Here, when the receiver 200 determines that the first subject has entered the captured display image (Y in step S209), the receiver 200 further determines whether or not a predetermined time (that is, a predetermined time) has passed (step S209). Step S210). That is, the receiver 200 determines whether or not a predetermined time has elapsed from when the first subject is removed from the captured display image until it enters again. If it is determined that the predetermined time has not elapsed (Y in step S210), the receiver 200 starts reproduction from the middle of the interrupted first moving image (step S211). It should be noted that the first picture to be resumed for reproduction, which is the first picture of the first moving picture that is displayed at the beginning of the reproduction from the middle, is the next picture that was last displayed when the reproduction of the first moving picture was interrupted. The pictures may be in the display order. Alternatively, the reproduction restart top picture may be a picture preceding the last displayed picture by n (n is an integer of 1 or more) in display order.

On the other hand, if it is determined that the predetermined time has elapsed (N in step S210), the receiver 200 starts playback of the interrupted first moving image from the beginning (step S212).

In the above example, the receiver 200 superimposes the AR image on the target area of the captured display image. At this time, the brightness of the AR image may be adjusted. That is, the receiver 200 determines whether or not the brightness of the AR image acquired from the server matches the brightness of the target area of the captured display image. If the receiver 200 determines that they do not match, the receiver 200 adjusts the brightness of the AR image to match the brightness of the AR image with the brightness of the target region. Then, the receiver 200 superimposes the AR image whose brightness has been adjusted on the target area of the captured display image. Thereby, the superimposed AR image can be brought closer to the image of the actual object, and the user's uncomfortable feeling with respect to the AR image can be suppressed. Note that the brightness of the AR image is the spatial average brightness of the AR image, and the brightness of the target area is also the spatial average brightness of the target area.

Further, as shown in FIG. 53, when the AR image is tapped, the receiver 200 may enlarge the AR image and display it on the entire display 201. In the example illustrated in FIG. 53, the receiver 200 switches the AR image on which the AR image is tapped to another AR image, but may automatically switch the AR image regardless of the tap. For example, when the AR image is displayed for a predetermined time, the receiver 200 switches the AR image to another AR image for display. Further, when the current time reaches a predetermined time, the receiver 200 switches the AR image that has been displayed so far to another AR image and displays it. Thereby, the user can easily see a new AR image without performing an operation.

[Modification 2 of Embodiment 4]
Hereinafter, a second modification of the fourth embodiment, that is, a second modification of the display method for realizing the AR using the optical ID will be described.

FIG. 78 is a diagram illustrating an example of a problem when displaying an AR image assumed in the receiver 200 according to the fourth embodiment or the modification 1 thereof.

For example, the receiver 200 in the fourth embodiment or its modification example 1 captures the subject at time t1. Note that the above-described subject is a transmitter such as a television that transmits a light ID according to a change in luminance, or a poster, a guide board, or a signboard illuminated by light from the transmitter. As a result, the receiver 200 displays the entire image obtained by the effective pixel area of the image sensor (hereinafter, referred to as all captured images) on the display 201 as a captured display image. At this time, the receiver 200 recognizes, in the captured display image, an area corresponding to the recognition information acquired based on the light ID as a target area on which the AR image is superimposed. The target area is an area indicating an image of a transmitter such as a television or an image of a poster, for example. Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201. The AR image may be a still image or a moving image, or a character string including one or more characters or symbols.

Here, when the user of the receiver 200 approaches the subject in order to display the AR image in a large size, a region corresponding to the target region in the image sensor (hereinafter referred to as a recognition region) protrudes from the effective pixel region at time t2. The recognition area is an area in which an image of the target area in the captured display image is projected in the effective pixel area of the image sensor. That is, the effective pixel area and the recognition area in the image sensor correspond to the captured display image and the target area on the display 201, respectively.

When the recognition area protrudes from the effective pixel area, the receiver 200 cannot recognize the target area from the captured display image and cannot display the AR image.

Therefore, the receiver 200 in the present modification acquires an image having a wider angle of view than the captured display image displayed on the entire display 201 as the entire captured image.

FIG. 79 is a diagram illustrating an example in which the receiver 200 in the second modification of the fourth embodiment displays an AR image.

The field angle of all captured images of the receiver 200 according to this modification, that is, the field angle of the effective pixel area of the image sensor is wider than the field angle of the captured display image displayed on the entire display 201. In the image sensor, an area corresponding to an image range displayed on the display 201 is hereinafter referred to as a display area.

For example, the receiver 200 images the subject at time t1. As a result, the receiver 200 displays, on the display 201, only the image obtained by the display area narrower than the effective pixel area among all the captured images obtained by the effective pixel area of the image sensor as the captured display image. At this time, similarly to the above, the receiver 200 recognizes an area corresponding to the recognition information acquired based on the light ID among all the captured images as a target area on which the AR image is superimposed. Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201.

Here, when the user of the receiver 200 approaches the subject in order to display the AR image in a large size, the recognition area in the image sensor is expanded. At time t2, the recognition area protrudes from the display area in the image sensor. That is, an image of the target region (for example, a poster image) protrudes from the captured display image displayed on the display 201. However, the recognition area in the image sensor does not protrude from the effective pixel area. That is, the receiver 200 acquires all captured images including the target area even at time t2. As a result, the receiver 200 can recognize the target area from all the captured images, and superimposes a part of the AR image corresponding to the target area only on a part of the target area in the captured display image. Are displayed on the display 201.

Thereby, even when the user approaches the subject to display the AR image in a large size and the target region protrudes from the captured display image, the display of the AR image can be continued.

FIG. 80 is a flowchart illustrating an example of a processing operation of the receiver 200 in the second modification of the fourth embodiment.

The receiver 200 acquires the entire captured image and the decoding image by the image sensor capturing the subject (step S301). Next, the receiver 200 acquires an optical ID by decoding the decoding image (step S302). Next, the receiver 200 transmits the optical ID to the server (step S303). Next, the receiver 200 acquires an AR image and recognition information corresponding to the optical ID from the server (step S304). Next, the receiver 200 recognizes an area corresponding to the recognition information among all captured images as a target area (step S305).

Here, the receiver 200 determines whether or not the recognition area that is the area corresponding to the image of the target area in the effective pixel area of the image sensor protrudes from the display area (step S306). Here, if it is determined that it is protruding (Yes in step S306), the receiver 200 extracts a part of the AR image corresponding to the area only in a part of the target area within the captured display image. It is displayed (step S307). On the other hand, when the receiver 200 determines that it does not protrude (No in step S306), the receiver 200 superimposes the AR image on the target area of the captured display image and displays the captured display image on which the AR image is superimposed. (Step S308).

Then, the receiver 200 determines whether or not the AR image display process should be terminated (step S309), and when it is determined that the AR image display process should not be terminated (No in step S309), the process from step S305 is repeatedly executed.

FIG. 81 is a diagram illustrating another example in which the receiver 200 in the second modification of the fourth embodiment displays an AR image.

The receiver 200 may switch the screen display of the AR image according to the ratio of the size of the recognition area to the display area.

When the horizontal width of the display area of the image sensor is w1, the vertical width is h1, the horizontal width of the recognition area is w2, and the vertical width is h2, the receiver uses the ratio (h2 / h1). ) And (w2 / w1), the larger ratio is compared with the threshold value.

For example, as shown in (screen display 1) in FIG. 81, the receiver 200 displays the captured display image in which the AR image is superimposed on the target region, and sets the larger ratio to the first threshold ( For example, compare with 0.9). When the larger ratio becomes 0.9 or more, the receiver 200 enlarges and displays the AR image on the entire display 201 as shown in (screen display 2) of FIG. Note that when the recognition area becomes larger than the display area and further when the recognition area becomes larger than the effective pixel area, the receiver 200 continues to enlarge and display the AR image on the entire display 201.

Further, for example, when the AR image is enlarged and displayed on the entire display 201 as shown in (screen display 2) of FIG. 81, the receiver 200 sets the larger ratio to the second threshold ( For example, compare with 0.7). The second threshold is smaller than the first threshold. When the larger ratio becomes 0.7 or less, the receiver 200 displays a captured display image in which the AR image is superimposed on the target area as shown in (screen display 1) of FIG.

FIG. 82 is a flowchart illustrating another example of the processing operation of the receiver 200 in the second modification of the fourth embodiment.

First, the receiver 200 performs optical ID processing (step S301a). This optical ID process is a process including steps S301 to S304 shown in FIG. Next, the receiver 200 recognizes an area corresponding to the recognition information in the captured display image as a target area (step S311). Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed (step S312).

Next, the receiver 200 has a recognition area ratio, that is, a larger ratio of the ratios (h2 / h1) and (w2 / w1) is equal to or greater than a first threshold value K (for example, K = 0.9). It is determined whether or not (step S313). If it is determined that the value is not equal to or greater than the first threshold value K (No in step S313), the receiver 200 repeatedly executes the processing from step S311. On the other hand, if it determines with it being more than the 1st threshold value K (Yes of step S313), the receiver 200 will expand and display the AR image on the entire display 201 (step S314). At this time, the receiver 200 periodically switches the power supply of the image sensor between on and off. Power saving of the receiver 200 can be achieved by periodically turning off the power supply of the image sensor.

Next, the receiver 200 determines whether or not the recognition area ratio is equal to or smaller than a second threshold L (for example, L = 0.7) when the image sensor is periodically turned on. To do. Here, if it is determined that it is not equal to or less than the second threshold L (No in step S315), the receiver 200 repeatedly executes the processing from step S314. On the other hand, if it is determined that it is equal to or smaller than the second threshold value L (Yes in step S315), the receiver 200 superimposes the AR image on the target area of the captured display image and displays the captured display image on which the AR image is superimposed. (Step S316).

Then, the receiver 200 determines whether or not to end the AR image display process (step S317). If the receiver 200 determines that the AR image display process should not be ended (No in step S317), the receiver 200 repeatedly executes the processes from step S313.

Thus, by setting the second threshold value L to be smaller than the first threshold value K, the screen display of the receiver 200 can be frequently switched between (screen display 1) and (screen display 2). Can be prevented and the state of the screen display can be stabilized.

In the example shown in FIGS. 81 and 82, the display area and the effective pixel area may be the same or different. In this example, the ratio of the size of the recognition area to the display area is used. However, if the display area and the effective pixel area are different, the size of the recognition area relative to the effective pixel area is used instead of the display area. A ratio may be used.

FIG. 83 is a diagram illustrating another example in which the receiver 200 according to the second modification of the fourth embodiment displays an AR image.

83, in the same manner as the example shown in FIG. 79, the image sensor of the receiver 200 has an effective pixel area wider than the display area.

For example, the receiver 200 images the subject at time t1. As a result, the receiver 200 displays, on the display 201, only the image obtained by the display area narrower than the effective pixel area among all the captured images obtained by the effective pixel area of the image sensor as the captured display image. At this time, similarly to the above, the receiver 200 recognizes an area corresponding to the recognition information acquired based on the light ID among all the captured images as a target area on which the AR image is superimposed. Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201.

Here, when the user changes the direction of the receiver 200 (specifically, the image sensor), the recognition area in the image sensor moves, for example, in the upper left direction in FIG. 83, and protrudes from the display area at time t2. That is, an image of the target region (for example, a poster image) protrudes from the captured display image displayed on the display 201. However, the recognition area in the image sensor does not protrude from the effective pixel area. That is, the receiver 200 acquires all captured images including the target area even at time t2. As a result, the receiver 200 can recognize the target area from all the captured images, and superimposes a part of the AR image corresponding to the area only on a part of the target area in the captured display image. And displayed on the display 201. Furthermore, the receiver 200 changes the size and position of a part of the displayed AR image in accordance with the movement of the recognition area in the image sensor, that is, the movement of the target area in all captured images.

When the recognition area protrudes from the display area as described above, the receiver 200 counts the number of pixels corresponding to the distance between the edge of the effective pixel area and the edge of the display area (hereinafter referred to as inter-area distance). Is compared to a threshold.

For example, the shorter distance (hereinafter referred to as the first distance) among the distance between the upper side of the effective pixel area and the upper side of the display area, and the distance between the lower side of the effective pixel area and the lower side of the display area. Let dh be the number of pixels corresponding to (distance). Further, the shorter distance (hereinafter referred to as the second distance) among the distance between the left side of the effective pixel area and the left side of the display area, and the distance between the right side of the effective pixel area and the right side of the display area. Let dw be the number of pixels corresponding to (distance). At this time, the above-mentioned inter-region distance is the shorter one of the first and second distances.

That is, the receiver 200 compares the smaller pixel number of the pixel numbers dw and dh with the threshold value N. Then, for example, when the smaller number of pixels becomes equal to or smaller than the threshold value N at time t2, the receiver 200 changes the size and position of a part of the AR image according to the position of the recognition area in the image sensor. Fix without. That is, the receiver 200 switches the screen display of the AR image. For example, the receiver 200 sets the size and position of a part of the displayed AR image to the size of the part of the AR image displayed on the display 201 when the smaller number of pixels reaches the threshold value N. Fix in position and position.

Therefore, even when the recognition area further moves and protrudes from the effective pixel area at time t3, the receiver 200 continues to display a part of the AR image similarly to time t2. That is, as long as the smaller number of pixels dw and dh is equal to or smaller than the threshold value N, the receiver 200 captures a part of the AR image whose size and position are fixed as at time t2. Continue to superimpose on the display image.

In the example shown in FIG. 83, the receiver 200 changes the size and position of a part of the displayed AR image in accordance with the movement of the recognition area in the image sensor, but the display magnification and position of the entire AR image are changed. It may be changed.

FIG. 84 is a diagram illustrating another example in which the receiver 200 according to the second modification of the fourth embodiment displays an AR image. Specifically, FIG. 84 shows an example in which the display magnification of the AR image is changed.

For example, as in the example shown in FIG. 83, when the user changes the orientation of the receiver 200 (specifically, the image sensor) from the state at time t1, the recognition area in the image sensor is, for example, in the upper left direction in FIG. It moves and protrudes from the display area at time t2. That is, an image of the target region (for example, a poster image) protrudes from the captured display image displayed on the display 201. However, the recognition area in the image sensor does not protrude from the effective pixel area. That is, the receiver 200 acquires all captured images including the target area even at time t2. As a result, the receiver 200 can recognize the target area from all captured images.

Therefore, in the example illustrated in FIG. 84, the receiver 200 displays the AR image display magnification so that the size of the entire AR image matches the size of a part of the target region in the captured display image. To change. That is, the receiver 200 reduces the AR image. Then, the receiver 200 superimposes the AR image whose display magnification has been changed (that is, reduced) on the area and displays the AR image on the display 201. Furthermore, the receiver 200 changes the display magnification and position of the displayed AR image in accordance with the movement of the recognition area in the image sensor, that is, the movement of the target area in all captured images.

In addition, when the recognition area protrudes from the display area as described above, the receiver 200 compares the smaller pixel number of the pixel numbers dw and dh with the threshold value N. Then, for example, if the smaller number of pixels becomes equal to or less than the threshold value N at time t2, the receiver 200 is fixed without changing the display magnification and position of the AR image according to the position of the recognition area in the image sensor. To do. That is, the receiver 200 switches the screen display of the AR image. For example, the receiver 200 fixes the display magnification and position of the displayed AR image to the display magnification and position of the AR image displayed on the display 201 when the smaller number of pixels reaches the threshold value N. .

Therefore, the receiver 200 continues to display the AR image similarly to the time t2 even when the recognition area further moves at the time t3 and protrudes from the effective pixel area. That is, as long as the smaller number of pixels dw and dh is equal to or smaller than the threshold value N, the receiver 200 converts an AR image with a fixed display magnification and position into a captured display image as at time t2. Continue to overlay and display.

In the above example, the smaller of the number of pixels dw and dh is compared with the threshold value, but the ratio of the smaller number of pixels may be compared with the threshold value. The ratio of the number of pixels dw is, for example, the ratio of the number of pixels dw to the number of pixels w0 in the horizontal direction of the effective pixel region (dw / w0). Similarly, the ratio of the number of pixels dh is, for example, the ratio of the number of pixels dh to the number of pixels h0 in the vertical direction of the effective pixel region (dh / h0). Alternatively, the ratio between the pixel numbers dw and dh may be expressed by using the number of pixels in the horizontal or vertical direction of the display area instead of the number of pixels in the horizontal or vertical direction of the effective pixel area. The threshold value compared with the ratio of the number of pixels dw and dh is, for example, 0.05.

Also, the smaller angle of view of the number of pixels dw and dh may be compared with a threshold value. When the number of diagonal pixels in the effective pixel area is m and the angle of view corresponding to the diagonal is θ (for example, 55 °), the angle of view corresponding to the number of pixels dw is θ × dw / m, The angle of view corresponding to the number of pixels dh is θ × dh / m.

In the example shown in FIGS. 83 and 84, the receiver 200 switches the screen display of the AR image based on the inter-region distance between the effective pixel region and the recognition region. The screen display of the AR image may be switched based on the relationship.

FIG. 85 is a diagram illustrating another example in which the receiver 200 in the second modification of the fourth embodiment displays an AR image. Specifically, FIG. 85 shows an example in which the screen display of the AR image is switched based on the relationship between the display area and the recognition area. In the example shown in FIG. 85, as in the example shown in FIG. 79, the image sensor of the receiver 200 has an effective pixel area wider than the display area.

For example, the receiver 200 images the subject at time t1. As a result, the receiver 200 displays, on the display 201, only the image obtained by the display area narrower than the effective pixel area among all the captured images obtained by the effective pixel area of the image sensor as the captured display image. At this time, similarly to the above, the receiver 200 recognizes an area corresponding to the recognition information acquired based on the light ID among all the captured images as a target area on which the AR image is superimposed. Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201.

Here, when the user changes the direction of the receiver 200, the receiver 200 changes the position of the displayed AR image according to the movement of the recognition area in the image sensor. For example, the recognition area in the image sensor moves in the upper left direction in FIG. 85, for example, and at time t2, a part of the edge of the recognition area coincides with a part of the edge of the display area. That is, an image of the target area (for example, an image such as a poster) is arranged at the corner of the captured display image displayed on the display 201. As a result, the receiver 200 superimposes the AR image on the target area at the corner of the captured display image and displays the AR image on the display 201.

When the recognition area further moves and protrudes from the display area, the receiver 200 fixes the AR image displayed at time t2 without changing the size and position. That is, the receiver 200 switches the screen display of the AR image.

Therefore, the receiver 200 continues to display the AR image similarly to the time t2 even when the recognition area further moves at the time t3 and protrudes from the effective pixel area. In other words, as long as the recognition area extends beyond the display area, the receiver 200 superimposes the AR image having the same size as that at time t2 on the same position as at time t2 in the captured display image. Continue to display.

Thus, in the example shown in FIG. 85, the receiver 200 switches the screen display of the AR image depending on whether or not the recognition area protrudes from the display area. Further, the receiver 200 may include a determination area that includes the display area and is larger than the display area and smaller than the effective pixel area instead of the display area. In this case, the receiver 200 switches the screen display of the AR image depending on whether or not the recognition area protrudes from the determination area.

As described above, the screen display of the AR image has been described with reference to FIGS. 79 to 85. However, when the receiver 200 cannot recognize the target area from all the captured images, the target that has been recognized until immediately before the target area is recognized. The AR image having the size of the area may be displayed superimposed on the captured display image.

FIG. 86 is a diagram illustrating another example in which the receiver 200 according to the second modification of the fourth embodiment displays an AR image.

Note that, in the example illustrated in FIG. 49, the receiver 200 acquires the captured display image Pe and the decoding image in the same manner as described above by imaging the guide plate 107 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the guide plate 107. However, if the entire surface of the guide plate 107 is a color that absorbs light (for example, dark color), the receiver 200 receives the light ID correctly because the surface is dark even when illuminated by the transmitter 100. It may not be possible. Alternatively, even if the entire surface of the guide plate 107 has a striped pattern such as a decoding image (that is, a bright line image), the receiver 200 may not be able to correctly receive the light ID.

Therefore, as shown in FIG. 86, a reflecting plate 109 may be arranged near the guide plate 107. Thereby, the receiver 200 can receive light reflected from the transmitter 100 by the reflector 109, that is, visible light (specifically, light ID) transmitted from the transmitter 100. As a result, the receiver 200 can appropriately receive the optical ID and display the AR image P5.

[Summary of Modifications 1 and 2 of Embodiment 4]
FIG. 87A is a flowchart illustrating a display method according to one embodiment of the present invention.

The display method according to one aspect of the present invention includes steps S41 to S43.

In step S41, a picked-up image is acquired by picking up an image of an object illuminated by a transmitter that transmits a signal according to a change in luminance of light as a subject with an image sensor. In step S42, a signal is decoded from the captured image. In step S43, a moving image corresponding to the decoded signal is read from the memory, and the moving image is superimposed on a target area corresponding to the subject in the captured image and displayed on the display. Here, in step S43, a predetermined image that is one of a plurality of images included in the moving image and includes display images of the image including the target object and the image including the target object. The moving image is displayed from any one of the plurality of images. For example, the predetermined number is 10 frames. Alternatively, the object is a still image, and in step S43, the moving image is displayed from the same image as the still image. The image from which the display of the moving image is started is not limited to the same image as the still image, but is the same number as the still image, that is, the predetermined number of frames in the display order from the image including the target object. It may be an image. Further, the object is not limited to a still image, and may be a doll or the like.

Note that the imaging sensor and the captured image are, for example, the image sensor and the entire captured image in the fourth embodiment. The still image to be lit up may be a still image displayed on the display panel of the image display device, or may be a poster, a guide board, a signboard, or the like illuminated by light from the transmitter.

Further, such a display method may further include a transmission step of transmitting a signal to the server and a reception step of receiving a moving image corresponding to the signal from the server.

Thus, for example, as shown in FIG. 71, a moving image can be displayed virtually so that a still image starts to move, and an image useful for the user can be displayed.

Further, the still image has an outer frame of a predetermined color, and the display method according to one aspect of the present invention may further include a recognition step of recognizing a target area from the captured image by the predetermined color. In this case, in step S43, the moving image is resized so as to be the same as the size of the recognized target region, and the resized moving image is superimposed on the target region in the captured image and displayed on the display. Good. For example, the outer frame of the predetermined color is a white or black rectangular frame surrounding a still image, and is indicated by the recognition information in the fourth embodiment. Then, the AR image in the fourth embodiment is resized and superimposed as a moving image.

Thereby, the moving image can be displayed more realistically so that the moving image actually exists as a subject.

Further, only the image projected on the display area which is smaller than the imaging area among the imaging areas of the imaging sensor is displayed on the display. In this case, in step S43, if the projection area onto which the subject is projected in the imaging area is larger than the display area, an image obtained by a portion of the projection area that exceeds the display area is displayed on the display. You don't have to. Here, for example, as shown in FIG. 79, the imaging region and the projection region are an effective pixel region and a recognition region of the image sensor.

As a result, for example, as shown in FIG. 79, the imaging sensor approaches the still image that is the subject, so that even if a part of the image obtained by the projection region (recognition region in FIG. 79) is not displayed on the display, the subject In some cases, the entire still image is projected onto the imaging region. Therefore, in this case, a still image that is a subject can be appropriately recognized, and a moving image can be appropriately superimposed on a target region corresponding to the subject in the captured image.

For example, the horizontal and vertical widths of the display area are w1 and h1, and the horizontal and vertical widths of the projection area are w2 and h2. In this case, in step S43, when h2 / h1 or w2 / w1 is greater than a predetermined value, a moving image is displayed on the entire screen of the display, and either h2 / h1 or w2 / w1 is displayed. If the larger value is smaller than the predetermined value, the moving image may be superimposed on the target area in the captured image and displayed on the display.

As a result, for example, as shown in FIG. 81, when the imaging sensor approaches the still image that is the subject, the moving image is displayed on the full screen. Therefore, the user enlarges the moving image by moving the imaging sensor closer to the still image. There is no need to display. Therefore, it is possible to prevent the signal from being unable to be decoded when the imaging sensor is too close to the still image and the projection area (recognition area in FIG. 81) protrudes from the imaging area (effective pixel area). .

In addition, the display method according to one aspect of the present invention may further include a control step of turning off the operation of the imaging sensor when a moving image is displayed on the entire screen of the display.

Thereby, for example, as shown in step S314 in FIG. 82, the power consumption of the image sensor can be suppressed by turning off the operation of the image sensor.

In step S43, if the target area cannot be recognized from the captured image due to the movement of the imaging sensor, the moving image is displayed in the same size as the size of the target area recognized immediately before it cannot be recognized. May be. Note that the target area cannot be recognized from the captured image, for example, is a situation in which at least a part of the target area corresponding to the still image that is the subject is not included in the captured image. Thus, when the target area cannot be recognized, a moving image having the same size as the size of the target area recognized immediately before is displayed, for example, at time t3 in FIG. Accordingly, it is possible to suppress the at least part of the moving image from being displayed because the imaging sensor has been moved.

Further, in step S43, when only a part of the target area is included in the area displayed on the display of the captured image due to the movement of the imaging sensor, it corresponds to a part of the target area. A part of the spatial area of the moving image to be displayed may be superimposed on a part of the target area and displayed on the display. A part of the spatial area of the moving image is a part of each picture constituting the moving image.

Thus, only a part of the spatial area of the moving image (AR image in FIG. 83) is displayed on the display, for example, at time t2 in FIG. As a result, the user can be informed that the image sensor is not properly directed to the still image that is the subject.

In step S43, if the target area cannot be recognized from the captured image due to the movement of the imaging sensor, the space of the moving image corresponding to a part of the target area displayed immediately before the target area cannot be recognized. A part of the area may be continuously displayed.

Thus, even when the user points the imaging sensor in a direction different from the still image as the subject, for example, at time t3 in FIG. 83, one space area of the moving image (AR image in FIG. 83) is displayed. The part is displayed continuously. As a result, it is possible to make it easier for the user to know how the image sensor is directed to display the entire moving image.

In step S43, the horizontal and vertical widths in the imaging area of the imaging sensor are w0 and h0, respectively, and the horizontal area between the projection area where the subject is projected in the imaging area and the imaging area. When the distances in the direction and the vertical direction are dh and dw, it is determined that the target region cannot be recognized when the smaller one of dw / w0 and dh / h0 is equal to or smaller than a predetermined value. Good. The projection area is a recognition area shown in FIG. 83, for example. Alternatively, in step S43, the angle of view corresponding to the shorter one of the horizontal and vertical distances between the projection area where the subject is projected in the imaging area of the imaging sensor and the imaging area is predetermined. If the value is less than or equal to the value, it may be determined that the target area cannot be recognized.

This makes it possible to appropriately determine whether or not the target area can be recognized.

FIG. 87B is a block diagram illustrating a structure of a display device according to one embodiment of the present invention.

The display device A10 according to an aspect of the present invention includes an imaging sensor A11, a decoding unit A12, and a display control unit A13.

The imaging sensor A11 acquires a captured image by capturing a still image illuminated by a transmitter that transmits a signal according to a change in the luminance of light as a subject.

The decoding unit A12 is a decoding unit that decodes a signal from the captured image.

The display control unit A13 reads out the moving image corresponding to the decoded signal from the memory, and displays the moving image on the display by superimposing the moving image on the target area corresponding to the subject in the captured image. Here, the display control unit A13 displays the plurality of images in order from the first image that is the same image as the still image among the plurality of images included in the moving image.

Thereby, the same effect as the above-described display method can be obtained.

Further, the imaging sensor A11 may include a plurality of micromirrors and a photosensor, and the display device A10 may further include an imaging control unit that controls the imaging sensor. In this case, the imaging control unit identifies a region including a signal as a signal region in the captured image, and controls the angle of the micromirror corresponding to the identified signal region among the plurality of micromirrors. And an imaging control part makes the above-mentioned photosensor receive only the reflected light by the micro mirror by which the angle was controlled among a plurality of micro mirrors.

Thus, even if a visible light signal, which is a signal represented by a change in light brightness, contains a high frequency component, the high frequency component can be correctly decoded.

In each of the above-described embodiments and modifications, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. For example, the program causes the computer to execute the display method shown by the flowcharts of FIGS. 77, 80, 82, and 87A.

As described above, the display method according to one or a plurality of aspects has been described based on the above-described embodiments and modifications. However, the present invention is not limited to the embodiments. Unless it deviates from the gist of the present invention, various modifications conceived by those skilled in the art have been made in the present embodiment, and forms constructed by combining components in different embodiments and modifications are also within the scope of the present invention. May be included.

[Modification 3 of Embodiment 4]
Hereinafter, a third modification of the fourth embodiment, that is, a third modification of the display method for realizing the AR using the optical ID will be described.

FIG. 88 is a diagram showing an example of expansion and movement of the AR image.

As shown in FIG. 88 (a), the receiver 200 superimposes the AR image P21 on the target area of the captured display image Ppre, as in the fourth embodiment or the first or second modification thereof. Then, the receiver 200 displays the captured display image Ppre on which the AR image P21 is superimposed on the display 201. For example, the AR image P21 is a moving image.

Here, as shown in FIG. 88 (b), when receiving an instruction to change the size, the receiver 200 changes the size of the AR image P21 in accordance with the instruction. For example, when receiving an enlargement instruction, the receiver 200 enlarges the AR image P21 according to the instruction. The size change instruction is given by, for example, a pinch operation, a double tap, or a long press on the AR image P21 by the user. Specifically, when receiving an enlargement instruction performed by pinching out, the receiver 200 enlarges the AR image P21 according to the instruction. Conversely, when receiving an instruction for reduction performed by pinch-in, the receiver 200 reduces the AR image P21 in accordance with the instruction.

Moreover, as shown in FIG. 88 (c), when the receiver 200 receives a position change instruction, the receiver 200 changes the position of the AR image P21 in accordance with the instruction. The instruction to change the position is given by, for example, swiping the AR image by the user. Specifically, when receiving an instruction to change the position performed by swiping, the receiver 200 changes the position of the AR image P21 according to the instruction. That is, the AR image P21 moves.

This makes it possible to make the AR image easier to see by enlarging the AR image that is a moving image, and to reduce the area of the captured display image Pre that is hidden in the AR image by reducing or moving the AR image that is the moving image. Can be displayed to the user.

FIG. 89 is a diagram illustrating an example of the enlargement of the AR image.

As shown in FIG. 89 (a), the receiver 200 superimposes the AR image P22 on the target area of the captured display image Ppre, as in the fourth embodiment or the first or second modification thereof. Then, the receiver 200 displays the captured display image Ppre on which the AR image P22 is superimposed on the display 201. For example, the AR image P22 is a still image in which a character string is described.

Here, as shown in FIG. 89 (b), when receiving an instruction to change the size, the receiver 200 changes the size of the AR image P22 in accordance with the instruction. For example, when receiving an enlargement instruction, the receiver 200 enlarges the AR image P22 according to the instruction. The size change instruction is performed by, for example, a pinch operation, double tap, or long press on the AR image P22 by the user, as described above. Specifically, when receiving an enlargement instruction performed by pinching out, the receiver 200 enlarges the AR image P22 according to the instruction. By enlarging the AR image P22, the character string described in the AR image P22 can be easily read by the user.

Further, as shown in (c) of FIG. 89, when the receiver 200 further receives a size change instruction, the receiver 200 changes the size of the AR image P22 according to the instruction. For example, when receiving a further enlargement instruction, the receiver 200 further enlarges the AR image P22 according to the instruction. By enlarging the AR image P22, it is possible to make it easier for the user to read the character string described in the AR image P22.

Note that, when receiving an enlargement instruction, the receiver 200 may acquire a high-resolution AR image if the enlargement ratio of the AR image corresponding to the instruction is equal to or greater than a threshold value. In this case, the receiver 200 may enlarge and display the high-resolution AR image up to the above-described enlargement factor instead of the original AR image that has already been displayed. For example, the receiver 200 displays an AR image of 1920 × 1080 pixels instead of the AR image of 640 × 480 pixels. As a result, the AR image can be enlarged and a high-resolution image that cannot be obtained by the optical zoom can be displayed so that the AR image is actually captured as a subject.

FIG. 90 is a flowchart illustrating an example of processing operations related to enlargement and movement of an AR image by the receiver 200.

First, the receiver 200 starts imaging based on the normal exposure time and the communication exposure time as in step S101 shown in the flowchart of FIG. 45 (step S401). When this imaging is started, a captured display image Ppre based on the normal exposure time and a decoding image (that is, a bright line image) Pdec based on the communication exposure time are periodically obtained. Then, the receiver 200 acquires the optical ID by decoding the decoding image Pdec.

Next, the receiver 200 performs an AR image superimposition process including the processes of steps S102 to S106 shown in the flowchart of FIG. 45 (step S402). When this AR image superimposition process is performed, the AR image is displayed superimposed on the captured display image Ppre. At this time, the receiver 200 decreases the optical ID acquisition rate (step S403). The light ID acquisition rate is a ratio of the number of decoding images (that is, bright line images) Pdec out of the number of captured images per unit time obtained by imaging started in step S401. For example, as the optical ID acquisition rate decreases, the number of decoding images Pdec obtained per unit time becomes smaller than the number of captured display images Ppre obtained per unit time.

Next, the receiver 200 determines whether or not a size change instruction has been received (step S404). If it is determined that the size change instruction has been received (Yes in step S404), the receiver 200 further determines whether the size change instruction is an enlargement instruction (step S405). If it is determined that the size change instruction is an enlargement instruction (Yes in step S405), the receiver 200 further determines whether it is necessary to reacquire the AR image (step S406). For example, when the receiver 200 determines that the AR image enlargement rate according to the enlargement instruction is equal to or greater than a threshold, the receiver 200 determines that the AR image needs to be reacquired. When the receiver 200 determines that reacquisition is necessary (Yes in step S406), the receiver 200 acquires a high-resolution AR image from, for example, a server, and converts the AR image displayed in a superimposed manner into the high-resolution AR image. Replace with the AR image (step S407).

Then, the receiver 200 changes the size of the AR image in accordance with the received size change instruction (step S408). That is, when a high-resolution AR image is acquired in step S407, the receiver 200 enlarges the high-resolution AR image. Further, when it is determined in step S406 that the re-acquisition of the AR image is unnecessary (No in step S406), the receiver 200 enlarges the superimposed AR image. If it is determined in step S405 that the size change instruction is a reduction instruction (No in step S405), the receiver 200 performs superimposition according to the received size change instruction, that is, the reduction instruction. Reduce the displayed AR image.

On the other hand, when the receiver 200 determines in step S404 that the size change instruction has not been received (No in step S404), the receiver 200 determines whether a position change instruction has been received (step S409). If it is determined that a position change instruction has been received (Yes in step S409), the receiver 200 changes the position of the superimposed AR image in accordance with the position change instruction (step S410). ). That is, the receiver 200 moves the AR image. If it is determined that the position change instruction has not been received (No in step S409), the receiver 200 repeatedly executes the processing from step S404.

When the size of the AR image is changed in step S408 or the position of the AR image is changed in step S410, the receiver 200 cannot acquire the light ID periodically acquired from step S401. It is determined whether or not (step S411). If it is determined that the light ID is no longer acquired (Yes in step S411), the receiver 200 ends the processing operation related to the expansion and movement of the AR image. On the other hand, if it is determined that the light ID is still acquired (No in step S411), the receiver 200 repeatedly executes the processing from step S404.

FIG. 91 is a diagram illustrating an example of superimposition of AR images by the receiver 200.

As described above, the receiver 200 superimposes the AR image P23 on the target area in the captured display image Ppre. Here, as shown in FIG. 91, the AR image P23 is configured such that the transmittance of each part of the AR image P23 increases as the part of the AR image P23 is closer to the end of the AR image P23. The transmittance is the degree to which the superimposed image is displayed through. For example, when the overall transmittance of the AR image is 100%, even if the AR image is superimposed on the target area of the captured display image, only the target area is displayed on the display 201 without displaying the AR image. Means that. Conversely, the transmittance of the entire AR image being 0% means that the target area of the captured display image is not displayed on the display 201 and only the AR image superimposed on the target area is displayed. .

For example, when the AR image P23 is rectangular, the transmittance of each part in the AR image P23 is higher as the part is closer to the upper end, lower end, left end, or right end of the rectangle. More specifically, the transmittance at those ends is 100%. In addition, in the central portion of the AR image P23, there is a rectangular area having a transmittance of 0% which is smaller than that of the AR image P23. In the rectangular area, for example, “Kyoto Station” is written in English. That is, at the peripheral edge of the AR image P23, the transmittance changes stepwise from 0% to 100% like a gradation.

The receiver 200 superimposes such an AR image P23 on the target area in the captured display image Ppre as shown in FIG. At this time, the receiver 200 matches the size of the AR image P23 with the size of the target area, and superimposes the resized AR image P23 on the target area. For example, an image of a station name sign having the same background color as that of the rectangular area in the center of the AR image P23 appears in the target area. The station name has “Kyoto” written in Japanese.

Here, as described above, the transmittance of each part of the AR image P23 is higher as the part is closer to the end of the AR image P23. Therefore, when the AR image P23 is superimposed on the target area, even if the rectangular area at the center of the AR image P23 is displayed, the end of the AR image P23 is not displayed, but the end of the target area, that is, the station name mark The edge of the image is displayed.

Thereby, the deviation between the AR image P23 and the target area can be made inconspicuous. That is, even when the AR image P23 is superimposed on the target area, a shift may occur between the AR image P23 and the target area due to the movement of the receiver 200 or the like. In this case, if the overall transmittance of the AR image P23 is 0%, the end of the AR image P23 and the end of the target area are displayed, and the shift becomes conspicuous. However, in the AR image P23 in the present modification, the closer to the end, the higher the transmittance of the part, so that the end of the AR image P23 can be made difficult to be displayed. As a result, the AR image P23 and the target region It is possible to make the gap inconspicuous. Furthermore, since the transmittance changes like gradation in the peripheral portion of the AR image P23, it is difficult to notice that the AR image P23 is superimposed on the target region.

FIG. 92 is a diagram illustrating an example of AR image superimposition performed by the receiver 200.

As described above, the receiver 200 superimposes the AR image P24 on the target area in the captured display image Ppre. Here, as shown in FIG. 92, the imaged subject is, for example, a restaurant menu. This menu is surrounded by a white frame, and the white frame is surrounded by a black frame. That is, the subject includes a menu, a white frame surrounding the menu, and a black frame surrounding the white frame.

The receiver 200 recognizes, as a target area, an area larger than the white frame image and smaller than the black frame image in the captured display image Pre. Then, the receiver 200 matches the size of the AR image P24 with the size of the target area, and superimposes the resized AR image P24 on the target area.

Thereby, even when the superimposed AR image P24 is displaced from the target area due to the movement of the receiver 200, the AR image P24 can be continuously displayed in a state surrounded by a black frame. Therefore, the shift between the AR image P24 and the target region can be made inconspicuous.

In the example shown in FIG. 92, the color of the frame is black or white, but is not limited to these colors and may be any color.

FIG. 93 is a diagram illustrating an example of superimposition of AR images by the receiver 200. FIG.

For example, the receiver 200 images a poster on which a castle illuminated in the night sky is drawn as a subject. For example, the poster is illuminated by the above-described transmitter 100 configured as a backlight, and a visible light signal (ie, a light ID) is transmitted by the backlight. The receiver 200 acquires the captured display image Ppre including the image of the subject that is the poster and the AR image P25 corresponding to the light ID by the imaging. Here, the AR image P25 has the same shape as the poster image from which the region where the castle is drawn is extracted. That is, the area corresponding to the castle of the poster image in the AR image P25 is masked. Further, the AR image P25 is configured so that the transmittance of each part of the AR image P25 is higher as the part of the AR image P25 is closer to the end of the AR image P25, similarly to the AR image P23 described above. In the central portion where the transmittance is 0% in the AR image P25, fireworks launched in the night sky are displayed as moving images.

The receiver 200 matches the size of the AR image P25 with the size of the target area that is the image of the subject, and superimposes the resized AR image P25 on the target area. As a result, the castle drawn on the poster is displayed as an image of the subject, not as an AR image, and a moving image of fireworks is displayed as an AR image.

Thereby, the captured display image Ppre can be displayed as if fireworks are actually being launched in the poster. Further, the transmittance of each part of the AR image P25 is higher as the part is closer to the end of the AR image P25. Therefore, when the AR image P25 is superimposed on the target area, even if the center portion of the AR image P25 is displayed, the end of the AR image P25 is not displayed, but the end of the target area is displayed. As a result, the deviation between the AR image P25 and the target region can be made inconspicuous. Furthermore, since the transmittance changes like a gradation at the peripheral portion of the AR image P25, it is difficult to notice that the AR image P25 is superimposed on the target region.

FIG. 94 is a diagram illustrating an example of superposition of AR images by the receiver 200. FIG.

For example, the receiver 200 images the transmitter 100 configured as a television as a subject. Specifically, the transmitter 100 displays a castle illuminated in the night sky on a display and transmits a visible light signal (that is, a light ID). The receiver 200 acquires the captured display image Ppre displayed by the transmitter 100 and the AR image P26 corresponding to the optical ID by the imaging. Here, the receiver 200 first displays the captured display image Ppre on the display 201. At this time, the receiver 200 also displays on the display 201 a message m that prompts the user to turn it off. Specifically, the message m is, for example, “Turn off the room lighting and darken the room”.

When the message m is displayed and the user is turned off and the room in which the transmitter 100 is installed becomes dark, the receiver 200 displays the AR image P26 superimposed on the captured display image Ppre. Here, the AR image P26 has the same size as the captured display image Ppre, and the area corresponding to the castle of the captured display image Ppre in the AR image P26 is cut out. That is, the area corresponding to the castle of the captured display image Ppre in the AR image P26 is masked. Therefore, the castle of the captured display image Ppre can be shown to the user from the area. Further, in the peripheral portion of the area in the AR image P26, similarly to the above, the transmittance may change stepwise from 0% to 100% like a gradation. In this case, the shift between the captured display image Ppre and the AR image P26 can be made inconspicuous.

In the above-described example, the AR image having a high peripheral edge transmittance is superimposed on the target area of the captured display image Ppre, so that the shift between the AR image and the target area is less noticeable. However, instead of such an AR image, an AR image that is the same size as the captured display image Ppre and is entirely translucent (that is, having a transmittance of 50%) may be superimposed on the captured display image Ppre. Even in this case, the shift between the AR image and the target region can be made inconspicuous. Further, when the captured display image Ppre is generally bright, the AR image having a uniform low transparency is superimposed on the captured display image Ppre. Conversely, when the captured display image Ppre is generally dark, the transparency is uniformly uniform. A high AR image may be superimposed on the captured display image Ppre.

It should be noted that objects such as the fireworks of the AR image P25 and the AR image P26 may be expressed by CG (computer graphics). In this case, masking can be eliminated. In the example illustrated in FIG. 94, the receiver 200 displays the message m that prompts the user to turn off the light, but the light may be automatically turned off without performing such display. For example, the receiver 200 outputs a turn-off signal to the lighting device in which the transmitter 100 that is a television is set by Bluetooth (registered trademark), ZigBee, a specific low-power radio station, or the like. Thereby, the lighting device is automatically turned off.

FIG. 95A is a diagram illustrating an example of a captured display image Ppre obtained by imaging by the receiver 200.

For example, the transmitter 100 is configured as a large display installed in a stadium. Then, the transmitter 100 displays a message indicating that, for example, fast food and drinks can be ordered with the light ID, and transmits a visible light signal (that is, a light ID). When such a message is displayed, the user images the receiver 200 toward the transmitter 100. That is, the receiver 200 images the transmitter 100 configured as a large display installed in the stadium as a subject.

The receiver 200 acquires the captured display image Ppre and the decoding image Pdec by the imaging. Then, the receiver 200 acquires the optical ID by decoding the decoding image Pdec, and transmits the optical ID and the captured display image Ppre to the server.

For each light ID, the server specifies the installation information of the captured large display associated with the light ID transmitted from the receiver 200 from the installation information associated with the light ID. For example, the installation information indicates the position and orientation where the large display is installed, the size of the large display, and the like. Further, the server identifies the seat number where the captured display image Ppre was captured in the stadium based on the size and orientation of the large display displayed in the captured display image Ppre and the installation information. To do. Then, the server causes the receiver 200 to display a menu screen including the seat number.

FIG. 95B is a diagram showing an example of a menu screen displayed on the display 201 of the receiver 200.

The menu screen m1 includes, for example, for each product, an input field ma1 in which the number of orders for the product is input, a seat field mb1 in which the seat number of the stadium specified by the server is described, and an order button mc1. . The user operates the receiver 200 to input the order quantity of the product in the input field ma1 corresponding to the desired product, and selects the order button mc1. As a result, the order is confirmed, and the receiver 200 transmits the order contents corresponding to the input result to the server.

When the server receives the order details, it instructs the stadium staff to deliver the number of products according to the order details to the seat of the number specified as described above.

FIG. 96 is a flowchart showing an example of processing operation between the receiver 200 and the server.

The receiver 200 first images the transmitter 100 configured as a large stadium display (step S421). The receiver 200 acquires the optical ID transmitted from the transmitter 100 by decoding the decoding image Pdec obtained by the imaging (step S422). The receiver 200 transmits the optical ID acquired in step S422 and the captured display image Ppre obtained by imaging in step S421 to the server (step S423).

When the server receives the light ID and the captured display image Pre (step S424), the server identifies installation information of a large display installed in the stadium based on the light ID (step S425). For example, for each light ID, the server holds a table indicating installation information of a large display associated with the light ID, and the installation information associated with the light ID transmitted from the receiver 200 is stored from the table. The installation information is specified by searching.

Next, the server acquires (that is, captures) the captured display image Ppre in the stadium based on the specified installation information and the size and orientation of the large display displayed in the captured display image Ppre. The assigned seat number is specified (step S426). Then, the server transmits the URL (Uniform Resource Locator) of the menu screen m1 including the identified seat number to the receiver 200 (step S427).

When the receiver 200 receives the URL of the menu screen m1 transmitted from the server (step S428), the receiver 200 accesses the URL and displays the menu screen m1 (step S429). Here, the user operates the receiver 200 to input the order contents into the menu screen m1, and selects the order button mc1, thereby confirming the order. Thereby, the receiver 200 transmits the order details to the server (step S430).

Upon receiving the order details transmitted from the receiver 200, the server performs an order receiving process according to the order details (step S431). At this time, for example, the server instructs the staff of the stadium to deliver the number of products corresponding to the order contents to the seat of the number specified in step S426.

Thus, since the seat number is specified based on the captured display image Ppre obtained by imaging by the receiver 200, the user of the receiver 200 bothers to input the seat number when ordering a product. There is no need to do. Therefore, the user can easily place an order for a product without inputting the seat number.

In the above example, the server specifies the seat number, but the receiver 200 may specify the seat number. In this case, the receiver 200 acquires the installation information from the server, and specifies the seat number based on the installation information and the size and orientation of the large display displayed in the captured display image Pre.

FIG. 97 is a diagram for explaining the volume of sound reproduced by the receiver 1800a.

23. Similarly to the example shown in FIG. 23, the receiver 1800a receives the light ID (visible light signal) transmitted from the transmitter 1800b configured as, for example, street digital signage. Then, the receiver 1800a reproduces sound at the same timing as the image reproduction by the transmitter 1800b. That is, the receiver 1800a reproduces sound so as to be synchronized with the image reproduced by the transmitter 1800b. Note that the receiver 1800a may reproduce the same image as the image reproduced by the transmitter 1800b (reproduced image) or an AR image (AR moving image) related to the reproduced image together with the sound.

Here, when reproducing the sound as described above, the receiver 1800a adjusts the volume of the sound according to the distance to the transmitter 1800b. Specifically, the receiver 1800a adjusts the volume smaller as the distance to the transmitter 1800b is longer, and conversely adjusts the volume larger as the distance to the transmitter 1800b is shorter.

The receiver 1800a may specify the distance to the transmitter 1800b using a GPS (Global Positioning System) or the like. Specifically, the receiver 1800a acquires position information of the transmitter 1800b associated with the optical ID from a server or the like, and further specifies the position of the receiver 1800a by GPS. Then, the receiver 1800a specifies the distance between the position of the transmitter 1800b indicated by the position information acquired from the server and the position of the specified receiver 1800a as the distance to the above-described transmitter 1800b. . Note that the receiver 1800a may specify the distance to the transmitter 1800b by using Bluetooth (registered trademark) instead of GPS.

Further, the receiver 1800a may specify the distance to the transmitter 1800b based on the size of the bright line pattern region of the above-described decoding image Pdec obtained by imaging. As in the example shown in FIGS. 51 and 52, the bright line pattern region is a region formed of a plurality of bright line patterns that appear by exposure at the exposure time for communication of a plurality of exposure lines included in the image sensor of the receiver 1800a. This bright line pattern area corresponds to the display area of the transmitter 1800b displayed in the captured display image Ppre. Specifically, the receiver 1800a specifies the shorter distance as the distance to the transmitter 1800b as the bright line pattern region is larger, and conversely specifies the longer distance as the distance to the transmitter 1800b as the bright line pattern region is smaller. . The receiver 1800a uses distance data indicating the relationship between the size of the bright line pattern region and the distance, and in the distance data, the distance associated with the size of the bright line pattern region in the captured display image Pre is The distance to the transmitter 1800b may be specified. Note that the receiver 1800a may transmit the optical ID received as described above to the server, and obtain distance data associated with the optical ID from the server.

As described above, since the volume is adjusted according to the distance to the transmitter 1800b, the user of the receiver 1800a makes the sound reproduced by the receiver 1800a like the sound actually reproduced by the transmitter 1800b. Can be heard.

FIG. 98 is a diagram showing the relationship between the distance from the receiver 1800a to the transmitter 1800b and the sound volume.

For example, when the distance to the transmitter 1800b is between L1 and L2 [m], the volume increases or decreases in proportion to the distance in the range from Vmin to Vmax [dB]. Specifically, the receiver 1800a linearly decreases the volume from Vmax [dB] to Vmin [dB] when the distance to the transmitter 1800b increases from L1 [m] to L2 [m]. Even if the distance to the transmitter 1800b is shorter than L1 [m], the receiver 1800a maintains the volume at Vmax [dB], and the distance to the transmitter 1800b is longer than L2 [m]. However, the volume is maintained at Vmin [dB].

As described above, the receiver 1800a stores the maximum volume Vmax, the longest distance L1 at which the sound of the maximum volume Vmax is output, the minimum volume Vmin, and the shortest distance L2 at which the sound of the minimum volume Vmin is output. is doing. The receiver 1800a may change the maximum volume Vmax, the minimum volume Vmin, the longest distance L1, and the shortest distance L2 according to the attributes set for the receiver 1800a. For example, when the attribute is the age of the user and the age indicates a high age, the receiver 1800a sets the maximum volume Vmax higher than the reference maximum volume and sets the minimum volume Vmin higher than the reference minimum volume. Also good. Further, the attribute may be information indicating whether audio output is performed from a speaker or an earphone.

As described above, since the minimum volume Vmin is set in the receiver 1800a, it is possible to prevent the receiver 1800a from being inaudible because the receiver 1800a is too far from the transmitter 1800b. Furthermore, since the maximum volume Vmax is set in the receiver 1800a, the receiver 1800a is too close to the transmitter 1800b, so that it is possible to suppress an excessively loud sound from being output.

FIG. 99 is a diagram illustrating an example of AR image superimposition performed by the receiver 200.

The receiver 200 images the illuminated signboard. Here, the signboard is lit up by the illumination device which is the above-described transmitter 100 that transmits the optical ID. Therefore, the receiver 200 acquires the captured display image Ppre and the decoding image Pdec by the imaging. Then, the receiver 200 acquires the optical ID by decoding the decoding image Pdec, and acquires a plurality of AR images P27a to P27c associated with the optical ID and the recognition information from the server. Based on the recognition information, the receiver 200 recognizes the periphery of the area m2 in which the signboard is displayed in the captured display image Ppre as a target area.

Specifically, as shown in (a) of FIG. 99, the receiver 200 recognizes an area in contact with the left side of the area m2 as the first target area, and superimposes the AR image P27a on the first target area. To do.

Next, as illustrated in FIG. 99B, the receiver 200 recognizes an area including the lower side of the area m2 as a second target area, and superimposes the AR image P27b on the second target area. .

Next, as shown in (c) of FIG. 99, the receiver 200 recognizes an area in contact with the upper side of the area m2 as the third target area, and superimposes the AR image P27c on the third target area.

Here, each of the AR images P27a to P27c is, for example, an image of a snowman character and may be a moving image.

Further, the receiver 200 switches the recognized target area to any one of the first to third target areas in a predetermined order and timing while continuously acquiring the optical ID. May be. That is, the receiver 200 may switch the recognized target area in the order of the first target area, the second target area, and the third target area. Alternatively, the receiver 200 may switch the recognized target area to any one of the first to third target areas in a predetermined order each time the above-described optical ID is acquired. That is, the receiver 200 first acquires the light ID, and while continuously acquiring the light ID, the receiver 200 recognizes the first target area as shown in FIG. Then, the AR image P27a is superimposed on the first target area. Then, when the receiver 200 cannot acquire the optical ID, the receiver 200 hides the AR image P27a. Next, when the receiver 200 acquires the light ID again, the receiver 200 recognizes the second target area as shown in FIG. 99 (b) while continuously acquiring the light ID. Then, the AR image P27b is superimposed on the second target area. Then, when the receiver 200 cannot acquire the optical ID again, the receiver 200 hides the AR image P27b. Next, when the receiver 200 acquires the light ID again, the receiver 200 recognizes the third target area as shown in (c) of FIG. 99 while continuously acquiring the light ID. Then, the AR image P27c is superimposed on the third target area.

In this way, when the target area to be recognized is switched every time the light ID is acquired, the receiver 200 displays the AR image displayed once every N (N is an integer of 2 or more) times. The color may be changed. N times is the number of times the AR image is displayed, and may be 200 times, for example. That is, the AR images P27a to P27c are images of the same white character, but an AR image of a pink character, for example, is displayed at a frequency of once every 200 times. When the AR image of the pink character is displayed, the receiver 200 may give points to the user when receiving an operation on the AR image by the user.

In this way, by switching the target area on which the AR image is superimposed or changing the color of the AR image at a predetermined frequency, the user's interest can be directed to the imaging of the signboard lit up by the transmitter 100. The user can repeatedly obtain the optical ID.

FIG. 100 is a diagram illustrating an example of superimposition of AR images by the receiver 200.

The receiver 200 functions as a so-called way finder (Way Finder) that presents a route to be followed by the user, for example, by imaging the mark M4 drawn on the floor surface at a position where a plurality of passages intersect in the building. Have The building is a hotel, for example, and the presented route is a route where the user who has checked in heads to his / her room.

The mark M4 is lit up by the illumination device that is the above-described transmitter 100 that transmits the light ID by a change in luminance. Therefore, the receiver 200 acquires the captured display image Ppre and the decoding image Pdec by capturing the mark M4. Then, the receiver 200 acquires the optical ID by decoding the decoding image Pdec, and transmits the optical ID and the terminal information of the receiver 200 to the server. The receiver 200 acquires a plurality of AR images P28 and recognition information associated with the optical ID and terminal information from the server. The optical ID and terminal information are stored in the server in association with a plurality of AR images P28 and recognition information at the time of user check-in.

Based on the recognition information, the receiver 200 recognizes a plurality of target areas around the area m4 in which the mark M4 is displayed in the captured display image Ppre. Then, as shown in FIG. 100, the receiver 200 superimposes and displays an AR image P28 such as an animal footprint on each of the plurality of target regions.

Specifically, the recognition information indicates the course of turning to the right at the position of the mark M4. Based on such recognition information, the receiver 200 identifies a route in the captured display image Ppre and recognizes a plurality of target regions arranged along the route. This route is a route that goes from the lower side of the display 201 to the region m4 and turns right in the region m4. The receiver 200 arranges the AR image P28 in each of the recognized plurality of target regions as if the animal walked along the route.

Here, when specifying the route in the captured display image Pre, the receiver 200 may use the geomagnetism detected by the 9-axis sensor provided in the receiver 200. In this case, the recognition information indicates the direction to proceed at the position of the mark M4 with reference to the direction of geomagnetism. For example, the recognition information indicates west as the direction to proceed at the position of the mark M4. Based on such recognition information, the receiver 200 specifies a route from the lower side of the display 201 toward the area m4 and toward the west in the area m4 in the captured display image Ppre. Then, the receiver 200 recognizes a plurality of target areas arranged along the route. Note that the receiver 200 identifies the lower side of the display 201 by detecting gravitational acceleration using a nine-axis sensor.

In this way, since the route of the user is presented by the receiver 200, the user can easily reach the destination by following the route. Moreover, since the course is displayed as an AR image in the captured display image Ppre, the course can be presented to the user in an easy-to-understand manner.

In addition, the illuminating device which is the transmitter 100 can appropriately transmit the light ID while suppressing the brightness by illuminating the mark M4 with a short pulse of light. In addition, the receiver 200 images the mark M4. However, the receiver 200 may image the illumination device using a camera (a so-called self-taking camera) disposed on the display 201 side. The receiver 200 may capture both the mark M4 and the illumination device.

FIG. 101 is a diagram for explaining an example of how the receiver 200 obtains the line scan time.

The receiver 200 performs decoding using the line scan time when decoding the decoding image Pdec. This line scan time is the time from the start of exposure of one exposure line included in the image sensor to the start of exposure of the next exposure line. If the line scan time is known, the receiver 200 decodes the decoding image Pdec using the known line scan time. However, when the line scan time is not known, the receiver 200 obtains the line scan time from the decoding image Pdec.

For example, as shown in FIG. 101, the receiver 200 finds a line having the minimum width from among a plurality of bright lines and a plurality of dark lines constituting a bright line pattern in the decoding image Pdec. The bright line is a line on the decoding image Pdec generated when each of one or a plurality of continuous exposure lines is exposed when the luminance of the transmitter 100 is high. Further, the dark line is a line on the decoding image Pdec generated by exposure of each of one or a plurality of continuous exposure lines when the luminance of the transmitter 100 is low.

When the receiver 200 finds the line with the minimum width, the receiver 200 specifies the number of exposure lines corresponding to the line with the minimum width, that is, the number of pixels. When the carrier frequency at which the luminance changes so that the transmitter 100 transmits the optical ID is 9.6 kHz, the time when the luminance of the transmitter 100 is high or low is 104 μs at the shortest. Therefore, the receiver 200 calculates the line scan time by dividing 104 μs by the number of pixels having the specified minimum width.

FIG. 102 is a diagram for explaining an example of how the receiver 200 obtains the line scan time.

The receiver 200 may perform a Fourier transform on the bright line pattern of the decoding image Pdec and obtain the line scan time based on the spatial frequency obtained by the Fourier transform.

For example, as shown in FIG. 102, the receiver 200 derives a spectrum indicating the relationship between the spatial frequency and the intensity of the component of the spatial frequency in the decoding image Pdec by the Fourier transform described above. Next, the receiver 200 sequentially selects each of the plurality of peaks indicated in the spectrum. Each time the receiver 200 selects a peak, the receiver 200 calculates a line scan time such that the spatial frequency of the selected peak (for example, the spatial frequency f2 in FIG. 102) is obtained by a time frequency of 9.6 kHz. Calculate as a scan time candidate. 9.6 kHz is the carrier frequency of the luminance change of the transmitter 100 as described above. Thereby, a plurality of line scan time candidates are calculated. The receiver 200 selects the most likely candidate among the plurality of line scan time candidates as the line scan time.

In order to select the most likely candidate, the receiver 200 calculates the allowable range of line scan time based on the frame rate in imaging and the number of exposure lines included in the image sensor. That is, the receiver 200 calculates the maximum value of the line scan time by 1 × 10 ⁶ [μs] / {(frame rate) × (number of exposure lines)}. Then, the receiver 200 determines the maximum value × constant K (K <1) to the maximum value as the allowable range of the line scan time. The constant K is, for example, 0.9 or 0.8.

The receiver 200 selects a candidate within this allowable range from among a plurality of line scan time candidates as a maximum likelihood candidate, that is, a line scan time.

Note that the receiver 200 may evaluate the reliability of the calculated line scan time depending on whether or not the line scan time calculated according to the example illustrated in FIG. 101 is within the above-described allowable range.

FIG. 103 is a flowchart showing an example of how to obtain the line scan time by the receiver 200.

The receiver 200 may obtain the line scan time by trying to decode the decoding image Pdec. Specifically, first, the receiver 200 starts imaging (step S441). Next, the receiver 200 determines whether or not the line scan time is known (step S442). For example, the receiver 200 may determine whether or not the line scan time is known by notifying the server of the type and model of the receiver 200 and inquiring the line scan time according to the type and model. . If it is determined that it is known (Yes in step S442), the receiver 200 sets the reference acquisition count of the optical ID to n (n is an integer equal to or larger than 2, for example, 4) (step S443). ). Next, the receiver 200 acquires the optical ID by decoding the decoding image Pdec using the known line scan time (step S444). At this time, the receiver 200 acquires a plurality of optical IDs by performing decoding on each of the plurality of decoding images Pdec obtained sequentially by the imaging started in step S441. Here, the receiver 200 determines whether or not the same optical ID has been acquired the reference acquisition times (that is, n times) (step S445). If it is determined that it has been acquired n times (Yes in step S445), the receiver 200 trusts the optical ID and starts processing using the optical ID (for example, superimposition of an AR image) (step S446). On the other hand, if it is determined that it has not been acquired n times (No in step S445), the receiver 200 does not trust the optical ID and ends the process.

If it is determined in step S442 that the line scan time is not known (No in step S442), the receiver 200 sets the optical ID reference acquisition count to n + k (k is an integer equal to or greater than 1) (step S447). . That is, when the line scan time is not known, the receiver 200 sets a larger reference acquisition count than when the line scan time is known. Next, the receiver 200 determines a temporary line scan time (step S448). Then, the receiver 200 acquires the optical ID by decoding the decoding image Pdec using the provisional line scan time (step S449). At this time, the receiver 200 acquires a plurality of optical IDs by performing decoding on each of the plurality of decoding images Pdec obtained sequentially by the imaging started in step S441, as described above. Here, the receiver 200 determines whether or not the same optical ID has been acquired the reference acquisition times (that is, (n + k) times) (step S450).

If it is determined that (n + k) times have been acquired (Yes in step S450), the receiver 200 determines that the provisional line scan time is the correct line scan time. Then, the receiver 200 notifies the server of the type and model of the receiver 200 and the line scan time (step S451). As a result, the server stores the type and model of the receiver in association with the line scan time suitable for the receiver. Therefore, when another receiver of the same type and type starts imaging, the other receiver can specify its own line scan time by making an inquiry to the server. That is, the other receivers can determine that the line scan time is known in the determination in step S442.

Then, the receiver 200 trusts the optical ID acquired (n + k) times, and starts processing using the optical ID (for example, superimposition of an AR image) (step S446).

If it is determined in step S450 that the same light ID has not been acquired (n + k) times (No in step S450), the receiver 200 further determines whether or not an end condition is satisfied (step S452). . The end condition is, for example, that a predetermined time has elapsed since the start of imaging, or that the optical ID has been acquired more than the maximum number of acquisitions. If it is determined that such an end condition is satisfied (Yes in step S452), the receiver 200 ends the process. On the other hand, when determining that the termination condition is not satisfied (No in step S452), the receiver 200 changes the provisional line scan time (step S453). Then, the receiver 200 repeatedly executes the processing from step S449 using the changed provisional line scan time.

Thus, even if the line scan time is not known, the receiver 200 can obtain the line scan time as in the examples shown in FIGS. Accordingly, regardless of the type and model of the receiver 200, the receiver 200 can appropriately decode the decoding image Pdec and obtain an optical ID.

FIG. 104 is a diagram illustrating an example of AR image superimposition performed by the receiver 200.

The receiver 200 images the transmitter 100 configured as a television. The transmitter 100 periodically transmits an optical ID and a time code by changing luminance while displaying a television program, for example. The time code is information indicating the time at the time of transmission each time it is transmitted, and may be, for example, a time packet shown in FIG.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec through the above-described imaging. The receiver 200 acquires the above-described optical ID and time code by decoding the decoding image Pdec while displaying the captured display image Ppre acquired periodically on the display 201. Next, the receiver 200 transmits the optical ID to the server 300. When the server 300 receives the optical ID, the server 300 transmits the audio data associated with the optical ID, the AR start time information, the AR image P29, and the recognition information to the receiver 200.

When the receiver 200 acquires the audio data, the receiver 200 reproduces the audio data in synchronization with the video of the TV program displayed on the transmitter 100. That is, the sound data is composed of a plurality of sound unit data, and the plurality of sound unit data includes a time code. The receiver 200 starts reproduction of a plurality of audio unit data from the audio unit data including the time code indicating the same time as the time code acquired from the transmitter 100 together with the optical ID in the audio data. Thereby, the reproduction of the audio data is synchronized with the video of the television program. It should be noted that such synchronization between audio and video may be performed by the same method as the audio synchronous reproduction shown in each of the drawings after FIG.

When the receiver 200 acquires the AR image P29 and the recognition information, the receiver 200 recognizes an area corresponding to the recognition information in the captured display image Ppre as a target area, and superimposes the AR image P29 on the target area. For example, the AR image P29 is an image showing a crack in the display 201 of the receiver 200, and the target area is an area that crosses the image of the transmitter 100 in the captured display image Ppre.

Here, the receiver 200 displays the captured display image Ppre on which the AR image P29 as described above is superimposed at a timing according to the AR start time information. The AR start time information is information indicating the time at which the AR image P29 is displayed. That is, the receiver 200 captures a display image in which the above-described AR image P29 is superimposed at the timing of receiving the time code indicating the same time as the AR start time information among the time codes transmitted from the transmitter 100 as needed. Display Pre. For example, the time indicated by the AR start time information is the time when a scene in which a magician girl applies ice magic appears in a television program. At this time, the receiver 200 may output from the speaker of the receiver 200 a sound in which the AR image P29 is cracked due to the reproduction of the audio data.

This allows the user to view the TV program scene with a sense of reality.

Further, the receiver 200 may vibrate a vibrator provided in the receiver 200 at a time indicated by the AR start time information, or may cause the light source to emit light like a flash, and the display 201 may be instantaneously displayed. It may be brightened or flashed. Further, the AR image P29 may include not only an image showing a crack but also an image showing a state in which condensation of the display 201 is frozen.

FIG. 105 is a diagram illustrating an example of AR image superimposition performed by the receiver 200.

The receiver 200 images the transmitter 100 configured as a toy cane, for example. The transmitter 100 includes a light source, and transmits an optical ID by changing the luminance of the light source.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec through the above-described imaging. The receiver 200 acquires the above-described optical ID by decoding the decoding image Pdec while displaying the captured display image Ppre acquired periodically on the display 201. Next, the receiver 200 transmits the optical ID to the server 300. When the server 300 receives the optical ID, the server 300 transmits the AR image P30 associated with the optical ID and the recognition information to the receiver 200.

Here, the recognition information further includes gesture information indicating a gesture (that is, an action) by a person holding the transmitter 100. The gesture information indicates, for example, a gesture in which a person moves the transmitter 100 from right to left. The receiver 200 compares the gesture by the person holding the transmitter 100 displayed on each captured display image Ppre with the gesture indicated by the gesture information. Then, when the gestures coincide with each other, the receiver 200 arranges the AR images P30 such that, for example, many star-shaped AR images P30 are arranged along the trajectory of the transmitter 100 moved by the gestures. Is superimposed on the captured display image Ppre.

FIG. 106 is a diagram illustrating an example of AR image superimposition performed by the receiver 200.

The receiver 200 images the transmitter 100 configured as, for example, a toy cane, as described above.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec by the imaging. The receiver 200 acquires the above-described optical ID by decoding the decoding image Pdec while displaying the captured display image Ppre acquired periodically on the display 201. Next, the receiver 200 transmits the optical ID to the server 300. When the server 300 receives the optical ID, the server 300 transmits the AR image P31 associated with the optical ID and the recognition information to the receiver 200.

Here, the recognition information includes gesture information indicating a gesture by a person holding the transmitter 100 as described above. The gesture information indicates, for example, a gesture in which a person moves the transmitter 100 from right to left. The receiver 200 compares the gesture by the person holding the transmitter 100 displayed on each captured display image Ppre with the gesture indicated by the gesture information. Then, when the gestures match, the receiver 200, for example, in the captured display image Ppre, an AR image P30 indicating a dress costume in a target area that is an area in which a person holding the transmitter 100 is projected. Is superimposed.

As described above, in the display method in the present modification, gesture information corresponding to the light ID is acquired from the server. Next, it is determined whether or not the movement of the subject indicated by the periodically acquired captured display image matches the movement indicated by the gesture information acquired from the server. And when it determines with matching, the picked-up display image Ppre on which AR image was superimposed is displayed.

Thereby, for example, an AR image can be displayed according to the movement of a subject such as a person. That is, the AR image can be displayed at an appropriate timing.

FIG. 107 is a diagram illustrating an example of the decoding image Pdec acquired according to the attitude of the receiver 200.

For example, as shown in (a) of FIG. 107, the receiver 200 images the transmitter 100 that transmits the optical ID according to the luminance change in the horizontal orientation. In the horizontal orientation, the longitudinal direction of the display 201 of the receiver 200 is an orientation along the horizontal direction. Further, each exposure line of the image sensor provided in the receiver 200 is orthogonal to the longitudinal direction of the display 201. By the imaging as described above, a decoding image Pdec including the bright line pattern region X with a small number of bright lines is acquired. In the bright line pattern region X of the decoding image Pdec, the number of bright lines is small. That is, there are few parts where the luminance changes to High or Low. Therefore, the receiver 200 may not be able to acquire the optical ID appropriately by decoding the decoding image Pdec.

Therefore, for example, as shown in FIG. 107 (b), the user changes the attitude of the receiver 200 from landscape to portrait. The vertical orientation is a posture in which the longitudinal direction of the display 201 of the receiver 200 is along the vertical direction. The receiver 200 having such an attitude can acquire the decoding image Pdec including the bright line pattern region Y having a large number of bright lines when the transmitter 100 that transmits the light ID is imaged.

As described above, the optical ID may not be appropriately acquired according to the attitude of the receiver 200. Therefore, when the receiver 200 acquires the optical ID, the attitude of the receiver 200 that is imaging is appropriately set. It is good to change. When the posture is changed, the receiver 200 can appropriately acquire the light ID at a timing when the posture is such that the light ID can be easily acquired.

FIG. 108 is a diagram illustrating another example of the decoding image Pdec acquired according to the attitude of the receiver 200.

For example, the transmitter 100 is configured as a digital signage of a coffee shop, displays a video relating to a coffee shop advertisement during the video display period, and transmits a light ID by a change in luminance during the light ID transmission period. That is, the transmitter 100 alternately and repeatedly performs video display during the video display period and transmission of the optical ID during the optical ID transmission period.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec by imaging of the transmitter 100. At this time, the decoding cycle including the bright line pattern region is synchronized with the repetition cycle of the video display period and the optical ID transmission period of the transmitter 100 and the repetition cycle of acquisition of the captured display image Ppre and the decoding image Pdec by the receiver 200. The image Pdec may not be acquired. Furthermore, the decoding image Pdec including the bright line pattern region may not be acquired depending on the attitude of the receiver 200.

For example, the receiver 200 images the transmitter 100 in a posture as shown in FIG. That is, the receiver 200 approaches the transmitter 100 and images the transmitter 100 so that the image of the transmitter 100 is projected on the entire image sensor of the receiver 200.

Here, if the timing at which the receiver 200 acquires the captured display image Ppre is within the video display period of the transmitter 100, the receiver 200 appropriately acquires the captured display image Ppre displayed by the transmitter 100. .

Even when the timing at which the receiver 200 acquires the decoding image Pdec extends between the video display period and the optical ID transmission period of the transmitter 100, the receiver 200 includes the bright line pattern region Z1. An image Pdec can be acquired.

That is, the exposure of each exposure line included in the image sensor is started sequentially from the exposure line at the upper end in the vertical direction downward. Therefore, even if the receiver 200 starts exposure of the image sensor to acquire the decoding image Pdec during the video display period, it is not possible to obtain the bright line pattern region. However, when the video display period is switched to the light ID transmission period, a bright line pattern region corresponding to each exposure line that is exposed in the light ID transmission period can be obtained.

Here, the receiver 200 images the transmitter 100 in a posture as shown in FIG. That is, the receiver 200 is separated from the transmitter 100 and images the transmitter 100 so that the image of the transmitter 100 is projected only on the area above the image sensor of the receiver 200. At this time, as described above, if the timing at which the receiver 200 acquires the captured display image Ppre is within the video display period of the transmitter 100, the receiver 200 captures the captured display image Prep on which the transmitter 100 is projected. Get properly. However, when the timing at which the receiver 200 acquires the decoding image Pdec extends between the video display period and the optical ID transmission period of the transmitter 100, the receiver 200 selects the decoding image Pdec including the bright line pattern region. You may not be able to get it. That is, even if the video display period of the transmitter 100 is switched to the optical ID transmission period, the image of the transmitter 100 whose luminance changes is displayed on each exposure line below the image sensor that is exposed during the optical ID transmission period. May not be projected. Therefore, the decoding image Pdec having the bright line pattern region cannot be acquired.

On the other hand, as shown in FIG. 108 (c), the receiver 200 projects the image of the transmitter 100 only on the lower region of the image sensor of the receiver 200 in a state of being separated from the transmitter 100. Then, the transmitter 100 is imaged. At this time, as described above, if the timing at which the receiver 200 acquires the captured display image Ppre is within the video display period of the transmitter 100, the receiver 200 captures the captured display image Prep on which the transmitter 100 is projected. Get properly. Furthermore, even when the timing at which the receiver 200 acquires the decoding image Pdec extends between the video display period and the optical ID transmission period of the transmitter 100, the receiver 200 acquires the decoding image Pdec including the bright line pattern region. There are things that can be done. That is, when the video display period of the transmitter 100 is switched to the optical ID transmission period, the image of the transmitter 100 whose luminance changes is displayed on each exposure line below the image sensor that is exposed during the optical ID transmission period. Projected. Therefore, the decoding image Pdec having the bright line pattern region Z2 can be acquired.

As described above, since the optical ID may not be appropriately acquired according to the attitude of the receiver 200, the receiver 200 may change the attitude of the receiver 200 when acquiring the optical ID. You may be encouraged. That is, when imaging starts, the receiver 200 displays, for example, a message “Please move” or “Shake” or output sound so that the attitude of the receiver 200 changes. Thereby, since the receiver 200 performs imaging while changing the posture, it can appropriately acquire the light ID.

FIG. 109 is a flowchart illustrating an example of processing operation of the receiver 200.

For example, the receiver 200 determines whether or not the receiver 200 is shaken during imaging (step S461). Specifically, the receiver 200 determines whether or not it is shaken based on the output of the 9-axis sensor provided in the receiver 200. Here, if the receiver 200 determines that it is being shaken during imaging (Yes in step S461), the receiver 200 increases the above-described optical ID acquisition rate (step S462). Specifically, the receiver 200 acquires all captured images per unit time obtained during imaging as decoding images (that is, bright line images) Pdec, and decodes all the acquired decoding images. . Alternatively, the receiver 200 starts acquisition and decoding when all the captured images are acquired as the captured display image Ppre, that is, when acquisition and decoding of the decoding image Pdec are stopped.

On the other hand, if the receiver 200 determines that it is not shaken during imaging (No in step S461), the receiver 200 acquires the decoding image Pdec at a low optical ID acquisition rate (step S463). Specifically, if the optical ID acquisition rate is increased in step S462 and is still a high optical ID acquisition rate, the receiver 200 sets the optical ID acquisition rate because the current optical ID acquisition rate is high. Lower. Thereby, since the frequency with which the decoding process of the decoding image Pdec by the receiver 200 is reduced, power consumption can be suppressed.

Then, the receiver 200 determines whether or not an end condition for ending the adjustment process of the optical ID acquisition rate is satisfied (step S464). When the receiver 200 determines that the end condition is not satisfied (No in step S464), The processing from S461 is repeatedly executed. On the other hand, when the receiver 200 determines that the end condition is satisfied (Yes in step S464), the receiver 200 ends the optical ID acquisition rate adjustment process.

FIG. 110 is a diagram illustrating an example of a camera lens switching process by the receiver 200.

The receiver 200 may include a wide-angle lens 211 and a telephoto lens 212 as camera lenses. A captured image obtained by imaging using the wide-angle lens 211 is an image with a wide angle of view, and a subject is projected to be small in the image. On the other hand, a captured image obtained by imaging using the telephoto lens 212 is an image with a narrow angle of view, and a subject is projected greatly in the image.

The receiver 200 as described above may switch the camera lens used for imaging by any one of the methods A to E shown in FIG.

In Method A, the receiver 200 always uses the telephoto lens 212 when imaging, whether in normal imaging or when receiving an optical ID. Here, the case of normal imaging is a case where all captured images are acquired as captured display images Ppre by imaging. The case where the optical ID is received is a case where the captured display image Ppre and the decoding image Pdec are periodically acquired by imaging.

In Method B, the receiver 200 uses the wide-angle lens 211 in the case of normal imaging. On the other hand, when receiving the optical ID, the receiver 200 first uses the wide-angle lens 211. The receiver 200 switches the camera lens from the wide-angle lens 211 to the telephoto lens 212 if the bright line pattern region is included in the decoding image Pdec acquired when the wide-angle lens 211 is used. After this switching, the receiver 200 can acquire a decoding image Pdec with a narrow angle of view, that is, a bright line pattern region appearing large.

In Method C, the receiver 200 uses the wide-angle lens 211 in the case of normal imaging. On the other hand, when receiving the optical ID, the receiver 200 switches the camera lens between the wide-angle lens 211 and the telephoto lens 212. That is, the receiver 200 acquires the captured display image Ppre using the wide-angle lens 211 and acquires the decoding image Pdec using the telephoto lens 212.

In Method D, the receiver 200 switches the camera lens between the wide-angle lens 211 and the telephoto lens 212 in accordance with the operation by the user regardless of whether it is normal imaging or receives an optical ID.

In the method E, when receiving the optical ID, the receiver 200 decodes the decoding image Pdec acquired using the wide-angle lens 211. If the decoding image Pdec cannot be correctly decoded, the camera lens is changed from the wide-angle lens 211 to the telephoto lens 212. Switch. Alternatively, the receiver 200 decodes the decoding image Pdec acquired using the telephoto lens 212, and switches the camera lens from the telephoto lens 212 to the wide-angle lens 211 if it cannot be decoded correctly. Note that when determining whether or not the decoding image Pdec has been correctly decoded, the receiver 200 first transmits an optical ID obtained by decoding the decoding image Pdec to the server. If the optical ID matches the optical ID registered in the server, the server notifies the receiver 200 of the matching information indicating that it matches, and if it does not match, the server does not match. Is sent to the receiver 200. The receiver 200 determines that the decoding image Pdec has been correctly decoded if the information notified from the server is coincidence information. If the information notified from the server is mismatch information, the receiver 200 correctly decodes the decoding image Pdec. Judge that it was not possible. Alternatively, the receiver 200 determines that the decoding image Pdec has been correctly decoded when the optical ID obtained by decoding the decoding image Pdec satisfies a predetermined condition. On the other hand, if the condition is not satisfied, the receiver 200 determines that the decoding image Pdec has not been correctly decoded.

By switching the camera lens in this way, an appropriate decoding image Pdec can be acquired.

FIG. 111 is a diagram illustrating an example of camera switching processing by the receiver 200.

For example, the receiver 200 includes an in camera 213 and an out camera (not shown in FIG. 111) as cameras. The in-camera 213 is also referred to as a face camera or a self-portrait camera, and is a camera arranged on the same surface as the display 201 in the receiver 200. The out camera is a camera arranged on the surface of the receiver 200 opposite to the surface of the display 201.

Such a receiver 200 images the transmitter 100 configured as a lighting device with the in-camera 213 with the in-camera 213 facing upward. By this imaging, the receiver 200 acquires the decoding image Pdec, and acquires the optical ID transmitted from the transmitter 100 by decoding the decoding image Pdec.

Next, the receiver 200 acquires the AR image and the recognition information associated with the optical ID from the server by transmitting the acquired optical ID to the server. The receiver 200 starts a process of recognizing a target area corresponding to the recognition information from the captured display images Ppre obtained by the out camera and the in camera 213, respectively. Here, the receiver 200 prompts the user to move the receiver 200 when the target area cannot be recognized from any of the captured display images Ppre obtained by the out camera and the in camera 213 respectively. The user who is prompted by the receiver 200 moves the receiver 200. Specifically, the user moves the receiver 200 so that the in-camera 213 and the out-camera face the front-rear direction of the user. As a result, the receiver 200 recognizes the target area from the captured display image Ppre acquired by the out camera. That is, the receiver 200 recognizes a region in which a person is projected as a target region, superimposes an AR image on the target region in the captured display image Ppre, and displays the captured display image Ppre on which the AR image is superimposed. To do.

FIG. 112 is a flowchart showing an example of processing operation between the receiver 200 and the server.

The receiver 200 acquires an optical ID transmitted from the transmitter 100 by capturing an image of the transmitter 100, which is a lighting device, with the in-camera 213, and transmits the optical ID to the server (step S471). The server receives the optical ID from the receiver 200 (step S472), and estimates the position of the receiver 200 based on the optical ID (step S473). For example, the server stores, for each light ID, a table indicating a room, a building, a space, or the like in which the transmitter 100 that transmits the light ID is arranged. Then, the server estimates the room associated with the optical ID transmitted from the receiver 200 as the position of the receiver 200 in the table. Further, the server transmits the AR image and recognition information associated with the estimated position to the receiver 200 (step S474).

The receiver 200 acquires the AR image and the recognition information transmitted from the server (Step S475). Here, the receiver 200 starts a process of recognizing a target area corresponding to the recognition information from each captured display image Ppre obtained by each of the out camera and the in camera 213. Then, the receiver 200 recognizes the target area from, for example, the captured display image Ppre acquired by the out-camera (step S476). The receiver 200 superimposes the AR image on the target area in the captured display image Ppre, and displays the captured display image Ppre on which the AR image is superimposed (step S477).

In the above-described example, when the receiver 200 acquires the AR image and the recognition information transmitted from the server, in step S476, the receiver 200 selects from the captured display images Ppre obtained by the out camera and the in camera 213, respectively. The process of recognizing the target area has started. However, the receiver 200 may start the process of recognizing the target area from the captured display image Ppre obtained only by the out-camera in step S476. That is, the camera for acquiring the light ID (in-camera 213 in the above example) and the camera for acquiring the captured display image Pre on which the AR image is superimposed (out-camera in the above example) are always different. It may be allowed.

In the above-described example, the receiver 200 images the transmitter 100 that is an illumination device with the in-camera 213, but the floor illuminated by the transmitter 100 may be captured with the out-camera. Even with such an out-camera imaging, the receiver 200 can acquire the optical ID transmitted from the transmitter 100.

FIG. 113 is a diagram illustrating an example of AR image superimposition performed by the receiver 200.

The receiver 200 images the transmitter 100 configured as a microwave oven installed in a store such as a convenience store. The transmitter 100 includes a camera for imaging the inside of a microwave oven and an illumination device that illuminates the inside of the oven. And the transmitter 100 recognizes the food / beverage (namely, warming object) accommodated in the store | warehouse | chamber by imaging with a camera. In addition, when the food or drink is warmed, the transmitter 100 transmits the light ID indicating the recognized food or drink by causing the lighting device to emit light and changing the luminance of the lighting device. In addition, although this illuminating device illuminates the inside of the store | warehouse | chamber of a microwave oven, the light of the illuminating device is emitted outside from the window part which has the transparency of a microwave oven. Accordingly, the light ID is transmitted from the lighting device to the outside of the microwave oven through the window portion of the microwave oven.

Here, the user purchases food and drink at a convenience store, and puts the food and drink into the transmitter 100, which is a microwave oven, in order to warm the food and drink. At this time, the transmitter 100 recognizes the food and drink with the camera, and starts warming the food and drink while transmitting the optical ID indicating the recognized food and drink.

The receiver 200 acquires the optical ID transmitted from the transmitter 100 by capturing an image of the transmitter 100 that has started the warming, and transmits the optical ID to the server. Next, the receiver 200 acquires an AR image, audio data, and recognition information associated with the optical ID from the server.

The above-mentioned AR image includes an AR image P32a that is a moving image showing a virtual state inside the transmitter 100, an AR image P32b that shows food and drink stored in the cabinet in detail, and steam from the transmitter 100. AR image P32c which shows a state of being heated by a moving image, and AR image P32d which shows the remaining time until the completion of warming of food and drink by a moving image.

For example, if the food and drink stored in the microwave oven is a pizza, the AR image P32a is a video in which a turntable with a pizza is rotating and a plurality of dwarfs are dancing around the pizza. It is. The AR image P32b is, for example, an image showing the product name “pizza” and the material of the pizza if the food and drink stored in the warehouse is a pizza.

Based on the recognition information, the receiver 200 recognizes the area in which the window of the transmitter 100 is projected in the captured display image Ppre as the target area of the AR image P32a, and the AR image P32a is displayed in the target area. Superimpose. Further, based on the recognition information, the receiver 200 recognizes the area above the area where the transmitter 100 is projected in the captured display image Ppre as the target area of the AR image P32b, and the target The AR image P32b is superimposed on the area. Furthermore, based on the recognition information, the receiver 200 recognizes an area between the target area of the AR image P32a and the target area of the AR image P32b in the captured display image Ppre as the target area of the AR image P32c. Then, the AR image P32c is superimposed on the target area. Further, based on the recognition information, the receiver 200 recognizes an area below the area where the transmitter 100 is projected in the captured display image Ppre as the target area of the AR image P32d, and sets the target area as the target area. The AR image P32d is superimposed.

Furthermore, the receiver 200 outputs sound generated when food or drink is heated by reproducing audio data.

The AR images P32a to P32d as described above are displayed by the receiver 200, and further, by outputting sound, it is possible to attract the user's interest to the receiver 200 until the warming of food and drink is completed. . As a result, the burden on the user who is waiting for completion of warming can be reduced. Further, the AR image P32c indicating steam or the like is displayed, and a sound generated when the food or drink is heated is output, so that a sizzle can be given to the user. In addition, the display of the AR image P32d allows the user to easily know the remaining time until the completion of the heating of the food and drink. Therefore, the user can read, for example, a book displayed in the store away from the transmitter 100 that is a microwave oven until the warming is completed. The receiver 200 may notify the user that the warming is completed when the remaining time becomes zero.

In the above-described example, the AR image P32a is a moving image in which a turntable on which a pizza is placed is rotating and a plurality of dwarfs are dancing around the pizza. May be an image that virtually represents. Moreover, although AR image P32b was an image which shows the brand name and material of the food / beverage accommodated in the store | warehouse | chamber, the image which shows a nutrient component or a calorie may be sufficient. Alternatively, the AR image P32b may be an image showing a discount ticket.

As described above, in the display method according to this modification, the subject is a microwave oven provided with an illumination device, and the illumination device illuminates the interior of the microwave oven and changes the luminance to change the light ID of the microwave oven. Send to the outside. In acquiring the captured display image Ppre and the decoding image Pdec, the captured display image Ppre and the decoding image Pdec are acquired by imaging the microwave oven that transmits the optical ID. In the recognition of the target area, the window portion of the microwave oven displayed in the captured display image Ppre is recognized as the target area. In the display of the captured display image Ppre, the captured display image Ppre on which the AR image indicating the state change in the warehouse is superimposed is displayed.

Thereby, since the change in the state of the microwave oven is displayed as an AR image, the state of the oven can be easily communicated to the user of the microwave oven.

FIG. 114 is a sequence diagram showing the processing operation of the system including the receiver 200, the microwave oven, the relay server, and the electronic settlement server. Note that the microwave oven includes a camera and a lighting device as described above, and transmits the light ID by changing the luminance of the lighting device. That is, the microwave oven has a function as the transmitter 100.

First, the microwave oven recognizes food and drink stored in the cabinet with the camera (step S481). Next, the microwave oven transmits a light ID indicating the recognized food and drink to the receiver 200 by a luminance change of the lighting device.

The receiver 200 receives the optical ID transmitted from the microwave oven by imaging the microwave oven (step S483), and transmits the optical ID and the card information to the relay server. The card information is information such as a credit card stored in advance in the receiver 200 and is information necessary for electronic payment.

The relay server holds a table indicating an AR image, recognition information, and product information corresponding to each optical ID. This merchandise information indicates the price of food and drink indicated by the light ID. When such a relay server receives the optical ID and card information transmitted from the receiver 200 (step S486), it finds product information associated with the optical ID from the above table. Then, the relay server transmits the merchandise information and card information to the electronic settlement server (step S486). When the electronic settlement server receives the merchandise information and the card information transmitted from the relay server (step S487), the electronic settlement server performs an electronic settlement process based on the merchandise information and the card information (step S488). Then, when the electronic payment processing is completed, the electronic settlement server notifies the relay server of the completion (step S489).

When the relay server confirms the payment completion notification from the electronic payment server (step S490), the relay server instructs the microwave to start heating the food (step S491). Further, the relay server transmits the AR image and the recognition information associated with the optical ID received in step S485 in the above table to the receiver 200 (step S493).

When the microwave oven receives a warming start instruction from the relay server, it starts warming the food and drink stored in the cabinet (step S492). Further, when the receiver 200 receives the AR image and the recognition information transmitted from the relay server, the target according to the recognition information from the captured display image Ppre periodically acquired by the imaging started from step S483. Recognize the area. Then, the receiver 200 superimposes the AR image on the target area (step S494).

Thereby, the user of the receiver 200 can easily settle the settlement and start warming the food and drink by placing the food and drink in the microwave oven and taking an image. Moreover, when payment cannot be performed, warming of food and drink by a user can be prohibited. Furthermore, when the warming is started, an AR image P32a shown in FIG. 113 can be displayed, and the user can be informed of the state in the warehouse.

FIG. 115 is a sequence diagram showing processing operations of a system including a POS terminal, a server, a receiver 200, and a microwave oven. Note that the microwave oven includes a camera and a lighting device as described above, and transmits the light ID by changing the luminance of the lighting device. That is, the microwave oven has a function as the transmitter 100. A POS (point-of-sale) terminal is a terminal installed in a store such as the same convenience store as the microwave oven.

First, the user of the receiver 200 selects a food or drink as a product at a store and heads to a place where a POS terminal is installed in order to purchase the food or drink. The store clerk operates the POS terminal and receives the price of food and drink from the user. By the operation of the POS terminal by the store clerk, the POS terminal acquires operation input data and sales information (step S501). The sales information indicates, for example, the name, number, and price of the product, the sales location, and the sales date and time. The operation input data indicates, for example, the sex and age of the user input by the store clerk. The POS terminal transmits the operation input data and sales information to the server (step S502). The server receives operation input data and sales information transmitted from the POS terminal (step S503).

On the other hand, when the user of the receiver 200 pays the clerk for the food and drink, the user puts the food and drink in the microwave oven to warm the food and drink. The microwave oven recognizes the food and drink stored in the cabinet with the camera (step S504). Next, the microwave oven transmits the light ID indicating the recognized food and drink to the receiver 200 by the luminance change of the lighting device (step S505). And a microwave oven starts warming of food and drink (step S507).

The receiver 200 receives the optical ID transmitted from the microwave oven by imaging the microwave oven (step S508), and transmits the optical ID and the terminal information to the server (step S509). The terminal information is information stored in advance in the receiver 200 and indicates, for example, the language type (for example, English or Japanese) displayed on the display 201 of the receiver 200.

When the server is accessed from the receiver 200 and receives the optical ID and the terminal information transmitted from the receiver 200, the server determines whether the access from the receiver 200 is the first access (step S510). The first access is the first access performed within a predetermined time from the time when the process of step S503 is performed. If the server determines that the access from the receiver 200 is the first access (Yes in step S510), the server associates and stores the operation input data and the terminal information (step S511).

The server determines whether or not the access from the receiver 200 is the first access, but may determine whether or not the product indicated by the sales information matches the food or drink indicated by the light ID. . In step S511, the server may store not only the operation input data and the terminal information but also the sales information in association with them.

(Indoor use)
FIG. 116 is a diagram showing a situation of indoor use such as an underground shopping mall.

The receiver 200 receives the light ID transmitted from the transmitter 100 configured as a lighting device, and estimates its current position. Further, the receiver 200 displays the current position on a map to provide route guidance, or displays information on nearby stores.

In an emergency, transmission of disaster information and evacuation information from the transmitter 100 can be used when communication is congested, when a communication base station fails, or when radio waves from the communication base station do not reach. Even this information can be obtained. This is effective for a hearing impaired person who has missed an emergency broadcast or cannot hear an emergency broadcast.

That is, the receiver 200 acquires the optical ID transmitted from the transmitter 100 by taking an image, and further acquires the AR image P33 and the recognition information associated with the optical ID from the server. Then, the receiver 200 recognizes the target area corresponding to the recognition information from the captured display image Ppre obtained by the above-described imaging, and superimposes the AR image P33 in the shape of an arrow on the target area. Thereby, receiver 200 can be used as the above-mentioned way finder (refer to Drawing 100).

(Display of augmented reality object)
FIG. 117 is a diagram illustrating a state in which an augmented reality object is displayed.

The stage 2718e that displays the augmented reality is configured as the transmitter 100 described above, and the light emitting patterns and position patterns of the light emitting units 2718a, 2718b, 2718c, and 2718d are the reference positions for displaying the augmented reality object information and the augmented reality object. Send.

The receiver 200 displays the augmented reality object 2718f, which is an AR image, superimposed on the captured image based on the received information.

Note that these comprehensive or specific modes may be realized by a recording medium such as an apparatus, a system, a method, an integrated circuit, a computer program, or a computer-readable CD-ROM. You may implement | achieve in arbitrary combinations of a circuit, a computer program, or a recording medium. Moreover, the computer program which performs the method concerning one Embodiment is preserve | saved at the recording medium of the server, and may be implement | achieved in the aspect delivered to a terminal from a server according to the request | requirement of a terminal.

[Modification 4 of Embodiment 4]
FIG. 118 is a diagram illustrating a configuration of a display system according to the fourth modification of the fourth embodiment.

This display system 500 performs object recognition and augmented reality (Augmented Reality / Mixed Reality) display using visible light signals.

The receiver 200 performs imaging, receives a visible light signal, and extracts feature quantities for object recognition or space recognition. The feature amount extraction is extraction of an image feature amount from a captured image obtained by imaging. The visible light signal may be a visible light adjacent carrier signal such as infrared light or ultraviolet light. In this modification, the receiver 200 is configured as a recognition device that recognizes an object on which an augmented reality image (that is, an AR image) is displayed. In the example shown in FIG. 118, the target object is, for example, the AR target object 501.

The transmitter 100 transmits information such as an ID for identifying itself or the AR object 501 as a visible light signal or a radio wave signal. The ID is identification information such as the above-described optical ID, for example, and the AR object 501 is the above-described target area. The visible light signal is a signal transmitted by a change in luminance of a light source included in the transmitter 100.

The receiver 200 or the server 300 holds the identification information transmitted from the transmitter 100 in association with the AR recognition information and the AR display information. The association may be one-to-one or one-to-many. The AR recognition information is the above-described recognition information, and is information for recognizing the AR object 501 for performing AR display. Specifically, the AR recognition information is the image feature amount (SIFT feature amount, SURF feature amount, ORB feature amount, etc.), color, shape, size, reflectance, transmittance, or three-dimensional of the AR object 501. Model etc. The AR recognition information may include identification information or a recognition algorithm indicating which recognition method is used for recognition. The AR display information is information for performing AR display, and is an image (that is, the above-described AR image), video, audio, three-dimensional model, motion data, display coordinates, display size, transmittance, or the like. Further, the AR display information may be the absolute value or change ratio of each of hue, saturation, and brightness.

The transmitter 100 may also function as the server 300. That is, the transmitter 100 may hold the AR recognition information and the AR display information and transmit the information by wired or wireless communication.

The receiver 200 captures an image with a camera (specifically, an image sensor). The receiver 200 receives a visible light signal or a radio wave signal such as WiFi or Bluetooth (registered trademark). Further, the receiver 200 acquires position information obtained by GPS or the like, information obtained by a gyro sensor or acceleration sensor, and information such as sound from a microphone, and integrates all or some of these information. AR objects existing in the vicinity may be recognized. Further, the receiver 200 may recognize the AR object using only one of the information without integrating the information.

FIG. 119 is a flowchart showing the processing operation of the display system according to the fourth modification of the fourth embodiment.

First, the receiver 200 determines whether or not a visible light signal has already been received (step S521). That is, for example, the receiver 200 determines whether or not the visible light signal indicating the identification information is acquired by photographing the transmitter 100 that transmits the visible light signal according to the luminance change of the light source. At this time, a captured image of the transmitter 100 is acquired by the shooting.

Here, when the receiver 200 determines that the visible light signal has already been received (Y in step S521), the AR object (object, reference point, spatial coordinate, or space in the space is determined from the received information. The position and orientation of the receiver 200 are specified. Furthermore, the receiver 200 recognizes the relative position of the AR object. This relative position is represented by the distance and direction from the receiver 200 to the AR object. For example, the receiver 200 identifies an AR object (that is, a target area that is a bright line pattern area) based on the size and position of the bright line pattern area shown in FIG. 50, and recognizes the relative position of the AR object. To do.

Then, the receiver 200 transmits the information such as an ID included in the visible light signal and the relative position to the server 300, and uses the information and the relative position as a key, thereby registering the AR recognition information in the server 300. And AR display information are acquired (step S522). At this time, the receiver 200 acquires not only the information on the recognized AR object but also information on other AR objects existing in the vicinity of the AR object (that is, the AR recognition information and the AR display information). Also good. Thereby, when another AR object existing in the vicinity is imaged by the receiver 200, the receiver 200 recognizes the other AR object existing in the vicinity quickly and without error. Can do. For example, other AR objects present in the vicinity are different from the first recognized AR object.

Note that the receiver 200 may acquire these pieces of information from the database in the receiver 200 instead of accessing the server 300. The receiver 200 receives these pieces of information after a certain period of time has elapsed from the time of acquisition or a specific process (for example, turning off the screen, pressing a button, ending or stopping an application, displaying an AR image, or another AR object. May be discarded after the recognition). Alternatively, the receiver 200 may decrease the reliability of the information every time a certain period of time has elapsed since the acquisition of the information, and use highly reliable information among the plurality of information. Good.

Here, the receiver 200 may preferentially acquire the AR recognition information of the AR object that is effective in relation to the relative position based on the relative position to each AR object. For example, the receiver 200 acquires a plurality of visible light signals (that is, identification information) by photographing the plurality of transmitters 100 in step S521, and corresponds to the plurality of visible light signals in step S522. A plurality of AR recognition information (that is, image feature amount) is acquired. At this time, in step S522, the receiver 200 selects an image feature amount of the AR object closest to the receiver 200 that performs imaging of the transmitter 100 among the plurality of AR objects. That is, the selected image feature amount is used to specify one AR object (that is, the first object) specified using the visible light signal. Thereby, even if a plurality of image feature amounts are acquired, an appropriate image feature amount can be used for specifying the first object.

On the other hand, when it is determined that the visible light signal is not received (N in step S521), the receiver 200 further determines whether or not the AR recognition information has already been acquired (step S523). If it is determined that the AR recognition information has not been acquired (N in step S523), the receiver 200 does not use identification information such as an ID indicated by the visible light signal, or by image processing, or position information or radio wave information. The AR object candidate is recognized using other information such as (step S524). This process may be performed only by the receiver 200. Alternatively, the receiver 200 may transmit information such as a captured image or an image feature amount of the captured image to the server 300, and the server 300 may recognize the AR object candidate. As a result, the receiver 200 acquires AR recognition information and AR display information corresponding to the recognized candidate from the server 300 or its own database.

After step S522, the receiver 200 determines whether or not the AR object is detected by another method that does not use identification information such as an ID indicated by the visible light signal, such as image recognition (step S525). ). That is, the receiver 200 determines whether or not the AR object is recognized by a plurality of methods. Specifically, the receiver 200 specifies the AR object (that is, the first object) from the captured image using the image feature amount acquired based on the identification information indicated by the visible light signal. Then, the receiver 200 determines whether or not the AR object (that is, the second object) is specified from the captured image by image processing without using such identification information.

Here, when the receiver 200 determines that the AR object has been recognized by a plurality of methods (Y in step S525), the recognition result by the visible light signal is prioritized. That is, the receiver 200 confirms whether or not the AR objects recognized by the respective methods match. If they do not match, the receiver 200 selects one AR object on which the AR image is superimposed in the captured image as the AR object recognized by the visible light signal. Determination is made (step S526). That is, when the first object is different from the second object, the receiver 200 gives priority to the first object and recognizes it as the object on which the AR image is displayed. The object on which the AR image is displayed is an object on which the AR image is superimposed.

Alternatively, the receiver 200 may prioritize a method having a higher priority order based on a priority order assigned to each of the plurality of methods. That is, the receiver 200 recognizes one AR object on which the AR image is superimposed in the captured image from among the AR objects recognized by each method, for example, by a method having the highest priority. AR target is determined. Alternatively, the receiver 200 may determine one AR object on which the AR image is superimposed in the captured image by a majority decision or a majority decision with priority. If the recognition result up to that point is overturned by this process, the receiver 200 performs an error handling process.

Next, the receiver 200 determines the state of the AR object in the captured image (specifically, the absolute position, the relative position from the receiver 200, the size, the angle, and the illumination status) based on the acquired AR recognition information. Or occlusion or the like) (step S527). Then, the receiver 200 displays the AR display information (that is, the AR image) superimposed on the captured image in accordance with the recognition result (step S528). That is, the receiver 200 superimposes the AR display information on the recognized AR object in the captured image. Alternatively, the receiver 200 displays only the AR display information.

These enable recognition or detection that is difficult with image processing alone. The difficult recognition or detection includes, for example, identification of AR objects that are image-similar (such as only different text content), detection of AR objects with fewer patterns, and AR objects with high reflectivity or transmittance. Detection of an object, detection of an AR object (for example, an animal) whose shape or pattern changes, or detection of an AR object from a wide angle (in various directions). That is, in this modification, recognition of these AR objects and AR display can be performed. In addition, in image processing that does not use a visible light signal, as the number of AR objects to be recognized increases, it takes time to search for neighborhoods of image feature values, which takes time for recognition processing, and the recognition rate also deteriorates. . However, in this modification, the influence of the increase in recognition time and the deterioration of the recognition rate due to the increase in recognition objects is not at all or extremely small, and effective AR object recognition is possible. Further, efficient recognition is possible by using the relative position of the AR object. For example, by using an approximate distance to the AR object, processing for making the AR object size independent of the calculation of the image feature amount can be omitted, or a size-dependent feature can be used. Can do. In addition, when the angle of the AR object is used and it is usually necessary to evaluate the image feature amount for many angles, only the retention and calculation of the image feature amount corresponding to the angle of the AR object is performed. The calculation speed or the memory efficiency can be improved.

[Summary of Modification 4 of Embodiment 4]
FIG. 120 is a flowchart illustrating a recognition method according to an aspect of the present invention.

The display method according to one aspect of the present invention is a method for recognizing an object on which an augmented reality image (AR image) is displayed, and includes steps S531 to S535.

In step S531, the receiver 200 acquires the identification information by photographing the transmitter 100 that transmits a visible light signal by a change in luminance of the light source. The identification information is, for example, an optical ID. In step S <b> 532, the receiver 200 transmits the identification information to the server 300 and acquires an image feature amount corresponding to the identification information from the server 300. The image feature amount is indicated as AR recognition information or recognition information.

In step S533, the receiver 200 specifies the first object from the captured image of the transmitter 100 using the image feature amount. In step S534, the receiver 200 specifies the second object from the captured image of the transmitter 100 by image processing without using the identification information (that is, the optical ID).

In step S535, when the first object specified in step S533 is different from the second object specified in step S534, the receiver 200 gives priority to the first object and augments reality. Recognize as an object to be displayed.

For example, the augmented reality image, the captured image, and the target object correspond to the AR image, the captured display image, and the target region in the fourth embodiment and the modifications thereof, respectively.

Thereby, as shown in FIG. 119, the first object specified by using the identification information indicated by the visible light signal, and the second object specified by the image processing without using the identification information, Are different, the first object is preferentially recognized as the object on which the augmented reality image is displayed. Therefore, the object on which the augmented reality image is displayed can be appropriately recognized from the captured image.

In addition to the image feature amount of the first object, the image feature amount includes an image feature amount of a third object that is located in the vicinity of the first object and is different from the first object. May be.

Thereby, as shown in step S522 of FIG. 119, not only the image feature amount of the first object but also the image feature amount of the third object is acquired. When appearing in the captured image, the third object can be quickly identified or recognized.

In addition, the receiver 200 may acquire a plurality of identification information by photographing a plurality of transmitters in step S531, and may acquire a plurality of image feature amounts corresponding to the plurality of identification information in step S532. is there. In such a case, in step S533, the receiver 200 determines the image feature amount of the object closest to the receiver 200 that performs imaging by the plurality of transmitters among the plurality of objects as the first object. It may be used to identify

Thereby, as shown in step S522 of FIG. 119, even if a plurality of image feature amounts are acquired, an appropriate image feature amount can be used to identify the first object.

Note that the recognition device in the present modification is, for example, a device provided in the receiver 200 described above, and includes a processor and a recording medium. On this recording medium, a program for causing the processor to execute the recognition method shown in FIG. 120 is recorded. Moreover, the program in this modification is a program which makes a computer perform the recognition method shown in FIG.

(Embodiment 5)
FIG. 121 is a diagram showing an example of an operation mode of a visible light signal according to this embodiment.

As shown in FIG. 121, there are two modes for the operation mode of the physical (PHY) layer of the visible light signal. The first operation mode is a mode in which packet PWM (Pulse Width Modulation) is performed, and the second operation mode is a mode in which packet PPM (Pulse-Position Modulation) is performed. The transmitter according to each of the above-described embodiments or modifications thereof generates and transmits a visible light signal by modulating a signal to be transmitted according to any one of these operation modes.

In the operation mode of the packet PWM, RLL (Run-Length Limited) encoding is not performed, the optical clock rate is 100 kHz, forward error correction (FEC) is repeatedly encoded, and a typical data rate is 5.5 kbps. It is.

In this packet PWM, the pulse width is modulated, and the pulse is represented by two brightness states. The two brightness states are a bright state (Bright or High) and a dark state (Dark or Low), but typically the light is on and off. A chunk of a physical layer signal called a packet (also referred to as a PHY packet) corresponds to a MAC (medium access control) frame. The transmitter can repeatedly transmit PHY packets and transmit a set of a plurality of PHY packets regardless of a special order.

Note that the packet PWM is used to generate a visible light signal transmitted from a normal transmitter.

In the operation mode of packet PPM, RLL encoding is not performed, the optical clock rate is 100 kHz, forward error correction (FEC) is repeatedly encoded, and a typical data rate is 8 kbps.

In this packet PPM, the position of a short time length pulse is modulated. That is, this pulse is a bright pulse of a bright pulse (High) and a dark pulse (Low), and the position of this pulse is modulated. The position of this pulse is indicated by the interval between the pulse and the next pulse.

Packet PPM realizes deep dimming. The format, waveform, and characteristics of the packet PPM not described in each embodiment and its modification are the same as those of the packet PWM. The packet PPM is used to generate a visible light signal transmitted from a transmitter having a light source that emits very bright light.

In each of the packet PWM and the packet PPM, dimming in the physical layer of the visible light signal is controlled by the average luminance of the optional field.

FIG. 122A is a flowchart showing another visible light signal generation method according to Embodiment 5. This visible light signal generation method is a method for generating a visible light signal transmitted by a change in luminance of a light source provided in a transmitter, and includes steps SE1 to SE3.

In step SE1, a preamble which is data in which each of the first and second luminance values, which are different luminance values, appears alternately along the time axis is generated.

In step SE2, in the data in which the first and second luminance values appear alternately along the time axis, an interval from when the first luminance value appears until the next first luminance value appears is set as a transmission target. 1st payload is produced | generated by determining according to the system according to this signal.

In step SE3, a visible light signal is generated by combining the preamble and the first payload.

FIG. 122B is a block diagram illustrating a configuration of another signal generation device according to Embodiment 5. The signal generation device E10 is a signal generation device that generates a visible light signal transmitted by the luminance change of the light source provided in the transmitter, and includes a preamble generation unit E11, a payload generation unit E12, and a combining unit E13. . In addition, the signal generation device E10 executes the processing of the flowchart shown in FIG. 122A.

That is, the preamble generation unit E11 generates a preamble that is data in which the first and second luminance values, which are different luminance values, appear alternately on the time axis.

In the data in which the first and second luminance values appear alternately along the time axis, the payload generation unit E12 sets an interval from when the first luminance value appears until the next first luminance value appears. A first payload is generated by determining according to a method according to a signal to be transmitted.

The combining unit E13 generates a visible light signal by combining the preamble and the first payload.

For example, the first and second luminance values are Bright (High) and Dark (Low), and the first payload is a PHY payload. By transmitting the visible light signal generated in this way, the number of received packets can be increased and the reliability can be increased. As a result, communication between various devices can be enabled.

For example, the time length of the first luminance value in each of the preamble and the first payload is 10 μsec or less.

This makes it possible to suppress the average luminance of the light source while performing visible light communication.

Also, the preamble is a header for the first payload, and the time length of the header includes three intervals from when the first luminance value appears until the next first luminance value appears. Here, each of the three intervals is 160 μs. That is, a pattern of intervals between pulses included in the header (SHR) in mode 1 of the packet PPM is defined. Each of the pulses is a pulse having a first luminance value, for example.

Also, the preamble is a header for the first payload, and the time length of the header includes three intervals from when the first luminance value appears until the next first luminance value appears. Here, the first interval among the three intervals is 160 μsec, the second interval is 180 μsec, and the third interval is 160 μsec. That is, a pattern of intervals between pulses included in the header (SHR) in mode 2 of the packet PPM is defined.

Also, the preamble is a header for the first payload, and the time length of the header includes three intervals from when the first luminance value appears until the next first luminance value appears. Here, the first one of the three intervals is 80 μs, the second interval is 90 μs, and the third interval is 80 μs. That is, a pattern of intervals between pulses included in the header (SHR) in mode 3 of the packet PPM is defined.

Thus, since the header patterns of the mode 1, mode 2 and mode 3 of the packet PPM are defined in this way, the receiver can appropriately receive the first payload in the visible light signal.

The signal to be transmitted consists of 6 bits from the first bit x ₀ to the bit x ₅ in the sixth, the time length of the first payload, first from the first luminance value appears in the following 1 Two intervals until the brightness value of 2 appears. Here, when the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4 (k is 0 or 1), in the generation of the first payload, in the first payload, Each of the two intervals is determined according to the interval P _k = 180 + 30 × y _k [μsec], which is the above-described method. That is, in the mode 1 of the packet PPM, the signal to be transmitted is modulated as an interval between each pulse included in the first payload (PHY payload).

The signal to be transmitted consists of 12 bits from the first bit x ₀ to bit x ₁₁ of the 12, the time length of the first payload, first from the first luminance value appears in the following 1 This includes four intervals until the luminance value appears. Here, when the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4 (k is 0, 1, 2, or 3), in the generation of the first payload, Each of the four intervals in one payload is determined according to the interval P _k = 180 + 30 × y _k [μsec], which is the above-described method. That is, in the mode 2 of the packet PPM, the signal to be transmitted is modulated as an interval between each pulse included in the first payload (PHY payload).

The signal to be transmitted consists of 3n bits from the first bit x ₀ through bit x _3n-1 of the 3n (n is an integer of 2 or more), the time length of the first payload, a first It includes n intervals from when the luminance value appears until the next first luminance value appears. Here, when the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4 (k is an integer from 0 to (n−1)), the first payload is generated. Then, each of the n intervals in the first payload is determined according to the interval P _k = 100 + 20 × y _k [μsec], which is the above-described method. That is, in the mode 3 of the packet PPM, the signal to be transmitted is modulated as an interval between pulses included in the first payload (PHY payload).

As described above, in the mode 1, mode 2 and mode 3 of the packet PPM, the signal to be transmitted is modulated as an interval between each pulse, so that the receiver appropriately transmits a visible light signal based on the interval. It can be demodulated into the signal of interest.

In the visible light signal generation method, a footer for the first payload may be further generated. In the generation of the visible light signal, the footer may be combined next to the first payload. That is, in mode 3 of the packet PWM and the packet PPM, the footer (SFT) is transmitted following the first payload (PHY payload). Thereby, since the end of the first payload can be clearly specified by the footer, visible light communication can be performed efficiently.

In addition, when generating a visible light signal, if a footer is not transmitted, a header for a signal next to a transmission target signal may be combined instead of the footer. That is, in the mode 3 of the packet PWM and the packet PPM, instead of the footer (SFT), the header (SHR) for the next first payload is transmitted following the first payload (PHY payload). Thus, the end of the first payload can be clearly specified by the header for the next first payload, and since the footer is not transmitted, visible light communication can be performed more efficiently.

In each of the above-described embodiments and modifications, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. For example, the program causes the computer to execute the visible light signal generation method shown by the flowchart of FIG. 122A.

As described above, the visible light signal generation method according to one or a plurality of aspects has been described based on the above-described embodiments and modifications. However, the present invention is not limited to the embodiments. Unless it deviates from the gist of the present invention, various modifications conceived by those skilled in the art have been made in the present embodiment, and forms constructed by combining components in different embodiments and modifications are also within the scope of the present invention. May be included.

(Embodiment 6)
In this embodiment, a visible light signal decoding method, an encoding method, and the like will be described.

FIG. 123 is a diagram showing a format of a MAC frame in MPM.

The format of the MAC (medium access control) header in MPM (Mirror Pulse Modulation) consists of MHR (medium access control header) and MSDU (medium access control service-data unit). The MHR field includes a sequence number subfield. The MSDU includes a frame payload and has a variable length. The bit length of MPDU (medium access control control protocol-data unit) including MHR and MSDU is set as macMpmMpduLength.

Note that the MPM is a modulation method in the fifth embodiment, for example, a method for modulating information or signals to be transmitted as shown in FIG.

FIG. 124 is a flowchart showing the processing operation of the encoding device for generating a MAC frame in MPM. Specifically, FIG. 124 is a diagram illustrating how to determine the bit length of the sequence number subfield. Note that the encoding device is provided in, for example, the above-described transmitter or transmission device that transmits a visible light signal.

The sequence number subfield includes a frame sequence number (also called a sequence number). The bit length of the sequence number subfield is set as macMpmSnLength. When the bit length of the sequence number subfield is set to a variable length, the first bit in the sequence number subfield is used as the last frame flag. That is, in this case, the sequence number subfield includes a final frame flag and a bit string indicating the sequence number. The final frame flag is set to 1 for the final frame, and is set to 0 for the other frames. That is, the final frame flag indicates whether or not the processing target frame is the final frame. This final frame flag corresponds to the above-described stop bit. The sequence number corresponds to the above address.

First, the encoding device determines whether or not SN is set to a variable length (step S101a). SN is the bit length of the sequence number subfield. That is, the encoding apparatus determines whether macMpmSnLength indicates 0xf. When macMpmSnLength indicates 0xf, SN is a variable length, and when macMpmSnLength indicates other than 0xf, SN is a fixed length. When determining that the SN is not set to a variable length, that is, the SN is set to a fixed length (N in step S101a), the encoding apparatus determines the SN to a value indicated by macMpmSnLength (step S102a). . At this time, the encoding apparatus does not use the final frame flag (that is, LFF).

On the other hand, when determining that the SN is set to a variable length (Y in step S101a), the encoding apparatus determines whether the processing target frame is the last frame (step S103a). Here, when the encoding apparatus determines that the processing target frame is the final frame (Y in step S103a), the encoding apparatus determines SN to be 5 bits (step S104a). At this time, the encoding apparatus determines a final frame flag indicating 1 as the first bit in the sequence number subfield.

Further, when the encoding apparatus determines that the processing target frame is not the final frame (N in step S103a), the encoding apparatus determines whether the value of the sequence number of the final frame is 1-15 (step S105a). The sequence number is an integer assigned to each frame in ascending order from 0. In the case of N in step S103a, the number of frames is 2 or more. Therefore, in this case, the value of the sequence number of the last frame can take any of 1-15 except 0.

If the encoding apparatus determines in step S105a that the value of the sequence number of the final frame is 1, it determines SN as 1 bit (step S106a). At this time, the encoding apparatus determines that the value of the last frame flag which is the first bit in the sequence number subfield is 0.

For example, when the sequence number value of the last frame is 1, the sequence number subfield of the last frame is represented as (1, 1) including the last frame flag (1) and the sequence number value (1). . At this time, the encoding apparatus determines that the bit length of the sequence number subfield of the processing target frame is 1 bit. That is, the encoding apparatus determines a sequence number subfield including only the last frame flag (0).

If the encoding device determines in step S105a that the value of the sequence number of the last frame is 2, it determines SN to 2 bits (step S107a). Also at this time, the encoding apparatus determines the value of the final frame flag as 0.

For example, when the sequence number value of the last frame is 2, the sequence number subfield of the last frame is represented as (1, 0, 1) including the last frame flag (1) and the sequence number value (2). Is done. The sequence number is indicated by a bit string. In the bit string, the leftmost bit is LSB (leastlesignificant bit) and the rightmost bit is MSB (most significant bit). Therefore, the value (2) of the sequence number is expressed as a bit string (0, 1). Thus, when the value of the sequence number of the last frame is 2, the encoding apparatus determines the bit length of the sequence number subfield of the processing target frame to be 2 bits. That is, the encoding apparatus determines the sequence number subfield including the last frame flag (0) and the bit (0) or (1) indicating the sequence number.

If the encoding device determines in step S105a that the value of the sequence number of the final frame is 3 or 4, it determines SN to 3 bits (step S108a). Also at this time, the encoding apparatus determines the value of the final frame flag as 0.

If the encoding apparatus determines in step S105a that the value of the sequence number of the last frame is any integer of 5-8, it determines SN as 4 bits (step S109a). Also at this time, the encoding apparatus determines the value of the final frame flag as 0.

When the encoding apparatus determines in step S105a that the sequence number value of the final frame is any integer of 9-15, it determines SN as 5 bits (step S110a). Also at this time, the encoding apparatus determines the value of the final frame flag as 0.

FIG. 125 is a flowchart showing the processing operation of the decoding device for decoding the MAC frame in MPM. Specifically, FIG. 125 is a diagram showing how to determine the bit length of the sequence number subfield. Note that the decoding device is provided, for example, in the above-described receiver or receiving device that receives a visible light signal.

Here, the decoding apparatus determines whether or not SN is set to a variable length (step S201a). That is, the decoding apparatus determines whether macMpmSnLength indicates 0xf. If the decoding apparatus determines that SN is not set to a variable length, that is, that SN is set to a fixed length (N in step S201a), SN is determined to be a value indicated by macMpmSnLength (step S202a). At this time, the decoding device does not use the final frame flag (ie, LFF).

On the other hand, when the decoding apparatus determines that the SN is set to a variable length (Y in step S201a), the decoding apparatus determines whether the value of the final frame flag of the decoding target frame is 1 or 0 (step S203a). . That is, the decoding apparatus determines whether or not the decoding target frame is the last frame. Here, when the decoding apparatus determines that the value of the final frame flag is 1 (1 in step S203a), it determines SN to be 5 bits (step S204a).

When the decoding apparatus determines that the value of the last frame flag is 0 (0 in step S203a), the value indicated by the bit string from the second bit to the fifth bit in the sequence number subfield of the last frame is 1. It is determined which of −15 (step S205a). The final frame has a final frame flag indicating 1 and is a frame generated from the same source as the decoding target frame. Each source is specified by a position in the captured image. Note that the source is divided into, for example, a plurality of frames (that is, packets). That is, the last frame is the last frame among a plurality of frames generated by dividing one source. The value indicated by the bit string from the second bit to the fifth bit in the sequence number subfield is the value of the sequence number.

If the decoding apparatus determines in step S205a that the value indicated by the bit string is 1, it determines the SN to be 1 bit (step S206a). For example, when the sequence number subfield of the last frame is 2 bits of (1, 1), the last frame flag is 1, and the sequence number of the last frame, that is, the value indicated by the bit string is 1. At this time, the decoding apparatus determines that the bit length of the sequence number subfield of the decoding target frame is 1 bit. That is, the decoding apparatus determines (0) as the sequence number subfield of the decoding target frame.

If the decoding apparatus determines in step S205a that the value indicated by the bit string is 2, the decoding apparatus determines SN to be 2 bits (step S207a). For example, if the sequence number subfield of the last frame is 3 bits (1, 0, 1), the last frame flag is 1, and the sequence number of the last frame, that is, the value indicated by the bit string (0, 1). Is 2. In the above bit string, the leftmost bit is LSB (least significant bit) and the rightmost bit is MSB (most） significant bit). At this time, the decoding apparatus determines that the bit length of the sequence number subfield of the decoding target frame is 2 bits. That is, the decoding apparatus determines the sequence number subfield of the decoding target frame to be (0, 0) or (0, 1).

If it is determined in step S205a that the value indicated by the bit string is 3 or 4, the decoding apparatus determines SN to be 3 bits (step S208a).

If the decoding apparatus determines in step S205a that the value indicated by the bit string is any integer of 5-8, it determines SN as 4 bits (step S209a).

When the decoding apparatus determines in step S205a that the value indicated by the bit string is any integer of 9-15, it determines SN as 5 bits (step S210a).

FIG. 126 is a diagram showing the attributes of the MAC PIB.

The MAC PIB (physical-layer personal-area-network information base) attributes include macMpmSnLength and macMpmMpduLength. macMpmSnLength is any integer value in the range of 0x0-0xf, and indicates the bit length of the sequence number subfield. Specifically, when macMpmSnLength is any integer value in the range of 0x0-0xe, the integer value is indicated as a fixed bit length of the sequence number subfield. When macMpmSnLength is 0xf, it indicates that the bit length of the sequence number subfield is variable.

MacMpmMpduLength is any integer value in the range of 0x00-0xff and indicates the bit length of the MPDU.

FIG. 127 is a diagram for explaining an MPM dimming method.

MPM has a dimming function. For example, FIG. 127 shows (a) an analog dimming method, (b) a PWM dimming method, (c) a VPPM dimming method, and (d) a field insertion dimming method.

In the analog dimming method, for example, as shown in (a2), the visible light signal is transmitted by changing the luminance. Here, when darkening the visible light signal, for example, as shown in (a1), the overall luminance of the visible light signal is lowered. Conversely, when the visible light signal is brightened, for example, as shown in (a3), the overall luminance of the visible light signal is increased.

In the PWM dimming method, for example, as shown in (b2), the visible light signal is transmitted by changing the luminance. Here, when darkening the visible light signal, for example, as shown in (b1), the luminance is lowered for a short period in the period in which the high-luminance light shown in (b2) is output. On the contrary, when the visible light signal is brightened, for example, as shown in (b3), the luminance is raised only for a short period in the period when the low-luminance light shown in (b2) is output. It should be noted that the short period described above must be less than 1/3 of the original pulse width and less than 50 μsec.

In the VPPM dimming method, for example, as shown in (c2), the visible light signal is transmitted by changing the luminance. Here, when darkening the visible light signal, for example, as shown in (c1), the timing of the fall of the brightness is advanced. On the other hand, when the visible light signal is brightened, for example, as shown in (c3), the timing of the fall of the luminance is delayed. The VPPM modulation method can be used only for the PHY PPM mode in MPM.

In the field insertion dimming method, for example, as shown in (d2), a visible light signal including a plurality of PPDUs (physical-layer data units) is transmitted. Here, when darkening the visible light signal, for example, as shown in (d1), a dimming field having a luminance lower than the luminance of the PPDU is inserted between the PPDUs. On the other hand, when the visible light signal is brightened, for example, as shown in (d3), a dimming field having a luminance higher than that of the PPDU is inserted between the PPDUs.

FIG. 128 is a diagram showing attributes of the PHY PIB.

PHY (physical layer) PIB attributes include phyMpmMode, phyMpmPlcpHeaderMode, phyMpmPlcpCenterMode, phyMpmSymbolSize, phyMpmOddSymbolBit, phyMpmEvenSymbolBit, phyMpmSymbolOffset, and phyMpmSymbolUnit.

PhyMpmMode is 0 or 1, and indicates the PHY mode of MPM. Specifically, when phyMpmMode is 0, it indicates that the PHY mode is the PWM mode, and when it is 1, it indicates that the PHY mode is the PWM mode.

PhyMpmPlcpHeaderMode is any integer value in the range of 0x0-0xf, and indicates a PLCP (Physical Layer Conversion Protocol) header subfield mode and a PLCP footer subfield mode.

PhyMpmPlcpCenterMode is any integer value in the range of 0x0-0xf and indicates the PLCP center subfield mode.

PhyMpmSymbolSize is any integer value in the range of 0x0-0xf, and indicates the number of symbols in the payload subfield. Specifically, when phyMpmSymbolSize is 0x0, it indicates that the number of symbols is variable, and is referred to as N.

phyMpmOddSymbolBit is an any integer value in the range of up to OxO-Oxf, shows the bit length included in each odd symbol in the payload field and are referred to as M _odd.

phyMpmEvenSymbolBit is any integer value in the range of 0x0-0xf, indicates the bit length included in each even symbol of the payload subfield, and is referred to as M _even .

phyMpmSymbolOffset is an any integer value in the range of up 0x00-0xFF, it indicates the offset value of the symbol of the payload field and are referred to as W _1.

phyMpmSymbolUnit is an any integer value in the range of up 0x00-0xFF, indicates the unit value of the symbol of the payload field and are referred to as W _2.

FIG. 129 is a diagram for explaining the MPM.

MPM consists only of PSDU (PHY service data unit) fields. The PSDU field includes an MPDU converted by the MPM PLCP.

MPM PLCP converts MPDU into 5 subfields as shown in FIG. The five subfields are a PLCP header subfield, a front payload subfield, a PLCP center subfield, a back payload subfield, and a PLCP footer subfield. The PHY mode of MPM is set as phyMpmMode.

As shown in FIG. 129, the MPM PLCP includes a bit rearrangement unit 301a, a duplication unit 302a, a front conversion unit 303a, and a back conversion unit 304a.

Here, (x ₀ , x ₁ , x ₂ ,...) Is each bit included in the MPDU, L _SN is the bit length of the sequence number subfield, and N is each payload subfield. The number of symbols. The bit rearrangement unit 301a rearranges (x ₀ , x ₁ , x ₂ ,...) To (y ₀ , y ₁ , y ₂ ,...) According to the following (Equation 1).

This relocation, each bit included in the sequence number subfield at the head of the MPDU, it moves rearward only L _SN. The duplication unit 302a duplicates the MPDU after the bit rearrangement.

Each of the front payload subfield and the back payload subfield consists of N symbols. Here, M _odd is the bit length included in the odd-numbered symbol, M _even is the bit length included in the even-numbered symbol, and W ₁ is the symbol value offset (the above-described offset value). , _{W 2} is the symbol value unit (unit value described above). N, M _odd , M _even , W ₁ , and W ₂ are set by the PHY PIB shown in FIG.

The front conversion unit 303a and the back conversion unit 304a convert payload bits (y0, y1, y2,...) Of the rearranged MPDU into z _i according to the following (Expression 2) to (Expression 5).

The front conversion unit 303a calculates the i-th symbol (that is, the symbol value) of the front payload subfield using z _i by the following (Equation 6).

The back conversion unit 304a calculates the i-th symbol (that is, the symbol value) of the back payload subfield using z _{i according} to (Equation 7) below.

FIG. 130 shows a PLCP header subfield.

As shown in FIG. 130, the PLCP header subfield is composed of four symbols in the PWM mode, and is composed of three symbols in the PPM mode.

FIG. 131 is a diagram showing a PLCP center subfield.

As shown in FIG. 131, the PLCP center subfield includes four symbols in the PWM mode and three symbols in the PPM mode.

FIG. 132 is a diagram showing a PLCP footer subfield.

As shown in FIG. 132, the PLCP footer subfield is composed of four symbols in the PWM mode, and is composed of three symbols in the PPM mode.

FIG. 133 is a diagram showing a PHY PWM mode waveform in the MPM.

In PWM mode, a symbol must be transmitted as one of two states of light intensity: bright or dark. In the PHY PWM mode in MPM, the symbol value corresponds to a continuous time in microseconds. For example, as shown in FIG. 133, the first symbol value corresponds to the continuous time of the first bright state, and the second symbol value corresponds to the continuous time of the next dark state. In the example shown in FIG. 133, the initial state of each subfield is a bright state, but may be a dark state.

FIG. 134 is a diagram showing a PHY PPM mode waveform in MPM.

In the PPM mode, as shown in FIG. 134, the symbol value represents the time from the start of the bright state to the start of the next bright state in units of microseconds. The bright state time must be shorter than 90% of the symbol value.

For both modes, the transmitter can transmit only some of the symbols. However, the transmitter must transmit all the symbols of the PLCP center subfield and at least N symbols. Each of the at least N symbols is a symbol included in either the front payload subfield or the back payload subfield.

(Summary of Embodiment 6)
FIG. 135 is a flowchart illustrating an example of the decoding method according to the sixth embodiment. Note that the flowchart shown in FIG. 135 corresponds to the flowchart shown in FIG.

This decoding method is a method for decoding a visible light signal composed of a plurality of frames, and includes step S310b, step S320b, and step S330b as shown in FIG. Each of the plurality of frames includes a sequence number and a frame payload.

In step S310b, based on macSnLength which is information for determining the bit length of the subfield in which the sequence number is stored in the decoding target frame, it is determined whether or not the bit length of the subfield is variable Process.

In step S320b, the bit length of the subfield is determined based on the result of the variable length determination process. In step S330b, the decoding target frame is decoded based on the determined bit length of the subfield.

Here, the determination of the bit length of the subfield in step S320b includes steps S321b to S324b.

That is, in the variable length determination process of step S310b, when it is determined that the bit length of the subfield is not variable, the bit length of the subfield is determined to the value indicated by the above macSnLength (step S321b). ).

On the other hand, if it is determined in the variable length determination process in step S310b that the bit length of the subfield is variable length, it is determined whether or not the decoding target frame is the last frame of the plurality of frames. A final determination process is performed (step S322b). If it is determined that the frame is the final frame (Y in step S322b), the bit length of the subfield is determined to be a predetermined value (step S323b). On the other hand, if it is determined that the frame is not the last frame (N in step S322b), the bit length of the subfield is determined based on the sequence number value of the last frame (step S324b).

As a result, as shown in FIG. 135, regardless of whether the bit length of the subfield in which the sequence number is stored (specifically, the sequence number subfield) is fixed or variable, the subfield The bit length can be determined appropriately.

Here, in the final determination process in step S322b, it may be determined whether the decoding target frame is the final frame based on the final frame flag indicating whether the decoding target frame is the final frame. Specifically, in the final determination process in step S322b, when the final frame flag indicates 1, when it is determined that the decoding target frame is the final frame, and when the final frame flag indicates 0, the decoding target frame May not be the last frame. For example, the last frame flag may be included in the first bit of the subfield.

Thereby, as shown in step S203a of FIG. 125, it is possible to appropriately determine whether or not the decoding target frame is the final frame.

More specifically, in the determination of the bit length of the subfield in step S320b, if it is determined in the final determination process in step S322b that the decoding target frame is the final frame, the bit length of the subfield is set as described above. The predetermined value of 5 bits may be determined. That is, as shown in step S204a of FIG. 125, the bit length SN of the subfield is determined to be 5 bits.

Further, in the determination of the bit length of the subfield in step S320b, when it is determined in the final determination process in step S322b that the decoding target frame is not the final frame, the sequence number value of the final frame is 1. The bit length of the subfield may be determined as 1 bit. Further, when the value of the sequence number of the last frame is 2, the bit length of the subfield may be determined to be 2 bits. Further, when the value of the sequence number of the last frame is 3 or 4, the bit length of the subfield may be determined to be 3 bits. Further, when the value of the sequence number of the last frame is any integer from 5 to 8, the bit length of the subfield may be determined to be 4 bits. Further, when the value of the sequence number of the last frame is any integer from 9 to 15, the bit length of the subfield may be determined to be 5 bits. That is, as shown in steps S206a to S210a in FIG. 125, the bit length SN of the subfield is determined to be any one of 1 to 5 bits.

FIG. 136 is a flowchart showing an example of the encoding method according to the sixth embodiment. The flowchart shown in FIG. 136 corresponds to the flowchart shown in FIG.

This encoding method is a method of encoding information to be encoded into a visible light signal composed of a plurality of frames. As shown in FIG. 136, step S410a, step S420a, step S430a, including. Each of the plurality of frames includes a sequence number and a frame payload.

In step S410a, based on macSnLength which is information for determining the bit length of the subfield in which the sequence number is stored in the processing target frame, it is determined whether or not the bit length of the subfield is variable. Process.

In step S420a, the bit length of the subfield is determined based on the result of the variable length determination process. In step S430a, based on the determined bit length of the subfield, a part of the encoding target information is encoded into the processing target frame.

Here, the determination of the bit length of the subfield in step S420a includes steps S421a to S424a.

That is, in the variable length determination process in step S410a, when it is determined that the bit length of the subfield is not variable length, the bit length of the subfield is determined to the value indicated by the above-described macSnLength (step S421a). ).

On the other hand, if it is determined in the variable length determination process in step S410a that the bit length of the subfield is variable length, it is determined whether or not the processing target frame is the last frame of the plurality of frames. A final determination process is performed (step S422a). If it is determined that the frame is the last frame (Y in step S422a), the bit length of the subfield is determined to be a predetermined value (step S423a). On the other hand, when it is determined that the frame is not the last frame (N in step S422a), the bit length of the subfield is determined based on the sequence number value of the last frame (step S424a).

As a result, as shown in FIG. 136, regardless of whether the bit length of the subfield in which the sequence number is stored (specifically, the sequence number subfield) is fixed or variable, the subfield The bit length can be determined appropriately.

Note that the decoding device according to the present embodiment includes a processor and a memory, and a program that causes the processor to execute the decoding method shown in FIG. 135 is recorded in the memory. The encoding apparatus according to the present embodiment includes a processor and a memory, and a program that causes the processor to execute the encoding method shown in FIG. 136 is recorded in the memory. Further, the program in the present embodiment is a program that causes a computer to execute the decoding method shown in FIG. 135 or the encoding method shown in FIG. 136.

(Embodiment 7)
In this embodiment, a transmission method for transmitting an optical ID by a visible light signal will be described. Note that the transmitter and the receiver in this embodiment may have the same functions and configurations as the transmitter (or the transmission device) and the receiver (or the reception device) in each of the above embodiments.

FIG. 137 is a diagram illustrating an example in which the receiver according to the present embodiment displays an AR image.

The receiver 200 in the present embodiment is a receiver including an image sensor and a display 201, and is configured as a smartphone, for example. Such a receiver 200 acquires the above-described captured display image Pa, which is the normal captured image, and the above-described decoding image, which is the visible light communication image or the bright line image, by capturing the subject with the image sensor.

Specifically, the image sensor of the receiver 200 images the transmitter 100. The transmitter 100 has a shape such as a light bulb, for example, and includes a glass bulb 141 and a light emitting unit 142 that sways while shining like a flame inside the glass bulb 141. The light emitting unit 142 emits light when one or more light emitting elements (for example, LEDs) provided in the transmitter 100 are turned on. The transmitter 100 changes in luminance by blinking the light emitting unit 142, and transmits an optical ID (optical identification information) by the luminance change. This light ID is the above-mentioned visible light signal.

The receiver 200 captures the transmitter 100 with the normal exposure time, thereby acquiring the captured display image Pa projected by the transmitter 100, and the transmitter 100 with a communication exposure time shorter than the normal exposure time. A decoding image is acquired by imaging. The normal exposure time is the exposure time in the above-described normal photographing mode, and the communication exposure time is the exposure time in the above-described visible light communication mode.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P42 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pa as a target area. Then, the receiver 200 superimposes the AR image P42 on the target area, and displays the captured display image Pa on which the AR image P42 is superimposed on the display 201.

For example, similarly to the example shown in FIG. 51, the receiver 200 recognizes the area at the upper left of the area where the transmitter 100 is projected as the target area according to the recognition information. As a result, for example, the AR image P <b> 42 showing a fairy is displayed so as to fly around the transmitter 100.

FIG. 138 is a diagram illustrating another example of the captured display image Pa on which the AR image P42 is superimposed.

As shown in FIG. 138, the receiver 200 displays a captured display image Pa on which the AR image P42 is superimposed on the display 201.

Here, the above-described recognition information indicates that a range having a luminance equal to or higher than a threshold value in the captured display image Pa is a reference region. Further, the recognition information indicates that the target area is in a predetermined direction with respect to the reference area, and that the target area is separated from the center (or center of gravity) of the reference area by a predetermined distance. Show.

Therefore, when the light emitting unit 142 of the transmitter 100 imaged by the receiver 200 fluctuates, the AR image P42 superimposed on the target area of the captured display image Pa also moves as shown in FIG. Move to sync. That is, when the light emitting unit 142 fluctuates, the image 142a of the light emitting unit 142 displayed in the captured display image Pa also fluctuates. This image 142a is a range having a luminance equal to or higher than the above-described threshold and is a reference area. That is, since the reference area moves, the receiver 200 moves the target area so that the distance between the reference area and the target area is maintained at a predetermined distance. The AR image P42 is superimposed. As a result, when the light emitting unit 142 shakes, the AR image P42 superimposed on the target area of the captured display image Pa also moves in synchronization with the movement of the light emitting unit 142. Note that the center position of the reference region may move even when the light emitting unit 142 is deformed. Therefore, even when the light emitting unit 142 is deformed, the AR image 42 may move so that the distance from the center position of the moving reference area is maintained at a predetermined distance.

In the above example, the receiver 200 recognizes the target area based on the recognition information and superimposes the AR image P42 on the target area. However, the AR image P42 may be vibrated around the target area. . That is, the receiver 200 vibrates the AR image P42 in the vertical direction, for example, according to a function indicating the change in amplitude with respect to time. The function is a trigonometric function such as a sine wave.

In addition, the receiver 200 may change the size of the AR image P42 according to the size of a range having a luminance equal to or higher than the above-described threshold. That is, the receiver 200 increases the size of the AR image P42 as the area of the bright region in the captured display image Pa increases, and conversely decreases the size of the AR image P42 as the area of the bright region decreases.

Alternatively, the receiver 200 increases the size of the AR image P42 as the average luminance in the range having the luminance equal to or higher than the above-described threshold is increased, and conversely decreases the size of the AR image P42 as the average luminance is lower. May be. Instead of the size of the AR image P42, the transparency of the AR image P42 may be changed according to the average luminance.

In the example shown in FIG. 138, any pixel in the image 142a of the light emitting unit 142 has a luminance equal to or higher than the threshold value, but any pixel may be less than the threshold value. That is, the range corresponding to the image 142a and having a luminance equal to or higher than the threshold value may be annular. Also in this case, a range having a luminance equal to or higher than the threshold is specified as the reference region, and the AR image P42 is superimposed on a target region that is separated from the center (or centroid) of the reference region by a predetermined distance.

FIG. 139 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

For example, as shown in FIG. 139, the transmitter 100 is configured as a lighting device, and transmits a light ID by changing the luminance while illuminating a figure 143 composed of, for example, three circles drawn on a wall. Since the figure 143 is illuminated by the light from the transmitter 100, the luminance changes in the same manner as the transmitter 100, and the light ID is transmitted.

The receiver 200 acquires the captured display image Pa and the decoding image in the same manner as described above by capturing the figure 143 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the graphic 143. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P43 and the recognition information corresponding to the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pa as a target area. For example, the receiver 200 recognizes an area where the graphic 143 is projected as a target area. Then, the receiver 200 superimposes the AR image P43 on the target area, and displays the captured display image Pa on which the AR image P43 is superimposed on the display 201. For example, the AR image P43 is a character face image.

Here, the figure 143 is composed of three circles as described above, but the figure 143 has few geometric features. Therefore, it is difficult to appropriately select and acquire an AR image corresponding to the graphic 143 from many images stored in the server from only the captured image obtained by imaging the graphic 143. However, in the present embodiment, the receiver 200 acquires an optical ID and acquires an AR image P43 corresponding to the optical ID from the server. Therefore, even if many images are stored in the server, the AR image P43 corresponding to the light ID can be appropriately selected and acquired as an AR image corresponding to the graphic 143 from the many images. it can.

FIG. 140 is a flowchart showing the operation of the receiver 200 in the present embodiment.

The receiver 200 in the present embodiment first acquires a plurality of AR image candidates (step S541). For example, the receiver 200 acquires a plurality of AR image candidates from the server through wireless communication (such as BTLE or Wi-Fi) different from visible light communication. Next, the receiver 200 captures an image of the subject (step S542). The receiver 200 acquires the captured display image Pa and the decoding image as described above by this imaging. However, when the subject is a photograph of the transmitter 100, since the optical ID is not transmitted from the subject, the receiver 200 does not acquire the optical ID even when decoding the decoding image. Can not.

Therefore, the receiver 200 determines whether or not the light ID has been acquired, that is, whether or not the light ID has been received from the subject (step S543).

Here, if it is determined that the optical ID is not received (No in step S543), the receiver 200 determines whether or not the AR display flag set for itself is 1 (step S544). The AR display flag is a flag indicating whether or not an AR image may be displayed only from the captured display image Pa even if the light ID is not acquired. When the AR display flag is 1, the AR display flag indicates that the AR image may be displayed only from the captured display image Pa. When the AR display flag is 0, the AR display flag is This indicates that the AR image should not be displayed only from the captured display image Pa.

If it is determined that the AR display flag is 1 (Yes in step S544), the receiver 200 selects a candidate corresponding to the captured display image Pa from the plurality of AR image candidates acquired in step S541 as an AR image. (Step S545). That is, the receiver 200 extracts a feature amount included in the captured display image Pa, and selects a candidate associated with the extracted feature amount as an AR image.

Then, the receiver 200 displays the selected candidate AR image superimposed on the captured display image Pa (step S546).

On the other hand, if it is determined that the AR display flag is 0 (No in step S544), the receiver 200 does not display the AR image.

If it is determined in step S543 that an optical ID has been received (Yes in step S543), the receiver 200 selects a candidate associated with the optical ID from among a plurality of AR image candidates acquired in step S541. It selects as AR image (step S547). Then, the receiver 200 displays the AR image that is the selected candidate so as to be superimposed on the captured display image Pa (step S546).

In the above example, the AR display flag is set in the receiver 200, but may be set in the server. In this case, the receiver 200 inquires of the server whether the AR display flag is 1 or 0 in step S544.

Thus, whether or not to display an AR image on the receiver 200 when the optical ID is not received even when the receiver 200 performs imaging can be controlled by the AR display flag.

FIG. 141 is a diagram for explaining the operation of the transmitter 100 in the present embodiment.

For example, the transmitter 100 is configured as a projector. Here, the intensity of the light emitted from the projector and reflected on the screen varies depending on factors such as the aging of the light source of the projector or the distance from the light source to the screen. When the light intensity is low, the optical ID transmitted from the transmitter 100 is difficult to be received by the receiver 200.

Therefore, the transmitter 100 according to the present embodiment adjusts a parameter for causing the light source to emit light in order to suppress a change in light intensity according to each factor. This parameter is at least one of the value of the current input to the light source for causing the light source to emit light and the light emission time (more specifically, the light emission time per unit time). For example, the light intensity of the light source increases as the current value increases or the light emission time increases.

That is, the transmitter 100 adjusts the parameter so that the light from the light source becomes stronger as the light source deteriorates with age. Specifically, the transmitter 100 includes a timer, and adjusts the parameter to increase the light of the light source as the usage time of the light source measured by the timer is longer. That is, the transmitter 100 increases the current value of the light source or lengthens the light emission time as the usage time is longer. Alternatively, the transmitter 100 detects the intensity of light emitted from the light source, and adjusts the parameters so that the detected light intensity does not decrease. That is, the transmitter 100 adjusts the parameter so as to increase the light intensity as the detected light intensity decreases.

In addition, the transmitter 100 adjusts the parameter so that the light from the light source becomes stronger as the irradiation distance from the light source to the screen becomes longer. Specifically, the transmitter 100 detects the intensity of the light that is irradiated and reflected by the screen, and adjusts the parameter so that the light from the light source increases as the detected light intensity decreases. That is, the transmitter 100 increases the current value of the light source or lengthens the light emission time as the detected light intensity decreases. Thereby, the parameters are adjusted so that the intensity of the reflected light is constant regardless of the irradiation distance. Alternatively, the transmitter 100 detects the irradiation distance from the light source to the screen by a distance measuring sensor, and adjusts the parameter so that the light from the light source is increased as the detected irradiation distance is longer.

Also, the transmitter 100 adjusts the parameter so that the lighter the light of the light source is, the darker the screen is. Specifically, the transmitter 100 detects the color of the screen by imaging the screen, and adjusts the parameter so that the light from the light source is increased as the detected color is blacker. That is, the transmitter 100 increases the current value of the light source or lengthens the light emission time as the detected color is black. Thus, the parameters are adjusted so that the intensity of the reflected light is constant regardless of the color of the screen.

In addition, the transmitter 100 adjusts the parameters so that the light from the light source increases as the external light increases. Specifically, the transmitter 100 detects the difference between the screen brightness when the light source is turned on and irradiated with light and the screen brightness when the light source is turned off and no light is irradiated. Then, the transmitter 100 adjusts the parameter so as to increase the light of the light source as the brightness difference is smaller. That is, the transmitter 100 increases the current value of the light source or lengthens the light emission time as the difference in brightness is smaller. Thus, the parameters are adjusted so that the S / N ratio of the light ID is constant regardless of the external light. Alternatively, when the transmitter 100 is configured as an LED display, for example, the intensity of sunlight may be detected, and the parameter may be adjusted to increase the light from the light source as the intensity of sunlight increases. .

It should be noted that the parameter adjustment as described above may be performed when an operation by the user is performed. For example, the transmitter 100 includes a calibration button, and performs the above-described parameter adjustment when the calibration button is pressed by the user. Alternatively, the transmitter 100 may periodically adjust the parameters described above.

FIG. 142 is a diagram for explaining another operation of the transmitter 100 in the present embodiment.

For example, the transmitter 100 is configured as a projector, and irradiates the screen with light from the light source through the front member. When the projector is a liquid crystal projector, the front member is a liquid crystal panel, and when the projector is a DLP (registered trademark) projector, the front member is a DMD (Digital Mirror Device). That is, the front member is a member that adjusts the luminance of the image for each pixel. The light source irradiates light toward the front member, and switches the intensity of the light between High and Low. In addition, the light source adjusts the time-average brightness by adjusting the High time per unit time.

Here, when the transmittance of the front member is 100%, for example, the light source is dark so that the image projected from the projector onto the screen is not too bright. That is, the light source shortens the High time per unit time.

At this time, the light source widens the pulse width of the light ID when transmitting the light ID due to a change in luminance.

On the other hand, when the transmittance of the front member is 20%, for example, the light source becomes bright so that the image projected from the projector onto the screen is not too dark. That is, the light source lengthens High time per unit time.

At this time, the light source narrows the pulse width of the light ID when transmitting the light ID due to a change in luminance.

As described above, when the light source is dark, the pulse width of the light ID is widened. Conversely, when the light source is bright, the pulse width of the light ID is narrowed. It is possible to prevent the intensity of the light from being too weak or too bright.

In the above example, the transmitter 100 is a projector, but may be configured as a large LED display. The large LED display includes a pixel switch and a common switch. An image is expressed by turning on and off the pixel switch, and an optical ID is transmitted by turning on and off the common switch. In this case, functionally, the pixel switch corresponds to the front member, and the common switch corresponds to the light source. When the average luminance by the pixel switch is high, the pulse width of the light ID by the common switch may be shortened.

FIG. 143 is a diagram for explaining another operation of the transmitter 100 in the present embodiment. Specifically, FIG. 143 shows the dimming degree of the transmitter 100 configured as a spotlight with dimming function, and the current (specifically, the peak current value) input to the light source of the transmitter 100. Show the relationship.

The transmitter 100 receives the dimming degree specified for the light source provided in the transmitter 100, and causes the light source to emit light at the specified dimming degree. The dimming degree is a ratio of the average luminance of the light source to the maximum average luminance. The average luminance is not instantaneous luminance but luminance in time average. In addition, the dimming degree is adjusted by adjusting the value of the current input to the light source or adjusting the time during which the luminance of the light source is low. The time when the luminance of the light source becomes Low may be the time when the light source is turned off.

Here, when transmitting the transmission target signal as an optical ID, the transmitter 100 generates an encoded signal by encoding the transmission target signal in a predetermined mode. Then, the transmitter 100 transmits the encoded signal as an optical ID (that is, a visible light signal) by changing the luminance of the light source provided in the transmitter 100 according to the encoded signal.

For example, when the designated dimming degree is 0% or more and x3 (%) or less, the transmitter 100 generates an encoded signal by encoding the transmission target signal in the PWM mode with a duty ratio of 35%. . x3 (%) is, for example, 50%. In the present embodiment, the PWM mode with a duty ratio of 35% is also referred to as a first mode, and the above x3 is also referred to as a first value.

That is, when the designated dimming degree is 0% or more and x3 (%) or less, the transmitter 100 sets the dimming degree of the light source to the value of the peak current while maintaining the duty ratio of the visible light signal at 35%. Adjust by.

When the designated dimming degree is greater than x3 (%) and equal to or less than 100%, the transmitter 100 encodes the encoded signal by encoding the transmission target signal in the PWM mode with a duty ratio of 65%. Generate. In the present embodiment, the PWM mode with a duty ratio of 65% is also referred to as a second mode.

That is, when the designated dimming degree is greater than x3 (%) and equal to or less than 100%, the transmitter 100 sets the dimming degree of the light source to the peak current while maintaining the duty ratio of the visible light signal at 65%. Adjust according to the value of.

Thus, the transmitter 100 according to the present embodiment accepts the dimming degree designated for the light source as the designated dimming degree. Then, when the designated dimming degree is equal to or less than the first value, the transmitter 100 transmits the signal encoded in the first mode by the luminance change while causing the light source to emit light at the designated dimming degree. In addition, when the value of the designated dimming degree is larger than the first value, the transmitter 100 transmits the signal encoded in the second mode by changing the luminance while causing the light source to emit light at the designated dimming degree. To do. Specifically, the duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode.

Here, since the duty ratio of the second mode is larger than the duty ratio of the first mode, the change rate of the peak current with respect to the dimming degree in the second mode is expressed as the change of the peak current with respect to the dimming degree in the first mode. Can be smaller than the rate.

Also, when the designated dimming degree exceeds x3 (%), the mode is switched from the first mode to the second mode. Accordingly, at this time, the peak current can be instantaneously reduced. That is, when the designated dimming degree is x3 (%), the peak current is y3 (mA). However, when the designated dimming degree slightly exceeds x3 (%), the peak current exceeds y2 (mA). Can be suppressed. In addition, y3 (mA) is 143 mA, for example, and y2 (mA) is 100 mA, for example. As a result, in order to increase the dimming degree, it is possible to suppress the peak current from becoming larger than y3 (mA), and it is possible to suppress the deterioration of the light source due to the large current flowing.

Also, when the specified dimming degree exceeds x4 (%), the peak current becomes larger than y3 (mA) even if the mode is the second mode. However, when the specified dimming rate is less than x4 (%), deterioration of the light source can be suppressed. In the present embodiment, x4 described above is also referred to as a second value. In the example shown in FIG. 143, x4 (%) is less than 100%, but may be 100%.

That is, in the transmitter 100 according to the present embodiment, when the designated dimming degree is larger than the first value and equal to or smaller than the second value, the signal encoded in the second mode is transmitted by the luminance change. When the specified dimming level is the first value, the peak current value of the light source is smaller than the peak current value of the light source for transmitting the signal encoded in the first mode by a change in luminance.

As a result, the peak current value of the light source when the designated dimming degree is larger than the first value and equal to or smaller than the second value by switching the mode for encoding the signal is the designated dimming degree being the first value. It becomes smaller than the peak current value of the light source in some cases. Therefore, as the designated dimming degree is increased, a large peak current can be suppressed from flowing to the light source. As a result, deterioration of the light source can be suppressed.

Furthermore, when the designated dimming degree is less than or equal to x1 (%) and less than x2 (%), the transmitter 100 in the present embodiment causes the light source to emit light at the designated dimming degree, while in the first mode. In addition to transmitting the signal encoded by the luminance change, the peak current value is maintained at a constant value with respect to the designated dimming change. x2 (%) is smaller than x3 (%). In the present embodiment, x2 described above is also referred to as a third value.

That is, when the designated dimming degree is smaller than x2 (%), the transmitter 100 increases the time for turning off the light source as the designated dimming degree becomes smaller, and the specified dimming degree becomes smaller. The light source is caused to emit light, and the peak current value is maintained at a constant value. Specifically, the transmitter 100 lengthens the cycle for transmitting each of the plurality of encoded signals while maintaining the duty ratio of the encoded signal at 35%. Thereby, the time to turn off the light source, that is, the extinguishing period becomes longer. As a result, the dimming degree can be reduced while keeping the peak current value constant. Even when the specified dimming level is small, the peak current value is kept constant, so that the visible light signal (that is, the light ID), which is a signal transmitted by a change in luminance, is easily received by the receiver 200. be able to.

Here, the transmitter 100 determines the time to turn off the light source so that one cycle obtained by adding the time to transmit the encoded signal due to the luminance change and the time to turn off the light source does not exceed 10 milliseconds. . For example, if the time to turn off the light source is too long and one period exceeds 10 milliseconds, the luminance change of the light source for transmitting the encoded signal may be recognized by the human eye as flickering. . Therefore, in this embodiment, since the time for turning off the light source is determined so that one period does not exceed 10 milliseconds, flicker can be prevented from being recognized by a person.

Further, even when the designated dimming degree is smaller than x1 (%), the transmitter 100 transmits the signal encoded in the first mode by the luminance change while causing the light source to emit light at the designated dimming degree. At this time, the transmitter 100 causes the light source to emit light at the specified dimming degree that becomes smaller by reducing the value of the peak current as the designated dimming degree becomes smaller. x1 (%) is smaller than x2 (%). In the present embodiment, the above x1 is also referred to as a fourth value.

Thereby, even if the designated dimming degree is smaller, the light source can be appropriately emitted with the designated dimming degree.

In the example shown in FIG. 143, the maximum peak current value in the first mode (ie, y3 (mA)) is greater than the maximum peak current value in the second mode (ie, y4 (mA)). Small but the same. That is, the transmitter 100 encodes the transmission target signal in the first mode until the designated dimming degree is x3a (%) larger than x3 (%). Then, when the designated dimming degree is x3a (%), the transmitter 100 turns on the light source with the same peak current value as the maximum peak current value in the second mode (ie, y4 (mA)). Make it emit light. In this case, x3a is the first value. Note that the maximum peak current value in the second mode is the peak current value when the specified dimming degree is the maximum value, that is, 100%.

In other words, in the present embodiment, the value of the peak current of the light source when the designated dimming degree is the first value is the same as the value of the peak current of the light source when the designated dimming degree is the maximum value. There may be. In this case, since the range of the dimming degree that causes the light source to emit light with a peak current of y3 (mA) or more is widened, it is possible to make the receiver 200 easily receive the light ID in a wide range of dimming degree. In other words, even in the first mode, since a large peak current can be passed through the light source, it is possible to make it easier for the receiver to receive a signal transmitted by a change in luminance of the light source. In this case, since the period during which a large peak current flows becomes long, the light source easily deteriorates.

FIG. 144 is a diagram showing a comparative example for explaining the ease of receiving the optical ID in the present embodiment.

In this embodiment, as shown in FIG. 143, the first mode is used when the dimming degree is small, and the second mode is used when the dimming degree is large. The first mode is a mode in which the increase in peak current is increased even if the increase in dimming degree is small, and the second mode is a mode in which the increase in peak current is suppressed even if the increase in dimming degree is large. Therefore, since the second mode suppresses a large peak current from flowing to the light source, deterioration of the light source can be suppressed. Furthermore, the first mode allows the receiver 200 to easily receive the optical ID because a large peak current flows through the light source even if the dimming degree is small.

On the other hand, when the second mode is used even when the dimming degree is small, as shown in FIG. 144, when the dimming degree is small, the value of the peak current is also small. It becomes difficult to make it receive.

Therefore, in transmitter 100 according to the present embodiment, it is possible to achieve both suppression of deterioration of the light source and ease of reception of the optical ID.

Further, when the value of the peak current of the light source exceeds the fifth value, the transmitter 100 may stop transmission of a signal due to a change in luminance of the light source. The fifth value may be, for example, y3 (mA).

Thereby, the deterioration of the light source can be further suppressed.

Also, the transmitter 100 may measure the usage time of the light source, as in the example shown in FIG. When the usage time is equal to or longer than the predetermined time, the transmitter 100 may transmit a signal by a change in luminance using a parameter value for causing the light source to emit light with a dimming degree greater than the specified dimming degree. . In this case, the parameter value may be a peak current value or a time for turning off the light source. Thereby, it is possible to prevent the optical ID from becoming difficult to be received by the receiver 200 due to deterioration of the light source over time.

Alternatively, the transmitter 100 may measure the usage time of the light source, and if the usage time is equal to or longer than the predetermined time, the transmitter 100 may increase the pulse width of the current of the light source as compared to the case where the usage time is less than the predetermined time. . Thereby, it can suppress that it becomes difficult to receive optical ID by deterioration of a light source like the above-mentioned.

In the above-described embodiment, the transmitter 100 is switched between the first mode and the second mode according to the designated dimming degree, but even if the mode is switched according to the operation by the user. Good. That is, when the switch is operated by the user, the transmitter 100 switches the first mode to the second mode, or conversely switches the second mode to the first mode. Further, the transmitter 100 may notify the user when the mode is switched. For example, the transmitter 100 may notify the user of the mode switching by sounding a sound, blinking a light source at a period that can be visually recognized by a person, or turning on a notification LED. Further, the transmitter 100 may notify the user that the relationship changes not only at the mode switching but also at the time when the relationship between the peak current and the dimming degree changes. The time point is, for example, a time point when the dimming degree shown in FIG. 143 changes from x1 (%) or a time point when the dimming degree changes from x2 (%).

FIG. 145A is a flowchart showing an operation of transmitter 100 in the present embodiment.

First, the transmitter 100 receives the dimming degree designated for the light source as the designated dimming degree (step S551). Next, the transmitter 100 transmits a signal according to a change in luminance of the light source (step S552). Specifically, when the designated dimming degree is equal to or less than the first value, the transmitter 100 causes the light source to emit light at the designated dimming degree, and changes the signal encoded in the first mode according to the luminance change. Send. In addition, when the designated dimming degree is larger than the first value, the transmitter 100 transmits the signal encoded in the second mode by the luminance change while causing the light source to emit light at the designated dimming degree. Here, when the designated dimming degree is larger than the first value and equal to or smaller than the second value, the value of the peak current of the light source for transmitting the signal encoded in the second mode by the luminance change is When the designated dimming level is the first value, it is smaller than the value of the peak current of the light source for transmitting the signal encoded in the first mode by the luminance change.

FIG. 145B is a block diagram showing a configuration of transmitter 100 in the present embodiment.

The transmitter 100 includes a reception unit 551 and a transmission unit 552. The accepting unit 551 accepts the dimming degree designated for the light source as the designated dimming degree (step S551). The transmission unit 552 transmits a signal according to a change in luminance of the light source. Specifically, when the designated dimming level is equal to or smaller than the first value, the transmission unit 552 causes the light source to emit light at the designated dimming level, and the signal encoded in the first mode is changed by the luminance change. Send. In addition, when the designated dimming degree is larger than the first value, the transmission unit 552 transmits the signal encoded in the second mode by the luminance change while causing the light source to emit light at the designated dimming degree. Here, when the designated dimming degree is larger than the first value and equal to or smaller than the second value, the value of the peak current of the light source for transmitting the signal encoded in the second mode by the luminance change is When the designated dimming level is the first value, it is smaller than the value of the peak current of the light source for transmitting the signal encoded in the first mode by the luminance change.

As a result, as shown in FIG. 143, the peak current value of the light source when the designated dimming degree is greater than the first value and equal to or less than the second value by switching the signal encoding mode is It becomes smaller than the value of the peak current of the light source when the luminous intensity is the first value. Therefore, as the designated dimming degree is increased, a large peak current can be suppressed from flowing to the light source. As a result, deterioration of the light source can be suppressed.

FIG. 146 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

The receiver 200 acquires the above-described captured display image Pk, which is the normal captured image, and the above-described decoding image, which is the visible light communication image or the bright line image, by capturing the subject with the image sensor.

Specifically, the image sensor of the receiver 200 images the transmitter 100 configured as signage and the person 21 adjacent to the transmitter 100. The transmitter 100 is the transmitter in each of the above embodiments, and includes one or a plurality of light emitting elements (for example, LEDs) and a light transmitting plate 144 having a light transmitting property like ground glass. The one or more light emitting elements emit light inside the transmitter 100, and light from the one or more light emitting elements is transmitted through the translucent plate 144 and irradiated outside. As a result, the translucent plate 144 of the transmitter 100 shines brightly. Such a transmitter 100 changes in luminance by blinking one or more light emitting elements, and transmits an optical ID (light identification information) by the change in luminance. This light ID is the above-mentioned visible light signal.

Here, a message “Please hold your smartphone here” is written on the translucent plate 144. Therefore, the user of the receiver 200 places the person 21 next to the transmitter 100 and instructs the person 21 to put his arm on the transmitter 100. Then, the user images the camera (that is, the image sensor) of the receiver 200 toward the person 21 and the transmitter 100. The receiver 200 captures the transmitter 100 and the person 21 with the normal exposure time, thereby obtaining a captured display image Pk on which they are projected. Furthermore, the receiver 200 acquires a decoding image by capturing the transmitter 100 and the person 21 with a communication exposure time shorter than the normal exposure time.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P44 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pk as a target area. For example, the receiver 200 recognizes an area in which signage as the transmitter 100 is displayed as a target area.

Then, the receiver 200 superimposes the AR image P44 on the captured display image Pk so that the target area is covered with the AR image P44, and displays the captured display image Pk on the display 201. For example, the receiver 200 acquires an AR image P44 indicating a soccer player. In this case, since the AR image P44 is superimposed so as to cover the target area of the captured display image Pk, the captured display image Pk can be displayed so that a soccer player actually exists next to the person 21. . As a result, even if the soccer player is not next to the person 21, the person 21 can be photographed together with the soccer player. More specifically, in the state where the arm of the person 21 is put on the shoulder of the soccer player, the photograph can be taken together with the soccer player.

(Embodiment 8)
In this embodiment, a transmission method for transmitting an optical ID by a visible light signal will be described. Note that the transmitter and the receiver in this embodiment may have the same functions and configurations as the transmitter (or the transmission device) and the receiver (or the reception device) in each of the above embodiments.

FIG. 147 is a diagram for explaining the operation of the transmitter 100 in the present embodiment. Specifically, FIG. 147 shows the dimming degree of the transmitter 100 configured as a spotlight with dimming function, and the current (specifically, the peak current value) input to the light source of the transmitter 100. Show the relationship.

When the designated dimming degree is 0% or more and x14 (%) or less, the transmitter 100 according to the present embodiment encodes a transmission target signal in a PWM mode with a duty ratio of 35%. Is generated. In other words, when the designated dimming level changes from 0% to x14 (%), the transmitter 100 increases the peak current value while maintaining the duty ratio of the visible light signal at 35%. The light source is caused to emit light at the specified dimming degree. Note that the PWM mode with a duty ratio of 35% is also referred to as a first mode, as in the seventh embodiment, and the above x14 is also referred to as a first value. For example, x14 (%) is a value within the range of 50 to 60%.

Further, when the designated dimming degree is not less than x13 (%) and not more than 100%, the transmitter 100 generates an encoded signal by encoding the transmission target signal in the PWM mode with a duty ratio of 65%. . That is, when the specified dimming level changes from 100% to x13 (%), the transmitter 100 suppresses the peak current value while maintaining the duty ratio of the visible light signal at 65%. The light source is caused to emit light at the specified dimming degree. Note that the PWM mode with a duty ratio of 65% is also referred to as a second mode, as in the seventh embodiment, and the above x13 is also referred to as a second value. Here, x13 (%) is a value smaller than x14 (%), for example, a value within a range of 40 to 50%.

As described above, in this embodiment, when the specified dimming degree increases, the PWM mode is changed from the PWM mode with the duty ratio of 35% to the PWM mode with the duty ratio of 65% at the dimming degree x14 (%). Can be switched. On the other hand, when the specified dimming degree decreases, the PWM mode is a dimming degree x13 (%) smaller than the dimming degree x14 (%), and the PWM mode from the PWM mode having a duty ratio of 65% to the PWM mode having a duty ratio of 35%. Can be switched to. That is, in the present embodiment, the dimming degree at which the PWM mode is switched is different when the designated dimming degree is increased and when the designated dimming degree is reduced. Hereinafter, the dimming degree at which the PWM mode is switched is referred to as a switching point.

Therefore, in this embodiment, frequent switching of the PWM mode can be suppressed. In the example shown in FIG. 143 of the seventh embodiment, the switching point of the PWM mode is 50%, and is the same when the designated dimming degree increases and when the designated dimming degree decreases. . As a result, in the example shown in FIG. 143, when the increase / decrease of the specified dimming degree is repeated around 50%, the PWM mode is frequently switched between the PWM mode with a duty ratio of 35% and the PWM mode with a duty ratio of 65%. Can be switched. However, in this embodiment, since the switching point of the PWM mode is different between the case where the designated dimming degree increases and the designated dimming degree decreases, such frequent switching of the PWM mode is performed. Can be suppressed.

In the present embodiment, similarly to the example shown in FIG. 143 of the seventh embodiment, when the designated dimming degree is small, the PWM mode with a small duty ratio is used, and conversely, the designated dimming degree is used. When is large, a PWM mode with a large duty ratio is used.

Therefore, when the specified dimming degree is large, the PWM mode with a large duty ratio is used, so the rate of change of the peak current with respect to the dimming degree can be reduced, and the light source emits light with a large dimming degree with the small peak current Can be made. For example, in the PWM mode with a duty ratio as small as 35%, unless the peak current is 250 mA, the light source cannot emit light with a dimming degree of 100%. However, in the present embodiment, for a large dimming degree, a PWM mode with a large duty ratio such as a duty ratio of 65% is used. Therefore, for example, the light source is set to 100 by simply setting the peak current to a smaller 154 mA. % Light can be emitted. That is, it is possible to suppress the overcurrent from flowing through the light source and shorten the life of the light source.

Also, when the specified dimming degree is small, the PWM mode with a small duty ratio is used, so that the rate of change of the peak current with respect to the dimming degree can be increased. As a result, a visible light signal can be transmitted with a large peak current while causing the light source to emit light with a small dimming degree. The light source emits light brighter as the input current increases. Therefore, when a visible light signal is transmitted with a large peak current, the visible light signal can be easily received by the receiver 200. In other words, the range of the dimming level in which the visible light signal receivable to the receiver 200 can be transmitted can be expanded to a smaller dimming level. For example, as shown in FIG. 195, if the peak current is equal to or greater than Ia (mA), the receiver 200 can receive a visible light signal transmitted by the peak current. In this case, in the PWM mode with a large duty ratio such as a duty ratio of 65%, the range of dimming degree capable of transmitting a receivable visible light signal is x12 (%) or more. However, in the PWM mode with a small duty ratio such as a duty ratio of 35%, the range of the dimming degree capable of transmitting a receivable visible light signal is set to x11 (%) or more smaller than x12 (%). Can do.

Thus, by switching the PWM mode, the life of the light source can be extended and a visible light signal can be transmitted in a wide dimming range.

FIG. 148A is a flowchart showing a transmission method according to the present embodiment.

The transmission method in the present embodiment is a transmission method for transmitting a signal by a change in luminance of a light source, and includes a reception step S561 and a transmission step S562. In the reception step S561, the transmitter 100 receives the dimming degree designated for the light source as the designated dimming degree. In the transmission step S562, the transmitter 100 transmits the signal encoded in the first mode or the second mode by the luminance change while causing the light source to emit light at the designated dimming degree. Here, the duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode. Further, in the transmission step S562, when the designated dimming level is changed from a small value to a large value, the transmitter 100 changes the mode used for signal encoding when the designated dimming level is the first value. Switch from the first mode to the second mode. Furthermore, when the designated dimming level is changed from a large value to a small value, the transmitter 100 changes the mode used for signal encoding from the second mode when the designated dimming level is the second value. Switch to the first mode. Here, the second value is smaller than the first value.

For example, the first mode and the second mode are the PWM mode with a duty ratio of 35% and the PWM mode with a duty ratio of 65% shown in FIG. 147, respectively. Further, the first value and the second value are x14 (%) and x15 (%) shown in FIG. 147, respectively.

Thereby, the designated dimming degree (that is, the switching point) at which switching between the first mode and the second mode is performed differs depending on whether the designated dimming degree increases or decreases. Therefore, frequent switching between these modes can be suppressed. That is, the occurrence of so-called chattering can be suppressed. As a result, the operation of the transmitter 100 that transmits a signal can be stabilized. Further, the duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode. Therefore, similarly to the transmission method shown in FIG. 143, the larger the specified dimming degree, the more the large peak current can be suppressed from flowing to the light source. As a result, deterioration of the light source can be suppressed. Further, since deterioration of the light source can be suppressed, communication between various devices can be performed for a long time. Further, when the designated dimming degree is small, the first mode with a small duty ratio is used. Therefore, the above-described peak current can be increased, and a signal that is easily received by the receiver 200 can be transmitted as a visible light signal.

In addition, in the transmission step S562, when switching from the first mode to the second mode is performed, the transmitter 100 determines the peak current of the light source for transmitting the encoded signal by the luminance change, The current value of 1 is changed to a second current value smaller than the first current value. Furthermore, when switching from the second mode to the first mode is performed, the transmitter 100 changes the peak current from the third current value to a fourth current value that is larger than the third current value. change. Here, the first current value is larger than the fourth current value, and the second current value is larger than the third current value.

For example, the first current value, the second current value, the third current value, and the fourth current value are the current value Ie, the current value Ic, the current value Ib, and the current value Id shown in FIG. 147, respectively. is there.

This makes it possible to appropriately switch between the first mode and the second mode.

FIG. 148B is a block diagram showing a configuration of transmitter 100 in the present embodiment.

The transmitter 100 in the present embodiment is a transmitter that transmits a signal according to a change in luminance of a light source, and includes a reception unit 561 and a transmission unit 562. The accepting unit 561 accepts the dimming degree designated for the light source as the designated dimming degree. The transmission unit 562 transmits the signal encoded in the first mode or the second mode by the luminance change while causing the light source to emit light at the designated dimming degree. Here, the duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode. Further, when the designated dimming level is changed from a small value to a large value, the transmission unit 562 changes the mode used for signal encoding from the first mode when the designated dimming level is the first value. Switch to the second mode. Furthermore, when the designated dimming level is changed from a large value to a small value, the transmission unit 562 changes the mode used for signal encoding from the second mode when the designated dimming level is the second value. Switch to the first mode. Here, the second value is smaller than the first value.

Such a transmitter 100 implements the transmission method of the flowchart shown in FIG. 148A.

FIG. 149 is a diagram illustrating an example of a detailed configuration of a visible light signal in the present embodiment.

Such a visible light signal is a PWM mode signal.

The visible light signal packet is composed of an L data part, a preamble, and an R data part. Note that each of the L data portion and the R data portion corresponds to a payload.

The preamble alternately indicates High and Low luminance values along the time axis. In other words, the preamble indicates a high luminance value only for the time length C ₀ , indicates a low luminance value for the next time length C ₁ , indicates a high luminance value for the next time length C ₂ , and displays the next time length C _2. Only ₃ indicates a low luminance value. The time lengths C ₀ and C ₃ are, for example, 100 μs. Further, the time lengths C ₁ and C ₂ are 90 μs shorter than the time lengths C ₁ and C ₃ by 10 μs, for example.

The L data portion alternately indicates High and Low luminance values along the time axis, and is arranged immediately before the preamble. That is, the L data portion indicates a high luminance value for the time length D ′ ₀ , indicates a low luminance value for the next time length D ′ ₁ , indicates a high luminance value for the next time length D ′ ₂ , The next time length D ′ ₃ indicates a low luminance value. Note that the time lengths D ′ ₀ to D ′ ₃ are determined according to a mathematical formula corresponding to the signal to be transmitted. This equation is expressed as D ′ ₀ = W ₀ + W ₁ × (3-y ₀ ), D ′ ₁ = W ₀ + W ₁ × (7−y ₁ ), D ′ ₂ = W ₀ + W ₁ × (3-y ₂₎ ), And D ′ ₃ = W ₀ + W ₁ × (7−y ₃ ). Here, the constant W ₀ is, for example, 110 μs, and the constant W ₁ is, for example, 30 μs. The variables y ₀ and y ₂ are any integer from 0 to 3 represented by 2 bits, and the variables y ₁ and y ₃ are any integer from 0 to 7 represented by 3 bits. Variables y ₀ to y ₃ are signals to be transmitted. In FIGS. 149 to 152, “*” is used as a symbol meaning multiplication.

The R data part is arranged immediately after the preamble, and the R data part alternately indicates High and Low luminance values along the time axis. That is, the R data portion indicates a high luminance value for the time length D ₀ , indicates a low luminance value for the next time length D ₁ , indicates a high luminance value for the next time length D ₂ , and displays the next time length. the length _{D 3} shows the luminance value of Low. Note that the time lengths D ₀ to D ₃ are determined according to a mathematical expression corresponding to the signal to be transmitted. This formula is D ₀ = W ₀ + W ₁ × y ₀ , D ₁ = W ₀ + W ₁ × y ₁ , D ₂ = W ₀ + W ₁ × y ₂ , and D ₃ = W ₀ + W ₁ × y ₃ .

Here, the L data portion and the R data portion have a complementary relationship with respect to brightness. That is, if the brightness of the L data portion is bright, the brightness of the R data portion is dark. Conversely, if the brightness of the L data portion is dark, the brightness of the R data portion is bright. That is, the sum of the time length of the L data portion and the time length of the R data portion is constant regardless of the signal to be transmitted. In other words, the brightness of the time average of the visible light signal transmitted from the light source can be made constant regardless of the signal to be transmitted.

Further, D ′ ₀ = W ₀ + W ₁ × (3-y ₀ ), D ′ ₁ = W ₀ + W ₁ × (7−y ₁ ), D ′ ₂ = W ₀ + W ₁ × (3-y ₂ ), And by changing the ratio of 3 and 7 in D ′ ₃ = W ₀ + W ₁ × (7−y ₃ ), the duty ratio of the PWM mode can be changed. The ratio between 3 and 7 corresponds to the ratio between the maximum values of the variables y ₀ and y _{2 and} the maximum values of the variables y ₁ and y ₃ . For example, when the ratio is 3: 7, a PWM mode with a small duty ratio is selected. Conversely, when the ratio is 7: 3, a PWM mode with a large duty ratio is selected. Therefore, by adjusting the ratio, the PWM mode can be switched between the PWM mode with a duty ratio of 35% and the PWM mode with a duty ratio of 65% shown in FIGS. 143 and 147. In addition, a preamble may be used to notify the receiver 200 which PWM mode is being switched to. For example, the transmitter 100 notifies the receiver 200 of the switched PWM mode by including in the packet the preamble of the pattern associated with the switched PWM mode. The preamble pattern is changed according to the time lengths C ₀ , C ₁ , C ₂ and C ₃ .

However, in the visible light signal shown in FIG. 149, since the packet includes two data parts, it takes time to transmit the packet. For example, when the transmitter 100 is a DLP projector, the transmitter 100 projects each video image of red, green, and blue in a time division manner. Here, the transmitter 100 desirably transmits a visible light signal when projecting a red video. This is because the visible light signal transmitted at this time has a red wavelength and thus is easily received by the receiver 200. The period during which the red video is continuously projected is, for example, 1.5 ms. This period is hereinafter referred to as a red projection period. In such a short red projection period, it is difficult to transmit a packet including the L data portion, the preamble, and the R data portion described above.

Therefore, a packet having only the R data portion of the two data portions is recalled.

FIG. 150 is a diagram showing another example of the detailed configuration of the visible light signal in the present embodiment.

150. The visible light signal packet shown in FIG. 150 does not include the L data portion, unlike the example shown in FIG. Instead, the visible light signal packet shown in FIG. 150 includes invalid data and an average luminance adjustment unit.

The invalid data alternately indicates High and Low luminance values along the time axis. In other words, the invalid data indicates a high luminance value for the time length A ₀ and indicates a low luminance value for the next time length A ₁ . The time length A ₀ is, for example, 100 μs, and the time length A ₁ is represented by, for example, A ₁ = W ₀ −W ₁ . Such invalid data indicates that the L data portion is not included in the packet.

The average luminance adjustment unit alternately indicates High and Low luminance values along the time axis. In other words, the invalid data indicates a high luminance value for the time length B ₀ and indicates a low luminance value for the next time length B ₁ . The time length B ₀ is represented by, for example, B ₀ = 100 + W ₁ × ((3-y ₀ ) + (3-y ₂ )), and the time length B ₁ is, for example, B ₁ = W ₁ × ((7−y ₁ ) + (7−y ₃ )).

By such an average luminance adjusting unit, the average luminance in the packet can be made constant regardless of the signals y ₀ to y ₃ to be transmitted. That is, the sum of the lengths of time when the luminance value is High in the packet (that is, the total ON time) can be set to A ₀ + C ₀ + C ₂ + D ₀ + D ₂ + B ₀ = 790. Furthermore, the sum of the time lengths when the luminance value is low in the packet (that is, the total OFF time) can be set to A ₁ + C ₁ + C ₃ + D ₁ + D ₃ + B ₁ = 910.

However, even in such a configuration of the visible light signal, it is impossible to shorten the effective length of time E ₁ which is a part of the time length of the total duration E ₀ in the packet. The effective duration E _1, first from appeared luminance values of High, a time until the brightness of the last occurrence High is completed, the receiver 200 demodulates or decodes the packets of the visible light signal in packet Is the time required to do. Specifically, the effective time length E ₁ is E ₁ = A ₀ + A ₁ + C ₀ + C ₁ + C ₂ + C ₃ + D ₀ + D ₁ + D ₂ + D ₃ + B ₀ . Note that the total time length E ₀ is E ₀ = E ₁ + B ₁ .

In other words, the effective duration E _1, even visible light signal configuration shown in FIG. 150, for a maximum 1700Myuesu, the transmitter 100, the red projection period described above, continues for its effective time length E ₁ It is difficult to transmit one packet.

Therefore, in order to shorten the effective time length E ₁ and make the average brightness of the packet constant regardless of the signal to be transmitted, not only the time lengths of the brightness values of High and Low but also the brightness value of High It is also recalled to adjust.

FIG. 151 is a diagram showing another example of a detailed configuration of a visible light signal in the present embodiment.

Packet visible light signal shown in FIG. 151, unlike the example shown in FIG. 150, in order to shorten the effective length of time E _1, the time length B ₀ of the luminance values of High average luminance adjusting unit transmitted signals Regardless, it is fixed at the shortest 100 μs. Instead, in the visible light signal packet shown in FIG. 151, the High luminance value depends on the variables y ₀ and y ₂ included in the transmission target signal, that is, according to the time lengths D ₀ and D _2. Adjusted. For example, when the time lengths D ₀ and D ₂ are short, the transmitter 100 adjusts the High luminance value to a large value as shown in FIG. When the time lengths D ₀ and D ₂ are long, the transmitter 100 adjusts the High luminance value to a small value as shown in FIG. 151 (b). Specifically, when the time lengths D ₀ and D ₂ are the shortest W ₀ (for example, 110 μs), the High luminance value is 100% brightness. When the time lengths D ₀ and D ₂ are the maximum “W ₀ + 3W ₁ ” (for example, 200 μs), the high luminance value is 77.2% brightness.

In such a visible light signal packet, the sum of the time lengths of the brightness value High (that is, the total ON time) can be set to, for example, A ₀ + C ₀ + C ₂ + D ₀ + D ₂ + B ₀ = 610 to 790. . Furthermore, the sum of the time lengths when the luminance value is Low (that is, the total OFF time) can be set to A ₁ + C ₁ + C ₃ + D ₁ + D ₃ + B ₁ = 910.

However, in the visible light signal shown in FIG. 151, the shortest time length of each of the total time length E ₀ and the effective time length E _{1 in} the packet can be made shorter than the example shown in FIG. The length cannot be shortened.

Therefore, to shorten the effective length of time E _1, and, in order to make constant regardless of the average brightness of the packet signal to be transmitted in response to a signal to be transmitted, and L data portion as the data portion included in the packet It is recalled that it can be understood using the R data part.

FIG. 152 is a diagram illustrating another example of the detailed configuration of the visible light signal according to the present embodiment.

In the visible light signal shown in FIG. 152, unlike the examples shown in FIGS. 149 to 151, in order to shorten the effective time length, the L data portion is selected in accordance with the sum of the variables y ₀ to y ₃ that are signals to be transmitted. And a packet including the R data portion are selectively used.

That is, when the sum of the variables y ₀ to y ₃ is 7 or more, the transmitter 100 generates a packet including only the L data portion of the two data portions, as shown in FIG. 152 (a). . Hereinafter, this packet is referred to as an L packet. In addition, when the sum of the variables y ₀ to y ₃ is 6 or less, the transmitter 100 generates a packet including only the R data portion of the two data portions as shown in FIG. 152 (b). . Hereinafter, this packet is referred to as an R packet.

The L packet includes an average luminance adjustment unit, an L data unit, a preamble, and invalid data, as shown in FIG. 152 (a).

The average luminance adjusting unit of the L packet indicates the low luminance value for the time length B ′ ₀ without indicating the high luminance value. The time length B ′ ₀ is represented by, for example, B ′ ₀ = 100 + W ₁ × (y ₀ + y ₁ + y ₂ + y ₃ −7).

The invalid data of the L packet alternately indicates high and low luminance values along the time axis. That is, the invalid data indicates a high luminance value for the time length A ′ ₀ and indicates a low luminance value for the next time length A ′ ₁ . The time length A ′ ₀ is indicated by A ′ ₀ = W ₀ −W ₁ and is, for example, 80 μs, and the time length A ′ ₁ is, for example, 150 μs. Such invalid data indicates that the R data portion is not included in the packet having the invalid data.

In such an L packet, the total time length E ′ ₀ is E ′ ₀ = 5W ₀ + 12W ₁ + 4b + 230 = 1540 μs regardless of the signal to be transmitted. The effective time length E ′ ₁ is a time length corresponding to a signal to be transmitted and is in the range of 900 to 1290 μs. In addition, while the total time length E ′ ₀ is a constant 1540 μs, the sum of the time lengths where the luminance value is High (that is, the total ON time) varies in the range of 490 to 670 μs depending on the signal to be transmitted. To do. Therefore, the transmitter 100 also sets the high luminance value to 100% to 73 in the L packet according to the total ON time, that is, according to the time lengths D ₀ and D ₂ , as in the example shown in FIG. Change in the range of 1%.

As in the example shown in FIG. 150, the R packet includes invalid data, a preamble, an R data portion, and an average luminance adjustment portion, as shown in FIG. 152 (b).

Here, the R packet shown in (b) in FIG. 152, in order to shorten the effective length of time E _1, the time length B ₀ of the luminance values of High in the average luminance adjusting unit, shortest regardless signal to be transmitted Of 100 μs. Further, the time length B ₁ of the Low luminance value in the average luminance adjustment unit is set to, for example, B ₁ = W ₁ × (6− (y ₀ + y ₁ + y ₂ + y ₃ ) in order to make the total time length E ₁ constant. Further, also in the R packet shown in (b) of Fig. 152, High according to the variables y ₀ and y ₂ included in the transmission target signal, that is, according to the time lengths D ₀ and D ₂ , The brightness value of is adjusted.

In such an R packet, the total time length E ₀ is E ₀ = 4W ₀ + 6W ₁ + 4b + 260 = 1280 μs regardless of the signal to be transmitted. The effective time length E ₁ is a time length corresponding to the signal to be transmitted, and is in the range of 1100 to 1280 μs. In addition, while the total time length E ₀ is a constant 1280 μs, the total time length (ie, the total ON time) when the luminance value is High varies within the range of 610 to 790 μs depending on the signal to be transmitted. . Accordingly, the transmitter 100, also in this L packets, similarly to the example shown in FIG. 151, in accordance with the total ON time, i.e. according to the time length _{D 0} and _{D 2,} the luminance value of the High 80.3% Change in the range of ~ 62.1%.

Thus, in the visible light signal shown in FIG. 152, the maximum value of the effective time length in the packet can be shortened. Therefore, the transmitter 100 can continuously transmit one packet during the above-described red projection period by the effective time length E ₁ or E ′ ₁ .

Here, in the example shown in FIG. 152, the transmitter 100 generates an L packet when the sum of the variables y ₀ to y ₃ is 7 or more, and when the sum of the variables y ₀ to y ₃ is 6 or less. , R packets are generated. In other words, since the sum of the variables y ₀ to y ₃ is an integer, the transmitter 100 generates an L packet when the sum of the variables y ₀ to y ₃ is greater than 6, and the variables y ₀ to y ₃ R packet is generated when the sum of is less than or equal to 6. That is, in this example, the threshold value for switching the packet type is 6. However, the threshold for switching the packet type is not limited to 6, and may be any value from 3 to 10.

FIG. 153 is a diagram showing the relationship between the sum of the variables y ₀ to y ₃ and the total time length and effective time length. The total time length shown in FIG. 153 is the larger time length of the total time length E ₀ of the R packet and the total time length E ′ ₀ of the L packet. The effective length of time shown in FIG. 153 is a time length of larger of the maximum value of the effective duration E ₁ of R packets, the maximum value of the effective time length E _'1 of L packets. In the example shown in FIG. 153, the constants W ₀ , W ₁ , and b are W ₀ = 110 μs, W ₁ = 15 μs, and b = 100 μs, respectively.

As shown in FIG. 153, the total time length changes according to the sum of the variables y ₀ to y ₃ , but the sum is about 10 and becomes the minimum. Further, as shown in FIG. 153, the effective time length changes according to the sum of the variables y ₀ to y ₃ , but the sum is about 3 and becomes the minimum.

Therefore, the packet type switching threshold may be set in the range of 3 to 10 depending on which of the total time length and the effective time length is to be shortened.

FIG. 154A is a flowchart showing the transmission method in the present embodiment.

The transmission method in the present embodiment is a transmission method for transmitting a visible light signal by a change in luminance of a light emitter, and includes a determination step S571 and a transmission step S572. In the determination step S571, the transmitter 100 modulates the signal to determine a luminance change pattern. In the transmission step S572, the transmitter 100 transmits a visible light signal by changing the luminance of red expressed by the light source included in the light emitter according to the determined pattern. Here, the visible light signal includes data, a preamble, and a payload. In the data, a first luminance value and a second luminance value smaller than the first luminance value appear along the time axis, and at least one of the first luminance value and the second luminance value is displayed. The length of time during which one continues is less than or equal to the first predetermined value. In the preamble, each of the first and second luminance values appears alternately along the time axis. In the payload, the first and second luminance values appear alternately along the time axis, the length of time that each of the first and second luminance values continues is greater than the first predetermined value, and It is determined according to the above signal and a predetermined method.

For example, data, preamble, and payload are invalid data, preamble, and L data portion or R data portion shown in FIGS. 152 (a) and (b), respectively. For example, the first predetermined value is 100 μs.

As a result, as shown in FIGS. 152A and 152B, the visible light signal has one waveform payload (ie, L data portion or R data portion) determined according to the signal to be modulated. Contains no two payloads. Therefore, the visible light signal, that is, the packet of the visible light signal can be shortened. As a result, for example, even if the light emission period of red light expressed by the light source included in the light emitter is short, a packet of visible light signals can be transmitted during the light emission period.

In the payload, the first luminance value of the first time length, the second luminance value of the second time length, the first luminance value of the third time length, the second luminance value of the fourth time length Each luminance value may appear in the order of the luminance value. In this case, in the transmission step S572, the transmitter 100 determines that the first time length and the third time length when the sum of the first time length and the third time length is smaller than the second predetermined value. Is larger than the second predetermined value, the value of the current flowing in the light source is increased. Here, the second predetermined value is larger than the first predetermined value. Note that the second predetermined value is, for example, a value larger than 220 μs.

Thereby, as shown in FIGS. 151 and 152, when the sum of the first time length and the third time length is small, the current value of the light source is increased, and the first time length and the third time length are increased. When the sum of the lengths is large, the current value of the light source is reduced. Therefore, the average luminance of the packet including data, preamble, and payload can be kept constant regardless of the signal.

In the payload, the first luminance value of the first time length D ₀ , the second luminance value of the second time length D ₁ , the first luminance value of the third time length D ₂ , the fourth in order of the second luminance value of the time length D _3, it may appear the respective luminance values. In this case, when the sum of the four parameters y _k (k = 0, 1, 2, 3) obtained from the signal is equal to or smaller than the third predetermined value, the first to fourth time lengths D ₀ to D ₃ Each is determined according to D _k = W ₀ + W ₁ × y _k (W ₀ , W ₁ is an integer of 0 or more). For example, as shown in FIG. 152 (b), the third predetermined value is 3.

As a result, as shown in FIG. 152 (b), it is possible to generate a payload having a short waveform according to the signal while setting each of the first to fourth time lengths D ₀ to D ₃ to be W ₀ or more.

When the sum of the four parameters y _k (k = 0, 1, 2, 3) is equal to or smaller than the third predetermined value, in the transmission step S572, the data, the preamble, and the payload are replaced with the data, the preamble, and the payload. You may transmit in order. In the case of the example shown in FIG. 152 (b), the payload is the R data portion.

Thereby, as shown in FIG. 152 (b), the fact that the packet of the visible light signal including the data (that is, invalid data) does not include the L data portion indicates that the receiver 200 receives the packet by the data. Can let you know.

When the sum of the four parameters y _k (k = 0, 1, 2, 3) is larger than the third predetermined value, each of the first to fourth time lengths D ₀ to D ₃ is D _0. = W ₀ + W ₁ × (A−y ₀ ), D ₁ = W ₀ + W ₁ × (By ₁ ), D ₂ = W ₀ + W ₁ × (A−y ₂ ), and D ₃ = W ₀ + W _It may be determined according to ₁ × (By ₃ ) (A and B are each integers of 0 or more).

Thereby, as shown in FIG. 152A, each of the first to fourth time lengths D ₀ to D ₃ (that is, the first to fourth time lengths D ′ ₀ to D ′ ₃ ) is set to W ₀ or more. However, even if the total sum is large, a short waveform payload can be generated according to the signal.

If the sum of the four parameters y _k (k = 0, 1, 2, 3) is larger than the third predetermined value, in step S572, the data, preamble, and payload are replaced with the payload, preamble, and data. You may transmit in order. In the case of the example shown in FIG. 152 (a), the payload is the L data portion.

As a result, as shown in FIG. 152 (a), the fact that the packet of visible light signal including data (that is, invalid data) does not include the R data portion is indicated to the receiving device that receives the packet by the data. I can inform you.

The light emitter has a plurality of light sources including a red light source, a blue light source, and a green light source. In the transmission step S572, a visible light signal is generated using only the red light source among the plurality of light sources. You may send it.

Thereby, the light emitter can display an image using a red light source, a blue light source, and a green light source, and can transmit a visible light signal having a wavelength that is easy to receive to the receiver 200.

The light emitter may be a DLP projector, for example. As described above, the DLP projector may include a plurality of light sources including a red light source, a blue light source, and a green light source, but may include only one light source. That is, the DLP projector may include one light source, a DMD (Digital Micromirror Device), and a color wheel disposed between the light source and the DMD. In this case, the DLP projector is visible during a period in which red light is output among red light, blue light, and green light that are output in time division from the light source to the DMD via the color wheel. Transmit an optical signal packet.

FIG. 154B is a block diagram showing a configuration of transmitter 100 in the present embodiment.

The transmitter 100 according to the present embodiment includes a transmitter that transmits a visible light signal according to a change in luminance of a light emitter, a determination unit 571, and a transmission unit 572. The determination unit 571 determines a luminance change pattern by modulating a signal. The transmission unit 572 transmits the visible light signal by changing the luminance of red expressed by the light source included in the light emitter according to the determined pattern. Here, the visible light signal includes data, a preamble, and a payload. In the data, a first luminance value and a second luminance value smaller than the first luminance value appear along the time axis, and at least one of the first luminance value and the second luminance value is displayed. The length of time during which one continues is less than or equal to the first predetermined value. In the preamble, each of the first and second luminance values appears alternately along the time axis. In the payload, the first and second luminance values appear alternately along the time axis, the length of time that each of the first and second luminance values continues is greater than the first predetermined value, and It is determined according to the above signal and a predetermined method.

Such a transmitter 100 realizes the transmission method of the flowchart shown in FIG. 154A.

(Embodiment 9)
In this embodiment, a display method and a display device that realize AR (Augmented Reality) using an optical ID will be described as in the fourth embodiment. Note that the transmitter and the receiver in this embodiment may have the same functions and configurations as the transmitter (or the transmission device) and the receiver (or the reception device) in each of the above embodiments. The receiver in this embodiment is configured as a display device.

FIG. 155 is a diagram illustrating a configuration of the display system according to the ninth embodiment.

This display system 500 performs object recognition and augmented reality (Augmented Reality / Mixed Reality) display using visible light signals.

For example, as illustrated in FIG. 155, the transmitter 100 is configured as a lighting device, and transmits a light ID by changing the luminance while illuminating the AR object 501. Since the AR object 501 is illuminated by the light from the transmitter 100, the luminance changes similarly to the transmitter 100, and the optical ID is transmitted.

The receiver 200 images the AR object 501. That is, the receiver 200 images the AR object 501 at the exposure times of the normal exposure time and the communication exposure time described above. Thereby, the receiver 200 acquires a captured display image and a decoding image that is a visible light communication image or a bright line image, as described above.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the AR object 501. The receiver 200 transmits the optical ID to the server 300. Then, the receiver 200 acquires the AR image P11 and the recognition information associated with the optical ID from the server 300. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image as a target area. For example, the receiver 200 recognizes the area where the AR object 501 is projected as the target area. Then, the receiver 200 superimposes the AR image P11 on the target area, and displays the captured display image on which the AR image P11 is superimposed on the display. For example, the AR image P11 is a moving image.

When the display or reproduction of all the moving images of the AR image P11 is completed, the receiver 200 notifies the server 300 of the completion of the reproduction of the moving images. The server 300 that has received the notification of the completion of reproduction gives a reward such as points to the receiver 200. When the receiver 200 notifies the server 300 of the completion of reproduction of the moving image, the receiver 200 may notify not only the completion of reproduction but also the personal information of the user of the receiver 200, and the wallet ID for storing the reward You may be notified. By receiving these notifications, the server 300 gives points to the receiver 200.

FIG. 156 is a sequence diagram showing processing operations of the receiver 200 and the server 300.

The receiver 200 acquires the light ID as a visible light signal by imaging the AR object 501 (step S51). Then, the receiver 200 transmits the optical ID to the server 300 (step S52).

When the server 300 acquires the optical ID (step S53), the server 300 transmits the recognition information associated with the optical ID and the AR image P11 to the receiver 200 (step S54).

In accordance with the recognition information, the receiver 200 recognizes, for example, an area in which the AR target 501 is projected in the captured display image as the target area, and displays the captured display image in which the AR image P11 is superimposed on the target area. To display. Then, the receiver 200 starts to reproduce the moving image that is the AR image P11 (step S56).

Next, the receiver 200 determines whether or not the reproduction of the moving image has been completed (step S57). When it is determined that all the reproduction has been completed (Yes in step S57), the receiver 200 notifies the server 300 of the completion of reproduction of the moving image (step S58).

When the server 300 receives the reproduction completion notification from the receiver 200, the server 300 gives points to the receiver 200 (step S59).

Here, as shown in FIG. 157, the server 300 may make conditions for giving points to the receiver 200 more stringent.

FIG. 157 is a flowchart showing the processing operation of the server 300.

The server 300 first acquires an optical ID from the receiver 200 (step S60). Next, the server 300 transmits the recognition information associated with the optical ID and the AR image P11 to the receiver 200 (step S61).

Then, the server 300 determines whether or not a notification of completion of reproduction of the moving image that is the AR image P11 has been received from the receiver 200 (step S62). Here, if the server 300 determines that the notification of the completion of the reproduction of the moving image has been received (Yes in step S62), it further determines whether or not the same AR image P11 has been reproduced in the receiver 200 in the past (step S63). ). If it is determined that the same AR image P11 has not been reproduced in the receiver 200 in the past (No in step S63), the server 300 gives points to the receiver 200 (step S66). On the other hand, when determining that the same AR image P11 has been reproduced in the receiver 200 in the past (Yes in step S63), the server 300 further determines whether or not a predetermined period has elapsed since the past reproduction. (Step S64). For example, the predetermined period may be one month, three months, one year, or any period.

Here, the server 300 determines that the predetermined period has not elapsed (No in step S64), and does not give points to the receiver 200. On the other hand, when the server 300 determines that the predetermined period has elapsed (Yes in step S64), the server 200 further determines that the current location of the receiver 200 is the location where the same AR image P11 has been reproduced in the past (hereinafter referred to as past playback). It is determined whether or not the location is different (step S65). When the server 300 determines that the current location of the receiver 200 is different from the past playback location (Yes in step S65), the server 300 gives points to the receiver 200 (step S66). On the other hand, if the server 300 determines that the current location of the receiver 200 is the same as the previous playback location (No in step S65), the server 300 does not give points to the receiver 200.

Thereby, since points are given to the receiver 200 by all reproduction of the AR image P11, the user of the receiver 200 can increase the willingness to reproduce all of the AR image P11. For example, in order to acquire the AR image P11 having a large amount of data from the server 300, a high packet fee is required, and the user may interrupt the reproduction of the AR image P11 on the way. However, it is possible to reproduce the entire AR image P11 by giving points. The points may be discounts on packet charges. Furthermore, points corresponding to the data amount of the AR image P11 may be given to the receiver 200.

FIG. 158 is a diagram illustrating an example of communication when the transmitter 100 and the receiver 200 are each mounted on a vehicle.

The vehicle 200n includes the receiver 200 described above, and the plurality of vehicles 100n include the transmitter 100 described above. In addition, the plurality of vehicles 100n are traveling in front of the vehicle 200n, for example. Furthermore, the vehicle 200n is wirelessly communicating with any one of the plurality of vehicles 100n.

Here, the vehicle 200n transmits a visible light signal to the communication partner vehicle 100n in order to grasp which of the plurality of vehicles 100n ahead is communicating with the vehicle 100n. Request by wireless communication.

When the communication partner vehicle 100n receives the request from the vehicle 200n, the vehicle 100n transmits a visible light signal backward. For example, the communication partner vehicle 100n transmits a visible light signal by blinking a backlight.

The vehicle 200n images the front with an image sensor. Thereby, the vehicle 200n acquires the captured display image and the decoding image as described above. In the captured display image, a plurality of vehicles 100n traveling in front of the vehicle 200n are displayed.

The vehicle 200n specifies the position of the bright line pattern region in the decoding image, and for example, superimposes the marker at the same position as the bright line pattern region in the captured display image. Then, the vehicle 200n displays the captured display image on which the marker is superimposed on a display inside the vehicle. For example, a captured display image in which a marker is superimposed on the backlight of any one of the plurality of vehicles 100n is displayed. Thereby, an occupant such as a driver of the vehicle 200n can easily grasp which vehicle 100n is the communication partner by looking at the captured display image.

FIG. 159 is a flowchart showing the processing operation of the vehicle 200n.

The vehicle 200n starts wireless communication with the vehicle 100n around the vehicle 200n (step S71). At this time, when a plurality of vehicles are displayed in an image obtained by imaging the periphery of the vehicle 200n by the image sensor of the vehicle 200n, any of the plurality of vehicles is wirelessly communicated. It is not possible to grasp whether it is a communication partner. Therefore, the vehicle 200n requests the communication partner vehicle 100n to transmit a visible light signal by wireless communication (step S72). The communication partner vehicle 100n that has received this request transmits a visible light signal. The vehicle 200n images the surroundings with an image sensor, and receives the visible light signal transmitted from the communication partner vehicle 100n by the imaging (step S73). That is, the vehicle 200n acquires the captured display image and the decoding image as described above. Then, the vehicle 200n specifies the position of the bright line pattern region in the decoding image, and superimposes the marker at the same position as the bright line pattern region in the captured display image. Thereby, even if a plurality of vehicles are displayed on the captured display image, the vehicle 200n can specify the vehicle on which the marker is superimposed among the plurality of vehicles as the communication partner vehicle 100n (step). S74).

FIG. 160 is a diagram illustrating an example in which the receiver 200 according to the present embodiment displays an AR image.

The receiver 200 repeatedly acquires a captured display image Pk and a decoding image, for example, as shown in FIG. 54, by imaging the subject by the image sensor.

Specifically, the image sensor of the receiver 200 images the transmitter 100 configured as signage and the person 21 adjacent to the transmitter 100. The transmitter 100 is the transmitter in each of the above embodiments, and includes one or a plurality of light emitting elements (for example, LEDs) and a light transmitting plate 144 having a light transmitting property like ground glass. The one or more light emitting elements emit light inside the transmitter 100, and light from the one or more light emitting elements is transmitted through the translucent plate 144 and irradiated outside. As a result, the translucent plate 144 of the transmitter 100 shines brightly. Such a transmitter 100 changes in luminance by blinking one or more light emitting elements, and transmits an optical ID (that is, optical identification information) by the change in luminance. This light ID is the above-mentioned visible light signal.

Here, a message “Please hold your smartphone here” is written on the translucent plate 144. Therefore, the user of the receiver 200 places the person 21 next to the transmitter 100 and instructs the person 21 to put out his arm to the transmitter 100. Then, the user images the camera (that is, the image sensor) of the receiver 200 toward the person 21 and the transmitter 100. The receiver 200 captures the transmitter 100 and the person 21 with the normal exposure time, thereby obtaining a captured display image Pk on which they are projected. Furthermore, the receiver 200 acquires a decoding image by capturing the transmitter 100 and the person 21 with a communication exposure time shorter than the normal exposure time.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P45 and the recognition information associated with the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pk as a target area. For example, the receiver 200 recognizes an area in which signage as the transmitter 100 is displayed as a target area.

Then, the receiver 200 superimposes the AR image P45 on the captured display image Pk so that the target area is covered with the AR image P45, and displays the captured display image Pk on the display 201. For example, the receiver 200 acquires an AR image P45 indicating a soccer player. In this case, since the AR image P45 is superimposed so as to cover the target area of the captured display image Pk, the captured display image Pk can be displayed so that a soccer player actually exists next to the person 21. . As a result, even if the soccer player is not next to the person 21, the person 21 can be photographed together with the soccer player.

Here, the AR image P45 shows a soccer player in a state of shaking hands. Therefore, the person 21 presents his / her hand to the transmitter 100 so as to obtain the captured display image Pk shaking hands with the AR image P45. However, the person 21 cannot see the AR image P45 superimposed on the captured display image Pk and does not know whether the AR image P45 is shaking hands with the soccer player.

Therefore, the receiver 200 in the present embodiment transmits the captured display image Pk to the display device D5 as a live view, and displays the captured display image Pk on the display of the display device D5. The display of the display device D5 is directed to the person 21. Accordingly, the person 21 can grasp that the AR image P45 is shaking hands with the soccer player by looking at the captured display image Pk displayed on the display device D5.

161 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

For example, as shown in FIG. 161, the transmitter 100 is configured as a digital signage of an album of music content, for example, and transmits an optical ID by changing the luminance.

The receiver 200 captures the transmitter 100 to repeatedly acquire the captured display image Pr and the decoding image in the same manner as described above. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires, from the server, the first AR image P46, the recognition information, the first music content, and the sub image Ps46 associated with the album specified by the optical ID.

The receiver 200 starts playing the first music content acquired from the server. As a result, the first song, which is the first music content, is output from the speaker of the receiver 200.

Furthermore, the receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pr as a target area. For example, the receiver 200 recognizes an area where the transmitter 100 is projected as a target area. Then, the receiver 200 superimposes the first AR image P46 on the target area, and further superimposes the sub image Ps46 on the outside of the target area. The receiver 200 displays the captured display image Pr on which the first AR image P46 and the sub-image Ps46 are superimposed on the display 201. For example, the first AR image P46 is a moving image related to the first song that is the first music content, and the sub image Ps46 is a still image related to the above-described album. The receiver 200 reproduces the moving image of the first AR image P46 in synchronization with the first music content.

FIG. 162 is a diagram illustrating processing operations of the receiver 200.

For example, similarly to the state shown in FIG. 161, the receiver 200 reproduces the first AR image P46 and the first music content in synchronization with each other as shown in FIG. 162 (a). Here, the user of the receiver 200 performs an operation on the receiver 200. For example, as shown in (b) of FIG. 162, the user performs a swipe. Specifically, the user moves the fingertip in the horizontal direction while placing the fingertip on the first AR image P46 displayed on the display 201 of the receiver 200. In other words, the user slides the first AR image P46 in the horizontal direction. In this case, the receiver 200 is associated with the second music content associated with the above-described optical ID after the first music content, and with the above-described optical ID after the first AR image P46. The second AR image P46c is acquired from the server. For example, the second music content is the second song, and the second AR image P46c is a moving image related to the second song.

Then, the receiver 200 switches the music content to be reproduced from the first music content to the second music content. That is, the receiver 200 stops playing the first music content and starts playing the second song, which is the second music content.

At this time, the receiver 200 switches the image superimposed on the target area of the captured display image Pr from the first AR image P46 to the second AR image P46c. That is, the receiver 20 stops the reproduction of the first AR image P46 and starts the reproduction of the second AR image P46c.

Here, the picture displayed at the beginning of the second AR image P46c is the same as the picture displayed at the beginning of the first AR image P46.

Therefore, as shown in FIG. 162 (a), the receiver 200 first displays the same picture as the first picture of the first AR image P46 when starting the reproduction of the second song. Thereafter, the receiver 200 sequentially displays the second and subsequent pictures included in the second AR image P46c, as shown in FIG. 162 (b).

Here again, the user swipes the receiver 200 as shown in FIG. 162 (b). In response to the operation, the receiver 200, as described above, receives the third music content associated with the above-described optical ID next to the second music content, and the above-mentioned second AR image P46c. The third AR image P46d associated with the optical ID is acquired from the server. For example, the third music content is the third song, and the third AR image P46d is a moving image related to the third song.

Then, the receiver 200 switches the music content to be reproduced from the second music content to the third music content. That is, the receiver 200 stops the reproduction of the second music content and starts reproducing the third song that is the third music content.

At this time, the receiver 200 switches the image superimposed on the target area of the captured display image Pr from the second AR image P46c to the third AR image P46d. That is, the receiver 20 stops the reproduction of the second AR image P46c and starts the reproduction of the third AR image P46d.

Here, the picture displayed at the beginning of the third AR image P46d is the same as the picture displayed at the beginning of the first AR image P46.

Therefore, as shown in FIG. 162 (a), the receiver 200 first displays the same picture as the first picture of the first AR image P46 when starting the reproduction of the third song. Thereafter, the receiver 200 sequentially displays the second and subsequent pictures included in the third AR image P46d, as shown in FIG. 162 (d).

In the above-described example, as illustrated in (b) of FIG. 162, when the receiver 200 receives an operation of sliding the AR image that is a moving image (that is, swipe), the receiver 200 displays the next moving image. However, the receiver 200 may display the next moving image when the optical ID is retaken instead of the operation. When the optical ID is re-acquired, the optical ID is acquired again by imaging with the image sensor. That is, the receiver 200 repeatedly acquires the captured display image and the decoding image by performing imaging, but when the bright line pattern region disappears from the repeatedly acquired decoding image and appears again, the optical ID Is retaken. For example, when the image sensor of the receiver 200 that is directed to the transmitter 100 is directed in another direction, the bright line pattern region disappears from the decoding image. When the image sensor is directed to the transmitter 100 again, a bright line pattern region appears in the decoding image. At this time, the optical ID is retaken.

As described above, in the display method according to the present embodiment, the receiver 200 acquires a visible light signal as an optical ID (that is, identification information) by imaging with the image sensor. Then, the receiver 200 displays the first AR image P46 that is a moving image associated with the optical ID. Next, when receiving an operation of sliding the first AR image P46, the receiver 200 receives a second AR image P46c that is a moving image associated with the light ID next to the first AR image P46. indicate. Therefore, an image useful for the user can be easily displayed.

Further, in the display method according to the present embodiment, in each of the first AR image P46 and the second AR image P46c, the object in the picture displayed first may be at the same position. For example, in the example shown in FIG. 162, the first displayed pictures of the first AR image P46 and the second AR image P46c are the same. Therefore, the objects in those pictures are at the same position. For example, as shown in FIG. 162 (a), the singer as the object is in the same position in the first displayed picture of each of the first AR image P46 and the second AR image P46c. As a result, the user can easily grasp that the first AR image P46 and the second AR image P46c are related to each other. In the example shown in FIG. 162, the first displayed pictures of the first AR image P46 and the second AR image P46c are the same, but if the objects in these pictures are at the same position, Those pictures may be different.

In the display method according to the present embodiment, when the receiver 200 acquires the light ID again by imaging with the image sensor, the receiver 200 displays the next moving image associated with the light ID next to the displayed moving image. indicate. Thereby, a moving image useful for a user can be displayed more easily.

In the display method according to the present embodiment, as shown in FIG. 161, receiver 200 has an area in which at least one moving image of first AR image P46 and second AR image P46c is displayed. The sub image Ps46 is displayed outside. This makes it possible to easily display a wide variety of images that are more useful to the user.

FIG. 163 is a diagram illustrating an example of operations on the receiver 200.

For example, as shown in FIGS. 161 and 162, when the AR image is displayed on the display 201 of the receiver 200, the user performs a vertical swipe as shown in FIG. Specifically, the user moves the fingertip in the vertical direction with the fingertip placed on the AR image displayed on the display 201 of the receiver 200. In other words, the user slides an AR image such as the first AR image P46 in the vertical direction, for example. In this case, the receiver 200 acquires another AR image associated with the above-described optical ID from the server.

FIG. 164 is a diagram illustrating an example of an AR image displayed on the receiver 200.

When a swipe operation as shown in FIG. 163 is performed, the receiver 200 displays the AR image P47 acquired from the server as the other AR image described above, superimposed on the captured display image Pr.

For example, the receiver 200 displays the AR image P47, which is a still image indicating the singer of the music content, superimposed on the captured display image Pr as in the examples shown in FIGS. 146 and 160. Here, the AR image P47 is superimposed on a target area in the captured display image Pr, that is, an area where the transmitter 100 that is digital signage is displayed. Therefore, similarly to the examples shown in FIGS. 146 and 160, when a person stands next to the transmitter 100, the captured display image Pr can be displayed so that a singer actually exists next to the person. . As a result, the person can be photographed with the singer even if the singer is not next to it.

As described above, in the display method according to the present embodiment, when receiving an operation of sliding the first AR image P46 in the horizontal direction, the receiver 200 displays the second AR image P46c and the first AR image. When an operation of sliding P46 in the vertical direction is received, an AR image P47 that is a still image associated with the light ID is displayed. Therefore, it is possible to easily display a wide variety of images useful to the user.

FIG. 165 is a diagram illustrating an example of an AR image superimposed on a captured display image.

As shown in FIG. 165, when superimposing the AR image P48 on the captured display image Pr1, the receiver 200 trims a part of the AR image P48 and superimposes only a part of the AR image P48 on the captured display image Pr1. Also good. For example, the receiver 200 may cut out a peripheral area of the rectangular AR image P48 and superimpose only the round area at the center of the AR image P48 on the captured display image Pr1.

FIG. 166 is a diagram illustrating an example of an AR image superimposed on a captured display image.

The receiver 200 images the transmitter 100 configured as digital signage of a coffee shop, for example. By the imaging, the receiver 200 acquires the captured display image Pr2 and the decoding image as described above. In the captured display image Pr2, the transmitter 100 as the digital signage is displayed as a signage image 100i. The receiver 200 acquires the optical ID by decoding the decoding image, and acquires the AR image P49 associated with the optical ID from the server. Then, the receiver 200 recognizes the region above the signage image 100i in the captured display image Pr2 as the target region, and superimposes the AR image P49 on the target region. The AR image P49 is a moving image in which coffee flows from a pot, for example. The AR image P49, which is the moving image, is formed such that the closer the position in the coffee region flowing down from the pot is to the lower end of the AR image P49, the higher the transparency at that position. Thereby, it is possible to display the AR image P49 so that the coffee actually flows down.

Note that such an AR image P49 may be any moving image as long as it has a vague outline, for example, a flame moving image. When the AR image P49 is a moving image of flame, the transparency at the peripheral portion of the AR image 49 gradually increases toward the outside. Moreover, the transparency may change with time. As a result, the AR image P49 can be displayed as a flickering flame so as to overflow with reality.

Also, at least one moving image of the first AR image P46, the second AR image P46c, and the third AR image P46d shown in FIG. 162 is formed to have transparency as shown in FIG. 166. May be.

That is, in the display method according to the present embodiment, at least one moving image of the first AR image P46 and the second AR image P46c has a position in the moving image that is closer to the end of the moving image. You may form so that the transparency in the said position may become high. Thus, when the moving image is displayed superimposed on the normal captured image, the captured display image is displayed so that an object with an ambiguous outline actually exists in the environment indicated by the normal captured image. Can do.

FIG. 167 is a diagram illustrating an example of the transmitter 100 in the present embodiment.

The transmitter 100 is configured to be able to transmit information as an image ID to a receiver that cannot be imaged in the visible light communication mode, that is, a receiver that does not support optical communication. That is, the transmitter 100 is configured as digital signage, for example, as described above, and transmits the optical ID by changing the luminance. Further, line patterns 151 to 154 are drawn on the transmitter 100. Each of these line patterns 151 to 154 is a short straight line arrangement pattern along a plurality of horizontal directions, and each of the plurality of straight lines is arranged apart from each other in the vertical direction. That is, each of the line patterns 151 to 154 is configured like a barcode. The line pattern 151 is arranged on the left side of the letter A drawn on the transmitter 100, and the line pattern 152 is arranged on the right side of the letter A. The line pattern 153 is arranged on the letter B drawn on the transmitter 100, and the line pattern 154 is arranged on the letter C drawn on the transmitter 100. The characters A, B, and C are examples, and any character or image may be drawn on the transmitter 100.

Since a receiver that does not support optical communication cannot set the exposure time of the image sensor to the above-described communication exposure time, even if the transmitter 100 is imaged, the optical ID cannot be acquired by the imaging. However, the receiver acquires normal captured images (that is, captured display images) on which the line patterns 151 to 154 are projected by imaging of the transmitter 100, and acquires image IDs from these line patterns 151 to 154. be able to. Therefore, a receiver that does not support optical communication can acquire an image ID even if it cannot acquire an optical ID from the transmitter 100. By using the image ID instead of the optical ID, it is possible to acquire the image ID. Similarly to the above, the AR image can be displayed superimposed on the captured display image.

Note that the same image ID may be acquired from each of the line patterns 151 to 154, or different image IDs may be acquired.

FIG. 168 is a diagram illustrating another example of the transmitter in this embodiment.

The transmitter 100e in the present embodiment includes a transmitter main body 115 and a lenticular lens 116. 168 (a) shows a top view of the transmitter 100e, and FIG. 168 (b) shows a front view of the transmitter 100e.

The transmitter main body 115 has the same configuration as the transmitter 100 shown in FIG. That is, on the front surface of the transmitter main body 115, characters A, B, and C and line patterns associated with these characters are drawn.

The lenticular lens 116 is attached to the transmitter main body 115 so as to cover the front surface of the transmitter main body 115, that is, the surface on which the letters A, B, and C and the line pattern are drawn.

Therefore, the line patterns 151 to 154 seen from the front left side of the transmitter 100e shown in FIG. 168 (c) are different from the line patterns 151 to 154 seen from the front right side of the transmitter 100e shown in FIG. 168 (d). Can be made.

FIG. 169 is a diagram illustrating another example of the transmitter 100 in the present embodiment. In addition, (a) of FIG. 169 shows an example in which the transmitter 100 configured as a real object is imaged by the receiver 200a. FIG. 169 (b) shows an example in which a transmitter 100f configured as a fake of the transmitter 100 is imaged by the receiver 200a.

As shown in FIG. 169 (a), the real transmitter 100 is configured to transmit an image ID to a receiver that does not support optical communication, as in the example shown in FIG. 167. . That is, characters A, B, and C, a line pattern 154, and the like are drawn on the front surface of the transmitter 100. Further, a character string 161 may be drawn on the front surface of the transmitter 100. The character string 161 is formed by applying an infrared reflecting paint, an infrared absorbing paint, or an infrared shielding paint. Therefore, the character string 161 is not visible to the human eye, but is displayed on a normal photographed image obtained by imaging by the image sensor of the receiver 200a.

The receiver 200a is a receiver that does not support optical communication. Therefore, even when the above-described visible light signal is transmitted from the transmitter 100, the receiver 200a cannot receive the visible light signal. However, if the receiver 200a captures an image of the transmitter 100, the receiver 200a can acquire the image ID from the line pattern displayed in the normal captured image obtained by the imaging. Furthermore, the receiver 200a can determine that the transmitter 100 is genuine if the character string 161 is displayed on the normal captured image, for example, “Please hold your smartphone here”. That is, the receiver 200 can determine that the acquired image ID is not invalid. In other words, the receiver 200 can authenticate the image ID depending on whether or not the character string 161 is displayed in the normal captured image. When the receiver 200 determines that the image ID is not invalid, the receiver 200 performs processing using the image ID, such as processing for transmitting the image ID to the server.

On the other hand, the transmitter 100 as described above may be illegally copied. That is, there is a case where a transmitter 100f configured as a fake of the transmitter 100 is installed instead of the real transmitter 100. On the front of such a fake transmitter 100f, characters A, B and C and a line pattern 154f are drawn. The letters A, B and C and the line pattern 154f are drawn by a malicious person to resemble the letters A, B and C drawn on the real transmitter 100 and the line pattern 154. It is. That is, the line pattern 154f is similar to the line pattern 154 but different.

However, a malicious person cannot visually recognize the character string 161 drawn by the infrared reflecting paint, the infrared absorbing paint, or the infrared shielding paint when the genuine transmitter 100 is illegally copied. Therefore, the character string 161 is not drawn in front of the fake transmitter 100f.

Therefore, if the receiver 200a images such a fake transmitter 100f, the receiver 200a acquires an illegal image ID from the line pattern displayed in the normal captured image obtained by the imaging. However, as shown in FIG. 169 (b), the receiver 200a can determine that the image ID is illegal because the character string 161 is not displayed in the normal photographed image. As a result, the receiver 200 can prohibit processing using the unauthorized image ID.

FIG. 170 is a diagram illustrating an example of a system using a receiver 200 that supports optical communication and a receiver 200a that does not support optical communication.

For example, the receiver 200a that does not support optical communication images the transmitter 100. In this transmitter 100, the line pattern 154 is drawn as in the example shown in FIG. 167, but the character string 161 shown in FIG. 168 is not drawn. Therefore, the receiver 200a can acquire the image ID from the line pattern displayed in the normal captured image acquired by imaging, but cannot authenticate the image ID. Therefore, even if the image ID is illegal, the receiver 200a trusts the image ID and performs processing using the image ID. For example, the receiver 200a requests the server 300 to perform a procedure associated with the image ID. The procedure is, for example, the transfer of money to an unauthorized bank account.

On the other hand, the receiver 200 compatible with optical communication captures an image of the transmitter 100, thereby acquiring an optical ID that is a visible light signal and also acquiring an image ID as described above. Therefore, the receiver 200 determines whether or not the image ID matches the light ID. If it is determined that the image ID is different from the light ID, the receiver 200 requests the server 300 to discard the request for the procedure associated with the image ID.

Therefore, even if the server 300 receives a request associated with the image ID from the optical communication incompatible receiver 200a, if the server 300 receives a request from the optical communication compatible receiver 200, the server 300 requests the procedure. Discard.

Accordingly, even if a line pattern 154 that can obtain an unauthorized image ID is drawn on the transmitter 100 by a malicious person, the request for the procedure associated with the image ID can be appropriately discarded. .

FIG. 171 is a flowchart showing the processing operation of the receiver 200.

The receiver 200 acquires a normal captured image by imaging of the transmitter 100 (step S81). Then, the receiver 200 acquires an image ID from the line pattern displayed in the normal captured image (step S82).

Next, the receiver 200 acquires the light ID from the transmitter 100 by visible light communication (step S83). That is, the receiver 200 acquires a decoding image by imaging of the transmitter 100 in the visible light communication mode, and acquires an optical ID by decoding the decoding image.

Then, the receiver 200 determines whether or not the image ID acquired in step S82 matches the optical ID acquired in step S83 (step S84). If it is determined that they match (Yes in step S84), the receiver 200 requests the server 300 for a procedure associated with the optical ID (step S85). On the other hand, if it is determined that they do not match (No in step S84), the receiver 200 requests the server 300 to discard the request for the procedure associated with the optical ID (step S86).

FIG. 172 is a diagram illustrating an example of display of an AR image.

For example, the transmitter 100 is formed like a saber and transmits a visible light signal as an optical ID by changing the luminance of a portion other than the saber handle.

The receiver 200 images the transmitter 100 from the vicinity of the transmitter 100 as shown in FIG. While performing the imaging, the receiver 200 repeatedly acquires the captured display image Pr3 and the decoding image as described above. Then, when the receiver 200 acquires the optical ID by decoding the decoding image, the receiver 200 transmits the optical ID to the server. As a result, the receiver 200 acquires the AR image P50 and the recognition information associated with the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pr3 as a target area. For example, the receiver 200 recognizes, as a target area, an area above the area where a portion other than the saber pattern is displayed in the captured display image Pr3.

Specifically, as shown in the examples of FIGS. 50 to 52, the identification information indicates reference information for specifying the reference area in the captured display image Pr3, and the relative position of the target area with respect to the reference area. Target information. For example, the reference information indicates that the position of the reference area in the captured display image Pr3 is the same as the position of the bright line pattern area in the decoding image. Further, the target information indicates that the position of the target area is above the reference area.

Therefore, the receiver 200 identifies the reference area from the captured display image Pr3 based on the reference information. That is, the receiver 200 specifies, as a reference area, an area that is in the same position as the bright line pattern area in the decoding image in the captured display image Pr3. That is, the receiver 200 specifies an area in which a portion other than the saber pattern is projected as a reference area in the captured display image Pr3.

Further, the receiver 200 recognizes, as the target area, an area at a relative position indicated by the target information with respect to the position of the reference area in the captured display image Pr3. In the above-described example, since the target information indicates that the position of the target area is above the reference area, the receiver 200 recognizes the area above the reference area in the captured display image Pr3 as the target area. . That is, the receiver 200 recognizes, as a target area, an area above the area where a portion other than the saber pattern is projected in the captured display image Pr3.

Then, the receiver 200 superimposes the AR image P50 on the target area, and displays the captured display image Pr3 on which the AR image P50 is superimposed on the display 201. For example, the AR image P50 is a moving image of a person.

Here, the receiver 200 moves away from the transmitter 100 as shown in FIG. Therefore, the saber displayed in the captured display image Pr3 becomes small. That is, the size of the bright line pattern area of the decoding image is reduced. As a result, the receiver 200 reduces the size of the AR image P50 so as to match the size of the bright line pattern region. That is, the receiver 200 adjusts the size of the AR image P50 so that the size ratio between the bright line pattern region and the AR image P50 is kept constant.

Thereby, the receiver 200 can display the captured display image Pr3 so that a person actually exists on the saber.

As described above, in the display method according to the present embodiment, the receiver 200 acquires a normal captured image by imaging based on the normal exposure time (that is, the first exposure time) by the image sensor. In addition, the receiver 200 acquires a decoding image including a bright line pattern region, which is a region composed of a plurality of bright line patterns, by imaging with a communication exposure time (that is, a second exposure time) shorter than the normal exposure time. Then, the optical ID is obtained by decoding the decoding image. Next, the receiver 200 identifies a reference area at the same position as the bright line pattern area in the decoding image from the normal captured image, and an area in which the moving image is superimposed on the normal captured image based on the reference area Is recognized as a target area. Then, the receiver 200 superimposes the moving image on the target area. The moving image may be a moving image of at least one of the first AR image P46 and the second AR image P46c shown in FIG. 162 and the like.

Further, the receiver 200 may recognize the upper, lower, left, or right area of the reference area in the normal captured image as the target area.

As a result, for example, as shown in FIGS. 50 to 52 and 172, the target area is recognized based on the reference area, and the moving image is superimposed on the target area. Can be easily increased.

Further, in the display method in the present embodiment, the receiver 200 may change the size of the moving image according to the size of the bright line pattern region. For example, the receiver 200 increases the size of the moving image as the size of the bright line pattern region increases.

As a result, as shown in FIG. 172, the size of the moving image changes according to the size of the bright line pattern region, so that the object indicated by the moving image is more compared to the case where the size of the moving image is fixed. The moving image can be displayed so that it exists in reality.

(Summary of Embodiment 9)
FIG. 173A is a flowchart illustrating a display method according to one embodiment of the present invention.

The display method according to an aspect of the present invention is a display method for displaying an image, and includes steps SG1 to SG3. That is, the display device that is the receiver 200 described above acquires a visible light signal as identification information (that is, a light ID) by imaging with an image sensor (step SG1). Next, the display device displays the first moving image associated with the light ID (step SG2). When the display device receives an operation of sliding the first moving image, the display device displays a second moving image associated with the light ID next to the first moving image (step SG3).

FIG. 173B is a block diagram illustrating a structure of a display device according to one embodiment of the present invention.

The display device G10 according to an aspect of the present invention is a device that displays an image, and includes an acquisition unit G11 and a display unit G12. The display device G10 is the receiver 200 described above. The acquisition unit G11 acquires a visible light signal as identification information (that is, a light ID) by imaging with an image sensor. Next, the display unit G12 displays the first moving image associated with the light ID. Then, when receiving an operation of sliding the first moving image, the display unit G12 displays the second moving image associated with the light ID next to the first moving image.

For example, each of the first moving image and the second moving image is the first AR image P46 and the second AR image P46c shown in FIG. In the display method and the display device G10 shown in FIGS. 173A and 173B, when an operation of sliding the first moving image, that is, a swipe is received, the second associated with the identification information next to the first moving image is received. A moving image is displayed. Therefore, an image useful for the user can be easily displayed.

In the above embodiment, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. For example, the program causes the computer to execute the display method shown by the flowcharts of FIGS. 156, 157, 159, 171 and 173A.

(Embodiment 10)
In this embodiment, a display method, a display device, and the like that realize AR (Augmented Reality) using an optical ID will be described as in Embodiments 4 and 9. Note that the transmitter and the receiver in this embodiment may have the same functions and configurations as the transmitter (or the transmission device) and the receiver (or the reception device) in each of the above embodiments. The receiver in this embodiment is configured as a display device.

FIG. 174 is a diagram illustrating an example of an image drawn on the transmitter according to the present embodiment. FIG. 175 is a diagram illustrating another example of an image drawn on the transmitter in this embodiment.

As in the example shown in FIG. 167, the transmitter 100 can transmit information as an image ID to a receiver that cannot capture an image in the visible light communication mode, that is, a receiver that does not support optical communication. It is configured. That is, the transmitter 100 has a substantially square transmission image Im1 or Im2. The transmitter 100 is configured as a signage, for example, as described above, and transmits an optical ID by changing the luminance. The transmitter 100 may include a light source, and may directly transmit the optical ID to the receiver 200 depending on the luminance change of the light source. Alternatively, the transmitter 100 includes a light source, irradiates light from the light source onto the transmission image Im1 or Im2, and reflects the light on the transmission image Im1 or Im2, thereby transmitting the light to the receiver 200 as an optical ID. May be.

The transmission image Im1 or Im2 of such a transmitter 100 is formed in a substantially rectangular shape as shown in FIGS. 174 and 175. The transmission image Im1 or Im2 has a substantially square base image Bi1 or Bi2 and a line pattern 155a or 155b added to the base image.

In the example shown in FIG. 174, the line pattern 155a is an array pattern of a plurality of short straight lines arranged along each of the four sides of the base image Bi1, and each of these short straight lines has its side. Is orthogonal to. That is, when a logotype is drawn on the base image of the transmitter 100, a signal is embedded around the logotype. A short straight line included in the line pattern is hereinafter referred to as a short line.

In the example shown in FIG. 174, the plurality of short lines included in the line pattern 155a are formed so that the darkness decreases toward the center of the transmitter 100, that is, the center of the base image Bi1. Thereby, even if the line pattern 155a is added to the base image Bi1, the line pattern 155a can be made inconspicuous.

In the example shown in FIG. 174, the line pattern 155a is not arranged at the corner of the base image Bi1, but may be arranged at the corner. Further, when the corner of the base image Bi1 is rounded, the line pattern 155a may not be arranged at the corner.

On the other hand, in the example shown in FIG. 175, the line pattern 155b is disposed within the frame line w at the periphery of the base image Bi2. For example, the base image Bi2 is formed by drawing a rectangular frame w so as to surround a logotype (specifically, a character string of ABC). The line pattern 155b is an array pattern of a plurality of short lines arranged along the rectangular frame line w, and these short lines are orthogonal to the frame line w. Moreover, these short lines are arrange | positioned in the frame line w.

In the example shown in FIG. 175, the line pattern 155b is not arranged at the corner of the frame line w, but may be arranged at the corner. Further, when the corner of the frame line w is rounded, the line pattern 155b may not be arranged at the corner.

FIG. 176 is a diagram illustrating an example of the transmitter 100 and the receiver 200 in the present embodiment.

For example, as in the example shown in FIG. 168, the transmitter 100 may include a lenticular lens 116 as shown in FIG. Such a lenticular lens 116 is attached to the transmitter 100 so as to cover an area excluding the frame line w of the transmission image Im2 drawn on the transmitter 100.

The receiver 200 acquires a normal captured image (that is, a captured display image) on which the line pattern 155b is projected by imaging of the transmitter 100, and acquires an image ID from the line pattern 155b. Here, the receiver 200 prompts the user of the receiver 200 to move the receiver 200. For example, the receiver 200 displays a message “Please move the receiver” when the transmitter 100 is imaging. As a result, the receiver 200 is moved by the user. At this time, the receiver 200 authenticates the acquired image ID by determining whether or not the base image Bi2 in the transmission image Im2 changes, that is, the transmitter 100 displayed in the normal captured image. . For example, if the receiver 200 determines that the logo type of the base image Bi2 has changed from “ABC” to “DEF”, the receiver 200 determines that the acquired image ID is the correct ID.

The above-described transmission image Im1 or Im2 may be drawn on the transmitter 100 that transmits the optical ID. Further, the transmission image Im1 or Im2 described above may be illuminated with light including the light ID from the transmitter 100, and the light ID may be transmitted by reflecting the light. In this case, the receiver 200 can acquire the image ID of the transmission image Im1 or Im2 and the light ID by imaging. At this time, the light ID and the image ID may be the same, or a part of each of the light ID and the image ID may be the same.

Further, the transmitter 100 may be turned on when the transmission switch is turned on, and may be turned off 10 seconds after the start of lighting. The transmitter 100 transmits the light ID during this lighting period. In such a case, the receiver 200 acquires the image ID, and when the brightness of the transmission image displayed in the normal captured image suddenly changes when the transmission switch is turned on, the image ID is received. May be determined to be the correct ID. Further, when the receiver 200 acquires the image ID and the transmission switch is turned on, the transmission image displayed in the normal photographed image becomes brighter and becomes darker after the elapse of a predetermined time. It may be determined that the ID is correct. As a result, it is possible to prevent the transmission image Im1 or Im2 from being illegally copied and used.

FIG. 177 is a diagram for explaining the fundamental frequency of the line pattern.

The encoding device for generating the transmission image Im1 or Im2 determines the fundamental frequency of the line pattern. At this time, for example, as shown in FIG. 177 (a), when the base image to which the line pattern is added is a horizontally long rectangle, the encoding device shows the base image in FIG. 177 (b). So that it transforms into a square. At this time, for example, the shape of the base image is deformed so that the short side of the rectangular base image has the same length as the long side.

Next, as shown in FIG. 177 (c), the encoding apparatus sets the length of the diagonal line of the base image transformed into a square as a basic period, and uses the frequency that is the reciprocal of the basic period as the basic frequency. decide. The base image transformed into a square is hereinafter referred to as a square base image.

FIG. 178A is a flowchart showing the processing operation of the encoding device. FIG. 178B is a diagram for explaining the processing operation of the encoding device.

First, the encoding device adds an error detection code (also referred to as an error correction code) to information to be processed (step S171). For example, as shown in FIG. 178B, the encoding apparatus adds an 8-bit error detection code to a 13-bit bit string that is information to be processed.

Next, the encoding apparatus divides the information to which the error detection code is added into (k + 1) values xk each consisting of N bits. Note that k is an integer of 1 or more. For example, as shown in FIG. 178B, when k = 6, the encoding apparatus divides the information into seven values xk each having N = 3 bits. That is, the information is divided into values x0, x1, x2,..., X6 each indicated by a 3-bit binary number. For example, the values x0, x1, and x2 are x0 = 010, x1 = 010, and x2 = 100.

Next, the encoding apparatus calculates a frequency fk corresponding to each value x0 to x6, that is, for the value xk, corresponding to the value xk (step S173). For example, the encoding apparatus calculates a value that is (A + B × xk) times the fundamental frequency with respect to the value xk as the frequency fk corresponding to the value xk. A and B are positive integers. As a result, as shown in FIG. 178B, frequencies f0 to f6 are calculated for values x0 to x6, respectively.

Next, the encoding device adds the positioning frequency fP to the head of the frequencies f0 to f6 (step S174). At this time, the encoding apparatus sets the positioning frequency fP to a value less than A times the fundamental frequency or larger than (A + B × 2 ^N-1 ) times the fundamental frequency. Thereby, as shown in FIG. 178B, a positioning frequency fP different from these frequencies is arranged at the head of the frequencies f0 to f6.

Next, the encoding apparatus sets (k + 2) designated areas on the periphery of the square base image. Then, the encoding device changes the luminance value (or color) of the designated area at the frequency fk along the direction of the side of the square base image with respect to the original color of the designated area for each designated area. (Step S175). For example, as shown in (a) or (b) of FIG. 178B, (k + 2) designated areas JP, J0 to J6 are set on the periphery of the square base image. When there is a frame line at the periphery of the square base image, each of the (k + 2) areas is set as a designated area by dividing the frame line into (k + 2) areas. More specifically, (k + 2) designated areas are set clockwise along the four sides of the square base image in the order of designated areas JP, JP0, JP1, JP2, JP3, JP4, JP5, and JP6. . The encoding device changes the luminance value (or color) at the frequency fk in each of the designated areas set in this way. A line pattern is added to the square base image by the change in the luminance value.

Next, the encoding apparatus returns the aspect ratio of the square base image with the line pattern to the aspect ratio of the original base image (step S176). For example, a square base image with a line pattern shown in (a) of FIG. 178B is transformed into a base image with a line pattern shown in (c) of FIG. 178B. In this case, the square base image with the line pattern is reduced in the vertical direction. Therefore, as shown in FIG. 178 (c), in the base image with the line pattern, the width of the line pattern above and below the base image is smaller than the width of the line pattern on the left and right.

Therefore, when adding a line pattern to the square base image in step S175, as shown in FIG. 178B (b), a line pattern added to the top and bottom of the square base image and a line pattern added to the left and right. The widths of and may be different. In order to make the widths different, for example, a ratio opposite to the aspect ratio of the original base image may be used. In other words, the encoding device adds the width obtained by multiplying the width of the specified area or line pattern added to the left and right of the square base image by the inverse ratio described above to the top and bottom of the square base image. Determine the width of the specified area or line pattern. As a result, even if the aspect ratio of the square base image with the line pattern is restored in step S176, as shown in FIG. 178B (d), the line pattern on the top and bottom of the base image and the line on the left and right Each width with the pattern can be made the same.

Further, the encoding apparatus may add a frame of a color different from those of the designated areas around the base image with the line pattern, that is, outside the (k + 2) designated areas (step S177). For example, as shown in FIG. 178B, a black frame Q1 is added. This makes it easy to detect (k + 2) designated areas.

FIG. 179 is a flowchart showing the processing operation of the receiver 200 which is a decoding device.

First, the receiver 200 captures a transmission image (step S181). Next, the receiver 200 performs edge detection from a normal captured image obtained by the imaging (step S182), and further extracts a contour (step S183).

Then, the receiver 200 applies the following to an area having a rectangular outline larger than a predetermined size or an area having a rounded square outline larger than a predetermined size from the extracted outlines: Steps S184 to S187 are executed.

That is, the receiver 200 perspective-transforms the area into a square area (step S184). Specifically, when the conversion target area is a quadrangular area, the receiver 200 performs perspective conversion with reference to the quadrangular vertex. Further, when the conversion target area is a rounded quadrangular area, the receiver 200 extends each side of the area and performs perspective conversion on the basis of the point where the two sides intersect.

Next, the receiver 200 obtains the frequency of luminance change in each designated area included in the square area (step S185).

Next, the receiver 200 finds the designated region of the frequency fP, and arranges the frequencies fk of the designated regions arranged in order in the clockwise direction at the periphery of the square region with reference to the designated region of the frequency fP ( Step S186).

Then, the receiver 200 decodes the line pattern by performing reverse processing of steps S171 to S174 shown in FIG. 178A on the frequency array (step S187). That is, the receiver 200 can acquire information to be processed.

In such a processing operation by the receiver 200, in step S184, the line pattern of the transmission image is obtained even when the transmission image is imaged not only from the front but also from a direction other than the front by performing perspective transformation into a square area. Can be decoded correctly. In step S186, the frequency of each designated region is arranged in order based on the fundamental frequency fP, so that the line pattern of the transmission image can be obtained even when the transmission image is imaged horizontally or upside down. It can be decoded correctly.

FIG. 180 is a flowchart showing the processing operation of the receiver 200.

First, the receiver 200 determines whether or not the exposure time can be set to a communication exposure time shorter than the normal exposure time (step S191). That is, the receiver 200 determines whether it is a device that does not support optical communication or a device that supports optical communication. If the receiver 200 determines that the communication exposure time cannot be set (N in step S191), the receiver 200 receives an image signal (that is, an image ID) (step S193). The communication exposure time is, for example, a time of 1/2000 second or less.

On the other hand, when the receiver 200 determines that the communication exposure time can be set (Y in step S191), the receiver 200 determines whether the line scan time is registered in the terminal (that is, the receiver 200) or the server. (Step S192). The line scan time is the time from the start of exposure of one exposure line included in the image sensor to the start of exposure of the next exposure line, as shown in the examples of FIGS. is there. If the line scan time is registered, the receiver 200 decodes the decoding image using the registered line scan time.

When the receiver 200 determines that the line scan time is not registered (N in step S192), the receiver 200 performs the process in step S193. On the other hand, when the receiver 200 determines that the line scan time is registered (Y in step S192), the receiver 200 receives the light ID that is a visible light signal using the line scan time (step S194).

When receiving the visible light signal, the receiver 200 verifies the identity between the image signal and the visible light signal if it is set to the visible light signal identity verification mode (step S195). Here, if the image signal is different from the visible light signal, the receiver 200 displays a message or an image indicating that the signals are different on the display. Alternatively, the receiver 200 notifies the server that the signals are different.

FIG. 181A is a diagram illustrating an example of a system configuration in the present embodiment.

The system in the present embodiment includes a plurality of transmitters 100 and a receiver 200. The transmitter 100 is configured as a self-propelled robot. For example, the robot is an automatic cleaning robot or a robot that communicates with a person. The receiver 200 is configured as a camera such as a surveillance camera or an environment-installed camera. Hereinafter, the transmitter 100 is referred to as a robot 100, and the receiver 200 is referred to as a camera 200.

Robot 100 transmits a light ID, which is a visible light signal, to camera 200. The camera 200 receives the light ID transmitted from the robot 100.

FIG. 181B is a diagram showing processing of the camera 200 in the present embodiment.

Each of the plurality of robots 100 is running automatically. In such a case, first, the camera 200 performs imaging in the normal imaging mode, and detects a moving object as the robot 100 from the normal imaging image obtained by the imaging (step S221). Next, the camera 200 transmits an ID transmission request signal that prompts the detected robot 100 to transmit an ID by radio wave communication (step S225). When the robot 100 receives this ID transmission request signal, the robot 100 starts transmitting the ID of the robot 100 (that is, the light ID) by visible light communication.

Next, the camera 200 changes the shooting mode from the normal shooting mode to the visible light recognition mode (step S226). This visible light recognition mode is a kind of visible light communication mode. Specifically, in the visible light recognition mode, among all the exposure lines included in the image sensor of the camera 200, only a plurality of specific exposure lines capturing the image of the robot 100 are lines at the exposure time for communication. Used for scanning. That is, the camera 200 performs line scanning only for the specific plurality of exposure lines, and does not perform exposure for other exposure lines. Through such a line scan, the camera 200 detects an ID (that is, an optical ID) from the robot 100 (step S227).

Next, the camera 200 recognizes the current position of the robot 100 based on the position of the visible light signal in the decoding image (that is, the bright line image), that is, the position where the bright line pattern appears and the imaging direction of the camera 200. (Step S228). Then, the camera 200 notifies the robot 100 and the server of the ID and current position of the robot 100 and the detection time of the ID.

Then, the camera 200 changes the shooting mode from the visible light recognition mode to the normal shooting mode (step S230).

Here, each of the plurality of robots 100 may automatically travel while transmitting a robot detection signal. The robot detection signal is a visible light signal, but is an optical signal having a frequency that can be recognized even in imaging in the normal shooting mode of the camera 200. That is, the frequency of the robot detection signal is lower than the frequency of the optical ID.

In such a case, instead of detecting the moving object as the robot 100, the camera 200 executes the processes of steps S225 to S230 when detecting a robot detection signal from the normal captured image (step S223). Also good.

Further, each of the plurality of robots 100 may automatically travel while transmitting a position recognition request signal by radio wave communication or the like and transmitting an ID by visible light communication.

In such a case, when receiving the position recognition request signal (step S224), the camera 200 may execute the processes of steps S226 to S230. When the camera 200 receives the position recognition request signal, the robot 100 may not be displayed in the normal captured image. In such a case, the camera 200 may notify the robot 100 that the robot 100 is not projected. That is, the camera 200 may notify the robot 100 that the position of the robot 100 cannot be recognized.

FIG. 182 is a diagram illustrating another example of a system configuration according to the present embodiment.

For example, the transmitter 100 may include a plurality of light sources 171 and transmit light IDs from the plurality of light sources 171 by changing the luminance of the light sources 171. Thereby, the blind spot of the camera 200 can be reduced. That is, the camera 200 can easily receive the optical ID. Further, when a plurality of light sources 171 are captured by the camera 200, the camera 200 can more appropriately recognize the position of the robot 100 by multipoint surveying. That is, the position recognition accuracy of the robot 100 can be improved.

In addition, the robot 100 may transmit different light IDs from each of the plurality of light sources 171. In this case, even when the camera 200 captures not all of the plurality of light sources 171 but only a part of the light sources 171 (for example, only one light source 171), the optical ID of the part of the light sources 171 is used. Thus, the position of the robot 100 can be accurately recognized.

In addition, when the current position of the robot 100 is notified from the camera 200, the robot 100 may give a reward such as a point to the camera 200.

FIG. 183 is a diagram illustrating another example of an image drawn on the transmitter in this embodiment.

Similarly to the examples shown in FIGS. 174 and 175, the transmitter 100 transmits information as an image ID to a receiver that cannot capture an image in the visible light communication mode, that is, a receiver that does not support optical communication. It is configured to be able to. The image ID is also referred to as a frame ID. That is, the transmitter 100 has a substantially rectangular transmission image Im3 drawn therein. The transmitter 100 is configured as a signage, for example, as described above, and transmits an optical ID by changing the luminance. The transmitter 100 may include a light source, and may directly transmit the optical ID to the receiver 200 depending on the luminance change of the light source. Specifically, the transmission image Im3 is drawn on the surface of a light-transmitting plate, and light from the light source is emitted toward the back surface of the plate. As a result, the luminance change of the light source appears as a luminance change of the transmission image Im3, and the light ID is transmitted to the receiver 200 as a visible light signal by the luminance change of the transmission image Im3. Alternatively, the transmitter 100 may be a display device including a display such as a liquid crystal display or an organic EL display. The transmitter 100 transmits the light ID by changing the luminance of the display while displaying the transmission image Im3 on the display. Alternatively, the transmitter 100 may include a light source, irradiate the transmission image Im3 with light from the light source, and reflect the light on the transmission image Im3, thereby transmitting the light to the receiver 200 as an optical ID.

The transmission image Im3 of such a transmitter 100 is formed in a substantially rectangular shape like the transmission images Im1 and Im2 shown in FIGS. 174 and 175. The transmission image Im3 includes a substantially square base image Bi3 and a line pattern 155c added to the base image Bi3.

In the example shown in FIG. 183, the line pattern 155c is an array pattern of a plurality of short lines arranged along the four sides of the base image Bi3, and these short lines (also referred to as short lines). Are each orthogonal to their sides. The line pattern 155c is composed of 32 blocks (the above-mentioned designated area). These blocks are also referred to as PHY symbols, as will be described later. The index of the frequency of each of the 32 blocks is −1, 0, 1, 2, or 3. An index of -1 indicates 200 times the fundamental frequency, an index of 0 indicates 210 times the fundamental frequency, an index of 1 indicates 220 times the fundamental frequency, and an index of 2 indicates 230 times the fundamental frequency. The index of 3 indicates 240 times the fundamental frequency. Here, as described above, the fundamental frequency is the reciprocal of the length of the diagonal line of the base image Bi3 (that is, the fundamental period). That is, in the block corresponding to the index of −1, short lines are arranged at a frequency of fundamental frequency × 200. In other words, the interval between the adjacent short lines in the block is 1/200 of the diagonal line of the base image Bi3. Therefore, each PHY symbol (that is, block) in the present embodiment indicates a numerical value of -1, 0, 1, 2, and 3 depending on the arrangement pattern.

Such a transmission image Im3 is captured as a subject by the image sensor of the receiver 200. In other words, the subject has a rectangular shape as viewed from the image sensor, and a visible light signal is transmitted when the light in the central region of the subject changes in luminance, and a barcode-like line pattern is arranged on the periphery of the subject. ing.

FIG. 184 is a diagram showing an example of the format of the MAC frame that constitutes the frame ID.

The MAC (medium access control) frame consists of a MAC header and a MAC payload. The MAC header is composed of 4 bits. The MAC payload includes variable length padding, variable length ID1, and fixed length ID2. ID2 consists of 5 bits when the MAC frame is composed of 44 bits, and 3 bits when the MAC frame is composed of 70 bits. The padding consists of, for example, “0000000000000001”, “0001”, “01” or “1”, and is a portion from the leftmost bit until 1 appears first.

ID1 is the above-described frame ID, and is the same information as the light ID that is identification information indicated by the visible light signal. That is, the visible light signal and the signal acquired from the line pattern are the same identification information. Thereby, even if the receiver 200 cannot receive the visible light signal, if it captures the transmission image Im3, the receiver 200 acquires the same identification information as the visible light signal from the line pattern 155c of the transmission image Im3. Can do.

FIG. 185 is a diagram illustrating an example of the configuration of the MAC header.

For example, the bit of the address “0” in the MAC header indicates the header version. Specifically, when the bit of the address “0” is “0”, the header version is 1.

2 bits of the address “1-2” in the MAC header indicate the protocol. Specifically, when the 2 bits of the address “1-2” are “00”, the MAC frame protocol conforms to IEC (International Electrotechnical Commission), and the 2 bits of the address “1-2” In the case of “01”, the protocol of the MAC frame conforms to LinkRay (registered trademark) Data. When the 2 bits of the address “1-2” are “10”, the protocol of the MAC frame conforms to IEEE (The Institute of Electrics and Electronics Electronics, Engineers, Inc.).

The bit of the address “3” in the MAC header indicates another protocol. Specifically, when the MAC frame conforms to IEC and the bit of the address “3” is “0”, the number of bits per packet is 4 bits. When the MAC frame conforms to IEC and the bit of the address “3” is “1”, the number of bits per packet is 8 bits. On the other hand, when the MAC frame conforms to LinkRay Data and the bit of the address “3” is “0”, the number of bits per packet is 32 bits. The number of bits per packet described above is the length of DATAPART (that is, the datapart length).

FIG. 186 is a diagram showing an example of a table for deriving the number of packet divisions.

The receiver 200 decodes the frame ID which is ID1 included in the MAC frame from the line pattern 155c and derives the number of divisions corresponding to the frame ID. This is because in visible light communication with a change in luminance, information to be transmitted and received is defined by the light ID and the number of packet divisions, and in order to maintain compatibility with the visible light communication even in communication using a transmission image. This is because the number of divisions is required.

Referring to the table shown in FIG. 186, receiver 200 in the present embodiment derives the number of divisions for the frame ID using a set of the number of bits of ID1 (hereinafter referred to as ID length) and datapart length. . For example, the receiver 200 specifies how many bits the datapart length is based on the bit of the address “3” of the MAC header, and further specifies an ID length that is the length of ID1 of the MAC frame. Then, the receiver 200 derives the number of divisions by finding out the number of divisions associated with the specified datapart length and ID length in the table shown in FIG. 186. Specifically, if the datapart length is 4 bits and the ID length is 10 bits, the division number “5” is derived.

When receiver 200 cannot derive the number of divisions from the table shown in FIG. 186, that is, when the number of divisions associated with the specified datapart length and ID length does not exist in the table, The number of divisions may be determined as 0.

In the table shown in FIG. 186, the number of divisions “6” and “7” are associated with the set of the datapart length “4 bits” and the ID length “14 bits”. Therefore, for example, when the frame ID is encoded, if the ID length is 15 bits, the number of divisions may be set to “7”. When the receiver 200 decodes the frame ID and the datapart length is 4 bits and the ID length is 15 bits, the receiver 200 derives the division number “7”. Furthermore, the receiver 200 may derive the 14-bit ID1 as the final frame ID by ignoring the first 1 bit in the 15-bit ID1.

Note that, when the frame ID conforms to the IEEE protocol, the receiver 200 may tentatively derive the division number “0”, for example. The division number “0” indicates that no division is performed.

Thereby, the light ID and the number of divisions used in visible light communication accompanied by a change in luminance can be appropriately applied to the communication using the transmission image as the frame ID and the number of divisions. That is, it is possible to maintain compatibility between visible light communication involving a change in luminance and communication using a transmission image.

FIG. 187 is a diagram illustrating PHY encoding.

First, the encoding device that encodes the frame ID adds ECC (Error Check Code) to the MAC frame. Next, the encoding apparatus divides the MAC frame to which the ECC is added into a plurality of blocks. The number of bits of the plurality of blocks is N (N is 2 or 3, for example). For each of the plurality of blocks, the encoding apparatus converts a value indicated by the N bits included in the block into a Gray code. Note that the Gray code is a code that differs only in one bit in the numerical values adjacent to each other. In other words, the Gray code has a characteristic that the Hamming distance between adjacent codes before and after is always 1. An error is most likely to occur between adjacent symbols, but if this Gray code is used, a plurality of bits do not differ between the symbols, so that error detection efficiency can be improved.

Then, the encoding device converts the value converted into the gray code for each of the plurality of blocks into a PHY symbol corresponding to the value. As a result, for example, 30 PHY symbols each assigned with a symbol number (0 to 29) are generated. This PHY symbol corresponds to the block of the line pattern 155c shown in FIG. 183, and is a pattern in which short lines are arranged at regular intervals (that is, a stripe pattern). For example, when the value converted to the gray code indicates 1, as shown in FIG. 183, a block (that is, a PHY symbol) having a frequency 220 times the fundamental frequency is generated.

FIG. 188 is a diagram illustrating an example of a transmission image Im3 having a PHY symbol.

As shown in FIG. 188, the above 30 PHY symbols and two header symbols are arranged around the base image Bi3. The header symbol is a symbol having a function as a header among the PHY symbols. The two header symbols include a header symbol for rotational alignment and a header symbol for specifying a PHY version. The index of the frequency of these header symbols is -1. That is, as shown in FIG. 183, the frequency of these header symbols is 200 times the fundamental frequency. The header symbol for rotational alignment is a symbol for making the receiver 200 recognize the arrangement of 30 PHY symbols. The receiver 200 recognizes the arrangement of each PHY symbol with reference to the position of the header symbol for rotational alignment. For example, such a header symbol for rotational alignment is arranged at the upper left corner of the base image Bi3.

The header symbol for specifying the PHY version is a symbol for specifying the PHY version. For example, the PHY version is specified by the relative position of the header symbol for specifying the PHY version from the header symbol for rotational alignment. The above 30 PHY symbols other than the header symbol are arranged in order from the smallest symbol number in the clockwise direction around the base image Bi3 from the right side of the header symbol for rotational alignment.

FIG. 189 is a diagram for explaining two PHY versions.

PHY version includes PHY version 1 and PHY version 2. In PHY version 1, a header symbol for specifying a PHY version is arranged on the right side of the header symbol for rotational alignment. In PHY version 2, the header symbol for specifying the PHY version is not arranged on the right side of the header symbol for rotational alignment. That is, in PHY version 2, the header symbol for specifying the PHY version is arranged so that the PHY symbol with the symbol number “0” is sandwiched between the header symbol for rotational alignment and the header symbol for specifying the PHY version. Is done. Thus, the header symbol for specifying the PHY version indicates the PHY version by its arrangement.

In PHY version 1, the number N of bits per PHY symbol is 2, the ECC is 16 bits, and the MAC frame is 44 bits. The PHY body is composed of a MAC frame and ECC, and is 60 bits. The maximum ID length (ID1 length) is 34 bits, and ID2 is 5 bits.

In PHY version 2, the number N of bits per PHY symbol is 3, the ECC is 20 bits, and the MAC frame is 70 bits. The PHY body is composed of a MAC frame and ECC, and is 90 bits. The maximum ID length (ID1 length) is 62 bits, and ID2 is 3 bits.

FIG. 190 is a diagram for explaining the Gray code.

In PHY version 1, the number of bits N is 2. In this case, in the Gray code conversion shown in FIG. 187, binary numbers “00, 01, 10, 11” corresponding to “0, 1, 2, 3” expressed in decimal numbers are gray codes “00, 01”. , 11, 10 ".

In PHY version 2, the number of bits N is 3. In this case, in the Gray code conversion shown in FIG. 187, binary numbers “000, 001, 010, 011,” corresponding to “0, 1, 2, 3, 4, 5, 6, 7” expressed in decimal numbers. 100, 101, 110, 111 "are converted into gray codes" 000, 001, 011, 010, 110, 111, 101, 100 ".

FIG. 191 is a diagram illustrating an example of a decoding process performed by the receiver 200.

The receiver 200 captures the transmission image Im3 of the transmitter 100, and recognizes the PHY version based on the position of the header symbol (PHY header symbol) included in the line pattern 155c of the captured transmission image Im3 (step) S601). Note that the receiver 200 may determine whether visible light communication is possible, and may capture the transmission image Im3 if it is determined that visible light communication is not possible. In this case, the receiver 200 acquires a captured image by capturing an image of the subject using an image sensor, and extracts at least one contour by performing edge detection of the captured image. Furthermore, the receiver 200 selects, as a selection area, an area having a rectangular outline larger than a predetermined size or an area having a rounded square outline larger than a predetermined size from among the at least one outline. select. There is a high possibility that the transmission image Im3 as the subject is displayed in this selection area. Therefore, in step S601, the receiver 200 recognizes the PHY version based on the position of the header symbol included in the line pattern 155c of the selected area.

In addition, when the receiver 200 determines that visible light communication is possible in the above-described determination of visible light communication, when the subject is imaged, the exposure time of the image sensor is set to the first exposure time as in the above embodiments. By setting the exposure time and imaging the subject with the first exposure time, a decoding image including identification information is acquired. Specifically, in the above-described determination of visible light communication, the receiver 200, when determining that visible light communication is possible, when imaging a subject, as in each of the above embodiments, includes a plurality of image sensors. A decoding image including a bright line pattern composed of a plurality of bright lines corresponding to the exposure line is acquired, and a visible light signal is acquired by decoding the bright line pattern. On the other hand, when the receiver 200 determines that visible light communication is not possible in the above-described determination of visible light communication, when the subject is imaged, the exposure time of the image sensor is set to the second exposure time. By capturing the subject with the second exposure time, a normal image is acquired as a captured image. Here, the first exposure time described above is shorter than the second exposure time.

Next, the receiver 200 restores the MAC frame to which the ECC is added from the plurality of PHY symbols constituting the line pattern 155c, and confirms the ECC (Step S602). Thereby, the receiver 200 receives the MAC frame from the transmitter 100. When the receiver 200 confirms that the same MAC frame has been received the designated number of times within the designated time (step S603), the receiver 200 calculates the number of divisions (that is, the number of packet divisions) (step S604). That is, the receiver 200 refers to the table shown in FIG. 186 and derives the number of divisions for the MAC frame using a set of the ID length and the datapart length in the MAC frame. Thereby, the number of divisions and the frame ID which is ID 1 of the MAC frame are decoded. That is, the receiver 200 acquires identification information from the line pattern of the selection area described above. Specifically, when it is determined that visible light communication is not possible in the above-described determination of visible light communication, the receiver 200 acquires a signal from a line pattern of a normal image when imaging a subject. Here, the visible light signal and the signal are the same identification information.

Incidentally, the transmitter 100 having the transmission image Im3 may be illegally copied. For example, a device such as a smartphone provided with a camera and a display may impersonate the transmitter 100 having the transmission image Im3. Specifically, the smartphone captures the transmission image Im3 of the transmitter 100 with a camera and displays the captured transmission image Im3 on the display. Thereby, the smart phone can transmit frame ID to the receiver 200 by the display of the transmission image Im3 like the transmitter 100. FIG.

Therefore, the receiver 200 determines whether or not the transmission image Im3 displayed on a device such as a smartphone is illegal. When the receiver 200 determines that the transmission image Im3 is illegal, the frame from the unauthorized transmission image Im3 is determined. The decryption of the ID or the use of the frame ID may be prohibited.

FIG. 192 is a diagram for explaining a method of detecting fraud of the transmission image Im3 by the receiver 200.

The transmission image Im3 is, for example, a quadrangle. If the transmission image Im3 is illegal, the rectangular frame of the transmission image Im3 is likely to be inclined in the same plane with respect to the display frame that displays the transmission image Im3. On the other hand, if the transmission image Im3 is valid, the square frame of the transmission image Im3 is not inclined in the same plane with respect to the above-described frame.

Further, if the transmission image Im3 is illegal, the rectangular frame of the transmission image Im3 is likely to be inclined in the depth direction with respect to the frame of the display that displays the transmission image Im3. On the other hand, if it is a legitimate transmission image Im3, the square frame of the transmission image Im3 is not inclined in the depth direction with respect to the above-described frame.

The receiver 200 detects the fraud of the transmission image Im3 based on the difference between the legitimate transmission image Im3 and the fraudulent transmission image Im3 as described above.

Specifically, as shown in FIG. 192 (a), the receiver 200 captures the frame of the transmission image Im3 (the dashed rectangle shown in FIG. 192 (a)) and the transmission image Im3 by imaging with the camera. For example, the display frame of the smartphone (solid square shown in FIG. 192 (a)) is recognized. Next, the receiver 200 is included in each combination including any one of the two diagonals of the frame of the transmission image Im3 and any one of the two diagonals of the frame of the display. An angle formed by two diagonal lines is calculated. Then, the receiver 200 determines whether or not the angle formed with the smallest absolute value among the angles calculated for each combination is equal to or greater than a first threshold (for example, 5 degrees), thereby transmitting the transmission image. It is determined whether Im3 is invalid. That is, the receiver 200 determines whether or not the transmission image Im is illegal depending on whether or not the square frame of the transmission image Im3 is inclined in the same plane with respect to the display frame. The receiver 200 determines that the transmission image Im3 is invalid if the angle formed by the minimum absolute value is equal to or greater than the first threshold, and if the angle formed is less than the first threshold, the transmission image Im3. Is determined to be valid.

Further, as shown in FIG. 192 (b), the receiver 200 determines the ratio (a / b) of two sides facing each other in the vertical direction of the frame of the transmission image Im3 and the vertical direction of the frame of the smartphone display. The ratio of two sides facing each other (A / B) is calculated. And the receiver 200 compares those ratios. Specifically, the receiver 200 divides a small ratio between the ratio (a / b) and the ratio (A / B) by a large ratio. Then, the receiver 200 determines whether or not the transmission image Im3 is illegal by determining whether or not the value obtained by the division is equal to or greater than a second threshold (for example, 0.9). That is, the receiver 200 determines whether or not the transmission image Im is illegal depending on whether or not the square frame of the transmission image Im3 is inclined in the depth direction with respect to the display frame. If the value obtained by the division is less than the second threshold, the receiver 200 determines that the transmission image Im3 is invalid. If the value is equal to or greater than the second threshold, the transmission image Im3 is valid. Judge that there is.

The receiver 200 decodes the frame ID from the transmission image Im3 only when the transmission image Im3 is valid, and prohibits the decoding of the frame ID from the transmission image Im3 if the transmission image Im3 is invalid.

FIG. 193 is a flowchart illustrating an example of a decoding process including fraud detection of the transmission image Im3 by the receiver 200.

First, the receiver 200 captures the transmission image Im3 and detects the frame of the transmission image Im3 (step S611). Next, the receiver 200 performs detection processing of a rectangular frame that includes the frame of the transmission image Im3 (step S612). The quadrangular frame is a frame formed from the outer periphery of a quadrangular display included in a device such as the smartphone described above. Here, the receiver 200 determines whether or not a rectangular frame is to be detected by the detection process in step S612 (step S613). If it is determined that the quadrangular frame is not detected (No in step S613), the receiver 200 prohibits decoding of the frame ID (step S619).

On the other hand, if it is determined that a quadrangular frame has been detected (Yes in step S613), the receiver 200 calculates an angle formed by each diagonal line between the frame of the transmission image Im3 and the detected quadrangular frame (step S614). Then, the receiver 200 determines whether or not the angle formed is less than the first threshold (step S615). If it is determined that the angle formed is equal to or greater than the first threshold (No in step S615), the receiver 200 prohibits decoding of the frame ID (step S619).

On the other hand, if it is determined that the angle formed is less than the second threshold (Yes in step S615), the receiver 200 determines the ratio (a / b) of the two sides in the frame of the transmission image Im3 and the two sides in the rectangular frame. Division using the ratio (A / B) is performed (step S616). Then, the receiver 200 determines whether or not the value obtained by the division is less than the second threshold value (step S617). Here, when it is determined that the obtained value is equal to or greater than the second threshold (No in step S617), the receiver 200 decodes the frame ID (step S618). On the other hand, when the receiver 200 determines that the obtained value is less than the second threshold value (Yes in step S617), the receiver 200 prohibits decoding of the frame ID (step S619).

In the above-described example, the receiver 200 prohibits decoding of the frame ID according to the determination result of step S613, S615, or S617. However, the receiver 200 may execute the above steps after decoding the frame ID first. In this case, the receiver 200 prohibits the use of the decoded frame ID or discards the frame ID according to the determination result of step S613, S615, or S617.

Further, a prism seal may be attached to the transmission image Im3. In this case, similarly to the example illustrated in FIG. 176, the receiver 200 determines whether or not the prism seal pattern or color of the transmission image Im3 is changed by the movement of the receiver 200. If the receiver 200 determines that the transmission is changed, the receiver 200 determines that the transmission image Im3 is valid, and decodes the frame ID from the transmission image Im3. On the other hand, if it is determined that there is no change, the receiver 200 determines that the transmission image Im3 is invalid, and prohibits decoding of the frame ID from the transmission image Im3. Note that, as described above, the receiver 200 may perform the frame ID decoding first, and then perform the pattern or color change determination. In this case, when the receiver 200 determines that the pattern or color does not change, the receiver 200 prohibits the use of the decoded frame ID or discards the frame ID.

Further, the receiver 200 may determine whether or not the transmission image Im3 is valid by causing the user to bring the receiver 200 close to the transmission image Im3. For example, the transmitter 100 transmits a visible light signal by turning on the transmission image Im3 and changing the luminance of the transmission image Im3. Therefore, when the receiver 200 captures the transmission image Im3, the receiver 200 displays a message prompting the user to bring the receiver 200 closer to the transmission image Im3 on the display. In response to the message, the user brings the camera (that is, the image sensor) of the receiver 200 closer to the transmission image Im3. At this time, the camera of the receiver 200 sets the exposure time of the image sensor to, for example, the minimum because the amount of light received from the transmission image Im3 increases. As a result, a striped pattern appears in the image displayed on the display when the receiver 200 captures the transmission image Im3. If the receiver 200 is compatible with optical communication, the striped pattern clearly appears as a bright line pattern. On the other hand, even if the receiver 200 is not compatible with optical communication, the stripe pattern does not appear clearly as a bright line pattern, but the transmission image Im3 is legitimate depending on whether or not the stripe pattern appears. It can be determined whether or not. That is, if a stripe pattern appears, the receiver 200 determines that the transmission image Im3 is valid, and if no stripe pattern appears, the receiver 200 determines that the transmission image Im3 is illegal.

Note that, similarly to the above, the receiver 200 may execute the decoding of the frame ID first, and then perform the determination of the stripe pattern. In this case, when the receiver 200 determines that the stripe pattern does not appear, use of the decoded frame ID is prohibited or the frame ID is discarded.

(Modification)
The receiver 200 in the present embodiment may be a display device having the function of the receiver 200 in the ninth embodiment. That is, the display device determines whether visible light communication is possible. If possible, the display device performs processing related to visible light or light ID as in the receiver 200 in each of the embodiments including the ninth embodiment. Do. On the other hand, when visible light communication is impossible, a display apparatus performs the process regarding the above-mentioned transmission image or frame ID. Note that the visible light communication is by decoding a bright line pattern corresponding to each exposure line of the image sensor, which is obtained by transmitting a signal according to a change in luminance of the subject and imaging the subject by the image sensor. This is a communication method for receiving the signal.

FIG. 194A is a flowchart showing a display method according to this modification.

The display method according to an aspect of the present invention is a display method for displaying an image, and includes steps SG1 to SG4. That is, the display device that is the above-described receiver 200 first determines whether or not visible light communication is possible (step SG4). Here, when it is determined that visible light communication is possible (Yes in step SG4), the display device acquires a visible light signal as identification information (that is, a light ID) by capturing an image of the subject with the image sensor (step ID). SG1). Next, the display device displays the first moving image associated with the light ID (step SG2). When the display device receives an operation of sliding the first moving image, the display device displays a second moving image associated with the light ID next to the first moving image (step SG3).

FIG. 194B is a block diagram showing a configuration of a display device according to this modification.

The display device G10 according to an aspect of the present invention is a device that displays an image, and includes a determination unit G13, an acquisition unit G11, and a display unit G12. The display device G10 is the receiver 200 described above. The determination unit G13 determines whether visible light communication is possible. When the determination unit G13 determines that visible light communication is possible, the acquisition unit G11 acquires a visible light signal as identification information (that is, a light ID) by imaging a subject with an image sensor. Next, the display unit G12 displays the first moving image associated with the light ID. Then, when receiving an operation of sliding the first moving image, the display unit G12 displays the second moving image associated with the light ID next to the first moving image.

For example, each of the first moving image and the second moving image is the first AR image P46 and the second AR image P46c shown in FIG. In the display method and the display device G10 shown in FIGS. 194A and 194B, when an operation of sliding the first moving image, that is, a swipe is received, the second associated with the identification information next to the first moving image is received. A moving image is displayed. Therefore, an image useful for the user can be easily displayed. In addition, since it is determined in advance whether or not visible light communication is possible, it is possible to omit a wasteful process of acquiring a visible light signal and to reduce a processing burden until it is impossible. it can.

Here, if the display device G10 determines that visible light communication is not possible in the determination of visible light communication, the display device G10 may acquire identification information (that is, a frame ID) from the transmission image Im3. In this case, the display device G10 acquires a captured image by capturing an image of the subject using an image sensor, and extracts at least one contour by performing edge detection of the captured image. Next, the display device G10 selects, from among the at least one outline, an area having a square outline larger than a predetermined size or an area having a rounded square outline larger than a predetermined size. Choose as. Then, the display device G10 acquires identification information from the line pattern of the selected area. The rounded quadrangular shape described above is a quadrangular shape in which the outer periphery of each of the four corners is an arc shape.

Thereby, for example, the transmission image shown in FIGS. 183 and 188 is captured as a subject, the area of the transmission image is selected as the selection area, and the identification information is acquired from the line pattern of the transmission image. Therefore, even when visible light communication is impossible, identification information can be acquired appropriately.

Further, when it is determined that visible light communication is possible in the determination of visible light communication, the display device G10 sets the exposure time of the image sensor to the first exposure time when imaging the subject, and the first exposure time is set. By capturing the subject with the exposure time, a decoding image including identification information is acquired. Further, when it is determined that visible light communication is not possible in the determination of visible light communication, the display device G10 sets the exposure time of the image sensor to the second exposure time when the subject is imaged, and the second exposure time. A normal image is acquired as a captured image by capturing the subject with the exposure time of. Here, the first exposure time described above is shorter than the second exposure time.

Thus, by switching the exposure time, it is possible to appropriately switch between acquisition of identification information by visible light communication and acquisition of identification information by capturing a transmission image.

Further, the above-mentioned subject has a rectangular shape as viewed from the image sensor, and the center region of the subject changes in luminance, so that a visible light signal is transmitted, and a barcode-like line pattern is arranged around the subject. ing. When the display device G10 determines that visible light communication is possible in the determination of visible light communication, when the subject is imaged, the bright line pattern composed of a plurality of bright lines corresponding to a plurality of exposure lines of the image sensor. And a visible light signal is obtained by decoding the bright line pattern. The visible light signal is, for example, a light ID. Further, when it is determined that visible light communication is not possible in the determination of visible light communication, the display device G10 acquires a signal from the line pattern of the normal image when imaging the subject. Here, the visible light signal and the signal are the same identification information.

As a result, the identification information indicated by the visible light signal is the same as the identification information indicated by the line pattern signal, and therefore the identification information indicated by the visible light signal is appropriate even if visible light communication is impossible. Can be obtained.

FIG. 194C is a flowchart showing a communication method according to this modification.

The communication method according to one aspect of the present invention is a communication method using a terminal including an image sensor, and includes steps SG11 to SG13. That is, the terminal that is the above-described receiver 200 first determines whether or not the terminal can perform visible light communication (step SG11). Here, when it is determined that the terminal can perform visible light communication (Yes in step SG11), the terminal executes the process of step SG12. That is, the terminal acquires a decoding image by imaging a subject whose luminance changes with an image sensor, and acquires first identification information transmitted by the subject from a striped pattern that appears in the decoding image (step). SG12). On the other hand, if the terminal determines that the terminal cannot perform visible light communication in the determination of visible light communication in step SG11 (No in step SG11), the terminal executes the process in step SG13. That is, the terminal acquires a captured image by capturing an image of a subject with an image sensor, extracts at least one contour by detecting an edge of the captured image, and selects a predetermined contour from the at least one contour. The second identification information transmitted by the subject is acquired from the line pattern of the specific area (step SG13). Note that the first identification information is, for example, an optical ID, and the second identification information is, for example, an image ID or a frame ID.

FIG. 194D is a block diagram showing a configuration of a communication apparatus according to this modification.

The communication device G20 according to an aspect of the present invention is a communication device using a terminal including an image sensor, and includes a determination unit G21, a first acquisition unit G22, and a second acquisition unit G23.

The determination unit G21 determines whether or not the terminal can perform visible light communication.

When the determination unit G21 determines that the terminal can perform visible light communication, the first acquisition unit G22 acquires a decoding image by imaging a subject whose luminance changes with the image sensor. First identification information transmitted from the subject is obtained from the striped pattern appearing in the decoding image.

When the determination unit G21 determines that the terminal cannot perform visible light communication, the second acquisition unit G23 acquires a captured image by capturing an image of the subject using the image sensor, and the captured image By detecting the edge, at least one contour is extracted, a predetermined specific area is specified from the at least one outline, and second identification information transmitted from the subject is obtained from the line pattern of the specific area To do.

Note that the terminal may be included in the communication device G20 or may be outside the communication device G20. Alternatively, the terminal may include a communication device G20. That is, each step of the flowchart illustrated in FIG. 194C may be executed by the terminal or the communication device G20.

Thereby, a terminal such as the receiver 200 can acquire the first identification information or the second identification information from a subject such as a transmitter regardless of whether or not visible light communication is possible. That is, when the terminal can perform visible light communication, the terminal acquires, for example, the light ID from the subject as the first identification information. On the other hand, even if the terminal cannot perform visible light communication, the terminal can acquire, for example, an image ID or a frame ID from the subject as the second identification information. Specifically, for example, the transmission image illustrated in FIGS. 183 and 188 is captured as a subject, the area of the transmission image is selected as a specific area (that is, a selection area), and second identification information is obtained from the line pattern of the transmission image. Is acquired. Therefore, even when visible light communication is impossible, the second identification information can be appropriately acquired.

In addition, in specifying the specific area described above, the terminal specifies, as the specific area, an area having a rectangular outline larger than a predetermined size or an area having a rounded square outline larger than a predetermined size. May be.

Thus, for example, as shown in FIG. 179, a quadrangular or rounded quadrangular region can be appropriately identified as the specific region.

Further, in the above-described determination of visible light communication, the terminal determines that visible light communication can be performed when the terminal specifies that the exposure time can be changed to a predetermined value or less. Then, when the terminal specifies that the exposure time cannot be changed to the predetermined value or less, it may be determined that the visible light communication cannot be performed.

Thereby, for example, as shown in FIG. 180, it is possible to appropriately determine whether or not a visible light signal can be performed.

In addition, when the terminal determines that the terminal can perform visible light communication in the determination of visible light communication, when imaging the subject, the terminal sets the exposure time of the image sensor to the first exposure time, A decoding image may be acquired by imaging the subject with the first exposure time. Further, when the terminal determines that the terminal is not capable of performing visible light communication in the determination of visible light communication, when imaging the subject, the exposure time of the image sensor is set to the second exposure time, The captured image may be acquired by capturing the subject with the second exposure time. Here, the first exposure time is shorter than the second exposure time.

Thereby, it is possible to acquire a decoding image having a bright line pattern area by imaging at the first exposure time, and appropriately acquire the first identification information by decoding the bright line pattern area. Furthermore, a normal captured image can be acquired as a captured image by imaging at the second exposure time, and the second identification information can be appropriately acquired from the line pattern appearing in the normal captured image. Accordingly, the terminal can acquire the first identification information or the second identification information suitable for the terminal by properly using the first exposure time and the second exposure time.

The subject has a rectangular shape as viewed from the image sensor, and the first identification information is transmitted by changing the luminance of the central region of the subject, and a barcode-like line pattern is arranged around the subject. ing. When the terminal determines that the terminal can perform visible light communication in the determination of visible light communication, the terminal includes a plurality of bright lines corresponding to the plurality of exposure lines of the image sensor when imaging the subject. The decoding image including the bright line pattern to be obtained is acquired, and the first identification information is acquired by decoding the bright line pattern. Further, when the terminal determines that the terminal cannot perform visible light communication in the determination of visible light communication, the terminal acquires second identification information from the line pattern of the captured image when imaging the subject. Also good.

Thus, the first identification information and the second identification information can be appropriately acquired from the subject whose central region changes in luminance.

Also, the first identification information obtained from the decoding image and the second identification information obtained from the line pattern may be the same information.

Thus, the same information can be acquired from the subject both in a terminal capable of visible light communication and a terminal incapable of visible light communication.

FIG. 194E is a block diagram showing a configuration of the transmitter according to Embodiment 10 and its modifications.

Transmitter G30 corresponds to the transmitter 100 described above. The transmitter G30 includes a light source G31, a microcontroller G32, and an illumination plate G33. The light source G31 irradiates light from the back side of the illumination plate 33. The microcontroller G32 changes the luminance of the light source G31. The illumination plate G33 is a plate that transmits light from the light source G31, that is, a light-transmitting plate. Moreover, the shape of the illumination plate G33 is, for example, a rectangular shape.

The microcontroller G32 transmits the first identification information from the light source G31 via the illumination plate G33 by changing the luminance of the light source G31. In addition, a barcode-like line pattern G34 is arranged around the front side of the illumination plate G33, and second identification information is encoded in the line pattern G34. Further, the first identification information and the second identification information are the same information.

Thereby, the same information can be transmitted to a terminal capable of performing visible light communication and a terminal capable of performing visible light communication.

In the above embodiment, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. For example, the program causes the computer to execute the display method shown by the flowcharts of FIGS. 191, 193, 194A, and 194C.

(Embodiment 11)
The server management method in the present embodiment is a method capable of providing appropriate services to the user of the mobile terminal.

FIG. 195 is a diagram illustrating an example of a configuration of a communication system including a server in the present embodiment.

This communication system includes a transmitter 100, a receiver 200, a first server 301, a second server 302, and a store system 310. The transmitter 100 and the receiver 200 in this embodiment may have the same functions as the transmitter 100 and the receiver 200 in each of the above embodiments. Further, the transmitter 100 is configured as, for example, a signage of a store, and transmits the light ID as a visible light signal by changing the luminance. The store system 310 has at least one computer for managing a store having the transmitter 100. The receiver 200 is a portable terminal configured as a smartphone including a camera and a display, for example.

For example, the user of the receiver 200 performs a reservation process in advance for the store system 310 by operating the receiver 200. This reservation process is a process for registering user information, such as the user's name, on user information and a menu ordered by the user in the store system 310 before the user visits the store. Note that the user does not have to perform such reservation processing.

Then, when the user visits the store, the receiver 200 images the transmitter 100 that is the signage of the store. Thereby, the receiver 200 receives the optical ID from the transmitter 100 by visible light communication. Then, the receiver 200 transmits the optical ID to the second server 302 via wireless communication. When the second server 302 receives the optical ID from the receiver 200, the second server 302 transmits store information associated with the optical ID to the receiver 200 via wireless communication. The store information is information regarding stores that have the signage.

When the receiver 200 receives the store information from the second server 302, the receiver 200 transmits the user information and the store information to the first server 301 via wireless communication. When the first server 301 receives the user information and the store information, the first server 301 makes an inquiry to the store system 310 indicated by the store information to determine whether or not the reservation process by the user indicated by the user information is completed. Determine.

Here, if the first server 301 determines that the reservation process has been completed, the first server 301 notifies the store system 310 of the arrival of the user at the store via wireless communication. On the other hand, if it is determined that the reservation process has not been completed, the first server 301 transmits the menu list of the store to the receiver 200 via wireless communication. When the receiver 200 receives the menu list, the receiver 200 displays the menu list on the display and accepts a menu selection from the user. Then, the receiver 200 notifies the first server 301 via wireless communication using the menu selected by the user as a selection menu.

When the first server 301 receives the notification of the selection menu from the receiver 200, the first server 301 notifies the store system 310 of the selection menu via wireless communication.

FIG. 196 is a flowchart showing a management method by the first server 301.

First, the first server 301 receives store information from the mobile terminal that is the receiver 200 (step S621). Next, the first server 301 determines whether or not the reservation process for the store indicated by the store information has been completed (step S622). If the first server 301 determines that the reservation process has been completed (Yes in step S622), the first server 301 notifies the store system 310 of the arrival of the mobile terminal user at the store (step S623). On the other hand, when the first server 301 determines that the reservation process has not been completed (No in step S622), the first server 301 notifies the portable terminal of the menu list of the store (step S624). Furthermore, when the selection menu which is the menu selected from the menu list is notified from the portable terminal, the first server 301 notifies the store system 310 of the selection menu (step S625).

Thus, in the management method of the server (that is, the first server 301) in the present embodiment, the server receives store information from the mobile terminal, and based on the store information, the store by the user of the mobile terminal It is determined whether or not the reservation process of the menu is completed, and if the reservation process is completed, a notification that the user of the mobile terminal has arrived at the store is sent to the store system I do. Further, in the management method, when the reservation process is not completed, the server notifies the mobile terminal of the menu list of the store, and receives a menu selection from the mobile terminal. The selected menu is notified to the store system. In the management method, the portable terminal acquires a visible light signal as identification information by photographing a subject installed in the store, transmits the identification information to another server, and transmits the identification information from the other server. Store information corresponding to the identification information is received, and the received store information is transmitted to the server.

As a result, if the user of the mobile terminal makes a reservation process in advance, when the user arrives at the store, the user can immediately start cooking the order menu for the store, and eat and drink freshly prepared dishes. it can. Further, even if the user does not perform the reservation process, when the user arrives at the store, the user can immediately select the menu from the menu list and place an order with respect to the store.

The receiver 200 transmits identification information (that is, an optical ID) instead of the store information to the first server 301, and the first server 301 indicates whether or not the reservation process has been completed based on the identification information. You may check. In this case, the identification information is transmitted from the mobile terminal to the first server 301 without transmitting the identification information to the second server 302.

(Embodiment 12)
In the present embodiment, a communication method and a communication apparatus using an optical ID will be described as in the above embodiments. Note that the transmitter and the receiver in this embodiment may have the same functions and configurations as the transmitter (or the transmission device) and the receiver (or the reception device) in each of the above embodiments.

FIG. 197 is a diagram illustrating an illumination system according to the present embodiment.

This illumination system includes, for example, a plurality of first illumination devices 100p and a plurality of second illumination devices 100q as shown in FIG. Such an illumination system is attached to the ceiling of a large store, for example. The plurality of first lighting devices 100p and the plurality of second lighting devices 100q are each formed in a long shape, and are arranged in a line along the longitudinal direction thereof. In addition, the first lighting device 100p and the second lighting device 100q are alternately arranged.

The first lighting device 100p is configured as the transmitter 100 in each of the above-described embodiments, emits light for illumination, and transmits a visible light signal as a light ID. In addition, the second illumination device 100q emits illumination light and transmits a dummy signal. In other words, the second illumination device 100q emits illumination light and periodically transmits a dummy signal by changing the luminance periodically. When the illumination system is imaged by the receiver in the visible light communication mode, the above-described visible light communication image or the decoding image that is the bright line image obtained by the imaging is included in an area corresponding to the first illumination device 100p. The bright line pattern area appears. However, the bright line pattern region does not appear in the region corresponding to the second illumination device 100q in the decoding image.

Thus, in the illumination system shown in FIG. 197 (a), the second illumination device 100q is disposed between the two first illumination devices 100p adjacent to each other. Accordingly, the receiver that receives the visible light signal can appropriately identify the edge of the bright line pattern region in the decoding image, and distinguishes and receives the visible light signal transmitted from each first illumination device 100p. can do.

Further, the average luminance when the second lighting device 100q emits light (that is, when a dummy signal is transmitted) and the average luminance when the first lighting device 100p emits light (that is, transmits a visible light signal). The average brightness is equal. Therefore, variation in the brightness of each lighting device in the lighting system can be suppressed. Note that the brightness of each lighting device is the brightness that humans see and feel. Thereby, the person in a store can make it difficult to sense the variation in the brightness in the lighting system. Further, in the case where the luminance change of the second lighting device 100q is performed by switching between lighting (on) and extinguishing (off), even if the second lighting device 100q does not have a dimming function, The average luminance of the second lighting device 100q can be adjusted by the duty ratio between on and off.

Further, for example, as shown in FIG. 197 (b), the lighting system includes a plurality of first lighting devices 100p and does not need to include the second lighting device 100q. In this case, the plurality of first lighting devices 100p are arranged in a line along the longitudinal direction of the first lighting devices 100p and separated from each other.

Therefore, even in the illumination system shown in FIG. 197 (b), similarly to the illumination system shown in FIG. 197 (a), the receiver that receives the visible light signal uses the edge of the bright line pattern region in the decoding image. Can be identified appropriately. As a result, the receiver can distinguish and receive visible light signals transmitted from the respective first lighting devices 100p.

Alternatively, the plurality of first lighting devices 100p may be arranged adjacent to each other, and a boundary portion between the two first lighting devices 100p adjacent to each other may be covered with a cover. This cover blocks light emitted from the boundary portion. Alternatively, each of the plurality of first lighting devices 100p may have a structure in which light is not irradiated from both end portions in the longitudinal direction of the first lighting device 100p.

In the illumination system shown in FIGS. 197 (a) and (b), the receiver uses the length in the longitudinal direction of the first illumination device 100p included in the illumination system to use the first illumination device. The distance from 100p can be calculated. Therefore, the receiver can accurately estimate the position of the receiver.

FIG. 198 is a diagram illustrating an example of the arrangement of lighting devices and an image for decoding.

For example, as shown in FIG. 198 (a), the first lighting device 100p and the second lighting device 100q are arranged adjacent to each other. Here, the second lighting device 100q transmits the dummy signal by switching on and off at a cycle of 100 μs or less.

The receiver captures the first illumination device 100p and the second illumination device 100q to capture the decoding image shown in FIG. 198 (b). Here, the on / off switching cycle in the second lighting device 100q is too short compared to the exposure time of the receiver. Therefore, the luminance of the region corresponding to the second illumination device 100q (hereinafter referred to as a dummy region) in the decoding image is uniform. In addition, the luminance of this dummy region is higher than the region corresponding to the background other than the first lighting device 100p and the second lighting device 100q. Further, the luminance of this dummy region is lower than the high luminance in the region corresponding to the first illumination device 100p, that is, the bright line pattern region.

Therefore, the receiver can distinguish the illumination device corresponding to the dummy area from the illumination device corresponding to the bright line pattern area.

FIG. 199 is a diagram showing another example of the arrangement of lighting devices and a decoding image.

For example, as shown in FIG. 199 (a), the first lighting device 100p and the second lighting device 100q are arranged adjacent to each other. Here, the second lighting device 100q transmits a dummy signal by switching on and off at a cycle exceeding 100 μs.

The receiver captures the first illumination device 100p and the second illumination device 100q to capture the decoding image shown in FIG. 199 (b). Here, the cycle of switching on and off in the second lighting device 100q is longer than the exposure time of the receiver. Therefore, the luminance of the dummy area in the decoding image is not uniform, and bright areas and dark areas appear alternately in the dummy area. For example, when a dark area wider than the predetermined maximum width appears in the decoding image, the receiver can recognize that there is a dummy area in the range including the dark area.

Therefore, the receiver can distinguish the illumination device corresponding to the dummy area from the illumination device corresponding to the bright line pattern area.

FIG. 200 is a diagram for explaining position estimation using the first lighting device 100p.

The receiver 200 can estimate the position of the receiver 200 by imaging the first lighting device 100p as described above.

However, the receiver 200 may notify the user of an error when the height from the floor at the estimated position is higher than the allowable range. For example, the receiver 200 determines the position of the first lighting device 100p based on the length in the longitudinal direction of the first lighting device 100p displayed in the decoding image or the normal captured image, the output of the acceleration sensor, and the like. And identify the orientation. Furthermore, the receiver 200 specifies the height from the floor at the position of the receiver 200 using the height from the floor to the ceiling where the first lighting device 100p is installed. Then, when the height at the position of the receiver 200 is higher than the allowable range, the receiver 200 notifies an error. Note that the position and orientation of the first lighting device 100p described above are relative to the receiver 200. Therefore, it can be said that the position and orientation of the receiver 200 are specified by specifying the position and orientation of the first lighting device 100p.

FIG. 201 is a flowchart showing the processing operation of the receiver 200.

First, the receiver 200 estimates the position of the receiver 200 as shown in FIG. 201 (a) (step S231). Next, the receiver 200 derives the height from the floor to the ceiling (step S232). For example, the receiver 200 derives the height from the floor to the ceiling by reading the height stored in the memory. Alternatively, the receiver 200 derives the height from the floor to the ceiling by receiving information transmitted by radio waves from transmitters in the vicinity.

Next, the receiver determines that the height from the floor to the receiver 200 is within an allowable range based on the position of the receiver 200 estimated in step S231 and the height from the floor to the ceiling derived in step S232. It is determined whether it is within (step S233). Here, when the receiver determines that the height is within the allowable range (Yes in step S233), the receiver displays the position and orientation of the receiver 200 (step S234). On the other hand, when the receiver determines that the height is not within the allowable range (No in step S233), only the direction of the receiver 200 is displayed (step S235).

Alternatively, the receiver 200 may execute step S236 instead of step S235 as shown in FIG. 201 (b). That is, when the receiver determines that the height is not within the allowable range (No in step S233), the receiver notifies the user that an error has occurred in the position estimation (step S236).

FIG. 202 is a diagram illustrating an example of a communication system according to the present embodiment.

The communication system includes a receiver 200 and a server 300. The receiver 200 receives position information or a transmitter ID transmitted by GPS, radio waves, or visible light signals. The position information is information indicating the position of the transmitter or the receiver, for example, and the transmitter ID is identification information for identifying the transmitter. Then, the receiver 200 transmits the received position information or transmitter ID to the server 300. The server 300 transmits a map or content associated with the position information or transmitter ID to the receiver 200.

FIG. 203 is a diagram for describing self-position estimation processing by the receiver 200 in the present embodiment.

The receiver 200 performs self-position estimation every predetermined cycle. This self-position estimation consists of a plurality of processes. The above-described cycle is, for example, a frame cycle of imaging by the receiver 200.

For example, the receiver 200 acquires the result of self-position estimation performed in the previous frame period as the previous self-position. Then, the receiver 200 estimates the moving distance and moving direction from the immediately preceding self-position based on the outputs from the acceleration sensor and the gyro sensor. Furthermore, the receiver 200 performs self-position estimation in the current frame period by changing the previous self-position according to the estimated moving distance and moving direction. Thereby, the first self-position estimation result is obtained. On the other hand, the receiver 200 performs self-position estimation in the current frame period based on at least one of radio waves, visible light signals, and outputs from the acceleration sensor and the azimuth sensor. Thereby, a second self-position estimation result is obtained. And the receiver 200 adjusts the 2nd self-position estimation result based on a 1st self-position estimation result using Kalman filter etc., for example. As a result, a final self-position estimation result in the current frame period is obtained.

FIG. 204 is a flowchart showing self-position estimation by receiver 200 in the present embodiment.

First, the receiver 200 estimates the position of the receiver 200 based on the strength of radio waves (step S241). Thereby, the estimated position A of the receiver 200 is obtained.

Next, the receiver 200 measures the moving direction and moving direction of the receiver 200 based on the outputs from the acceleration sensor, the gyro sensor, the azimuth sensor, and the like (step S242).

Next, the receiver 200 receives the visible light signal, and measures the position of the receiver 200 based on the received visible light signal and outputs from the acceleration sensor and the azimuth sensor (step S243).

Then, the receiver 200 uses the moving distance and moving direction of the receiver 200 measured in step S242 and the position of the receiver 200 measured in step S243 to obtain the estimated position A obtained in step S241. Update (step S243). For updating the estimated position A, an algorithm such as Kalman filter is used. And the process after step S242 is repeatedly performed.

FIG. 205 is a flowchart showing an outline of self-position estimation processing of the receiver 200 in the present embodiment.

First, the receiver 200 estimates the approximate position of the receiver 200 based on, for example, the strength of radio waves such as Bluetooth (registered trademark) (step S251). Next, the receiver 200 estimates the detailed position of the receiver 200 using a visible light signal or the like (step S252). Thereby, for example, the self-position can be estimated within an error range of ± 10 cm.

Note that the number of optical IDs that can be assigned to each transmitter is small, and a unique optical ID cannot be given to each transmitter in the world. However, in the present embodiment, as in the process of step S251 described above, the area where the transmitter that transmits the radio wave exists is narrowed down based on the strength of the radio wave. If a plurality of transmitters each having the same optical ID does not exist in the area, the receiver 200 identifies one transmitter from the area by the processing in step S252, that is, based on the optical ID. be able to.

The server stores, for each transmitter, the optical ID of the transmitter, the position information indicating the position of the transmitter, and the radio wave ID of the transmitter in association with each other.

FIG. 206 is a diagram showing the relationship between radio wave IDs and optical IDs in the present embodiment.

For example, the same information as the optical ID is included in the radio wave ID. The radio wave ID is identification information used for Bluetooth (registered trademark) or Wi-Fi (registered trademark). That is, the transmitter transmits the radio wave ID by radio wave, and transmits information that matches at least a part of the radio wave ID as an optical ID. For example, the lower few bits included in the radio wave ID match the optical ID. Thus, the server can centrally manage the radio wave ID and the optical ID.

In addition, the receiver 200 can check whether there are a plurality of transmitters that transmit the same optical ID in the vicinity of the receiver 200 via radio waves. Then, when the receiver 200 confirms that there are a plurality of transmitters, the receiver 200 may change the optical ID of any transmitter via radio waves.

FIG. 207 is a diagram for describing an example of imaging by the receiver 200 in this embodiment.

For example, as shown in FIG. 207 (a), the receiver 200 images the first lighting device 100p in the visible light communication mode at the position A. Furthermore, the receiver 200 images the first lighting device 100p in the visible light communication mode at the position B. Here, the position A and the position B are point-symmetric with respect to the first lighting device 100p. In this case, the receiver 200 generates the same decoding image by imaging as shown in (b) of FIG. 207 at both the position A and the position B. Therefore, the receiver 200 cannot distinguish whether the receiver 200 is at the position A or the position B only from the decoding image shown in FIG. 207 (b). Therefore, the receiver 200 may present position A and position B as candidates for self-position estimation. The receiver 200 may narrow down one candidate from a plurality of candidates based on the past position of the receiver 200 and the moving direction from the position. In addition, when two or more illumination devices are displayed in the decoding image, the decoding image obtained at the position A is different from the decoding image obtained at the position B. Therefore, in this case, the candidate positions of the receiver 200 can be narrowed down to one.

Note that the receiver 200 can narrow down the position A and the position B to one position based on the output from the direction sensor. However, even in such a case, when the reliability of the azimuth sensor is low, the receiver 200 may present each of the position A and the position B as position candidates for the receiver 200.

FIG. 208 is a diagram for describing another example of imaging by the receiver 200 in the present embodiment.

For example, a mirror 901 may be disposed around the first lighting device 100p. Thereby, the decoding image obtained by imaging at the position A can be made different from the decoding image obtained by imaging at the position B. That is, it is possible to suppress the occurrence of a situation where the position of the receiver 200 cannot be narrowed down to one by self-position estimation based on the decoding image.

FIG. 209 is a diagram for describing a camera used by the receiver 200 in the present embodiment.

For example, the receiver 200 includes a plurality of cameras, and selects a camera used for visible light communication from the plurality of cameras. Specifically, the receiver 200 identifies the direction of the receiver 200 based on output data from the acceleration sensor, and selects an upward camera from a plurality of cameras. Alternatively, the receiver 200 may select one or a plurality of cameras that can perform upward imaging from the horizontal direction based on the orientation of the receiver 200 and the angles of view of the plurality of cameras. In addition, when a plurality of cameras are selected, the receiver 200 may further select one camera having the widest upward area in the angle of view from the plurality of cameras. In addition, the receiver 200 may not perform processing for self-position estimation or reception of an optical ID for a part of an image obtained by imaging by the camera. The partial area may be an area that is downward from the horizontal direction, or may be an area that is further downward from a direction inclined by a predetermined angle from the horizontal direction.

Thereby, the calculation load of the receiver 200 can be reduced.

FIG. 210 is a flowchart illustrating an example of processing in which the receiver 200 in the present embodiment changes the visible light signal of the transmitter.

First, the receiver 200 receives the visible light signal A as a visible light signal (step S261).

Next, the receiver 200 transmits an instruction “change to visible light signal B when visible light signal A is transmitted” by radio waves (step S262).

Then, the transmitter 100 receives the command transmitted in step S262 from the receiver. If the visible light signal transmitted by the transmitter, which is the first lighting device 100p, is set to the visible light signal A based on the command, the visible light signal A that is set is visible. The optical signal is changed to B (step S263).

FIG. 211 is a flowchart showing another example of processing in which receiver 200 in the present embodiment changes the visible light signal of the transmitter.

First, the receiver 200 receives the visible light signal A as a visible light signal (step S271).

Next, the receiver 200 searches for transmitters capable of radio wave communication by receiving surrounding radio waves, and creates a list of the transmitters (step S272).

Next, the receiver 200 rearranges the order of the plurality of transmitters shown in the created list in a predetermined order (step S273). This predetermined order is, for example, the order in which the radio field intensity is strong, the random order, or the order in which the transmitter ID is small.

Next, the receiver 200 instructs the first transmitter shown in the list by radio waves to transmit the visible light signal B for a predetermined time (step S274). Then, the receiver 200 determines whether or not the visible light signal A received in step S271 is changed to the visible light signal B (step S275). Here, when the receiver 200 determines that the change has been made (Y in step S275), the receiver 200 instructs the first transmitter shown in the list to continue the transmission of the visible light signal B (step S276). .

On the other hand, if it is determined that the visible light signal A is not changed to the visible light signal B (N in step S275), the receiver 200 returns the visible light signal to the signal before the change for the first transmitter in the list. (Step S277). Then, the receiver 200 deletes the first transmitter shown in the list from the list, and increments the order of the second and subsequent transmitters by one (step S278). Then, the receiver 200 repeatedly executes the processes from step S274.

By such processing, the receiver 200 can appropriately identify the transmitter that is transmitting the visible light signal that is currently received by the receiver 200, and cause the transmitter to change the visible light signal. it can.

(Embodiment 13)
The receiver 200 performs self-position estimation and navigation using the estimation result, as in the example shown in FIGS. 18A to 18C of the second embodiment. When performing the self-position estimation, the receiver 200 uses the size and position of the bright line pattern area included in the decoding image. That is, the receiver 200 receives signals from the transmitter 100 based on the attitude of the receiver 200, the size and shape of the transmitter 100, and the size, shape, and position of the bright line pattern region included in the decoding image. The relative position of the machine 200 is specified. Then, the receiver 200 estimates its own position using the position of the transmitter 100 on the map specified by the visible light signal from the transmitter 100 and the specified relative position described above. Note that the attitude of the receiver 200 is, for example, the orientation of the camera of the receiver 200 specified by output data from sensors such as an acceleration sensor and an orientation sensor provided in the receiver 200.

FIG. 212 is a diagram for explaining navigation by the receiver 200.

The transmitter 100 is configured as a digital signage for guiding a bus stop, for example, as shown in FIG. 212 (a), and is set in an underground mall. Then, transmitter 100 transmits a visible light signal as in the example shown in FIGS. 18A to 18C of the second embodiment. Here, the transmitter 100 displays an image prompting AR navigation. When the user of the receiver 200 looks at the transmitter 100 and wants to be guided to the bus stop by AR navigation, the user starts the AR navigation application installed in the receiver 200 configured as a smartphone. . By this activation, the receiver 200 causes the built-in camera to alternately perform imaging in the visible light communication mode and imaging in the normal imaging mode. And every time imaging in the normal imaging mode is performed, the receiver 200 displays a normal captured image obtained by the imaging on the display of the receiver 200. Then, the user points the camera of the receiver 200 toward the transmitter 100. Thereby, the receiver 200 acquires the decoding image at the timing when the imaging in the visible light communication mode is performed, and decodes the bright line pattern region included in the decoding image, thereby receiving the signal from the transmitter 100. A visible light signal is received. And the receiver 200 transmits the information (namely, light ID) shown with the visible light signal to a server, and receives the data which shows the position on the map of the transmitter 100 linked | related with the information from the server. . Furthermore, the receiver 200 performs self-position estimation using the position of the transmitter 100 on the map, and transmits the estimated self-position to the server. The server searches the location of the receiver 200 and the route to the destination bus stop, and transmits a map and data indicating the route to the receiver 200. Note that the position of the receiver 200 obtained by the self-position estimation at this time is a starting point for guiding the user to the destination.

Next, as shown in FIG. 212 (b), the receiver 200 starts navigation according to the searched route. At this time, the receiver 200 superimposes the direction indication image 431 on the normal captured image and displays it on the display. This direction indication image 431 is generated based on the searched route, the current position of the receiver 200, and the direction of the camera, and is configured as an arrow directed in the direction toward the destination.

Then, when the receiver 200 moves in the underground shopping street, as shown in FIGS. 212 (c) and (d), the receiver 200 estimates the current self-position based on the movement of the feature points displayed in the normal photographed image. To do.

Furthermore, as shown in (e) of FIG. 212, when the receiver 200 receives a visible light signal from a transmitter 100 different from the transmitter 100 shown in (a) of FIG. Correct the self-position. That is, the receiver 200 updates the self position by performing self position estimation using the visible light signal again.

Then, as shown in FIG. 212 (f), the receiver 200 guides the user to the bus stop that is the destination.

Thus, at the start point, the receiver 200 may first perform self-position estimation based on the visible light signal and periodically update the self-position obtained by the estimation. For example, as shown in (c) and (d) of FIG. 212, the receiver 200 captures a normal captured image at a constant frame rate when capturing images in the normal capturing mode. The self position may be updated from the amount of movement of the feature point displayed in the captured image. Then, the receiver 200 periodically performs imaging in the visible light communication mode while imaging in the normal imaging mode is being performed. As shown in FIG. 212 (e), if the bright line pattern area is displayed in the decoding image obtained by imaging in the visible light communication mode, the receiver 200 updates the self that has been updated recently at that time. The position may be corrected based on the projected bright line pattern region.

Here, the receiver 200 can estimate the self-position by decoding the bright line pattern area even if it cannot receive the visible light signal. That is, even if the receiver 200 cannot completely decode the bright line pattern area displayed in the decoding image, the receiver 200 is based on the bright line pattern area or a striped area such as the bright line pattern area. Based on the above, self-position estimation may be performed.

FIG. 213 is a flowchart illustrating an example of self-position estimation by the receiver 200.

The receiver 200 acquires the map and the transmitter data of each of the plurality of transmitters 100 from the server or the recording medium included in the receiver 200 (step S341). The transmitter data indicates the position on the map where the transmitter 100 is arranged, and the shape and size of the transmitter 100.

Next, the receiver 200 captures an image in the visible light communication mode (that is, short exposure), and detects a stripe-shaped region (that is, the region A) from the decoding image obtained by the imaging (step S342).

Then, the receiver 200 determines whether or not there is a possibility that the striped region is a visible light signal (step S343). That is, the receiver 200 determines whether or not the stripe-shaped region is a bright line pattern region that appears due to a visible light signal. Here, when the receiver 200 determines that there is no possibility of being a visible light signal (N in step S343), the process ends. On the other hand, when the receiver 200 determines that there is a possibility of being a visible light signal (Y in step S343), it further determines whether or not the visible light signal has been received (step S344). That is, the receiver 200 decodes the bright line pattern region of the decoding image and determines whether or not the light ID has been acquired as a visible light signal by the decoding.

Here, when the receiver 200 determines that the visible light signal has been received (Y in step S344), the receiver 200 acquires the shape, size, and position of the region A in the decoding image (step S347). That is, the receiver 200 acquires the shape, size, and position of the transmitter 100 that is displayed as a striped image on the decoding image by imaging in the visible light communication mode.

Then, the receiver 200 calculates the relative position between the transmitter 100 and the receiver 200 based on the transmitter data of the transmitter 100 and the acquired shape, size, and position of the area A. The current position of 200 (that is, the current self position) is updated (step S348). For example, the receiver 200 selects transmitter data of the transmitter 100 corresponding to the received visible light signal from the transmitter data of each transmitter 100 acquired in step S341. That is, the receiver 200 selects the transmitter 100 corresponding to the visible light signal from among the plurality of transmitters 100 shown on the map as the imaging target transmitter 100 displayed as the image of the area A. . Based on the shape, size, and position of the transmitter 100 acquired in step S347 and the shape and size indicated in the transmitter data of the imaging target transmitter 100, the receiver 200 The relative position with respect to the transmitter 100 is calculated. Thereafter, the receiver 200 updates its own position based on the relative position, the map acquired in step S341, and the position on the map indicated by the transmitter data of the imaging target transmitter 100.

On the other hand, when the receiver 200 determines that the visible light signal cannot be received in step S344 (N in step S344), it estimates which position or range on the map is captured by the camera of the receiver 200. (Step S345). That is, the receiver 200 estimates the position or range imaged on the map based on the current self-position estimated at that time and the direction or direction of the camera that is the imaging unit of the receiver 200. To do. In the receiver 200, the transmitter 100 that is most likely to be imaged among the plurality of transmitters 100 shown on the map is the transmitter 100 that is displayed as an image of the area A. Deemed (step S346). That is, the receiver 200 selects the transmitter 100 having the highest possibility of being imaged as the transmitter 100 to be imaged, from among the plurality of transmitters 100 shown on the map. Note that the transmitter 100 having the highest possibility of being imaged is, for example, the transmitter 100 closest to the imaging position or range estimated in step S345.

FIG. 214 is a diagram for explaining a visible light signal received by the receiver 200.

The bright line pattern area included in the decoding image appears in two cases. In the first case, the bright line pattern region appears when the receiver 200 directly images the transmitter 100 such as a lighting device disposed on the ceiling, for example. In other words, in the first case, the light that causes the bright line pattern region to appear is direct light. In the second case, the bright line pattern region appears when the receiver 200 indirectly images the transmitter 100. That is, the receiver 200 images a reflection area such as a wall or a floor reflecting light from the transmitter 100 without imaging the transmitter 100 such as a lighting device. As a result, a bright line pattern region appears in the decoding image. In other words, in the second case, the light that causes the bright line pattern region to appear is reflected light.

Therefore, if there is a bright line pattern area in the decoding image, the receiver 200 according to the present embodiment determines whether the case where the bright line pattern area appears is the first case or the second case. That is, the receiver 200 determines whether the bright line pattern region appears due to the direct light of the transmitter 100 or the reflected light of the transmitter 100.

Then, when the receiver 200 determines that this is the first case, the bright line pattern region in the decoding image is used as the transmitter 100 displayed in the decoding image, and the receiver 200 with respect to the transmitter 100 uses the bright line pattern region. Specify the relative position. That is, the receiver 200 uses the orientation and field angle of the camera used for imaging, the shape, size and position of the bright line pattern region, and the shape and size of the transmitter 100 to perform triangulation or geometrical measurement. The relative position of the receiver 200 is specified by the surveying method.

On the other hand, when determining that the receiver 200 is in the second case, the receiver 200 uses the bright line pattern area in the decoding image as a reflection area displayed in the decoding image, and the receiver 200 is relative to the transmitter 100. Identify the location. That is, the receiver 200 uses the orientation and angle of view of the camera used for imaging, the shape, size and position of the bright line pattern region, the position and orientation of the floor or wall indicated by the map, and the shape of the transmitter 100. The relative position of the receiver 200 is specified by triangulation or a geometric surveying method using and the size. At this time, the receiver 200 may use the center of the bright line pattern region as the position of the bright line pattern region.

FIG. 215 is a flowchart showing another example of self-position estimation by the receiver 200.

First, the receiver 200 receives a visible light signal by imaging in the visible light communication mode (step S351). Then, the receiver 200 acquires the map and the transmitter data of each of the plurality of transmitters 100 from the recording medium (that is, the database) included in the server or the receiver 200 (step S352).

Next, the receiver 200 determines whether or not the visible light signal received in step S351 is received by reflected light (step S353).

If the receiver 200 determines that it has been received by the reflected light in step S353 (Y in step S353), the center of the striped region in the decoding image obtained by the imaging in step S351 is placed on the floor or wall. It is regarded as the position of the transmitter 100 being projected (step S354).

Next, the receiver 200 calculates the relative position between the transmitter 100 and the receiver 200 and updates the current position of the receiver 200 (step S355), as in step S348 in FIG. On the other hand, if the receiver 200 determines that it is not received by the reflected light in step S353 (N in step S353), the receiver 200 updates the current position of the receiver 200 without considering the floor or wall reflection.

FIG. 216 is a flowchart illustrating an example of determination of reflected light by the receiver 200.

The receiver 200 detects a stripe-shaped area or bright line pattern area as the area A from the decoding image (step S641). Next, the receiver 200 specifies the orientation of the camera when the decoding image is captured by the acceleration sensor (step S642). Next, the receiver 200 determines from the map data whether the transmitter 100 exists in the direction of the camera specified in step S642 from the position on the map of the receiver 200 that has already been estimated at that time. Specify (step S643). That is, the receiver 200 transmits based on the estimated position of the receiver 200 on the map, the direction or direction of imaging of the receiver 200, and the position of each transmitter 100 on the map. It is determined whether the apparatus 100 is directly imaged.

When the receiver 200 determines that the transmitter 100 is present (Yes in step S644), the receiver 200 determines that the light in the area A, that is, the light used for receiving the visible light signal is direct light (step S645). . On the other hand, if the receiver 200 determines that the transmitter 100 does not exist (No in step S644), the receiver 200 determines that the light in the region A, that is, the light used for receiving the visible light signal is reflected light (step S646). ).

Thus, the receiver 200 determines whether the light that causes the bright line pattern region to appear is direct light or reflected light by using the acceleration sensor. The receiver 200 may determine that the light is direct light if the camera is directed upward, and may determine that the light is reflected light if the camera is directed downward.

Further, the receiver 200 determines whether the light is direct light or reflected light based on the intensity, position, or size of light in the bright line pattern region included in the decoding image instead of the output of the acceleration sensor. Also good. For example, if the intensity of light is less than a predetermined intensity, the receiver 200 determines that the light that causes the bright line pattern region to appear is reflected light. Alternatively, if the position of the bright line pattern region is below the decoding image, the receiver 200 determines that the light is reflected light. Alternatively, if the size of the bright line pattern region is larger than a predetermined size, the receiver 200 determines that the light is reflected light.

FIG. 217 is a flowchart showing an example of navigation by the receiver 200.

The receiver 200 is, for example, a smartphone including an out camera, an in camera, and a display, and performs navigation by displaying an image for guiding the user to the destination on the display. That is, the receiver 200 performs AR navigation as in the example illustrated in FIGS. 18A to 18C of the second embodiment. At this time, the receiver 200 captures an image with an out-camera, and performs surrounding danger detection based on an image obtained by the imaging. Then, the receiver 200 determines whether or not the user is in a dangerous situation (step S361).

Here, when the receiver 200 determines that the user is in a dangerous situation (Y in step S361), the receiver 200 displays a warning message on the display of the receiver 200 or stops navigation (step S364). .

On the other hand, when the receiver 200 determines that the user is not in a dangerous situation (N in step S361), the receiver 200 determines whether walking in the area where the receiver 200 is located is prohibited (step S362). For example, the receiver 200 refers to the map data and determines whether or not the current position of the receiver 200 is included in the walking smartphone prohibition range indicated by the map data. Here, when the receiver 200 determines that the walking smartphone is not prohibited (N in Step S362), the navigation is continued (Step S366). On the other hand, when the receiver 200 determines that the walking smartphone is prohibited (Y in step S362), the receiver 200 determines whether the user is looking at the receiver 200 by recognizing the user's line of sight with the in-camera. (Step S363). Here, if the receiver 200 determines that the receiver 200 is not seen (N in step S363), the navigation is continued (step S366). On the other hand, when it is determined that the receiver 200 is looking at the receiver 200 (Y in step S363), a warning message is displayed on the display of the receiver 200 or the navigation is stopped (step S364).

Then, the receiver 200 determines whether or not the user has escaped the dangerous situation or has left the state in which the user is gazing at the receiver 200 (step S365). Here, when the receiver 200 determines that the situation or state has been removed (Y in step S365), the navigation is continued (step S366). On the other hand, if the receiver 200 determines that the situation or state has not been removed (N in step S365), the receiver 200 repeatedly executes the process in step S364.

Further, the receiver 200 may detect the moving speed based on outputs from an acceleration sensor, a gyro sensor, and the like. In this case, the receiver 200 determines whether or not the moving speed is equal to or higher than the threshold, and may stop the navigation if the moving speed is above the threshold. At this time, the receiver 200 may display a message for notifying that walking at the moving speed is dangerous. Thereby, the danger by walking smartphone can be avoided.

Here, the transmitter 100 may be configured as a projector.

FIG. 218 is a diagram illustrating an example of the transmitter 100 configured as a projector.

For example, the transmitter 100 projects the image 441 on the floor or wall. Further, the transmitter 100 transmits a visible light signal by projecting the image 441 and changing the luminance of light used for the projection. The projected image 441 may display characters that promote AR navigation. The receiver 200 receives a visible light signal by capturing an image 441 projected on its floor or wall. Then, the receiver 200 may perform self-position estimation using the projected image 441. For example, the receiver 200 acquires the position on the map of the image 441 associated with the visible light signal from the server, and performs self-position estimation using the position of the image 441. Alternatively, the receiver 200 acquires the position on the map of the transmitter 100 associated with the visible light signal from the server, and displays the image 441 projected on the floor or wall in the second case described above. As described above, self-position estimation may be performed by treating it as reflected light.

FIG. 219 is a flowchart showing another example of self-position estimation by the receiver 200.

The receiver 200 first images the transmitter 100, a predetermined image, or a predetermined code (such as a two-dimensional code) (step S371). Note that in imaging by the transmitter 100, the receiver 200 receives a visible light signal from the transmitter 100.

Next, the receiver 200 acquires the position of the subject imaged in step S371 (that is, the position on the map). Then, the receiver 200 estimates the position of the receiver 200, that is, the self position based on the position, shape, and size of the object and the position, shape, and size of the subject in the image obtained by the imaging in step S371. (Step S372).

Next, the receiver 200 starts navigation for guiding the user to a predetermined position indicated by the image obtained by imaging in step S371 (step S373). If the subject is the transmitter 100, the predetermined position is a position specified by a visible light signal. If the subject is a predetermined image, the predetermined position is a position obtained by analyzing the predetermined image. If the subject is a code, the predetermined position is a position obtained by decoding the code. When performing navigation, the receiver 200 repeatedly captures images with the camera, and sequentially displays normal captured images obtained by the imaging in real time, while displaying a direction indication image such as an arrow indicating the destination of the user. Superimpose on. While carrying the receiver 200, the user starts moving according to the displayed direction instruction image.

Next, the receiver 200 determines whether or not position information such as GPS (that is, GPS data) can be received (step S374). If the receiver 200 determines that it can receive the signal (Y in step S374), the receiver 200 estimates the current position of the receiver 200 using the position information such as GPS (step S375). On the other hand, when the receiver 200 determines that position information such as GPS cannot be received (N in step S374), the receiver 200 receives the information based on the movement of the object or feature point displayed in each of the above normal captured images. The self-position of the machine 200 is estimated (step S376). For example, the receiver 200 detects the movement of the object or feature point displayed in each of the above-described normal captured images, and estimates the moving direction and moving distance of the receiver 200 based on the movement. Then, receiver 200 estimates the current position of receiver 200 based on the estimated moving direction and moving distance, and the position estimated in step S372.

Next, the receiver 200 determines whether or not the recently estimated self-position is within a predetermined distance from a predetermined position as the destination (step S377). Here, when the receiver 200 determines that its own position is within the distance (Y in step S377), the receiver 200 determines that the user has arrived at the destination and ends the navigation process. On the other hand, if the receiver 200 determines that the self-position is not within the distance (N in step S377), the receiver 200 determines that the user has not arrived at the destination, and repeatedly executes the processing from step S374.

Further, when the receiver 200 loses sight of the current self-position during navigation, that is, when the self-position cannot be estimated, the receiver 200 stops superimposing the direction indication image on the normal captured image, and finally The self-position estimated in (1) may be displayed on a map. Alternatively, the receiver 200 may display a map of the surrounding area including the last estimated self-location.

FIG. 220 is a flowchart illustrating an example of processing performed by the transmitter 100. In the example shown in FIG. 220, the transmitter 100 is a lighting device installed in an elevator.

The transmitter 100 determines whether or not elevator operation information indicating the operation status of the elevator can be obtained from the elevator (step S381). The elevator operation information may indicate a situation such as whether the elevator is rising, falling or stopped, a floor where the elevator is currently located, a floor scheduled to stop, and the like.

Here, when the transmitter 100 determines that the elevator operation information has been obtained (Y in step S381), the transmitter 100 transmits all or part of the elevator operation information by a visible light signal (step S386). Alternatively, the transmitter 100 may associate the elevator operation information with the visible light signal (that is, the light ID) transmitted from the transmitter 100 and cause the server to hold the elevator operation information.

On the other hand, if the transmitter 100 determines that the elevator operation information cannot be obtained (N in step S381), the transmitter 100 recognizes whether the elevator is in a stopped state, in an ascending state, or in a descending state (step S381). Step S382). Further, the transmitter 100 determines whether or not the floor where the elevator is currently located can be specified from the floor display section of the elevator (step S383). The floor display unit corresponds to the floor display unit shown in FIG. 18C. If it is determined that the floor can be specified (Y in step S383), the transmitter 100 executes the process in step S386 described above. On the other hand, when determining that the floor cannot be specified (N in step S383), the transmitter 100 further captures the floor display unit with a camera, and the elevator is currently located from the image obtained by the imaging. It is determined whether or not the floor can be recognized (step S384).

Here, when the transmitter 100 determines that the floor has been recognized (Y in step S384), the transmitter 100 executes the process in step S386 described above. On the other hand, if the transmitter 100 determines that the floor cannot be recognized (N in step S384), the transmitter 100 transmits a predetermined visible light signal (step S385).

FIG. 221 is a flowchart showing another example of navigation by the receiver 200. In the example illustrated in FIG. 221, the transmitter 100 is a lighting device installed in an elevator.

The receiver 200 first determines whether or not the current position of the receiver 200 is on the escalator (step S391). The escalator may be a slope escalator or a horizontal escalator.

Here, if it is determined that it is on the escalator (Y in step S391), the receiver 200 estimates the movement of the receiver 200 (step S392). This movement is a movement of the receiver 200 based on a fixed floor or wall other than the escalator. That is, the receiver 200 first acquires the direction and speed of the escalator movement from the server. The receiver 200 adds the movement of the escalator to the movement of the receiver 200 on the escalator recognized by the inter-frame image processing such as SLAM (SimultaneousaneLocalization and Mapping). Estimate movement.

On the other hand, when it is determined that the receiver 200 is not on the escalator (N in Step S391), it is determined whether or not the current position of the receiver 200 is in the elevator (Step S393). Here, if the receiver 200 determines that it is not in the elevator (N in step S393), the process is terminated. On the other hand, when the receiver 200 determines that it is in the elevator (Y in step S393), the floor where the elevator (specifically, the elevator car) is currently located is determined by a visible light signal, a radio signal, or other means. It is determined whether it can be identified (step S394).

Here, if it is determined that the floor cannot be specified (N in step S394), the receiver 200 displays the floor on which the user is scheduled to get off the elevator (step S395). Furthermore, the receiver 200 recognizes whether or not the receiver 200 has exited the elevator when the user gets out of the elevator, and determines the floor where the receiver 200 is currently located by a visible light signal, a radio wave signal, or other means. recognize. Then, if the recognized floor is different from the planned floor, the receiver 200 notifies the user that the floor on which the user has descended is incorrect (step S396).

Further, when the receiver 200 determines in step S394 that the floor on which the elevator is currently located can be specified (Y in step S394), the receiver 200 is placed on the floor on which the user is to exit the elevator, that is, the target floor. It is determined whether or not there is (step S397). If it is determined that the receiver 200 is on the target floor (Y in step S397), the receiver 200 displays a message that prompts the user to get off (step S399). Alternatively, the receiver 200 displays an advertisement related to the target floor. Further, the receiver 200 may display a warning message when the user is not going to get off.

On the other hand, when the receiver 200 determines that the receiver 200 is not on the target floor (N in step S397), the receiver 200 displays a message for alerting the user not to get off (step S398). Alternatively, the receiver 200 may display an advertisement. The receiver 200 may display a warning message when the user is about to get off.

FIG. 222 is a flowchart illustrating an example of processing performed by the receiver 200.

222, the receiver 200 uses a visible light signal and normal exposure image information (that is, a normal captured image) in combination.

For example, the receiver 200 configured as a wearable device such as a smartphone or smart glass acquires the image A (that is, the above-described decoding image) by imaging with an exposure time shorter than the normal exposure time (step S631). Then, the receiver 200 receives the visible light signal by decoding the image A (step S632). For example, the receiver 200 identifies the current position of the receiver 200 based on the received visible light signal, and starts navigation to a predetermined position.

Next, the receiver 200 acquires an image B (that is, the above-described normal captured image) by imaging with an exposure time longer than the above-described short exposure time (for example, the exposure time based on the automatic exposure setting) (step S633). Here, the image A is not suitable for object detection or feature amount extraction. Therefore, the receiver 200 alternately repeats acquisition of the image A by imaging with the above short exposure time and acquisition of the image B by imaging with the above long exposure time by a predetermined number of times. Thereby, the receiver 200 performs image processing such as the above-described object detection or feature amount extraction using the obtained plurality of images B (step S634). For example, the receiver 200 corrects the position of the receiver 200 by detecting a specific object from the image B. Further, for example, the receiver 200 extracts feature points from each of two or more images B, and identifies how the same feature points have moved between images. As a result, the receiver 200 can recognize the distance and direction of movement of the receiver 200 between the respective imaging points of the two or more images B, and can correct the current position of the receiver 200. .

FIG. 223 is a diagram illustrating an example of a screen displayed on the display of the receiver 200.

When the navigation application is activated, the receiver 200 displays the logo mark of the transmitter 100 as shown in FIG. 223, for example. The logo mark is, for example, a logo mark described as “AR Navi”. Then, the receiver 200 may guide the user to take an image of the logo mark. The transmitter 100 is configured as, for example, digital signage, and displays the logo mark while changing the luminance in order to transmit a visible light signal. Alternatively, the transmitter 100 is configured as a projector, for example, and projects the above logo mark on the floor or wall while changing the luminance in order to transmit visible light. The receiver 200 receives the visible light signal from the transmitter 100 by imaging the logo mark in the visible light communication mode. Note that the receiver 200 may display a picture of a nearby lighting device or landmark configured as the transmitter 100 instead of the logo mark.

Also, the receiver 200 may display a call center telephone number for when the user is in trouble. At this time, the receiver 200 may notify the call center server of the user's language and the estimated self-location. The language used by the user may be registered in advance in the receiver 200, for example, or may be set by an operation by the user. As a result, the call center can quickly respond to a user when a call is received from the user of the receiver 200. For example, a call center can guide a user to a destination by telephone.

Further, the receiver 200 may correct the self-position based on the form of the landmark registered in advance, the size of the landmark, and the position of the landmark on the map. That is, when a normal captured image is acquired, the receiver 200 detects an image area where a landmark is projected from the normal captured image. Then, the receiver 200 performs self-position estimation based on the shape, size, and position of the image area in the normal captured image, and the size of the landmark and the position on the map.

Also, the receiver 200 may recognize or detect a landmark on the ceiling or behind using the in-camera. In addition, the receiver 200 may use, for example, only an area that is greater than or equal to a predetermined angle (or less than or equal to a predetermined angle) with respect to the horizontal direction, among images obtained by imaging with a camera. For example, if the transmitter 100 or landmark is often arranged on the ceiling side, the receiver 200 uses only the area of the camera image in which the subject above the horizontal direction is projected. The receiver 200 detects the bright line pattern area or the landmark image area only from the area. Thereby, the processing amount of the receiver 200 can be reduced.

Further, as shown in the example of FIG. 212, the receiver 200 superimposes the direction indication image 431 on the normal captured image when performing AR navigation, but may further superimpose a character.

FIG. 224 is a diagram illustrating a display example of characters by the receiver 200.

When the receiver 200 receives a visible light signal from the transmitter 100 configured as digital signage, for example, the receiver 200 acquires the character 432 corresponding to the visible light signal from the server, for example, as the AR image. Then, as shown in FIG. 224, the receiver 200 displays not only the direction indication image 431 but also the character 432 superimposed on the normal captured image. For example, the character 432 is an advertising character for a drinking water manufacturing and sales company, and is displayed as an image of a can in which the drinking water is enclosed. The character 432 is an advertisement character for drinking water sold on the route to the user's destination. Such a character 432 may be displayed in a direction or position indicated by the direction indication image 431 so as to lead the user. The advertisement by such a character may be realized by an affiliate.

Also, the character may be in the form of an animal or a person. In this case, the receiver 200 may superimpose a character that walks on the direction indication image on the normal captured image. A plurality of characters may be superimposed on the normal captured image. Furthermore, the receiver 200 may superimpose a moving image for advertisement on a normal captured image as a commercial instead of or together with the character.

Further, the receiver 200 may change the size and display time of the character for advertisement according to the advertisement fee paid for the advertisement of the company. Further, when a plurality of advertising characters are displayed, the receiver 200 may determine the display order of the characters in the depth direction according to the advertising fee paid for each character. Further, when the receiver 200 enters the store where the displayed character product is sold, the receiver 200 may charge the store by electronic payment.

In addition, when the receiver 200 receives another visible light signal from another digital signage while displaying the character 432, the receiver 200 changes the displayed character 432 according to the other visible light signal. You may change to a different character.

The receiver 200 may superimpose a moving image that is a commercial of the company on a normal captured image. The advertiser may be charged according to the display time and the number of times of the commercial moving image or advertisement. The receiver 200 may display the language of the user as the language of the commercial, and a sentence or a voice link for the user to inform the store person of the intention to purchase the commercial product is displayed in the language of the store person. It may be displayed. The receiver 200 may display the price of the product in the user's currency.

FIG. 225 is a diagram illustrating another example of a screen displayed on the display of the receiver 200.

For example, as shown in FIG. 225 (a), the receiver 200 displays a message for notifying that the bookstore with the store name “XYZ” is ahead in English, which is the user's language. Such a message may be displayed as a moving image commercial as described above. Then, when the receiver 200 enters the bookstore, for example, as shown in FIG. 225 (b), a sentence to convey to the bookstore clerk, the language of the clerk (for example, Japanese), and the use of the user You may display with English which is a language.

Also, the receiver 200 may prompt the user to take a detour during the navigation. In this case, the receiver 200 may propose a detour according to the spare time. In the example of FIG. 212, the margin time is the difference between the departure time of the bus from the bus stop and the time when the user arrives at the bus stop.

Further, the receiver 200 may display advertisements of nearby stores. In this case, the receiver 200 may display an advertisement for a store next to it or a store along the way to go. Further, the receiver 200 may calculate the timing for starting the playback of the moving image so that the receiver 200 exists next to the store corresponding to the commercial when the playback of the moving image of the commercial is completed. Further, the receiver 200 may stop the advertisement display of the store that has passed.

Further, when the user detours to the store or the like, the store is provided with a lighting device or the like as the transmitter 100 for obtaining the starting point so that the receiver 200 can return to the guidance to the original destination. Also good. Alternatively, the receiver 200 may display a button labeled “Restart from the front of the XYZ store”. Further, the receiver 200 may apply the discount price only to the person who visits the store after seeing the commercial, or may display a coupon. In addition, the receiver 200 may display the barcode by an application and perform electronic settlement in order to pay for the purchase of the product.

Further, the server may perform a flow line analysis based on the result of navigation by the receiver 200 of each user.

Further, when the camera is not used for navigation, the receiver 200 may switch the self-position estimation method to PDR (Pedestrian Dead Reckoning) using an acceleration sensor or the like. For example, when the navigation application is turned off, or when the receiver 200 enters the user's pocket or the like and the image obtained by the camera becomes dark, the self-position estimation method is switched to the PDR. The receiver 200 may use radio waves (Bluetooth or Wi-Fi) or sound waves for self-position estimation.

In addition, if the user tries to go in the wrong direction, the receiver 200 may notify the user with vibration or sound. For example, the receiver 200 may change the type of vibration or sound depending on whether the user tries to go in the correct direction or an incorrect direction at the intersection. Note that the receiver 200 may perform the above-described vibration or sound notification when the receiver 200 is directed in the wrong direction or the correct direction without moving. Thereby, usability can be improved even for visually impaired people. The correct direction is a direction toward the destination along the searched route, and the wrong direction is a direction other than the correct direction.

In addition, the receiver 200 is configured as a smartphone in the above-described example, but may be configured as a smart watch or a smart glass. When the receiver 200 is configured as a smart glass, navigation using the camera by the receiver 200 can be prevented from being interrupted by an application not related to the navigation.

Further, the receiver 200 may end the navigation when a certain time has elapsed after starting the navigation. The certain time may be changed according to the distance to the destination. Alternatively, the receiver 200 may end the navigation when entering a place where GPS data reaches. Alternatively, the receiver 200 may end the navigation when the GPS location is a certain distance away. The receiver 200 may display the estimated arrival time or the remaining distance until arrival. Furthermore, in the example of FIG. 212, the receiver 200 may display the time at which the bus departs at the destination bus stop.

In addition, the receiver 200 may alert the user at a position such as a staircase or an intersection, and may guide the user to an elevator or the like instead of the staircase according to the user's desire or health condition. For example, if the user is elderly (for example, in his 80s), the receiver 200 may guide the elevator away from the stairs. If the receiver 200 determines that the user has a large baggage, the receiver 200 may guide the elevator away from the stairs. For example, the receiver 200 may determine whether the user's walking speed is slower than the normal speed based on the output from the acceleration sensor, and may determine that the user has a large baggage when it is slow. . Alternatively, the receiver 200 may determine whether or not the user's stride is shorter than the normal stride based on the output from the acceleration sensor, and may determine that the user has a large baggage when it is short. Furthermore, if the user is a woman, the receiver 200 may guide the user to a safe course. The safe course is shown in the map data.

Further, the receiver 200 may recognize an obstacle such as a person or a car around the receiver 200 based on an image obtained by imaging with a camera. Then, when the user is likely to collide with the obstacle, the receiver 200 may prompt the user to avoid the obstacle. For example, the receiver 200 may prompt the user to stop or avoid an obstacle by making a sound.

Further, when performing the navigation, the receiver 200 may correct the estimated arrival time from the travel time of other users in the past. At this time, the receiver 200 may correct based on the age and sex of the user. For example, the receiver 200 corrects the estimated arrival time before if the user is in his 20s, and corrects the estimated arrival time later if the user is in his 80s.

In addition, the receiver 200 may change the destination depending on the user even when the digital signage that is the same transmitter 100 is imaged at the start of navigation. For example, the receiver 200 may vary the position of the toilet serving as the destination according to the gender of the user, or may vary the counter of entry or return as the destination according to the nationality of the user. Alternatively, the receiver 200 may change the destination of the destination train or airplane according to the user's ticket. In addition, the receiver 200 may change the seat of the show that is the destination according to the user's ticket. In addition, the receiver 200 may vary the prayer space serving as a destination depending on the sect of the user.

Further, when starting the navigation, the receiver 200 may display a dialog such as “Do you want to start guiding to XYZ? Yes / No” without starting the navigation suddenly. Further, the receiver 200 may inquire of the user where the guidance destination is (such as an entrance gate, a lounge, or a store).

In addition, the receiver 200 may suppress notifications or incoming calls from other applications when performing navigation. Thereby, it can suppress that navigation is interrupted.

Also, the receiver 200 may guide the user to a meeting place as a destination.

FIG. 226 is a diagram showing a system configuration for performing navigation to a meeting place.

For example, a user having the receiver 200a and a user having the receiver 200b gather at a meeting place. Note that each of the receiver 200a and the receiver 200b has the function of the receiver 200 described above.

When waiting as described above is performed, the receiver 200a, as shown in FIG. 226 (a), the position obtained by the self-position estimation, the number of the receiver 200a, and the partner of the waiting (that is, the receiver 200b). ) Is sent to the server 300. The number may be a telephone number or any identification number that can identify the receiver. Moreover, information other than a number may be sufficient.

When the server 300 receives the above information from the receiver 200a, the server 300 transmits the position and number of the receiver 200a to the receiver 200b as shown in FIG. 226 (b). Then, the server 300 inquires of the receiver 200b whether or not to accept the meeting invitation from the receiver 200a. Here, the user of the receiver 200b accepts the invitation by operating the receiver 200b. That is, the receiver 200b informs the server 300 that the waiting is accepted as shown in FIG. 226 (c). Then, as illustrated in FIG. 226 (d), the receiver 200b notifies the server 300 of the position of the receiver 200b obtained by the self-position estimation.

As a result, the server 300 identifies the positions of the receiver 200a and the receiver 200b. Then, the server 300 sets an intermediate point of each position as a meeting place (that is, a destination), and notifies the receiver 200a and the receiver 200b of a route to the meeting place. Thereby, the AR navigation to the meeting place is executed in each of the receiver 200a and the receiver 200b. In the above-described example, an intermediate point between the positions of the receiver 200a and the receiver 200b is set as the destination, but another place may be set as the destination. For example, a place where the movement time is the shortest among a plurality of places where the landmarks are installed may be set as the destination. The travel time is an estimated time until the receiver 200a and the receiver 200b reach the place.

This makes it possible to wait smoothly.

Here, when the receiver 200a arrives near the destination, an image for identifying the user having the receiver 200b may be superimposed on the normal captured image.

FIG. 227 is a diagram illustrating an example of a screen displayed on the display of the receiver 200a.

For example, the server 300 periodically transmits the position of the receiver 200b to the receiver 200a. The position of the receiver 200b is a position obtained by self-position estimation by the receiver 200b. Therefore, the receiver 200a knows the position of the receiver 200b on the map. Therefore, when the position of the receiver 200b is displayed in the normal captured image, the receiver 200a may superimpose an arrow 433 indicating the position on the normal captured image as illustrated in FIG. Note that, similarly to the receiver 200a, the receiver 200b may superimpose an arrow indicating the position of the receiver 200a on the normal captured image.

This makes it easy to find a meeting partner even if there are many people at the meeting place.

In the above example, the indicator such as the arrow 433 is used for waiting, but may be used for other than waiting. The user of the receiver 200b may notify the server 300 of this by operating the receiver 200b when there is something wrong with the destination, regardless of the waiting time. In this case, the server 300 may display the image shown in the example of FIG. 227 on the display of the receiver 200a owned by the call center staff. At this time, the server 300 may display a question mark instead of the arrow 433. Thereby, the staff of the call center can easily confirm that the user of the receiver 200b is in trouble.

Further, the receiver 200 may guide the inside of the concert hall.

FIG. 228 shows the inside of the concert hall.

The receiver 200 may acquire, for example, a map inside the concert hall shown in FIG. 228 and a route 434 from the entrance of the concert hall to the seat from the server. For example, the receiver 200 estimates a self-position by receiving a visible light signal from the transmitter 100 arranged at the entrance, and guides the user along the route 434 to the user's seat. Here, if there is a staircase inside the concert hall, the receiver 200 identifies how many steps the user has climbed or descended on the basis of the output of the acceleration sensor or the like, and is identified. The self position may be updated based on the number of steps.

In the above-described example, the receiver 200 performs self-position estimation based on the movement of the feature point when no visible light signal is received. However, the receiver 200 can detect the feature point from the normal captured image. When it becomes impossible, the output from the acceleration sensor may be used. Specifically, when the receiver 200 can detect a feature point from a normal captured image, the receiver 200 estimates the movement distance as described above, and the movement distance and output data from the moving acceleration sensor To learn the relationship. For this learning, for example, machine learning such as DNN (Deep Neural 例 え ば Network) may be used. When the feature point cannot be detected, the receiver 200 derives the moving distance using the learning result and output data from the moving acceleration sensor. Alternatively, when it becomes impossible to detect feature points, the receiver 200 assumes that the receiver 200 is moving at a constant speed at the immediately preceding moving speed, and derives a moving distance based on the assumption. Also good.

[First embodiment]
In the communication method, it is determined whether the inclination of the terminal is greater than a predetermined angle with respect to a plane parallel to the ground, and is determined to be smaller than the predetermined angle, and an object whose brightness changes is captured by the out-camera. In this case, the exposure time of the image sensor of the out-camera is set to a first exposure time, and the subject is imaged at the first exposure time using the image sensor, thereby obtaining a decoding image, When the first signal transmitted from the subject can be decoded from the decoding image, the first signal is decoded from the decoding image, the position specified by the first signal is acquired, and the decoding is performed. When the signal transmitted from the subject cannot be decoded from the image for use, the map is stored in the terminal, and the map information including the positions of a plurality of transmitters and the position of the terminal are used to determine the predetermined position from the terminal position. Range Identifying a location associated with a transmitter.

FIG. 229 is a flowchart showing an example of a communication method according to the first aspect of the present invention.

First, the terminal which is the receiver 200 determines whether or not the inclination of the terminal is larger than a predetermined angle with respect to a plane parallel to the ground (step SG21). The surface parallel to the ground may be a horizontal plane, for example. Specifically, the terminal determines whether the inclination is large by detecting the inclination of the terminal using output data from the acceleration sensor. This tilt is the tilt of the front or back of the terminal.

Here, when the terminal determines that the inclination is larger than a predetermined angle and images the subject whose luminance changes with the out camera (Yes in step SG21), the exposure time of the image sensor of the out camera is set to the first time. (Step SG22). Then, the terminal captures the subject with the first exposure time using the image sensor, thereby acquiring a decoding image (step SG23).

Since the decoding image acquired here is an image obtained when the inclination of the terminal is larger than a predetermined angle with respect to a plane parallel to the ground, an image obtained by imaging a subject on the ground side is not. Therefore, in capturing the decoding image, a transmitter 100 capable of transmitting a visible light signal such as a lighting device arranged on the ceiling side or a digital signage installed on a wall surface is imaged as a subject. There is a high possibility. In other words, in capturing a decoding image, there is a low possibility that reflected light from the transmitter 100 is captured as a subject. Therefore, it is possible to appropriately acquire a decoding image that has a high possibility that the transmitter 100 is captured as a subject. That is, as shown in step S353 of FIGS. 214 and 215, it is possible to appropriately determine whether the reflected light from the floor or the wall is imaged or the direct light from the transmitter 100 is imaged. it can.

Next, the terminal determines whether or not the first signal transmitted from the subject can be decoded from the decoding image (step SG24). Here, when the first signal can be decoded (Yes in Step SG24), the terminal decodes the first signal from the decoding image (Step SG25), and acquires the position specified by the first signal. (Step SG26). On the other hand, when the signal transmitted from the subject cannot be decoded from the decoding image (No in step SG24), the terminal stores the map information including the positions of a plurality of transmitters and the position of the terminal. Are used to identify a position related to a transmitter within a predetermined range from the position of the terminal (step SG27).

As a result, as shown in steps S344 to S348 in FIG. 213, for example, the position of the transmitter that is the subject can be received even if the first signal that is a visible light signal can be received or not. Can be specified. As a result, the current self-position of the terminal can be estimated appropriately.

[Second embodiment]
In the communication method of the second aspect, in the first communication method, the first exposure time is set such that bright lines corresponding to a plurality of exposure lines included in the image sensor appear in the decoding image. The

Thereby, the first signal can be appropriately decoded from the decoding image.

[Third Aspect]
In the communication method according to the third aspect, in the communication method according to the first aspect, the subject is reflected light in which light from the first transmitter that transmits a signal due to a change in luminance is reflected on the floor surface.

Thereby, even when the decoding image is obtained by imaging the reflected light and the first signal cannot be decoded from the decoding image, the position of the first transmitter can be specified.

[Fourth aspect]
According to a communication method of a fourth aspect, in the communication method of the first aspect, an exposure time of an image sensor of the out-camera is set to a second exposure time longer than the first exposure time, and the first By capturing images with an exposure time of 2, a plurality of normal images are acquired, a plurality of spatial feature amounts are calculated from the plurality of normal images, and a position of the terminal is calculated using the plurality of spatial feature amounts. .

Note that the normal image is the above-described normal photographed image.

As a result, as shown in FIGS. 212 (c) and (d), even if there is a terminal in an underground mall where GPS data does not reach, the current self-position of the terminal is appropriately determined from a plurality of normal images. Can be estimated. The spatial feature amount may be a feature point.

[Fifth aspect]
In the communication method according to the fifth aspect, in the communication method according to the fourth aspect, a decoding image is obtained by capturing an image of the second transmitter at the first exposure time, and the decoding image is acquired from the decoding image. Decoding the second signal transmitted by the second transmitter, obtaining the position specified by the second signal, and setting the position specified by the second signal as the movement start position in the map information; The position of the terminal is specified by calculating the amount of movement of the terminal using the plurality of spatial feature amounts.

Thereby, as shown in FIG. 212 (a), the position of the terminal is specified based on the movement amount from the starting point which is the movement start position, so that more accurate self-position estimation can be performed.

The communication method of the present invention has an effect that communication between various devices can be performed, and can be used for a display device such as a smartphone, a glass, or a tablet.

G20 communication device G21 determination unit G22 first acquisition unit G23 second acquisition unit

Claims (20)

1. A communication method using a terminal equipped with an image sensor,
   Determining whether the terminal is capable of visible light communication;
   When it is determined that the terminal can perform visible light communication, the image sensor captures a subject whose luminance changes, thereby obtaining a decoding image, and from the striped pattern appearing in the decoding image, Obtaining first identification information transmitted by the subject;
   In the determination of visible light communication, when it is determined that the terminal cannot perform visible light communication, the image sensor acquires a captured image by capturing the subject, and edge detection of the captured image To extract at least one contour, identify a predetermined specific region from the at least one contour, and obtain second identification information transmitted by the subject from a line pattern of the specific region ,
   Communication method.
2. In specifying the specific area,
   An area having a quadrangular outline of a predetermined size or more, or an area having a rounded quadrangular outline of a predetermined size or more is specified as the specific area.
   The communication method according to claim 1.
3. In the determination of the visible light communication,
   When it is determined that the terminal is a terminal that can change the exposure time to a predetermined value or less, it is determined that visible light communication can be performed,
   When it is determined that the terminal is a terminal that cannot change the exposure time below the predetermined value, it is determined that visible light communication is not possible.
   The communication method according to claim 1.
4. In the determination of the visible light communication, when the terminal determines that it is possible to perform visible light communication, when imaging the subject, the exposure time of the image sensor is set to a first exposure time, and the first By capturing the subject with an exposure time of 1, the decoding image is obtained,
   In the determination of the visible light communication, when the terminal determines that the visible light communication is not possible, when imaging the subject, the exposure time of the image sensor is set to a second exposure time, By capturing the subject with a second exposure time, the captured image is acquired,
   The first exposure time is shorter than the second exposure time;
   The communication method according to any one of claims 1 to 3.
5. The subject has a rectangular shape when viewed from the image sensor, and the central area of the subject changes in luminance, so that the first identification information is transmitted, and a barcode-like line pattern is arranged around the subject. Has been
   In the determination of the visible light communication, when the terminal determines that the visible light communication is possible, the image sensor is configured by a plurality of bright lines corresponding to a plurality of exposure lines of the image sensor. Obtaining the decoding image including the bright line pattern, obtaining the first identification information by decoding the bright line pattern,
   In the determination of the visible light communication, when the terminal determines that the visible light communication cannot be performed, the second identification information is acquired from the line pattern of the captured image when capturing the subject. ,
   The communication method according to claim 4.
6. The first identification information obtained from the decoding image and the second identification information obtained from the line pattern are the same information.
   The communication method according to claim 5.
7. In the determination of the visible light communication, when it is determined that the terminal can perform visible light communication, the first moving image associated with the first identification information is displayed,
   When an operation of sliding the first moving image is received, a second moving image associated with the first identification information is displayed next to the first moving image.
   The communication method according to claim 1.
8. In the display of the second moving image,
   When an operation of sliding the first moving image in the horizontal direction is received, the second moving image is displayed,
   The communication method according to claim 7, wherein a still image associated with the first identification information is displayed when an operation of sliding the first moving image in the vertical direction is received.
9. In each of the first moving image and the second moving image, the objects in the picture displayed first are in the same position.
   The communication method according to claim 8.
10. When the first identification information is acquired again by imaging by the image sensor, the next moving image associated with the first identification information is displayed next to the displayed moving image.
    The communication method according to claim 7 or 8.
11. In each of the displayed moving image and the next moving image, the object in the picture displayed first is in the same position.
    The communication method according to claim 10.
12. At least one of the first moving image and the second moving image is formed such that the closer the position in the moving image is to the end of the moving image, the higher the transparency at that position is. ing,
    The communication method according to claim 11.
13. Displaying an image outside an area in which at least one of the first moving image and the second moving image is displayed;
    The communication method according to claim 12.
14. A normal captured image is obtained by imaging with the first exposure time by the image sensor,
    The image for decoding including a bright line pattern region, which is a region composed of a plurality of bright line patterns, is acquired by imaging with a second exposure time shorter than the first exposure time, and the first image is obtained by decoding the decoding image. 1 identification information,
    In displaying the moving image of at least one of the first moving image and the second moving image, a reference region located at the same position as the bright line pattern region in the decoding image is specified from the normal captured image. And
    Recognizing a region in which the moving image is superimposed in the normal captured image as a target region based on the reference region, and superimposing the moving image on the target region;
    The communication method according to claim 7.
15. In displaying the moving image of at least one of the first moving image and the second moving image, the upper, lower, left or right region of the reference region in the normal captured image is recognized as the target region. To
    The communication method according to claim 14.
16. In displaying the moving image of at least one of the first moving image and the second moving image, the size of the moving image is increased as the size of the bright line pattern region is increased.
    The communication method according to claim 14.
17. A communication device using a terminal equipped with an image sensor,
    A determination unit that determines whether the terminal can perform visible light communication; and
    In the determination unit, when it is determined that the terminal can perform visible light communication, the image sensor captures a subject whose luminance changes, thereby obtaining a decoding image, and the decoding image is obtained as the decoding image. A first acquisition unit for acquiring first identification information transmitted by the subject from the appearing stripe pattern;
    When the determination unit determines that the terminal is not capable of visible light communication, the image sensor acquires a captured image by capturing the subject and performs edge detection of the captured image. Thus, at least one contour is extracted, a predetermined specific area is specified from the at least one outline, and second identification information transmitted by the subject is acquired from a line pattern of the specific area. An acquisition unit,
    Communication device.
18. A lighting board;
    A light source that emits light from the back side of the illumination plate;
    A microcontroller for changing the luminance of the light source,
    The microcontroller transmits the first identification information from the light source via the illumination plate by changing the brightness of the light source,
    A barcode-like line pattern is arranged around the front side of the lighting plate, and second identification information is encoded in the line pattern,
    The first identification information and the second identification information are the same information.
    Transmitter.
19. The lighting plate has a rectangular shape,
    The transmitter according to claim 18.
20. A program for executing a communication method using a terminal equipped with an image sensor,
    Determining whether the terminal is capable of visible light communication;
    When it is determined that the terminal can perform visible light communication, the image sensor captures a subject whose luminance changes, thereby obtaining a decoding image, and the subject from the striped pattern appearing in the decoding image Obtains the first identification information transmitted by
    In the determination of visible light communication, when it is determined that the terminal cannot perform visible light communication, the image sensor acquires a captured image by capturing the subject, and edge detection of the captured image To extract at least one contour, identify a predetermined specific region from the at least one contour, and obtain second identification information transmitted by the subject from a line pattern of the specific region ,
    A program that causes a computer to execute.

Images