JP7023239B2 - Transmission method, transmitter, and program - Google Patents

Description

The present invention relates to a method for transmitting a visible light signal, a transmission device, a program, and the like.

In recent home networks, in addition to the cooperation of AV home appliances by IP (Internet Protocol) connection via Ethernet (registered trademark) and wireless LAN (Local Area Network), management of power consumption in response to environmental problems and from outside the home With the home energy management system (HEMS) that has functions such as power ON / OFF, the introduction of a home appliance cooperation function that connects various home appliances to a network is progressing. However, there are some home appliances that do not have sufficient computing power to have a communication function, and some home appliances that are difficult to install a communication function in terms of cost.

In order to solve such a problem, in Patent Document 1, in an optical space transmission device that transmits information to free space using light, limited transmission is performed by performing communication using a plurality of monochromatic light sources of illumination light. Among the devices, the technology for efficiently realizing communication between devices is described.

Japanese Unexamined Patent Publication No. 2002-290335

However, the conventional method is limited to the case where the applied device has a three-color light source such as lighting.

The present invention solves such a problem and provides a transmission method that enables communication between various devices including devices other than lighting having a three-color light source.

The transmission method according to one embodiment of the present invention is a transmission method for transmitting a signal by changing the brightness of a light source, that is, a reception step that accepts a dimming degree designated for the light source as a designated dimming degree, and the designated dimming degree. When is equal to or less than the first value, the light source is made to emit light at the designated dimming degree, and the signal encoded in the first mode is transmitted by the luminance change, and the designated dimming degree is the first. When it is larger than the value of, the specified dimming degree includes the transmission step of transmitting the signal encoded in the second mode by the luminance change while causing the light source to emit light at the designated dimming degree. When the value is larger than the first value and equal to or less than the second value, the value of the peak current of the light source for transmitting the signal encoded in the second mode by the luminance change is the specified dimming degree. Is smaller than the value of the peak current of the light source for transmitting the signal encoded in the first mode by the luminance change when is the first value.

It should be noted that these comprehensive or specific embodiments may be realized in a recording medium such as a system, method, integrated circuit, computer program or computer-readable CD-ROM, and the system, method, integrated circuit, computer program. And may be realized by any combination of recording media. Further, a computer program for executing the method according to the embodiment may be stored in a recording medium of the server, and may be realized in a mode of distribution from the server to the terminal in response to a request of the terminal.

According to the present invention, it is possible to realize a transmission method that enables communication between devices including devices other than lighting having a three-color light source.

FIG. 1 is a diagram showing an example of a method of observing the brightness of a light emitting unit according to the first embodiment. FIG. 2 is a diagram showing an example of a method of observing the brightness of the light emitting portion in the first embodiment. FIG. 3 is a diagram showing an example of a method of observing the brightness of the light emitting portion in the first embodiment. FIG. 4 is a diagram showing an example of a method of observing the brightness of the light emitting portion in the first embodiment. FIG. 5A is a diagram showing an example of a method of observing the brightness of the light emitting portion according to the first embodiment. FIG. 5B is a diagram showing an example of a method of observing the brightness of the light emitting portion in the first embodiment. FIG. 5C is a diagram showing an example of a method of observing the brightness of the light emitting portion in the first embodiment. FIG. 5D is a diagram showing an example of a method of observing the brightness of the light emitting portion according to the first embodiment. FIG. 5E is a diagram showing an example of a method of observing the brightness of the light emitting portion according to the first embodiment. FIG. 5F is a diagram showing an example of a method of observing the brightness of the light emitting portion in the first embodiment. FIG. 5G is a diagram showing an example of a method of observing the brightness of the light emitting portion in the first embodiment. FIG. 5H is a diagram showing an example of a method of observing the brightness of the light emitting portion in the first embodiment. FIG. 6A is a flowchart of the information communication method according to the first embodiment. FIG. 6B is a block diagram of the information communication device according to the first embodiment. FIG. 7 is a diagram showing an example of the photographing operation of the receiver in the second embodiment. FIG. 8 is a diagram showing another example of the photographing operation of the receiver in the second embodiment. FIG. 9 is a diagram showing another example of the photographing operation of the receiver in the second embodiment. FIG. 10 is a diagram showing an example of the display operation of the receiver in the second embodiment. FIG. 11 is a diagram showing an example of the display operation of the receiver in the second embodiment. FIG. 12 is a diagram showing an example of the operation of the receiver according to the second embodiment. FIG. 13 is a diagram showing another example of the operation of the receiver in the second embodiment. FIG. 14 is a diagram showing another example of the operation of the receiver in the second embodiment. FIG. 15 is a diagram showing another example of the operation of the receiver in the second embodiment. FIG. 16 is a diagram showing another example of the operation of the receiver in the second embodiment. FIG. 17 is a diagram showing another example of the operation of the receiver in the second embodiment. FIG. 18 is a diagram showing an example of the operation of the receiver, the transmitter, and the server in the second embodiment. FIG. 19 is a diagram showing another example of the operation of the receiver in the second embodiment. FIG. 20 is a diagram showing another example of the operation of the receiver in the second embodiment. FIG. 21 is a diagram showing another example of the operation of the receiver in the second embodiment. FIG. 22 is a diagram showing an example of the operation of the transmitter according to the second embodiment. FIG. 23 is a diagram showing another example of the operation of the transmitter according to the second embodiment. FIG. 24 is a diagram showing an application example of the receiver according to the second embodiment. FIG. 25 is a diagram showing another example of the operation of the receiver in the second embodiment. FIG. 26 is a diagram showing an example of processing operations of the receiver, the transmitter, and the server in the third embodiment. FIG. 27 is a diagram showing an example of the operation of the transmitter and the receiver in the third embodiment. FIG. 28 is a diagram showing an example of the operation of the transmitter, receiver, and server in the third embodiment. FIG. 29 is a diagram showing an example of the operation of the transmitter and the receiver in the third embodiment. FIG. 30 is a diagram showing an example of the operation of the transmitter and the receiver in the fourth embodiment. FIG. 31 is a diagram showing an example of the operation of the transmitter and the receiver in the fourth embodiment. FIG. 32 is a diagram showing an example of the operation of the transmitter and the receiver in the fourth embodiment. FIG. 33 is a diagram showing an example of the operation of the transmitter and the receiver in the fourth embodiment. FIG. 34 is a diagram showing an example of the operation of the transmitter and the receiver in the fourth embodiment. FIG. 35 is a diagram showing an example of the operation of the transmitter and the receiver in the fourth embodiment. FIG. 36 is a diagram showing an example of the operation of the transmitter and the receiver in the fourth embodiment. FIG. 37 is a diagram for explaining the notification of visible light communication to humans in the fifth embodiment. FIG. 38 is a diagram for explaining an application example to the route guidance in the fifth embodiment. FIG. 39 is a diagram for explaining an application example to utilization log storage and analysis in the fifth embodiment. FIG. 40 is a diagram for explaining an application example to screen sharing according to the fifth embodiment. FIG. 41 is a diagram showing an application example of the information communication method according to the fifth embodiment. FIG. 42 is a diagram showing an application example of the transmitter and the receiver in the sixth embodiment. FIG. 43 is a diagram showing an application example of the transmitter and the receiver in the sixth embodiment. FIG. 44 is a diagram showing an example of the receiver according to the seventh embodiment. FIG. 45 is a diagram showing an example of the receiving system according to the seventh embodiment. FIG. 46 is a diagram showing an example of the signal transmission / reception system according to the seventh embodiment. FIG. 47 is a flowchart showing a reception method excluding interference in the seventh embodiment. FIG. 48 is a flowchart showing a method of estimating the orientation of the transmitter according to the seventh embodiment. FIG. 49 is a flowchart showing a method of starting reception in the seventh embodiment. FIG. 50 is a flowchart showing a method of generating an ID in combination with information of another medium according to the seventh embodiment. FIG. 51 is a flowchart showing a method of selecting a reception method by frequency separation in the seventh embodiment. FIG. 52 is a flowchart showing a signal receiving method when the exposure time is long in the seventh embodiment. FIG. 53 is a diagram showing an example of a dimming (adjusting the brightness) method of the transmitter according to the seventh embodiment. FIG. 54 is a diagram showing an example of a method for configuring the dimming function of the transmitter according to the seventh embodiment. FIG. 55 is a diagram for explaining EX zoom. FIG. 56 is a diagram showing an example of the signal receiving method according to the ninth embodiment. FIG. 57 is a diagram showing an example of the signal receiving method according to the ninth embodiment. FIG. 58 is a diagram showing an example of the signal receiving method according to the ninth embodiment. FIG. 59 is a diagram showing an example of the screen display method of the receiver according to the ninth embodiment. FIG. 60 is a diagram showing an example of the signal receiving method according to the ninth embodiment. FIG. 61 is a diagram showing an example of the signal receiving method according to the ninth embodiment. FIG. 62 is a flowchart showing an example of the signal receiving method according to the ninth embodiment. FIG. 63 is a diagram showing an example of the signal receiving method according to the ninth embodiment. FIG. 64 is a flowchart showing the processing of the receiving program according to the ninth embodiment. FIG. 65 is a block diagram of the receiving device according to the ninth embodiment. FIG. 66 is a diagram showing an example of a display of a receiver when a visible light signal is received. FIG. 67 is a diagram showing an example of a display of a receiver when a visible light signal is received. FIG. 68 is a diagram showing an example of displaying the acquired data image. FIG. 69 is a diagram showing an example of an operation when the acquired data is saved or destroyed. FIG. 70 is a diagram showing a display example when viewing the acquired data. FIG. 71 is a diagram showing an example of the transmitter according to the ninth embodiment. FIG. 72 is a diagram showing an example of the receiving method according to the ninth embodiment. FIG. 73 is a flowchart showing an example of the receiving method according to the tenth embodiment. FIG. 74 is a flowchart showing an example of the receiving method according to the tenth embodiment. FIG. 75 is a flowchart showing an example of the receiving method according to the tenth embodiment. FIG. 76 is a diagram for explaining a receiving method in which the receiver in the tenth embodiment uses an exposure time longer than the cycle of the modulation frequency (modulation cycle). FIG. 77 is a diagram for explaining a receiving method in which the receiver in the tenth embodiment uses an exposure time longer than the cycle of the modulation frequency (modulation cycle). FIG. 78 is a diagram showing an efficient number of divisions with respect to the size of transmission data in the tenth embodiment. FIG. 79A is a diagram showing an example of the setting method according to the tenth embodiment. FIG. 79B is a diagram showing another example of the setting method in the tenth embodiment. FIG. 80 is a flowchart showing the processing of the information processing program according to the tenth embodiment. FIG. 81 is a diagram for explaining an application example of the transmission / reception system according to the tenth embodiment. FIG. 82 is a flowchart showing the processing operation of the transmission / reception system according to the tenth embodiment. FIG. 83 is a diagram for explaining an application example of the transmission / reception system according to the tenth embodiment. FIG. 84 is a flowchart showing the processing operation of the transmission / reception system according to the tenth embodiment. FIG. 85 is a diagram for explaining an application example of the transmission / reception system according to the tenth embodiment. FIG. 86 is a flowchart showing the processing operation of the transmission / reception system according to the tenth embodiment. FIG. 87 is a diagram for explaining an application example of the transmitter according to the tenth embodiment. FIG. 88 is a diagram for explaining an application example of the transmission / reception system according to the eleventh embodiment. FIG. 89 is a diagram for explaining an application example of the transmission / reception system according to the eleventh embodiment. FIG. 90 is a diagram for explaining an application example of the transmission / reception system according to the eleventh embodiment. FIG. 91 is a diagram for explaining an application example of the transmission / reception system according to the eleventh embodiment. FIG. 92 is a diagram for explaining an application example of the transmission / reception system according to the eleventh embodiment. FIG. 93 is a diagram for explaining an application example of the transmission / reception system according to the eleventh embodiment. FIG. 94 is a diagram for explaining an application example of the transmission / reception system according to the eleventh embodiment. FIG. 95 is a diagram for explaining an application example of the transmission / reception system according to the eleventh embodiment. FIG. 96 is a diagram for explaining an application example of the transmission / reception system according to the eleventh embodiment. FIG. 97 is a diagram for explaining an application example of the transmission / reception system according to the eleventh embodiment. FIG. 98 is a diagram for explaining an application example of the transmission / reception system according to the eleventh embodiment. FIG. 99 is a diagram for explaining an application example of the transmission / reception system according to the eleventh embodiment. FIG. 100 is a diagram for explaining an application example of the transmission / reception system according to the eleventh embodiment. FIG. 101 is a diagram for explaining an application example of the transmission / reception system according to the eleventh embodiment. FIG. 102 is a diagram for explaining the operation of the receiver according to the twelfth embodiment. FIG. 103A is a diagram for explaining other operations of the receiver in the twelfth embodiment. FIG. 103B is a diagram showing an example of an indicator displayed by the output unit 1215 in the twelfth embodiment. FIG. 103C is a diagram showing an example of displaying AR in the twelfth embodiment. FIG. 104A is a diagram for explaining an example of the transmitter according to the twelfth embodiment. FIG. 104B is a diagram for explaining another example of the transmitter according to the twelfth embodiment. FIG. 105A is a diagram for explaining an example of synchronous transmission by a plurality of transmitters in the twelfth embodiment. FIG. 105B is a diagram for explaining another example of synchronous transmission by a plurality of transmitters in the twelfth embodiment. FIG. 106 is a diagram for explaining another example of synchronous transmission by a plurality of transmitters in the twelfth embodiment. FIG. 107 is a diagram for explaining signal processing of the transmitter according to the twelfth embodiment. FIG. 108 is a flowchart showing an example of the receiving method according to the twelfth embodiment. FIG. 109 is an explanatory diagram for explaining an example of the receiving method according to the twelfth embodiment. FIG. 110 is a flowchart showing another example of the receiving method according to the twelfth embodiment. FIG. 111 is a diagram showing an example of a transmission signal according to the thirteenth embodiment. FIG. 112 is a diagram showing another example of the transmission signal in the thirteenth embodiment. FIG. 113 is a diagram showing another example of the transmission signal in the thirteenth embodiment. FIG. 114A is a diagram for explaining the transmitter according to the fourteenth embodiment. FIG. 114B is a diagram showing the respective luminance changes of RGB in the fourteenth embodiment. FIG. 115 is a diagram showing the afterglow characteristics of the green fluorescent component and the red fluorescent component in the fourteenth embodiment. FIG. 116 is a diagram for explaining a problem newly generated in order to suppress the occurrence of a barcode reading error in the fourteenth embodiment. FIG. 117 is a diagram for explaining downsampling performed by the receiver in the fourteenth embodiment. FIG. 118 is a flowchart showing the processing operation of the receiver according to the fourteenth embodiment. FIG. 119 is a diagram showing a processing operation of the receiving device (imaging device) according to the fifteenth embodiment. FIG. 120 is a diagram showing a processing operation of the receiving device (imaging device) according to the fifteenth embodiment. FIG. 121 is a diagram showing a processing operation of the receiving device (imaging device) according to the fifteenth embodiment. FIG. 122 is a diagram showing a processing operation of the receiving device (imaging device) in the fifteenth embodiment. FIG. 123 is a diagram showing an example of the application according to the sixteenth embodiment. FIG. 124 is a diagram showing an example of the application according to the sixteenth embodiment. FIG. 125 is a diagram showing an example of a transmission signal and an example of a voice synchronization method in the 16th embodiment. FIG. 126 is a diagram showing an example of a transmission signal according to the 16th embodiment. FIG. 127 is a diagram showing an example of the processing flow of the receiver in the 16th embodiment. FIG. 128 is a diagram showing an example of the user interface of the receiver according to the sixteenth embodiment. FIG. 129 is a diagram showing an example of the processing flow of the receiver in the 16th embodiment. FIG. 130 is a diagram showing another example of the processing flow of the receiver according to the sixteenth embodiment. FIG. 131A is a diagram for explaining a specific method of synchronous reproduction in the sixteenth embodiment. FIG. 131B is a block diagram showing a configuration of a reproduction device (receiver) that performs synchronous reproduction according to the sixteenth embodiment. FIG. 131C is a flowchart showing the processing operation of the reproduction device (receiver) that performs synchronous reproduction in the sixteenth embodiment. FIG. 132 is a diagram for explaining advance preparation for synchronous reproduction in the sixteenth embodiment. FIG. 133 is a diagram showing an application example of the receiver according to the sixteenth embodiment. FIG. 134A is a front view of the receiver held in the holder in the sixteenth embodiment. FIG. 134B is a rear view of the receiver held in the holder in the 16th embodiment. FIG. 135 is a diagram for explaining the use case of the receiver held in the holder in the 16th embodiment. FIG. 136 is a flowchart showing the processing operation of the receiver held in the holder in the 16th embodiment. FIG. 137 is a diagram showing an example of an image displayed by the receiver in the 16th embodiment. FIG. 138 is a diagram showing another example of the holder in the sixteenth embodiment. FIG. 139A is a diagram showing an example of a visible light signal according to the 17th embodiment. FIG. 139B is a diagram showing an example of a visible light signal according to the 17th embodiment. FIG. 139C is a diagram showing an example of a visible light signal according to the 17th embodiment. FIG. 139D is a diagram showing an example of a visible light signal according to the 17th embodiment. FIG. 140 is a diagram showing a configuration of a visible light signal according to the 17th embodiment. FIG. 141 is a diagram showing an example of a emission line image obtained by imaging the receiver in the 17th embodiment. FIG. 142 is a diagram showing another example of the emission line image obtained by imaging the receiver in the 17th embodiment. FIG. 143 is a diagram showing another example of the emission line image obtained by imaging the receiver in the 17th embodiment. FIG. 144 is a diagram for explaining the adaptation of the receiver in the 17th embodiment to the camera system that performs HDR synthesis. FIG. 145 is a diagram for explaining the processing operation of the visible light communication system according to the seventeenth embodiment. FIG. 146A is a diagram showing an example of vehicle-to-vehicle communication using visible light in the 17th embodiment. FIG. 146B is a diagram showing another example of vehicle-to-vehicle communication using visible light in the seventeenth embodiment. FIG. 147 is a diagram showing an example of a method for determining the positions of a plurality of LEDs according to the seventeenth embodiment. FIG. 148 is a diagram showing an example of a emission line image obtained by imaging a vehicle in the 17th embodiment. FIG. 149 is a diagram showing an application example of the receiver and the transmitter in the 17th embodiment. Note that FIG. 149 is a view of the automobile from behind. FIG. 150 is a flowchart showing an example of processing operations of the receiver and the transmitter in the 17th embodiment. FIG. 151 is a diagram showing an application example of the receiver and the transmitter in the 17th embodiment. FIG. 152 is a flowchart showing an example of the processing operation of the receiver 7007a and the transmitter 7007b in the seventeenth embodiment. FIG. 153 is a diagram showing a configuration of a visible light communication system applied to the inside of a train according to the seventeenth embodiment. FIG. 154 is a diagram showing a configuration of a visible light communication system applied to a facility such as an amusement park in the 17th embodiment. FIG. 155 is a diagram showing an example of a visible light communication system including a playset and a smartphone according to the seventeenth embodiment. FIG. 156 is a diagram showing an example of a transmission signal according to the eighteenth embodiment. FIG. 157 is a diagram showing an example of a transmission signal according to the eighteenth embodiment. FIG. 158 is a diagram showing an example of a transmission signal according to the nineteenth embodiment. FIG. 159 is a diagram showing an example of a transmission signal according to the nineteenth embodiment. FIG. 160 is a diagram showing an example of a transmission signal according to the nineteenth embodiment. FIG. 161 is a diagram showing an example of a transmission signal according to the nineteenth embodiment. FIG. 162 is a diagram showing an example of a transmission signal according to the nineteenth embodiment. FIG. 163 is a diagram showing an example of a transmission signal according to the nineteenth embodiment. FIG. 164 is a diagram showing an example of a transmission / reception system according to the nineteenth embodiment. FIG. 165 is a flowchart showing an example of processing of the transmission / reception system according to the nineteenth embodiment. FIG. 166 is a flowchart showing the operation of the server according to the nineteenth embodiment. FIG. 167 is a flowchart showing an example of the operation of the receiver according to the nineteenth embodiment. FIG. 168 is a flowchart showing a calculation method of the progress status in the simplified mode in the nineteenth embodiment. FIG. 169 is a flowchart showing a calculation method of the progress status in the maximum likelihood estimation mode in the nineteenth embodiment. FIG. 170 is a flowchart showing a display method in which the progress status in the nineteenth embodiment does not decrease. FIG. 171 is a flowchart showing a method of displaying a progress status when there are a plurality of packet lengths in the nineteenth embodiment. FIG. 172 is a diagram showing an example of the operating state of the receiver according to the nineteenth embodiment. FIG. 173 is a diagram showing an example of a transmission signal according to the nineteenth embodiment. FIG. 174 is a diagram showing an example of a transmission signal according to the nineteenth embodiment. FIG. 175 is a diagram showing an example of a transmission signal according to the nineteenth embodiment. FIG. 176 is a block diagram showing an example of the transmitter according to the nineteenth embodiment. FIG. 177 is a diagram showing a timing chart when the LED display according to the nineteenth embodiment is driven by the optical ID modulation signal of the present invention. FIG. 178 is a diagram showing a timing chart when the LED display according to the nineteenth embodiment is driven by the optical ID modulation signal of the present invention. FIG. 179 is a diagram showing a timing chart when the LED display according to the nineteenth embodiment is driven by the optical ID modulation signal of the present invention. FIG. 180A is a flowchart showing a transmission method according to one aspect of the present invention. FIG. 180B is a block diagram showing a functional configuration of a transmission device according to an aspect of the present invention. FIG. 181 is a diagram showing an example of a transmission signal according to the nineteenth embodiment. FIG. 182 is a diagram showing an example of a transmission signal according to the nineteenth embodiment. FIG. 183 is a diagram showing an example of a transmission signal according to the nineteenth embodiment. FIG. 184 is a diagram showing an example of a transmission signal according to the nineteenth embodiment. FIG. 185 is a diagram showing an example of a transmission signal according to the nineteenth embodiment. FIG. 186 is a diagram showing an example of a transmission signal according to the nineteenth embodiment. FIG. 187 is a diagram showing an example of the configuration of a visible light signal according to the twentieth embodiment. FIG. 188 is a diagram showing an example of a detailed configuration of a visible light signal according to the twentieth embodiment. FIG. 189A is a diagram showing another example of the visible light signal according to the twentieth embodiment. FIG. 189B is a diagram showing another example of the visible light signal according to the twentieth embodiment. FIG. 189C is a diagram showing the signal length of the visible light signal according to the twentieth embodiment. FIG. 190 is a diagram showing a comparison result of the luminance values of the visible light signal in the 20th embodiment and the visible light signal of the standard IEC. FIG. 191 is a diagram showing a comparison result of the number of received packets and the reliability with respect to the angle of view between the visible light signal in the 20th embodiment and the visible light signal of the standard IEC. FIG. 192 is a diagram showing a comparison result of the number of received packets and the reliability with respect to noise between the visible light signal in the 20th embodiment and the visible light signal of the standard IEC. FIG. 193 is a diagram showing a comparison result of the number of received packets and the reliability with respect to the clock error on the receiving side between the visible light signal in the 20th embodiment and the visible light signal of the standard IEC. FIG. 194 is a diagram showing a configuration of a signal to be transmitted according to the twentieth embodiment. FIG. 195A is a diagram showing a method of receiving a visible light signal according to the twentieth embodiment. FIG. 195B is a diagram showing the rearrangement of visible light signals in the 20th embodiment. FIG. 196 is a diagram showing another example of the visible light signal according to the twentieth embodiment. FIG. 197 is a diagram showing another example of the detailed configuration of the visible light signal according to the twentieth embodiment. FIG. 198 is a diagram showing another example of the detailed configuration of the visible light signal according to the twentieth embodiment. FIG. 199 is a diagram showing another example of the detailed configuration of the visible light signal according to the twentieth embodiment. FIG. 200 is a diagram showing another example of the detailed configuration of the visible light signal according to the twentieth embodiment. FIG. 201 is a diagram showing another example of the detailed configuration of the visible light signal according to the twentieth embodiment. FIG. 202 is a diagram showing another example of the detailed configuration of the visible light signal according to the twentieth embodiment. FIG. 203 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. 197. FIG. 204 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. 197. FIG. 205 is a diagram for explaining a method of determining the values of x1 to x4 of FIG. 197. FIG. 206 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. 197. FIG. 207 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. 197. FIG. 208 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. 197. FIG. 209 is a diagram for explaining a method of determining the values of x1 to x4 of FIG. 197. FIG. 210 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. 197. FIG. 211 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. 197. FIG. 212 is a diagram showing an example of a detailed configuration of a visible light signal according to a modification 1 of the twenty embodiment. FIG. 213 is a diagram showing another example of the visible light signal according to the first modification of the twenty embodiment. FIG. 214 is a diagram showing still another example of the visible light signal according to the first modification of the twenty embodiment. FIG. 215 is a diagram showing an example of packet modulation according to the first modification of the twenty embodiment. FIG. 216 is a diagram showing a process of dividing the original data into one according to the first modification of the twenty embodiment. FIG. 217 is a diagram showing a process of dividing the original data into two according to the first modification of the twenty embodiment. FIG. 218 is a diagram showing a process of dividing the original data into three according to the first modification of the twenty embodiment. FIG. 219 is a diagram showing another example of the process of dividing the original data into three according to the first modification of the twenty embodiment. FIG. 220 is a diagram showing another example of the process of dividing the original data into three according to the first modification of the twenty embodiment. FIG. 221 is a diagram showing a process of dividing the original data into four according to the first modification of the twenty embodiment. FIG. 222 is a diagram showing a process of dividing the original data into five according to the first modification of the twenty embodiment. FIG. 223 is a diagram showing a process of dividing the original data into 6, 7 or 8 according to the first modification of the 20th embodiment. FIG. 224 is a diagram showing another example of the process of dividing the original data into 6, 7 or 8 according to the first modification of the twenty embodiment. FIG. 225 is a diagram showing a process of dividing the original data into 9 parts according to the first modification of the 20th embodiment. FIG. 226 is a diagram showing a process of dividing the original data into any number of 10 to 16 according to the first modification of the twenty embodiment. FIG. 227 is a diagram showing an example of the relationship between the number of divisions of the original data, the data size, and the error correction code according to the first modification of the twenty embodiment. FIG. 228 is a diagram showing another example of the relationship between the number of divisions of the original data, the data size, and the error correction code according to the first modification of the twenty embodiment. FIG. 229 is a diagram showing still another example of the relationship between the number of divisions of the original data, the data size, and the error correction code according to the first modification of the twenty embodiment. FIG. 230A is a flowchart showing a method of generating a visible light signal according to the twentieth embodiment. FIG. 230B is a block diagram showing the configuration of the signal generator according to the twentieth embodiment. FIG. 231 is a diagram showing a method of receiving a high frequency visible light signal according to the 21st embodiment. FIG. 232A is a diagram showing another method of receiving a high frequency visible light signal according to the 21st embodiment. FIG. 232B is a diagram showing another method of receiving a high frequency visible light signal according to the 21st embodiment. FIG. 233 is a diagram showing a method of outputting a high frequency signal according to the 21st embodiment. FIG. 234 is a diagram for explaining the autonomous flight device according to the 22nd embodiment. FIG. 235 is a diagram showing an example in which the receiver in the 23rd embodiment displays an AR image. FIG. 236 is a diagram showing an example of the display system according to the 23rd embodiment. FIG. 237 is a diagram showing another example of the display system according to the 23rd embodiment. FIG. 238 is a diagram showing another example of the display system according to the 23rd embodiment. FIG. 239 is a flowchart showing an example of the processing operation of the receiver in the 23rd embodiment. FIG. 240 is a diagram showing another example in which the receiver in the 23rd embodiment displays an AR image. FIG. 241 is a diagram showing another example in which the receiver in the 23rd embodiment displays an AR image. FIG. 242 is a diagram showing another example in which the receiver in the 23rd embodiment displays an AR image. FIG. 243 is a diagram showing another example in which the receiver in the 23rd embodiment displays an AR image. FIG. 244 is a diagram showing another example in which the receiver in the 23rd embodiment displays an AR image. FIG. 245 is a diagram showing another example in which the receiver in the 23rd embodiment displays an AR image. FIG. 246 is a flowchart showing another example of the processing operation of the receiver in the 23rd embodiment. FIG. 247 is a diagram showing another example in which the receiver in the 23rd embodiment displays an AR image. FIG. 248 is a diagram showing an image capture display image Ppre and a decoding image Pdec acquired by image pickup of the receiver in the 23rd embodiment. FIG. 249 is a diagram showing an example of the captured display image Ppre displayed on the receiver in the 23rd embodiment. FIG. 250 is a flowchart showing another example of the processing operation of the receiver in the 23rd embodiment. FIG. 251 is a diagram showing another example in which the receiver in the 23rd embodiment displays an AR image. FIG. 252 is a diagram showing another example in which the receiver in the 23rd embodiment displays an AR image. FIG. 253 is a diagram showing another example in which the receiver in the 23rd embodiment displays an AR image. FIG. 254 is a diagram showing another example in which the receiver in the 23rd embodiment displays an AR image. FIG. 255 is a diagram showing an example of recognition information in the 23rd embodiment. FIG. 256 is a flowchart showing another example of the processing operation of the receiver in the 23rd embodiment. FIG. 257 is a diagram showing an example in which the receiver in the 23rd embodiment identifies the emission line pattern region. FIG. 258 is a diagram showing another example of the receiver in the 23rd embodiment. FIG. 259 is a flowchart showing another example of the processing operation of the receiver in the 23rd embodiment. FIG. 260 is a diagram showing an example of a transmission system including a plurality of transmitters according to the 23rd embodiment. FIG. 261 is a diagram showing an example of a transmission system including a plurality of transmitters and receivers according to the 23rd embodiment. FIG. 262A is a flowchart showing an example of the processing operation of the receiver in the 23rd embodiment. FIG. 262B is a flowchart showing an example of the processing operation of the receiver in the 23rd embodiment. FIG. 263A is a flowchart showing the display method in the 23rd embodiment. FIG. 263B is a block diagram showing the configuration of the display device according to the 23rd embodiment. FIG. 264 is a diagram showing an example in which the receiver in the first modification of the 23rd embodiment displays an AR image. FIG. 265 is a diagram showing another example in which the receiver 200 in the first modification of the second embodiment 23 displays an AR image. FIG. 266 is a diagram showing another example in which the receiver 200 in the first modification of the second embodiment 23 displays an AR image. FIG. 267 is a diagram showing another example in which the receiver 200 in the first modification of the second embodiment 23 displays an AR image. FIG. 268 is a diagram showing another example of the receiver 200 in the first modification of the 23rd embodiment. FIG. 269 is a diagram showing another example in which the receiver 200 in the first modification of the second embodiment 23 displays an AR image. FIG. 270 is a diagram showing another example in which the receiver 200 in the first modification of the second embodiment 23 displays an AR image. FIG. 271 is a flowchart showing an example of the processing operation of the receiver 200 in the modification 1 of the embodiment 23. FIG. 272 is a diagram showing an example of a problem when displaying an AR image assumed in the receiver in the 23rd embodiment or the first modification thereof. FIG. 273 is a diagram showing an example in which the receiver in the second modification of the second embodiment displays an AR image. FIG. 274 is a flowchart showing an example of the processing operation of the receiver in the second modification of the second embodiment. FIG. 275 is a diagram showing another example in which the receiver in the second modification of the second embodiment displays an AR image. FIG. 276 is a flowchart showing another example of the processing operation of the receiver in the second modification of the second embodiment. FIG. 277 is a diagram showing another example in which the receiver in the second modification of the second embodiment displays an AR image. FIG. 278 is a diagram showing another example in which the receiver in the second modification of the second embodiment displays an AR image. FIG. 279 is a diagram showing another example in which the receiver in the second modification of the second embodiment displays an AR image. FIG. 280 is a diagram showing another example in which the receiver in the second modification of the second embodiment displays an AR image. FIG. 281A is a flowchart showing a display method according to one aspect of the present invention. FIG. 281B is a block diagram showing a configuration of a display device according to an aspect of the present invention. FIG. 282 is a diagram showing an example of enlargement and movement of the AR image in the modification 3 of the embodiment 23. FIG. 283 is a diagram showing an example of enlargement of the AR image in the modification 3 of the embodiment 23. FIG. 284 is a flowchart showing an example of a processing operation related to enlargement and movement of the AR image by the receiver in the third modification of the second embodiment. FIG. 285 is a diagram showing an example of superimposition of AR images in the third modification of the 23rd embodiment. FIG. 286 is a diagram showing an example of superimposition of AR images in the third modification of the 23rd embodiment. FIG. 287 is a diagram showing an example of superimposition of AR images in the third modification of the 23rd embodiment. FIG. 288 is a diagram showing an example of superimposition of AR images in the third modification of the 23rd embodiment. FIG. 289A is a diagram showing an example of an image capture display image obtained by image pickup by a receiver in the modification 3 of the embodiment 23. FIG. 289B is a diagram showing an example of a menu screen displayed on the display of the receiver in the modification 3 of the second embodiment 23. FIG. 290 is a flowchart showing an example of the processing operation between the receiver and the server in the modification 3 of the embodiment 23. FIG. 291 is a diagram for explaining the volume of the sound reproduced by the receiver in the modification 3 of the embodiment 23. FIG. 292 is a diagram showing the relationship between the distance from the receiver to the transmitter and the volume in the third modification of the 23rd embodiment. FIG. 293 is a diagram showing an example of superimposition of an AR image by a receiver in the modification 3 of the second embodiment 23. FIG. 294 is a diagram showing an example of superimposition of an AR image by a receiver in the third modification of the 23rd embodiment. FIG. 295 is a diagram for explaining an example of how to obtain the line scan time by the receiver in the modification 3 of the embodiment 23. FIG. 296 is a diagram for explaining an example of how to obtain the line scan time by the receiver in the modification 3 of the embodiment 23. FIG. 297 is a flowchart showing an example of how to obtain the line scan time by the receiver in the third modification of the second embodiment. FIG. 298 is a diagram showing an example of superimposition of an AR image by a receiver in the third modification of the 23rd embodiment. FIG. 299 is a diagram showing an example of superimposition of an AR image by a receiver in the third modification of the 23rd embodiment. FIG. 300 is a diagram showing an example of superimposition of an AR image by a receiver in the third modification of the 23rd embodiment. FIG. 301 is a diagram showing an example of a decoding image acquired according to the posture of the receiver in the modification 3 of the embodiment 23. FIG. 302 is a diagram showing another example of the decoding image acquired according to the posture of the receiver in the modification 3 of the embodiment 23. FIG. 303 is a flowchart showing an example of the processing operation of the receiver in the modification 3 of the embodiment 23. FIG. 304 is a diagram showing an example of a camera lens switching process by the receiver in the third modification of the 23rd embodiment. FIG. 305 is a diagram showing an example of a camera switching process by the receiver in the third modification of the 23rd embodiment. FIG. 306 is a flowchart showing an example of the processing operation between the receiver and the server in the third modification of the second embodiment. FIG. 307 is a diagram showing an example of superimposition of an AR image by a receiver in the modification 3 of the second embodiment 23. FIG. 308 is a sequence diagram showing a processing operation of a system including a receiver, a microwave oven, a relay server, and an electronic payment server in the third modification of the 23rd embodiment. FIG. 309 is a sequence diagram showing the processing operation of the system including the POS terminal, the server, the receiver 200, and the microwave oven in the third modification of the 23rd embodiment. FIG. 310 is a diagram showing an example of indoor use in the third modification of the 23rd embodiment. FIG. 311 is a diagram showing an example of the display of the augmented reality object in the modification 3 of the embodiment 23. FIG. 312 is a diagram showing a configuration of a display system according to a modification 4 of the 23rd embodiment. FIG. 313 is a flowchart showing the processing operation of the display system according to the modified example 4 of the 23rd embodiment. FIG. 314 is a flowchart showing a recognition method according to one aspect of the present invention. FIG. 315 is a diagram showing an example of an operation mode of the visible light signal according to the twenty-fourth embodiment. FIG. 316 is a diagram showing an example of the PPDU format in the mode 1 of the packet PWM according to the twenty-fourth embodiment. FIG. 317 is a diagram showing an example of the PPDU format in the packet PWM mode 2 according to the 24th embodiment. FIG. 318 is a diagram showing an example of the PPDU format in the packet PWM mode 3 according to the twenty-fourth embodiment. FIG. 319 is a diagram showing an example of a pulse width pattern in each SHR of the packet PWM modes 1 to 3 according to the twenty-fourth embodiment. FIG. 320 is a diagram showing an example of the PPDU format in mode 1 of the packet PPM according to the twenty-fourth embodiment. FIG. 321 is a diagram showing an example of the PPDU format in the mode 2 of the packet PPM according to the twenty-fourth embodiment. FIG. 322 is a diagram showing an example of the PPDU format in the mode 3 of the packet PPM according to the twenty-fourth embodiment. FIG. 323 is a diagram showing an example of an interval pattern in each of the modes 1 to 3 of the packet PPM according to the twenty-fourth embodiment. FIG. 324 is a diagram showing an example of 12-bit data included in the PHY payload according to the twenty-fourth embodiment. FIG. 325 is a diagram showing a process of accommodating a PHY frame in one packet according to the twenty-fourth embodiment. FIG. 326 is a diagram showing a process of dividing the PHY frame into two packets according to the twenty-fourth embodiment. FIG. 327 is a diagram showing a process of dividing the PHY frame into three packets according to the twenty-fourth embodiment. FIG. 328 is a diagram showing a process of dividing the PHY frame into four packets according to the twenty-fourth embodiment. FIG. 329 is a diagram showing a process of dividing the PHY frame into 5 packets according to the 24th embodiment. FIG. 330 is a diagram showing a process of dividing the PHY frame into N (N = 6, 7 or 8) packets according to the twenty-fourth embodiment. FIG. 331 is a diagram showing a process of dividing the PHY frame into 9 packets according to the 24th embodiment. FIG. 332 is a diagram showing a process of dividing the PHY frame into N (N = 10 to 16) packets according to the twenty-fourth embodiment. FIG. 333A is a flowchart showing a method of generating a visible light signal according to the twenty-fourth embodiment. FIG. 333B is a block diagram showing the configuration of the signal generation device according to the twenty-fourth embodiment. FIG. 334 is a diagram showing the format of the MAC frame of the MPM in the 25th embodiment. FIG. 335 is a flowchart showing the processing operation of the coding device that generates the MAC frame of the MPM in the 25th embodiment. FIG. 336 is a flowchart showing the processing operation of the decoding device that decodes the MAC frame of the MPM in the 25th embodiment. FIG. 337 is a diagram showing the attributes of the PIB of the MAC in the 25th embodiment. FIG. 338 is a diagram for explaining a method of dimming the MPM in the 25th embodiment. FIG. 339 is a diagram showing the attributes of the PHY PIB in the 25th embodiment. FIG. 340 is a diagram for explaining MPM in the 25th embodiment. FIG. 341 is a diagram showing a PLCP header subfield according to the 25th embodiment. FIG. 342 is a diagram showing a PLCP center subfield according to the 25th embodiment. FIG. 343 is a diagram showing a PLCP footer subfield according to the 25th embodiment. FIG. 344 is a diagram showing a waveform of the PWM mode of the PHY in the MPM according to the 25th embodiment. FIG. 345 is a diagram showing a waveform of the PPM mode of PHY in MPM in the 25th embodiment. FIG. 346 is a flowchart showing an example of the decoding method of the 25th embodiment. FIG. 347 is a flowchart showing an example of the coding method of the 25th embodiment. FIG. 348 is a diagram showing an example in which the receiver in the 26th embodiment displays an AR image. FIG. 349 is a diagram showing an example of a captured display image on which an AR image is superimposed in the 26th embodiment. FIG. 350 is a diagram showing another example in which the receiver in the 26th embodiment displays an AR image. FIG. 351 is a flowchart showing the operation of the receiver in the 26th embodiment. FIG. 352 is a diagram for explaining the operation of the transmitter in the 26th embodiment. FIG. 353 is a diagram for explaining other operations of the transmitter in the 26th embodiment. FIG. 354 is a diagram for explaining other operations of the transmitter in the 26th embodiment. FIG. 355 is a diagram showing a comparative example for explaining the ease of receiving the optical ID in the 26th embodiment. FIG. 356A is a flowchart showing the operation of the transmitter according to the 26th embodiment. FIG. 356B is a block diagram showing the configuration of the transmitter according to the 26th embodiment. FIG. 357 is a diagram showing another example in which the receiver in the 26th embodiment displays an AR image. FIG. 358 is a diagram for explaining the operation of the transmitter according to the 27th embodiment. FIG. 359A is a flowchart showing a transmission method according to the 27th embodiment. FIG. 359B is a block diagram showing the configuration of the transmitter according to the 27th embodiment. FIG. 360 is a diagram showing an example of a detailed configuration of a visible light signal according to the 27th embodiment. FIG. 361 is a diagram showing another example of the detailed configuration of the visible light signal in the 27th embodiment. FIG. 362 is a diagram showing another example of the detailed configuration of the visible light signal in the 27th embodiment. FIG. 363 is a diagram showing another example of the detailed configuration of the visible light signal in the 27th embodiment. FIG. 364 is a diagram showing the relationship between the total of the variables y ₀ to y ₃ and the total time length and the effective time length in the 27th embodiment. FIG. 365A is a flowchart showing a transmission method according to the 27th embodiment. FIG. 365B is a block diagram showing the configuration of the transmitter according to the 27th embodiment.

The transmission method according to one aspect of the present invention is a transmission method for transmitting a signal by changing the brightness of a light source, that is, a reception step that accepts a dimming degree designated for the light source as a designated dimming degree, and the designated dimming degree. When is equal to or less than the first value, the light source is made to emit light at the designated dimming degree, and the signal encoded in the first mode is transmitted by the luminance change, and the designated dimming degree is the first. When it is larger than the value of, the specified dimming degree includes the transmission step of transmitting the signal encoded in the second mode by the luminance change while causing the light source to emit light at the designated dimming degree. When the value is larger than the first value and equal to or less than the second value, the value of the peak current of the light source for transmitting the signal encoded in the second mode by the luminance change is the specified dimming degree. Is smaller than the value of the peak current of the light source for transmitting the signal encoded in the first mode by the luminance change when is the first value.

As a result, as shown in FIG. 354, the value of the peak current of the light source when the specified dimming intensity is larger than the first value and equal to or less than the second value by switching the mode for encoding the signal is set to the specified adjustment. It is smaller than the peak current value of the light source when the luminous intensity is the first value. Therefore, the larger the designated luminous intensity, the more the large peak current can be suppressed from flowing to the light source. As a result, deterioration of the light source can be suppressed. In addition, since deterioration of the light source is suppressed, communication between various devices can be performed for a long period of time.

When the designated dimming intensity is smaller than the third value, the signal encoded in the first mode is transmitted by changing the brightness while the light source is made to emit light at the designated dimming intensity. The value of the peak current may be maintained at a constant value with respect to the change in the designated dimming intensity, and the third value may be smaller than the first value. Specifically, when the designated dimming intensity is smaller than the third value, the designated dimming intensity becomes smaller by lengthening the time for turning off the light source as the designated dimming intensity becomes smaller. The light source may be made to emit light at a luminous intensity, and the value of the peak current may be maintained at a constant value.

As a result, even when the specified luminous intensity becomes small, the value of the peak current is kept constant, so that it is easy for the receiver to receive the visible light signal (that is, the optical ID) which is a signal transmitted by the change in luminance. be able to.

Further, the time for turning off the light source may be determined so that one cycle obtained by adding the time for transmitting the signal due to the change in brightness and the time for turning off the light source does not exceed 10 milliseconds.

For example, if one cycle exceeds 10 milliseconds, there is a risk that the change in brightness of the light source for transmitting the encoded signal will be perceived by the human eye as flickering. Therefore, in the present disclosure, since the time for turning off the light source is determined so that one cycle does not exceed 10 milliseconds, it is possible to suppress the recognition of flicker by a person.

When the designated dimming intensity is smaller than the fourth value, the light source is emitted at the designated dimming intensity, and the signal encoded in the first mode is transmitted by changing the luminance. By reducing the value of the peak current as the designated dimming intensity becomes smaller, the light source is made to emit light at the designated dimming intensity that becomes smaller, and the fourth value is smaller than the second value. May be good.

As a result, even if the designated luminous intensity is smaller, the light source can be appropriately made to emit light at the designated luminous intensity.

Further, the value of the peak current of the light source when the designated luminous intensity is the first value and the value of the peak current of the light source when the designated luminous intensity is the maximum value are the same. You may. For example, the maximum value of the designated luminous intensity is 100%.

As a result, even in the first mode, a large peak current can be passed through the light source, so that the signal transmitted by the change in the luminance of the light source can be easily received by the receiver.

Further, the duty ratio of the signal encoded in the second mode may be larger than the duty ratio of the signal encoded in the first mode.

The first mode is a mode in which the increase in peak current is large even if the increase in dimming intensity is small, and the second mode is a mode in which the increase in peak current is suppressed even if the increase in dimming intensity is large. Therefore, the second mode suppresses the flow of a large peak current to the light source, so that deterioration of the light source can be suppressed. Further, in the first mode, a large peak current flows through the light source even if the dimming intensity is small, so that the receiver can easily receive the signal transmitted by the change in the luminance of the light source. Therefore, in the present disclosure, it is possible to achieve both suppression of deterioration of the light source and ease of receiving a signal.

Further, when the value of the peak current of the light source exceeds the fifth value, the transmission of the signal may be stopped due to the change in the brightness of the light source.

As a result, deterioration of the light source can be further suppressed.

Further, when the usage time of the light source is measured and the usage time is longer than a predetermined time, the signal is brightened by using the value of the parameter for causing the light source to emit light at a dimming intensity larger than the designated dimming intensity. It may be transmitted depending on the change.

As a result, it is possible to prevent the receiver from becoming difficult to receive the signal transmitted due to the change in luminance due to the deterioration of the light source over time.

Further, the usage time of the light source may be measured, and when the usage time is longer than the predetermined time, the pulse width of the current of the light source may be larger than when the usage time is less than the predetermined time.

As a result, even if the light source deteriorates over time, the pulse width of the current of the light source becomes large, so that it is possible to prevent the receiver from becoming difficult to receive the signal transmitted due to the change in the brightness of the light source.

Further, the transmission method according to another aspect of the present invention is a transmission method for transmitting a signal by changing the brightness of the light source, the reception step of accepting the dimming intensity designated for the light source as the designated dimming intensity, and the above-mentioned. The encoded in the second mode comprises a transmission step of transmitting the signal encoded in the first mode or the second mode by a change in brightness while causing the light source to emit light at a specified luminous intensity. The duty ratio of the signal is larger than the duty ratio of the signal encoded in the first mode, and the designated dimming intensity is changed from a small value to a large value in the transmission step. When is the first value, the mode used for encoding the signal is switched from the first mode to the second mode, and the designated luminous intensity is changed from a large value to a small value. When the designated luminous intensity is a second value, the mode used for encoding the signal is switched from the second mode to the first mode, and the second value is the first value. Is less than the value of.

As a result, as shown in FIG. 358, the designated luminous intensity (that is, the switching point) at which the first mode and the second mode are switched is different depending on whether the designated luminous intensity increases or decreases. Therefore, frequent switching between these modes can be suppressed. That is, the occurrence of so-called chattering can be suppressed. As a result, the operation of the transmitting device that transmits the signal can be stabilized. Also, the duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode. Therefore, as in the transmission method according to one aspect of the present invention, the larger the designated luminous intensity, the more the large peak current can be suppressed from flowing to the light source. As a result, deterioration of the light source can be suppressed. In addition, since deterioration of the light source is suppressed, communication between various devices can be performed for a long period of time. When the designated luminous intensity is small, the first mode with a small duty ratio is used. Therefore, the above-mentioned peak current can be increased, and a signal that is easily received by the receiver can be transmitted as a visible light signal.

Further, in the transmission method according to another aspect of the present invention, in the transmission step, when switching from the first mode to the second mode is performed, the encoded signal is transmitted by a change in brightness. The peak current of the light source is changed from the first current value to a second current value smaller than the first current value, and switching from the second mode to the first mode can be performed. When this is done, the peak current is changed from the third current value to a fourth current value larger than the third current value, and the first current value is from the fourth current value. The second current value is larger than the third current value.

This makes it possible to appropriately switch between the first mode and the second mode.

Further, the transmission method according to still another aspect of the present invention is a transmission method for transmitting a visible light signal by changing the luminance of the light emitter, and is a determination step of determining a pattern of the luminance change by modulating the signal. The visible light signal comprises a transmission step of transmitting the visible light signal by varying the luminance of the red color represented by the light source contained in the light emitter according to the determined pattern. In the data including the preamble and the payload, a first luminance value and a second luminance value smaller than the first luminance value appear along the time axis, and the first luminance value. And the length of time that at least one of the second luminance values continues is less than or equal to the first predetermined value, and in the preamble, each of the first and second luminance values is along the time axis. In the payload, the first and second luminance values appear alternately along the time axis, and the time length in which each of the first and second luminance values continues is the first. It is larger than a predetermined value and is determined according to the signal and a predetermined method.

Thereby, as shown in FIG. 363, the visible light signal includes one payload (ie, L data section or R data section) of the waveform determined according to the signal to be modulated, and includes two payloads. not present. Therefore, the visible light signal, that is, the packet of the visible light signal can be shortened. That is, the visible light signal can be transmitted in a short time, and communication between various devices can be performed in a short time. As a result, for example, even if the emission period of the red light represented by the light source included in the light emitting body is short, the packet of the visible light signal can be transmitted during the emission period.

Further, in the payload, the first luminance value of the first time length, the second luminance value of the second time length, the first luminance value of the third time length, and the fourth luminance value. Each luminance value appears in the order of the second luminance value, and in the transmission step, the sum of the first time length and the third time length is smaller than the second predetermined value. The sum of the first time length and the third time length is larger than the case where the sum of the first time length and the third time length is larger than the second predetermined value. , May be larger than the first predetermined value.

As a result, as shown in FIGS. 362 and 363, when the sum of the first time length and the third time length is small, the current value of the light source is increased, and the first time length and the third time length are increased. When the sum of the lengths is large, the current value of the light source is reduced. Therefore, the average luminance of a packet consisting of data, preamble and payload can be kept constant regardless of the signal.

Further, in the payload, the first brightness value of the first time length D ₀ , the second brightness value of the second time length D ₁ , and the first brightness value of the third time length D ₂ , Each brightness value appears in the order of the second brightness value of the fourth time length D ₃ , and the sum of the four parameters y _k (k = 0, 1, 2, 3) obtained from the signal is When it is equal to or less than the third predetermined value, each of the time lengths D ₀ to D ₃ of the first to the fourth is D _k = W ₀ + W ₁ × y _k (W ₀ and W ₁ are 0 or more, respectively). It may be determined according to (integer).

As a result, as shown in FIG. 363 (b), it is possible to generate a payload having a short waveform according to the signal while setting each of the first to fourth time lengths D ₀ to D ₃ to W ₀ or more.

Further, when the sum of the four parameters y _k (k = 0, 1, 2, 3) is equal to or less than the third predetermined value, the data, the preamble, and the payload are used in the transmission step. The data, the preamble, and the payload may be transmitted in this order.

As a result, as shown in FIG. 363 (b), the receiving device that receives the packet by the data indicates that the packet of the visible light signal containing the data (that is, invalid data) does not include the L data unit. I can tell you.

Further, when the sum of the four parameters y _k (k = 0, 1, 2, 3) is larger than the third predetermined value, each of the time lengths D ₀ to D ₃ of the first to fourth is , D ₀ = W ₀ + W ₁ x (A-y ₀ ), D ₁ = W ₀ + W ₁ x (By ₁ ), D ₂ = W ₀ + W ₁ x (A-y ₂ ), and D ₃ = It may be determined according to W ₀ + W ₁ × (By ₃ ) (A and B are integers of 0 or more, respectively).

As a result, as shown in FIG. 363 (a), each of the first to fourth time lengths D ₀ to D ₃ (that is, the first to fourth time lengths D' ₀ to D' ₃ ) is W ₀ or more. However, even if the sum is large, it is possible to generate a payload having a short waveform depending on the signal.

Further, when the sum of the four parameters y _k (k = 0, 1, 2, 3) is larger than the third predetermined value, the data, the preamble, and the payload are used in the transmission step. The payload, the preamble, and the data may be transmitted in this order.

As a result, as shown in FIG. 363 (a), the receiving device that receives the packet by the data indicates that the packet of the visible light signal containing the data (that is, invalid data) does not include the R data unit. I can tell you.

Further, the light emitter has a plurality of light sources including a red light source, a blue light source, and a green light source, and in the transmission step, the visible light source is used only by the red light source among the plurality of light sources. An optical signal may be transmitted.

As a result, the light emitter can display an image using a red light source, a blue light source, and a green light source, and can transmit a visible light signal having a wavelength that is easy to receive to the receiving device.

It should be noted that these comprehensive or specific embodiments may be realized in a recording medium such as a device, system, method, integrated circuit, computer program or computer readable CD-ROM, and the device, system, method, integrated. It may be realized by any combination of circuits, computer programs or recording media.

Hereinafter, embodiments will be specifically described with reference to the drawings.

It should be noted that all of the embodiments described below are comprehensive or specific examples. The numerical values, shapes, materials, components, arrangement positions and connection forms of the components, steps, the order of steps, etc. shown in the following embodiments are examples, and are not intended to limit the present invention. Further, among the components in the following embodiments, the components not described in the independent claim indicating the highest level concept are described as arbitrary components.

(Embodiment 1)
Hereinafter, the first embodiment will be described.

(Observation of the brightness of the light emitting part)
When capturing a single image, we propose an imaging method that starts and ends exposure at different times for each image sensor, instead of exposing all the image sensors at the same timing. FIG. 1 is an example in which the image pickup devices arranged in a row are exposed at the same time and the exposure start times are shifted in the order in which the rows are closer to each other to take an image. Here, it is called an exposure line of an image pickup element that is exposed at the same time, and a line of pixels on an image corresponding to the image pickup element is called an emission line.

When a blinking light source is projected onto the entire surface of an image sensor for imaging using this imaging method, bright lines (bright and dark lines of pixel values) along the exposure line are generated on the captured image as shown in FIG. .. By recognizing this emission line pattern, it is possible to estimate a change in light source luminance at a speed higher than the imaging frame rate. As a result, by transmitting the signal as a change in the brightness of the light source, it is possible to perform communication at a speed higher than the imaging frame rate. When a light source expresses a signal by taking two kinds of luminance values, the lower luminance value is called low (LO) and the higher luminance value is called high (HI). Low may be in a state where the light source is not shining, or may be shining weaker than high.

By this method, information is transmitted at a speed exceeding the imaging frame rate.

When there are 20 exposure lines in one captured image in which the exposure times do not overlap and the frame rate of imaging is 30 fps, a change in brightness with a period of 1.67 milliseconds can be recognized. When there are 1000 exposure lines where the exposure times do not overlap, it is possible to recognize a change in brightness with a period of 1 / 30,000 second (about 33 microseconds). The exposure time is set to be shorter than, for example, 10 milliseconds.

FIG. 2 shows a case where the exposure of one exposure line is completed and then the exposure of the next exposure line is started.

In this case, when the number of frames per second (frame rate) is f and the number of exposure lines constituting one image is l, if information is transmitted depending on whether each exposure line receives a certain amount of light or more, the maximum is transmitted. Information can be transmitted at a speed of fl bits per second.

In addition, when exposure is performed with a time difference for each pixel instead of each line, communication is possible at a higher speed.

At this time, when the number of pixels per exposure line is m pixels and information is transmitted depending on whether or not each pixel receives a certain amount of light or more, the maximum transmission speed is flm bits per second.

As shown in FIG. 3, if the exposure state of each exposure line due to the light emission of the light emitting unit can be recognized at a plurality of levels, the light emission time of the light emitting unit can be controlled in a unit time shorter than the exposure time of each exposure line. , More information can be transmitted.

When the exposure state can be recognized in the Elv stage, information can be transmitted at a speed of up to fEllv bits per second.

Further, the basic cycle of transmission can be recognized by causing the light emitting unit to emit light at a timing slightly deviated from the exposure timing of each exposure line.

FIG. 4 shows a case where the exposure of the next exposure line is started before the exposure of one exposure line is completed. That is, the exposure times of the adjacent exposure lines partially overlap with each other. With such a configuration, it is possible to increase the number of samples within a predetermined time as compared with the case of (1) waiting for the end of the exposure time of one exposure line and starting the exposure of the next exposure line. By increasing the number of samples within a predetermined time, it becomes possible to more appropriately detect an optical signal generated by an optical transmitter as a subject. That is, it is possible to reduce the error rate when detecting an optical signal. Further, (2) the exposure time of each exposure line can be lengthened as compared with the case where the exposure of the next exposure line is started after waiting for the end of the exposure time of one exposure line, so that the subject is dark. However, it is possible to obtain a brighter image. That is, it is possible to improve the S / N ratio. It should be noted that, in all the exposure lines, it is not necessary that the exposure times of the adjacent exposure lines partially overlap with each other, and the exposure times of some of the exposure lines do not partially overlap with each other. It is also possible to. By configuring some exposure lines so that they do not partially overlap in time, it is possible to suppress the generation of neutral colors due to the overlap of exposure times on the imaging screen, and it becomes possible to detect bright lines more appropriately. ..

In this case, the exposure time is calculated from the brightness of each exposure line, and the state of light emission of the light emitting unit is recognized.

When the brightness of each exposure line is determined by the binary value of whether or not the brightness is equal to or higher than the threshold value, in order to recognize the state of not emitting light, the light emitting unit indicates the state of not emitting light for each line. Must continue for longer than the exposure time of.

FIG. 5A shows the influence of the difference in the exposure time when the exposure start times of the exposure lines are the same. 7500a is a case where the exposure end time of the previous exposure line is equal to the exposure start time of the next exposure line, and 7500b is a case where the exposure time is longer than that. By making the exposure time of adjacent exposure lines partially overlapped with each other as in 7500b, it is possible to increase the exposure time. That is, the light incident on the image sensor increases, and a bright image can be obtained. Further, since the imaging sensitivity for capturing an image having the same brightness can be suppressed to a low level, an image with less noise can be obtained, and communication errors can be suppressed.

FIG. 5B shows the influence of the difference in the exposure start time of each exposure line when the exposure times are the same. 7501a is a case where the exposure end time of the previous exposure line and the exposure start time of the next exposure line are equal, and 7501b is a case where the exposure of the next exposure line is started earlier than the end of the exposure of the previous exposure line. By configuring the exposure times of adjacent exposure lines to partially overlap with each other as in 7501b, it is possible to increase the number of lines that can be exposed per hour. As a result, the resolution becomes higher and a large amount of information can be obtained. By increasing the sample interval (= difference in exposure start time), the change in light source brightness can be estimated more accurately, the error rate can be reduced, and the change in light source brightness in a shorter time can be recognized. be able to. By having the exposure times overlap, it is possible to recognize the blinking of the light source shorter than the exposure time by utilizing the difference in the exposure amount of the adjacent exposure lines.

Further, when the number of samples described above is small, that is, when the sample interval (time difference t _D shown in FIG. 5B) is long, there is a high possibility that the change in the brightness of the light source cannot be detected accurately. In this case, the possibility can be suppressed by shortening the exposure time. That is, the change in the brightness of the light source can be accurately detected. Further, it is desirable that the exposure time satisfies the exposure time> (sample interval-pulse width). The pulse width is the pulse width of light during which the brightness of the light source is high. Thereby, the brightness of High can be appropriately detected.

As described with reference to FIGS. 5A and 5B, in a configuration in which each exposure line is sequentially exposed so that the exposure times of adjacent exposure lines partially overlap with each other, the exposure time is set to be longer than that in the normal shooting mode. By using the emission line pattern generated by setting it short for signal transmission, it is possible to dramatically improve the communication speed. Here, by setting the exposure time during visible light communication to 1/480 seconds or less, it is possible to generate an appropriate emission line pattern. Here, the exposure time needs to be set to the exposure time <1/8 × f, where frame frequency = f. The blanking that occurs during shooting is up to half the size of one frame. That is, since the blanking time is less than half of the shooting time, the actual shooting time is 1 / 2f in the shortest time. Further, since it is necessary to receive the information of four values within the time of 1 / 2f, at least the exposure time needs to be shorter than 1 / (2f × 4). Since the normal frame rate is 60 frames / sec or less, by setting the exposure time to 1/480 sec or less, it is possible to generate an appropriate emission line pattern in the image data and perform high-speed signal transmission. Become.

FIG. 5C shows the advantages of short exposure times when the exposure times of the exposure lines do not overlap. When the exposure time is long, even if the light source has a binary brightness change like 7502a, a neutral color part is formed like 7502e in the captured image, and it tends to be difficult to recognize the brightness change of the light source. There is. However, as in 7502d, the brightness of the light source is changed by providing a free time (predetermined waiting time) t _D2 in which a predetermined exposure is not performed until the start of exposure of the next exposure line after the end of exposure of one exposure line. Can be easily recognized. That is, it becomes possible to detect a more appropriate emission line pattern such as 7502f. A configuration such as 7502d in which a predetermined free time without exposure is provided can be realized by making the exposure time t _E smaller than the time difference t _D of the exposure start time of each exposure line. When the normal shooting mode has a configuration in which the exposure times of adjacent exposure lines partially overlap with each other, the exposure time is set shorter than in the normal shooting mode until a predetermined free time without exposure occurs. By doing so, it can be realized. Further, even when the normal shooting mode is equal to the exposure end time of the previous exposure line and the exposure start time of the next exposure line, the exposure time is set short until a predetermined non-exposure time occurs. It can be realized. Further, by increasing the interval _tD of the exposure start time of each exposure line as in 7502 g, a predetermined free time (predetermined) during which a predetermined exposure is not performed until the start of the exposure of the next exposure line after the end of the exposure of one exposure line. (Waiting time) t _D2 can be provided. In this configuration, since the exposure time can be lengthened, a bright image can be captured and noise is reduced, so that error tolerance is high. On the other hand, in this configuration, since the number of exposure lines that can be exposed within a certain period of time is reduced, there is a drawback that the number of samples is reduced as in 7502h, so it is desirable to use them properly depending on the situation. For example, by using the former configuration when the imaging target is bright and using the latter configuration when it is dark, it is possible to reduce the estimation error of the light source luminance change.

It should be noted that, in all the exposure lines, it is not necessary that the exposure times of the adjacent exposure lines partially overlap with each other, and the exposure times of some of the exposure lines do not partially overlap with each other. It is also possible to. Further, it is not necessary to provide a predetermined free time (predetermined waiting time) for all exposure lines from the end of exposure of one exposure line to the start of exposure of the next exposure line, and it is not necessary to provide a partial exposure. It is also possible to have a configuration in which the lines partially overlap in time. With such a configuration, it is possible to take advantage of each configuration. Further, the same readout method is used in the normal shooting mode in which shooting is performed at a normal frame rate (30 fps, 60 fps) and the visible light communication mode in which shooting is performed with an exposure time of 1/480 second or less for visible light communication. Alternatively, the signal may be read out by the circuit. By reading the signal by the same reading method or circuit, it is not necessary to use different circuits for the normal shooting mode and the visible light communication mode, and the circuit scale can be reduced.

FIG. 5D shows the relationship between the minimum change time t _S of the light source luminance, the exposure time t _E , the time difference t _D of the exposure start time of each exposure line, and the captured image. When t _E + t _D <t _S , one or more exposure lines are always imaged with the light source unchanged from the start to the end of the exposure, so an image with clear brightness such as 7503d can be obtained. It is easy to recognize changes in the brightness of the light source. When 2t _E > t _S , a bright line having a pattern different from the change in the brightness of the light source may be obtained, and it becomes difficult to recognize the change in the brightness of the light source from the captured image.

FIG. 5E shows the relationship between the transition time t _T of the light source luminance and the time difference t _D of the exposure start time of each exposure line. As t _D is larger than t _T , the number of exposure lines that become neutral colors is reduced, and it becomes easier to estimate the light source brightness. When t _D > t _T , the number of exposure lines for neutral colors is continuously 2 or less, which is desirable. Since t _T is 1 microsecond or less when the light source is an LED and about 5 microseconds when the light source is an organic EL, setting t _D to 5 microseconds or more facilitates estimation of the light source brightness. be able to.

FIG. 5F shows the relationship between the high frequency noise _tHT of the light source luminance and the exposure time t _E. The larger t _E than t _HT , the less the influence of high frequency noise on the captured image, and the easier it is to estimate the brightness of the light source. When t _E is an integral multiple of t _HT , the influence of high frequency noise disappears, and the estimation of the light source brightness becomes the easiest. For the estimation of the light source brightness, it is desirable that t _E > t _HT . The main cause of high-frequency noise comes from the switching power supply circuit, and since _tHT is 20 microseconds or less in many switching power supplies for lamps, light source brightness can be estimated by setting t _E to 20 microseconds or more. It can be done easily.

FIG. 5G is a graph showing the relationship between the exposure time t _E and the magnitude of high frequency noise when t _HT is 20 microseconds. Considering that t _HT varies depending on the light source, from the graph, t _E is a value equal to the value when the amount of noise is maximized, which is 15 microseconds or more, 35 microseconds or more, or. It can be confirmed that the efficiency is good when it is set to 54 microseconds or more or 74 microseconds or more. From the viewpoint of reducing high-frequency noise, it is desirable that t _E is large, but as described above, the smaller t _E is, the less likely it is that a neutral color portion is generated, which makes it easier to estimate the brightness of the light source. Therefore, when the cycle of change in light source brightness is 15 to 35 microseconds, t _E is 15 microseconds or more, and when the cycle of change in light source brightness is 35 to 54 microseconds, t _E is 35 microseconds or more, and the light source brightness. When the change cycle of is 54 to 74 microseconds, t _E should be set to 54 microseconds or more, and when the change cycle of the light source brightness is 74 microseconds or more, t _E should be set to 74 microseconds or more.

_FIG . 5H shows the relationship between the exposure time tE and the recognition success rate. Since the exposure time t _E has a relative meaning to the time when the brightness of the light source is constant, the horizontal axis is the value obtained by dividing the period t _S at which the brightness of the light source changes by the exposure time t _E (relative exposure time). There is. From the graph, it can be seen that if the recognition success rate is to be almost 100%, the relative exposure time should be 1.2 or less. For example, when the transmission signal is 1 kHz, the exposure time may be about 0.83 ms or less. Similarly, if the recognition success rate is 95% or more, the relative exposure time should be 1.25 or less, and if the recognition success rate is 80% or more, the relative exposure time should be 1.4 or less. Recognize. In addition, the recognition success rate drops sharply when the relative exposure time is around 1.5, and becomes almost 0% at 1.6, so it is clear that the relative exposure time should not exceed 1.5. .. Further, it can be seen that after the recognition rate becomes 0 at 7507c, it increases again at 7507d, 7507e, and 7507f. Therefore, if you want to take a bright image with a long exposure time, use an exposure time with a relative exposure time of 1.9 to 2.2, 2.4 to 2.6, 2.8 to 3.0. Just do it. For example, it is good to use these exposure times as an intermediate mode.

FIG. 6A is a flowchart of the information communication method according to the present embodiment.

The information communication method in the present embodiment is an information communication method for acquiring information from a subject, and includes steps SK91 to SK93.

That is, in this information communication method, a plurality of emission lines corresponding to a plurality of exposure lines included in the image sensor are generated in the image obtained by photographing the subject by the image sensor according to the change in the brightness of the subject. The first exposure time setting step SK91 for setting the first exposure time of the image sensor, and the image sensor taking a picture of the subject whose brightness changes at the set first exposure time. The first image acquisition step SK92 for acquiring an emission line image including a plurality of emission lines, and information acquisition for acquiring information by demolishing the data specified by the plurality of emission line patterns included in the acquired emission line image. In the first image acquisition step SK92 including step SK93, each of the plurality of exposure lines starts exposure at different times in sequence, and the exposure of the adjacent exposure line adjacent to the exposure line ends. Then, after a predetermined free time has elapsed, the exposure is started.

FIG. 6B is a block diagram of the information communication device according to the present embodiment.

The information communication device K90 in the present embodiment is an information communication device that acquires information from a subject, and includes components K91 to K93.

That is, in this information communication device K90, a plurality of emission lines corresponding to a plurality of exposure lines included in the image sensor are generated in the image obtained by photographing the subject by the image sensor according to the change in the brightness of the subject. The image sensor that acquires an emission line image including the plurality of emission lines by taking an exposure time setting unit K91 that sets the exposure time of the image sensor and the subject whose brightness changes at the set exposure time. The plurality of exposure lines are provided with an image acquisition unit K92 having an image acquisition unit K92 and an information acquisition unit K93 for acquiring information by demodulating the data specified by the pattern of the plurality of emission lines included in the acquired emission line image. Each of the above starts exposure at different times in sequence, and starts exposure after a predetermined free time elapses after the exposure of the adjacent exposure line adjacent to the exposure line is completed.

In such an information communication method and information communication device K90 shown by FIGS. 6A and 6B, as shown in, for example, FIG. 5C, each of the plurality of exposure lines has an exposure of an adjacent exposure line adjacent to the exposure line. Since the exposure is started after a predetermined free time has elapsed from the end, it is possible to easily recognize the change in the brightness of the subject. As a result, information can be appropriately acquired from the subject.

In the above embodiment, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. For example, the program causes the computer to execute the information communication method shown by the flowchart of FIG. 6A.

(Embodiment 2)
In this embodiment, each application example using a receiver such as a smartphone which is the information communication device K90 in the first embodiment and a transmitter which transmits information as a blinking pattern of a light source such as an LED or an organic EL is provided. explain.

In the following description, shooting in the normal shooting mode or the normal shooting mode is referred to as normal shooting, and shooting in the visible light communication mode or the visible light communication mode is referred to as visible light shooting (visible light communication). Further, instead of normal shooting and visible light shooting, shooting in the intermediate mode may be used, or an intermediate image may be used instead of the composite image described later.

FIG. 7 is a diagram showing an example of the photographing operation of the receiver in the present embodiment.

The receiver 8000 switches the shooting mode to normal shooting, visible light communication, normal shooting, and so on. Then, the receiver 8000 generates a composite image in which the emission line pattern and the subject and its surroundings are clearly projected by synthesizing the normally captured image and the visible light communication image, and displays the composite image on the display. .. This composite image is an image generated by superimposing a bright line pattern of a visible light communication image on a portion of a normally captured image in which a signal is transmitted. Further, the emission line pattern projected by this composite image, the subject and its surroundings are each clear, and have a sharpness sufficiently recognized by the user. By displaying such a composite image, the user can know more clearly from where or from what position the signal is transmitted.

FIG. 8 is a diagram showing another example of the photographing operation of the receiver in the present embodiment.

The receiver 8000 includes a camera Ca1 and a camera Ca2. In such a receiver 8000, the camera Ca1 takes a normal picture, and the camera Ca2 takes a visible light picture. As a result, the camera Ca1 acquires the normal captured image as described above, and the camera Ca2 acquires the visible light communication image as described above. Then, the receiver 8000 generates the above-mentioned composite image by synthesizing the normally captured image and the visible light communication image and displays it on the display.

FIG. 9 is a diagram showing another example of the photographing operation of the receiver in the present embodiment.

In the receiver 8000 having two cameras, the camera Ca1 switches the shooting mode to normal shooting, visible light communication, normal shooting, and so on. On the other hand, the camera Ca2 continuously performs normal shooting. Then, when normal shooting is performed by the camera Ca1 and the camera Ca2 at the same time, the receiver 8000 receives from the normal shot images acquired by those cameras by using stereo vision (principle of triangulation). The distance from the machine 8000 to the subject (hereinafter referred to as the subject distance) is estimated. By using the subject distance estimated in this way, the receiver 8000 can superimpose the emission line pattern of the visible light communication image on an appropriate position of the normally captured image. That is, an appropriate composite image can be generated.

FIG. 10 is a diagram showing an example of the display operation of the receiver in the present embodiment.

As described above, the receiver 8000 switches the shooting mode as visible light communication, normal shooting, visible light communication, and so on. Here, the receiver 8000 starts the application program when the visible light communication is first performed. Then, the receiver 8000 estimates its position based on the signal received by the visible light communication. Next, when the receiver 8000 performs normal shooting, AR (Augmented Reality) information is displayed on the normal shooting image acquired by the normal shooting. This AR information is acquired based on the position estimated as described above. Further, the receiver 8000 estimates the movement and the change in the direction of the receiver 8000 based on the detection result by the 9-axis sensor and the motion detection of the normally captured image, and matches the estimated movement and the change in the direction. And move the display position of AR information. As a result, the AR information can be made to follow the subject image of the normally captured image.

Further, when the receiver 8000 switches the shooting mode from normal shooting to visible light communication, the receiver 8000 superimposes AR information on the latest normal shooting image acquired at the time of the immediately preceding normal shooting at the time of the visible light communication. Then, the receiver 8000 displays a normally captured image on which AR information is superimposed. Further, the receiver 8000 estimates the movement and the change in the direction of the receiver 8000 based on the detection result by the 9-axis sensor as in the normal shooting, and AR according to the estimated movement and the change in the direction. Move information and normally captured images. As a result, even during visible light communication, AR information can be made to follow the subject image of the normal shooting image in accordance with the movement of the receiver 8000 or the like, as in the case of normal shooting. In addition, the normal image can be enlarged or reduced in accordance with the movement of the receiver 8000 or the like.

FIG. 11 is a diagram showing an example of the display operation of the receiver in the present embodiment.

For example, the receiver 8000 may display the composite image on which the emission line pattern is projected, as shown in FIG. 11A. Further, as shown in FIG. 11B, the receiver 8000 normally captures a signal explicit object which is an image having a predetermined color for notifying that a signal is being transmitted instead of the emission line pattern. A composite image may be generated by superimposing on the image, and the composite image may be displayed.

Further, in the receiver 8000, as shown in FIG. 11C, the place where the signal is transmitted is usually indicated by a dotted line frame and an identifier (for example, ID: 101, ID: 102, etc.). The captured image may be displayed as a composite image. Further, as shown in FIG. 11D, the receiver 8000 is a signal identification image having a predetermined color for notifying that a specific type of signal is transmitted instead of the emission line pattern. A composite image may be generated by superimposing an object on a normally captured image, and the composite image may be displayed. In this case, the color of the signal identification object depends on the type of signal output from the transmitter. For example, if the signal output from the transmitter is position information, the red signal identification object is superimposed, and if the signal output from the transmitter is a coupon, the green signal identification object is superimposed. It is superimposed.

FIG. 12 is a diagram showing an example of the operation of the receiver in the present embodiment.

For example, when the receiver 8000 receives a signal by visible light communication, it may display a normally captured image and output a sound for notifying the user that the transmitter has been found. In this case, the receiver 8000 may have different output sound types, output times, or output times depending on the number of transmitters found, the type of signal received, the type of information specified by the signal, and the like. You may let me.

FIG. 13 is a diagram showing another example of the operation of the receiver in the present embodiment.

For example, when the user touches the emission line pattern projected on the composite image, the receiver 8000 generates an information notification image based on the signal transmitted from the subject corresponding to the touched emission line pattern, and notifies the information. Display the image. This information notification image shows, for example, a coupon or a place of a store. The emission line pattern may be a signal explicit object, a signal identification object, a dotted line frame, or the like shown in FIG. The same applies to the emission line pattern described below.

FIG. 14 is a diagram showing another example of the operation of the receiver in the present embodiment.

For example, when the user touches the emission line pattern projected on the composite image, the receiver 8000 generates an information notification image based on the signal transmitted from the subject corresponding to the touched emission line pattern, and notifies the information. Display the image. This information notification image shows, for example, the current location of the receiver 8000 by a map or the like.

FIG. 15 is a diagram showing another example of the operation of the receiver in the present embodiment.

For example, when the user swipes on the receiver 8000 on which the composite image is displayed, the receiver 8000 has a dotted line frame and an identifier similar to the normal shot image shown in FIG. 11 (c). Display an image and a list of information to follow the swipe operation. This list shows the information identified by the signal transmitted from the location (transmitter) indicated by each identifier. Further, the swipe may be, for example, an operation of moving a finger from the outside to the inside of the right side of the display in the receiver 8000. The swipe may be an operation of moving the finger from the upper side, the lower side, or the left side of the display.

Further, when the information included in the list is tapped by the user, the receiver 8000 may display an information notification image (for example, an image showing a coupon) showing the information in more detail.

FIG. 16 is a diagram showing another example of the operation of the receiver in the present embodiment.

For example, when the user swipes the receiver 8000 on which the composite image is displayed, the receiver 8000 superimposes and displays the information notification image on the composite image so as to follow the swipe operation. This information notification image shows the subject distance together with an arrow in an easy-to-understand manner to the user. Further, the swipe may be, for example, an operation of moving a finger from the outside to the inside of the lower side of the display in the receiver 8000. The swipe may be an operation of moving the finger from the left side of the display, from the upper side, or from the right side to the inside.

FIG. 17 is a diagram showing another example of the operation of the receiver in the present embodiment.

For example, the receiver 8000 shoots a transmitter, which is a signage indicating a plurality of stores, as a subject, and displays a normal shot image acquired by the shooting. Here, when the user taps the image of the signage of one store included in the subject projected in the normally captured image, the receiver 8000 generates an information notification image based on the signal transmitted from the signage of the store. Then, the information notification image 8001 is displayed. The information notification image 8001 is an image showing, for example, the availability of seats in a store.

FIG. 18 is a diagram showing an example of the operation of the receiver, the transmitter, and the server in the present embodiment.

First, the transmitter 8012 configured as a television transmits a signal to the receiver 8011 by changing the luminance. This signal contains, for example, information to encourage the user to purchase content related to the program being watched. When the receiver 8011 receives the signal by visible light communication, the receiver 8011 displays an information notification image prompting the user to purchase the content based on the signal. When the user performs an operation for purchasing the content, the receiver 8011 receives information, a user ID, a terminal ID, a credit card information, and a charge included in the SIM (Subscriber Identity Module) card inserted in the receiver 8011. At least one of the information, password, and transmitter ID for is transmitted to the server 8013. The server 8013 manages the user ID and the payment information in association with each user. Then, the server 8013 identifies the user ID based on the information transmitted from the receiver 8011, and confirms the payment information associated with the user ID. Based on this confirmation, the server 8013 determines whether or not to allow the user to purchase the content. Then, when the server 8013 determines that the permission is permitted, the server 8013 transmits the permission information to the receiver 8011. When the receiver 8011 receives the permission information, the receiver 8011 transmits the permission information to the transmitter 8012. The transmitter 8012 that has received the permission information acquires and reproduces the content, for example, via a network.

Further, the transmitter 8012 may transmit information including the ID of the transmitter 8012 to the receiver 8011 by changing the brightness. In this case, the receiver 8011 transmits the information to the server 8013. When the server 8013 acquires the information, it can determine that, for example, a television program is being viewed by the transmitter 8012, and can conduct an audience rating survey of the television program.

Further, the receiver 8011 includes the content operated by the user (voting, etc.) in the above information and transmits the content to the server 8013, so that the server 8013 can reflect the content in the television program. That is, it is possible to realize a viewer participation type program. Further, when the receiver 8011 accepts the writing by the user, the receiver 8011 includes the content of the writing in the above-mentioned information and transmits the writing to the server 8013, so that the server 8013 sends the writing to the TV program or the bulletin board on the network. It can be reflected in such as.

Further, when the transmitter 8012 transmits the above-mentioned information, the server 8013 can charge for viewing a television program by pay broadcasting or an on-demand program. Further, the server 8013 displays an advertisement on the receiver 8011, displays detailed information of the TV program displayed on the transmitter 8012, and displays the URL of the site showing the detailed information. Can be done. Further, the server 8013 obtains the number of times the advertisement is displayed by the receiver 8011, the amount of the product purchased by the advertisement, and the like, and charges the advertiser according to the number of times or the amount. be able to. Billing based on such an amount can be performed without the user who sees the advertisement purchasing the product immediately. Further, when the server 8013 acquires information indicating the manufacturer of the transmitter 8012 from the transmitter 8012 via the receiver 8011, the server 8013 provides a service to the manufacturer indicated by the information (for example, a reward for selling the above-mentioned product). Payment) can be made.

FIG. 19 is a diagram showing another example of the operation of the receiver in the present embodiment.

The receiver 8030 is configured as, for example, a head-mounted display equipped with a camera. When the start button is pressed, the receiver 8030 starts shooting in the visible light communication mode, that is, visible light communication. Then, when the signal is received by the visible light communication, the receiver 8030 notifies the user of the information corresponding to the received signal. This notification is performed, for example, by outputting sound from a speaker provided in the receiver 8030, or by displaying an image. Further, in the visible light communication, other than when the start button is pressed, when the input of the voice instructing the start is received by the receiver 8030, or the signal instructing the start is received by the receiver 8030 by wireless communication. It may be started as soon as it is done. In addition, visible light communication is started when the range of change in the value obtained by the 9-axis sensor provided in the receiver 8030 exceeds a predetermined range, or when a bright line pattern appears even a little in the normally captured image. You may.

FIG. 20 is a diagram showing another example of the operation of the receiver in the present embodiment.

The receiver 8030 displays the composite image 8034 in the same manner as described above. Here, the user performs an operation of moving the fingertip so as to surround the bright line pattern in the composite image 8034. Upon receiving this operation, the receiver 8030 identifies the emission line pattern targeted for the operation, and displays the information notification image 8032 based on the signal transmitted from the portion corresponding to the emission line pattern.

FIG. 21 is a diagram showing another example of the operation of the receiver in the present embodiment.

The receiver 8030 displays the composite image 8034 in the same manner as described above. Here, the user performs an operation of applying the fingertip to the bright line pattern in the composite image 8034 for a predetermined time or longer. Upon receiving this operation, the receiver 8030 identifies the emission line pattern targeted for the operation, and displays the information notification image 8032 based on the signal transmitted from the portion corresponding to the emission line pattern.

FIG. 22 is a diagram showing an example of the operation of the transmitter in the present embodiment.

The transmitter alternately transmits the signal 1 and the signal 2 at a predetermined cycle, for example. The transmission of the signal 1 and the transmission of the signal 2 are performed by changing the luminance such as blinking of visible light, respectively. Further, the pattern of the luminance change for transmitting the signal 1 and the pattern of the luminance change for transmitting the signal 2 are different from each other.

FIG. 23 is a diagram showing another example of the operation of the transmitter in the present embodiment.

As described above, when the transmitter repeatedly transmits the signal sequence of the structural unit including the block 1, the block 2, and the block 3, the arrangement of the blocks included in the signal sequence is changed for each signal string. May be good. For example, each block is arranged in the order of block 1, block 2, and block 3 in the first signal sequence, and each block is arranged in the order of block 3, block 1, and block 2 in the next signal sequence. This makes it possible to avoid acquiring only the same block by a receiver that requires a periodic blanking period.

FIG. 24 is a diagram showing an application example of the receiver in this embodiment.

For example, the receiver 7510a configured as a smartphone captures the light source 7510b with the back camera (out camera) 7510c, receives the signal transmitted from the light source 7510b, and acquires the position and orientation of the light source 7510b from the received signal. The receiver 7510a estimates the position and orientation of the receiver 7510a itself from the appearance of the light source 7510b in the captured image and the sensor value of the 9-axis sensor provided in the receiver 7510a. The receiver 7510a captures the user 7510e with the front camera (face camera, in-camera) 7510f, and estimates the position and orientation of the head of the 7510e and the line-of-sight direction (position and orientation of the eyeball) by image processing. .. The receiver 7510a transmits the estimation result to the server. The receiver 7510a changes its behavior (display contents and reproduced sound) according to the line-of-sight direction of the user 7510e. The image pickup by the back camera 7510c and the image pickup by the front camera 7510f may be performed simultaneously or alternately.

FIG. 25 is a diagram showing another example of the operation of the receiver in the present embodiment.

The receiver displays the emission line pattern by a composite image or an intermediate image as described above. At this time, the receiver may not be able to receive the signal from the transmitter corresponding to this emission line pattern. Here, when the bright line pattern is selected by the user performing an operation (for example, tapping) on the bright line pattern, the receiver performs an optical zoom to enlarge the portion of the bright line pattern or an intermediate image. Display the image. By performing such optical zooming, the receiver can appropriately receive the signal from the transmitter corresponding to the emission line pattern. That is, even if the image obtained by imaging is too small to acquire a signal, the signal can be appropriately received by performing optical zooming. Further, even when an image having a size capable of acquiring a signal is displayed, fast reception can be performed by performing optical zooming.

(Summary of this embodiment)
The information communication method in the present embodiment is an information communication method for acquiring information from a subject, and an image obtained by photographing the subject by an image sensor has a bright line corresponding to an exposure line included in the image sensor. The first exposure time setting step of setting the exposure time of the image sensor so as to occur according to the change in the brightness of the subject, and the image sensor taking a picture of the subject whose brightness changes with the set exposure time. Thereby, the subject and the subject can be identified in such a manner that the spatial position of the portion where the bright line appears can be identified based on the bright line image acquisition step of acquiring the bright line image which is an image including the bright line and the bright line image. An image display step of displaying a display image in which the surroundings of the subject are projected, and an information acquisition step of acquiring transmission information by demolishing data specified by the emission line pattern included in the acquired emission line image. And include.

For example, a composite image or an intermediate image as shown in FIGS. 7, 8 and 11 is displayed as a display image. Further, in the display image in which the subject and the surroundings of the subject are projected, the spatial position of the portion where the emission line appears is identified by the emission line pattern, the signal explicit object, the signal identification object, the dotted line frame, or the like. Therefore, the user can easily find the subject transmitting the signal by the change in luminance by looking at such a display image.

Further, the information communication method further includes a second exposure time setting step of setting an exposure time longer than the exposure time, and the image sensor captures the subject and the periphery of the subject with the long exposure time. By doing so, the normal image acquisition step for acquiring a normal shot image and the portion where the bright line appears in the normal shot image are specified based on the bright line image, and the signal object which is an image pointing to the portion is the normal. In the image display step, the composite image may be displayed as the display image, including a composite step of generating a composite image by superimposing the composite image on the captured image.

For example, the signal object is a bright line pattern, a signal explicit object, a signal identification object, a dotted line frame, or the like, and as shown in FIGS. 7, 8 and 11, a composite image is displayed as a display image. This allows the user to more easily find the subject transmitting the signal due to the change in luminance.

Further, in the first exposure time setting step, the exposure time is set to 1/3000 second, in the emission line image acquisition step, the emission line image in which the surroundings of the subject are projected is acquired, and in the image display step, the emission line image is acquired. , The emission line image may be displayed as the display image.

For example, the emission line image is acquired and displayed as an intermediate image. Therefore, it is not necessary to perform processing such as acquiring and synthesizing a normally captured image and a visible light communication image, and the processing can be simplified.

Further, the image sensor includes a first image sensor and a second image sensor, and in the normal image acquisition step, the first image sensor captures the image to acquire the normal captured image, and the emission line is captured. In the image acquisition step, the emission line image may be acquired by the second image sensor taking an image at the same time as the image taken by the first image sensor.

For example, as shown in FIG. 8, a normal photographed image and a visible light communication image which is an emission line image are acquired by each camera. Therefore, as compared with the case of acquiring the normally captured image and the visible light communication image with one camera, those images can be acquired faster and the processing can be speeded up.

Further, when the portion where the emission line appears in the display image is designated by an operation by the user, the information communication method further uses the transmission information acquired from the emission line pattern of the designated portion. It may include an information presentation step that presents the based presentation information. For example, the operation by the user includes tapping, swiping, an operation of continuously touching the portion with a fingertip for a predetermined time or longer, an operation of continuously directing the line of sight to the portion for a predetermined time or longer, and an operation associated with the portion. The pointer displayed on the display image is applied to the part by moving a part of the user's body to the arrow, touching the part with a pen tip whose brightness changes, or touching the touch sensor. It is an operation.

For example, as shown in FIGS. 13 to 17, 20 and 21, the presented information is displayed as an information notification image. This makes it possible to present desired information to the user.

Further, the image sensor is provided in the head-mounted display, and in the image display step, the projector mounted on the head-mounted display may display the display image.

Thereby, for example, as shown in FIGS. 19 to 21, information can be easily presented to the user.

Further, it is an information communication method for acquiring information from a subject, and a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in the brightness of the subject. As described above, the first exposure time setting step for setting the exposure time of the image sensor and the image sensor taking an image of the subject whose brightness changes at the set exposure time, thereby including the bright line. The bright line image includes a bright line image acquisition step for acquiring a bright line image, and an information acquisition step for acquiring information by demodulating the data specified by the bright line pattern included in the acquired bright line image. In the acquisition step, by photographing a plurality of the subjects while the image sensor is being moved, the emission line image including a plurality of portions where the emission lines appear is acquired, and in the information acquisition step, for each of the portions. In addition, by demodulating the data specified by the pattern of the emission line of the portion, the positions of the plurality of the subjects are acquired, and the information communication method further obtains each of the acquired plurality of the subjects. A position estimation step for estimating the position of the image sensor based on the position and the moving state of the image sensor may be included.

As a result, the position of the receiver including the image sensor can be accurately estimated by the change in brightness due to the subject such as a plurality of lights.

Further, it is an information communication method for acquiring information from a subject, and a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in the brightness of the subject. As described above, the first exposure time setting step for setting the exposure time of the image sensor and the image sensor taking a picture of the subject whose brightness changes at the set exposure time, thereby including the bright line. The emission line image acquisition step for acquiring the emission line image, the information acquisition step for acquiring information by demodulating the data specified by the emission line pattern included in the acquired emission line image, and the acquired information. In the information presentation step, an image prompting a predetermined gesture may be presented as the information to the user of the image sensor.

As a result, it is possible to authenticate the user depending on whether or not the user performs the gesture as prompted, and the convenience can be enhanced.

Further, it is an information communication method for acquiring information from a subject, and a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in the brightness of the subject. As described above, the exposure time setting step for setting the exposure time of the image sensor and the image sensor taking a picture of the subject whose brightness changes at the set exposure time obtains a bright line image including the bright line. The image acquisition step includes an information acquisition step of acquiring information by demodulating the data specified by the emission line pattern included in the acquired emission line image, and the image acquisition step reflects the image on the reflective surface. The bright line image is acquired by photographing a plurality of the subjects, and in the information acquisition step, the bright line is converted into a bright line corresponding to each of the plurality of subjects according to the intensity of the bright line included in the bright line image. Information may be acquired by separating and demodulating the data specified by the pattern of the emission line corresponding to the subject for each subject.

As a result, even when the brightness of each subject such as a plurality of lights changes, appropriate information can be acquired from each of the subjects.

Further, it is an information communication method for acquiring information from a subject, and a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in the brightness of the subject. As described above, the exposure time setting step for setting the exposure time of the image sensor and the image sensor taking a picture of the subject whose brightness changes at the set exposure time obtains a bright line image including the bright line. The image acquisition step includes an information acquisition step of acquiring information by demodulating the data specified by the emission line pattern included in the acquired emission line image, and the image acquisition step reflects the image on the reflective surface. The emission line image may be acquired by photographing the subject, and the information communication method may further include a position estimation step of estimating the position of the subject based on the brightness distribution in the emission line image.

This makes it possible to estimate an appropriate subject position based on the luminance distribution.

Further, it is an information communication method for transmitting a signal by a change in luminance, that is, a first determination step of determining a first pattern of a change in luminance by modulating a first signal of a transmission target, and a first determination step of the transmission target. The second determination step of determining the second pattern of luminance change by modulating the signal of 2, the luminance change according to the determined first pattern, and the determined first. It may include a transmission step of transmitting the first and second signals by alternately performing the luminance change according to the pattern 2.

Thereby, for example, as shown in FIG. 22, the first signal and the second signal can be transmitted without delay.

Further, in the transmission step, when the luminance change is switched between the luminance change according to the first pattern and the luminance change according to the second pattern, the luminance change may be switched after a buffering time.

As a result, interference between the first signal and the second signal can be suppressed.

Further, it is an information communication method for transmitting a signal by a change in brightness, in which a determination step of determining a pattern of change in brightness by modulating a signal to be transmitted and a light emitting body change the brightness according to the determined pattern. The signal comprises a transmission step of transmitting the signal to be transmitted, the signal is composed of a plurality of large blocks, and each of the plurality of large blocks is a preamble for the first data and the first data. And a check signal for the first data, the first data is composed of a plurality of small blocks, the small blocks are a second data, a preamble for the second data, and the first. It may include a check signal for the data of 2.

As a result, data can be appropriately acquired by both the receiver that requires a blanking period and the receiver that does not require a blanking period.

Further, it is an information communication method in which a signal is transmitted by a change in brightness, and a determination step of determining a pattern of change in brightness by modulating a signal to be transmitted by each of a plurality of transmitters, and a determination step for each transmitter. A light emitter provided in a transmitter includes a transmission step of transmitting a signal to be transmitted by changing the brightness according to the determined pattern, and the transmission step transmits signals having different frequencies or protocols from each other. You may.

This makes it possible to suppress signal interference from a plurality of transmitters.

Further, it is an information communication method for transmitting a signal by changing the brightness, and a determination step of determining a pattern of the change in brightness by modulating a signal to be transmitted by each of a plurality of transmitters, and a determination step for each transmitter. A light emitting body provided in a transmitter includes a transmission step of transmitting a signal to be transmitted by changing the brightness according to the determined pattern, and in the transmission step, one of the plurality of transmitters is used. One transmitter may receive a signal transmitted from the other transmitter and transmit another signal in a manner that does not interfere with the received signal.

This makes it possible to suppress signal interference from a plurality of transmitters.

(Embodiment 3)
In this embodiment, each application example using a receiver such as a smartphone in the above-described first or second embodiment and a transmitter that transmits information as a blinking pattern such as an LED or an organic EL will be described.

FIG. 26 is a diagram showing an example of processing operations of the receiver, the transmitter, and the server in the third embodiment.

For example, the receiver 8142 configured as a smartphone acquires the position information indicating its own position and transmits the position information to the server 8141. The receiver 8142 acquires the position information when, for example, GPS is used or another signal is received. The server 8141 transmits the ID list associated with the position indicated by the position information to the receiver 8142. The ID list includes the ID and the information associated with the ID for each ID such as "abcd".

The receiver 8142 receives a signal from a transmitter 8143 configured as, for example, a lighting device. At this time, the receiver 8142 may be able to receive only a part of the ID (for example, "b") as the above-mentioned signal. In this case, the receiver 8142 searches the ID list for an ID including a part of the ID. If no unique ID is found, receiver 8142 further receives a signal from transmitter 8143 that includes other parts of that ID. As a result, the receiver 8142 acquires a larger portion (for example, "bc") of the ID. Then, the receiver 8142 searches the ID list again for an ID including a part of the ID (for example, "bc"). By performing such a search, the receiver 8142 can specify all of the IDs even if only a part of the IDs can be acquired. When the receiver 8142 receives a signal from the transmitter 8143, the receiver 8142 receives not only a part of the ID but also a check portion such as a CRC (Cyclic Redundancy Check).

FIG. 27 is a diagram showing an example of the operation of the transmitter and the receiver in the third embodiment.

For example, the transmitter 8165 configured as a television acquires an image and an ID (ID 1000) associated with the image from the control unit 8166. Then, the transmitter 8165 displays the image and transmits the ID (ID 1000) to the receiver 8167 by changing the brightness. By taking an image, the receiver 8167 receives the ID (ID 1000) and displays the information associated with the ID (ID 1000). Here, the control unit 8166 changes the image output to the transmitter 8165 to another image. At this time, the control unit 8166 also changes the ID output to the transmitter 8165. That is, the control unit 8166 outputs another ID (ID 1001) associated with the other image to the transmitter 8165 together with the other image. As a result, the transmitter 8165 displays other images and transmits another ID (ID 1001) to the receiver 8167 by changing the luminance. The receiver 8167 receives the other ID (ID 1001) and displays the information associated with the other ID (ID 1001) by taking an image.

FIG. 28 is a diagram showing an example of the operation of the transmitter, receiver, and server in the third embodiment.

For example, the transmitter 8185 configured as a smartphone displays information indicating, for example, a "coupon 100 yen discount" by changing the brightness of the portion of the display 8185a excluding the barcode portion 8185b, that is, by visible light communication. Send. Further, the transmitter 8185 causes the barcode portion 8185b to display the barcode without changing the brightness of the barcode portion 8185b. This barcode indicates the same information as the information transmitted by the visible light communication described above. Further, the transmitter 8185 displays characters or pictures indicating information transmitted by visible light communication, for example, the characters "coupon 100 yen discount" on the portion of the display 8185a excluding the barcode portion 8185b. By displaying such characters or pictures, the user of the transmitter 8185 can easily grasp what kind of information is being transmitted.

By taking an image, the receiver 8186 acquires the information transmitted by the visible light communication and the information indicated by the barcode, and transmits the information to the server 8187. The server 8187 determines whether or not the information is matched or related, and when it is determined that the information is matched or related, the server 8187 executes processing according to the information. Alternatively, the server 8187 transmits the determination result to the receiver 8186, and causes the receiver 8186 to execute the process according to the information.

The transmitter 8185 may transmit a part of the information indicated by the barcode by visible light communication. Further, the URL of the server 8187 may be indicated on the barcode. Further, the transmitter 8185 may acquire the ID as a receiver and transmit the ID to the server 8187 to acquire the information associated with the ID. The information associated with this ID is the same as the information transmitted by the above-mentioned visible light communication or the information indicated by the barcode. Further, the server 8187 may transmit an ID associated with the information (visible light communication information or barcode information) transmitted from the transmitter 8185 via the receiver 8186 to the transmitter 8185.

FIG. 29 is a diagram showing an example of the operation of the transmitter and the receiver in the third embodiment.

For example, the receiver 8183 captures a subject including a plurality of people 8197 and a streetlight 8195. The street lamp 8195 includes a transmitter 8195a that transmits information by changing the brightness. By this imaging, the receiver 8183 acquires an image in which the image of the transmitter 8195a appears as the above-mentioned emission line pattern. Further, the receiver 8183 acquires the AR object 8196a associated with the ID indicated by the emission line pattern from, for example, a server. Then, the receiver 8183 superimposes the AR object 8196a on the normal shooting image 8196 obtained by the normal shooting, and displays the normal shooting image 8196 on which the AR object 8196a is superposed.

(Summary of this embodiment)
The information communication method in the present embodiment is an information communication method in which a signal is transmitted by a change in luminance, and a determination step for determining a pattern of change in luminance by modulating the signal to be transmitted and a light emitter are determined. Including a transmission step of transmitting a signal to be transmitted by changing the luminance according to the pattern, the pattern of the luminance change has two luminances different from each other at arbitrary positions in a predetermined time width. It is a pattern in which one of the values appears, and in the determination step, the luminance change position, which is the luminance rising position or the luminance falling position in the time width, is different from each other for each of the signals to be transmitted which are different from each other. Moreover, the pattern of the luminance change is determined so that the integrated value of the luminance of the luminous body in the time width becomes the same value according to the preset luminance.

For example, for each of the signals "00", "01", "10", and "11" that are different from each other to be transmitted, the rising position (luminance change position) of the luminance is different from each other, and the predetermined time width is set. The pattern of luminance change is determined so that the integral value of the luminance of the illuminant in (unit time width) becomes the same value according to the predetermined brightness (for example, 99% or 1%). As a result, the brightness of the light emitter can be kept constant for each of the signals to be transmitted, flicker can be suppressed, and the receiver that images the light emitter is based on the brightness change position. , The pattern of the brightness change can be appropriately demodulated. Further, since the luminance change pattern is a pattern in which one of two different luminance values (luminance H (High) or luminance L (Low)) appears at any position in the unit time width, light emission is emitted. The brightness of the body can be changed continuously.

Further, the information communication method further includes an image display step of sequentially switching and displaying each of the plurality of images, and in the determination step, each time an image is displayed in the image display step, the displayed image is displayed. By modulating the identification information corresponding to the above as the signal to be transmitted, the pattern of the change in brightness with respect to the identification information is determined, and in the transmission step, the image is displayed every time the image is displayed in the image display step. The identification information may be transmitted by changing the brightness of the light emitter according to the pattern of the brightness change determined for the identification information corresponding to the image.

As a result, as shown in FIG. 27, for example, each time an image is displayed, the identification information corresponding to the displayed image is transmitted, so that the user receives the information to the receiver based on the displayed image. The identification information to be used can be easily selected.

Further, in the transmission step, each time an image is displayed in the image display step, the light emitter becomes brighter according to a pattern of luminance change determined for identification information corresponding to an image displayed in the past. The identification information may be transmitted by changing.

As a result, even if the receiver cannot receive the identification signal transmitted before the switching due to the switching of the displayed image, the image displayed in the past together with the identification information corresponding to the currently displayed image. Since the identification information corresponding to the above is also transmitted, the identification information transmitted before the switching can be appropriately received by the receiver again.

Further, in the determination step, each time an image is displayed in the image display step, the identification information corresponding to the displayed image and the time when the image is displayed are modulated as signals to be transmitted. Thereby, the identification information and the pattern of the brightness change with respect to the time are determined, and in the transmission step, each time the image is displayed in the image display step, the identification information and the time corresponding to the displayed image are obtained. The identification information and the time are transmitted by changing the brightness of the light emitter according to the pattern of the brightness change determined in the above-mentioned, and further determined for the identification information and the time corresponding to the image displayed in the past. The identification information and the time may be transmitted by changing the brightness of the light emitter according to the pattern of the change in brightness.

As a result, each time the image is displayed, a plurality of ID time information (information consisting of identification information and time) is transmitted, so that the receiver transmits the received multiple ID time information in the past. The identification signal that has been received and could not be received can be easily selected based on the time included in each of the ID time information.

Further, each of the light emitters has a plurality of regions that emit light, and the light of the regions adjacent to each other among the plurality of regions interferes with each other, and only one of the plurality of regions is determined. When the luminance changes according to the luminance change pattern, in the transmission step, only the region arranged at the end of the plurality of regions may change the luminance according to the determined luminance change pattern.

As a result, since only the region (light emitting part) arranged at the end changes the brightness, the influence on the brightness change due to the light from other regions is compared with the case where only the region arranged other than the end changes the brightness. Can be suppressed. As a result, the receiver can appropriately capture the pattern of the brightness change by shooting.

Further, when only two of the plurality of regions change the luminance according to the determined pattern of the luminance change, in the transmission step, the region arranged at the end of the plurality of regions and the region described above. The area adjacent to the area arranged at the end may change the luminance according to the determined pattern of the luminance change.

As a result, the brightness of the region arranged at the end (light emitting portion) and the region adjacent to the region arranged at the end (light emitting portion) change, so that the brightness of the regions separated from each other changes as compared with the case where the brightness changes. It is possible to keep a wide area in the range where the brightness changes continuously in space. As a result, the receiver can appropriately capture the pattern of the brightness change by shooting.

The information communication method in the present embodiment is an information communication method for acquiring information from a subject, and includes a position information transmission step for transmitting position information indicating the position of an image sensor used for photographing the subject, and the position information. Corresponds to the exposure line included in the image sensor in the list receiving step of receiving the ID list including a plurality of identification information associated with the position indicated by the image sensor and the image obtained by photographing the subject by the image sensor. The exposure time setting step for setting the exposure time of the image sensor and the image sensor taking a picture of the subject whose brightness changes at the set exposure time so that the emission line is generated according to the change in the brightness of the subject. An image acquisition step of acquiring an emission line image including the emission line, an information acquisition step of acquiring information by demodulating the data specified by the emission line pattern included in the acquired emission line image, and the like. It includes a search step of searching the ID list for the acquired identification information including the information.

As a result, for example, as shown in FIG. 26, since the ID list has been received in advance, even if the acquired information "bc" is only a part of the identification information, the appropriate identification information "" is based on the ID list. "Abcd" can be specified.

If the acquired identification information including the information is not uniquely specified in the search step, new information is acquired by repeating the image acquisition step and the information acquisition step, and the information communication is performed. The method may further include a re-search step of searching the ID list for identification information including the acquired information and the new information.

As a result, as shown in FIG. 26, for example, even if the acquired information "b" is only a part of the identification information and the identification information is not uniquely specified by the information alone, the new information "b" Since "c" is acquired, appropriate identification information "abcd" can be specified based on the new information and the ID list.

The information communication method in the present embodiment is an information communication method for acquiring information from a subject, and an image obtained by photographing the subject by an image sensor has a bright line corresponding to an exposure line included in the image sensor. An exposure time setting step for setting the exposure time of the image sensor so as to occur in response to a change in the brightness of the subject, and the image sensor taking a picture of the subject whose brightness changes with the set exposure time. An image acquisition step for acquiring an emission line image including the emission line, an information acquisition step for acquiring identification information by demodulating the data specified by the emission line pattern included in the acquired emission line image, and acquisition. The identification acquired in an ID list including a plurality of identification information associated with the transmission step of transmitting the identification information and the position information indicating the position of the image sensor and the position indicated by the position information. If there is no information, it includes an error receiving step of receiving error notification information for notifying an error.

As a result, if the acquired identification information is not in the ID list, the error notification information is received, so that the user of the receiver who has received the error notification information can use the information associated with the acquired identification information. You can easily figure out what you cannot get.

(Embodiment 4)
In this embodiment, an application example using a receiver such as a smartphone in the above-described first to fourth embodiments and a transmitter that transmits information as a blinking pattern such as an LED or an organic EL will be described.

FIG. 30 is a diagram showing an example of the operation of the transmitter and the receiver in the fourth embodiment.

The transmitter includes an ID storage unit 8361, a random number generation unit 8362, an addition unit 8363, a encryption unit 8364, and a transmission unit 8365. The ID storage unit 8361 stores the ID of the transmitter. The random number generation unit 8362 generates different random numbers at regular intervals. The addition unit 8363 combines the latest random numbers generated by the random number generation unit 8362 with the ID stored in the ID storage unit 8361, and outputs the result as an edit ID. The encryption unit 8364 generates an encrypted edit ID by encrypting the edit ID. The transmission unit 8365 transmits the encrypted edit ID to the receiver by changing the luminance.

The receiver includes a receiving unit 8366, a decoding unit 8637, and an ID acquisition unit 8368. The receiving unit 8366 receives the encrypted edit ID from the transmitter by imaging (visible light photography) the transmitter. The decryption unit 8367 restores the edit ID by decrypting the received encrypted edit ID. The ID acquisition unit 8368 acquires the ID by extracting the ID from the restored edit ID.

For example, the ID storage unit 8361 stores the ID "100", and the random number generation unit 8362 generates the latest random number "817" (Example 1). In this case, the addition unit 8363 generates and outputs the edit ID "100817" by combining the random number "817" with the ID "100". The encryption unit 8364 generates an encrypted edit ID "abced" by encrypting the edit ID "100817". The decryption unit 8367 of the receiver restores the edit ID "100817" by decrypting the encrypted edit ID "abced". Then, the ID acquisition unit 8368 extracts the ID "100" from the restored edit ID "100817". In other words, the ID acquisition unit 8368 acquires the ID "100" by deleting the last three digits of the edit ID.

Next, the random number generation unit 8362 generates a new random number "619" (Example 2). In this case, the addition unit 8363 generates and outputs the edit ID "100619" by combining the random number "619" with the ID "100". The encryption unit 8364 generates an encrypted editing ID "diffia" by encrypting the editing ID "100619". The decryption unit 8367 of the transmitter restores the edit ID "100619" by decrypting the encrypted edit ID "diffia". Then, the ID acquisition unit 8368 extracts the ID "100" from the restored edit ID "100619". In other words, the ID acquisition unit 8368 acquires the ID "100" by deleting the last three digits of the edit ID.

In this way, the transmitter does not simply encrypt the ID, but encrypts a combination of random numbers that are changed at regular intervals, so that the ID can be easily decrypted from the signal transmitted from the transmitter 8365. It can be prevented from being done. That is, when a simply encrypted ID is transmitted from the transmitter to the receiver several times, even if the ID is encrypted, if the ID is the same, the transmitter to the receiver. Since the transmitted signal is the same, the ID may be decrypted. However, in the example shown in FIG. 30, a random number changed at regular time intervals is combined with the ID, and the ID in which the random numbers are combined is encrypted. Therefore, even if the same ID is transmitted to the receiver several times, if the transmission timing of those IDs is different, the signal transmitted from the transmitter to the receiver can be different. As a result, it is possible to prevent the ID from being easily decrypted.

When the receiver shown in FIG. 30 acquires the encrypted edit ID, the receiver may transmit the encrypted edit ID to the server and acquire the ID from the server.

(Information at the station)
FIG. 31 shows an example of the usage pattern of the present invention in a train platform. A user holds a mobile terminal over an electronic bulletin board or lighting and acquires information displayed on the electronic bulletin board, train information at a station where the electronic bulletin board is installed, station premises information, etc. by visible light communication. Here, the information itself displayed on the electronic bulletin board may be transmitted to the mobile terminal by visible light communication, or the ID information corresponding to the electronic bulletin board is transmitted to the mobile terminal and the ID information acquired by the mobile terminal. You may get the information displayed on the electronic bulletin board by inquiring to the server. When the ID information is transmitted from the mobile terminal, the server transmits the content displayed on the electronic bulletin board to the mobile terminal based on the ID information. The train ticket information stored in the memory of the mobile terminal is compared with the information displayed on the electronic bulletin board, and when the ticket information corresponding to the user's ticket is displayed on the electronic bulletin board, the mobile terminal The display shows an arrow indicating the destination to the platform where the train to be boarded by the user arrives. When getting off, the route to the vehicle near the exit or the transfer route may be displayed. If a seat is designated, the route to that seat may be displayed. When displaying the arrow, it can be displayed more clearly by displaying the arrow using the same color as the color of the train line in the map or the train guidance information. In addition to displaying the arrow, the user's reservation information (home number, vehicle number, departure time, seat number) can also be displayed. By displaying the user's reservation information together, it is possible to prevent erroneous recognition. If the ticket information is stored in the server, contact the server from the mobile terminal to obtain the ticket information and compare it, or compare the ticket information with the information displayed on the electronic bulletin board on the server side. Allows information related to ticket information to be obtained. The target route may be estimated from the history of the transfer search by the user, and the route may be displayed. Further, not only the contents displayed on the electronic bulletin board but also the train information and the premises information of the station where the electronic bulletin board is installed may be acquired and compared. Information related to the user may be highlighted or rewritten to be displayed on the display of the electronic bulletin board on the display. If the user's boarding schedule is unknown, an arrow for guidance to the platform of each line may be displayed. When the station yard information is acquired, an arrow to guide the shop / restroom may be displayed on the display. The behavioral characteristics of the user may be managed in advance on the server, and when the user often stops at the shop / restroom in the station yard, an arrow guiding the shop / restroom may be displayed on the display. It is possible to reduce the amount of processing because the arrow that guides the shop / restroom is displayed only to the user who has the behavioral characteristic of stopping at the shop / restroom, and the arrow is not displayed to other users. .. The color of the arrow that guides you to the shop / restroom may be different from the color of the arrow that guides you to the home. When displaying both arrows at the same time, it is possible to prevent erroneous recognition by using different colors. Although an example of a train is shown in FIG. 31, it is possible to display the same configuration on an airplane, a bus, or the like.

(Coupon pop-up)
FIG. 32 shows an example in which coupon information acquired by visible light communication is displayed or a pop-up is displayed on the display of a mobile terminal when the user approaches the store. The user acquires the coupon information of the store from an electronic bulletin board system or the like by visible light communication using a mobile terminal. Next, when the user enters within a predetermined range from the store, the coupon information of the store or the pop-up is displayed. Whether or not the user has entered the predetermined range from the store is determined by using the GPS information of the mobile terminal and the store information included in the coupon information. Not limited to coupon information, ticket information may be used. Since it automatically alerts when a store where coupons and tickets can be used approaches, users can use coupons and tickets appropriately.

(Starting the operation application)
FIG. 33 shows an example in which a user acquires information from a home appliance by visible light communication using a mobile terminal. When ID information or information about the home appliance is acquired from the home appliance by visible light communication, an application for operating the home appliance is automatically started. FIG. 33 shows an example using a television. With such a configuration, it is possible to start an application for operating a home appliance simply by holding the mobile terminal over the home appliance.

(Database)
FIG. 34 shows an example of the configuration of the database held by the server that manages the ID transmitted by the transmitter.

The database has an ID-data table that holds data to be passed to inquiries using an ID as a key, and an access log table that holds a record of inquiries using an ID as a key. In the ID-data table, the ID sent by the transmitter, the data passed to the inquiry using the ID as the key, the conditions for passing the data, the number of times the access was made using the ID as the key, and the conditions are cleared and the data is passed. Have a number of times. The conditions for passing data include the date and time, the number of accesses, the number of successful accesses, the information of the terminal of the inquiry source (terminal model, application that made the inquiry, the current position of the terminal, etc.), and the user information of the inquiry source (user information of the inquiry source). Age, gender, occupation, nationality, language used, religion, etc.). By setting the number of successful accesses as a condition, a service method such as "1 yen per access, but no data is returned after 100 yen is possible" becomes possible. In the log table, when there is an access using the ID as a key, the ID, the ID of the requesting user, the time, other incidental information, whether or not the data is passed after clearing the conditions, and the passed data. Record the contents of.

(Communication protocol different for each zone)
FIG. 35 is a diagram showing an example of the operation of the transmitter and the receiver in the fourth embodiment.

The receiver 8420a receives zone information from the base station 8420h, recognizes which zone it is located in, and selects a reception protocol. The base station 8420h is configured as, for example, a communication base station of a mobile phone, a Wi-Fi access point, an IEMS transmitter, a speaker, or a wireless transmitter (Bluetooth (registered trademark), ZigBee, a specified low power wireless station, etc.). The receiver 8420a may specify the zone based on the position information obtained from GPS or the like. As an example, it is assumed that zone A communicates at a signal frequency of 9.6 kHz, and zone B communicates at a signal frequency of 15 kHz for ceiling lighting and 4.8 kHz for signage. At the position 8420j, the receiver 8420a recognizes that the current location is the zone A from the information of the base station 8420h, receives the signal at the signal frequency of 9.6 kHz, and receives the signal transmitted by the transmitters 8420b and 8420c. At position 8420l, the receiver 8420a recognizes that the current location is zone B from the information of the base station 8420i, and further, since the in-camera is directed upward, it is trying to receive a signal from the ceiling lighting. Is estimated, reception is performed at a signal frequency of 15 kHz, and the signal transmitted by the transmitters 8420e and 8420f is received. At the position 8420m, the receiver 8420a recognizes that the current location is zone B from above the base station 8420i, and further estimates that it is trying to receive the signal transmitted by the signage from the movement of protruding the out-camera, 4 It receives at a signal frequency of 8.8 kHz and receives the signal transmitted by the transmitter 8420 g. At position 8420k, the receiver 8420a receives signals from both base station 8420h and base station 8420i, and cannot determine whether the current location is zone A or zone B. Therefore, reception processing is performed at both 9.6 kHz and 15 kHz. I do. It should be noted that the part where the protocol differs depending on the zone is not limited to the frequency, but the server that inquires about the modulation method, signal format, and ID of the transmission signal may be different. The base stations 8420h and 8420i may transmit the protocol in the zone to the receiver, or may transmit only the ID indicating the zone, and the receiver may acquire the protocol information from the server using the zone ID as a key. good.

The transmitters 8420b to 8420f receive the zone ID and protocol information transmitted by the base stations 8420h and 8420i, and determine the signal transmission protocol. The transmitter 8420d capable of receiving the signals transmitted by both the base station 8420h and the base station 8420i utilizes the protocol of the zone of the base station having a stronger signal strength, or uses both protocols alternately.

(Zone recognition and service for each zone)
FIG. 36 is a diagram showing an example of the operation of the receiver and the transmitter in the fourth embodiment.

The receiver 8421a recognizes the zone to which its position belongs from the received signal. The receiver 8421a provides services (coupon distribution, point awarding, directions, etc.) defined for each zone. As an example, the receiver 8421a receives a signal transmitted from the left side of the transmitter 8421b and recognizes that it is in the zone A. Here, the transmitter 8421b may transmit a signal different depending on the transmission direction. Further, the transmitter 8421b may transmit a signal so that a different signal is received depending on the distance to the receiver by using a signal having a light emission pattern such as 2217a. Further, the receiver 8421a may recognize the positional relationship with the transmitter 8421b from the image pickup direction and size of the transmitter 8421b, and may recognize the zone in which the receiver 8421b is located.

Some of the signals indicating that they are located in the same zone may be shared. For example, the ID representing the zone A transmitted from the transmitter 8421b and the transmitter 8421c has the same first half. As a result, the receiver 8421a can recognize the zone in which it is located only by receiving the first half of the signal.

(Summary of this embodiment)
The information communication method in the present embodiment is an information communication method in which a signal is transmitted by a change in luminance, and is a determination step for determining a pattern of a plurality of luminance changes by modulating each of a plurality of signals to be transmitted. , Each of the plurality of light emitters transmits a transmission target signal corresponding to the one of the above patterns by changing the luminance according to any one of the determined patterns of the plurality of luminance changes. In the transmission step, each of the two or more light emitters of the plurality of light emitters has two types having different brightnesses for each predetermined time unit for the light emitter. Luminance changes at different frequencies so that one of the two or more light emitters is output and the predetermined time units for each of the two or more light emitters are different from each other. do.

As a result, the brightness of each of the two or more light emitters (for example, a transmitter configured as a lighting device) changes at different frequencies from each other, so that the signal to be transmitted from those light emitters (for example, the ID of the light emitter). The receiver that receives the signal can easily distinguish and acquire the signals to be transmitted.

Further, in the transmission step, the luminance of each of the plurality of light emitters changes at any one of at least four kinds of frequencies, and the light emitters of two or more of the plurality of light emitters change their brightness. The brightness may change at the same frequency. For example, in the transmission step, when the plurality of light emitting bodies are projected on the light receiving surface of the image sensor for receiving the plurality of transmission target signals, all the light emitting bodies adjacent to each other on the light receiving surface are projected. Each of the plurality of light emitters changes its brightness so that the frequency of the change in brightness differs between the two.

As a result, if there are at least four types of frequencies used for the brightness change, even if there are two or more light emitters whose brightness changes at the same frequency, that is, the number of frequency types is the number of multiple light emitters. Based on the four-color problem or the four-color theorem, it is possible to ensure that the frequencies of luminance changes are different among all emitters adjacent to each other on the light receiving surface of the image sensor, even if less than. As a result, the receiver can easily distinguish and acquire each of the signals to be transmitted transmitted from the plurality of light emitters.

Further, in the transmission step, each of the plurality of light emitters may transmit the signal to be transmitted by changing the luminance at a frequency specified by the hash value of the signal to be transmitted.

As a result, the luminance of each of the plurality of light emitters changes at the frequency specified by the hash value of the signal to be transmitted (for example, the ID of the light emitter). Therefore, when the receiver receives the signal to be transmitted, the receiver receives the signal to be transmitted. It is possible to determine whether or not the frequency specified by the actual luminance change and the frequency specified by the hash value match. That is, the receiver can determine whether or not there is an error in the received signal (for example, the ID of the light emitter).

Further, the information communication method further calculates a frequency corresponding to the signal of the transmission target as the first frequency from the signal of the transmission target stored in the signal storage unit according to a predetermined function. A calculation step, a frequency determination step for determining whether or not the second frequency stored in the frequency storage unit and the calculated frequency 1 match, and the first frequency and the second frequency. When it is determined that the frequencies do not match, a frequency error notification step for notifying an error is included, and when it is determined that the first frequency and the second frequency match, the determination step is included. Then, the pattern of the brightness change is determined by modulating the signal of the transmission target stored in the signal storage unit, and in the transmission step, any one of the plurality of light emitters is used. The signal of the transmission target stored in the signal storage unit may be transmitted by changing the brightness at the first frequency according to the determined pattern.

As a result, it is determined whether or not the frequency stored in the frequency storage unit and the frequency calculated from the transmission target signal stored in the signal storage unit (ID storage unit) match, and if they do not match. If it is determined, an error is notified, so that it is possible to easily detect an abnormality in the signal transmission function by the light emitter.

Further, the information communication method further includes a check value calculation step of calculating a first check value from a signal to be transmitted stored in the signal storage unit according to a predetermined function, and a check value storage unit. The check value determination step for determining whether or not the stored second check value and the calculated first check value match, and the first check value and the second check value are If it is determined that they do not match, a check value error notification step for notifying an error is included, and if it is determined that the first check value and the second check value match, the determination step is included. Then, the pattern of the brightness change is determined by modulating the signal of the transmission target stored in the signal storage unit, and in the transmission step, any one of the plurality of light emitters is used. The signal of the transmission target stored in the signal storage unit may be transmitted by changing the brightness according to the determined pattern.

As a result, it is determined whether or not the check value stored in the check value storage unit and the check value calculated from the transmission target signal stored in the signal storage unit (ID storage unit) match. If it is determined that they do not match, an error is notified, so that it is possible to easily detect an abnormality in the signal transmission function by the light emitter.

Further, the information communication method in the present embodiment is an information communication method for acquiring information from a subject, and corresponds to a plurality of exposure lines included in the image sensor in an image obtained by photographing the subject by an image sensor. An exposure time setting step for setting the exposure time of the image sensor so that a plurality of emission lines are generated in response to a change in the brightness of the subject, and the exposure time set for the subject whose brightness is changed by the image sensor. The information is acquired by taking a picture with the image acquisition step of acquiring the emission line image including the plurality of emission lines and demolating the data specified by the pattern of the plurality of emission lines included in the acquired emission line image. The information acquisition step is included, and a frequency specifying step for specifying the frequency of the brightness change of the subject based on the pattern of the plurality of emission lines included in the acquired emission line image is included. For example, in the frequency specifying step, a plurality of header patterns, which are a plurality of predetermined patterns for indicating headers, which are included in the plurality of emission line patterns, are specified, and the number of pixels between the plurality of header patterns is specified. The frequency corresponding to the above is specified as the frequency of the change in the brightness of the subject.

As a result, the frequency of the change in the brightness of the subject is specified. Therefore, when a plurality of subjects having different frequencies of the change in the brightness are photographed, the information from those subjects can be easily distinguished and acquired.

Further, in the image acquisition step, by photographing a plurality of subjects whose luminance changes, the emission line image including a plurality of patterns represented by the plurality of emission lines is acquired, and in the information acquisition step, the acquisition is acquired. When a part of each of the plurality of patterns included in the emission line image overlaps, the plurality of patterns are demodulated by demodulating the data specified by the part excluding the part from each of the plurality of patterns. Information may be obtained from each of the patterns.

As a result, since the data is not demodulated from the portion where the plurality of patterns (plurality of emission line patterns) overlap, it is possible to prevent the acquisition of erroneous information.

Further, in the image acquisition step, a plurality of emission line images are acquired by photographing the plurality of subjects a plurality of times at different timings, and in the frequency specifying step, each emission line image is included in the emission line image. The frequency for each of the plurality of patterns is specified, and in the information acquisition step, a plurality of patterns in which the same frequency is specified are searched from the preceding plurality of emission line images, and the searched plurality of patterns are combined and combined. Information may be acquired by demodulating the data specified by the plurality of patterns.

As a result, a plurality of patterns (plurality of emission line patterns) in which the same frequency is specified are searched from a plurality of emission line images, the plurality of searched patterns are combined, and information is acquired from the plurality of combined patterns. Therefore, even when a plurality of subjects are moving, information from the plurality of subjects can be easily distinguished and acquired.

Further, in the information communication method, the identification information of the subject included in the information acquired in the information acquisition step and the frequency identification are further specified for the server in which the frequency is registered for each of the identification information. A transmission step of transmitting specific frequency information indicating the frequency specified in the step, and a related information acquisition step of acquiring related information associated with the identification information and the frequency indicated by the specific frequency information from the server. May include.

As a result, the identification information (ID) acquired based on the brightness change of the subject (transmitter) and the related information associated with the frequency of the brightness change are acquired. Therefore, by changing the frequency of the brightness change of the subject and updating the frequency registered in the server to the changed frequency, the receiver that acquired the identification information before the frequency change acquires the related information from the server. You can prevent that. That is, by changing the frequency registered in the server in accordance with the change in the frequency of the change in the brightness of the subject, the receiver that has acquired the subject identification information in the past can acquire the related information from the server indefinitely. It is possible to prevent it from becoming a state.

Further, the information communication method is further acquired in the identification information acquisition step of acquiring the identification information of the subject by extracting a part from the information acquired in the information acquisition step, and the information acquisition step. The information may include a set frequency specifying step that specifies the number indicated by the remaining part other than the part as the set frequency of the luminance change set for the subject.

As a result, the information obtained from the patterns of the plurality of emission lines can include the identification information of the subject and the set frequency of the luminance change set for the subject without depending on each other. The degree of freedom can be increased.

(Embodiment 5)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

(Dissemination of visible light communication to humans)
FIG. 37 is a diagram showing an example of the operation of the transmitter according to the fifth embodiment.

As shown in FIG. 37 (a), the light emitting unit of the transmitter 8921a repeats blinking visible to humans and visible light communication. By blinking so that it can be visually recognized by humans, it is possible to inform humans that visible light communication is possible. The user notices that visible light communication is possible because the transmitter 8921a is blinking, performs visible light communication with the receiver 8921b toward the transmitter 8921a, and registers the user of the transmitter 8921a.

That is, the transmitter in the present embodiment alternately repeats a step in which the light emitting body transmits a signal by changing the brightness and a step in which the light emitting body blinks so as to be visually recognized by a human eye.

As shown in FIG. 37 (b), the transmitter may be provided with a visible light communication unit and a blinking unit (communication status display unit) separately.

By operating the transmitter as shown in FIG. 37 (c), it is possible to make the light emitting portion appear to be blinking to a human while performing visible light communication. That is, the transmitter alternately repeats, for example, high-intensity visible light communication having a brightness of 75% and low-intensity visible light communication having a brightness of 1%. For example, when an abnormality occurs in the transmitter and a signal different from the usual one is transmitted, the operation shown in FIG. 37 (c) is performed to call attention to the user without stopping the visible light communication. Can be done.

(Example of application to directions)
FIG. 38 is a diagram showing an example of an application example of the transmission / reception system according to the fifth embodiment.

The receiver 8955a receives, for example, the transmission ID of the transmitter 8955b configured as a guide plate, acquires the map data displayed on the guide plate from the server, and displays it. At this time, the server may transmit an advertisement suitable for the user of the receiver 8955a, and the receiver 8955a may also display this advertisement information. The receiver 8955a displays the route from the current location to the location specified by the user.

(Example of application to usage log storage and analysis)
FIG. 39 is a diagram showing an example of an application example of the transmission / reception system according to the fifth embodiment.

The receiver 8957a receives the ID transmitted by the transmitter 8957b configured as a signboard, for example, and acquires and displays the coupon information from the server. The receiver 8957a is a server for subsequent user actions such as saving a coupon, moving to a store displayed on the coupon, shopping at the store, and leaving without saving the coupon. Store in 8957c. Thereby, the subsequent behavior of the user who acquired the information from the signboard 8957b can be analyzed, and the advertising value of the signboard 8957b can be estimated.

(Application example for screen sharing)
FIG. 40 is a diagram showing an example of an application example of the transmission / reception system according to the fifth embodiment.

For example, the transmitter 8960b configured as a projector or a display transmits information for wirelessly connecting to itself (SSID, password for wireless connection, IP address, password for operating the transmitter). Alternatively, a key ID for accessing this information is transmitted. For example, the receiver 8960a configured as a smartphone, a tablet, a laptop computer, or a camera receives the signal transmitted from the transmitter 8960b, acquires the information, and establishes a wireless connection with the transmitter 8960b. This wireless connection may be connected via a router, or may be directly connected by Wi-Fi Direct, Bluetooth (registered trademark), Wireless Home Digital Interface, or the like. The receiver 8960a transmits the screen displayed by the transmitter 8960b. This makes it possible to easily display the image of the receiver on the transmitter.

When the transmitter 8960b is connected to the receiver 8960a, the transmitter 8960b informs the receiver 8960a that a password is required in addition to the information transmitted by the transmitter in order to display the screen, and the correct password is used. If is not sent, the sent screen may not be displayed. At this time, the receiver 8960a displays a password input screen such as 8960d, and causes the user to input the password.

Although the information communication method according to one or more embodiments has been described above based on the embodiment, the present invention is not limited to this embodiment. As long as it does not deviate from the gist of the present invention, a form in which various modifications conceived by those skilled in the art are applied to the present embodiment or a form constructed by combining components in different embodiments is also within the scope of one or a plurality of embodiments. May be included within.

Further, as shown in FIG. 41, the information communication method according to one aspect of the present invention may be applied.

FIG. 41 is a diagram showing an example of an application example of the transmission / reception system according to the fifth embodiment.

A camera configured as a receiver for visible light communication normally performs imaging in the imaging mode (Step 1). By this imaging, the camera acquires an image file composed of a format such as EXIF (Exchangeable image file format). Next, the camera performs imaging in the visible light communication imaging mode (Step 2). The camera acquires a signal (visible light communication information) transmitted by visible light communication from the transmitter which is the subject, based on the pattern of the emission line in the image obtained by this imaging (Step 3). Further, the camera treats the signal (received information) as a key and accesses the server to acquire the information corresponding to the key from the server (Step 4). Then, the camera is a signal transmitted from the subject by visible light communication (visible light reception data), information acquired from the server, and data indicating the position where the transmitter, which is the subject, is projected in the image shown by the image file. And, data indicating the time when the signal transmitted by the visible light communication is received (time in the moving image) and the like are saved as the metadata in the above-mentioned image file, respectively. When a plurality of transmitters are projected as subjects in an image (image file) obtained by imaging, the camera displays some metadata corresponding to the transmitters for each transmitter. Save to file.

When displaying the image represented by the above-mentioned image file, the display or projector configured as a transmitter of visible light communication transmits a signal corresponding to the metadata contained in the image file by visible light communication. For example, the display or projector may transmit the metadata itself by visible light communication, or may transmit a signal associated with the transmitter projected in the image as a key.

A mobile terminal (smartphone) configured as a receiver for visible light communication receives a signal transmitted by visible light communication from the display or projector by capturing an image of the display or projector. If the received signal is the above-mentioned key, the mobile terminal uses the key to acquire the metadata of the transmitter associated with the key from the display, the projector, or the server. Further, if the received signal is a signal (visible light reception data or visible light communication information) transmitted by visible light communication from an existing transmitter, the mobile terminal has its visible light from a display, a projector or a server. Acquires information corresponding to received light data or visible light communication information.

(Summary of the present embodiment, etc.)
The information communication method in the present embodiment is an information communication method for acquiring information from a subject, and each exposure included in the image sensor on an image obtained by photographing the first subject, which is the subject, by the image sensor. The first exposure time setting step for setting the first exposure time of the image sensor and the image sensor have the brightness so that a plurality of emission lines corresponding to the lines are generated in response to the change in the brightness of the first subject. A first emission line image acquisition step of acquiring a first emission line image which is an image including the plurality of emission lines by photographing the changing first subject at a set first exposure time. The first information acquisition step of acquiring the first transmission information by demodulating the data specified by the pattern of the plurality of emission lines included in the acquired first emission line image, and the first transmission information. Includes a door control step that causes the door open / close drive device to open the door by transmitting a control signal after the acquisition.

As a result, the receiver equipped with the image sensor can be used like a door key, and a special electronic lock can be eliminated. As a result, communication can be performed between various devices including devices having low computing power.

Further, the information communication method is a second image including a plurality of emission lines by the image sensor taking a picture of a second subject whose brightness changes with the set first exposure time. The second transmission information is acquired by the second emission line image acquisition step of acquiring the emission line image and the data specified by the plurality of emission line patterns included in the acquired second emission line image. The second information acquisition step includes an approach determination step of determining whether or not the receiving device provided with the image sensor is approaching the door based on the acquired first and second transmission information. In the door control step, the control signal may be transmitted when it is determined that the receiving device is approaching the door.

As a result, the door can be opened only when the receiver (receiver) approaches the door, that is, only at an appropriate timing.

Further, in the information communication method, a second exposure time setting step for setting a second exposure time longer than the first exposure time and the image sensor set a third subject. The normal image acquisition step of acquiring a normal image on which the third subject is projected by taking a picture with the second exposure time is included, and the normal image acquisition step includes a region including the optical black of the image sensor. For each of the plurality of exposure lines in the above, the charge is read out after a predetermined time has elapsed from the time when the charge is read out to the exposure line adjacent to the exposure line, and the first emission line image acquisition step is performed. Then, without using the optical black for reading the charge, the charge is read out to the exposure line next to the exposure line for each of the plurality of exposure lines in the region other than the optical black in the image sensor. The charge may be read out after a lapse of a time longer than the predetermined time from the time when it is broken.

As a result, when the first emission line image is acquired, the charge reading (exposure) for the optical black is not performed, so that the charge reading (exposure) for the effective pixel region, which is a region other than the optical black in the image sensor, is performed. The time required can be increased. As a result, the time for receiving the signal in the effective pixel region can be lengthened, and many signals can be acquired.

Further, in the information communication method, the length of the plurality of emission lines included in the first emission line image in the direction perpendicular to each of the plurality of emission lines is less than a predetermined length. The length determination step for determining the presence or absence, and when it is determined that the length of the pattern is less than the predetermined length, the frame rate of the image sensor is set to the first emission line. The frame rate change step of changing to a second frame rate slower than the first frame rate when the image is acquired, and the image sensor changing the brightness of the first subject at the second frame rate. A third emission line image acquisition step of acquiring a third emission line image, which is an image including a plurality of emission lines, by taking a picture at the set first exposure time, and the acquired third emission line image. It may include a third information acquisition step of acquiring the first transmission information by demolishing the data specified by the pattern of the plurality of emission lines included in the emission line image.

As a result, if the signal length indicated by the bright line pattern (bright line region) included in the first bright line image is less than, for example, one block of the transmitted signal, the frame rate is reduced and the bright line is renewed. The image is acquired as a third emission line image. As a result, the length of the emission line pattern included in the third emission line image can be lengthened, and the transmitted signal can be acquired for one block.

Further, the information communication method further includes a ratio setting step for setting the ratio of the vertical width and the horizontal width of the image obtained by the image sensor, and the first luminance image acquisition step is based on the set ratio. The clipping determination step of determining whether or not the end of the image in the direction perpendicular to each exposure line is clipped, and the ratio set in the ratio setting step when it is determined that the end is clipped. The first emission line of the non-clipping ratio by the ratio changing step of changing the edge to the non-clipping ratio, which is the ratio at which the edges are not clipped, and the image sensor taking a picture of the first subject whose brightness changes. It may include an acquisition step to acquire an image.

As a result, for example, when the ratio of the horizontal width to the vertical width of the effective pixel area of the image sensor is 4: 3, the ratio of the horizontal width to the vertical width of the image is set to 16: 9, and a bright line along the horizontal direction appears. That is, when the exposure line is along the horizontal direction, it is determined that the upper end and the lower end of the above-mentioned image are clipped. That is, it is determined that the edge of the first emission line image is missing. In this case, the ratio of the image is changed to, for example, 4: 3, which is a ratio that is not clipped. As a result, it is possible to prevent the edge of the first emission line image from being missing, and it is possible to acquire a lot of information from the first emission line image.

Further, in the information communication method, a compression step of generating a compressed image by further compressing the first emission line image in a direction parallel to each of the plurality of emission lines included in the first emission line image. And the compressed image transmission step of transmitting the compressed image may be included.

As a result, the first emission line image can be appropriately compressed without losing the information indicated by the plurality of emission lines.

Further, the information communication method further determines that the receiving device including the image sensor has been moved in a predetermined mode and a gesture determination step for determining whether or not the receiver has been moved in the predetermined mode. Occasionally, it may include an activation step of activating the image sensor.

As a result, the image sensor can be easily activated only when necessary, and the power consumption efficiency can be improved.

(Embodiment 6)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

FIG. 42 is a diagram showing an application example of the transmitter and the receiver in the sixth embodiment.

The robot 8970 has, for example, a function as a self-propelled vacuum cleaner and a function as a receiver in each of the above embodiments. The lighting devices 8971a and 8971b each have a function as a transmitter in each of the above embodiments.

For example, the robot 8970 cleans the room while moving, and photographs the lighting device 8971a that illuminates the room. The lighting device 8971a transmits the ID of the lighting device 8971a by changing the brightness. As a result, the robot 8970 receives the ID from the lighting device 8971a and estimates its own position (self-position) based on the ID, as in each of the above embodiments. That is, the robot 8970 moves and self-positions based on the detection result by the 9-axis sensor, the relative position of the lighting device 8971a reflected in the image obtained by shooting, and the absolute position of the lighting device 8971a specified by the ID. Is estimated.

Further, when the robot 8970 moves away from the lighting device 8971a, the robot 8970 transmits a signal (turning off command) instructing the lighting device 8971a to turn off the light. For example, the robot 8970 transmits a turn-off command when it is separated from the lighting device 8971a by a predetermined distance. Alternatively, the robot 8970 transmits a turn-off command to the lighting device 8971a when the lighting device 8971a does not appear in the image obtained by shooting, or when another lighting device appears in the image. When the lighting device 8971a receives the turn-off command from the robot 8970, the lighting device 8971a turns off in response to the turn-off command.

Next, the robot 8970 detects that it has approached the lighting device 8971b based on the estimated self-position while moving and cleaning. That is, the robot 8970 holds information indicating the position of the lighting device 8971b, and when the distance between the self-position and the position of the lighting device 8971b becomes equal to or less than a predetermined distance, the lighting device 8971b Detects that you are approaching. Then, the robot 8970 transmits a signal (lighting command) instructing the lighting device 8971b to light. When the lighting device 8971b receives a lighting command, it lights up in response to the lighting command.

As a result, the robot 8970 can easily perform cleaning by brightening only the surroundings of the robot while moving.

FIG. 43 is a diagram showing an application example of the transmitter and the receiver in the sixth embodiment.

The lighting device 8974 has a function as a transmitter in each of the above embodiments. The lighting device 8974 illuminates a line bulletin board 8975 at, for example, a railway station while changing the brightness. The receiver 8973 pointed at the route bulletin board 8975 by the user photographs the route bulletin board 8975. As a result, the receiver 8973 acquires the ID of the route bulletin board 8975, and acquires the detailed information about each route described in the route bulletin board 8975, which is the information associated with the ID. Then, the receiver 8973 displays the guide image 8973a showing the detailed information. For example, the guide image 8973a shows the distance to the line described on the line bulletin board 8975, the direction toward the line, and the time when the next train arrives on the line.

Here, the receiver 8973 displays the supplementary guide image 8973b when the guide image 8973a is touched by the user. The supplementary guide image 8973b allows the user to select any one of, for example, a railway timetable, information on a line different from the line shown by the guide image 8973a, and detailed information on the station. It is an image to be displayed accordingly.

(Embodiment 7)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

(Reception of signals from multiple directions by multiple light receiving units)
FIG. 44 is a diagram showing an example of the receiver according to the seventh embodiment.

For example, the receiver 9020a configured as a wristwatch includes a plurality of light receiving units. For example, as shown in FIG. 44, the receiver 9020a is arranged near the light receiving portion 9020b arranged at the upper end of the rotating shaft supporting the long hand and the short hand of the wristwatch and the character indicating 12 o'clock on the peripheral portion of the wristwatch. It is provided with a light receiving unit 9020c. The light receiving unit 9020b receives light directed toward the light receiving unit 9020b along the direction of the above-mentioned rotation axis, and the light receiving unit 9020c receives light directed toward the light receiving unit 9020c along the direction connecting the rotation axis and the character indicating 12 o'clock. Receive. As a result, when the receiver 9020a is held in front of the chest as when the user confirms the time, the light receiving unit 9020b can receive light from above. As a result, the receiver 9020a can receive the signal from the ceiling lighting. Further, when the receiver 9020a is held in front of the chest as when the user confirms the time, the light receiving unit 9020c can receive light from the front direction. As a result, the receiver 9020a can receive the signal from the signage or the like on the front surface.

By providing these light receiving units 9020b and 9020c with directivity, it is possible to receive signals without interference even when a plurality of transmitters are located close to each other.

(Directions using a wristwatch-type display)
FIG. 45 is a diagram showing an example of the receiving system according to the seventh embodiment.

For example, the receiver 9023b configured as a wristwatch is connected to the smartphone 9022a via wireless communication such as Bluetooth (registered trademark). The dial of the receiver 9023b is composed of a display such as a liquid crystal display, and can display information other than the time. The smartphone 9022a recognizes the current location from the signal received by the receiver 9023b, and displays the route and distance to the destination on the display surface of the receiver 9023b.

FIG. 46 is a diagram showing an example of the signal transmission / reception system according to the seventh embodiment.

The signal transmission / reception system includes a smartphone (smartphone) which is a multifunctional mobile phone, an LED light emitter which is a lighting device, a home electric appliance such as a refrigerator, and a server. The LED light emitter performs communication using BTLE (Bluetooth (registered trademark) Low Energy) and visible light communication using an LED (Light Emitting Diode). For example, the LED light emitter controls the refrigerator or communicates with the air conditioner by BTLE. In addition, the LED light emitter controls the power supply of a microwave oven, an air purifier, a television (TV), or the like by visible light communication.

The television is provided with, for example, a solar power generation element, and the solar power generation element is used as an optical sensor. That is, when the LED light emitter transmits a signal due to the change in brightness, the television detects the change in brightness of the LED light emitter by the change in the electric power generated by the photovoltaic power generation element. Then, the television acquires the signal transmitted from the LED light emitter by demodulating the signal indicated by the detected luminance change. When the signal is a command indicating power ON, the television switches its main power to ON, and when the signal is a command indicating power OFF, the television switches its main power to OFF.

In addition, the server can communicate with the air conditioner via a router and a specified low power radio station (extra small). Further, since the air conditioner can communicate with the LED light emitter via BTLE, the server can communicate with the LED light emitter. Therefore, the server can switch the power of the TV to ON and OFF via the LED light emitter. Further, the smartphone can control the power supply of the TV via the server by communicating with the server via, for example, Wi-Fi (Wireless Fidelity).

As shown in FIG. 46, in the information communication method in the present embodiment, the mobile terminal (smartphone) uses a wireless communication (BTLE, Wi-Fi, etc.) different from the visible light communication to control signals (transmission data string or user command). ) To the lighting device (light emitter), the visible light communication step in which the lighting device performs visible light communication by changing the brightness according to the control signal, and the controlled device (microwaver, etc.) Includes an execution step of detecting a change in brightness of the lighting device, acquiring a control signal by demodulating the signal specified by the detected change in brightness, and executing processing according to the control signal. As a result, even if the mobile terminal cannot change the brightness for visible light communication, the lighting device can change the brightness instead of the mobile terminal by wireless communication, and the controlled device can be appropriately controlled. can do. The mobile terminal may be a wristwatch instead of a smartphone.

(Reception without interference)
FIG. 47 is a flowchart showing a reception method excluding interference in the seventh embodiment.

First, a start is performed in step 9001a to confirm whether or not there is a periodic change in the intensity of the light received in step 9001b, and if YES, the process proceeds to step 9001c. If NO, the process proceeds to step 9001d, the lens of the light receiving unit is wide-angled to receive a wide range of light, and the process returns to step 9001b. It is confirmed whether the signal can be received in step 9001c, and if YES, the process proceeds to step 9001e, the signal is received, and the process ends in step 9001g. If NO, the process proceeds to step 9001f, the lens of the light receiving unit is telephoto, light in a narrow range is received, and the process returns to step 9001c.

By this method, it is possible to receive signals from transmitters in a wide direction while eliminating signal interference from a plurality of transmitters.

(Estimation of transmitter orientation)
FIG. 48 is a flowchart showing a method of estimating the orientation of the transmitter according to the seventh embodiment.

First, start in step 9002a, set the lens of the light receiving part to the maximum telephoto in step 9002b, check whether there is a periodic change in the intensity of the light received in step 9002c, and if YES, step. Proceed to 9002d. If NO, the process proceeds to step 9002e, the lens of the light receiving unit is wide-angled to receive a wide range of light, and the process returns to step 9002c. In step 9002d, the signal is received, in step 9002f, the lens of the light receiving unit is set to the maximum telephoto, the light receiving direction is changed along the boundary of the light receiving range, the direction in which the light receiving intensity is maximized is detected, and the transmitter detects the direction. It is presumed to be in the direction and ends in step 9002d.

By this method, the direction in which the transmitter is present can be estimated. The maximum wide-angle lens may be set first, and then the telephoto lens may be gradually set.

(Start of reception)
FIG. 49 is a flowchart showing a method of starting reception in the seventh embodiment.

First, start in step 9003a, confirm in step 9003b whether a signal from a base station such as Wi-Fi, Bluetooth (registered trademark), or IMES has been received, and if YES, proceed to step 9003c. If NO, the process returns to step 9003b. In step 9003c, it is confirmed whether or not the base station is registered in the receiver or server as a trigger for starting reception. If YES, the process proceeds to step 9003d, signal reception is started, and the process ends in step 9003e. .. If NO, the process returns to step 9003b.

By this method, reception can be started without the user performing a reception start operation. In addition, power consumption can be suppressed as compared with constant reception.

(Generation of ID using information from other media)
FIG. 50 is a flowchart showing a method of generating an ID in combination with information of another medium according to the seventh embodiment.

First, start in step 9004a, and the carrier communication network connected in step 9004b, the ID of Wi-Fi, Bluetooth (registered trademark), etc., or the location information obtained from the above ID, the location information obtained from GPS, etc. To the high-order bit ID index server. In step 9004c, the high-order bit of the visible light ID is received from the high-order bit ID index server, and in step 9004d, the signal from the transmitter is received as the low-order bit of the visible light ID. In step 9004e, the upper bit and the lower bit of the visible light ID are combined and transmitted to the ID resolution server, and the process ends in step 9004f.

By this method, the high-order bits commonly used in the vicinity of the receiver can be obtained, and the amount of data transmitted by the transmitter can be reduced. In addition, the speed at which the receiver receives can be increased.

The transmitter may transmit both the high-order bit and the low-order bit. In this case, the receiver using this method can synthesize the ID when the lower bits are received, and the receiver not using this method obtains the ID by receiving the entire ID from the transmitter. ..

(Selection of reception method by frequency separation)
FIG. 51 is a flowchart showing a method of selecting a reception method by frequency separation in the seventh embodiment.

First, it is started in step 9005a, and the optical signal received in step 9005b is applied to a frequency filter circuit, or discrete Fourier series expansion is performed to perform frequency decomposition. It is confirmed in step 9005c whether or not a low frequency component is present, and if YES, the process proceeds to step 9005d, the signal expressed in the low frequency region such as frequency modulation is decoded, and the process proceeds to step 9005e. If NO, the process proceeds to step 9005e. In step 9005e, it is confirmed whether or not the base station is registered in the receiver or server as a trigger for starting reception, and if YES, the process proceeds to step 9005f, and the signal expressed in the high frequency region such as pulse position modulation is performed. Is decoded, and the process proceeds to step 9005 g. If NO, the process proceeds to step 9005 g. Reception of the signal is started in step 9005g and ends in step 9005h.

By this method, it is possible to receive a signal modulated by a plurality of modulation methods.

(Signal reception when the exposure time is long)
FIG. 52 is a flowchart showing a signal receiving method when the exposure time is long in the seventh embodiment.

First, start in step 9030a, and if the sensitivity can be set in step 9030b, set the sensitivity to the maximum. If the exposure time can be set in step 9030c, it is set to a shorter time than the normal shooting mode. In step 9030d, two images are taken and the difference in brightness is obtained. If the position or direction of the image pickup unit changes during the acquisition of two images, the change is canceled and an image as if taken from the same position / direction is generated and the difference is obtained. In step 9030e, the value obtained by averaging the luminance values in the direction parallel to the exposure line of the difference image or the captured image is obtained. The averaged values in step 9030f are arranged in a direction perpendicular to the exposure line, and a discrete Fourier transform is performed to recognize whether or not there is a peak near a predetermined frequency in step 9030g, and the process ends in step 9030h.

By this method, the signal can be received even when the exposure time is long, such as when the exposure time cannot be set or when a normal image is simultaneously captured.

When the exposure time is set automatically, when the camera is pointed at the transmitter configured as lighting, the exposure time is set from 1/60 second to 1/480 second by the automatic exposure compensation function. If the exposure time cannot be set, a signal is received under these conditions. In the experiment, when the illumination is blinked periodically, if the time of one cycle is about 1/16 or more of the exposure time, stripes can be visually recognized in the direction perpendicular to the exposure line, and the blinking cycle is determined by image processing. I was able to recognize it. At this time, since the brightness of the part where the illumination is reflected is too high and it is difficult to confirm the stripes, it is better to obtain the signal cycle from the part where the illumination light is reflected.

When a method such as a frequency shift keying method or a frequency multiplex modulation method, in which the light emitting part is periodically turned on and off, is used, even if the modulation frequency is the same, a human being is used, as compared with the case where the pulse position modulation method is used. The flicker is hard to see for the user, and the flicker is hard to appear in the video taken with the video camera. Therefore, a low frequency can be used as the modulation frequency. Since the time resolution of human vision is about 60 Hz, a frequency higher than this frequency can be used as the modulation frequency.

When the modulation frequency is an integral multiple of the image pickup frame rate of the receiver, the pixels at the same position in the two images are imaged when the optical pattern of the transmitter is in the same phase, so a bright line appears in the difference image. It is difficult to receive. Since the image frame rate of the receiver is usually 30 fps, if the modulation frequency is set to a frequency other than an integral multiple of 30 Hz, reception is easy. In addition, since there are various imaging frame rates of the receiver, two modulation frequencies that are mutually exclusive are assigned to the same signal, and the transmitter transmits by alternately using the two modulation frequencies to receive the signal. The machine can easily restore the signal by receiving at least one signal.

FIG. 53 is a diagram showing an example of a dimming (adjusting the brightness) method of the transmitter.

By adjusting the ratio of the section with high brightness and the section with low brightness, the average brightness changes and the brightness can be adjusted. At this time, the frequency peak can be kept constant by keeping the period T ₁ that repeats high and low brightness constant. For example, in each of (a), (b), and (c) of FIG. 53, transmission is performed while keeping the time T1 between the first luminance change and the second luminance change, which are brighter than the average luminance, constant. When dimming the machine darkly, shorten the lighting time to be brighter than the average brightness. On the other hand, when dimming the transmitter brightly, the time for illuminating the transmitter brighter than the average brightness is lengthened. (B) and (c) of FIG. 53 are dimmed darker than (a), and (c) of FIG. 53 is dimmed darkest. This makes it possible to perform dimming while transmitting signals having the same meaning.

The average luminance may be changed by changing the values of the luminance in the high-luminance section, the luminance in the low-luminance section, or both of them.

FIG. 54 is a diagram showing an example of a method for configuring the dimming function of the transmitter.

Due to the limited accuracy of the components, the brightness will be slightly different from other transmitters even with the same dimming settings. However, when the transmitters are arranged side by side, if the brightness of the adjacent transmitters is different, an unnatural feeling is felt. Therefore, the user adjusts the brightness of the transmitter by operating the dimming correction operation unit. The dimming correction unit holds the correction value, and the dimming control unit controls the brightness of the light emitting unit according to the correction value. When the degree of dimming is changed by the user operating the dimming operation unit, the dimming control unit is based on the changed dimming set value and the correction value held in the dimming correction unit. In addition, the brightness of the light emitting part is controlled. Further, the dimming control unit transmits the dimming set value to other transmitters through the interlocking dimming unit. When the dimming set value is transmitted from another device through the interlocking dimming unit, the dimming control unit uses the light emitting unit based on the dimming set value and the correction value held in the dimming correction unit. Control the brightness of.

According to one embodiment of the present invention, it is a control method for controlling an information communication device that transmits a signal by changing the brightness of a light emitter, and includes a plurality of different signals with respect to the computer of the information communication device. , A determination step in which a pattern of brightness change at a different frequency is determined for each different signal by modulating the signal to be transmitted, and a brightness change in which a single signal is modulated at a time corresponding to a single frequency. It may be a control method including a transmission step of transmitting a signal to be transmitted by changing the brightness of the light emitting body so as to include only a pattern.

For example, when a pattern of luminance change in which a plurality of signals are modulated is included in a time corresponding to a single frequency, the waveform of the luminance change due to the passage of time becomes complicated, and it becomes difficult to properly receive the waveform. However, by controlling so as to include only the pattern of luminance change in which a single signal is modulated at a time corresponding to a single frequency, it becomes possible to perform reception more appropriately at the time of reception.

According to one embodiment of the present invention, in the determination step, the number of transmissions of transmitting one of a plurality of different signals is different from the number of transmissions of transmitting another signal within a predetermined time. May determine the number of transmissions.

Since the number of transmissions for transmitting one signal is different from the number of transmissions for transmitting another signal, it is possible to prevent flicker during transmission.

According to one embodiment of the present invention, the determination step may cause the number of transmissions of the signal corresponding to the high frequency to be larger than the number of transmissions of the other signals within a predetermined time.

When frequency conversion is performed on the receiving side, the luminance of a signal corresponding to a high frequency becomes small, but by increasing the number of transmissions, it is possible to increase the luminance value at the time of frequency conversion.

According to one embodiment of the present invention, the pattern of the luminance change may be a pattern in which the waveform of the luminance change with the passage of time is any of a square wave, a triangular wave, and a sawtooth wave.

By using a square wave or the like, it becomes possible to perform reception more appropriately.

According to one embodiment of the present invention, when the value of the average brightness of the illuminant is increased, the time during which the brightness of the illuminant becomes larger than a predetermined value in the time corresponding to a single frequency is set. The value of the average luminance of the light emitter may be increased as opposed to the case of decreasing the value.

By adjusting the time during which the brightness of the light emitter becomes larger than a predetermined value in the time corresponding to a single frequency, it is possible to transmit a signal and adjust the average brightness of the light emitter. For example, when a light emitter is used as lighting, it is possible to transmit a signal while dimming or brightening the overall brightness.

The receiver can set the exposure time to a predetermined value by using the API (abbreviation of application programming interface, which refers to the means for using the functions of the OS) that sets the exposure time. , The visible light signal can be received stably. In addition, the receiver can set the sensitivity to a predetermined value by using the API that sets the sensitivity, and can stably receive the visible light signal even when the brightness of the transmission signal is dark or bright. be able to.

(Embodiment 8)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

Here, the EX zoom will be described.

FIG. 55 is a diagram for explaining EX zoom.

Zoom, that is, the method of obtaining a large image, is an optical zoom that adjusts the focal length of the lens to change the size of the image captured by the image sensor, and digitally interpolates the image captured by the image sensor to obtain a large image. There is a digital zoom that obtains a large image and an EX zoom that obtains a large image by changing a plurality of image pickup elements used for image pickup. The EX zoom can be used when the number of image pickup elements included in the image sensor is large compared to the resolution of the captured image.

For example, in the image sensor 10080a shown in FIG. 55, 32 × 24 image pickup elements are arranged in a matrix. That is, 32 image pickup devices are arranged horizontally and 24 images are arranged vertically. When an image having a resolution of 16 horizontal × 12 vertical is obtained by image pickup with the image sensor 10080a, as shown in FIG. 55 (a), among the 32 × 24 image pickup elements included in the image sensor 10080a, the image sensor Only 16 × 12 image pickup elements (for example, the image pickup elements indicated by the black squares in the image sensor 1080a in FIG. 55A) are used for imaging in 10080a. That is, of the plurality of image pickup elements arranged in each of the vertical direction and the horizontal direction, only the odd-numbered or even-numbered image pickup elements are used for imaging. As a result, an image 10080b having a desired resolution can be obtained. In FIG. 55, the subject appears on the image sensor 1008a in order to make it easy to understand the correspondence between each image sensor and the image obtained by the image pickup.

The receiver provided with the image sensor 10080a captures a wide range, and when searching for a transmitter or receiving information from many transmitters, the image sensor 10080a is used as a whole evenly as a whole. Imaging is performed using only some of the image pickup elements arranged in a dispersed manner.

Further, when the receiver performs EX zoom, as shown in FIG. 55 (b), a part of the image sensor (for example, FIG. 55 (b)) is locally densely arranged in the image sensor 10080a. Only 16 × 12 image pickup devices (indicated by the black squares in the image sensor 1080a) in the image sensor 1080a are used for imaging. As a result, the portion of the image 10080b corresponding to a part of the image pickup element is zoomed, and the image 10080d is obtained. With such an EX zoom, the visible light signal can be received for a long time by taking a large image of the transmitter, the reception speed is improved, and the visible light signal can be received from a distance.

With digital zoom, it is not possible to increase the number of exposure lines that receive visible light signals, and the reception time of visible light signals does not increase, so it is better to use other zooms as much as possible. Optical zoom requires physical movement time of the lens and image sensor, but EX zoom has the advantage that the time required for zooming is short because it is performed only by changing electronic settings. From this point of view, the priority of each zoom is (1) EX zoom, (2) optical zoom, and (3) digital zoom. The receiver may select and use any one or more zooms depending on this priority and the need for zoom magnification. In the image pickup methods shown in FIGS. 55 (a) and 55, image noise can be suppressed by using an unused image pickup device.

(Embodiment 9)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

In the present embodiment, the exposure time is set for each exposure line or each image sensor.

56, 57, and 58 are diagrams showing an example of the signal receiving method according to the ninth embodiment.

As shown in FIG. 56, in the image sensor 10010a, which is an image pickup unit provided in the receiver, the exposure time is set for each exposure line. That is, a long exposure time for normal imaging is set for a predetermined exposure line (white exposure line in FIG. 56), and a long exposure time for normal imaging is set for another exposure line (black exposure line in FIG. 56) for visible light imaging. A short exposure time is set. For example, long exposure times and short exposure times are alternately set for each exposure line arranged in the vertical direction. As a result, when imaging a transmitter that transmits a visible light signal due to a change in luminance, normal imaging and visible light imaging (visible light communication) can be performed almost at the same time. The two exposure times may be set alternately for each line, may be set for every several lines, or different exposure times may be set for the upper part and the lower part of the image sensor 10010a. By using the two exposure times in this way, the data obtained by imaging a plurality of exposure lines set to the same exposure time can be summarized as a normal image 10010b and an emission line image showing a pattern of a plurality of emission lines. The visible light captured image 10010c is obtained. Since the normally captured image 10010b lacks a portion that has not been captured with a long exposure time (that is, an image corresponding to a plurality of exposure lines set to a short exposure time), the preview image is obtained by interpolating that portion. 10010d can be displayed. Here, the information obtained by the visible light communication can be superimposed on the preview image 10010d. This information is information associated with the visible light signal obtained by decoding a plurality of emission line patterns included in the visible light captured image 10010c. The receiver saves the normal captured image 10010b or an image interpolated with the normally captured image 10010b as an captured image, and receives the received visible light signal or information associated with the visible light signal. As additional information, it can be added to the captured image to be saved.

As shown in FIG. 57, the image sensor 10011a may be used instead of the image sensor 10010a. In the image sensor 1011a, the exposure time is set not for each exposure line but for each row (hereinafter referred to as a vertical line) composed of a plurality of image pickup elements arranged along the direction perpendicular to the exposure line. That is, a predetermined vertical line (white vertical line in FIG. 57) is normally set with a long exposure time for imaging, and another vertical line (black vertical line in FIG. 57) is set for visible light imaging. A short exposure time is set. In this case, in the image sensor 10011a, as in the image sensor 10010a, the exposure is started at different timings for each exposure line, but the exposure time of each image sensor included in the exposure line is different for each exposure line. .. The receiver obtains a normal image captured image 10011b and a visible light image captured image 10011c by imaging with the image sensor 10011a. Further, the receiver generates and displays the preview image 10011d based on the normal captured image 10011b and the information associated with the visible light signal obtained from the visible light captured image 10011c.

In this image sensor 10011a, unlike the image sensor 10010a, all exposure lines can be used for visible light imaging. As a result, since the visible light captured image 10011c obtained by the image sensor 10011a contains more bright lines than the visible light captured image 10010c, the reception accuracy of the visible light signal can be improved.

Further, as shown in FIG. 58, the image sensor 10012a may be used instead of the image sensor 10010a. In the image sensor 10012a, the exposure time is set for each image sensor so that the same exposure time is not continuously set for each image sensor along the horizontal direction and the vertical direction. That is, the exposure time for each image sensor is such that the plurality of image sensors for which a long exposure time is set and the plurality of image sensors for which a short exposure time is set are distributed in a grid pattern or a checkered pattern. Is set. In this case as well, the exposure is started at different timings for each exposure line as in the image sensor 10010a, but the exposure time of each image sensor included in the exposure line is different for each exposure line. The receiver obtains a normal image captured image 10012b and a visible light image captured image 10012c by imaging with the image sensor 10012a. Further, the receiver generates and displays the preview image 10012d based on the normal captured image 10012b and the information associated with the visible light signal obtained from the visible light captured image 10012c.

Since the normal image sensor 10012b obtained by the image sensor 10012a has data of a plurality of image sensors arranged in a grid pattern or uniformly arranged, the normal image sensor 10010b and the normal image sensor 10011b can be more accurately interpolated. You can resize. Further, the visible light captured image 10012c is generated by imaging using all the exposure lines of the image sensor 10012a. That is, in this image sensor 10012a, unlike the image sensor 10010a, all exposure lines can be used for visible light imaging. As a result, the visible light captured image 10012c obtained by the image sensor 10012a contains more bright lines than the visible light captured image 10010c, as in the visible light captured image 10011c, so that the reception of the visible light signal is high. It can be done with precision.

Here, the interlaced display of the preview image will be described.

FIG. 59 is a diagram showing an example of the screen display method of the receiver according to the ninth embodiment.

The receiver including the image sensor 10010a shown in FIG. 56 described above is set to an exposure time set to an odd-numbered exposure line (hereinafter referred to as an odd-numbered line) and an even-numbered exposure line (hereinafter referred to as an even-numbered line). The exposure time is replaced at predetermined time intervals. For example, as shown in FIG. 59, the receiver sets a long exposure time for each image sensor of the odd line and a short exposure time for each image sensor of the even line at time t1. Image pickup is performed using the set exposure time of. Further, at time t2, the receiver sets a short exposure time for each image sensor of the odd-numbered line, sets a long exposure time for each image sensor of the even-numbered line, and sets the set exposure time. Perform the imaging using. Then, the receiver performs imaging at time t3 using each exposure time set in the same manner as at time t1, and uses each exposure time set at time t4 in the same manner as at time t2. Take an image.

Further, at time t1, the receiver has an image obtained from each of the plurality of odd-numbered lines by imaging (hereinafter referred to as an odd-numbered line image) and an image obtained from each of the plurality of even-numbered lines by imaging (hereinafter referred to as an even-numbered line image). Image 1 including) and is acquired. At this time, since the exposure time is short in each of the plurality of even-numbered lines, the subject is not clearly projected on each of the even-numbered line images. Therefore, the receiver generates a plurality of interpolated line images by interpolating the pixel values for the plurality of odd-numbered line images. Then, the receiver displays a preview image including a plurality of interpolated line images instead of the plurality of even line images. That is, the odd-numbered line image and the interpolated line image are alternately arranged in the preview image.

At time t2, the receiver acquires Image 2 including a plurality of odd-numbered line images and even-numbered line images by imaging. At this time, since the exposure time is short in each of the plurality of odd-numbered lines, the subject is not clearly projected on each of the odd-numbered line images. Therefore, the receiver displays a preview image including the odd-numbered line image of Image1 instead of the odd-numbered line image of Image2. That is, in the preview image, the odd-numbered line image of Image1 and the even-numbered line image of Image2 are alternately arranged.

Further, at time t3, the receiver acquires an Image 3 including a plurality of odd-numbered line images and even-numbered line images by imaging. At this time, as in the case of time t1, since the exposure time is short in each of the plurality of even-numbered lines, the subject is not clearly projected on each of the even-numbered line images. Therefore, the receiver displays a preview image including the even-numbered line image of Image2 instead of the even-numbered line image of Image3. That is, in the preview image, the even-numbered line image of Image2 and the odd-numbered line image of Image3 are alternately arranged. Then, at time t4, the receiver acquires Image 4 including a plurality of odd-numbered line images and even-numbered line images by imaging. At this time, as in the case of time t2, since the exposure time is short in each of the plurality of odd-numbered lines, the subject is not clearly projected on each of the odd-numbered line images. Therefore, the receiver displays a preview image including the odd line image of Image 3 instead of the odd line image of Image 4. That is, in the preview image, the odd-numbered line image of Image 3 and the even-numbered line image of Image 4 are alternately arranged.

In this way, the receiver performs so-called interlaced display, which displays an image including an even-numbered line image and an odd-numbered line image whose acquired timings are different from each other.

A receiver such as this can display a fine preview image while performing visible light imaging. The plurality of image pickup elements for which the same exposure time is set may be a plurality of image pickup elements arranged along a direction horizontal to the exposure line such as the image sensor 10010a, or the exposure line such as the image sensor 10011a. A plurality of image pickup elements arranged along a direction perpendicular to the image sensor may be used, or a plurality of image pickup elements arranged according to a checkered pattern such as the image sensor 10012a may be used. In addition, the receiver may save the preview image as imaging data.

Next, the spatial ratio between normal imaging and visible light imaging will be described.

FIG. 60 is a diagram showing an example of the signal receiving method according to the ninth embodiment.

In the image sensor 10014b provided in the receiver, a long exposure time or a short exposure time is set for each exposure line, similarly to the image sensor 10010a described above. In this image sensor 10014b, the ratio of the number of image pickup elements for which a long exposure time is set to the number of image pickup elements for which a short exposure time is set is 1: 1. It should be noted that this ratio is a ratio between normal imaging and visible light imaging, and is hereinafter referred to as a spatial ratio.

However, in this embodiment, the spatial ratio does not have to be 1: 1. For example, the receiver may include an image sensor 10014a. In this image sensor 10014a, the image sensor having a short exposure time has more images than the image sensor having a long exposure time, and the spatial ratio is 1: N (N> 1). Further, the receiver may be provided with an image sensor 10014c. In this image sensor 10014c, the image sensor having a short exposure time has less than the image sensor having a long exposure time, and the spatial ratio is N (N> 1): 1. Further, in the receiver, instead of the image sensors 10014a to 10014c, the exposure time is set for each of the above-mentioned vertical lines, and the image sensors 10015a to 10015c have a spatial ratio of 1: N, 1: 1 or N: 1, respectively. Any of the above may be provided.

In such image sensors 10014a and 10015a, since many image pickup devices have a short exposure time, it is possible to improve the reception accuracy or the reception speed of the visible light signal. On the other hand, the image sensors 10014c and 10015c can display a fine preview image because many image sensors have a long exposure time.

Further, the receiver may use the image sensors 10014a, 10014c, 10015a, 10015c to perform interlaced display as shown in FIG. 59.

Next, the time ratio between normal imaging and visible light imaging will be described.

FIG. 61 is a diagram showing an example of the signal receiving method according to the ninth embodiment.

As shown in FIG. 61 (a), the receiver may switch the imaging mode between the normal imaging mode and the visible light imaging mode for each frame. The normal image pickup mode is an image pickup mode in which a long exposure time for normal image pickup is set for all image pickup elements of the image sensor of the receiver. The visible light image pickup mode is an image pickup mode in which a short exposure time for visible light image pickup is set for all image pickup elements of the image sensor of the receiver. By switching between long and short exposure times in this way, it is possible to display a preview image by imaging with a long exposure time while receiving a visible light signal by imaging with a short exposure time.

When the receiver determines a long exposure time by automatic exposure, the receiver ignores the image obtained by imaging with a short exposure time and only the brightness of the image obtained by imaging with a long exposure time. Automatic exposure may be performed as a reference. Thereby, a long exposure time can be determined to be an appropriate time.

Further, as shown in FIG. 61 (b), the receiver may switch the imaging mode between the normal imaging mode and the visible light imaging mode for each set of a plurality of frames. If it takes time to switch the exposure time or it takes time for the exposure time to stabilize, as shown in FIG. 61 (b), the exposure time can be changed for each set of a plurality of frames. It is possible to achieve both visible light imaging (reception of visible light signals) and normal imaging. Further, as the number of frames included in the set increases, the number of times the exposure time is switched decreases, so that power consumption and heat generation in the receiver can be suppressed.

Here, the number of at least one frame continuously generated by imaging with a long exposure time in the normal imaging mode and at least one frame continuously generated by imaging with a short exposure time in the visible light imaging mode. The ratio to the number of (hereinafter referred to as time ratio) does not have to be 1: 1. That is, in the case shown in FIGS. 61A and 61, the time ratio is 1: 1 but the time ratio does not have to be 1: 1.

For example, the receiver may have more frames in the visible light imaging mode than frames in the normal imaging mode, as shown in FIG. 61 (c). As a result, the reception speed of the visible light signal can be increased. If the frame rate of the preview image is equal to or higher than a predetermined rate, the difference in the preview image due to the frame rate is not recognized by the human eye. When the frame rate of imaging is sufficiently high, for example, when the frame rate is 120 fps, the receiver sets the visible light imaging mode for three consecutive frames, and then performs visible light imaging for one frame. Set the mode. As a result, the receiver can receive the visible light signal at high speed while displaying the preview image at a frame rate of 30 fps, which is sufficiently higher than the above-mentioned predetermined rate. Further, since the number of switchings is reduced, the effect described in FIG. 61 (b) can be obtained.

Further, as shown in FIG. 61 (d), the receiver may have more frames in the normal imaging mode than frames in the visible light imaging mode. In this way, the preview image can be displayed smoothly by increasing the number of frames in the normal imaging mode, that is, the frames obtained by imaging with a long exposure time. In addition, since the number of times of receiving the visible light signal is reduced, there is a power saving effect. Further, since the number of switchings is reduced, the effect described in FIG. 61 (b) can be obtained.

Further, as shown in FIG. 61 (e), the receiver first switches the imaging mode for each frame as in the case shown in FIG. 61 (a), and then receives the visible light signal. When completed, the number of frames in the normal imaging mode may be increased as in the case shown in FIG. 61 (d). As a result, after the reception of the visible light signal is completed, the preview image can be displayed smoothly, and the search for the existence of a new visible light signal can be continued. Further, since the number of switchings is reduced, the effect described in FIG. 61 (b) can be obtained.

FIG. 62 is a flowchart showing an example of the signal receiving method according to the ninth embodiment.

The receiver starts visible light reception, which is a process of receiving the visible light signal (step S10017a), and sets the exposure time length / short setting ratio to a value specified by the user (step S10017b). The exposure time length setting ratio is at least one of the above-mentioned spatial ratio and time ratio. The user may specify only the spatial ratio, only the time ratio, or both the spatial ratio and the time ratio, or the receiver may automatically set the values regardless of the user's specification.

Next, the receiver determines whether or not the reception performance is equal to or less than a predetermined value (step S10017c). If it is determined that the value is equal to or less than a predetermined value (Y in step S10017c), the receiver sets a high ratio of the short exposure time (step S10017d). This makes it possible to improve the reception performance. In the case of the spatial ratio, the ratio of the short exposure time is the ratio of the number of image sensors for which the short exposure time is set to the number of image sensors for which the long exposure time is set, and in the case of the time ratio, normal imaging is performed. It is the ratio of the number of frames continuously generated in the visible light imaging mode to the number of frames continuously generated in the mode.

Next, the receiver receives at least a part of the visible light signal, and determines whether or not a priority is set for at least a part of the received visible light signal (hereinafter referred to as a received signal) (hereinafter referred to as a received signal). Step S10017e). When the priority is set, the received signal includes an identifier indicating the priority. When the receiver determines that the priority is set (Y in step S10017e), the receiver sets the exposure time length / short ratio according to the priority (step S10017f). That is, if the priority is high, the receiver sets a high ratio of short exposure times. For example, an emergency light configured as a transmitter emits an identifier indicating a high priority by changing the brightness. In this case, the receiver can increase the reception speed by increasing the ratio of the short exposure time, and can promptly display the evacuation route or the like.

Next, the receiver determines whether or not all reception of the visible light signal has been completed (step S10017g). Here, when it is determined that the process is not completed (N in step S10017g), the receiver repeatedly executes the process from step S10017c. On the other hand, when it is determined that the completion is completed (Y in step S10017g), the receiver sets a high ratio of the long exposure time and shifts to the power saving mode (step S10017h). The ratio of the long exposure time is the ratio of the number of image sensors for which a long exposure time is set to the number of image sensors for which a short exposure time is set in the case of a spatial ratio, and the visible light in the case of a time ratio. It is the ratio of the number of frames continuously generated in the normal image pickup mode to the number of frames continuously generated in the image pickup mode. As a result, the preview image can be displayed smoothly without receiving unnecessary visible light.

Next, the receiver determines whether or not another visible light signal has been found (step S10017i). Here, when it is determined that the discovery has been made (Y in step S10017i), the receiver repeatedly executes the process from step S10017b.

Next, simultaneous execution of visible light imaging and normal imaging will be described.

FIG. 63 is a diagram showing an example of the signal receiving method according to the ninth embodiment.

The receiver may set the image sensor to two or more exposure times. That is, as shown in FIG. 63 (a), each of the exposure lines included in the image sensor is continuously exposed for the longest exposure time among the two or more set exposure times. The receiver reads out the image pickup data obtained by the exposure of the exposure line at the time when the above-mentioned two or more exposure times set for each exposure line have elapsed. Here, the receiver does not reset the read imaging data until the longest exposure time has elapsed. Therefore, by recording the cumulative value of the read imaging data, the receiver can obtain the imaging data at a plurality of exposure times only by the exposure with the longest exposure time. The image sensor may or may not record the cumulative value of the imaging data. If the image sensor does not, the component of the receiver that reads the data from the image sensor performs the cumulative calculation, that is, the recording of the cumulative value of the imaged data.

For example, when two exposure times are set, as shown in FIG. 63 (a), the receiver receives visible light imaging data including a visible light signal generated by exposure with a short exposure time. Read, and then read the normal imaging data generated by the exposure with a long exposure time.

As a result, visible light imaging, which is an imaging for receiving a visible light signal, and normal imaging can be performed at the same time, and normal imaging can be performed while receiving a visible light signal. Further, by using the data of a plurality of exposure times, it is possible to recognize the signal frequency higher than the sampling theorem, and it is possible to receive the high frequency signal and the high density modulation signal.

Further, when the receiver outputs the image pickup data, as shown in FIG. 63 (b), the receiver outputs a data string including the image pickup data as an image pickup data body. That is, the receiver has an image pickup mode identifier indicating an image pickup mode (visible light image pickup or normal image pickup), an image pickup element identifier for specifying an image pickup element or an exposure line to which the image pickup element belongs, and the exposure number of the image pickup data body. By adding additional information including an imaging data number indicating whether it is time-based imaging data and an imaging data length indicating the size of the imaging data body to the imaging data body, the above-mentioned data string is generated and output. .. In the method of reading out the imaging data described with reference to FIG. 63A, the respective imaging data are not always output in the order of the exposure lines. Therefore, by adding the additional information shown in FIG. 63 (b), it is possible to specify which exposure line the imaging data is.

FIG. 64 is a flowchart showing the processing of the receiving program according to the ninth embodiment.

This receiving program is a program that causes a computer provided in the receiver to execute, for example, the processes shown in FIGS. 56 to 63.

That is, this receiving program is a receiving program for receiving information from a light emitter whose brightness changes. Specifically, this receiving program causes the computer to execute step SA31, step SA32, and step SA33. In step SA31, the first exposure time is set for a plurality of a plurality of K image sensors (K is an integer of 4 or more) included in the image sensor, and the K image sensors are used. A second exposure time shorter than the first exposure time is set for the remaining plurality of image pickup devices. In step SA32, the subject, which is a light emitter whose brightness changes, is imaged by the image sensor at the set first and second exposure times, so that the output from the plurality of image pickup elements for which the first exposure time is set is set. It is an image corresponding to the output from the plurality of image sensors for which the second exposure time is set, and the emission line corresponding to each of the plurality of exposure lines included in the image sensor is obtained. Acquires a bright line image which is an image including the image. In step SA33, information is acquired by decoding a plurality of emission line patterns included in the acquired emission line image.

As a result, the image pickup is performed by the plurality of image pickup elements for which the first exposure time is set and the plurality of image pickup elements for which the second exposure time is set. And the emission line image can be obtained. That is, it is possible to simultaneously capture a normal image and acquire information by visible light communication.

Further, in the exposure time setting step SA31, the first exposure time is set for a plurality of image sensor trains among the L image sensor trains (L is an integer of 4 or more) included in the image sensor. Then, the second exposure time is set for the remaining plurality of image pickup element trains in the L image pickup element trains. Here, each of the L image pickup element rows is composed of a plurality of image pickup elements arranged in a row, which are included in the image sensor.

As a result, the exposure time can be set for each image sensor row, which is a large unit, without individually setting the exposure time for each of the image sensor, which is a small unit, and the processing load can be reduced. ..

For example, each of the L image sensor rows is an exposure line included in the image sensor, as shown in FIG. 56. Alternatively, as shown in FIG. 57, each of the L image pickup element trains comprises a plurality of image pickup elements arranged along the direction perpendicular to the exposure line included in the image sensor.

Further, as shown in FIG. 59, in the exposure time setting step SA31, the exposure time is the same for each of the odd-third image sensor trains among the L image sensor trains included in the image sensor. The first and second exposures in which one of the first and second exposure times is set and the exposure time is the same for each of the even-order image sensor trains in the L image sensor trains. The other of the times may be set. When the exposure time setting step SA31, the image acquisition step SA32, and the information acquisition step SA33 are repeated, in the repeated exposure time setting step SA31, the exposure time setting step SA31 immediately before is set to each of the odd-th image pickup element trains. The exposed exposure time may be replaced with the exposure time set for each of the even-th image pickup element trains.

As a result, each time a normal image is acquired, the plurality of image pickup element sequences used for the acquisition can be switched between the odd-numbered plurality of image pickup element sequences and the even-numbered plurality of image pickup element sequences. As a result, each of the normally acquired images can be displayed by interlacing. Further, by complementing two consecutively acquired normal images with each other, a new normal image including an image by a plurality of odd-numbered image sensor trains and an image by an even-numbered plurality of image sensor trains is generated. be able to.

Further, as shown in FIG. 60, in the exposure time setting step SA31, the setting mode is switched between the normal priority mode and the visible light priority mode, and when the setting mode is switched to the normal priority mode, the first exposure time is set. The number of image pickup elements may be larger than the number of image pickup elements for which the second exposure time is set. Further, when the mode is switched to the visible light priority mode, the number of image pickup elements for which the first exposure time is set may be smaller than the number of image pickup elements for which the second exposure time is set.

As a result, when the setting mode is switched to the normal priority mode, the image quality of the normal image can be improved, and when the setting mode is switched to the visible light priority mode, the efficiency of receiving information from the light emitter is improved. can do.

Further, as shown in FIG. 58, in the exposure time setting step SA31, the plurality of image pickup elements in which the first exposure time is set and the plurality of image pickup elements in which the second exposure time is set are formed in a checkered pattern (a checkered pattern). The exposure time of the image sensor may be set for each image sensor included in the image sensor so that the image sensor is distributed as in the Checkered pattern).

As a result, the plurality of image pickup elements for which the first exposure time is set and the plurality of image pickup elements for which the second exposure time is set are uniformly distributed, so that the image quality is biased in the horizontal direction and the vertical direction. It is possible to obtain no normal image and bright line image.

FIG. 65 is a block diagram of the receiving device according to the ninth embodiment.

The receiver A30 is the above-mentioned receiver that executes the processes shown in FIGS. 56 to 63, for example.

That is, the receiving device A30 is a receiving device that receives information from a light emitting body whose brightness changes, and includes a plurality of exposure time setting units A31, an imaging unit A32, and a decoding unit A33. The multiple exposure time setting unit A31 sets the first exposure time for a plurality of a plurality of K image sensors (K is an integer of 4 or more) included in the image sensor, and K sets the first exposure time. A second exposure time shorter than the first exposure time is set for the remaining plurality of image pickup elements. The image pickup unit A32 causes an image sensor to take an image of a subject, which is a light emitting body whose brightness changes, at set first and second exposure times, so that the image sensor A32 receives images from a plurality of image pickup elements for which the first exposure time is set. A normal image corresponding to the output is acquired, and an image corresponding to the output from the plurality of image sensors for which the second exposure time is set, and the emission line corresponding to each of the plurality of exposure lines included in the image sensor. Acquires a bright line image which is an image including. The decoding unit A33 acquires information by decoding a plurality of emission line patterns included in the acquired emission line image. In such a receiving device A30, the same effect as the above-mentioned receiving program can be obtained.

Next, the display of the contents related to the received visible light signal will be described.

66 and 67 are diagrams showing an example of a display of a receiver when a visible light signal is received.

As shown in FIG. 66 (a), when the receiver takes an image of the transmitter 10020d, the receiver displays the image 10020a on which the transmitter 10020d is projected. Further, the receiver generates and displays the image 10020b by superimposing the object 10020e on the image 10020a. The object 10020e is an image showing a place where the image of the transmitter 10020d is located and receiving a visible light signal from the transmitter 10020d. The object 10020e may be an image different depending on the reception state of the visible light signal (state during reception, state of searching for a transmitter, degree of progress of reception, reception speed, error rate, etc.). For example, the receiver changes the color, line thickness, line type (single line, double line, dotted line, etc.), dotted line spacing, and the like of the object 1020e. This makes it possible for the user to recognize the reception state. Next, the receiver generates and displays the image 10020c by superimposing the image showing the content of the acquired data on the image 10020a as the acquired data image 10020f. The acquired data is the received visible light signal or the data associated with the ID indicated by the received visible light signal.

When displaying the acquired data image 10020f, the receiver may display the acquired data image 10020f like a balloon from the transmitter 10020d, or near the transmitter 10020d, as shown in FIG. 66 (a). The acquired data image 10020f is displayed on the screen. Further, as shown in FIG. 66B, the receiver may display the acquired data image 10020f so that the acquired data image 10020f gradually approaches the receiver side from the transmitter 10020d. This makes it possible for the user to recognize from which transmitter the acquired data image 10020f is based on the visible light signal received. Further, as shown in FIG. 67, the receiver may display the acquired data image 10020f so that the acquired data image 10020f gradually emerges from the edge of the display of the receiver. This makes it possible for the user to easily recognize that the visible light signal was acquired at that time.

Next, AR (Augmented Reality) will be described.

FIG. 68 is a diagram showing an example of display of the acquired data image 10020f.

When the image of the transmitter moves in the display, the receiver moves the acquired data image 10020f in accordance with the movement of the image of the transmitter. This makes it possible for the user to recognize that the acquired data image 10020f corresponds to the transmitter. Further, the receiver may display the acquired data image 10020f in association with another image instead of the image of the transmitter. As a result, AR display can be performed.

Next, the storage and destruction of the acquired data will be described.

FIG. 69 is a diagram showing an example of an operation when the acquired data is saved or destroyed.

For example, as shown in FIG. 69 (a), the receiver stores the acquired data indicated by the acquired data image 10020f when the user swipes downward on the acquired data image 10020f. .. The receiver arranges the acquired data image 10020f indicating the stored acquired data at the end of the acquired data image indicating one or more other already stored acquired data. This makes it possible for the user to recognize that the acquired data indicated by the acquired data image 10020f is the last stored acquired data. For example, as shown in FIG. 69 (a), the receiver arranges the acquired data image 10020f in the foreground of the plurality of acquired data images.

Further, as shown in FIG. 69 (b), when the user swipes the acquired data image 10020f to the right, the receiver discards the acquired data indicated by the acquired data image 10020f. Alternatively, the receiver may discard the acquired data indicated by the acquired data image 10020f when the image of the transmitter is framed out of the display by the user moving the receiver. It should be noted that the same effect as described above can be obtained regardless of the direction of swiping, up, down, left or right. The receiver may display the swipe direction corresponding to the save or discard. This makes it possible for the user to recognize that the operation can be saved or destroyed.

Next, browsing of the acquired data will be described.

FIG. 70 is a diagram showing a display example when viewing the acquired data.

As shown in FIG. 70A, the receiver superimposes the acquired data images of the plurality of stored acquired data on the lower end of the display and displays them in a small size. At this time, when the user taps a part of the acquired data image displayed, the receiver displays each of the plurality of acquired data images in a large size as shown in FIG. 70 (b). As a result, the acquired data images can be displayed in a large size only when it is necessary to view each acquired data, and the display can be effectively used for other displays when it is not necessary.

When the user taps the acquired data image to be displayed in the state shown in FIG. 70 (b), the receiver displays the tapped acquired data image in a larger size as shown in FIG. 70 (c). A lot of information is displayed in the acquired data image. Further, when the user taps the back surface display button 10024a, the receiver displays the back surface of the acquired data image and displays another data related to the acquired data.

Next, turning off the image stabilization at the time of accident position estimation will be described.

The receiver obtains an accurate imaging direction and accurately estimates its own position by disabling (turning off) the image stabilization or converting the captured image according to the correction direction and correction amount of the image stabilization. Can be done. The captured image is an image obtained by imaging by the imaging unit of the receiver. In addition, self-position estimation means that the receiver estimates its own position. In self-position estimation, specifically, the receiver identifies the position of the transmitter based on the received visible light signal, and receives based on the size, position, or shape of the transmitter reflected in the captured image. Identify the relative positional relationship between the machine and the transmitter. Then, the receiver estimates the position of the receiver based on the position of the transmitter and the relative positional relationship between the receiver and the transmitter.

Further, at the time of partial readout in which imaging is performed using only a part of the exposure lines shown in FIG. 56 and the like, that is, when imaging is performed as shown in FIG. 56 and the like, the transmitter is framed out with a slight blur of the receiver. It ends up. In such a case, the receiver can continuously receive the signal by enabling the image stabilization.

Next, self-position estimation using an asymmetrical light emitting portion will be described.

FIG. 71 is a diagram showing an example of the transmitter according to the ninth embodiment.

The above-mentioned transmitter includes a light emitting unit, and transmits a visible light signal by changing the brightness of the light emitting unit. In the self-position estimation described above, the receiver is the relative positional relationship between the receiver and the transmitter based on the shape of the transmitter (specifically, the light emitting part) in the captured image. Find the relative angle to the transmitter. Here, for example, as shown in FIG. 71, when the transmitter is provided with a light emitting unit 10090a having a rotationally symmetric shape, the transmitter and the transmitter are based on the shape of the transmitter in the captured image as described above. The relative angle to the receiver cannot be determined accurately. Therefore, it is desirable that the transmitter is provided with a light emitting unit having a shape that is not rotationally symmetric. This allows the receiver to accurately determine the relative angle described above. That is, since the error of the measurement result is large in the directional sensor for acquiring the angle, the receiver can perform accurate self-position estimation by using the relative angle obtained by the above method.

Here, as shown in FIG. 71, the transmitter may include a light emitting unit 10090b that does not have a perfectly rotationally symmetric shape. The shape of the light emitting portion 10090b is symmetric with respect to a rotation of 90 °, but is not completely rotationally symmetric. In this case, the receiver can uniquely limit the relative angle between the receiver and the transmitter by obtaining a rough angle with the orientation sensor and using the shape of the transmitter in the captured image. It is possible to perform accurate self-position estimation.

Further, the transmitter may include a light emitting unit 10090c shown in FIG. 71. The shape of the light emitting portion 10090c is basically a rotationally symmetric shape. However, since a light guide plate or the like is installed in a part of the light emitting unit 10090c, the shape of the light emitting unit 10090c is not rotationally symmetric.

Further, the transmitter may include a light emitting unit 10090d shown in FIG. 71. Each of the light emitting units 10090d is provided with illumination having a rotationally symmetric shape. However, the overall shape of the light emitting unit 10090d formed by arranging them in combination is not a rotationally symmetric shape. Therefore, the receiver can perform accurate self-position estimation by imaging the transmitter. Further, all the lights included in the light emitting unit 10090d do not have to be the lights for visible light communication whose brightness changes in order to transmit the visible light signal, and only some of the lights are the lights for visible light communication. May be.

Further, the transmitter may include a light emitting unit 10090e and an object 10090f shown in FIG. 71. Here, the object 10090f is an object (for example, a fire alarm, a pipe, or the like) configured so that the positional relationship with the light emitting unit 10090e does not change. Since the shape of the combination of the light emitting unit 10090e and the object 10090f is not a rotationally symmetric shape, the receiver can accurately estimate the self-position by imaging the light emitting unit 10090e and the object 10090f.

Next, the time series processing of self-position estimation will be described.

Each time the receiver takes an image, it can estimate its own position from the position and shape of the transmitter in the captured image. As a result, the receiver can estimate the moving direction and distance of the receiver during imaging. In addition, the receiver can perform more accurate self-position estimation by performing triangulation using a plurality of frames or images. By integrating the estimation results using a plurality of images and the estimation results using a plurality of different combinations of images, the receiver can perform self-position estimation more accurately. At this time, the receiver can perform self-position estimation more accurately by emphasizing the results estimated from the recent captured images and integrating them.

Next, skipping the reading of Optical Black will be described.

FIG. 72 is a diagram showing an example of the receiving method according to the ninth embodiment. The horizontal axis of the graph shown in FIG. 72 indicates the time, and the vertical axis indicates the position of each exposure line in the image sensor. Further, the solid arrow in the graph indicates the time (exposure timing) at which the exposure of each exposure line in the image sensor is started.

During normal imaging, the receiver reads out the horizontal optical black signal in the image sensor as shown in FIG. 72 (a), but skips the horizontal optical black signal as shown in FIG. 72 (b). You may. This makes it possible to receive a continuous visible light signal.

Horizontal optical black is optical black that is horizontal to the exposure line. The vertical optical black is a portion of the optical black other than the horizontal optical black.

Since the receiver adjusts the black level by the signal read from the optical black, the black level can be adjusted by using the optical black at the start of visible light imaging as in the normal imaging. If vertical optical black is available, the receiver can perform continuous reception and black level adjustment by using only vertical optical black to adjust the black level. During the continuation of visible light imaging, the receiver may adjust the black level using horizontal optical black at predetermined time intervals. In the case where normal imaging and visible light imaging are alternately performed, the receiver skips the horizontal optical black signal when performing visible light imaging continuously, and reads out the horizontal optical black signal at other times. Then, the receiver can adjust the black level while continuously receiving the visible light signal by adjusting the black level based on the read signal. The receiver may adjust the black level by setting the darkest part of the visible light captured image as black.

As described above, by limiting the optical black from which the signal is read to only the vertical optical black, it is possible to continuously receive the visible light signal. Further, by providing a mode for skipping the horizontal optical black signal, it is possible to adjust the black level at the time of normal imaging and perform continuous communication as needed at the time of visible light imaging. Further, by skipping the horizontal optical black signal, the difference in the timing of starting the exposure between the exposure lines becomes large, so that the visible light signal from the transmitter, which is only small, can be received.

Next, an identifier indicating the type of transmitter will be described.

The transmitter may transmit by adding a transmitter identifier indicating the type of the transmitter to the visible light signal. In this case, the receiver can perform the receiving operation according to the type of the transmitter when the transmitter identifier is received. For example, when the transmitter identifier indicates digital signage, the transmitter displays a content ID indicating which content is currently displayed in addition to the transmitter ID for identifying the individual transmitter. Is being sent as. By handling these IDs separately based on the transmitter identifier, the receiver can display information according to the content currently displayed by the transmitter. Further, for example, when the transmitter identifier indicates digital signage or an emergency light, the receiver can reduce the reception error by increasing the sensitivity and taking an image.

(Embodiment 10)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

Here, a receiving method for comparing the data units of the same address will be described.

FIG. 73 is a flowchart showing an example of the receiving method according to the present embodiment.

The receiver receives the packet (step S10101) and corrects the error (step S10102). Then, the receiver determines whether or not a packet having the same address as the received packet has already been received (step S10103). Here, if it is determined that the data has been received (Y in step S10103), the receiver compares the data. That is, the receiver determines whether or not the data units are equal (step S10104). Here, when it is determined that they are not equal (N in step S10104), the receiver further determines whether the difference in the plurality of data units is a predetermined number or more, specifically, the number of different bits or the luminance. It is determined whether or not the number of slots having different states is a predetermined number or more (step S10105). Here, if it is determined that the number is equal to or greater than a predetermined number (N in step S10105), the receiver discards the packets that have already been received (step S10106). As a result, when a packet is started to be received from another transmitter, interference with a packet received from the previous transmitter can be avoided. On the other hand, if it is determined that the number is not equal to or more than a predetermined number (N in step S10105), the receiver uses the data of the data part having the largest number of packets having the same data part as the data of the address (step S10107). Alternatively, the receiver takes the bit with the most equal bits as the value of that bit at that address. Alternatively, the receiver sets the luminance state having the most equal luminance states as the luminance state of the slot at that address, and demodulates the data at that address.

As described above, in the present embodiment, the receiver first acquires the first packet including the data unit and the address unit from the plurality of emission line patterns. Next, the receiver is at least one second packet that includes the same address part as the address part of the first packet among at least one packet that has already been acquired before the first packet. Determines if a packet of Next, when the receiver determines that the at least one second packet exists, whether or not the data parts of the at least one second packet and the first packet are all equal or not. judge. If it is determined that all the data parts are not equal, the receiver receives the first packet of each part included in the data part of the second packet in each of the at least one second packet. It is determined whether or not the number of parts different from each part included in the data part of the above is equal to or more than a predetermined number. Here, the receiver, if there is a second packet in which it is determined that the number of different portions is equal to or more than a predetermined number among the at least one second packet, the receiver has at least one second packet. Is destroyed. On the other hand, if there is no second packet in which it is determined that the number of different portions is equal to or greater than a predetermined number among the at least one second packet, the receiver determines the first packet and at least one second packet. Among the packets of, the plurality of packets having the same data part are specified. Then, the receiver decodes the data part included in each of the plurality of packets as the data part corresponding to the address part included in the first packet, thereby performing at least a part of the visible light identifier (ID). get.

As a result, when multiple packets having the same address part are received, even if the data parts of those packets are different, the appropriate data part can be decoded, and at least a part of the visible light identifier can be used. You can get it correctly. That is, a plurality of packets having the same address unit transmitted from the same transmitter basically have the same data unit. However, when the receiver switches the transmitter that is the source of the packet, the receiver may receive a plurality of packets having the same address part but different data parts from each other. In such a case, in the present embodiment, as in step S10106 of FIG. 73, the packet (second packet) that has already been received is discarded, and the data unit of the latest packet (first packet). Can be decoded as the correct data part corresponding to the address part. Further, even when the transmitter is not switched as described above, the data unit of a plurality of packets having the same address unit may be slightly different depending on the transmission / reception status of the visible light signal. In such a case, in the present embodiment, an appropriate data unit can be decoded by a so-called majority vote as in step S10107 of FIG. 73.

Here, a receiving method for demodulating the data of the data unit from a plurality of packets will be described.

FIG. 74 is a flowchart showing an example of the receiving method according to the present embodiment.

First, the receiver receives the packet (step S10111) and corrects the error in the address unit (step S10112). At this time, the receiver does not demodulate the data unit and keeps the pixel value obtained by the imaging as it is. Then, the receiver determines whether or not there are a predetermined number or more of packets having the same address in the plurality of packets already received (step S10113). Here, if it is determined that it exists (Y in step S10113), the receiver performs demodulation processing by matching the pixel values of the portions corresponding to the data portions of the plurality of packets having the same address (step S10114).

As described above, in the receiving method in the present embodiment, the first packet including the data unit and the address unit is acquired from the pattern of the plurality of emission lines. Then, among at least one packet already acquired before the first packet, there are a predetermined number or more of second packets that include the same address part as the address part of the first packet. Judge whether or not. When it is determined that there are more than a predetermined number of the second packets, the pixel value of a part of the emission line image corresponding to each data part of the predetermined number or more of the second packets and the first. Matches with the pixel value of a part of the emission line image corresponding to the data part of the packet. That is, the pixel values are added. The composite pixel value is calculated by the addition, and at least a part of the visible light identifier (ID) is acquired by decoding the data unit including the composite pixel value.

Since the timing at which a plurality of packets are received is different, the pixel value of the data unit is a value that reflects the brightness of the transmitter at slightly different time points. Therefore, the portion to be demodulated as described above includes a larger amount of data (number of samples) than the data portion of a single packet. As a result, the data unit can be demodulated more accurately. Also, by increasing the number of samples, it is possible to demodulate a signal modulated at a higher modulation frequency.

The data section and its error-correcting code section are modulated at a higher frequency than the header section, the address section, and the error-correcting code section of the address section. With the above demodulation method, demodulation is possible even if the data section and subsequent parts are modulated at a high modulation frequency. Therefore, with this configuration, the transmission time of the entire packet can be shortened, and even from a greater distance, from a smaller light source. However, the visible light signal can be received faster.

Next, a receiving method for receiving the data of the variable length address will be described.

FIG. 75 is a flowchart showing an example of the receiving method according to the present embodiment.

The receiver receives the packet (step S10121), and determines whether or not a packet in which all the bits of the data unit are 0 (hereinafter referred to as a 0-terminated packet) has been received (step S10122). Here, if it is determined that the packet has been received, that is, if it is determined that the 0-terminated packet exists (Y in step S10122), the receiver has whether or not all the packets having addresses equal to or lower than the address of the 0-terminated packet are available, that is, It is determined whether or not the packet is received (step S10123). The address is set to a value that increases for each of the packets generated by dividing the transmitted data according to the transmission order of those packets. When the receiver determines that all the packets are complete (Y in step S10123), the receiver determines that the address of the 0-terminated packet is the last address of the packet transmitted from the transmitter. Then, the receiver restores the data by connecting the data of the packets of each address up to the 0-terminated packet (step S10124). Further, the receiver performs an error check of the restored data (step S10125). This makes it possible to send and receive variable-length address data even when it is not known how many pieces of data to be transmitted, that is, the address is variable-length instead of fixed-length, and fixed-length address data. More IDs can be sent and received with high efficiency.

As described above, in the present embodiment, the receiver acquires a plurality of packets including a data unit and an address unit from the plurality of emission line patterns, respectively. Then, the receiver determines whether or not there is a 0-terminated packet in which all the bits included in the data unit indicate 0 among the acquired plurality of packets. When it is determined that the 0-terminated packet exists, the receiver receives N packets including the address portion associated with the address portion of the 0-terminated packet among a plurality of packets (N is an integer of 1 or more). ) Is determined whether or not all the related packets exist. Next, when the receiver determines that all N related packets exist, the receiver acquires the visible light identifier (ID) by arranging and decoding the data portions of the N related packets side by side. Here, the address part associated with the address part of the 0-terminated packet is an address part that is smaller than the address shown in the address part of the 0-terminated packet and indicates an address of 0 or more.

Next, a reception method using an exposure time longer than the cycle of the modulation frequency will be described.

76 and 77 are diagrams for explaining a receiving method in which the receiver in the present embodiment uses an exposure time longer than the cycle of the modulation frequency (modulation cycle).

For example, as shown in FIG. 76 (a), if the exposure time is set to a time equal to the modulation period, the visible light signal may not be received correctly. The modulation period is the time of the above-mentioned one slot. That is, in such a case, there are few exposure lines (exposure lines shown in black in FIG. 76) that reflect the state of brightness of a certain slot. As a result, it is difficult to estimate the brightness of the transmitter when a lot of noise is accidentally included in the pixel values of these exposure lines.

On the other hand, for example, as shown in FIG. 76 (b), when the exposure time is set to a time longer than the modulation period, the visible light signal can be correctly received. That is, in such a case, since there are many exposure lines that reflect the brightness of a certain slot, the brightness of the transmitter can be estimated from the pixel values of many exposure lines, and it is resistant to noise.

On the contrary, if the exposure time is too long, the visible light signal cannot be received correctly.

For example, as shown in FIG. 77 (a), when the exposure time is equal to the modulation period, the luminance change received by the receiver (that is, the change in the pixel value of each exposure line) is used for transmission. Follows changes in brightness. However, as shown in FIG. 77 (b), when the exposure time is three times the modulation period, the luminance change received by the receiver can sufficiently follow the luminance change used for transmission. Can not. Further, as shown in FIG. 77 (c), when the exposure time is 10 times the modulation cycle, the luminance change received by the receiver cannot follow the luminance change used for transmission at all. That is, the longer the exposure time, the higher the noise immunity because the luminance can be estimated from many exposure lines, but the longer the exposure time, the lower the discrimination margin or the smaller the discrimination margin, and the lower the noise immunity. With these balances, the noise immunity can be maximized by setting the exposure time to about 2 to 5 times the modulation period.

Next, the number of packet divisions will be described.

FIG. 78 is a diagram showing an efficient number of divisions with respect to the size of transmission data.

When the transmitter transmits data by changing the brightness, if all the transmitted data (transmission data) is included in one packet, the data size of the packet is large. However, if the transmission data is divided into a plurality of partial data and the partial data is included in each packet, the data size of each packet becomes smaller. Here, the receiver receives the packet by imaging. However, the larger the data size of the packet, the more difficult it is for the receiver to receive the packet by one imaging, and it is necessary to repeat the imaging.

Therefore, as shown in FIGS. 78A and 78, it is desirable that the transmitter increases the number of divisions of the transmission data as the data size of the transmission data increases. However, if the number of divisions is too large, the transmission data cannot be restored unless all the partial data is received, so that the reception efficiency is conversely lowered.

Therefore, as shown in FIG. 78 (a), the data size (address size) of the address is variable, and the data size of the transmission data is 2-16 bits, 16-24 bits, 24-6 bits, 66-. In the case of 78 bits, 78-128 bits, 128 bits or more, transmission data to 1-2 pieces, 2-4 pieces, 4 pieces, 4-6 pieces, 6-8 pieces, 7 or more partial data, respectively. By dividing the data, the transmission data can be efficiently transmitted by the visible light signal. Further, as shown in FIG. 78 (b), the data size (address size) of the address is fixed to 4 bits, and the data size of the transmission data is 2-8 bits, 8-16 bits, 16-30 bits. In the case of 30-64 bits, 66-80 bits, 80-96 bits, 96-132 bits, 132 bits or more, 1-2 pieces, 2-3 pieces, 2-4 pieces, 4-5 pieces, respectively. By dividing the transmission data into 4-7, 6, 6-8, and 7 or more partial data, the transmission data can be efficiently transmitted by a visible light signal.

Further, the transmitter sequentially performs a luminance change based on each packet including each of the plurality of partial data. For example, the transmitter changes the luminance based on the packet in the order of the address of each packet. Further, the transmitter may perform the luminance change again based on the plurality of partial data in an order different from the address order. As a result, each partial data can be reliably received by the receiver.

Next, a method of setting the notification operation by the receiver will be described.

FIG. 79A is a diagram showing an example of the setting method in the present embodiment.

First, the receiver acquires the notification operation identifier for identifying the notification operation and the priority of the notification operation identifier (specifically, the identifier indicating the priority) from the server near the receiver. (Step S10131). Here, the notification operation is a reception that notifies the user of the receiver that each packet containing each of the plurality of partial data is transmitted by the luminance change and received by the receiver. It is the operation of the machine. For example, the action may be sounding, vibration, or screen display.

Next, the receiver receives the packetized visible light signal, that is, each packet including each of the plurality of partial data (step S10132). Here, the receiver acquires the notification operation identifier and the priority (specifically, the identifier indicating the priority) of the notification operation identifier included in the visible light signal (step S10133).

Further, the receiver has a setting content of the current notification operation of the receiver, that is, a notification operation identifier preset in the receiver and a priority (specifically, an identifier indicating the priority) of the notification operation identifier. ) And read (step S10134). The notification operation identifier preset in the receiver is set by, for example, a user operation.

Then, the receiver selects the identifier having the highest priority from the preset notification operation identifier and the notification operation identifier acquired in each of step S10131 and step S10133 (step S10135). Next, the receiver performs the operation indicated by the selected notification operation identifier by resetting the selected notification operation identifier to itself, and notifies the user of the reception of the visible light signal (step S10136).

The receiver may not perform either one of step S10131 and step S10133, and may select a notification operation identifier having a higher priority from the two notification operation identifiers.

The priority of the notification operation identifier transmitted from the server installed in the theater or the museum, or the priority of the notification operation identifier included in the visible light signal transmitted in those facilities is set high. May be good. As a result, regardless of the user's setting, it is possible to prevent the sound for reception notification from being sounded in the facility. Further, in other facilities, by lowering the priority of the notification operation identifier, the receiver can notify the reception by the operation according to the user's setting.

FIG. 79B is a diagram showing another example of the setting method in the present embodiment.

First, the receiver acquires the notification operation identifier for identifying the notification operation and the priority of the notification operation identifier (specifically, the identifier indicating the priority) from the server near the receiver. (Step S10141). Next, the receiver receives the packetized visible light signal, that is, each packet including each of the plurality of partial data (step S10142). Here, the receiver acquires the notification operation identifier and the priority (specifically, the identifier indicating the priority) of the notification operation identifier included in the visible light signal (step S10143).

Further, the receiver has a setting content of the current notification operation of the receiver, that is, a notification operation identifier preset in the receiver and a priority (specifically, an identifier indicating the priority) of the notification operation identifier. ) And read (step S10144).

Then, the receiver includes the operation notification identifier indicating the operation of prohibiting the generation of the notification sound in the preset notification operation identifier and the notification operation identifier acquired in each of steps S10141 and S10143. It is determined whether or not this is the case (step S10145). Here, if it is determined that it is included (Y in step S10145), the receiver sounds a notification sound for notifying the completion of reception (step S10146). On the other hand, if it is determined that it is not included (N in step S10145), the receiver notifies the user of the completion of reception by, for example, vibration (step S10147).

The receiver does not perform either step S10141 or step S10143, and determines whether or not the two notification operation identifiers include an operation notification identifier indicating an operation for prohibiting the generation of the notification sound. You may.

Further, the receiver may perform self-position estimation based on the image obtained by imaging, and notify the user of the reception by the operation associated with the estimated position or the facility at the estimated position.

FIG. 80 is a flowchart showing the processing of the information processing program according to the tenth embodiment.

This information processing program is a program for changing the brightness of the light emitter of the above-mentioned transmitter according to the number of divisions shown in FIG. 78.

That is, this information processing program is an information processing program that causes a computer to process the information to be transmitted in order to transmit the information to be transmitted by changing the brightness. Specifically, this information processing program has a coding step SA41 that generates a coded signal by coding information to be transmitted, and the number of bits of the generated coded signal is in the range of 24 to 64 bits. In some cases, the computer is made to perform a division step SA42 for dividing the coded signal into four partial signals and an output step SA43 for sequentially outputting the four partial signals. These partial signals are output as packets. Further, the information processing program may specify the number of bits of the coded signal and let the computer determine the number of partial signals based on the specified number of bits. In this case, the information processing program causes the computer to generate a determined number of partial signals by dividing the coded signal.

As a result, when the number of bits of the coded signal is in the range of 24 to 64 bits, the coded signal is divided into four partial signals and output. As a result, when the luminance of the light emitter changes according to the four output partial signals, the four partial signals are transmitted as visible light signals and received by the receiver. Here, as the number of bits of the output signal increases, it becomes difficult for the receiver to properly receive the signal by imaging, and the reception efficiency decreases. Therefore, it is desirable to divide the signal into a signal having a small number of bits, that is, a small signal. However, if the signal is divided into many small signals too finely, the receiver cannot receive the original signal unless each of all the small signals is received individually, resulting in a decrease in reception efficiency. Therefore, as described above, when the number of bits of the coded signal is in the range of 24 to 64 bits, the coded signal is divided into four partial signals and sequentially output to indicate the information to be transmitted. The coded signal can be transmitted as a visible light signal with the best reception efficiency. As a result, communication between various devices can be enabled.

Further, in the output step SA43, the four partial signals may be output according to the first order, and further, the four partial signals may be output again according to the second order different from the first order.

As a result, these four partial signals are repeatedly output in a different order. Therefore, when each output signal is transmitted to the receiver as a visible light signal, the reception efficiency of those four partial signals is reduced. It can be further enhanced. That is, even if the four partial signals are repeatedly output in the same order, the same partial signal may not be received by the receiver, but by changing the order, such a case can be suppressed.

Further, as shown in FIGS. 79A and 79B, in the output step SA43, the four partial signals may be further accompanied by the notification operation identifier and output. The notification operation identifier is for identifying the operation of the receiver that notifies the user of the receiver that the four partial signals have been received when the four partial signals are transmitted by the luminance change and received by the receiver. It is an identifier of.

As a result, when the notification operation identifier is transmitted as a visible light signal and received by the receiver, the receiver receives the four partial signals to the user according to the operation identified by the notification operation identifier. You can be notified. That is, the notification operation by the receiver can be set on the side that transmits the information to be transmitted.

Further, as shown in FIGS. 79A and 79B, in the output step SA43, a priority identifier for identifying the priority of the notification operation identifier may be further output attached to the four partial signals.

As a result, when the priority identifier and the notification operation identifier are transmitted as a visible light signal and received by the receiver, the receiver handles the notification operation identifier according to the priority identified by the priority identifier. be able to. That is, when the receiver has acquired another notification operation identifier, the receiver is identified by the notification operation identified by the notification operation identifier transmitted as a visible light signal and by the other notification operation identifier. One of the notification actions can be selected based on its priority.

The information processing program according to one aspect of the present invention is an information processing program that causes a computer to process the information of the transmission target in order to transmit the information of the transmission target by changing the brightness, and encodes the information of the transmission target. A coding step for generating a coded signal by the above means, and a dividing step for dividing the coded signal into four partial signals when the number of bits of the generated coded signal is in the range of 24 to 64 bits. , The computer is made to execute the output step of sequentially outputting the four partial signals.

As a result, as shown in FIGS. 77 to 80, when the number of bits of the coded signal is in the range of 24 to 64 bits, the coded signal is divided into four partial signals and output. As a result, when the luminance of the light emitter changes according to the four output partial signals, the four partial signals are transmitted as visible light signals and received by the receiver. Here, as the number of bits of the output signal increases, it becomes difficult for the receiver to properly receive the signal by imaging, and the reception efficiency decreases. Therefore, it is desirable to divide the signal into a signal having a small number of bits, that is, a small signal. However, if the signal is divided into many small signals too finely, the receiver cannot receive the original signal unless each of all the small signals is received individually, resulting in a decrease in reception efficiency. Therefore, as described above, when the number of bits of the coded signal is in the range of 24 to 64 bits, the coded signal is divided into four partial signals and sequentially output to indicate the information to be transmitted. The coded signal can be transmitted as a visible light signal with the best reception efficiency. As a result, communication between various devices can be enabled.

Further, in the output step, the four partial signals may be output according to the first order, and further, the four partial signals may be output again according to the second order different from the first order.

As a result, these four partial signals are repeatedly output in a different order. Therefore, when each output signal is transmitted to the receiver as a visible light signal, the reception efficiency of those four partial signals is reduced. It can be further enhanced. That is, even if the four partial signals are repeatedly output in the same order, the same partial signal may not be received by the receiver, but by changing the order, such a case can be suppressed.

Further, in the output step, the four partial signals are further output with the notification operation identifier attached, and the notification operation identifier is obtained when the four partial signals are transmitted by the luminance change and received by the receiver. In addition, it may be an identifier for identifying the operation of the receiver that notifies the user of the receiver that the four partial signals have been received.

As a result, when the notification operation identifier is transmitted as a visible light signal and received by the receiver, the receiver receives the four partial signals to the user according to the operation identified by the notification operation identifier. You can be notified. That is, the notification operation by the receiver can be set on the side that transmits the information to be transmitted.

Further, in the output step, a priority identifier for identifying the priority of the notification operation identifier may be attached to the four partial signals and output.

As a result, when the priority identifier and the notification operation identifier are transmitted as a visible light signal and received by the receiver, the receiver handles the notification operation identifier according to the priority identified by the priority identifier. be able to. That is, when the receiver has acquired another notification operation identifier, the receiver is identified by the notification operation identified by the notification operation identifier transmitted as a visible light signal and by the other notification operation identifier. One of the notification actions can be selected based on its priority.

Next, the registration of the network connection of the electronic device will be described.

FIG. 81 is a diagram for explaining an application example of the transmission / reception system according to the present embodiment.

This transmission / reception system includes a transmitter 10131b configured as an electronic device such as a washing machine, a receiver 10131a configured as a smartphone, for example, and a communication device 10131c configured as an access point or a router.

FIG. 82 is a flowchart showing the processing operation of the transmission / reception system according to the present embodiment.

When the start button is pressed (step S10165), the transmitter 10131b provides information for connecting to itself, such as an SSID, password, IP address, MAC address, or encryption key, to Wi-Fi, Bluetooth (registered trademark). ), Or via Ethernet (registered trademark) or the like (step S10166), and waits for a connection. The transmitter 10131b may transmit these information directly or indirectly. When transmitting indirectly, the transmitter 10131b transmits the ID associated with the information. The receiver 10131a that has received the ID downloads, for example, the information associated with the ID from a server or the like.

The receiver 10131a receives the information (step S10151), connects to the transmitter 10131b, and has information (SSID, password, IP address, MAC address, etc.) for connecting to the communication device 10131c configured as an access point or a router. Alternatively, the encryption key or the like) is transmitted to the transmitter 10131b (step S10152). The receiver 10131a registers information (MAC address, IP address, encryption key, etc.) for the transmitter 10131b to connect to the communication device 10131c in the communication device 10131c, and causes the communication device 10131c to listen for the connection. Further, the receiver 10131a notifies the transmitter 10131b that the preparation for connection from the transmitter 10131b to the communication device 10131c is completed (step S10153).

The transmitter 10131b disconnects from the receiver 10131a (step S10168) and connects to the communication device 10131c (step S10169). If the connection is successful (Y in step S10170), the transmitter 10131b notifies the receiver 10131a of the connection success via the communication device 10131c, and notifies the user of the connection success by the screen display, the LED status, the voice, or the like. (Step S10171). If the connection fails (N in step S10170), the transmitter 10131b notifies the receiver 10131a of the connection failure by visible light communication, and notifies the user in the same manner as when the connection fails (step S10172). The connection success may be notified by visible light communication.

The receiver 10131a is connected to the communication device 10131c (step S10154), and if there is no notification of connection success or failure (N in step S10155 and N in step S10156), the receiver 10131b can be accessed via the communication device 10131c. Check whether or not (step S10157). If this is not possible (N in step S10157), the receiver 10131a determines whether or not the connection to the transmitter 10131b using the information received from the transmitter 10131b has been made a predetermined number of times or more (step S10158). Here, if it is determined that the process has not been performed more than a predetermined number of times (N in step S10158), the receiver 10131a repeats the process from step S10152. On the other hand, if it is determined that the process has been performed more than a predetermined number of times (Y in step S10158), the receiver 10131a notifies the user of the processing failure (step S10159). Further, when the receiver 10131a determines in step S10156 that the connection success is notified (Y in step S10156), the receiver 10131a notifies the user of the processing success (step S10160). That is, the receiver 10131a notifies the user by screen display, voice, or the like whether or not the transmitter 10131b could be connected to the communication device 10131c. As a result, the transmitter 10131b can be connected to the communication device 10131c without requiring the user to input complicatedly.

Next, the registration of the network connection of the electronic device (when connecting via another electronic device) will be described.

FIG. 83 is a diagram for explaining an application example of the transmission / reception system according to the present embodiment.

This transmission / reception system includes an air conditioner 10133b, a transmitter 10133c configured as an electronic device such as a wireless adapter connected to the air conditioner 10133b, a receiver 10133a configured as a smartphone, for example, and a communication device configured as an access point or a router. The 10133d is provided with another electronic device 10133e configured as, for example, a wireless adapter, a wireless access point, a router, or the like.

FIG. 84 is a flowchart showing the processing operation of the transmission / reception system according to the present embodiment. Hereinafter, the air conditioner 10133b or the transmitter 10133c will be referred to as an electronic device A, and the electronic device 10133e will be referred to as an electronic device B.

First, when the start button is pressed (step S10188), the electronic device A transmits information (individual ID, password, IP address, MAC address, encryption key, etc.) for connecting to itself (step S10189). , Waiting for connection (step S10190). The electronic device A may directly transmit or indirectly transmit these information in the same manner as described above.

The receiver 10133a receives the information from the electronic device A (step S10181) and transmits the information to the electronic device B (step S10182). When the electronic device B receives the information (step S10196), the electronic device B connects to the electronic device A according to the received information (step S10197). Then, the electronic device B determines whether or not the connection with the electronic device A has been made (step S10198), and notifies the receiver 10133a of the success or failure (step S10199 or step S101200).

If the electronic device A is connected to the electronic device B within a predetermined time (Y in step S10191), the electronic device A notifies the receiver 10133a via the electronic device B (step S10192), and if it is not connected (step S10192). N) of step S10191, the receiver 10133a is notified of the connection failure by visible light communication (step S10193). Further, the electronic device A notifies the user of the success or failure of the connection by the screen display, the light emitting state, the voice, or the like. As a result, the electronic device A (transmitter 10133c) can be connected to the electronic device B (electronic device 10133e) without causing the user to perform complicated input. The air conditioner 10133b and the transmitter 10133c shown in FIG. 83 may be integrally configured, and similarly, the communication device 10133d and the electronic device 10133e may be integrally configured.

Next, transmission of appropriate imaging information will be described.

FIG. 85 is a diagram for explaining an application example of the transmission / reception system according to the present embodiment.

This transmission / reception system includes, for example, a receiver 10135a configured as a digital still camera or a digital video camera, and a transmitter 10135b configured as, for example, lighting.

FIG. 86 is a flowchart showing the processing operation of the transmission / reception system according to the present embodiment.

First, the receiver 10135a sends an image pickup information transmission command to the transmitter 10135b (step S10211). Next, when the transmitter 10135b receives the image pickup information transmission command, the image pickup information transmission button is pressed, the image pickup information transmission switch is on, and the power is turned on (step S10221). Y), the imaging information is transmitted (step S10222). The image pickup information transmission command is a command for transmitting the image pickup information, and the image pickup information indicates, for example, the color temperature, the spectral distribution, the illuminance, or the light distribution of the illumination. The transmitter 10135b may transmit the image pickup information directly or indirectly as described above. When transmitting indirectly, the transmitter 10135b transmits the ID associated with the image pickup information. The receiver 10135a that has received the ID downloads, for example, the image pickup information associated with the ID from a server or the like. At this time, the transmitter 10135b is a method for transmitting a transmission stop command to itself (radio wave, infrared ray, or sound wave frequency for transmitting the transmission stop command, SSID, password, IP address, etc. for connecting to self-confidence, etc. ) May be sent.

When the receiver 10135a receives the image pickup information (step S10212), the receiver 10135a transmits a transmission stop command to the transmitter 10135b (step S10213). Here, when the transmitter 10135b receives the transmission stop command from the receiver 10135a (in step S10223), the transmitter 10135b stops the transmission of the imaging information and emits light uniformly (step S10224).

Further, the receiver 10135a sets the imaging parameters according to the imaging information received in step S10212 (step S10214), or notifies the user of the imaging information. Imaging parameters are, for example, white balance, exposure time, focal length, sensitivity or scene mode. As a result, it is possible to take an image with the optimum setting according to the lighting. Next, the receiver 10135a takes an image (step S10216) after the transmission of the image pickup information from the transmitter 10135b is stopped (Y in step S10215). This makes it possible to perform imaging without changing the brightness of the subject due to signal transmission. After step S10216, the receiver 10135a may transmit a transmission start command prompting the start of transmission of the imaging information to the transmitter 10135b (step S10217).

Next, the display of the charging status will be described.

FIG. 87 is a diagram for explaining an application example of the transmitter according to the present embodiment.

For example, the transmitter 10137b configured as a charger includes a light emitting unit, and transmits a visible light signal indicating the state of charge of the battery from the light emitting unit. This makes it possible to notify the state of charge of the battery without having to provide an expensive display device. When a small LED is used as the light emitting unit, the visible light signal cannot be received unless the LED is imaged from a close distance. Further, in the transmitter 10137c having a protrusion near the LED, the protrusion is an obstacle and it is difficult to take a close-up shot of the LED. Therefore, the visible light signal from the transmitter 10137b, which has no protrusion near the LED, can be more easily received than the visible light signal from the transmitter 10137c.

(Embodiment 11)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

First, transmission in the demo mode and in the event of a failure will be described.

FIG. 88 is a diagram illustrating an example of the operation of the transmitter in the present embodiment.

If an error has occurred, the transmitter sends a signal indicating that an error has occurred, or a signal corresponding to the error code, so that the receiver has an error. You can tell the details of the error. The receiver can repair the error or report the error content appropriately to the service center by showing an appropriate response according to the error content.

The transmitter sends a demo code if it is in demo mode. As a result, for example, when a transmitter, which is a product, is being demonstrated at a store, a visitor can receive the demo code and obtain a product description associated with the demo code. Judgment as to whether or not it is in demo mode is that the transmitter operation setting is in demo mode, the storefront CAS card is inserted, the CAS card is not inserted, or the recording media is not inserted. It can be judged from the point.

Next, signal transmission from the remote controller will be described.

FIG. 89 is a diagram illustrating an example of the operation of the transmitter according to the present embodiment.

For example, when a transmitter configured as a remote controller for an air conditioner receives main unit information, the transmitter transmits the main unit information, so that the receiver receives information from a distant main unit from a nearby transmitter. can do. The receiver can also receive information from the main body that exists in a place where visible light communication is impossible, such as over a network.

Next, the process of transmitting only when the place is bright will be described.

FIG. 90 is a diagram illustrating an example of the operation of the transmitter in the present embodiment.

The transmitter performs transmission when the ambient brightness is above a certain level, and stops transmission when the ambient brightness is below a certain level. As a result, for example, the transmitter configured as an advertisement for a train can automatically stop the operation when the vehicle enters the garage, and the battery consumption can be suppressed.

Next, content distribution (change of association / scheduling) according to the display of the transmitter will be described.

FIG. 91 is a diagram illustrating an example of the operation of the transmitter according to the present embodiment.

The transmitter associates the content to be acquired by the receiver with the transmission ID according to the display timing of the content to be displayed. Every time the displayed content is changed, the change of the association is registered in the server.

When the display timing of the display content is known, the transmitter sets the server so that another content is passed to the receiver according to the change timing of the display content. When the receiver requests the content associated with the transmission ID, the server transmits the content according to the set schedule to the receiver.

As a result, for example, when the transmitter configured as digital signage changes the display contents one after another, the receiver can acquire the contents according to the contents displayed by the transmitter.

Next, content distribution (synchronization by time) according to the display of the transmitter will be described.

FIG. 92 is a diagram illustrating an example of the operation of the transmitter according to the present embodiment.

It is registered in advance in the server so that different contents are passed according to the time in response to the content acquisition request associated with the predetermined ID.

The transmitter synchronizes the time with the server, adjusts the timing so that a predetermined part is displayed at a predetermined time, and displays the content.

As a result, for example, when the transmitter configured as digital signage changes the display contents one after another, the receiver can acquire the contents according to the contents displayed by the transmitter.

Next, content distribution (transmission of display time) according to the display of the transmitter will be described.

FIG. 93 is a diagram illustrating an example of the operation of the transmitter and the receiver in the present embodiment.

The transmitter transmits the display time of the displayed content in addition to the ID of the transmitter. The content display time is information that can identify the currently displayed content, and can be expressed by, for example, the elapsed time from the start time of the content.

The receiver acquires the content associated with the received ID from the server and displays the content according to the received display time. As a result, for example, when the transmitter configured as digital signage changes the display contents one after another, the receiver can acquire the contents according to the contents displayed by the transmitter.

In addition, the receiver changes the displayed content over time. As a result, when the display content of the transmitter changes, the content matching the display content is displayed without receiving the signal again.

Next, uploading data according to the user's permission status will be described.

FIG. 94 is a diagram illustrating an example of the operation of the receiver in the present embodiment.

If the user has registered an account, the receiver will be the information that the user has permission to access when registering an account (receiver location, phone number, ID, installed apps, and user's information. Age, gender, occupation, preference, etc.) are sent to the server together with the received ID.

If the account is not registered, if the user allows uploading of information as described above, it will be sent to the server in the same way, and if not, only the received ID will be sent to the server. ..

As a result, the user can receive the content according to the situation at the time of reception and his / her personality, and the server can use it for data analysis by obtaining the user's information.

Next, the activation of the content playback application will be described.

FIG. 95 is a diagram illustrating an example of the operation of the receiver in the present embodiment.

The receiver acquires the content associated with the received ID from the server. If the running app can handle (display and play) the acquired content, the acquired content is displayed and played by the running app. If it cannot be handled, check whether the app that can be handled is installed in the receiver, and if it is installed, start the app and display / play the acquired content. If it is not installed, it will be installed automatically, a display prompting you to install it, a download screen will be displayed, and the acquired content will be displayed and played after installation.

As a result, the acquired content can be appropriately handled (displayed, played back, etc.).

Next, the start of the designated application will be described.

FIG. 96 is a diagram illustrating an example of the operation of the receiver in the present embodiment.

The receiver acquires the content associated with the received ID and the information (app ID) that specifies the application to be started from the server. If the running app is the specified app, the acquired content is displayed / played. If the specified application is installed in the receiver, start the specified application to display / play the acquired content. If it is not installed, it will be installed automatically, a display prompting you to install it, a download screen will be displayed, and the acquired content will be displayed and played after installation.

The receiver may acquire only the application ID from the server and start the designated application.

The receiver may make the specified settings. The receiver may set the specified parameters and launch the specified application.

Next, the selection between streaming reception and normal reception will be described.

FIG. 97 is a diagram illustrating an example of the operation of the receiver in the present embodiment.

When the value of the predetermined address of the received data is a predetermined value or the received data includes a predetermined identifier, the receiver determines that the signal is being streamed and receives the streaming data. Receive with. If not, it will be received by the normal reception method.

As a result, reception can be performed regardless of whether the signal is transmitted by either streaming distribution or normal distribution.

Next, private data will be described.

FIG. 98 is a diagram illustrating an example of the operation of the receiver in the present embodiment.

The receiver refers to the table in the application when the value of the received ID is within the predetermined range or includes the predetermined identifier, and if the received ID exists in the table, it is specified in the table. Get the content. If this is not the case, the content set as the reception ID is acquired from the server.

As a result, the content can be received without registering with the server. In addition, since it does not communicate with the server, a quick response can be obtained.

Next, the setting of the exposure time according to the frequency will be described.

FIG. 99 is a diagram illustrating an example of the operation of the receiver in the present embodiment.

The receiver detects the signal and recognizes the modulation frequency of the signal. The receiver sets the exposure time according to the cycle of the modulation frequency (modulation cycle). For example, the signal can be easily received by setting the exposure time to be about the same as the modulation period. Further, for example, by setting the exposure time to an integral multiple of the modulation cycle or a value close to it (about ± 30%), it is possible to facilitate reception of the signal by convolutional decoding.

Next, the optimum parameter setting of the transmitter will be described.

FIG. 100 is a diagram illustrating an example of operation of the receiver according to the present embodiment.

In addition to the data received from the transmitter, the receiver transmits the current location information and information related to the user (address, gender, age, preference, etc.) to the server. The server sends the parameters for the transmitter to operate optimally to the receiver according to the received information. The receiver sets the received parameters if it can be set in the transmitter. If it cannot be set, it displays the parameter and prompts the user to set the parameter.

This allows, for example, to operate the washing machine by optimizing the nature of the water in the area where the transmitter is used, or to operate the rice cooker to cook rice in the most suitable way for the type of rice used by the user. You can make it.

Next, an identifier indicating the structure of the data will be described.

FIG. 101 is a diagram illustrating an example of the configuration of transmission data in the present embodiment.

The information transmitted includes an identifier, from which the receiver can know the configuration of the subsequent part. For example, the length of the data, the type and length of the error correction code, the division point of the data, and the like can be specified.

As a result, the transmitter can change the type and length of the data body and the error correction code according to the nature of the transmitter and the communication path. Further, the transmitter can make the receiver acquire an ID corresponding to the content ID by transmitting the content ID in addition to the ID of the transmitter.

(Embodiment 12)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

FIG. 102 is a diagram for explaining the operation of the receiver in the present embodiment.

The receiver 1210a in the present embodiment switches the shutter speed between high speed and low speed, for example, in frame units when performing continuous shooting by the image sensor. Further, the receiver 1210a switches the processing for the frame into the barcode recognition processing and the visible light recognition processing based on the frame obtained by the photographing. Here, the barcode recognition process is a process of decoding a barcode displayed in a frame obtained by a low shutter speed. The visible light recognition process is a process of decoding the above-mentioned emission line pattern reflected in a frame obtained by a high shutter speed.

Such a receiver 1210a includes a video input unit 1211, a barcode / visible light identification unit 1212, a barcode recognition unit 1212a, a visible light recognition unit 1212b, and an output unit 1213.

The video input unit 1211 includes an image sensor and switches the shutter speed for shooting by the image sensor. That is, the video input unit 1211 alternately switches the shutter speed between low speed and high speed in units of frames, for example. More specifically, the video input unit 1211 switches the shutter speed to a high speed for odd-numbered frames and a low shutter speed for even-numbered frames. Shooting at a low shutter speed is shooting in the above-mentioned normal shooting mode, and shooting at a high shutter speed is shooting in the above-mentioned visible light communication mode. That is, when the shutter speed is low, the exposure time of each exposure line included in the image sensor is long, and a normally captured image on which the subject is projected can be obtained as a frame. Further, when the shutter speed is high, the exposure time of each exposure line included in the image sensor is short, and a visible light communication image on which the above-mentioned emission line is projected can be obtained as a frame.

The barcode / visible light identification unit 1212 switches the processing for the image by determining whether or not the barcode appears or whether or not the emission line appears in the image obtained by the video input unit 1211. For example, if a barcode appears in a frame obtained by shooting at a low shutter speed, the barcode / visible light identification unit 1212 causes the barcode recognition unit 1212a to perform processing on the image. On the other hand, if a bright line appears in an image obtained by shooting at a high shutter speed, the barcode / visible light identification unit 1212 causes the visible light recognition unit 1212b to perform processing on the image.

The barcode recognition unit 1212a decodes the barcode appearing in the frame obtained by shooting at a low shutter speed. The barcode recognition unit 1212a acquires barcode data (for example, a barcode identifier) by its decoding, and outputs the barcode identifier to the output unit 1213. The barcode may be a one-dimensional code or a two-dimensional code (for example, a QR code (registered trademark)).

The visible light recognition unit 1212b decodes the pattern of the emission line appearing in the frame obtained by shooting at a high shutter speed. The visible light recognition unit 1212b acquires visible light data (for example, a visible light identifier) by its decoding, and outputs the visible light identifier to the output unit 1213. The visible light data is the above-mentioned visible light signal.

The output unit 1213 displays only the frames obtained by shooting at a low shutter speed. Therefore, when the subject photographed by the video input unit 1211 is a barcode, the output unit 1213 displays the barcode. Further, when the subject photographed by the video input unit 1211 is a digital signage or the like that transmits a visible light signal, the output unit 1213 displays the image of the digital signage without displaying the pattern of the emission line. Then, when the output unit 1213 acquires the barcode identifier, the output unit 1213 acquires the information associated with the barcode identifier from, for example, a server and displays the information. When the visible light identifier is acquired, the output unit 1213 acquires the information associated with the visible light identifier from, for example, a server and displays the information.

That is, the receiver 1210a, which is a terminal device, includes an image sensor, and the shutter speed of the image sensor is switched between the first speed and the second speed, which is faster than the first speed, while switching the image sensor. Take continuous shots with. Then, (a) when the subject photographed by the image sensor is a bar code, the receiver 1210a acquires an image in which the bar code is reflected by shooting when the shutter speed is the first speed. The barcode identifier is acquired by decoding the barcode shown in the image. Further, (b) when the subject photographed by the image sensor is a light source (for example, a digital signage), the receiver 1210a includes a plurality of receivers 1210a included in the image sensor by photographing when the shutter speed is the second speed. An emission line image, which is an image including an emission line corresponding to each of the exposure lines of the above, is acquired. Then, the receiver 1210a acquires a visible light signal as a visible light identifier by decoding a plurality of emission line patterns included in the acquired emission line image. Further, the receiver 1210a displays an image obtained by shooting when the shutter speed is the first speed.

In the receiver 1210a of the present embodiment as described above, by switching between the barcode recognition process and the visible light recognition process, the barcode can be decoded and the visible light signal can be received. Furthermore, power consumption can be suppressed by switching.

The receiver in the present embodiment may perform image recognition processing at the same time as visible light processing instead of barcode recognition processing.

FIG. 103A is a diagram for explaining other operations of the receiver in the present embodiment.

The receiver 1210b in the present embodiment switches the shutter speed between high speed and low speed, for example, in frame units when performing continuous shooting by the image sensor. Further, the receiver 1210b simultaneously executes the image recognition process and the above-mentioned visible light recognition process on the image (frame) obtained by the photographing. The image recognition process is a process of recognizing a subject reflected in a frame obtained by a low shutter speed.

Such a receiver 1210b includes a video input unit 1211, an image recognition unit 1212c, a visible light recognition unit 1212b, and an output unit 1215.

The video input unit 1211 includes an image sensor and switches the shutter speed for shooting by the image sensor. That is, the video input unit 1211 alternately switches the shutter speed between low speed and high speed in units of frames, for example. More specifically, the video input unit 1211 switches the shutter speed to a high speed for odd-numbered frames and a low shutter speed for even-numbered frames. Shooting at a low shutter speed is shooting in the above-mentioned normal shooting mode, and shooting at a high shutter speed is shooting in the above-mentioned visible light communication mode. That is, when the shutter speed is low, the exposure time of each exposure line included in the image sensor is long, and a normally captured image on which the subject is projected can be obtained as a frame. Further, when the shutter speed is high, the exposure time of each exposure line included in the image sensor is short, and a visible light communication image on which the above-mentioned emission line is projected can be obtained as a frame.

The image recognition unit 1212c recognizes the subject appearing in the frame obtained by shooting at a low shutter speed, and identifies the position of the subject in the frame. As a result of recognition, the image recognition unit 1212c determines whether or not the subject is an object of AR (Augmented Reality) (hereinafter referred to as an AR object). Then, when the image recognition unit 1212c determines that the subject is an AR object, the image recognition unit 1212c generates image recognition data which is data for displaying information about the subject (for example, the position of the subject and an AR marker), and the image recognition data thereof is generated. The AR marker is output to the output unit 1215.

The output unit 1215 displays only the frames obtained by shooting at a low shutter speed, similarly to the output unit 1213 described above. Therefore, when the subject photographed by the video input unit 1211 is a digital signage or the like that transmits a visible light signal, the output unit 1213 displays the image of the digital signage without displaying the pattern of the emission line. Further, when the output unit 1215 acquires the image recognition data from the image recognition unit 1212c, the output unit 1215 superimposes a white frame-shaped indicator surrounding the subject on the frame based on the position of the subject in the frame indicated by the image recognition data. ..

FIG. 103B is a diagram showing an example of an indicator displayed by the output unit 1215.

The output unit 1215 superimposes, for example, a white frame-shaped indicator 1215b surrounding the image 1215a of the subject configured as digital signage on the frame. That is, the output unit 1215 displays the indicator 1215b indicating the image-recognized subject. Further, when the output unit 1215 acquires the visible light identifier from the visible light recognition unit 1212b, the output unit 1215 changes the color of the indicator 1215b from, for example, white to red.

FIG. 103C is a diagram showing an AR display example.

The output unit 1215 further acquires information about the subject associated with the visible light identifier as related information from, for example, a server. The output unit 1215 describes the related information on the AR marker 1215c indicated by the image recognition data, and displays the AR marker 1215c on which the related information is described in association with the image 1215a of the subject in the frame.

In the receiver 1210b in the present embodiment as described above, AR using visible light communication can be realized by simultaneously performing the image recognition process and the visible light recognition process. The receiver 1210a shown in FIG. 103A may also display the indicator 1215b shown in FIG. 103B in the same manner as the receiver 1210b. In this case, the receiver 1210a displays a white frame-shaped indicator 1215b surrounding the barcode when the barcode is recognized in the frame obtained by shooting at a low shutter speed. Then, when the barcode is decoded, the receiver 1210a changes the color of the indicator 1215b from white to red. Similarly, when the receiver 1210a recognizes a bright line pattern in a frame obtained by photographing at a high shutter speed, the receiver 1210a identifies a portion in the low-speed frame corresponding to the portion having the bright line pattern. For example, if the digital signage is transmitting a visible light signal, the image of the digital signage in the slow frame is identified. The low-speed frame is a frame obtained by shooting at a low shutter speed. Then, the receiver 1210a superimposes and displays the white frame-shaped indicator 1215b surrounding the specified portion in the low-speed frame (for example, the image of the above-mentioned digital signage) on the low-speed frame. Then, when the pattern of the emission line is decoded, the receiver 1210a changes the color of the indicator 1215b from white to red.

FIG. 104A is a diagram for explaining an example of a transmitter according to the present embodiment.

The transmitter 1220a in the present embodiment transmits a visible light signal in synchronization with the transmitter 1230. That is, the transmitter 1220a transmits the same visible light signal as the visible light signal at the timing when the transmitter 1230 transmits the visible light signal. The transmitter 1230 includes a light emitting unit 1231, and the light emitting unit 1231 transmits a visible light signal by changing the luminance.

Such a transmitter 1220a includes a light receiving unit 1221, a signal analysis unit 1222, a transmission clock adjusting unit 1223a, and a light emitting unit 1224. The light emitting unit 1224 transmits the same visible light signal as the visible light signal transmitted from the transmitter 1230 by changing the luminance. The light receiving unit 1221 receives the visible light signal from the transmitter 1230 by receiving the visible light from the transmitter 1230. The signal analysis unit 1222 analyzes the visible light signal received by the light receiving unit 1221 and transmits the analysis result to the transmission clock adjustment unit 1223a. The transmission clock adjusting unit 1223a adjusts the timing of the visible light signal transmitted from the light emitting unit 1224 based on the analysis result. That is, the transmission clock adjusting unit 1223a uses the light emitting unit 1224 so that the timing at which the visible light signal is transmitted from the light emitting unit 1231 of the transmitter 1230 coincides with the timing at which the visible light signal is transmitted from the light emitting unit 1224. Adjust the timing of the brightness change.

Thereby, the waveform of the visible light signal transmitted by the transmitter 1220a and the waveform of the visible light signal transmitted by the transmitter 1230 can be timely matched.

FIG. 104B is a diagram for explaining another example of the transmitter in this embodiment.

The transmitter 1220b in the present embodiment transmits a visible light signal in synchronization with the transmitter 1230, similarly to the transmitter 1220a. That is, the transmitter 1200b transmits the same visible light signal as the visible light signal at the timing when the transmitter 1230 transmits the visible light signal.

Such a transmitter 1220b includes a first light receiving unit 1221a, a second light receiving unit 1221b, a comparison unit 1225, a transmission clock adjusting unit 1223b, and a light emitting unit 1224.

Similar to the light receiving unit 1221, the first light receiving unit 1221a receives the visible light signal from the transmitter 1230 by receiving the visible light from the transmitter 1230. The second light receiving unit 1221b receives visible light from the light emitting unit 1224. The comparison unit 1225 compares the first timing at which visible light is received by the first light receiving unit 1221a with the second timing at which visible light is received by the second light receiving unit 1221b. Then, the comparison unit 1225 outputs the difference (that is, the delay time) between the first timing and the second timing to the transmission clock adjustment unit 1223b. The transmission clock adjusting unit 1223b adjusts the timing of the visible light signal transmitted from the light emitting unit 1224 so that the delay time is shortened.

Thereby, the waveform of the visible light signal transmitted by the transmitter 1220b and the waveform of the visible light signal transmitted by the transmitter 1230 can be matched more accurately in terms of timing.

In the example shown in FIGS. 104A and 104B, the two transmitters transmit the same visible light signal, but different visible light signals may be transmitted. That is, when the two transmitters transmit the same visible light signal, they transmit in synchronization as described above. Then, when the two transmitters transmit different visible light signals, only one of the two transmitters transmits the visible light signal, while the other transmitter is uniformly turned on or off. do. After that, one transmitter is uniformly turned on or off, while only the other transmitter transmits a visible light signal. It should be noted that the two transmitters may simultaneously transmit different visible light signals.

FIG. 105A is a diagram for explaining an example of synchronous transmission by a plurality of transmitters in the present embodiment.

The plurality of transmitters 1220 in this embodiment are arranged, for example, in a row as shown in FIG. 105A. These transmitters 1220 have the same configuration as the transmitter 1220a shown in FIG. 104A or the transmitter 1220b shown in FIG. 104B. Each of such a plurality of transmitters 1220 transmits a visible light signal in synchronization with one of the transmitters 1220 on both sides.

This allows many transmitters to transmit visible light signals synchronously.

FIG. 105B is a diagram for explaining an example of synchronous transmission by a plurality of transmitters in the present embodiment.

The transmitter 1220 of the plurality of transmitters 1220 in the present embodiment serves as a reference for synchronizing the visible light signal, and the remaining plurality of transmitters 1220 serve as a reference for the visible light signal so as to match the reference. To send.

This allows many transmitters to more accurately synchronize and transmit visible light signals.

FIG. 106 is a diagram for explaining another example of synchronous transmission by a plurality of transmitters in the present embodiment.

Each of the plurality of transmitters 1240 in the present embodiment receives a synchronization signal and transmits a visible light signal according to the synchronization signal. As a result, visible light signals are transmitted synchronously from each of the plurality of transmitters 1240.

Specifically, each of the plurality of transmitters 1240 includes a control unit 1241, a synchronous control unit 1242, a photocoupler 1243, an LED drive circuit 1244, an LED 1245, and a photodiode 1246.

The control unit 1241 receives the synchronization signal and outputs the synchronization signal to the synchronization control unit 1242.

The LED1245 is a light source that emits visible light, and blinks (that is, changes in brightness) in response to control by the LED drive circuit 1244. As a result, the visible light signal is transmitted from the LED 1245 to the outside of the transmitter 1240.

The photocoupler 1243 electrically insulates between the synchronization control unit 1242 and the LED drive circuit 1244, and transmits a signal between them. Specifically, the photocoupler 1243 transmits a transmission start signal, which will be described later, transmitted from the synchronization control unit 1242 to the LED drive circuit 1244.

When the LED drive circuit 1244 receives the transmission start signal from the synchronization control unit 1242 via the photocoupler 1243, the LED 1245 starts the transmission of the visible light signal at the timing when the transmission start signal is received.

The photodiode 1246 detects visible light emitted from the LED 1245 and outputs a detection signal indicating that the visible light has been detected to the synchronization control unit 1242.

When the synchronization control unit 1242 receives the synchronization signal from the control unit 1241, the synchronization control unit 124 transmits the transmission start signal to the LED drive circuit 1244 via the photocoupler 1243. By transmitting this transmission start signal, transmission of the visible light signal is started. Further, when the synchronization control unit 1242 receives the detection signal from the photodiode 1246 by transmitting the visible light signal, the delay is the difference between the timing of receiving the detection signal and the timing of receiving the synchronization signal from the control unit 1241. Calculate the time. When the synchronization control unit 1242 receives the next synchronization signal from the control unit 1241, the synchronization control unit 124 adjusts the timing of transmitting the next transmission start signal based on the calculated delay time. That is, the synchronization control unit 1242 adjusts the timing of transmitting the next transmission start signal so that the delay time for the next synchronization signal becomes a predetermined set delay time. As a result, the synchronization control unit 1242 transmits the next transmission start signal at the adjusted timing.

FIG. 107 is a diagram for explaining signal processing in the transmitter 1240.

Upon receiving the synchronization signal, the synchronization control unit 1242 generates a delay time setting signal in which a delay time setting pulse is generated at a predetermined timing. Note that receiving a synchronization signal specifically means receiving a synchronization pulse. That is, the synchronization control unit 1242 generates a delay time setting signal so that the delay time setting pulse rises at the timing when the above-mentioned set delay time elapses from the fall of the synchronization pulse.

Then, the synchronization control unit 1242 transmits the transmission start signal to the LED drive circuit 1244 via the photocoupler 1243 at a timing delayed by the correction value N obtained last time from the fall of the synchronization pulse. As a result, the LED drive circuit 1244 transmits a visible light signal from the LED 1245. Here, the synchronization control unit 1242 receives the detection signal from the photodiode 1246 at a timing delayed by the sum of the intrinsic delay time and the correction value N from the fall of the synchronization pulse. That is, the transmission of the visible light signal is started from that timing. Hereinafter, the timing is referred to as a transmission start timing. The above-mentioned inherent delay time is a delay time caused by a circuit such as the photocoupler 1243, and is a delay time that occurs even if the synchronization control unit 1242 receives the synchronization signal and immediately transmits the transmission start signal. ..

The synchronization control unit 1242 specifies the time difference from the transmission start timing to the rise of the delay time setting pulse as the correction correction value N. Then, the synchronization control unit 1242 calculates and holds the correction value (N + 1) by the correction value (N + 1) = correction value N + correction correction value N. As a result, when the synchronization control unit 1242 receives the next synchronization signal (synchronization pulse), the synchronization control unit 1242 transmits the transmission start signal to the LED drive circuit 1244 at a timing delayed by the correction value (N + 1) from the fall of the synchronization pulse. do. The correction correction value N can be not only a positive value but also a negative value.

As a result, each of the plurality of transmitters 1240 transmits the visible light signal after the set delay time elapses after receiving the synchronization signal (synchronization pulse), so that the visible light signal can be transmitted in exact synchronization. can. That is, even if there is a variation in the intrinsic delay time due to a circuit such as the photocoupler 1243 in each of the plurality of transmitters 1240, it is visible from each of the plurality of transmitters 1240 without being affected by the variation. The transmission of optical signals can be accurately synchronized.

The LED drive circuit consumes a large amount of electric power, and is electrically isolated from the control circuit that handles the synchronization signal by using a photocoupler or the like. Therefore, when such a photocoupler is used, it is difficult to synchronize the transmission of visible light signals from a plurality of transmitters due to the variation in the inherent delay time described above. However, in the plurality of transmitters 1240 in the present embodiment, the emission timing of the LED 1245 is detected by the photodiode 1246, the delay time from the synchronization signal is detected by the synchronization control unit 1242, and the delay time is set in advance. It is adjusted to be the time (the above-mentioned set delay time). As a result, even if there are individual variations in the photocouplers provided in a plurality of transmitters, each of which is configured as LED lighting, a visible light signal (for example, a visible light ID) is synchronized with high accuracy from a plurality of LED lights. You can send it with.

The LED lighting may be turned on or off except during the visible light signal transmission period. When lighting other than the visible light signal transmission period, the first falling edge of the visible light signal may be detected. When the light is turned off except for the visible light signal transmission period, the first rising edge of the visible light signal may be detected.

In the above example, the transmitter 1240 transmits a visible light signal each time it receives a synchronization signal, but it may transmit a visible light signal without receiving the synchronization signal. That is, once the visible light signal is transmitted in response to the reception of the synchronization signal, the transmitter 1240 may sequentially transmit the visible light signal without receiving the synchronization signal. Specifically, the transmitter 1240 may sequentially transmit the visible light signal two to several thousand times for one reception of the synchronization signal. Further, the transmitter 1240 may transmit a visible light signal according to the synchronization signal at a rate of once every 100 ms or once every several seconds.

Further, when the transmission of the visible light signal corresponding to the synchronization signal is repeatedly performed, the continuity of light emission of the LED 1245 may be lost due to the above-mentioned set delay time. In other words, a slightly longer blanking period may occur. As a result, the blinking of the LED 1245 may be visually recognized by a person, and so-called flicker may occur. Therefore, the transmitter 1240 may transmit a visible light signal according to the synchronization signal at a cycle of 60 Hz or more. As a result, the blinking is performed at high speed, and the blinking becomes difficult for humans to see. As a result, the occurrence of flicker can be suppressed. Alternatively, the transmitter 1240 may transmit the visible light signal according to the synchronization signal at a sufficiently long cycle such as once every few minutes. As a result, the blinking is visually recognized by a person, but it is possible to prevent the blinking from being repeatedly visually recognized, and it is possible to reduce the discomfort that the flicker gives to the person.

(Pre-processing of reception method)
FIG. 108 is a flowchart showing an example of the receiving method according to the present embodiment. Further, FIG. 109 is an explanatory diagram for explaining an example of the receiving method in the present embodiment.

First, the receiver calculates the average value of the pixel values of the plurality of pixels arranged in the direction parallel to the exposure line (step S1211). By averaging the pixel values of N pixels by the central limit theorem, the expected value of the amount of noise is N minus 1/2, and the SN ratio is improved.

Next, the receiver leaves only the portion where the pixel value changes in the same direction in the vertical direction for each of the colors, and removes the change in the pixel value in the portion where the pixel value changes differently (step S1212). When the transmission signal (visible light signal) is expressed by the brightness of the light emitting unit provided in the transmitter, the brightness of the illumination of the transmitter or the backlight of the display changes. At this time, as shown in the portion (b) of FIG. 109, the pixel values change in the same direction for each of all the colors. In the parts (a) and (c) of FIG. 109, the pixel values change differently for each color. In these parts, the pixel values fluctuate due to reception noise or the picture of the display or signage, so the SN ratio can be improved by removing these fluctuations.

Next, the receiver obtains the luminance value (step S1213). Since the brightness is not easily changed by the color, the influence of the picture on the display or signage can be eliminated, and the SN ratio can be improved.

Next, the receiver applies a low-pass filter to the luminance value (step S1214). In the receiving method in the present embodiment, since the moving average filter according to the length of the exposure time is applied, almost no signal is included in the high frequency region, and noise becomes dominant. Therefore, the SN ratio can be improved by using a low-pass filter that cuts the high frequency region. Since there are many signal components up to the frequency up to the reciprocal of the exposure time, the effect of improving the SN ratio can be increased by blocking frequencies beyond that. When the frequency component contained in the signal is finite, the SN ratio can be improved by blocking the frequency higher than the frequency. A filter that does not contain a frequency vibration component (Butterworth filter, etc.) is suitable for the low-pass filter.

(Reception method by convolution maximum likelihood decoding)
FIG. 110 is a flowchart showing another example of the receiving method according to the present embodiment. Hereinafter, the reception method when the exposure time is longer than the transmission cycle will be described with reference to this figure.

When the exposure time is an integral multiple of the transmission cycle, reception can be performed with the highest accuracy. Even if it is not an integral multiple, reception can be performed as long as it is in the range of (N ± 0.33) times (N is an integer).

First, the receiver sets the transmission / reception offset to 0 (step S1221). The transmission / reception offset is a value for correcting the difference between the transmission timing and the reception timing. Since this deviation is unknown, the receiver gradually changes the candidate value of the transmission / reception offset, and adopts the value that best matches the transmission / reception offset for the transmission / reception offset.

Next, the receiver determines whether or not the transmission / reception offset is less than the transmission cycle (step S1222). Here, since the reception cycle and the transmission cycle are not synchronized, the reception value according to the transmission cycle is not always obtained. Therefore, when the receiver determines in step S1222 that it is less than the transmission cycle (Y in step S1222), the receiver uses the reception value in the vicinity thereof and uses the reception value in the vicinity thereof for each transmission cycle to match the reception value (for example, the pixel value). ) Is calculated by interpolation (step S1223). As the interpolation method, linear interpolation, nearest neighbor value, spline interpolation, or the like can be used. Next, the receiver obtains the difference between the received values obtained for each transmission cycle (step S1224).

The receiver adds a predetermined value to the transmission / reception offset (step S1226), and repeatedly executes the process from step S1222. Further, when the receiver determines in step S1222 that the likelihood is not less than the transmission cycle (N in step S1222), the receiver identifies the highest likelihood of the received signal calculated for each transmission / reception offset. Then, the receiver determines whether or not the highest likelihood is equal to or higher than a predetermined value (step S1227). When it is determined that the value is equal to or higher than a predetermined value (Y in step S1227), the receiver uses the received signal having the highest likelihood as the final estimation result. Alternatively, the receiver uses a received signal having a likelihood equal to or higher than the value obtained by subtracting a predetermined value from the highest likelihood as a received signal candidate (step S1228). On the other hand, if it is determined in step S1227 that the highest likelihood is less than a predetermined value (N in step S1227), the receiver discards the estimation result (step S1229).

If there is too much noise, the received signal cannot be estimated properly, and at the same time, the likelihood becomes low. Therefore, when the likelihood is low, the reliability of the received signal can be improved by discarding the estimation result. Further, even if the input image does not contain a valid signal, there is a problem that the maximum likelihood decoding outputs a valid signal as an estimation result. However, since the likelihood is low in this case as well, this problem can be avoided by discarding the estimation result when the likelihood is low.

(Embodiment 13)
In this embodiment, a protocol transmission method for visible light communication will be described.

(Multi-value amplitude pulse signal)
111, 112 and 113 are diagrams showing an example of a transmission signal according to the present embodiment.

By giving meaning to the pulse amplitude, more information can be expressed per unit time. For example, when the amplitude is classified into three stages, as shown in FIG. 111, the three values can be expressed by the transmission time of two slots while keeping the average brightness at 50%. However, when (c) of FIG. 111 is continuously transmitted, there is no change in luminance, so it is difficult to understand the existence of the signal. In addition, the three values are a little difficult to handle in digital processing.

Therefore, by using the four types of symbols (a) to (d) in FIG. 112, it is possible to express four values with an average transmission time of three slots while keeping the average brightness at 50%. Although the transmission time differs depending on the symbol, the end point of the symbol can be recognized by setting the final state of the symbol to a low brightness state. The same effect can be obtained by switching between the high brightness state and the low brightness state. (E) in FIG. 112 is not suitable because it is indistinguishable from transmitting (a) twice. (F) and (g) in FIG. 112 are somewhat difficult to recognize because the intermediate luminance is continuous, but they can be used.

Consider using the patterns (a) and (b) in FIG. 113 as headers. Since these patterns have a strong specific frequency component in frequency analysis, signal detection can be performed by frequency analysis by using these patterns as headers.

As shown in FIG. 113 (c), the transmission packet is configured by using the patterns (a) and (b). Data can be separated by using a pattern of a specific length as the header of the entire packet and using patterns of different lengths as the separator. Further, by including this pattern in the middle portion, signal detection can be facilitated. As a result, even if one packet is longer than the imaging time of one frame of image, the receiver can connect and decode the data. Further, by adjusting the number of separators, the length of the packet can be made variable. The length of the entire packet may be expressed by the length of the pattern of the packet header. Further, by using the separator as the packet header and the length of the separator as the address of the data, the receiver can partially synthesize the received data.

The transmitter repeatedly transmits the packet configured in this way. The contents of packets 1 to 4 in FIG. 113 (c) may be all the same, or may be combined as different data on the receiving side.

(Embodiment 14)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

FIG. 114A is a diagram for explaining a transmitter according to the present embodiment.

The transmitter in the present embodiment is configured as, for example, a backlight of a liquid crystal display, and includes a blue LED 2303 and a phosphor 2310 composed of a green fluorescent component 2304 and a red fluorescent component 2305.

The blue LED 2303 emits blue (B) light. When the phosphor 2310 receives the blue light emitted from the blue LED 2303 as the excitation light, the phosphor 2310 emits yellow light (Y). That is, the phosphor 2310 emits yellow light. Specifically, since the fluorescent substance 2130 is composed of a green fluorescent component 2304 and a red fluorescent component 2305, it emits yellow light by emitting light of these fluorescent components. Of these two fluorescent components, the green fluorescent component 2304 emits green light when it receives the blue light emitted from the blue LED 2303 as excitation light. That is, the green fluorescent component 2304 emits green (G) light. Of the above two fluorescent components, the red fluorescent component 2305 emits red light when it receives the blue light emitted from the blue LED 2303 as excitation light. That is, the red fluorescent component 2305 emits red (R) light. As a result, RGB or Y (RG) B light is emitted, so that the transmitter outputs white light as a backlight.

This transmitter transmits a visible light signal of white light by changing the brightness of the blue LED 2303 in the same manner as in each of the above embodiments. At this time, a visible light signal having a predetermined carrier frequency is output by changing the brightness of the white light.

Here, the barcode reader irradiates the barcode with a red laser beam and reads the barcode based on the change in the brightness of the red laser beam reflected from the barcode. The reading frequency of the barcode in this red laser light may match or approximate the carrier frequency of the visible light signal output from a general transmitter currently in practical use. Therefore, in such a case, when the barcode reader tries to read the barcode illuminated by the white light which is a visible light signal from the general transmitter, the brightness of the red light contained in the white light is increased. The change may cause the read to fail. That is, a barcode reading error occurs due to the interference between the carrier frequency of the visible light signal (particularly red light) and the barcode reading frequency.

Therefore, for the red fluorescent component 2305 in the present embodiment, a fluorescent material having a longer afterglow duration than the green fluorescent component 2304 is used. That is, the red fluorescent component 2305 in the present embodiment changes its brightness at a frequency sufficiently lower than the frequency of the brightness change of the blue LED 2303 and the green fluorescent component 2304. In other words, the red fluorescent component 2305 blunts the frequency of the red luminance change contained in the visible light signal.

FIG. 114B is a diagram showing each luminance change of RGB.

The blue light from the blue LED 2303 is included in the visible light signal and output as shown in FIG. 114B (a). As shown in FIG. 114B (b), the green fluorescent component 2304 emits green light when it receives blue light from the blue LED 2303. The duration of the afterglow in this green fluorescent component 2304 is short. Therefore, when the brightness of the blue LED 2303 is changed, the green fluorescent component 2304 emits green light whose brightness changes at substantially the same frequency as the frequency of the brightness change of the blue LED 2303 (that is, the carrier frequency of the visible light signal). ..

As shown in FIG. 114B (c), the red fluorescent component 2305 emits red light when it receives the blue light from the blue LED 2303. The duration of the afterglow in this red fluorescent component 2305 is long. Therefore, when the brightness of the blue LED 2303 is changed, the red fluorescence component 2305 emits red light whose brightness changes at a frequency lower than the frequency of the brightness change of the blue LED 2303 (that is, the carrier frequency of the visible light signal). ..

FIG. 115 is a diagram showing the afterglow characteristics of the green fluorescent component 2304 and the red fluorescent component 2305 in the present embodiment.

For example, when the blue LED 2303 is lit without changing the brightness, the green fluorescence component 2304 emits green light having an intensity I = I0 without changing the brightness (that is, light having a brightness change frequency f = 0). .. Further, even if the brightness of the blue LED 2303 changes at a low frequency, the green fluorescence component 2304 emits green light having an intensity I = I0, which changes the brightness at substantially the same frequency f as the low frequency. However, when the brightness of the blue LED 2303 changes at a high frequency, the intensity I of the green light emitted from the green fluorescence component 2304, which changes the brightness at substantially the same frequency f as the high frequency, is affected by the afterglow in the green fluorescence component 2304. Will be smaller than the intensity I0. As a result, the intensity I of the green light emitted from the green fluorescent component 2304 is kept at I = I0 when the frequency f of the brightness change of the light is less than the threshold value fb, as shown by the dotted line in FIG. 115. However, when the frequency f becomes higher than the threshold value fb, it gradually becomes smaller.

Further, the duration of the afterglow of the red fluorescent component 2305 in the present embodiment is longer than the duration of the afterglow of the green fluorescent component 2304. Therefore, as shown by the solid line in FIG. 115, the intensity I of the red light emitted from the red fluorescent component 2305 is such that the frequency f of the brightness change of the light is less than the threshold value fa lower than the threshold value fb. It is kept at I0, but gradually decreases as the frequency f becomes higher than the threshold value fb. In other words, the red light emitted from the red fluorescent component 2305 does not exist in the high frequency region of the frequency band of the green light emitted from the green fluorescent component 2304, but exists only in the low frequency region.

More specifically, for the red fluorescent component 2305 in the present embodiment, a fluorescent material having an intensity I of red light emitted at the same frequency f as the carrier frequency f1 of the visible light signal is I = I1 is used. Be done. The carrier frequency f1 is a carrier frequency for changing the luminance by the blue LED 2303 provided in the transmitter. Further, the above-mentioned strength I1 is 1/3 of the strength I0 or -10 dB of the strength I0. For example, the carrier frequency f1 is 10 kHz, or 5 to 100 kHz.

That is, the transmitter in the present embodiment is a transmitter that transmits a visible light signal, and receives a blue LED that emits blue light whose brightness changes as light included in the visible light signal and the blue light. This comprises a green fluorescent component that emits green light as light contained in the visible light signal, and a red fluorescent component that emits red light as light contained in the visible light signal by receiving the blue light. The duration of the afterglow in the red fluorescent component is longer than the duration of the afterglow in the green fluorescent component. The green fluorescent component and the red fluorescent component may be contained in a single phosphor that emits yellow light as light included in the visible light signal by receiving the blue light. Alternatively, the green fluorescent component may be contained in the green fluorescent substance, and the red fluorescent component may be contained in a red fluorescent substance different from the green fluorescent substance.

As a result, since the duration of the afterglow in the red fluorescent component is long, the brightness of the red light can be changed at a frequency lower than the frequency of the change in the brightness of the blue and green lights. Therefore, even if the frequency in the brightness change of the blue and green light contained in the visible light signal of white light is the same as or close to the reading frequency of the bar code in the red laser light, it is included in the visible light signal of white light. The frequency of the red light can be significantly different from the reading frequency of the bar code. As a result, it is possible to suppress the occurrence of barcode reading errors.

Here, the red fluorescent component may emit red light whose luminance changes at a frequency lower than the frequency of the luminance change of the light emitted from the blue LED.

Further, the red fluorescent component may include a red fluorescent material that emits red light by receiving blue light, and a low-pass filter that transmits only light in a predetermined frequency band. For example, the low-pass filter transmits only the light in the low frequency band among the blue light emitted from the blue LED and applies it to the red fluorescent material. The red fluorescent material may have the same afterglow characteristics as the green fluorescent component. Alternatively, the low-pass filter transmits only the light in the low frequency band among the red light emitted from the red fluorescent material when the blue light emitted from the blue LED hits the red fluorescent material. Let me. Even when such a low-pass filter is used, it is possible to suppress the occurrence of a barcode reading error as described above.

Further, the red fluorescent component may be made of a fluorescent material having predetermined afterglow characteristics. For example, the predetermined afterglow characteristics are (a) the intensity of the red light when the frequency f of the brightness change of the red light emitted from the red fluorescent component is 0, and (b). When the carrier frequency in the change in the brightness of the light emitted from the blue LED is f1, the intensity of the red light is 1/3 or less of the I0 when the frequency f of the red light is f = f1. , Or -10 dB or less, which is a characteristic.

As a result, the frequency of the red light contained in the visible light signal can be surely greatly different from the reading frequency of the barcode. As a result, it is possible to reliably suppress the occurrence of barcode reading errors.

Further, the carrier frequency f1 may be approximately 10 kHz.

As a result, since the carrier frequency used for transmitting the visible light signal currently in practical use is 9.6 kHz, it is effective to generate a barcode reading error in the transmission of the visible light signal in practical use. Can be suppressed.

Further, the carrier frequency f1 may be approximately 5 to 100 kHz.

With the advancement of image sensors (imaging elements) of receivers that receive visible light signals, it is expected that carrier frequencies such as 20 kHz, 40 kHz, 80 kHz, or 100 kHz will be used in future visible light communication. Therefore, by setting the above-mentioned carrier frequency f1 to approximately 5 to 100 kHz, it is possible to effectively suppress the occurrence of barcode reading errors even in future visible light communication.

In the present embodiment, regardless of whether the green fluorescent component and the red fluorescent component are contained in a single fluorescent substance, or whether each of the two fluorescent components is contained in a separate fluorescent body. Each of the above effects can be achieved. That is, even when a single phosphor is used, the afterglow characteristics, that is, the frequency characteristics of the red light and the green light emitted from the phosphor are different. Therefore, each of the above effects can be achieved by using a single phosphor having poor afterglow characteristics or frequency characteristics in red light and superior in afterglow characteristics or frequency characteristics in green light. Inferior afterglow characteristics or frequency characteristics mean that the duration of afterglow is long or the intensity of light in the high frequency band is weak, and superior afterglow characteristics or frequency characteristics mean afterglow. The duration of the light is short, or the light intensity in the high frequency band is strong.

Here, in the examples shown in FIGS. 114A to 115, the occurrence of a barcode reading error is suppressed by blunting the frequency of the red luminance change included in the visible light signal, but the carrier frequency of the visible light signal is reduced. By increasing the value, the occurrence of the read error may be suppressed.

FIG. 116 is a diagram for explaining a new problem that occurs in order to suppress the occurrence of a barcode reading error.

As shown in FIG. 116, when the carrier frequency fc of the visible light signal is about 10 kHz, the reading frequency of the red laser light used for reading the barcode is also about 10 to 20 kHz, so that the frequencies interfere with each other. Bar code reading error occurs.

Therefore, by increasing the carrier frequency fc of the visible light signal from about 10 kHz to, for example, 40 kHz, it is possible to suppress the occurrence of a barcode reading error.

However, if the carrier frequency fc of the visible light signal is about 40 kHz, the sampling frequency fs for the receiver to sample the visible light signal by photographing needs to be 80 kHz or more.

That is, since the sampling frequency fs required in the receiver is high, a new problem arises that the processing load on the receiver increases. Therefore, in order to solve this new problem, the receiver in the present embodiment performs downsampling.

FIG. 117 is a diagram for explaining downsampling performed by the receiver in the present embodiment.

The transmitter 2301 in this embodiment is configured as, for example, a liquid crystal display, digital signage, or a lighting device. Then, the transmitter 2301 outputs a frequency-modulated visible light signal. At this time, the transmitter 2301 switches the carrier frequency fc of the visible light signal between, for example, 40 kHz and 45 kHz.

The receiver 2302 in the present embodiment captures the transmitter 2301 at a frame rate of, for example, 30 fps. At this time, the receiver 2302 takes a picture with a short exposure time so that a bright line is generated in each image (specifically, each frame) obtained by the picture taking, as in the case of the receiver in each of the above-described embodiments. Further, the image sensor used for photographing the receiver 2302 has, for example, 1000 exposure lines. Therefore, in one-frame shooting, the visible light signal is sampled by starting exposure at different timings for each of the 1000 exposure lines. As a result, sampling (30 ks / sec) of 30 fps × 1000 lines = 30,000 times is performed in 1 second. In other words, the sampling frequency fs of the visible light signal is 30 kHz.

According to a general sampling theorem, at a sampling frequency fs = 30 kHz, only a visible light signal having a carrier frequency of 15 kHz or less can be demodulated.

However, the receiver 2302 in the present embodiment downsamples a visible light signal having a carrier frequency fc = 40 kHz or 45 kHz at a sampling frequency fs = 30 kHz. This downsampling causes an alias in the frame, and the receiver 2302 in the present embodiment estimates the carrier frequency fc of the visible light signal by observing and analyzing the alias.

FIG. 118 is a flowchart showing the processing operation of the receiver 2302 in the present embodiment.

First, the receiver 2302 performs downsampling of the sampling frequency fs = 30 kHz with respect to the visible light signal having the carrier frequency fc = 40 kHz or 45 kHz by photographing the subject (step S2310).

Next, the receiver 2302 observes and analyzes the alias generated in the frame obtained by the downsampling (step S2311). Thereby, the receiver 2302 specifies the frequency of the alias as, for example, 5.1 kHz or 5.5 kHz.

Then, the receiver 2302 estimates the carrier frequency fc of the visible light signal based on the frequency of the specified alias (step S2311). That is, the receiver 2302 restores the original frequency from the alias. As a result, the receiver 2302 estimates the carrier frequency fc of the visible light signal as, for example, 40 kHz or 45 kHz.

As described above, the receiver 2302 in the present embodiment can appropriately receive the visible light signal having a high carrier frequency by performing downsampling and restoring the frequency based on the alias. For example, the receiver 2302 can receive a visible light signal having a carrier frequency of 30 kHz to 60 kHz even if the sampling frequency is fs = 30 kHz. Therefore, the carrier frequency of the visible light signal can be increased from the currently practical frequency (about 10 kHz) to 30 kHz to 60 kHz. As a result, the carrier frequency of the visible light signal and the reading frequency of the barcode (10 to 20 kHz) can be greatly different from each other, and interference between the frequencies can be suppressed. As a result, it is possible to suppress the occurrence of barcode reading errors.

Such a receiving method in the present embodiment is a receiving method for acquiring information from a subject, and corresponds to a plurality of exposure lines included in the image sensor in a frame obtained by photographing the subject by the image sensor. The exposure time setting step for setting the exposure time of the image sensor and the plurality of exposure lines included in the image sensor are sequentially exposed at different times so that a plurality of emission lines are generated according to the change in the brightness of the subject. By repeating the process of starting, the image sensor takes a picture of the subject whose brightness changes at a predetermined frame rate and at a set exposure time, and for each frame obtained by the picture. , The information acquisition step of acquiring information by demodulating the data specified by the pattern of the plurality of emission lines included in the frame. Then, in the shooting step, the plurality of exposure lines are sequentially started to be exposed at different times, so that the sampling frequency is lower than the carrier frequency of the visible light signal transmitted by the change in the brightness of the subject. , The visible light signal is downsampled, and in the information acquisition step, the frequency of the alias specified by the pattern of the plurality of emission lines included in the frame is specified and specified for each frame obtained by the photographing. The information is acquired by estimating the frequency of the visible light signal from the frequency of the alias and demodulating the estimated frequency of the visible light signal.

In such a receiving method, a visible light signal having a high carrier frequency can be appropriately received by performing downsampling and frequency restoration based on the alias.

Further, in the downsampling, a visible light signal having a carrier frequency higher than 30 kHz may be downsampled. As a result, it is possible to avoid interference between the carrier frequency of the visible light signal and the reading frequency of the barcode (10 to 20 kHz), and it is possible to more effectively suppress the reading error of the barcode.

(Embodiment 15)
FIG. 119 is a diagram showing a processing operation of the receiving device (imaging device). Specifically, FIG. 119 is a diagram for explaining an example of a process of switching between a normal imaging mode and a macro imaging mode in the case of receiving visible light communication.

Here, the receiving device 1610 receives the visible light emitted by the transmitting device composed of a plurality of light sources (four light sources in FIG. 119).

First, when the receiving device 1610 transitions to the mode for performing visible light communication, the receiving device 1610 activates the imaging unit in the normal imaging mode (S1601). When the receiving device 1610 shifts to the mode for performing visible light communication, the receiving device 1610 displays the frame 1611 for capturing the light source on the screen.

After a predetermined time, the receiving device 1610 switches the imaging mode of the imaging unit to the macro imaging mode (S1602). The timing of switching from step S1601 to step S1602 may be determined not after a predetermined time from step S1601 but when it is determined that the receiving device 1610 has been imaged so that the light source fits within the frame 1611. By switching to the macro imaging mode in this way, the user can easily accommodate the light source in the frame 1611 with a clear image in the normal imaging mode before the image is blurred by the macro imaging mode. Can be stored in.

Next, the receiving device 1610 determines whether or not a signal from the light source has been received (S1603). If it is determined that the signal from the light source has been received (Yes in S1603), the normal imaging mode of step S1601 is returned, and if it is determined that the signal from the light source has not been received (No in S1603), the macro imaging in step 1602 is performed. Continue mode. If Yes in step S1603, processing based on the received signal (for example, processing for displaying an image shown in the received signal) may be performed.

According to the receiving device 1610, the user can switch from the normal imaging mode to the macro imaging mode by touching the display unit of the light source 1611 of the smartphone with a finger, so that a plurality of light sources can be imaged in a blurred state. Therefore, the image captured in the macro imaging mode includes more bright regions than the image captured in the normal imaging mode. In particular, since the light from the two light sources overlaps between two adjacent light sources among the plurality of light sources, the striped images are separated as shown in the left figure of FIG. 119 (a), so that they are continuous. The problem of not being able to receive as a signal can be demodulated as a continuous reception signal for forming a continuous stripe as shown in the right figure. Since a long code can be received at one time, there is an effect that the response time is shortened. As shown in FIG. 119 (b), when a photographed image is first photographed with a normal shutter and a normal focus, a beautiful normal image can be obtained. However, if the light source is far away like letters, even if the shutter speed is increased, continuous data cannot be obtained and demodulation cannot be performed. Next, when the shutter speed is increased and the focal drive unit of the lens is set to a short distance (macro), the light source is blurred and expanded, and the four light sources are connected, so that data can be received. Then refocus and return the shutter speed to normal to get the original beautiful image. As shown in (c), the display unit has the effect of recording and displaying a beautiful image in the memory so that only the beautiful image is displayed on the display unit. The image captured in the macro imaging mode contains more areas brighter than the predetermined brightness than the image captured in the normal imaging mode. Therefore, in the macro imaging mode, it is possible to increase the number of exposure lines capable of generating emission lines for the subject.

FIG. 120 is a diagram showing a processing operation of a receiving device (imaging device). Specifically, FIG. 120 is a diagram for explaining another example of the switching process between the normal imaging mode and the macro imaging mode in the case of receiving visible light communication.

Here, the receiving device 1620 receives the visible light emitted by the transmitting device composed of a plurality of light sources (four light sources in FIG. 120).

First, when the receiving device 1620 shifts to the mode for performing visible light communication, the receiving device 1620 activates the imaging unit in the normal imaging mode and captures an image 1623 in a wider range than the image 1622 displayed on the screen of the receiving device 1620. .. Then, the image data showing the captured image 1623 and the posture information indicating the posture of the receiving device 1620 detected by the gyro sensor, the geomagnetic sensor, and the acceleration sensor of the receiving device 1620 when the image 1623 is captured are stored in the memory. (S1611). The captured image 1623 is an image having a wide range by a predetermined width in the vertical direction and the horizontal direction with reference to the image 1622 displayed on the screen of the receiving device 1620. Further, when the receiving device 1620 shifts to the mode for performing visible light communication, the receiving device 1620 displays a frame 1621 for imaging the light source on the screen.

After a predetermined time, the receiving device 1620 switches the imaging mode of the imaging unit to the macro imaging mode (S1612). The timing of switching from step S1611 to step S1612 is not after a predetermined time from step S1611, but even when the image 1623 is imaged and it is determined that the image data indicating the captured image 1623 is held in the memory. good. At this time, the receiving device 1620 displays an image 1624 having a size corresponding to the screen size of the receiving device 1620 among the images 1623 based on the image data held in the memory.

The image 1624 displayed on the receiving device 1620 at this time is a part of the image of the image 1623, and the posture of the receiving device 1620 (indicated by a white dashed line) indicated by the posture information acquired in step S1611. It is an image of a region predicted to be captured by the current receiving device 1620 from the difference between the position) and the posture of the current receiving device 1620. That is, the image 1624 is an image of a part of the image 1623, and is an image of a region corresponding to the image pickup target of the image 1625 actually captured in the macro imaging mode. That is, in step S1612, the posture (imaging direction) changed from the time of step S1611 is acquired, and the imaging target presumed to be currently imaged from the acquired current posture (imaging direction) is specified and imaged in advance. An image 1624 corresponding to the current posture (imaging direction) is specified from the image 1623, and a process of displaying the image 1624 is performed. Therefore, as shown in the image 1623 of FIG. 120, the receiving device 1620 is an image cut out from the image 1623 according to the movement amount when the receiving device 1620 moves in the direction of the white arrow from the position indicated by the white broken line. A region of 1624 can be determined and an image 1624, which is an image 1623 in the determined region, can be displayed.

As a result, the receiving device 1620 does not display the image 1625 captured in the macro imaging mode even when the image is captured in the macro imaging mode, and the image 1623 captured in the clearer normal imaging mode is used. , The image 1624 cut out according to the posture of the current receiving device 1620 can be displayed. In the method of the present invention in which continuous visible light information is obtained from a plurality of light sources that are far from the defocused image and at the same time the stored normal plane image is displayed on the display unit, the user takes a picture using a smartphone. At that time, it is expected that camera shake will occur and the direction of the actual captured image and the still image displayed from the memory will be deviated, and the user will not be able to adjust the direction to the target light source. In this case, data from the light source cannot be received, so countermeasures are required. However, according to the improved invention, even if the camera shakes, the camera shake is detected by the image shake detecting means or the shaking detecting means of the vibration gyro, and the target image in the still image is shifted in a predetermined direction to the camera. The user can see the deviation from the direction of. This display allows the user to point the camera at the target light source, so multiple divided light sources can be optically connected and photographed while displaying a normal image, and signals are continuously received. can do. As a result, since a normal image is displayed, it is possible to receive a light source divided into a plurality of parts. In this case, the posture of the receiving device 1620 can be easily adjusted so that the plurality of light sources fit the frame 1621. When the focus is blurred, the light source is dispersed and the brightness is equivalently reduced. Therefore, by increasing the sensitivity of the ISO of the camera or the like, there is an effect that visible light data can be received more reliably.

Next, the receiving device 1620 determines whether or not a signal from the light source has been received (S1613). If it is determined that the signal from the light source has been received (Yes in S1613), the normal imaging mode of step S1611 is returned, and if it is determined that the signal from the light source has not been received (No in S1613), the macro imaging in step 1612 is performed. Continue mode. If Yes in step S1613, processing based on the received signal (for example, processing for displaying an image shown in the received signal) may be performed.

Similar to the receiving device 1610, the receiving device 1620 can also capture an image including a brighter region in the macro imaging mode. Therefore, in the macro imaging mode, it is possible to increase the number of exposure lines capable of generating emission lines for the subject.

FIG. 121 is a diagram showing a processing operation of a receiving device (imaging device).

Here, the transmission device 1630 is, for example, a display device such as a television, and transmits different transmission IDs by visible light communication at a predetermined time interval Δ1630. Specifically, at times t1631, t1632, t1633, and t1634, ID1631, ID1632, ID1633, and ID1634, which are transmission IDs associated with the data corresponding to the displayed images 1631, 1632, 1633, and 1634, respectively, are transmitted. do. That is, ID1631 to ID1634 are transmitted one after another from the transmission device 1630 at a predetermined time interval Δt1630.

The receiving device 1640 requests the data associated with each transmission ID from the server 1650 based on the transmission ID received by the visible light communication, receives the data from the server, and displays the image corresponding to the data. Specifically, the images 1641, 1642, 1644 and 1644 corresponding to ID1631, ID1632, ID1633 and ID1634 are displayed at the times t1631, t1632, t1633 and t1634, respectively.

When the receiving device 1640 acquires the ID 1631 received at the time t1631, the receiving device 1640 may acquire the ID information indicating the transmission ID scheduled to be transmitted from the transmitting device 1630 at the subsequent times t1632 to t1634 from the server 1650. In this case, the receiving device 1640 uses the acquired ID information to transmit the data associated with the IDs 1632 to ID1634 at the times t1632 to t1634 to the server 1650 without receiving the transmission ID from the transmitting device 1630 each time. The received data can be displayed at each time t1632 to t1634.

Further, if the receiving device 1640 requests the data corresponding to the ID 1631 at the time t1631 without acquiring the information indicating the transmission ID scheduled to be transmitted from the transmitting device 1630 from the server 1650 at the subsequent times t1632 to t1634. , The data associated with the transmission ID corresponding to the subsequent times t1632 to t1634 may be received from the server 1650, and the received data may be displayed at each time t1632 to t1634. That is, when the server 1650 receives the request for the data associated with the ID 1631 transmitted from the receiving device 1640 to the time t1631, the server 1650 receives the data associated with the transmission ID corresponding to the subsequent times t1632 to t1634. Even if there is no request from 1640, it is transmitted to the receiving device 1640 at each time t1632 to t1634. That is, in this case, the server 1650 holds the association information associated with each time t1631 to 1634 and the data associated with the transmission ID corresponding to each time t1631 to 1634, and is based on the association information. The predetermined data associated with the predetermined time is transmitted at the predetermined time.

As described above, if the receiving device 1640 can acquire the transmission ID 1631 by visible light communication at time t1631, the data corresponding to each time t1632 to t1634 is performed from the server 1650 at the subsequent times t1632 to t1634 without performing visible light communication. Can be received. Therefore, the user does not need to keep pointing the receiving device 1640 to the transmitting device 1630 in order to acquire the transmission ID by visible light communication, and the receiving device 1640 can easily display the data acquired from the server 1650. In this case, when the receiving device 1640 acquires the data corresponding to the ID from the server each time, a time delay from the server occurs and the response time becomes long. Therefore, in order to speed up the response, the data corresponding to the ID is stored in advance from the server or the like in the storage unit of the receiver, and the data corresponding to the ID in the storage unit is displayed to display the response time. Can be done quickly. In this method, if the time information for outputting the next ID is included in the transmission signal from the visible light transmitter, the receiver side can be at that time even if the visible light signal is not continuously received. For this reason, since the transmission time of the next ID can be known, there is an effect that it is not necessary to point the receiving device toward the light source all the time. This method only synchronizes the time information (clock) on the transmitter side with the time information (clock) on the receiver side when visible light is received, and does not receive the data from the transmitter after synchronization. However, it has the effect of being able to continuously display the screen synchronized with the transmitter.

Further, in the above example, the receiving device 1640 displays the images 1641, 1642, 1643, 1644 corresponding to the transmission IDs ID1631, ID1632, ID1633 and ID1634, respectively, at the times t1631, t1632, t1633, and t1634, respectively. Each was displayed. Here, as shown in FIG. 122, the receiving device 1640 may present not only the image but also other information at each of the above times. That is, the receiving device 1640 displays the image 1641 corresponding to the ID 1631 at the time t1631 and outputs the sound or the voice corresponding to the ID 1631. At this time, the receiving device 1640 may further display, for example, a purchase site of the product displayed in the image. Such sound output and display of the purchase site are similarly performed at each of the times t1632, t1633, and t1634 other than the time t1631.

Next, in the case of a smartphone equipped with two cameras for stereoscopic use, as shown in FIG. 119 (b), an image with a normal shutter speed for the left eye and a normal image quality at a normal focus is displayed. At the same time, in the camera for the right eye, the shutter is faster than the left eye, and / or the focal point or macro is set at a short distance to obtain the striped emission line of the present invention, and the data is demodulated. As a result, an image of normal image quality is displayed on the display unit, and the right-eye camera can receive optical communication data of a plurality of light sources divided in distance.

(Embodiment 16)
Here, an application example of voice synchronous reproduction will be described below.

FIG. 123 is a diagram showing an example of the application according to the sixteenth embodiment.

For example, the receiver 1800a configured as a smartphone receives a signal (visible light signal) transmitted from the transmitter 1800b configured as a street digital signage, for example. That is, the receiver 1800a receives the timing of image reproduction by the transmitter 1800b. The receiver 1800a reproduces the sound at the same timing as the image reproduction. In other words, the receiver 1800a performs synchronous reproduction of the sound so that the image and sound reproduced by the transmitter 1800b are synchronized. The receiver 1800a may reproduce the same image as the image (reproduced image) reproduced by the transmitter 1800b, or a related image related to the reproduced image, together with the sound. Further, the receiver 1800a may cause a device connected to the receiver 1800a to reproduce audio or the like. Further, after receiving the visible light signal, the receiver 1800a may download the content such as the voice or the related image associated with the visible light signal from the server. The receiver 1800a performs synchronous reproduction after the download.

As a result, even if the sound from the transmitter 1800b cannot be heard or the sound from the transmitter 1800b is not played because the street sound reproduction is prohibited, the user can adjust the sound to the display of the transmitter 1800b. Can be heard. In addition, even if there is a distance that it takes time to reach the voice, the voice can be heard according to the display.

Here, multilingual support by voice synchronous reproduction will be described below.

FIG. 124 is a diagram showing an example of the application according to the sixteenth embodiment.

Each of the receiver 1800a and the receiver 1800c acquires the sound of the language set in the receiver and corresponds to the image of a movie or the like displayed on the transmitter 1800d from the server and reproduces it. do. Specifically, the transmitter 1800d transmits a visible light signal indicating an ID for identifying the displayed image to the receiver. When the receiver receives the visible light signal, it transmits a request signal including the ID shown in the visible light signal and the language set in itself to the server. The receiver acquires the voice corresponding to the request signal from the server and reproduces it. As a result, the user can enjoy the work displayed on the transmitter 1800d in the language set by the user.

Here, the voice synchronization method will be described below.

125 and 126 are diagrams showing an example of a transmission signal and an example of a voice synchronization method according to the sixteenth embodiment.

The different data (for example, the data shown in FIG. 125: 1 to 6, etc.) are associated with the time at regular time intervals (N seconds). These data may be, for example, an ID for identifying time, time, or voice data (for example, 64 kbps data). Hereinafter, the description will be made on the assumption that the data is an ID. The different IDs may have different additional information portions attached to the IDs.

It is desirable that the packets constituting the ID are different. Therefore, it is desirable that the IDs are not continuous. Alternatively, when packetizing the ID, a packetization method in which the discontinuous portion is configured as one packet is desirable. Since the error correction signal tends to have a different pattern even if it is a continuous ID, the error correction signal may be distributed in a plurality of packets instead of being combined in one packet.

The transmitter 1800d transmits an ID according to, for example, the reproduction time of the displayed image. The receiver can recognize the reproduction time (synchronization time) of the image of the transmitter 1800d by detecting the timing when the ID is changed.

In the case of (a), since the change time points of ID: 1 and ID: 2 are received, the synchronization time can be accurately recognized.

When the time N in which the ID is transmitted is long, such an opportunity is small, and the ID may be received as in (b). Even in this case, the synchronization time can be recognized by the following method.

(B1) The midpoint of the reception section where the ID has changed is assumed to be the ID change point. Further, the time after an integral multiple of the time N from the ID change point estimated in the past is also estimated as the ID change point, and the midpoint of the plurality of ID change points is estimated as the more accurate ID change point. With such an estimation algorithm, it is possible to gradually estimate an accurate ID change point.

(B2) In addition to the above, the reception section in which the ID does not change and the time after an integral multiple of the time N may be gradually the ID change point by presuming that the ID change point is not included. The number of sections with is reduced, and an accurate ID change point can be estimated.

Accurate synchronization can be achieved by setting N to 0.5 seconds or less.

By setting N to 2 seconds or less, synchronization can be performed without causing the user to feel a delay.

By setting N to 10 seconds or less, it is possible to suppress the waste of ID and synchronize.

FIG. 126 is a diagram showing an example of a transmission signal according to the 16th embodiment.

In FIG. 126, ID waste can be avoided by performing synchronization by time packets. A time packet is a packet that holds the time of transmission. When it is necessary to express a long time, a time packet is formed by dividing it into a time packet 1 representing a fine time and a time packet 2 representing a coarse time. For example, the time packet 2 indicates the hours and minutes of the time, and the time packet 1 indicates only the seconds of the time. The packet indicating the time may be divided into three or more time packets. Since coarse time is less necessary, the receiver can recognize the synchronization time quickly and accurately by sending more fine time packets than coarse time packets.

That is, in the present embodiment, the visible light signal includes the second information (time packet 2) indicating the hour and minute of the time and the first information (time packet 1) indicating the second of the time. By including, the time when the visible light signal is transmitted from the transmitter 1800d is indicated. Then, the receiver 1800a receives the second information and receives the first information more times than the number of times the second information is received.

Here, the synchronization time adjustment will be described below.

FIG. 127 is a diagram showing an example of the processing flow of the receiver 1800a according to the 16th embodiment.

After the signal is transmitted, it is processed by the receiver 1800a, and it takes some time for the audio or video to be played. Therefore, by performing the processing to play the audio or video in anticipation of this processing time, it will be accurate. Synchronous playback can be performed.

First, the processing delay time is designated for the receiver 1800a (step S1801). This may be retained in the processing program or may be specified by the user. By making corrections by the user, more accurate synchronization can be realized according to the individual receiver. By changing this processing delay time for each receiver model, the temperature of the receiver, and the CPU usage ratio, synchronization can be performed more accurately.

The receiver 1800a determines whether or not a time packet has been received, or whether or not an ID associated with voice synchronization has been received (step S1802). Here, when the receiver 1800a determines that the image has been received (Y in step S1802), it further determines whether or not there is an image waiting to be processed (step S1804). When it is determined that there is a processing-waiting image (Y in step S1804), the receiver 1800a discards the processing-waiting image or postpones the processing of the processing-waiting image to receive processing from the latest acquired image. (Step S1805). As a result, it is possible to avoid an unexpected delay due to the amount of waiting for processing.

The receiver 1800a measures the position of the visible light signal (specifically, the emission line) in the image (step S1806). That is, by measuring at which position the signal appears in the direction perpendicular to the exposure line from the first exposure line in the image sensor, the time difference (delay time in the image) from the image acquisition start time to the signal reception time can be obtained. Can be calculated.

The receiver 1800a can accurately perform synchronous reproduction by reproducing the audio or moving image at the time obtained by adding the processing delay time and the in-image delay time to the recognized synchronous time (step S1807).

On the other hand, in step S1802, if it is determined that the receiver 1800a has not received the time packet or the voice synchronization ID, the receiver 1800a receives a signal from the image obtained by the imaging (step S1803).

FIG. 128 is a diagram showing an example of the user interface of the receiver 1800a according to the sixteenth embodiment.

As shown in FIG. 128 (a), the user can adjust the above-mentioned processing delay time by pressing any of the buttons Bt1 to Bt4 displayed on the receiver 1800a. Further, it may be possible to set the processing delay time by swiping as shown in FIG. 128 (b). As a result, synchronous reproduction can be performed more accurately based on the user's feeling.

Here, the earphone limited reproduction will be described below.

FIG. 129 is a diagram showing an example of the processing flow of the receiver 1800a according to the 16th embodiment.

By the earphone limited reproduction shown by this processing flow, it is possible to perform audio reproduction without disturbing the surroundings.

The receiver 1800a confirms whether or not the earphone-only setting is made (step S1811). When the earphone-only setting is made, for example, the receiver 1800a is set to the earphone-only setting. Alternatively, the received signal (visible light signal) is set to be limited to earphones. Alternatively, the earphone limitation is recorded in the server or receiver 1800a in association with the received signal.

When the receiver 1800a confirms that the earphones are limited (Y in step S1811), it determines whether or not the earphones are connected to the receiver 1800a (step S1813).

When the receiver 1800a confirms that the earphones are not limited (N in step S1811) or determines that the earphones are connected (Y in step S1813), the receiver 1800a reproduces the sound (step S1812). When reproducing the sound, the receiver 1800a adjusts the volume so that the volume is within the set range. This setting range is set in the same manner as the earphone-only setting.

When the receiver 1800a determines that the earphones are not connected (N in step S1813), the receiver 1800a notifies the user to connect the earphones (step S1814). This notification is made, for example, by screen display, audio output or vibration.

Further, when the receiver 1800a is not set to prohibit the forced playback, the receiver 1800a prepares an interface for the forced playback and determines whether or not the user has performed the forced playback operation (). Step S1815). Here, if it is determined that the forced reproduction operation has been performed (Y in step S1815), the receiver 1800a reproduces the sound even when the earphone is not connected (step S1812).

On the other hand, when it is determined that the forced playback operation is not performed (N in step S1815), the receiver 1800a holds the audio data received in advance and the analyzed synchronization time, so that when the earphones are connected, the earphones are connected. Synchronized playback of audio is performed promptly.

FIG. 130 is a diagram showing another example of the processing flow of the receiver 1800a in the 16th embodiment.

The receiver 1800a first receives an ID from the transmitter 1800d (step S1821). That is, the receiver 1800a receives a visible light signal indicating the ID of the transmitter 1800d or the ID of the content displayed on the transmitter 1800d.

Next, the receiver 1800a downloads the information (content) associated with the received ID from the server (step S1822). Alternatively, the receiver 1800a reads the information from the data holding unit inside the receiver 1800a. Hereinafter, this information is referred to as related information.

Next, the receiver 1800a determines whether or not the synchronous reproduction flag included in the related information indicates ON (step S1823). Here, if it is determined that the synchronous reproduction flag does not indicate ON (N in step S1823), the receiver 1800a outputs the content indicated by the related information (step S1824). That is, when the content is an image, the receiver 1800a displays the image, and when the content is audio, the receiver 1800a outputs audio.

On the other hand, when the receiver 1800a determines that the synchronous reproduction flag is ON (Y in step S1823), is the time adjustment mode included in the related information set to the transmitter reference mode? , It is determined whether the absolute time mode is set (step S1825). If it is determined that the absolute time mode is set, the receiver 1800a determines whether or not the last time adjustment has been performed within a certain time from the current time (step S1826). The time adjustment at this time is a process of obtaining time information by a predetermined method and using the time information to adjust the time of the clock provided in the receiver 1800a to the absolute time of the reference clock. The predetermined method is, for example, a method using a GPS (Global Positioning System) radio wave or an NTP (Network Time Protocol) radio wave. The above-mentioned current time may be the time when the receiver 1800a, which is a terminal device, receives the visible light signal.

When the receiver 1800a determines that the last time adjustment has been performed within a certain time (Y in step S1826), the receiver 1800a is displayed on the transmitter 1800d by outputting related information based on the clock time of the receiver 1800a. Content and related information are synchronized (step S1827). When the content indicated by the related information is, for example, a moving image, the receiver 1800a displays the moving image so as to be synchronized with the content displayed on the transmitter 1800d. When the content indicated by the related information is, for example, voice, the receiver 1800a outputs the voice so as to synchronize with the content displayed on the transmitter 1800d. For example, when the related information indicates a voice, the related information includes each frame constituting the voice, and those frames are time-stamped. The receiver 1800a outputs the sound synchronized with the contents of the transmitter 1800d by reproducing the frame with the type stamp corresponding to the time of its own clock.

When the receiver 1800a determines that the final time adjustment has not been performed within a certain time (N in step S1826), the receiver 1800a attempts to obtain the time information by a predetermined method, and whether or not the time information can be obtained. (Step S1828). Here, if it is determined that the time information can be obtained (Y in step S1828), the receiver 1800a updates the clock time of the receiver 1800a using the time information (step S1829). Then, the receiver 1800a executes the process of step S1827 described above.

Further, when it is determined in step S1825 that the time adjustment mode is the transmitter reference mode, or when it is determined in step S1828 that the time information could not be obtained (N in step S1828), the receiver 1800a , Acquire time information from the transmitter 1800d (step S1830). That is, the receiver 1800a acquires the time information, which is a synchronization signal, from the transmitter 1800d by visible light communication. For example, the synchronization signals are the time packet 1 and the time packet 2 shown in FIG. 126. Alternatively, the receiver 1800a acquires time information from the transmitter 1800d by radio waves such as Bluetooth (registered trademark) or Wi-Fi. Then, the receiver 1800a executes the processes of steps S1829 and S1827 described above.

In the present embodiment, as in steps S1829 and S1830, processing (time adjustment) for synchronizing the clock of the terminal device, which is the receiver 1800a, with the reference clock is performed by GPS radio waves or NTP radio waves. When the time is earlier than the predetermined time from the time when the terminal device receives the visible light signal, the clock of the terminal device and the clock of the transmitter are set according to the time indicated by the visible light signal transmitted from the transmitter 1800d. Synchronize between. As a result, the terminal device can reproduce the content (video or audio) at the timing synchronized with the transmitter-side content reproduced by the transmitter 1800d.

FIG. 131A is a diagram for explaining a specific method of synchronous reproduction in the sixteenth embodiment. Synchronous reproduction methods include methods a to e shown in FIG. 131A.

(Method a)
In the method a, the transmitter 1800d outputs a visible light signal indicating a content ID and a content reproduction time by changing the brightness of the display, as in each of the above embodiments. The content playback time is the playback time of data that is a part of the content that is being played back by the transmitter 1800d when the content ID is transmitted from the transmitter 1800d. If the content is a moving image, the data is a picture or sequence that constitutes the moving image, and if the content is audio, it is a frame that constitutes the audio. The reproduction time indicates, for example, the reproduction time from the beginning of the content as a time. If the content is a moving image, the reproduction time is included in the content as a PTS (Presentation Time Stamp). That is, the content includes the reproduction time (display time) of the data for each data constituting the content.

The receiver 1800a receives the visible light signal by photographing the transmitter 1800d in the same manner as in each of the above embodiments. Then, the receiver 1800a transmits a request signal including the content ID indicated by the visible light signal to the server 1800f. The server 1800f receives the request signal and transmits the content associated with the content ID included in the request signal to the receiver 1800a.

When the receiver 1800a receives the content, the receiver 1800a reproduces the content from the time point (time during content reproduction + elapsed time from ID reception). The elapsed time from ID reception is the elapsed time from the time when the content ID is received by the receiver 1800a.

(Method b)
In the method b, the transmitter 1800d outputs a visible light signal indicating the content ID and the content reproduction time by changing the brightness of the display, as in each of the above-described embodiments. The receiver 1800a receives the visible light signal by photographing the transmitter 1800d in the same manner as in each of the above embodiments. Then, the receiver 1800a transmits a request signal including the content ID indicated by the visible light signal and the content reproduction time to the server 1800f. The server 1800f receives the request signal, and among the contents associated with the content ID included in the request signal, only a part of the content after the content reproduction time is transmitted to the receiver 1800a.

When the receiver 1800a receives a part of the content, the receiver 1800a reproduces the part of the content from the time point (elapsed time from ID reception).

(Method c)
In the method c, the transmitter 1800d outputs a visible light signal indicating a transmitter ID and a content reproduction time by changing the brightness of the display, as in each of the above embodiments. The transmitter ID is information for identifying the transmitter.

The receiver 1800a receives the visible light signal by photographing the transmitter 1800d in the same manner as in each of the above embodiments. Then, the receiver 1800a transmits a request signal including the transmitter ID indicated by the visible light signal to the server 1800f.

The server 1800f holds a playback schedule, which is a timetable of the content played by the transmitter of the transmitter ID, for each transmitter ID. Further, the server 1800f is equipped with a clock. When such a server 1800f receives the request signal, the content associated with the transmitter ID included in the request signal and the clock time (server time) of the server 1800f is used as the content being played. , Identify from the playback schedule. Then, the server 1800f transmits the content to the receiver 1800a.

When the receiver 1800a receives the content, the receiver 1800a reproduces the content from the time point (time during content reproduction + elapsed time from ID reception).

(Method d)
In the method d, the transmitter 1800d outputs a visible light signal indicating a transmitter ID and a transmitter time by changing the brightness of the display, as in each of the above embodiments. The transmitter time is the time indicated by the clock provided in the transmitter 1800d.

The receiver 1800a receives the visible light signal by photographing the transmitter 1800d in the same manner as in each of the above embodiments. Then, the receiver 1800a transmits a request signal including the transmitter ID and the transmitter time indicated by the visible light signal to the server 1800f.

The server 1800f holds the above-mentioned reproduction schedule. When such a server 1800f receives the request signal, the content associated with the transmitter ID and the transmitter time included in the request signal is specified from the reproduction schedule as the content being reproduced. Further, the server 1800f specifies the content reproduction time from the transmitter time. That is, the server 1800f finds the reproduction start time of the specified content from the reproduction schedule, and specifies the time between the transmitter time and the reproduction start time as the content reproduction time. Then, the server 1800f transmits the content and the content reproduction time to the receiver 1800a.

When the receiver 1800a receives the content and the content playback time, the receiver 1800a reproduces the content from the time point (content playback time + elapsed time from ID reception).

As described above, in the present embodiment, the visible light signal indicates the time when the visible light signal is transmitted from the transmitter 1800d. Therefore, the receiver 1800a, which is a terminal device, can receive the content associated with the time (transmitter time) at which the visible light signal is transmitted from the transmitter 1800d. For example, if the transmitter time is 5:43, the content to be played at 5:43 can be received.

Further, in the present embodiment, the server 1800f has a plurality of contents associated with each time. However, the content associated with the time indicated by the visible light signal may not exist in the server 1800f. In such a case, the receiver 1800a, which is a terminal device, is associated with the time closest to the time indicated by the visible light signal and after the time indicated by the visible light signal among the plurality of contents. You may receive the content. As a result, even if the content associated with the time indicated by the visible light signal does not exist in the server 1800f, the appropriate content can be received from the plurality of contents in the server 1800f.

Further, the reproduction method in the present embodiment includes a signal receiving step of receiving a visible light signal from a transmitter 1800d that transmits a visible light signal due to a change in brightness of a light source by a sensor of a receiver 1800a (terminal device), and a receiver. A transmission step of transmitting a request signal for requesting the content associated with the visible light signal from the 1800a to the server 1800f, a content receiving step in which the receiver 1800a receives the content from the server 1800f, and the content reproduction. Includes playback steps. The visible light signal indicates a transmitter ID and a transmitter time. The transmitter ID is ID information. Further, the transmitter time is a time indicated by the clock of the transmitter 1800d, and is a time when the visible light signal is transmitted from the transmitter 1800d. Then, in the content receiving step, the receiver 1800a receives the content associated with the transmitter ID and the transmitter time indicated by the visible light signal. As a result, the receiver 1800a can reproduce the content appropriate for the transmitter ID and the transmitter time.

(Method e)
In the method e, the transmitter 1800d outputs a visible light signal indicating the transmitter ID by changing the brightness of the display, as in each of the above-described embodiments.

The receiver 1800a receives the visible light signal by photographing the transmitter 1800d in the same manner as in each of the above embodiments. Then, the receiver 1800a transmits a request signal including the transmitter ID indicated by the visible light signal to the server 1800f.

The server 1800f holds the above-mentioned reproduction schedule and further includes a clock. When such a server 1800f receives the request signal, the content associated with the transmitter ID and the server time included in the request signal is specified from the reproduction schedule as the content being reproduced. The server time is the time indicated by the clock of the server 1800f. Further, the server 1800f also finds the reproduction start time of the specified content from the reproduction schedule. Then, the server 1800f transmits the content and the content reproduction start time to the receiver 1800a.

When the receiver 1800a receives the content and the content reproduction start time, the receiver 1800a reproduces the content from the time point of (receiver time-content reproduction start time). The receiver time is the time indicated by the clock provided in the receiver 1800a.

As described above, the reproduction method in the present embodiment includes a signal receiving step of receiving a visible light signal from a transmitter 1800d that transmits a visible light signal due to a change in the brightness of the light source by a sensor of the receiver 1800a (terminal device). The receiver 1800a includes a transmission step of transmitting a request signal for requesting the content associated with the visible light signal to the server 1800f, and the receiver 1800a includes each time and data reproduced at each time. The content receiving step of receiving the content from the server 1800f and the reproduction step of reproducing the data corresponding to the time of the clock provided in the receiver 1800a among the contents are included. Therefore, the receiver 1800a can appropriately reproduce the data in the content at the correct time shown in the content without reproducing the data at the wrong time. Further, even in the transmitter 1800d, if the content related to the content (transmitter side content) is reproduced, the receiver 1800a can appropriately synchronize the content with the transmitter side content and reproduce the content. ..

Even in the above methods c to e, the server 1800f may transmit only a part of the contents after the content reproduction time to the receiver 1800a as in the method b.

Further, in the above methods a to e, the receiver 1800a transmits a request signal to the server 1800f and receives necessary data from the server 1800f, but the data in the server 1800f is previously transmitted without such transmission / reception. You may keep it.

FIG. 131B is a block diagram showing a configuration of a reproduction device that performs synchronous reproduction by the above-mentioned method e.

The reproduction device B10 is a receiver 1800a or a terminal device that performs synchronous reproduction by the above-mentioned method e, and includes a sensor B11, a request signal transmission unit B12, a content reception unit B13, a clock B14, and a reproduction unit B15. I have.

The sensor B11 is, for example, an image sensor, and receives the visible light signal from the transmitter 1800d that transmits the visible light signal due to the change in the brightness of the light source. The request signal transmission unit B12 transmits a request signal for requesting the content associated with the visible light signal to the server 1800f. The content receiving unit B13 receives content including each time and data reproduced at each time from the server 1800f. The reproduction unit B15 reproduces the data corresponding to the time of the clock B14 among the contents.

FIG. 131C is a flowchart showing the processing operation of the terminal device that performs synchronous reproduction by the above-mentioned method e.

The reproduction device B10 is a receiver 1800a or a terminal device that performs synchronous reproduction by the above-mentioned method e, and executes each process of steps SB11 to SB15.

In step SB11, the visible light signal is received from the transmitter 1800d that transmits the visible light signal due to the change in the brightness of the light source. In step SB12, a request signal for requesting the content associated with the visible light signal is transmitted to the server 1800f. In step SB13, the content including each time and the data to be reproduced at each time is received from the server 1800f. In step SB15, among the contents, the data corresponding to the time of the clock B14 is reproduced.

As described above, in the reproduction device B10 and the reproduction method in the present embodiment, the data in the content can be appropriately reproduced at the correct time shown in the content without being reproduced at the wrong time.

In the present embodiment, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software that realizes the reproduction device B10 and the like of the present embodiment is a program that causes a computer to execute each step included in the flowchart shown in FIG. 131C.

FIG. 132 is a diagram for explaining advance preparation for synchronous reproduction in the sixteenth embodiment.

The receiver 1800a adjusts the time of the clock provided in the receiver 1800a to the time of the reference clock in order to perform synchronous reproduction. For this time adjustment, the receiver 1800a performs the following processes (1) to (5).

(1) The receiver 1800a receives the signal. This signal may be a visible light signal transmitted by a change in the luminance of the display of the transmitter 1800d, or may be a radio signal based on Wi-Fi or Bluetooth® from a wireless device. Alternatively, instead of receiving such a signal, the receiver 1800a acquires position information indicating the position of the receiver 1800a by, for example, GPS. Then, the receiver 1800a recognizes that the receiver 1800a has entered a predetermined place or building based on the position information.

(2) When the receiver 1800a receives the above signal or recognizes that it has entered a predetermined place, the receiver 1800a sends a request signal requesting data (related information) associated with the signal or place. It is transmitted to the server (visible light ID resolution server) 1800f.

(3) The server 1800f transmits the above-mentioned data and the time adjustment request for causing the receiver 1800a to adjust the time to the receiver 1800a.

(4) When the receiver 1800a receives the data and the time adjustment request, the receiver 1800a transmits the time adjustment request to the GPS time server, the NTP server, or the base station of the telecommunications carrier (carrier).

(5) When the server or the base station receives the time adjustment request, it transmits time data (time information) indicating the current time (reference clock time or absolute time) to the receiver 1800a. The receiver 1800a adjusts the time by adjusting the time of the clock provided in the receiver to the current time shown in the time data.

As described above, in the present embodiment, a GPS (Global Positioning System) radio wave or an NTP (Network Time Protocol) radio wave is used between the clock provided in the receiver 1800a (terminal device) and the reference clock. It is synchronized. Therefore, the receiver 1800a can reproduce the data corresponding to the time at an appropriate time according to the reference clock.

FIG. 133 is a diagram showing an application example of the receiver 1800a according to the sixteenth embodiment.

The receiver 1800a is configured as a smartphone as described above, and is held and used in a holder 1810 made of, for example, a member such as a translucent resin or glass. The holder 1810 has a back plate portion 1810a and a locking portion 1810b erected on the back plate portion 1810a. The receiver 1800a is inserted between the back plate portion 1810a and the locking portion 1810b so as to be along the back plate portion 1810a.

FIG. 134A is a front view of the receiver 1800a held in the holder 1810 in the sixteenth embodiment.

The receiver 1800a is held in the holder 1810 in the inserted state as described above. At this time, the locking portion 1810b is locked with the lower portion of the receiver 1800a, and the lower portion thereof is sandwiched between the back plate portion 1810a. Further, the back surface of the receiver 1800a faces the back plate portion 1810a, and the display 1801 of the receiver 1800a is exposed.

FIG. 134B is a rear view of the receiver 1800a held in the holder 1810 in the sixteenth embodiment.

Further, a through hole 1811 is formed in the back plate portion 1810a, and a variable filter 1812 is attached near the through hole 1811. When the receiver 1800a is held by the holder 1810, the camera 1802 of the receiver 1800a is exposed from the back plate portion 1810a through the through hole 1811. Further, the flashlight 1803 of the receiver 1800a faces the variable filter 1812.

The variable filter 1812 is formed, for example, in a disk shape and has three color filters (red filter, yellow filter, and green filter), each of which is fan-shaped and has the same size. Further, the variable filter 1812 is rotatably attached to the back plate portion 1810a about the center of the variable filter 1812. Further, the red filter is a filter having red translucency, the yellow filter is a filter having yellow translucency, and the green filter is a filter having green translucency.

Therefore, the variable filter 1812 is rotated so that, for example, the red filter is placed at a position facing the flashlight 1803a. In this case, the light emitted from the flashlight 1803a is diffused inside the holder 1810 as red light by passing through the red filter. As a result, substantially the entire holder 1810 emits red light.

Similarly, the variable filter 1812 is rotated so that, for example, the yellow filter is placed at a position facing the flashlight 1803a. In this case, the light emitted from the flashlight 1803a is diffused inside the holder 1810 as yellow light by passing through the yellow filter. As a result, substantially the entire holder 1810 emits yellow light.

Similarly, the variable filter 1812 is rotated so that, for example, the green filter is placed at a position facing the flashlight 1803a. In this case, the light emitted from the flashlight 1803a is diffused inside the holder 1810 as green light by passing through the green filter. As a result, substantially the entire holder 1810 emits green light.

That is, the holder 1810 lights up in red, yellow, or green, like a penlight.

FIG. 135 is a diagram for explaining a use case of the receiver 1800a held in the holder 1810 in the 16th embodiment.

For example, a receiver with a holder, which is a receiver 1800a held in the holder 1810, is used in an amusement park or the like. That is, the plurality of receivers with holders directed to the moving float in the amusement park blink in synchronization with the music played from the float. That is, the float is configured as a transmitter in each of the above embodiments, and transmits a visible light signal by changing the brightness of a light source attached to the float. For example, the float transmits a visible light signal indicating the ID of the float. Then, the receiver with a holder receives the visible light signal, that is, the ID by taking a picture of the camera 1802 of the receiver 1800a, as in each of the above-described embodiments. The receiver 1800a that has received the ID acquires the program associated with the ID from, for example, the server. This program consists of an instruction to turn on the flashlight 1803 of the receiver 1800a at each predetermined time. Each of these predetermined times is set to match (synchronize with) the music played from the float. Then, the receiver 1800a blinks the flashlight 1803a according to the program.

As a result, the holder 1810 of each receiver 1800a that has received the ID repeatedly lights up at the same timing according to the music played from the float of the ID.

Here, each receiver 1800a blinks the flashlight 1803 according to the set color filter (hereinafter referred to as a setting filter). The setting filter is a color filter facing the flashlight 1803 of the receiver 1800a. Further, each receiver 1800a recognizes the current setting filter based on the operation by the user. Alternatively, each receiver 1800a recognizes the current setting filter based on the color of the image obtained by the shooting of the camera 1802 and the like.

That is, among the plurality of receivers 1800a that have received the ID, only the holders 1810 of the plurality of receivers 1800a that recognize that the setting filter is the red filter are lit at the same time at a predetermined time. At the next time, only the holders 1810 of the plurality of receivers 1800a that recognize that the setting filter is the green filter are lit at the same time. Further, at the next time, only the holders 1810 of the plurality of receivers 1800a that recognize that the setting filter is the yellow filter are turned on at the same time.

In this way, the receiver 1800a held in the holder 1810 synchronizes with the music of the float and the receiver 1800a held in the other holder 1810, as in the synchronous reproduction shown in FIGS. 123 to 129 described above. Then, the flashlight 1803, that is, the holder 1810 is blinked.

FIG. 136 is a flowchart showing the processing operation of the receiver 1800a held in the holder 1810 in the 16th embodiment.

The receiver 1800a receives the ID of the float indicated by the visible light signal from the float (step S1831). Next, the receiver 1800a acquires the program associated with the ID from the server (step S1832). Next, the receiver 1800a turns on the flashlight 1803 at each predetermined time according to the setting filter by executing the program (step S1833).

Here, the receiver 1800a may display the image corresponding to the received ID or the acquired program on the display 1801.

FIG. 137 is a diagram showing an example of an image displayed by the receiver 1800a in the 16th embodiment.

When the receiver 1800a receives the ID from, for example, the float of Santa Claus, the receiver 1800a displays an image of Santa Claus as shown in FIG. 137 (a). Further, as shown in FIG. 137 (b), the receiver 1800a may change the background color of the image of Santa Claus to the color of the setting filter at the same time as the flashlight 1803 is turned on. For example, when the color of the setting filter is red, the flashlight 1803 turns on the holder 1810 in red, and at the same time, an image of Santa Claus having a red background color is displayed on the display 1801. That is, the blinking of the holder 1810 and the display of the display 1801 are synchronized.

FIG. 138 is a diagram showing another example of the holder in the sixteenth embodiment.

The holder 1820 is configured similarly to the holder 1810 described above, but without the through holes 1811 and the variable filter 1812. Such a holder 1820 holds the receiver 1800a in a state where the display 1801 of the receiver 1800a is directed to the back plate portion 1820a. In this case, the receiver 1800a causes the display 1801 to emit light instead of the flashlight 1803. As a result, the light from the display 1801 is diffused to substantially the entire holder 1820. Therefore, when the receiver 1800a causes the display 1801 to emit light with red light according to the above program, the holder 1820 lights up in red. Similarly, when the receiver 1800a causes the display 1801 to emit light with yellow light according to the above program, the holder 1820 lights up in yellow. When the receiver 1800a causes the display 1801 to emit light with green light according to the above program, the holder 1820 lights up in green. By using such a holder 1820, it is possible to omit the setting of the variable filter 1812.

(Embodiment 17)
(Visible light signal)
139A to 139D are diagrams showing an example of a visible light signal according to the 17th embodiment.

Similar to the above, the transmitter generates a visible light signal of 4PPM, for example, as shown in FIG. 139A, and the luminance changes according to the visible light signal. Specifically, the transmitter allocates four slots to one signal unit and generates a visible light signal composed of a plurality of signal units. The signal unit indicates High (H) or Low (L) for each slot. Then, the transmitter emits bright light in the H slot and dimly emits or turns off in the L slot. For example, one slot is a period corresponding to a time of 1/9600 seconds.

Further, the transmitter may generate a visible light signal in which the number of slots assigned to one signal unit is variable, for example, as shown in FIG. 139B. In this case, the signal unit includes a signal indicating H in one or more continuous slots and a signal indicating L in one slot following the signal of H. Since the number of slots of H is variable, the total number of slots in the signal unit is variable. For example, as shown in FIG. 139B, the transmitter generates a visible light signal including a signal unit of 3 slots, a signal unit of 4 slots, and a signal unit of 6 slots in this order. Then, in this case as well, the transmitter emits bright light in the slot H and emits light in the slot L darkly or turns off.

Further, as shown in FIG. 139C, for example, the transmitter may allocate an arbitrary period (signal unit period) to one signal unit without allocating a plurality of slots to one signal unit. This signal unit period consists of a period of H and a period of L following the period of H. The period of H is adjusted according to the signal before modulation. The period of L may be fixed and may be a period corresponding to the slot. Further, the period of H and the period of L are, for example, 100 μs or more, respectively. For example, as shown in FIG. 139C, a transmitter includes a visible light signal including a signal unit having a signal unit period of 210 μs, a signal unit having a signal unit period of 220 μs, and a signal unit having a signal unit period of 230 μs in this order. To send. Then, in this case as well, the transmitter emits bright light in the period of H and emits light in the dark or turns off in the period of L.

Further, the transmitter may generate a signal indicating L and H alternately as a visible light signal, for example, as shown in FIG. 139D. In this case, in the visible light signal, the period L and the period H are adjusted according to the signal before modulation, respectively. For example, as shown in FIG. 139D, the transmitter indicates H for a period of 100 μs, then L for a period of 120 μs, then H for a period of 110 μs, and further L for a period of 200 μs. The visible light signal shown is transmitted. Then, in this case as well, the transmitter emits bright light in the period of H and emits light in the dark or turns off in the period of L.

FIG. 140 is a diagram showing a configuration of a visible light signal according to the 17th embodiment.

The visible light signal includes, for example, a signal 1, a brightness adjustment signal corresponding to the signal 1, a signal 2, and a brightness adjustment signal corresponding to the signal 2. When the transmitter generates the signal 1 and the signal 2 by modulating the unmodulated signal, the transmitter generates a brightness adjustment signal for those signals and generates the above-mentioned visible light signal.

The brightness adjustment signal corresponding to the signal 1 is a signal that compensates for the increase / decrease in brightness due to the change in brightness according to the signal 1. The brightness adjustment signal corresponding to the signal 2 is a signal that compensates for the increase / decrease in brightness due to the change in brightness according to the signal 2. Here, the brightness B1 is expressed by the brightness change according to the signal 1 and the brightness adjustment signal of the signal 1, and the brightness change according to the signal 2 and the brightness adjustment signal of the signal 2. Brightness B2 is expressed. The transmitter in the present embodiment generates the brightness adjustment signals of the signal 1 and the signal 2 as a part of the visible light signal so that the brightness B1 and the brightness B2 are equal to each other. As a result, the brightness is kept constant and flicker can be suppressed.

Further, when generating the above-mentioned signal 1, the transmitter generates a signal 1 including data 1, a preamble (header) following the data 1, and data 1 following the printable. Here, the preamble is a signal corresponding to the data 1 arranged before and after the preamble. For example, this preamble is a signal that serves as an identifier for reading data 1. In this way, since the signal 1 is composed of the two data 1 and the preamble arranged between them, even if the receiver reads the visible light signal from the middle of the preceding data 1, the signal 1 is the same. Data 1 (ie, signal 1) can be demodulated correctly.

(Bright line image)
FIG. 141 is a diagram showing an example of a emission line image obtained by imaging the receiver in the 17th embodiment.

As described above, the receiver acquires a bright line image including a visible light signal transmitted from the transmitter as a bright line pattern by imaging a transmitter whose luminance changes. By such imaging, the visible light signal is received by the receiver.

For example, as shown in FIG. 141, the receiver uses the N exposure lines included in the image sensor to take an image at time t1, and the emission line including the region a and the region b in which the emission line pattern appears, respectively. Get an image. Each of the region a and the region b is a region in which a bright line pattern appears due to a change in the luminance of the transmitter, which is the subject.

Here, the receiver demodulates the visible light signal from the emission line patterns in the regions a and b. However, if the receiver determines that the demodulated visible light signal alone is not sufficient, the receiver uses only M (M <N) continuous exposure lines corresponding to the region a out of the N exposure lines. , The image is taken at time t2. As a result, the receiver acquires an emission line image including only the region a of the region a and the region b. The receiver repeatedly performs such an imaging even at times t3 to t5. As a result, it is possible to receive a visible light signal having a sufficient amount of data from the subject corresponding to the region a at high speed. Further, the receiver uses only L (L <N) continuous exposure lines corresponding to the region b among the N exposure lines to take an image at time t6. As a result, the receiver acquires an emission line image including only the area b of the area a and the area b. The receiver repeatedly performs such imaging even at times t7 to t9. As a result, it is possible to receive a visible light signal having a sufficient amount of data from the subject corresponding to the area b at high speed.

Further, the receiver may acquire a bright line image including only the region a by performing the same imaging as the times t2 to t5 at the times t10 and t11. Further, the receiver may acquire an emission line image including only the region b at times t12 and t13 by performing the same imaging as at times t6 to t9.

Further, in the above example, when the receiver determines that the visible light signal is insufficient, the receiver continuously shoots the emission line image including only the region a at times t2 to t5, but the image is taken at time t1. If a bright line appears in the image obtained by the above-mentioned continuous shooting, the above-mentioned continuous shooting may be performed. Similarly, when the receiver determines that the visible light signal is insufficient, the receiver continuously shoots the emission line image including only the region b at times t6 to t9, but is obtained by imaging at time t1. If a bright line appears in the image, the above-mentioned continuous shooting may be performed. Further, the receiver may alternately acquire the emission line image including only the area a and the emission line image including only the area b.

The M continuous exposure lines corresponding to the above region a are exposure lines that contribute to the generation of the region a, and the L continuous exposure lines corresponding to the above region b are the generation of the region b. It is an exposure line that contributes to.

FIG. 142 is a diagram showing another example of the emission line image obtained by imaging the receiver in the 17th embodiment.

For example, as shown in FIG. 142, the receiver uses the N exposure lines included in the image sensor to take an image at time t1, and the emission line including the region a and the region b in which the emission line pattern appears, respectively. Get an image. Each of the region a and the region b is a region in which a bright line pattern appears due to a change in luminance of the transmitter as a subject, as described above. Further, each of the region a and the region b has a region (hereinafter referred to as an overlapping region) that overlaps with each other along the direction of the emission line or the exposure line.

Here, when the receiver determines that the visible light signal demodulated from the emission line patterns of the region a and the region b is insufficient, P (P <N) corresponding to the overlapping region among the N exposure lines. Imaging is performed at time t2 using only one continuous exposure line. As a result, the receiver acquires an emission line image including only the overlapping regions of the regions a and b. The receiver repeatedly performs such imaging at times t3 and t4. As a result, it is possible to receive a sufficient amount of visible light signals from the subject corresponding to each of the area a and the area b at substantially the same time and at high speed.

FIG. 143 is a diagram showing another example of the emission line image obtained by imaging the receiver in the 17th embodiment.

For example, as shown in FIG. 143, the receiver uses the N exposure lines included in the image sensor to take an image at time t1 to clearly show the portion a in which the emission line pattern appears indistinctly. An emission line image including a region including the appearing portion b is acquired. Similar to the above, this region is a region in which a bright line pattern appears when the transmitter, which is the subject, changes its luminance.

In such a case, when the receiver determines that the visible light signal demodulated from the emission line pattern in the above region is insufficient, Q (Q <N) corresponding to the portion b of the N exposure lines. The image is taken at time t2 using only the continuous exposure lines of. As a result, the receiver acquires an emission line image including only the portion b of the above-mentioned region. The receiver repeatedly performs such imaging at times t3 and t4. As a result, it is possible to receive a visible light signal with a sufficient amount of data from the subject corresponding to the above-mentioned region at high speed.

Further, the receiver may perform continuous shooting of the bright line image including only the portion a after the continuous shooting of the bright line image including only the portion b is performed.

As described above, when the emission line image contains a plurality of areas (or parts) in which the emission line pattern appears, the receiver assigns an order to each area, and only that area is ordered according to the order. Continuous shooting of emission line images including. In this case, the order may be an order according to the size of the signal (the size of the area or a portion) or an order according to the intelligibility of the emission line. Further, the order may be an order according to the color of the light from the subject corresponding to those areas. For example, the first continuous shooting is performed on the region corresponding to the red light, and the second continuous shooting is performed on the region corresponding to the white light. Also, only continuous shooting of the area for red light may be performed.

(HDR synthesis)
FIG. 144 is a diagram for explaining the adaptation of the receiver in the 17th embodiment to the camera system that performs HDR synthesis.

The vehicle is equipped with a camera system to prevent collisions. This camera system performs HDR (High Dynamic Range) composition using the image obtained by the image pickup of the camera. By this HDR composition, an image having a wide dynamic range of brightness can be obtained. The camera system recognizes surrounding vehicles, obstacles, people, etc. based on this wide dynamic range image.

For example, the camera system has a normal setting mode and a communication setting mode as setting modes. When the setting mode is the normal setting mode, for example, as shown in FIG. 144, the camera system performs four imaging times at the same shutter speed of 1/100 second and different sensitivities at times t1 to t4. .. The camera system performs HDR composition using the four images obtained by these four imagings.

On the other hand, when the setting mode is the communication setting mode, for example, as shown in FIG. 144, the camera system takes three images at the same shutter speed of 1/100 second and different sensitivities at times t5 to t7. I do. Further, the camera system takes an image at a time t8 with a shutter speed of 1/10000 seconds and with the maximum sensitivity (for example, ISO = 1600). The camera system performs HDR composition using three images obtained by the first three imagings out of the four imagings. Further, the camera system receives the visible light signal by the last imaging of the above four imagings and demodulates the emission line pattern appearing in the image obtained by the imaging.

Further, when the setting mode is the communication setting mode, the camera system does not have to perform HDR composition. For example, as shown in FIG. 144, the camera system performs imaging at a time t9 with a shutter speed of 1/100 second and a low sensitivity (eg, ISO = 200). Further, the camera system performs three times of imaging at time t10 to t12 with a shutter speed of 1/10000 second and with different sensitivities. The camera system recognizes surrounding vehicles, obstacles, people, etc. from the image obtained by the first one of the four images. Further, the camera system receives the visible light signal by the last three imagings out of the four imagings described above, and demodulates the emission line pattern appearing in the image obtained by the imaging.

In the example shown in FIG. 144, the images are taken with different sensitivities at the times t10 to t12, but the images may be taken with the same sensitivity.

In such a camera system, HDR synthesis can be performed and a visible light signal can be received.

(Security)
FIG. 145 is a diagram for explaining the processing operation of the visible light communication system according to the seventeenth embodiment.

This visible light communication system includes, for example, a transmitter arranged at a cash register, a smartphone as a receiver, and a server. The communication between the smartphone and the server and the communication between the transmitter and the server are performed via secure communication lines, respectively. In addition, communication between the transmitter and the smartphone is performed by visible light communication. The visible light communication system in the present embodiment secures security by determining whether or not the visible light signal from the transmitter is accurately received by the smartphone.

Specifically, the transmitter transmits a visible light signal indicating, for example, a value "100" to the smartphone by changing the luminance at time t1. When the smartphone receives the visible light signal at time t2, it transmits a radio signal indicating the value "100" to the server. The server receives the radio signal from the smartphone at time t3. At this time, the server performs a process for determining whether or not the value "100" indicated by the radio wave signal is the value of the visible light signal received from the transmitter to the smartphone. That is, the server transmits, for example, a radio signal indicating a value "200" to the transmitter. The transmitter that has received the radio wave signal transmits a visible light signal indicating the value "200" to the smartphone by changing the brightness at time t4. When the smartphone receives the visible light signal at time t5, it transmits a radio signal indicating the value "200" to the server. The server receives the radio signal from the smartphone at time t6. The server determines whether or not the value indicated by the received radio wave signal is the same as the value indicated by the transmitted radio wave signal at time t3. If they are the same, the server determines that the value "100" indicated by the visible light signal received at time t3 is the value of the visible light signal transmitted from the transmitter to the smartphone and received. On the other hand, if they are not the same, the server determines that the value "100" indicated by the visible light signal received at time t3 is suspicious as the value of the visible light signal transmitted from the transmitter to the smartphone and received.

This allows the server to determine if the smartphone has reliably received the visible light signal from the transmitter. That is, it is possible to prevent the smartphone from transmitting the signal to the server by pretending to have received the visible light signal even though the smartphone has not received the visible light signal from the transmitter.

In the above example, communication using radio signals is performed between the smartphone, the server, and the transmitter, but communication using optical signals other than visible light signals or communication using electric signals may also be performed. good. Further, the visible light signal transmitted from the transmitter to the smartphone indicates, for example, a billing value, a coupon value, a monster value, a bingo value, or the like.

(Vehicle related)
FIG. 146A is a diagram showing an example of vehicle-to-vehicle communication using visible light in the 17th embodiment.

For example, the leading vehicle recognizes that there is an accident in the traveling direction by a sensor (camera, etc.) mounted on the vehicle. Upon recognizing the accident in this way, the leading vehicle transmits a visible light signal by changing the brightness of the tail lamp. For example, the leading vehicle transmits a visible light signal prompting the following vehicle to decelerate. When the following vehicle receives the visible light signal by the image taken by the camera mounted on the vehicle, the following vehicle decelerates according to the visible light signal and further transmits a visible light signal prompting the following vehicle to decelerate. do.

In this way, the visible light signal prompting deceleration is sequentially transmitted from the beginning to a plurality of vehicles traveling in a row, and the vehicle receiving the visible light signal decelerates. Since the transmission of the visible light signal to each vehicle is fast, those multiple vehicles can decelerate at about the same time. Therefore, it is possible to alleviate the traffic congestion caused by the accident.

FIG. 146B is a diagram showing another example of vehicle-to-vehicle communication using visible light in the seventeenth embodiment.

For example, the front vehicle may transmit a visible light signal indicating a message (eg, "thank you") to the rear vehicle by varying the brightness of the tail lamp. This message is generated, for example, by a user operating a smartphone. Then, the smartphone transmits a signal indicating the message to the vehicle in front of the above. As a result, the front vehicle can transmit a visible light signal indicating the message to the rear vehicle.

FIG. 147 is a diagram showing an example of a method for determining the positions of a plurality of LEDs according to the seventeenth embodiment.

For example, a vehicle headlight has a plurality of LEDs (Light Emitting Diodes). The transmitter of this vehicle transmits a visible light signal from each of the plurality of LEDs of the headlight by individually changing the brightness of each of the plurality of LEDs. The receivers of other vehicles receive visible light signals from their plurality of LEDs by imaging the vehicle having their headlights.

At this time, the receiver determines the position of each of the plurality of LEDs from the image obtained by the imaging in order to recognize which LED the received visible light signal is transmitted from. .. Specifically, the receiver utilizes an accelerometer mounted on the same vehicle as the receiver and has a plurality of reference to the direction of gravity indicated by the accelerometer (eg, the down arrow in FIG. 147). Determine the position of each of the LEDs.

In the above example, the LED is given as an example of the light emitting body whose brightness changes, but a light emitting body other than the LED may be used.

FIG. 148 is a diagram showing an example of a emission line image obtained by imaging a vehicle in the 17th embodiment.

For example, the receiver mounted on the traveling vehicle acquires the emission line image shown in FIG. 148 by taking an image of the rear vehicle (following vehicle). The transmitter mounted on the following vehicle transmits a visible light signal to the preceding vehicle by changing the brightness of the two headlights of the vehicle. A camera that captures the rear is attached to the rear of the vehicle in front or the side mirrors. The receiver acquires a emission line image by imaging with the camera of the following vehicle as a subject, and demodulates the emission line pattern (visible light signal) included in the emission line image. As a result, the visible light signal transmitted from the transmitter of the following vehicle is received by the receiver of the preceding vehicle.

Here, the receiver acquires the ID of the vehicle having the headlights, the speed of the vehicle, and the vehicle type of the vehicle from each of the visible light signals transmitted from the two headlights and demodulated. If the IDs of the two visible light signals are the same, the receiver determines that the two visible light signals are signals transmitted from the same vehicle. Then, the receiver specifies the length (distance between lights) between the two headlights of the vehicle from the vehicle type of the vehicle. Further, the receiver measures the distance L1 between the two regions in which the emission line pattern appears, which is included in the emission line image. Then, the receiver calculates the distance (inter-vehicle distance) from the vehicle equipped with the receiver to the following vehicle by triangulation using the distance L1 and the distance between lights. The receiver determines the danger of collision based on the inter-vehicle distance and the speed of the vehicle acquired from the visible light signal, and notifies the driver of the vehicle of a warning according to the determination result. This makes it possible to avoid a vehicle collision.

In the above example, the receiver specifies the distance between lights from the vehicle type included in the visible light signal, but the distance between lights may be specified from information other than the vehicle type. Further, in the above example, the receiver issues a warning when it determines that there is a risk of collision, but even if the receiver outputs a control signal for causing the vehicle to perform an operation to avoid the risk. good. For example, the control signal is a signal for accelerating the vehicle or a signal for causing the vehicle to change lanes.

Further, in the above example, the camera captures the following vehicle, but an oncoming vehicle may be captured. Further, the receiver is in a mode of receiving the visible light signal as described above when it is determined from the image obtained by the image taken by the camera that fog is trapped around the receiver (that is, the vehicle equipped with the receiver). You may. Thereby, the receiver of the vehicle can identify the position and speed of the oncoming vehicle by receiving the visible light signal transmitted from the headlight of the oncoming vehicle even if the fog is in the vicinity.

FIG. 149 is a diagram showing an application example of the receiver and the transmitter in the 17th embodiment. Note that FIG. 149 is a view of the automobile from behind.

For example, a transmitter (car) 7006a having two tail lamps (light emitting units or lights) of a car transmits the identification information (ID) of the transmitter 7006a to a receiver configured as, for example, a smartphone. When the receiver receives the ID, it acquires the information associated with the ID from the server. For example, the information includes the ID of the car or transmitter, the distance between the light emitting parts, the size of the light emitting part, the size of the car, the shape of the car, the weight of the car, the number of the car, the appearance in front, or the danger. This is information indicating the presence or absence of. Further, the receiver may acquire such information directly from the transmitter 7006a.

FIG. 150 is a flowchart showing an example of the processing operation of the receiver and the transmitter 7006a in the 17th embodiment.

The ID of the transmitter 7006a and the information to be passed to the receiver that received the ID are associated and stored in the server (7106a). The information passed to the receiver includes the size of the light emitting part of the transmitter 7006a, the distance between the light emitting parts, the shape of the object having the transmitter 7006a as a component, the weight, the vehicle body number, and the like. Information such as the number, the state of a place that is difficult to observe from the receiver, and the presence or absence of danger may be included.

The transmitter 7006a transmits an ID (7106b). The transmission content may include the URL of the server and information stored in the server.

The receiver receives information such as the transmitted ID (7106c). The receiver acquires the information associated with the received ID from the server (7106d). The receiver displays the received information and the information acquired from the server (7106e).

The receiver is a receiver and a light emitting part from the size information of the light emitting part and the visible size of the imaged light emitting part, or from the distance information between the light emitting parts and the distance between the imaged light emitting parts in the manner of triangulation. The distance to and from is calculated (7106f). The receiver gives a warning of danger based on information such as the state of a place that is difficult to observe from the receiver and the presence or absence of danger (7106 g).

FIG. 151 is a diagram showing an application example of the receiver and the transmitter in the 17th embodiment.

For example, a transmitter (car) 7007b having two tail lamps (light emitting units or lights) of a car transmits information of the transmitter 7007b to a receiver 7007a configured as, for example, a transmitter / receiver in a parking lot. The information of the transmitter 7007b indicates the identification information (ID) of the transmitter 7007b, the number of the car, the size of the car, the shape of the car, or the weight of the car. Upon receiving the information, the receiver 7007a transmits parking availability, billing information, or parking position. The receiver 7007a may receive the ID and acquire information other than the ID from the server.

FIG. 152 is a flowchart showing an example of the processing operation of the receiver 7007a and the transmitter 7007b in the seventeenth embodiment. Since the transmitter 7007b not only transmits but also receives, the transmitter 7007b includes an in-vehicle transmitter and an in-vehicle receiver.

The ID of the transmitter 7007b and the information to be passed to the receiver 7007a that has received the ID are associated and stored in the server (parking lot management server) (7107a). The information passed to the receiver 7007a includes the shape and weight of an object whose component is the transmitter 7007b, an identification number such as a vehicle body number, an identification number of a user of the transmitter 7007b, and information for payment. May be included.

The transmitter 7007b (vehicle-mounted transmitter) transmits an ID (7107b). The transmission content may include the URL of the server and the information stored in the server. The parking lot receiver 7007a (parking lot transmission / reception device) transmits the received information to a server (parking lot management server) that manages the parking lot (7107c). The parking lot management server uses the ID of the transmitter 7007b as a key to acquire the information associated with the ID (7107d). The parking lot management server investigates the availability of the parking lot (7107e).

The parking lot receiver 7007a (parking lot transmitter / receiver) transmits parking availability, parking position information, or the address of a server that holds such information (7107f). Alternatively, the parking lot management server sends this information to another server. The transmitter (vehicle-mounted receiver) 7007b receives the information transmitted above (7107 g). Alternatively, the in-vehicle system acquires this information from another server.

The parking lot management server controls the parking lot so that parking can be easily performed (7107h). For example, it controls a multi-story parking lot. The transmission / reception device of the parking lot transmits an ID (7107i). The in-vehicle receiver (transmitter 7007b) makes an inquiry to the parking lot management server based on the user information of the in-vehicle receiver and the received ID (7107j).

The parking lot management server charges according to the parking time and the like (7107k). The parking lot management server controls the parking lot so that the parked vehicle can be easily accessed (7107 m). For example, it controls a multi-story parking lot. The in-vehicle receiver (transmitter 7007b) displays a map to the parking position and navigates from the current location (7107n).

(In the train)
FIG. 153 is a diagram showing a configuration of a visible light communication system applied to the inside of a train according to the seventeenth embodiment.

The visible light communication system includes, for example, a plurality of lighting devices 1905 arranged in the train, a smartphone 1906 held by the user, a server 1904, and a camera 1903 arranged in the train.

Each of the plurality of lighting devices 1905 is configured as the above-mentioned transmitter, illuminates the light, and transmits a visible light signal by changing the brightness. This visible light signal indicates the ID of the lighting device 1905 that transmits the visible light signal.

The smartphone 1906 is configured as the above-mentioned receiver, and receives an image of the lighting device 1905 to receive a visible light signal transmitted from the lighting device 1905. For example, a user causes a smartphone 1906 to receive a visible light signal when he / she is involved in a trouble (for example, a molester or a fight) on a train. When the smartphone 1906 receives the visible light signal, it notifies the server 1904 of the ID indicated by the visible light signal.

Upon receiving the notification of the ID, the server 1904 identifies the camera 1903 whose imaging range is the range illuminated by the lighting device 1905 identified by the ID. Then, the server 1904 causes the specified camera 1903 to take an image of the range illuminated by the lighting device 1905.

The camera 1903 takes an image in response to an instruction from the server 1904, and transmits the image obtained by the image taking to the server 1904.

This makes it possible to acquire an image showing the situation of trouble in the train. This image can be used as evidence of trouble.

Further, the user may operate the smartphone 1906 to transmit the image obtained by the image pickup of the camera 1903 from the server 1904 to the smartphone 1906.

Further, the smartphone 1906 may display an image pickup button on the screen, and when the image pickup button is touched by the user, a signal prompting the image pickup may be transmitted to the server 1904. This allows the user to determine the timing of imaging by himself / herself.

FIG. 154 is a diagram showing a configuration of a visible light communication system applied to a facility such as an amusement park in the 17th embodiment.

The visible light communication system includes, for example, a plurality of cameras 1903 arranged in a facility and jewelry 1907 attached to a person.

The accessory 1907 is, for example, a headband having a ribbon to which a plurality of LEDs are attached. Further, the accessory 1907 is configured as the above-mentioned transmitter, and transmits a visible light signal by changing the brightness of a plurality of LEDs.

Each of the plurality of cameras 1903 is configured as the above-mentioned receiver and has a visible light communication mode and a normal imaging mode. In addition, each of these plurality of cameras 1903 is arranged at different points in the path in the facility.

Specifically, when the camera 1903 is set to the visible light communication mode and the accessory 1907 is imaged as a subject, the camera 1903 receives a visible light signal from the accessory 1907. Upon receiving the visible light signal, the camera 1903 switches the set mode from the visible light communication mode to the normal imaging mode. As a result, the camera 1903 takes an image of a person wearing the accessory 1907 as a subject.

Therefore, when a person wearing jewelry 1907 is walking along the path in the facility, the cameras 1903 near the person will take pictures of the person one after another. As a result, it is possible to automatically acquire and save an image showing the person having fun at the facility.

The camera 1903 does not perform imaging in the normal imaging mode immediately after receiving the visible light signal, but may perform imaging in the normal imaging mode when, for example, an instruction to start imaging is received from a smartphone. As a result, the user can have the camera 1903 image the image at the timing of touching the image pickup start button displayed on the screen of the smartphone.

FIG. 155 is a diagram showing an example of a visible light communication system including a playset and a smartphone according to the seventeenth embodiment.

The playset 1901 is configured as the above-mentioned transmitter including, for example, a plurality of LEDs. That is, the playset 1901 transmits a visible light signal by changing the brightness of the plurality of LEDs.

The smartphone 1902 receives the visible light signal transmitted from the playset 1901 by imaging the playset 1901. Then, as shown in FIG. 155 (a), when the smartphone 1902 receives the visible light signal for the first time, the moving image 1 associated with the visible light signal and the first time is transmitted from, for example, a server. Download and play. On the other hand, when the smartphone 1902 receives the visible light signal for the second time, as shown in FIG. 155 (b), the smartphone 1902 outputs the moving image 2 associated with the visible light signal and the second time from, for example, a server. Download and play.

That is, even if the smartphone 1902 receives the same visible light signal, the moving image to be played is switched according to the number of times the visible light signal is received. The number of times the visible light signal is received may be counted by the smartphone 1902 or may be counted by the server. Alternatively, the smartphone 1902 does not continuously play the same moving image even if the same visible light signal is received a plurality of times. Alternatively, the smartphone 1902 lowers the appearance probability of the video already played among the plurality of videos associated with the same visible light signal, and preferentially downloads and plays the video having a high appearance probability. You may.

Further, the smartphone 1902 may receive a visible light signal transmitted from a touch panel provided in an information center of a facility having a plurality of stores and display an image corresponding to the visible light signal. For example, when the touch panel displays the initial screen showing the outline of the facility, the touch panel transmits a visible light signal showing the outline of the facility by changing the luminance. Therefore, when the smartphone receives the visible light signal by imaging the touch panel displaying the initial screen, the smartphone can display an image showing the outline of the facility on its own display. Here, when the touch panel is operated by the user, the touch panel displays, for example, a store image showing information on a specific store. At this time, the touch panel transmits a visible light signal indicating the information of the specific store. Therefore, the smartphone can display a store image showing information of a specific store when it receives the visible light signal by capturing the touch panel displaying the store image. In this way, the smartphone can display an image synchronized with the touch panel.

(Summary of the above embodiment)
The reproduction method according to one aspect of the present invention includes a signal receiving step of receiving a visible light signal from a transmitter that transmits a visible light signal due to a change in brightness of a light source by a sensor of a terminal device, and the visible from the terminal device. The transmission step of transmitting a request signal for requesting the content associated with the optical signal to the server, and the content including the time and the data reproduced at each time by the terminal device are transmitted from the server. The content receiving step to be received and the reproduction step of reproducing the data corresponding to the time of the clock provided in the terminal device among the contents are included.

As a result, as shown in FIG. 131C, the content including each time and the data to be reproduced at each time is received by the terminal device, and the data corresponding to the clock time of the terminal device is reproduced. Therefore, the terminal device can appropriately reproduce the data in the content at the correct time shown in the content without reproducing the data at the wrong time. Specifically, as in the method e of FIG. 131A, the receiver, which is a terminal device, reproduces the content from the time point of (receiver time-content reproduction start time). The data corresponding to the time of the clock of the terminal device described above is the data at the time of (receiver time-content reproduction start time) of the contents. Further, even in the transmitter, if the content related to the content (transmitter side content) is reproduced, the terminal device can appropriately synchronize the content with the transmitter side content and reproduce the content. The content is audio or image.

Further, the clock provided in the terminal device and the reference clock may be synchronized by GPS (Global Positioning System) radio waves or NTP (Network Time Protocol) radio waves.

As a result, as shown in FIGS. 130 and 132, since the clock of the terminal device (receiver) and the reference clock are synchronized, the time corresponds to an appropriate time according to the reference clock. You can play back the data you want.

Further, the visible light signal may indicate the time when the visible light signal is transmitted from the transmitter.

As a result, as shown in the method d of FIG. 131A, the terminal device (receiver) can receive the content associated with the time (transmitter time) when the visible light signal is transmitted from the transmitter. For example, if the transmitter time is 5:43, the content to be played at 5:43 can be received.

Further, in the reproduction method, the terminal device can see the time when the processing for synchronizing the clock of the terminal device and the reference clock is performed by the GPS radio wave or the NTP radio wave. When the time before the time when the optical signal is received is earlier than a predetermined time, the clock of the terminal device and the clock of the transmitter are synchronized according to the time indicated by the visible light signal transmitted from the transmitter. Very good.

For example, if a predetermined time elapses after the process for synchronizing the clock of the terminal device and the reference clock is performed, the synchronization may not be properly maintained. In such a case, the terminal device may not be able to reproduce the content at the time synchronized with the transmitter-side content reproduced by the transmitter. Therefore, in the reproduction method according to one aspect of the present invention, as in steps S1829 and S1830 of FIG. 130, when a predetermined time elapses, between the clock of the terminal device (receiver) and the clock of the transmitter. Synchronized. Therefore, the terminal device can reproduce the content at the time synchronized with the transmitter-side content reproduced by the transmitter.

Further, each of the servers has a plurality of contents associated with the time, and in the content receiving step, when the contents associated with the time indicated by the visible light signal do not exist in the server, the content is present. Of the plurality of contents, the content that is closest to the time indicated by the visible light signal and is associated with the time after the time indicated by the visible light signal may be received.

As a result, as shown in the method d of FIG. 131A, even if the content associated with the time indicated by the visible light signal does not exist on the server, the appropriate content is received from the plurality of contents on the server. be able to.

Further, the signal receiving step of receiving the visible light signal from the transmitter that transmits the visible light signal due to the change in the brightness of the light source by the sensor of the terminal device, and the content associated with the visible light signal from the terminal device. The visible light signal includes a transmission step of transmitting a request signal for requesting to a server, a content reception step in which the terminal device receives content from the server, and a reproduction step of reproducing the content. The ID information and the time when the visible light signal is transmitted from the transmitter are shown, and in the content receiving step, even if the ID information indicated by the visible light signal and the content associated with the time are received. good.

As a result, as in the method d of FIG. 131A, the visible light signal is associated with the time (transmitter time) transmitted from the transmitter from among the plurality of contents associated with the ID information (transmitter ID). The received content is received and played. Therefore, it is possible to play the content appropriate for the transmitter ID and the transmitter time.

Further, the visible light signal is transmitted from the transmitter by including the second information indicating the hour and minute of the time and the first information indicating the second of the time. In the signal receiving step, the second information may be received and the first information may be received more times than the number of times the second information is received.

As a result, for example, when notifying the terminal device of the time when each packet included in the visible light signal is transmitted in seconds, a packet indicating the current time expressed using all of hours, minutes, and seconds. Can be reduced from the trouble of transmitting the message to the terminal device every second. That is, as shown in FIG. 126, a packet indicating only seconds (time packet 1) if the hours and minutes of the time the packet is transmitted are not updated from the hours and minutes indicated in the previously transmitted packet. ), Only the first information needs to be transmitted. Therefore, by reducing the second information, which is a packet indicating hours and minutes (time packet 2), from the first information, which is a packet indicating seconds (time packet 1), transmitted by the transmitter. It is possible to suppress the transmission of packets containing redundant contents.

Further, the sensor of the terminal device is an image sensor, and in the signal receiving step, the shutter speed of the image sensor is set to a first speed and a second speed higher than the first speed. Continuous shooting is performed by the image sensor while switching alternately, and (a) when the subject of the shooting by the image sensor is a bar code, the shooting when the shutter speed is the first speed is performed. An image showing a bar code is acquired, and a bar code identifier is obtained by decoding the bar code shown in the image. (B) When the subject photographed by the image sensor is the light source. By shooting when the shutter speed is the second speed, a bright line image which is an image including a bright line corresponding to each of the plurality of exposure lines included in the image sensor is acquired, and the acquired bright line image is used. The visible light signal is acquired as a visible light identifier by decoding a plurality of emission line patterns included, and in the reproduction method, an image obtained by shooting when the shutter speed is the first speed is further obtained. It may be displayed.

As a result, as shown in FIG. 102, identifiers corresponding to the barcodes and visible light signals can be appropriately acquired, and an image on which the barcode or the light source as the subject is projected can be obtained. Can be displayed.

Further, in the acquisition of the visible light identifier, the first packet including the data unit and the address unit is acquired from the plurality of emission line patterns, and at least one that has already been acquired before the first packet. It is determined whether or not there are a predetermined number or more of the second packets, which are packets including the same address part as the address part of the first packet, among the packets, and the second packet is the predetermined number. When it is determined that the above is present, the pixel value of a part of the area of the emission line image corresponding to each data part of the predetermined number or more of the second packet and the data part of the first packet The composite pixel value is calculated by matching with the pixel value of a part of the corresponding bright line image, and at least a part of the visible light identifier is acquired by decoding the data portion including the composite pixel value. You may.

As a result, as shown in FIG. 74, even if the data part is slightly different for each of the plurality of packets including the same address part, the appropriate data part can be obtained by matching the pixel values of the data part of those packets. It can be decoded and at least part of the visible light identifier can be obtained correctly.

Further, the first packet further includes a first error correction code for the data unit and a second error correction code for the address unit, and in the signal receiving step, the second from the transmitter. The address unit and the data unit that receives the second error correction code transmitted by the brightness change according to the frequency of the above and are transmitted by the brightness change according to the first frequency higher than the second frequency. And the first error correction code may be received.

As a result, it is possible to prevent the address unit from being erroneously received and to quickly acquire the data unit having a large amount of data.

Further, in the acquisition of the visible light identifier, a first packet including a data unit and an address unit is acquired from the plurality of emission line patterns, and at least one that has already been acquired before the first packet. It is determined whether or not at least one second packet, which is a packet containing the same address portion as the address portion of the first packet, exists among the packets, and the presence of the at least one second packet is determined. When it is determined, it is determined whether or not the data units of the at least one second packet and the first packet are all equal, and when it is determined that the data units are not all equal. In each of the at least one second packet, the number of parts different from each part included in the data part of the first packet among the parts included in the data part of the second packet is determined. If there is a second packet in which it is determined whether or not there are a predetermined number or more, and it is determined that the number of different portions is present in the predetermined number or more among the at least one second packet, the said If at least one second packet is discarded and there is no second packet for which it is determined that the number of different portions is equal to or greater than the predetermined number among the at least one second packet, the first packet is described. Among the packet and the at least one second packet, a plurality of packets having the same number of packets having the same data part are specified, and the data part included in each of the plurality of packets is referred to as the first packet. At least a part of the visible light identifier may be acquired by decoding as a data part corresponding to the address part included in.

As a result, as shown in FIG. 73, when a plurality of packets having the same address part are received, even if the data parts of the packets are different, the appropriate data part can be decoded and visible. At least part of the optical identifier can be obtained correctly. That is, a plurality of packets having the same address unit transmitted from the same transmitter basically have the same data unit. However, when the terminal device switches the transmitter that is the source of the packet, the terminal device may receive a plurality of packets having the same address unit but different data units from each other. In such a case, in the reproduction method according to one aspect of the present invention, as in step S10106 of FIG. 73, the packet (second packet) that has already been received is discarded, and the latest packet (first packet) is discarded. The data part of the packet) can be decoded as the correct data part corresponding to the address part. Further, even when the transmitter is not switched as described above, the data unit of a plurality of packets having the same address unit may be slightly different depending on the transmission / reception status of the visible light signal. In such a case, in the reproduction method according to one aspect of the present invention, an appropriate data unit can be decoded by a so-called majority vote as in step S10107 of FIG. 73.

Further, in the acquisition of the visible light identifier, a plurality of packets including a data unit and an address unit are acquired from the plurality of emission line patterns, and all of the acquired packets included in the data unit. It is determined whether or not there is a 0-terminated packet, which is a packet in which the bit of is 0, and if it is determined that the 0-terminated packet exists, the address portion of the 0-terminated packet among the plurality of packets It is determined whether or not all N related packets (N is an integer of 1 or more) which are packets including the associated address part exist, and when it is determined that all the N related packets exist. , The visible light identifier may be obtained by arranging and decoding each data part of the N related packets. For example, the address part associated with the address part of the 0-terminated packet is an address part indicating an address of 0 or more, which is smaller than the address shown in the address part of the 0-terminated packet.

Specifically, as shown in FIG. 75, it is determined whether or not all the packets having addresses equal to or less than the address of the 0-terminated packet are aligned as related packets, and if it is determined that they are aligned, they are related. Each data part of the packet is decoded. As a result, the terminal device does not know in advance how many related packets are required to obtain the visible light identifier, and further, it does not know in advance the addresses of those related packets. However, it can be easily known when the 0-terminated packet is acquired. As a result, the terminal device can acquire an appropriate visible light identifier by arranging and decoding each data unit of the N related packets.

(Embodiment 18)
Hereinafter, a protocol corresponding to a variable length and a variable number of divisions will be described.

FIG. 156 is a diagram showing an example of a transmission signal according to the present embodiment.

The transmitted packet is composed of a preamble, a TYPE, a payload, and a check unit. Packets may be transmitted continuously or intermittently. By providing a period during which packets are not transmitted, the state of the liquid crystal can be changed when the backlight is turned off, and the dynamic resolution of the liquid crystal display can be improved. Interference can be avoided by making the packet transmission interval random.

For the preamble, a pattern that does not appear in 4PPM is used. The reception process can be simplified by using a short basic pattern.

By expressing the number of data divisions by the type of preamble, the number of data divisions can be made variable without using an extra transmission slot.

By changing the payload length according to the TYPE value, the transmission data can be made variable length. In TYPE, the payload length may be expressed, or the data length before division may be expressed. By expressing the address of the packet by the value of TYPE, the receiver can correctly arrange the received packets. Further, the payload length (data length) expressed by the TYPE value may be changed depending on the type of preamble or the number of divisions.

Efficient error correction (detection) can be performed by changing the length of the check unit according to the payload length. By setting the shortest length of the check unit to 2 bits, it can be efficiently converted to 4PPM. Further, by changing the type of error correction (detection) code depending on the payload length, error correction (detection) can be performed efficiently. The length of the check unit and the type of error correction (detection) code may be changed depending on the type of preamble or the value of TYPE.

There are combinations with the same data length for combinations with different payloads and divisions. In such a case, even if the data values are the same, more values can be expressed by giving different meanings to each combination.

Hereinafter, the high-speed transmission / luminance modulation protocol will be described.

FIG. 157 is a diagram showing an example of a transmission signal according to the present embodiment.

The transmission packet is composed of a preamble unit, a body unit, and a brightness adjustment unit. The body includes an address part, a data part, and an error correction (detection) code part. By allowing intermittent transmission, the same effect as described above can be obtained.

(Embodiment 19)
(Frame configuration of Single frame transition)
FIG. 158 is a diagram showing an example of a transmission signal according to the present embodiment.

The transmission frame is composed of a preamble (PRE), a frame length (FLEN), an ID type (IDTYPE), a content (ID / DATA), and a check code (CRC), and may include a content type (CONTENTTYPE). The number of bits in each area is an example.

By specifying the length of ID / DATA in FLEN, variable length content can be transmitted.

CRC is an inspection code for correcting or detecting an error in a part other than PRE. By changing the CRC length according to the length of the inspection area, the inspection ability can be maintained above a certain level. Further, by using different inspection codes according to the length of the inspection region, the inspection ability per CRC length can be improved.

(Frame structure of Multiple frame transmission)
FIG. 159 is a diagram showing an example of a transmission signal according to the present embodiment.

The transmission frame is composed of a preamble (PRE), an address (ADDR), and a part of divided data (DATAPART), and may include each of a number of divisions (PARTNUM) and an address flag (ADDRFRAG). The number of bits in each area is an example.

By dividing the content into multiple parts and transmitting it, long-distance communication can be performed.

By dividing the size into equal parts, the maximum frame length can be reduced and stable communication can be performed.

When equal division is not possible, data of an appropriate size can be transmitted by making a part of the divided portion smaller than the other divided portions.

More information can be transmitted by setting the division sizes to different sizes and giving meaning to the combination of division sizes. For example, even if the data has the same value of 32 bits, when 8 bits are transmitted four times, when 16 bits are transmitted twice, and when 15 bits are transmitted once and 17 is transmitted once. Then, by treating it as different information, it is possible to express a larger amount of information.

By indicating the number of divisions in PARTNUM, the receiver can immediately know the number of divisions and can accurately display the progress of reception.

When ADDRFRAG is 0, it is not the last address, but when it is 1, it is the last address. Therefore, the area indicating the number of divisions becomes unnecessary, and transmission can be performed in a shorter time.

The CRC is an inspection code for correcting or detecting an error in a portion other than PRE, as described above. By this inspection, interference can be detected when transmission frames from a plurality of sources are received. By setting the CRC length to an integral multiple of the DATAPART length, interference can be detected most efficiently.

An inspection code for inspecting a portion other than PRE of each frame may be added to the end of the divided frames (frames indicated by (a), (b) or (c) in FIG. 159).

The IDTYPE shown by (d) in FIG. 159 may have a fixed length such as 4 bits or 5 bits, as in (a) to (d) of FIG. 158, or the IDTYPE length may be changed by the ID / DATA length. good. As a result, the same effect as described above can be obtained.

(Designation of ID / DATA length)
FIG. 160 is a diagram showing an example of a transmission signal according to the present embodiment.

In the case of (a) to (d) of FIG. 158, by setting as shown in the tables (a) and (b) shown in FIG. 160, the ocode can be represented at 128 bits.

(CRC length and generated polynomial)
FIG. 161 is a diagram showing an example of a transmission signal according to the present embodiment.

By setting the CRC length in this way, the inspection ability can be maintained regardless of the length of the inspection target.

The generated polynomial is an example, and another generated polynomial may be used. Further, an inspection code other than CRC may be used. These can improve the inspection ability.

(Specifying the DATAPART length and the last address according to the preamble type)
FIG. 162 is a diagram showing an example of a transmission signal according to the present embodiment.

By indicating the DATAPART length by the type of preamble, the area indicating the DATAPART length is not required, and the information can be transmitted in a shorter transmission time. Further, by indicating whether or not the address is the last address, an area indicating the number of divisions is not required, and information can be transmitted in a shorter transmission time. Further, in the case of (b) of FIG. 162, since the DATAPART length is not known in the case of the last address, it is estimated that it is the same as the DATAPART length of the frame which is not the last address received immediately before or immediately after the frame reception. By performing the reception process, it is possible to receive normally.

The address length may differ depending on the type of preamble. As a result, it is possible to increase the number of combinations of transmission information lengths and to transmit in a short time.

In the case of (c) of FIG. 162, the number of divisions is specified by the preamble, and a region indicating the DATAPART length is added.

(Specify address)
FIG. 163 is a diagram showing an example of a transmission signal according to the present embodiment.

By indicating the address of the frame with the value of ADDR, the receiver can reconstruct the correctly transmitted information.

By indicating the number of divisions by the value of PARTNUM, the receiver can always know the number of divisions when the first frame is received, and can accurately display the progress of reception.

(Prevention of interference due to the difference in the number of divisions)
164 and 165 are diagrams and flowcharts showing an example of a transmission / reception system according to the present embodiment.

When the transmission information is equally divided and transmitted, the signals from the transmitter A and the transmitter B in FIG. 164 have different preambles, so that the receiver confuses the transmitter even when these signals are received at the same time. The transmitted information can be reconstructed without any need.

By providing the transmitters A and B with a division number setting unit, the user can set the number of divisions of the transmitters installed nearby to be different, and interference can be prevented.

By registering the number of divisions of the received signal in the server, the receiver can know the number of divisions set by the transmitter, and other receivers can acquire the information from the server. The progress of reception can be displayed accurately.

The receiver obtains from the server or from the storage unit of the receiver whether or not the signal from the nearby or corresponding transmitter is divided into equal lengths. When the acquired information is divided into equal lengths, the signal is restored only from the frames having the same DATAPART length. If this is not the case, or if all the addresses are not aligned in the same DATAPART length frame for a predetermined time or longer, the signals are restored by combining the frames with different DATAPART lengths.

(Prevention of interference due to the difference in the number of divisions)
FIG. 166 is a flowchart showing the operation of the server in the present embodiment.

The server receives from the receiver the ID received by the receiver and the split configuration (what combination of DATAPART lengths received the signal). When the ID is the target of expansion by the division configuration, the digitized pattern of the division configuration is used as the auxiliary ID, and the information associated with the expansion ID obtained by combining the ID and the auxiliary ID is used as the key to receive the receiver. Pass to.

If it is not the target of expansion by the divided configuration, it is confirmed whether the divided configuration associated with the ID exists in the storage unit, and whether it is the same as the received divided configuration. If they are different, a reconfirmation command is sent to the receiver. This makes it possible to prevent incorrect information from being presented due to a reception error in the receiver.

If the same divided configuration is received with the same ID within a predetermined time after the reconfirmation command is transmitted, it is determined that the divided configuration has been changed, and the divided configuration associated with the ID is updated. Thereby, as described as the description of FIG. 164, it is possible to cope with the case where the divided configuration is changed.

If the split configuration is not stored, if the received split configuration matches the stored split configuration, or if the split configuration is updated, the associated information is passed to the receiver using the ID as a key. The divided configuration is associated with the ID and stored in the storage unit.

(Display of reception progress)
167 to 172 are flowcharts and diagrams showing an example of the operation of the receiver in the present embodiment.

The receiver acquires the type and ratio of the number of divisions of the transmitter supported by the receiver or the transmitter in the vicinity of the receiver from the storage area of the server or the receiver. If a part of the divided data has already been received, the type and ratio of the number of divided parts of the transmitter transmitting the information matching the part is acquired.

The receiver receives the divided frames.

If the last address has already been received, if the acquired number of divisions is only one type, or if the number of divisions supported by the running receiving application is only one type, the number of divisions is Since it is known, the progress is displayed based on the number of divisions.

If this is not the case and the available processing resources are low or in energy saving mode, the receiver will calculate and display the progress in simple mode. On the other hand, when there are many processing resources available or the mode is not the energy saving mode, the progress status is calculated and displayed in the maximum likelihood estimation mode.

FIG. 168 is a flowchart showing a calculation method of the progress status in the simple mode.

First, the receiver acquires the standard number of divisions Ns from the server. Alternatively, the receiver reads the standard division number Ns from its own internal data holding unit. The standard number of divisions is (a) the most frequent value or the expected value of the number of transmitters transmitted by the number of divisions, (b) the number of divisions determined for each packet length, and (c) the number of divisions determined for each application. The number, or (d) the location where the receiver is located, and the number of divisions determined for each identifiable range.

Next, the receiver determines whether or not it has received a packet indicating that it is the final address. If it is determined that the packet is being received, the address of the last packet is set to N. On the other hand, if it is determined that the signal has not been received, Ne is defined as the maximum received address Amax plus a number of 1 or 2 or more. Here, the receiver determines whether or not Ne> Ns. If it is determined that Ne> Ns, the receiver sets N = Ne. On the other hand, if it is determined that Ne> Ns, the receiver sets N = Ns.

Then, the receiver calculates the ratio of the number of received packets to the packets required for receiving the entire signal, assuming that the number of divisions of the signal being received is N.

In such a simple mode, the progress can be calculated by a simpler calculation than in the maximum likelihood estimation mode, which is advantageous in terms of processing time or energy consumption.

FIG. 169 is a flowchart showing a calculation method of the progress status in the maximum likelihood estimation mode.

First, the receiver acquires the prior distribution of the number of divisions from the server. Alternatively, the receiver reads the prior distribution from its internal data holding unit. The prior distribution is (a) defined as the distribution of the number of transmitters transmitted by the number of divisions, (b) defined for each packet length, (c) defined for each application, or. , (D) The location where the receiver is located, and is defined for each identifiable range.

Next, the receiver receives the packet x and calculates the probability P (x | y) of receiving the packet x when the number of divisions is y. Then, the receiver obtains the probability P (y | x) that the number of divisions of the transmission signal is y when the packet x is received as P (x | y) × P (y) ÷ A (note that A). Is a normalized multiplier). Further, the receiver is set to P (y) = P (y | x).

Here, the receiver determines whether the division number estimation mode is the maximum likelihood mode or the likelihood average mode. In the maximum likelihood mode, the receiver calculates the ratio of the number of received packets with y having the maximum P (y) as the number of divisions. On the other hand, in the likelihood average mode, the receiver calculates the ratio of the number of received packets by using the sum of y × P (y) as the number of divisions.

In such a maximum likelihood estimation mode, it is possible to calculate the degree of progress more accurately than in the simple mode.

If the division number estimation mode is the maximum likelihood mode, the likelihood of the last address is calculated from the addresses received so far, and the maximum likelihood is estimated to be the number of divisions. Display the progress of reception. This display method can display the progress status closest to the actual progress status.

FIG. 170 is a flowchart showing a display method in which the progress status does not decrease.

First, the receiver calculates the ratio of the number of received packets to the packets required for receiving the entire signal. Then, the receiver determines whether or not the calculated ratio is smaller than the displayed ratio. If it is determined that the ratio is smaller than the displayed ratio, the receiver further determines whether or not the displayed ratio is the calculation result before a predetermined time. If it is determined that the calculation result is earlier than a predetermined time, the receiver displays the calculated ratio. On the other hand, if it is determined that the calculation result is not more than a predetermined time ago, the receiver continues to display the displayed ratio.

Further, when the receiver determines that the calculated ratio is equal to or higher than the displayed ratio, Ne is defined as the number obtained by adding a number of 1 or 2 or more to the maximum received address Amax. Then, the receiver displays the calculated ratio.

It is unnatural that the calculation result of the progress status becomes smaller than before when the final packet is received, that is, the displayed progress status (progress degree) decreases. However, with the above-mentioned display method, such an unnatural display can be suppressed.

FIG. 171 is a flowchart showing a method of displaying the progress status when there are a plurality of packet lengths.

First, the receiver calculates the ratio P of the number of received packets for each packet length. Here, the receiver determines whether the display mode is the maximum mode, the full display mode, or the latest mode. If it is determined that the mode is the maximum mode, the receiver displays the maximum ratio of the respective ratios P of the plurality of packet lengths. If it is determined that the mode is all display mode, the receiver displays all the ratios P. If it is determined that the mode is the latest mode, the receiver displays the ratio P of the packet length of the last received packet.

In FIG. 172, (a) is the progress status calculated as the simple mode, (b) is the progress status calculated as the maximum likelihood mode, and (c) is calculated by using the smallest number of acquired divisions as the number of divisions. The progress of the case. Since the progress status increases in the order of (a), (b), and (c), all the progress status can be displayed at the same time by displaying (a), (b), and (c) in this way. ..

(Light emission control by common switch and pixel switch)
In the transmission method according to the present embodiment, for example, each LED included in the LED display for displaying an image is changed in brightness according to the switching of a common switch and a pixel switch, so that a visible light signal (also known as a visible light communication signal) is changed. ) Is sent.

The LED display is configured as, for example, a large display arranged outdoors. Further, the LED display includes a plurality of LEDs arranged in a matrix, and displays an image by blinking these LEDs according to an image signal. Such an LED display is composed of a plurality of common lines (COM lines) and a plurality of pixel lines (SEG lines). Each common line consists of a plurality of LEDs arranged in a row in the horizontal direction, and each pixel line consists of a plurality of LEDs arranged in a row in the vertical direction. Further, each of the plurality of common lines is connected to the common switch corresponding to the common line. The common switch is, for example, a transistor. Each of the plurality of pixel lines is connected to the pixel switch corresponding to the pixel line. A plurality of pixel switches corresponding to a plurality of pixel lines are provided in, for example, an LED driver circuit (constant current circuit). The LED driver circuit is configured as a pixel switch control unit that switches a plurality of pixel switches.

More specifically, one of the anode and cathode of each LED included in the common line is connected to a terminal such as a collector of a transistor corresponding to the common line. Further, the other of the anode and cathode of each LED included in the pixel line is connected to the terminal (pixel switch) corresponding to the pixel line in the LED driver circuit.

When such an LED display displays an image, a common switch control unit that controls a plurality of common switches turns on those common switches in a time-division manner. For example, the common switch control unit turns on only the first common switch among the plurality of common switches during the first period, and the second common among the plurality of common switches during the next second period. Turn on only the switch. Then, the LED driver circuit turns on each pixel switch according to the video signal while any common switch is turned on. As a result, the LED corresponding to the common switch and the pixel switch is lit only during the period when the common switch is on and the pixel switch is on. The brightness of the pixels in the image is expressed by this lighting period. That is, the brightness of the pixels of the video is PWM controlled.

In the transmission method according to the present embodiment, the visible light signal is transmitted by using such an LED display, a common switch and a pixel switch, and a common switch control unit and a pixel switch control unit. Further, the transmitter (also referred to as a transmitter) in the present embodiment that transmits a visible light signal by such a transmission method includes a common switch control unit and a pixel switch control unit thereof.

FIG. 173 is a diagram showing an example of a transmission signal according to the present embodiment.

The transmitter transmits each symbol included in the visible light signal according to a predetermined symbol period. For example, when the symbol "00" is transmitted by 4PPM, the transmitter switches the common switch according to the symbol (luminance change pattern of "00") in the symbol cycle consisting of 4 slots. Then, the transmitter switches the pixel switch according to the average luminance indicated by the video signal or the like.

More specifically, when the average brightness in the symbol period is set to 75% ((a) in FIG. 173), the transmitter turns off the common switch during the period of the first slot, and the second slot to the fourth slot are turned off. Turn on the common switch during the period up to the slot. Further, the transmitter turns off the pixel switch during the period of the first slot and turns on the pixel switch during the period from the second slot to the fourth slot. As a result, the LED corresponding to the common switch and the pixel switch is lit only during the period when the common switch is on and the pixel switch is on. That is, the brightness of the LED changes by lighting with the brightness of LO (Low), HI (High), HI, and HI in each of the four slots. As a result, the symbol "00" is transmitted.

When the average brightness in the symbol period is 25% ((e) in FIG. 173), the transmitter turns off the common switch during the period of the first slot, and during the period from the second slot to the fourth slot. , Turn on the common switch. Further, the transmitter turns off the pixel switch during the period of the first slot, the third slot and the fourth slot, and turns on the pixel switch during the period of the second slot. As a result, the LED corresponding to the common switch and the pixel switch is lit only during the period when the common switch is on and the pixel switch is on. That is, the brightness of the LED changes by lighting like LO (Low), HI (High), LO, LO in each of the four slots. As a result, the symbol "00" is transmitted. Since the transmitter in the present embodiment transmits a visible light signal close to the above-mentioned V4PPM (variable 4PPM), the average brightness can be made variable even when the same symbol is transmitted. That is, when transmitting the same symbol (for example, “00”) with different average luminances, the transmitter raises the luminance unique to the symbol (timing) as shown in FIGS. 173 (a) to (e). ) Is constant regardless of the average brightness. As a result, the receiver can receive the visible light signal without being aware of the brightness.

The common switch is switched by the above-mentioned common switch control unit, and the pixel switch is switched by the above-mentioned pixel switch control unit.

As described above, the transmission method in the present embodiment is a transmission method for transmitting a visible light signal by changing the luminance, and includes a determination step, a common switch control step, and a first pixel switch control step. In the determination step, the luminance change pattern is determined by modulating the visible light signal. In the common switch control step, the brightness of a common switch for commonly lighting a plurality of light sources (LEDs) included in a light source group (common line) provided in the display for representing pixels in the image. Switch according to the change pattern. In the first pixel switch control step, the common switch is turned on by turning on the first pixel switch for turning on the first light source among the plurality of light sources included in the light source group. , The visible light signal is transmitted by turning on the first light source only during the period when the first pixel switch is on.

This makes it possible to appropriately transmit a visible light signal from a display provided with a plurality of LEDs or the like as a light source. Therefore, it is possible to communicate between devices including devices other than lighting. Further, when the display is a display for displaying an image under the control of the common switch and the first pixel switch, the visible light signal can be transmitted by using the common switch and the first pixel switch. can. Therefore, the visible light signal can be easily transmitted without making a significant change to the configuration for displaying the image on the display.

Further, by matching the control timing of the pixel switch with the transmission symbol (for one 4PPM) and controlling as shown in FIG. 173, the visible light signal can be transmitted from the LED display without flicker. The image signal (that is, the video signal) usually changes in a cycle of 1/30 second or 1/60 second, but by changing the image signal according to the symbol transmission cycle (symbol cycle), it is realized without changing the circuit. can do.

As described above, in the above-mentioned determination step of the transmission method in the present embodiment, the luminance change pattern is determined for each symbol period. Further, in the first pixel switch control step, the pixel switch is switched in synchronization with the symbol period. As a result, even if the symbol period is, for example, 1/2400 seconds, the visible light signal can be appropriately transmitted according to the symbol period.

When the signal (symbol) is "10" and the average luminance is around 50%, the luminance change pattern is close to 0101 and the luminance rises at two points. However, in that case, the receiver can correctly receive the signal by giving priority to the later rising point. That is, the later rising point is the timing at which the rising edge of the luminance peculiar to the symbol "10" is obtained.

The higher the average luminance, the closer to the signal modulated by 4PPM can be output. Therefore, when the brightness of the entire screen or the part where the power supply line is common is low, the HI section can be lengthened and the error can be reduced by reducing the current and lowering the instantaneous value of the brightness. .. In this case, the maximum brightness of the screen will decrease, but if high brightness is not required in the first place, such as for indoor use, or if visible light communication is prioritized, you can enable communication by enabling the switch that enables this. The balance between quality and image quality can be optimally set.

Further, in the first pixel switch control step of the transmission method in the present embodiment, when displaying an image on the display (LED display), the pixel value of the pixel in the image corresponding to the first light source is expressed. Of the lighting period for lighting, the first pixel switch is switched so as to supplement the lighting period only during the period when the first light source is turned off for transmission of the visible light signal. That is, in the transmission method of the present embodiment, the visible light signal is transmitted when the image is displayed on the LED display. Therefore, during the period in which the LED should be turned on in order to express the pixel value (specifically, the luminance value) indicated by the video signal, the LED may be turned off for the transmission of the visible light signal. In such a case, in the transmission method of the present embodiment, the first pixel switch is switched so as to supplement the lighting period only during the period when the LED is turned off.

For example, when displaying the image indicated by the video signal without transmitting the visible light signal, the common switch is turned on during one symbol cycle, and the pixel switch is the average which is the pixel value indicated by the image signal. It is turned on only for the period according to the brightness. When the average brightness is 75%, the common switch is turned on in the first to fourth slots of the symbol period. Further, the pixel switch is turned on in the first slot to the third slot of the symbol cycle. As a result, during the symbol cycle, the LED lights up in the first slot to the third slot, so that the above-mentioned pixel value can be expressed. However, for the transmission of the symbol "01", the second slot is turned off. Therefore, in the transmission method of the present embodiment, the pixel switch is switched so that only the second slot in which the LED is turned off supplements the lighting period of the LED, that is, the LED is turned on in the fourth slot. ..

Further, in the transmission method in the present embodiment, the lighting period is supplemented by changing the pixel value of the pixel in the video. For example, in the above case, the pixel value with an average brightness of 75% is changed to the pixel value with an average brightness of 100%. When the average brightness is 100%, the LED tries to turn on in the first slot to the fourth slot, but the first slot is turned off for the transmission of the symbol "01". Therefore, even when the visible light signal is transmitted, the LED can be turned on with the original pixel value (average brightness 75%).

As a result, it is possible to prevent the image from being distorted due to the transmission of the visible light signal.

(Light emission control shifted for each pixel)
FIG. 174 is a diagram showing an example of a transmission signal according to the present embodiment.

When the transmitter in the present embodiment transmits the same symbol (for example, "10") from pixel A and pixels in the vicinity of the pixel A (for example, pixel B and pixel C), as shown in FIG. 174, they are transmitted. The emission timing of the pixels of is shifted. However, the transmitter causes the pixels to emit light without shifting the timing of the rise of the luminance peculiar to the symbol between the pixels. Each of the pixels A to C corresponds to a light source (specifically, an LED). Further, the timing of the rise of the luminance peculiar to the symbol is the timing of the boundary between the third slot and the fourth slot if the symbol is “10”. Further, such timing is hereinafter referred to as symbol-specific timing. By specifying the symbol-specific timing, the receiver can receive the symbol according to the timing.

By shifting the light emission timing in this way, the waveform showing the average luminance transition between pixels has a gradual rising edge or falling edge except for the rising edge at the symbol-specific timing, as shown in FIG. 174. That is, the rising edge of the symbol-specific timing is steeper than the rising edge of other timings. Therefore, the receiver can specify an appropriate symbol-specific timing by giving priority to the steepest rising edge among the plurality of rising edges, and as a result, can suppress reception errors.

That is, when the symbol "10" is transmitted from a predetermined pixel and the brightness of the predetermined pixel is an intermediate value of 25% to 75%, the transmitter opens the pixel switch corresponding to the predetermined pixel. Set the section short or long. Further, the transmitter reversely adjusts the open section of the pixel switch corresponding to the pixel in the vicinity of the predetermined pixel. In this way, it is possible to suppress an error by setting the open section of each pixel switch so that the overall brightness including the predetermined pixel and the nearby pixel does not change. The open section is a section in which the pixel switch is turned on.

As described above, the transmission method in the present embodiment further includes a second pixel switch control step. In this second pixel switch control step, by turning on the second pixel switch for turning on the second light source around the first light source included in the above-mentioned light source group (common line). , The visible light signal is transmitted by turning on the second light source only during the period when the common switch is on and the second pixel switch is on. The second light source is, for example, a light source next to the first light source.

Then, in the first and second pixel switch control steps, when the same symbol included in the visible light signal is simultaneously transmitted from each of the first and second light sources, the first and second pixel switches are used. Of the multiple timings that each is turned on or off to transmit the same symbol, the timing at which the rise in brightness unique to that same symbol is obtained is the same for each of the first and second pixel switches. The other timings are different for each of the first and second pixel switches, and the overall average brightness of the first and second light sources during the period in which the same symbol is transmitted is set to a predetermined brightness. Match.

As a result, as in the transition of the average brightness between pixels shown in FIG. 174, the rise can be steepened only at the timing when the rise of the brightness peculiar to the symbol is obtained in the spatially averaged brightness, and the reception can be performed. It is possible to suppress the occurrence of errors. That is, it is possible to suppress the reception error of the visible light signal by the receiver.

Further, when the symbol "10" is transmitted from a predetermined pixel and the brightness of the predetermined pixel is an intermediate value of 25% to 75%, the transmitter transmits the symbol "10" to the predetermined pixel in the first period. Set the open section of the corresponding pixel switch short or long. Further, the transmitter reversely adjusts the open section of the pixel switch in the first period and the second period (eg, frame) before or after the time. As described above, the error can be suppressed by setting the open section of the pixel switch so that the time average brightness of the entire time including the first period and the second period does not change in the predetermined pixel.

That is, in the above-mentioned first pixel switch control step in the transmission method in the present embodiment, for example, the first period and the second period following the first period are included in the visible light signal. Send the same symbol. At this time, in each of the first and second periods, the rise of the luminance peculiar to the same symbol among the plurality of timings in which the first pixel switch is turned on or off to transmit the same symbol is generated. Make the obtained timings the same and make the other timings different. Then, the average brightness of the first light source in the whole of the first and second periods is made to match the predetermined brightness. The first period and the second period may be a period for displaying a frame and a period for displaying the next frame, respectively. Further, the first period and the second period may each have a symbol period. That is, the first period and the second period may be a period for transmitting one symbol and a period for transmitting the next symbol, respectively.

As a result, similar to the inter-pixel average luminance transition shown in FIG. 174, in the temporally averaged luminance, the rising edge can be steepened only at the timing when the rising edge of the luminance peculiar to the symbol is obtained. , The occurrence of reception error can be suppressed. That is, it is possible to suppress the reception error of the visible light signal by the receiver.

(Light emission control when the pixel switch can be driven at double speed)
FIG. 175 is a diagram showing an example of a transmission signal according to the present embodiment.

When the pixel switch can be opened and closed at half the transmission symbol cycle, that is, when the pixel switch can be driven at double speed, the same emission pattern as V4PPM can be used as shown in FIG. 175.

In other words, when the symbol period (the period during which the symbol is transmitted) consists of 4 slots, the pixel switch control unit such as the LED driver circuit that controls the pixel switch can control the pixel switch every 2 slots. That is, the pixel switch control unit can turn on the pixel switch for an arbitrary time in the period of two slots from the first time point of the symbol cycle. Further, the pixel switch control unit can turn on the pixel switch for an arbitrary time in a period of two slots from the first time of the third slot of the symbol cycle.

That is, in the transmission method in the present embodiment, the pixel value may be changed at a cycle of 1/2 of the above-mentioned symbol cycle.

In this case, there is a possibility that the fineness of opening and closing the pixel switch will be reduced (accuracy will be reduced). Therefore, by performing this only when the transmission priority switch is enabled, the balance between image quality and transmission quality can be optimally set.

(Block of light emission control by pixel value adjustment)
FIG. 176 is a block diagram showing an example of a transmitter according to the present embodiment.

FIG. 176 (a) is a block diagram showing a configuration of a device that displays only an image without transmitting a visible light signal, that is, a display device that displays an image on the above-mentioned LED display. As shown in FIG. 176 (a), this display device includes an image / video input unit 1911, an N-speed speedup unit 1912, a common switch control unit 1913, and a pixel switch control unit 1914.

The image / video input unit 1911 outputs a video signal indicating an image or video at a frame rate of, for example, 60 Hz to the N-fold speed unit 1912.

The N times speeding unit 1912 increases the frame rate of the video signal input from the image / video input unit 1911 by N (N> 1) times, and outputs the video signal. For example, the N times speeding unit 1912 increases the frame rate by 10 times (N = 10), that is, to a frame rate of 600 Hz.

The common switch control unit 1913 switches the common switch based on the image having a frame rate of 600 Hz. Similarly, the pixel switch control unit 1914 switches the pixel switch based on the image having a frame rate of 600 Hz. In this way, by increasing the frame rate by the N times speeding unit 1912, it is possible to avoid flickering due to opening and closing of a switch such as a common switch or a pixel switch. Further, even when the LED display is imaged by the image pickup device with a high-speed shutter, the image pickup device can capture an image without missing pixels or flicker.

FIG. 176 (b) is a block diagram showing a configuration of a display device, that is, a transmitter (transmitter) that not only displays an image but also transmits the above-mentioned visible light signal. This transmitter includes an image / video input unit 1911, a common switch control unit 1913, a pixel switch control unit 1914, a signal input unit 1915, and a pixel value adjustment unit 1916. The signal input unit 1915 outputs a visible light signal composed of a plurality of symbols to the pixel value adjusting unit 1916 at a symbol rate (frequency) of 2400 symbols / second.

The pixel value adjusting unit 1916 duplicates the image input from the image / video input unit 1911 according to the symbol rate of the visible light signal, and adjusts the pixel value according to the above method. As a result, the common switch control unit 1913 and the pixel switch control unit 1914 in the subsequent stages from the pixel value adjustment unit 1916 can output a visible light signal without changing the brightness of the image or the moving image.

For example, in the case of the example shown in FIG. 176, if the symbol rate of the visible light signal is 2400 symbols / sec, the pixel value adjusting unit 1916 is included in the video signal so that the frame rate of 60 Hz of the video signal becomes 4800 Hz. Duplicate the image. For example, the value of the symbol included in the visible light signal is "00", and the pixel value (luminance value) of the pixel included in the first image before duplication is 50%. In this case, the pixel value adjusting unit 1916 adjusts the pixel value to 100% for the first image after duplication and to 50% for the second image. As a result, the luminance becomes 50% due to the AND of the common switch and the pixel switch, as in the case of the luminance change in the case of the symbol “00” shown in FIG. 175 (c). As a result, the visible light signal can be transmitted while keeping the brightness equal to that of the original image. Note that the AND of the common switch and the pixel switch means that the light source (that is, LED) corresponding to the common switch and the pixel switch is turned on only during the period when the common switch is on and the pixel switch is on. ..

Further, in the transmission method of the present embodiment, the video display and the visible light signal transmission may not be performed at the same time, but may be performed separately for the signal transmission period and the video display time.

That is, in the above-mentioned first pixel switch control step in the present embodiment, the first pixel switch is turned on during the signal transmission period in which the common switch is switching according to the luminance change pattern. Then, the transmission method in the present embodiment further turns on the common switch during the video display period different from the signal transmission period, and turns on the first pixel switch according to the video to be displayed during the video display period. Thereby, the image display step of displaying the pixels in the image by turning on the first light source only during the period when the common switch is on and the first pixel switch is on may be included. ..

As a result, since the display of the video and the transmission of the visible light signal are performed in different periods from each other, the display and transmission can be easily performed.

(Timing of power change)
When changing the power line, a signal off section will occur, but since the last part of 4PPM does not affect reception even if it does not emit light, changing the power line according to the transmission cycle of the 4PPM symbol , The power line can be changed without affecting the reception quality.

Further, by changing the power supply line during the LO period of 4PPM, the power supply line can be changed without affecting the reception quality. In this case, it is possible to further transmit while keeping the maximum luminance high.

(Drive timing)
Further, in the present embodiment, the LED display may be driven at the timings shown in FIGS. 177 to 179.

17 to 179 are timing charts when the LED display is driven by the optical ID modulation signal of the present invention.

For example, as shown in FIG. 178, when the common switch (COM1) is turned off (period t1) in order to transmit a visible light signal (optical ID), the LED is turned on with the brightness indicated by the video signal. Therefore, the LED is turned on after the period t1. As a result, it is possible to appropriately display the image while appropriately transmitting the visible light signal without destroying the image indicated by the image signal.

(summary)
FIG. 180A is a flowchart showing a transmission method according to one aspect of the present invention.

The transmission method according to one aspect of the present invention is a transmission method for transmitting a visible light signal by changing the luminance, and includes steps SC11 to SC13.

In step SC11, the luminance change pattern is determined by modulating the visible light signal in the same manner as in each of the above-described embodiments.

In step SC12, a common switch for commonly lighting a plurality of light sources included in the light source group provided in the display for representing pixels in the image is switched according to the luminance change pattern.

In step S13, the common switch is turned on by turning on the first pixel switch (that is, the pixel switch) for turning on the first light source among the plurality of light sources included in the light source group. , The visible light signal is transmitted by turning on the first light source only during the period when the first pixel switch is on.

FIG. 180B is a block diagram showing a functional configuration of a transmission device according to an aspect of the present invention.

The transmitter C10 according to one aspect of the present invention is a transmitter (or transmitter) that transmits a visible light signal by changing the luminance, and includes a determination unit C11, a common switch control unit C12, and a pixel switch control unit C13. To prepare for. The determination unit C11 determines the luminance change pattern by modulating the visible light signal in the same manner as in each of the above-described embodiments. The determination unit C11 is provided in, for example, the signal input unit 1915 shown in FIG. 176.

The common switch control unit C12 switches the common switch according to its luminance change pattern. This common switch is a switch for commonly lighting a plurality of light sources for representing pixels in an image, which are included in a group of light sources provided in the display.

The pixel switch control unit C13 turns on the common switch and turns on the pixel switch by turning on the pixel switch for turning on the light source to be controlled among the plurality of light sources included in the light source group. A visible light signal is transmitted by turning on the light source to be controlled only for a certain period of time. The light source to be controlled is the above-mentioned first light source.

This makes it possible to appropriately transmit a visible light signal from a display provided with a plurality of LEDs or the like as a light source. Therefore, it is possible to communicate between devices including devices other than lighting. Further, when the display is a display for displaying an image by controlling the common switch and the pixel switch, the visible light signal can be transmitted by using the common switch and the pixel switch. Therefore, the visible light signal can be easily transmitted without making a significant change to the configuration (that is, the display device) for displaying the image on the display.

(Frame configuration of Single frame transition)
FIG. 181 is a diagram showing an example of a transmission signal according to the present embodiment.

As shown in FIG. 181 (a), the transmission frame is composed of a preamble (PRE), an ID length (IDLEN), an ID type (IDTYPE), a content (ID / DATA), and a check code (CRC). The number of bits in each area is an example.

By using the preamble as shown in FIG. 181 (b), the receiver can be distinguished from other parts encoded by 4PPM, I-4PPM or V4PPM and can find the signal delimiter. ..

As shown in FIG. 181 (c), variable length content can be transmitted by specifying the ID / DATA length in IDLEN.

CRC is an inspection code for correcting or detecting an error in a part other than PRE. By changing the CRC length according to the length of the inspection area, the inspection ability can be maintained above a certain level. Further, by using different inspection codes according to the length of the inspection region, the inspection ability per CRC length can be improved.

(Frame structure of Multiple frame transmission)
182 and 183 are diagrams showing an example of a transmission signal according to the present embodiment.

A partition type (PTYPE) and an inspection code (CRC) are added to the transmission data (BODY) to obtain Jointed data. The Joined data is divided into several DATAPARTs, and the preamble (PRE) and the address (ADDR) are added and transmitted.

PTYPE (or partition mode (PMODE)) indicates the method or meaning of BODY division. By setting it to 2 bits as shown in FIG. 182 (a), it can be coded just right with 4PPM. As shown in FIG. 182 (b), the transmission time can be shortened by setting the number to 1 bit.

CRC is an inspection code for inspecting PTYPE and BODY. As defined in FIG. 161 by changing the code length of the CRC according to the length of the portion to be inspected, the inspection ability can be maintained above a certain level.

By defining the preamble as shown in FIG. 162, it is possible to shorten the transmission time while ensuring the variation of the division pattern.

By defining the address as shown in FIG. 163, the receiver can restore the data regardless of the order in which the frames are received.

FIG. 183 is a combination of possible joined data lengths and the number of frames. The underlined combination is the combination used when PTYPE described later is Single frame compatible.

(Structure of BODY field)
FIG. 184 is a diagram showing an example of a transmission signal according to the present embodiment.

By setting BODY as a field configuration as shown in the figure, it is possible to transmit an ID similar to that for single frame transmission.

If the same IDTYPE has the same ID, it has the same meaning regardless of whether it is single-frame transmission or multi-frame transmission, and regardless of the combination of packet transmission, so that signals can be flexibly signaled when continuous transmission / reception time is short. Can be sent.

The length of the ID is specified by IDLEN, and PADDING is transmitted to the remaining part. All of this part may be 0 or 1, data for expanding the ID may be transmitted, or a check code may be used. PADDING may be left-justified.

In (b), (c) or (d) of FIG. 184, the transmission time can be shorter than that of (a) of FIG. 184. At this time, it is assumed that the length of the ID is the largest of the lengths that can be taken as the ID.

In the case of (b) or (c) of FIG. 184, the number of bits of IDTYPE is an odd number, but by combining with the 1-bit PTYPE shown in FIG. 182 (b), it becomes an even number and can be efficiently encoded by 4PPM. Can be done.

In (c) of FIG. 184, a longer ID can be transmitted.

In FIG. 184 (d), more IDTYPEs can be represented.

(PTYPE)
FIG. 185 is a diagram showing an example of a transmission signal according to the present embodiment.

When PTYPE is a predetermined bit, it indicates that BODY is in Single frame compact mode. As a result, the same ID as in the case of single frame transmission can be transmitted.

For example, when PTYPE = 00, the ID or ID type corresponding to the PTYPE can be treated in the same manner as the ID or ID type transmitted by single frame transmission, and the management of the ID or ID type can be simplified. can.

When PTYPE is a predetermined bit, BODY indicates that it is in Data stream mode. At this time, all combinations can be used for the number of transmission frames and the DATAPART length, and the data of different combinations can have different meanings. Depending on the bit of PTYPE, the different combinations may have the same meaning or may maintain different meanings. This makes it possible to flexibly select the transmission method.

For example, when PTYPE = 01, an ID having a size not defined for single frame transmission can be transmitted. Further, even if the ID corresponding to the PTYPE is the same as the ID for the single frame transmission, the ID corresponding to the PTYPE can be treated as an ID different from the ID for the single frame transmission. As a result, the number of IDs that can be expressed can be increased.

(Field configuration of Single frame compact mode)
FIG. 186 is a diagram showing an example of a transmission signal according to the present embodiment.

When (a) of FIG. 184 is used, in the single frame compact mode, it is most efficient to transmit in the combination of the tables shown in FIG. 186.

When (b), (c) or (d) of FIG. 184 is used, when the ID is 32 bits, a combination of 13 frames and a DATAPART length of 4 bits is efficient. When the ID is 64 bits, the combination of 11 frames and 8 bits of DATAPART length is efficient.

By assuming that only the combinations in the table are transmitted, it becomes possible to determine that different combinations are reception errors, and it is possible to reduce the reception error rate.

(Summary of Embodiment 19)
The transmission method according to one aspect of the present invention is a transmission method for transmitting a visible light signal by changing the luminance, and is provided in a determination step for determining a luminance change pattern by modulating the visible light signal and a display. A common switch control step for switching a common switch for commonly lighting a plurality of light sources for representing pixels in an image, which are included in the light source group, according to the luminance change pattern, and a plurality of common switches included in the light source group. By turning on the first pixel switch for turning on the first light source among the light sources of the above, only during the period when the common switch is on and the first pixel switch is on. It includes a first pixel switch control step of transmitting the visible light signal by turning on the first light source.

Thereby, for example, as shown in FIGS. 173 to 180B, a visible light signal can be appropriately transmitted from a display provided with a plurality of LEDs or the like as a light source. Therefore, it is possible to communicate between devices including devices other than lighting. Further, when the display is a display for displaying an image under the control of the common switch and the first pixel switch, the visible light signal can be transmitted by using the common switch and the first pixel switch. can. Therefore, the visible light signal can be easily transmitted without making a significant change to the configuration for displaying the image on the display.

Further, in the determination step, the luminance change pattern may be determined for each symbol cycle, and in the first pixel switch control step, the first pixel switch may be switched in synchronization with the symbol cycle.

Thereby, for example, as shown in FIG. 173, even if the symbol period is, for example, 1/2400 seconds, the visible light signal can be appropriately transmitted according to the symbol period.

Further, in the first pixel switch control step, when displaying an image on the display, the visible period of the lighting period for expressing the pixel value of the pixel in the image corresponding to the first light source. The first pixel switch may be switched so as to supplement the lighting period only during the period when the first light source is turned off for transmission of an optical signal. For example, the lighting period may be supplemented by changing the pixel value of the pixel in the video.

As a result, as shown in FIGS. 173 and 175, even when the first light source is turned off for transmission of a visible light signal, the lighting period is supplemented, so that the original image is properly displayed without being disturbed. can do.

Further, the pixel value may be changed in a cycle of 1/2 of the symbol cycle.

Thereby, for example, as shown in FIG. 175, it is possible to appropriately display an image and transmit a visible light signal.

Further, in the transmission method, the common switch is further turned on by turning on a second pixel switch for lighting a second light source around the first light source, which is included in the light source group. The present invention includes a second pixel switch control step of transmitting the visible light signal by turning on the second light source only during the period when the second pixel switch is on and the second pixel switch is on. In the first and second pixel switch control steps, when the same symbol included in the visible light signal is simultaneously transmitted from each of the first and second light sources, the first and second pixel switches Of the plurality of timings each of which is turned on or off to transmit the same symbol, the timing at which the rise of the luminance peculiar to the same symbol is obtained is the same for each of the first and second pixel switches. The other timings are different for each of the first and second pixel switches, and the overall average brightness of the first and second light sources during the period when the same symbol is transmitted is predetermined. It may match the brightness.

As a result, as shown in FIG. 174, for example, in the spatially averaged luminance, the rising edge of the luminance peculiar to the symbol can be made steep only at the timing when the rising edge is obtained, and the occurrence of a reception error can occur. It can be suppressed.

Further, in the first pixel switch control step, when the same symbol included in the visible light signal is transmitted in the first period and the second period following the first period, the first period. And in each of the second period, among the plurality of timings in which the first pixel switch is turned on or off in order to transmit the same symbol, the timing at which the rise in brightness peculiar to the same symbol is obtained. They may be the same, with different timings, to match the average brightness of the first light source over the entire first and second periods to a predetermined brightness.

As a result, as shown in FIG. 174, for example, in the temporally averaged luminance, the rising edge of the luminance peculiar to the symbol can be made steep only at the timing when the rising edge is obtained, and the occurrence of a reception error can occur. It can be suppressed.

Further, in the first pixel switch control step, the first pixel switch is turned on during the signal transmission period in which the common switch is switching according to the brightness change pattern, and the transmission method further comprises the signal. By turning on the common switch during the video display period different from the transmission period and turning on the first pixel switch according to the video to be displayed during the video display period, the common switch is turned on and An image display step of displaying the pixels in the image may be included by turning on the first light source only during the period when the first pixel switch is on.

As a result, since the display of the video and the transmission of the visible light signal are performed in different periods from each other, the display and transmission can be easily performed.

(Embodiment 20)
In this embodiment, details or modifications of the visible light signal in each of the above embodiments will be specifically described. The trend of cameras is to increase the resolution (4K) and the frame rate (60 fps). By increasing the frame rate, the frame scan time is reduced. As a result, the reception distance decreases and the reception time increases. Therefore, in a transmitter that transmits a visible light signal, it is necessary to shorten the packet transmission time. In addition, the time resolution of reception is increased by reducing the line scan time. The exposure time is 1/8000 second. In 4PPM, since signal expression and dimming are performed at the same time, the signal density is low and the efficiency is poor. Therefore, in the visible light signal in the present embodiment, the signal portion and the dimming portion are separated, and the portion required for reception is shortened.

FIG. 187 is a diagram showing an example of the configuration of a visible light signal in the present embodiment.

As shown in FIG. 187, the visible light signal includes a plurality of combinations of a signal unit and a dimming unit. The time length of this combination is, for example, 2 ms or less (frequency is 500 Hz or more).

FIG. 188 is a diagram showing an example of a detailed configuration of a visible light signal according to the present embodiment.

The visible light signal includes a data L (DataL), a preamble (Preamble), a data R (DataR), and a dimming unit (Dimming). The signal unit is composed of the data L, the preamble, and the data R.

The preamble alternately shows the high and low luminance values along the time axis. That is, the preamble shows the high brightness value _only for the time length P1, the low brightness value for the _next time length P2 _, the high brightness value for the next time length P3, and the next time length P. Only ₄ shows the brightness value of Low. The time _lengths P1 to _P4 are, for example, 100 μs.

The data R alternately shows the luminance values of High and Low along the time axis, and is arranged immediately after the preamble. That is, the data R shows the high luminance value only for the time length D _R1 , shows the low luminance value only for the next time length D _R2 , shows the high luminance value only for the next time length D _R3 , and shows the next luminance value. Only _DR4 shows the low brightness value. The time lengths _DR1 to _DR4 are determined according to a mathematical formula according to the signal to be transmitted. This formula is D _Ri = 120 + 20x _i (i ∈ 1 to 4, x _i ∈ 0 to 15). Numerical values such as 120 and 20 indicate time (μs). Moreover, these numerical values are an example.

The data L alternately shows the luminance values of High and Low along the time axis, and is arranged immediately before the preamble. That is, the data _L shows the high luminance value only for the time length _DL1 , the low luminance value only for the next time length DL2, shows the high luminance value only for the next time length _DL3 , and shows the next luminance value. Only _DL4 shows the low brightness value. The time lengths _DL1 to _DL4 are determined according to a mathematical formula according to the signal to be transmitted. This formula is D _Li = 120 + 20 × (15−x _i ). As in the above, numerical values such as 120 and 20 indicate time (μs). Moreover, these numerical values are an example.

The signal to be transmitted is composed of 4 × 4 = 16 bits, and _xi is a signal of 4 bits among the signals to be transmitted. In the visible light signal, the numerical value of _xi (4-bit signal) is indicated by each of the time lengths _DR1 to _DR4 in the data _R or the time lengths DL1 to _DL4 in the data L. Of the 16 bits of the signal to be transmitted, 4 bits indicate an address, 8 bits indicate data, and 4 bits are used for error detection.

Here, the data R and the data L have a complementary relationship with respect to the brightness. That is, if the brightness of the data R is bright, the brightness of the data L is dark, and conversely, if the brightness of the data R is dark, the brightness of the data L is bright. That is, the sum of the total time length of the data R and the time length of the data L is constant regardless of the signal to be transmitted.

The dimming unit is a signal for adjusting the brightness (luminance) of the visible light signal, and shows a high brightness value only for a time length C ₁ and a Low signal for the next time length C ₂ . The time lengths C ₁ and C ₂ are arbitrarily adjusted. The dimming unit may or may not be included in the visible light signal.

Further, in the example shown in FIG. 188, the data R and the data L are included in the visible light signal, but only one of the data R and the data L may be included. When it is desired to brighten the visible light signal, only the bright data of the data R and the data L may be transmitted. Further, the arrangement of the data R and the data L may be reversed. Further, when the data R is included, the time length C ₁ of the dimming unit is longer than, for example, 100 μs, and when the data L is included, the time length C ₂ of the dimming unit is, for example, 100 μs. Longer than.

FIG. 189A is a diagram showing another example of the visible light signal in the present embodiment.

In the visible light signal shown in FIG. 188, the signal to be transmitted is represented by each of the time length indicating the luminance value of High and the time length indicating the luminance value of Low. As shown in FIG. 189A (a). , The signal to be transmitted may be expressed only by the time length indicating the luminance value of Low. Note that FIG. 189A (b) shows the visible light signal of FIG. 188.

For example, as shown in FIG. 189A (a), in the preamble, the time lengths indicating the high luminance values are all equal and relatively short, and the time _lengths P1 to P4 indicating the low luminance values are, for example, 100 μs, _respectively . Is. Further, in the data _R , the time lengths indicating the high luminance values are all equal and relatively short, and the time lengths DR1 to _DR4 indicating the low luminance values are adjusted according to the signals x _i , respectively. In the preamble and the data R, the time length indicating the high luminance value is, for example, 10 μs or less.

FIG. 189B is a diagram showing another example of the visible light signal in the present embodiment.

For example, as shown in FIG. _189B , in the preamble, the time lengths indicating the high luminance values are all equal and relatively short, and the time _lengths P1 to P3 indicating the low luminance values are, for example, 160 μs, 180 μs, and 160 μs, respectively. Is. Further, in the data _R , the time lengths indicating the high luminance values are all equal and relatively short, and the time lengths DR1 to _DR4 indicating the low luminance values are adjusted according to the signals x _i , respectively. In the preamble and the data R, the time length indicating the high luminance value is, for example, 10 μs or less.

FIG. 189C is a diagram showing the signal length of the visible light signal in the present embodiment.

FIG. 190 is a diagram showing a comparison result of the luminance values of the visible light signal in the present embodiment and the visible light signal of the standard IEC (International Electrotechnical Commission). The standard IEC is specifically "VISIBLE LIGHT BEACON SYSTEM FOR MULTIMEDIA APPLICATIONS".

In the visible light signal in the present embodiment (method of the embodiment (Data one side)), a maximum brightness of 82% higher than the maximum brightness of the visible light signal of the standard IEC can be obtained, and the visible light signal of the standard IEC can be obtained. A minimum brightness of 18%, which is lower than the minimum brightness, can be obtained. The maximum luminance 82% and the minimum luminance 18% are numerical values obtained by a visible light signal including only one of data R and data L in the present embodiment.

FIG. 191 is a diagram showing a comparison result of the number of received packets and the reliability with respect to the angle of view between the visible light signal in the present embodiment and the visible light signal of the standard IEC.

In the visible light signal in the present embodiment (method of the embodiment (bottom)), even if the angle of view is small, that is, even if the distance from the transmitter that transmits the visible light signal to the receiver is long, The number of received packets is larger than that of the visible light signal of the standard IEC, and high reliability can be obtained. The numerical value of the method (both) of the embodiment shown in FIG. 191 is a numerical value obtained by a visible light signal including both data R and data L.

FIG. 192 is a diagram showing a comparison result of the number of received packets and the reliability with respect to noise between the visible light signal in the present embodiment and the visible light signal of the standard IEC.

In the visible light signal (IEEE) in the present embodiment, the number of received packets is larger than that of the visible light signal of the standard IEC, regardless of the noise (dispersion value of noise), and high reliability can be obtained.

FIG. 193 is a diagram showing a comparison result of the number of received packets and the reliability with respect to the clock error on the receiving side between the visible light signal in the present embodiment and the visible light signal of the standard IEC.

In the visible light signal (IEEE) in the present embodiment, the number of received packets is larger than that of the visible light signal of the standard IEC in a wide range of the clock error on the receiving side, and high reliability can be obtained. The clock error on the receiving side is an error in the timing at which the exposure line in the image sensor of the receiver starts exposure.

FIG. 194 is a diagram showing a configuration of a signal to be transmitted in the present embodiment.

The signal to be transmitted includes four 4-bit signals ( _xi ) (4 × 4 = 16 bits) as described above. For example, the signal to be transmitted includes signals x ₁ to x ₄ . The signal x ₁ is composed of bits x ₁₁ to x ₁₄ , and the signal x ₂ is composed of bits x ₂₁ to x ₂₄ . Further, the signal x ₃ is composed of bits x ₃₁ to x ₃₄ , and the signal x ₄ is composed of bits x ₄₁ to x ₄₄ . Here, bit x ₁₁ , bit x ₂₁ , bit x ₃₁ and bit x ₄₁ are easy to make a mistake, and other bits are hard to make a mistake. Therefore, the bits x ₄₂ to x ₄₄ included in the signal x ₄ are used for the parity of the bit x ₁₁ of the signal x ₁ , the bit x ₂₁ of the signal x ₂ , and the bit x ₃₁ of the signal x ₃ , respectively, and the signal x ₄ Bit x ₄₁ contained in is not used and always indicates 0. The mathematical formula shown in FIG. 194 is used for the calculation of each of the bits x ₄₂ , x ₄₃ , and x ₄₄ . Bits x ₄₂ , x ₄₃ , and x ₄₄ are calculated by this formula, such as bit x ₄₂ = bit x ₁₁ , bit x ₄₃ = bit x ₂₁ , bit x ₄₄ = bit x ₃₁ .

FIG. 195A is a diagram showing a method of receiving a visible light signal according to the present embodiment.

The receiver sequentially acquires the signal unit of the above-mentioned visible light signal. The signal unit includes a 4-bit address (Addr) and 8-bit data (Data). The receiver combines the data of each of the signal units to generate an ID composed of a plurality of data and a parity composed of one or a plurality of data.

FIG. 195B is a diagram showing the rearrangement of visible light signals in the present embodiment.

FIG. 196 is a diagram showing another example of the visible light signal in the present embodiment.

The visible light signal shown in FIG. 196 is configured by superimposing a high frequency signal on the visible light signal shown in FIG. 188. The frequency of the high frequency signal is, for example, 1 to several Gbps. As a result, data can be transmitted at a higher speed than the visible light signal shown in FIG. 188.

FIG. 197 is a diagram showing another example of the detailed configuration of the visible light signal in the present embodiment. The configuration of the visible light signal shown in FIG. 197 is the same as the configuration shown in FIG. 188, but the time lengths C1 and C2 of the dimming unit in the visible light signal shown in FIG. 197 are the time length C1 shown in FIG. 188. And different from C2.

FIG. 198 is a diagram showing another example of the detailed configuration of the visible light signal in the present embodiment. In the visible light signal shown in FIG. 198, the data R and the data L contain eight symbols of V4PPM. The rising position or falling position of the symbol D _Li included in the data L is the same as the rising position or falling position of the symbol D _Ri included in the data R. However, the average luminance of the symbol D _Li and the average luminance of the symbol D _Ri may be the same or different.

FIG. 199 is a diagram showing another example of the detailed configuration of the visible light signal in the present embodiment. The visible light signal shown in FIG. 199 is a signal for ID communication or low average luminance application, and is the same as the visible light signal shown in FIG. 189B.

FIG. 200 is a diagram showing another example of the detailed configuration of the visible light signal in the present embodiment. In the visible light signal shown in FIG. 200, the even-numbered time length D _2i and the odd-numbered time length D _{2i + 1} in the data (Data) are equal to each other.

FIG. 201 is a diagram showing another example of the detailed configuration of the visible light signal in the present embodiment. The data (Data) in the visible light signal shown in FIG. 201 includes a plurality of symbols which are signals of pulse position modulation.

FIG. 202 is a diagram showing another example of the detailed configuration of the visible light signal in the present embodiment. The visible light signal shown in FIG. 202 is a signal for continuous communication and is the same as the visible light signal shown in FIG. 198.

203 to 211 are diagrams for explaining a method of determining the values of x1 to x4 of FIG. 197. Note that x1 to x4 shown in FIGS. 203 to 211 are determined by the same method as the method for determining the values (W1 to W4) of the reference numerals w1 to w4 shown in the following modification. However, each of x1 to x4 is a code consisting of 4 bits, and is different from the codes w1 to w4 shown in the following modification in that the first bit includes parity.

(Modification 1)
FIG. 212 is a diagram showing an example of a detailed configuration of a visible light signal according to a modification 1 of the present embodiment. The visible light signal according to the first modification is the same as the visible light signal shown in FIG. 188 of the above embodiment, but the time length indicating the brightness value of High or Low is different from the visible light signal shown in FIG. 188. There is. For example, in the visible light signal according to this modification, the preamble _time lengths P2 and P3 _are 90 μs. Further, in the visible light signal according to the present modification, the time lengths DR1 to _DR4 in the data _R are determined according to a mathematical formula according to the signal to be transmitted, as in the above embodiment. However, the mathematical formula in this modification is D _Ri = 120 + 30 × wi (i ∈ 1 to 4, wi ∈ 0 to 7). Note that wi is a code consisting of 3 bits and is a signal to be transmitted indicating an integer value of any of 0 to 7. Further, in the visible light signal according to the present modification, the time lengths DL1 to _DL4 in the data _L are determined according to a mathematical formula according to the signal to be transmitted, as in the above embodiment. However, the mathematical formula in this modification is D _Li = 120 + 30 × (7-wi).

Further, in the example shown in FIG. 212, the data R and the data L are included in the visible light signal, but only one of the data R and the data L may be included in the visible light signal. When it is desired to brighten the visible light signal, only the bright data of the data R and the data L may be transmitted. Further, the arrangement of the data R and the data L may be reversed.

FIG. 213 is a diagram showing another example of the visible light signal according to this modification.

As the visible light signal according to the first modification, the signal to be transmitted may be represented only by the time length indicating the luminance value of Low, as in the example shown in FIG. 189A (a) and FIG. 189B.

For example, as shown in FIG. 213, in the preamble, the time length indicating the high luminance value is, for example, less than 10 μs, and the time _lengths P1 to P3 indicating the low luminance value are, for example, 160 μs, 180 μs, and 160 μs _, respectively. Is. Further, in the data (Data), the time length indicating the high luminance value is less than 10 μs, and the time lengths D ₁ to D ₃ indicating the low luminance value are adjusted according to the signal wi, respectively. Specifically, the time length Di indicating the luminance value of Low is Di _{= 180 + 30 × wi (i ∈} 1 to 4, wi ∈ 0 to 7).

FIG. 214 is a diagram showing still another example of the visible light signal according to this modification.

The visible light signal according to this modification may include a preamble and data as shown in FIG. 214. Similar to the preamble shown in FIG. 212, the preamble alternately shows the brightness values of High and Low along the time axis. Further, the time _lengths P1 to P4 in the preamble are 50 μs, ₄₀ μs, 40 μs, and 50 μs, respectively. The data (Data) alternately show the high and low luminance values along the time axis. For example, the data L shows the high luminance value only for the time length D ₁ , shows the low luminance value only for the next time length D ₂ , shows the high luminance value only for the next time length D ₃ , and shows the next luminance value. Only D ₄ shows the low brightness value.

Here, the time length D _2i-1 + D _2i is determined according to a mathematical formula corresponding to the signal to be transmitted. That is, the sum of the time length indicating the high luminance value and the time length indicating the low luminance value following the luminance value is determined according to the mathematical formula. This formula is, for example, D _2i-1 + D _2i = 100 + 20 × x _i (i ∈ 1 to N, x _i ∈ 0 to 7, D _2i > 50 μs, D _{2i + 1} > 50 μs).

FIG. 215 is a diagram showing an example of packet modulation.

The signal generation device generates a visible light signal by the method of generating a visible light signal according to this modification. In the visible light signal generation method according to this modification, the packet is modulated (that is, converted) into the above-mentioned signal wi to be transmitted. The above-mentioned signal generation device may or may not be provided in the transmitter in each of the above-described embodiments.

For example, the signal generator converts the packet into a signal of interest including the numerical values indicated by the symbols w1, w2, w3 and w4, respectively, as shown in FIG. 215. These codes w1, w2, w3 and w4 are codes consisting of 3 bits from the first bit to the third bit, respectively, and indicate integer values from 0 to 7 as shown in FIG. 212.

Here, in each of the reference numerals w1 to w4, the value of the first bit is b1, the value of the second bit is b2, and the value of the third bit is b3. Note that b1, b2 and b3 are 0 or 1. In this case, each of the numerical values W1 to W4 indicated by the reference numerals w1 to w4 is, for example, b1 × 2 ⁰ + b2 × 2 ¹ + b3 × 2 ² .

The packet includes address data (A1 to A4) consisting of 0 to 4 bits, main data Da (Da1 to Da7) consisting of 4 to 7 bits, and sub data Db (Db1 to Db4) consisting of 3 to 4 bits. The value (S) of the stop bit is included as data. It should be noted that each of Da1 to Da7, A1 to A4, Db1 to Db4, and S indicates a bit value, that is, 0 or 1.

That is, when the signal generator modulates a packet into a signal to be transmitted, the signal generator allocates the data contained in the packet to any bit of the reference numerals w1, w2, w3, and w4. As a result, the packet is converted into a signal to be transmitted including the numerical values indicated by the symbols w1, w2, w3, and w4, respectively.

When the signal generator allocates the data contained in the packet, specifically, at least one of the main data Da included in the packet is set in the first bit string composed of the first bits (bit1) of the reference numerals w1 to w4. Units (Da1 to Da4) are assigned. Further, the signal generator assigns the value (S) of the stop bit included in the packet to the second bit (bit2) of the reference numeral w1. Further, the signal generator is included in a part of the main data Da (Da5 to Da7) included in the packet or in the packet in the second bit string composed of the second bits (bit2) of the reference numerals w2 to w4. Allocate at least a part (A1 to A3) of the address data. Further, the signal generator includes at least a part (Db1 to Db3) of the sub data Db included in the packet and the sub data Db thereof in the third bit string composed of the third bits (bit3) of the reference numerals w1 to w4. (Db4) or a part of the address data (A4) is assigned.

When the third bits (bit3) of the reference numerals w1 to w4 are all 0, the numerical values indicated by those codes are 3 according to the above-mentioned "b1 × 2 ⁰ + b2 × 2 ¹ + b3 × 2 ² ". It can be suppressed to the following. Therefore, the time length D _Ri can be shortened by the mathematical formula D _Ri = 120 + 30 × wi ( _i ∈ 1 to 4, wi ∈ ₀ to 7) shown in FIG. 212. As a result, the time for transmitting one packet can be shortened, and the packet can be received even from a greater distance.

FIGS. 216 to 226 are diagrams showing a process of generating a packet from the original data.

The signal generation device according to this modification determines whether or not to divide the original data according to the bit length of the original data. Then, the signal generation device generates at least one packet from the original data by performing processing according to the result of the determination. That is, the signal generator divides the original data into many packets as the bit length of the original data becomes longer. On the contrary, if the bit length of the original data is shorter than the predetermined bit length, the signal generator generates a packet without dividing the original data.

When the signal generation device generates at least one packet from the original data in this way, each of the at least one packet is converted into the above-mentioned signal to be transmitted, that is, the codes w1 to w4.

In FIGS. 216 to 226, Data indicates the original data, Dataa indicates the main element data included in the original data, and Data indicates the sub-element data included in the original data. Further, Da (k) indicates the k-th part of the main element data itself or a plurality of parts constituting the data including the main element data and the parity. Similarly, Db (k) indicates the sub-source data itself or the k-th portion of a plurality of parts constituting the data including the sub-source data and parity. For example, Da (2) indicates the second part of a plurality of parts constituting the data including the main element data and the parity. Further, S indicates a start bit, and A indicates address data.

The uppermost notation shown in each block is a label for identifying the original data, the main original data, the secondary original data, the start bit, the address data, and the like. The numerical value in the center shown in each block is the bit size (number of bits), and the numerical value in the lowermost row is the value of each bit.

FIG. 216 is a diagram showing a process of dividing the original data into one.

For example, if the bit length of the original data (Data) is 7 bits, the signal generator generates one packet without dividing the original data. Specifically, the original data includes the 4-bit main source data Dataa (Da1 to Da4) and the 3-bit sub-source data Dataab (Db1 to Db3), respectively, as the main data Da (1) and the sub-data Db (1). Included as 1). In this case, the signal generator adds a packet to the original data by adding the start bit S (S = 1) and the address data (A1 to A4) consisting of 4 bits and indicating "0000" to the original data. Generate. Note that the start bit S = 1 indicates that the packet including the start bit is the terminal packet.

By converting this packet, the signal generator has the reference numerals w1 = (Da1, S = 1, Db1), the reference numerals w2 = (Da2, A1 = 0, Db2), and the reference numerals w3 = (Da3, A2 = 0, Db3). ), And the reference numeral w4 = (Da4, A3 = 0, A4 = 0). Further, the signal generator generates a signal to be transmitted including the numerical values W1, W2, W3 and W4 indicated by the symbols w1, w2, w3 and w4, respectively.

In this modification, wi is expressed as a 3-bit code and also as a decimal value. Therefore, in this modification, in order to make the explanation easy to understand, wi (w1 to w4) used as a decimal value is expressed as a numerical value Wi (W1 to W4).

FIG. 217 is a diagram showing a process of dividing the original data into two.

For example, if the bit length of the original data (Data) is 16 bits, the signal generator generates two intermediate data by dividing the original data. Specifically, the original data includes 10-bit main original data Data and 6-bit sub-source data Data. In this case, the signal generator generates the first intermediate data including the main source data Dataa and the 1-bit parity corresponding to the main element data Dataa, and corresponds to the secondary source data Datab and the secondary source data Datab. The second intermediate data including the 1-bit parity to be generated is generated.

Next, the signal generator divides the first intermediate data into a main data Da (1) consisting of 7 bits and a main data Da (2) consisting of 4 bits. Further, the signal generator divides the second intermediate data into sub data Db (1) consisting of 4 bits and sub data Db (2) consisting of 3 bits. The main data is one part of a plurality of parts constituting the data including the main source data and the parity. Similarly, the sub-data is one of a plurality of parts constituting the data including the sub-source data and the parity.

Next, the signal generator generates a 12-bit first packet including the start bit S (S = 0), the main data Da (1), and the sub data Db (1). As a result, the first packet containing no address data is generated.

Further, the signal generator is composed of a start bit S (S = 1), an address data indicating "1000" consisting of 4 bits, a main data Da (2), and a 12-bit data Db (2). Generate a second packet. Note that the start bit S = 0 indicates that, among the plurality of generated packets, the packet including the start bit is not at the end. Further, the start bit S = 1 indicates that, among the plurality of generated packets, the packet including the start bit is the packet at the end.

As a result, the original data is divided into a first packet and a second packet.

By converting the first packet, the signal generator has reference numerals w1 = (Da1, S = 0, Db1), reference numerals w2 = (Da2, Da7, Db2), reference numerals w3 = (Da3, Da6, Db3), and The reference numeral w4 = (Da4, Da5, Db4) is generated. Further, the signal generator generates a signal to be transmitted including the numerical values W1, W2, W3 and W4 indicated by the symbols w1, w2, w3 and w4, respectively.

Further, the signal generator converts the second packet into reference numerals w1 = (Da1, S = 1, Db1), reference numeral w2 = (Da2, A1 = 1, Db2), reference numeral w3 = (Da3, A2 =). 0, Db3), and the sign w4 = (Da4, A3 = 0, A4 = 0) are generated. Further, the signal generator generates a signal to be transmitted including the numerical values W1, W2, W3 and W4 indicated by the symbols w1, w2, w3 and w4, respectively.

FIG. 218 is a diagram showing a process of dividing the original data into three parts.

For example, if the bit length of the original data (Data) is 17 bits, the signal generator generates two intermediate data by dividing the original data. Specifically, the original data includes 10-bit main original data Data and 7-bit sub-source data Data. In this case, the signal generator generates the first intermediate data including the main element data Dataa and the 6-bit parity corresponding to the original element data Dataa. Further, the signal generator generates the second intermediate data including the sub-source data Data and the 4-bit parity corresponding to the sub-source data Data. For example, the signal generator generates parity by CRC (Cyclic Redundancy Check).

Next, the signal generator uses the first intermediate data as the main data Da (1) consisting of 6-bit parity, the main data Da (2) consisting of 6 bits, and the main data Da (3) consisting of 4 bits. Divide into and. Further, the signal generator uses the second intermediate data as sub-data Db (1) consisting of 4-bit parity, sub-data Db (2) consisting of 4-bit, and sub-data Db (3) consisting of 3-bit. Divide into.

Next, the signal generator has 12 bits including a start bit S (S = 0), an address data consisting of 1 bit indicating "0", a main data Da (1), and a sub data Db (1). Generates the first packet of. Further, the signal generator is a 12-bit signal generation device including a start bit S (S = 0), an address data consisting of 1 bit indicating “1”, a main data Da (2), and a sub data Db (2). Generate a second packet. Further, the signal generator is composed of a start bit S (S = 1), an address data indicating "0100" consisting of 4 bits, a main data Da (3), and a 12-bit data Db (3). Generate a third packet.

As a result, the original data is divided into a first packet, a second packet, and a third packet.

By converting the first packet, the signal generator has reference numeral w1 = (Da1, S = 0, Db1), reference numeral w2 = (Da2, A1 = 0, Db2), reference numeral w3 = (Da3, Da6, Db3). , And the reference numeral w4 = (Da4, Da5, Db4). Further, the signal generator generates a signal to be transmitted including the numerical values W1, W2, W3 and W4 indicated by the symbols w1, w2, w3 and w4, respectively.

Similarly, the signal generator converts the second packet into reference numerals w1 = (Da1, S = 0, Db1), reference numeral w2 = (Da2, A1 = 1, Db2), reference numeral w3 = (Da3, Da6). , Db3), and the reference numeral w4 = (Da4, Da5, Db4). Further, the signal generator generates a signal to be transmitted including the numerical values W1, W2, W3 and W4 indicated by the symbols w1, w2, w3 and w4, respectively.

Similarly, the signal generator converts the third packet into reference numerals w1 = (Da1, S = 1, Db1), reference numerals w2 = (Da2, A1 = 0, Db2), and reference numeral w3 = (Da3, A2). = 1, Db3), and the reference numeral w4 = (Da4, A3 = 0, A4 = 0) are generated. Further, the signal generator generates a signal to be transmitted including the numerical values W1, W2, W3 and W4 indicated by the symbols w1, w2, w3 and w4, respectively.

FIG. 219 is a diagram showing another example of the process of dividing the original data into three parts.

In the example shown in FIG. 218, CRC generates 6-bit or 4-bit parity, but 1-bit parity may be generated.

In this case, if the bit length of the original data (Data) is 25 bits, the signal generator generates two intermediate data by dividing the original data. Specifically, the original data includes a 15-bit main original data Dataa and a 10-bit sub-source data Datab. In this case, the signal generator generates the first intermediate data including the main source data Dataa and the 1-bit parity corresponding to the main element data Dataa, and corresponds to the secondary source data Datab and the secondary source data Datab. The second intermediate data including the 1-bit parity to be generated is generated.

Next, the signal generator uses the first intermediate data as the main data Da (1) consisting of 6 bits including parity, the main data Da (2) consisting of 6 bits, and the main data Da (3) consisting of 4 bits. ) And. Further, the signal generator uses the second intermediate data as the sub data Db (1) consisting of 4 bits including parity, the sub data Db (2) consisting of 4 bits, and the sub data Db (3) consisting of 3 bits. Divide into and.

Next, the signal generator generates a first packet, a second packet, and a third packet from the first intermediate data and the second intermediate data, as in the example shown in FIG. 218.

FIG. 220 is a diagram showing another example of the process of dividing the original data into three parts.

In the example shown in FIG. 218, the CRC for the main element data Dataa generates a 6-bit parity, and the CRC for the secondary source data Dataa generates a 4-bit parity. However, parity may be generated by CRC for the entire primary data Data and secondary data Data.

In this case, if the bit length of the original data (Data) is 22 bits, the signal generator generates two intermediate data by dividing the original data.

Specifically, the original data includes a 15-bit main original data Data and a 7-bit sub-source data Datab. The signal generator generates the first intermediate data including the main element data Dataa and the 1-bit parity corresponding to the original element data Dataa. In addition, the signal generator generates its 4-bit parity by CRC for the entire primary data Data and secondary data Data. Then, the signal generator generates the second intermediate data including the sub-source data Datab and the 4-bit parity.

Next, the signal generator uses the first intermediate data as the main data Da (1) consisting of 6 bits including parity, the main data Da (2) consisting of 6 bits, and the main data Da (3) consisting of 4 bits. ) And. Further, the signal generator uses the second intermediate data as the sub data Db (1) consisting of 4 bits, the sub data Db (2) consisting of 4 bits including a part of the parity of the CRC, and the remainder of the parity of the CRC. It is divided into sub data Db (3) consisting of 3 bits including.

Next, the signal generator generates a first packet, a second packet, and a third packet from the first intermediate data and the second intermediate data, as in the example shown in FIG. 218.

Of the specific examples of the process of dividing the original data into three, the process shown in FIG. 218 is referred to as version 1, the process shown in FIG. 219 is referred to as version 2, and the process shown in FIG. 220 is referred to as version 3.

FIG. 221 is a diagram showing a process of dividing the original data into four parts. Further, FIG. 222 is a diagram showing a process of dividing the original data into five parts.

The signal generator divides the original data into four or five parts in the same manner as the process of dividing the original data into three, that is, the same as the process shown in FIGS. 218 to 220.

FIG. 223 is a diagram showing a process of dividing the original data into 6, 7 or 8 divisions.

For example, if the bit length of the original data (Data) is 31 bits, the signal generator generates two intermediate data by dividing the original data. Specifically, the original data includes 16-bit main original data Data and 15-bit secondary original data Data. In this case, the signal generator generates the first intermediate data including the main element data Dataa and the 8-bit parity corresponding to the original element data Dataa. Further, the signal generator generates the second intermediate data including the sub-source data Data and the 8-bit parity corresponding to the sub-source data Data. For example, a signal generator generates parity by a Reed-Solomon code.

Here, when 4 bits are treated as one symbol in the Reed-Solomon code, the bit lengths of the main element data Dataa and the secondary element data Datab must be an integral multiple of 4 bits. However, the secondary source data Datab is 15 bits as described above, which is one bit less than 16 bits, which is an integral multiple of 4 bits.

Therefore, when the signal generator generates the second intermediate data, it pads the sub-source data Data and reads the 8-bit parity corresponding to the 16-bit sub-source data Data on which the padding is performed. Generated by the Solomon code.

Next, the signal generator divides each of the first intermediate data and the second intermediate data into six parts (4 bits or 3 bits) by the same method as described above. Then, the signal generator generates the first packet including the start bit, the address data consisting of 3 bits or 4 bits, the first main data, and the first sub data. Similarly, the signal generator generates the second packet to the sixth packet.

FIG. 224 is a diagram showing another example of the process of dividing the original data into 6, 7 or 8 divisions.

In the example shown in FIG. 223, the parity is generated by the Reed-Solomon code, but the parity may be generated by the CRC.

For example, if the bit length of the original data (Data) is 39 bits, the signal generator generates two intermediate data by dividing the original data. Specifically, the original data includes 20-bit main original data Data and 19-bit sub-source data Data. In this case, the signal generator generates the first intermediate data including the main source data Dataa and the 4-bit parity corresponding to the main element data Dataa, and corresponds to the secondary source data Datab and the secondary source data Datab. The second intermediate data including the 4-bit parity is generated. For example, a signal generator generates parity by CRC.

Next, the signal generator divides each of the first intermediate data and the second intermediate data into six parts (4 bits or 3 bits) by the same method as described above. Then, the signal generator generates the first packet including the start bit, the address data consisting of 3 bits or 4 bits, the first main data, and the first sub data. Similarly, the signal generator generates the second packet to the sixth packet.

Of the specific examples of the process of dividing the original data into 6, 7 or 8, the process shown in FIG. 223 is referred to as version 1, and the process shown in FIG. 224 is referred to as version 2.

FIG. 225 is a diagram showing a process of dividing the original data into nine parts.

For example, if the bit length of the original data (Data) is 55 bits, the signal generator generates nine packets from the first packet to the ninth packet by dividing the original data. In FIG. 225, the first intermediate data and the second intermediate data are omitted.

Specifically, the bit length of the original data (Data) is 55 bits, which is one bit less than 56 bits, which is an integral multiple of 4 bits. Therefore, the signal generator performs padding on the original data, and generates parity (16 bits) on the original data consisting of 56 bits on which the padding is performed by a Reed-Solomon code.

Next, the signal generator divides the entire data including the 16-bit parity and the 55-bit original data into nine data DaDb (1) to DaDb (9).

Each of the data DaDb (k) includes a portion consisting of the k-th 4-bit included in the main source data Dataa and a portion consisting of the k-th 4-bit included in the sub-source data Dataa. In addition, k is an integer of any of 1 to 8. Further, the data DaDb (9) includes a portion consisting of the 9th 4 bits included in the main source data Dataa and a portion consisting of the 9th 3 bits included in the secondary source data Dataa.

Next, the signal generator generates the first packet to the ninth packet by adding the start bit S and the address data to each of the nine data DaDb (1) to DaDb (9).

FIG. 226 is a diagram showing a process of dividing the original data into any number of 10 to 16.

For example, if the bit length of the original data (Data) is 7 × (N-2) bits, the signal generator divides the original data into N packets from the first packet to the Nth packet. To generate. In addition, N is an integer of any of 10 to 16. Further, in FIG. 226, the first intermediate data and the second intermediate data are omitted.

Specifically, the signal generator generates parity (14 bits) for the original data consisting of 7 × (N-2) bits by a Reed-Solomon code. In this Reed-Solomon code, 7 bits are treated as one symbol.

Next, the signal generator divides the entire data including the 14-bit parity and the 7 × (N-2) bit original data into N data DaDb (1) to DaDb (N).

Each of the data DaDb (k) includes a portion consisting of the k-th 4 bits included in the main source data Dataa and a portion consisting of the k-th 3 bits included in the sub-source data Dataa. In addition, k is an integer of any one of 1 to (N-1).

Next, the signal generator generates the first packet to the Nth packet by adding the start bit S and the address data to each of the nine data DaDb (1) to DaDb (N).

227 to 229 are diagrams showing an example of the relationship between the number of divisions of the original data, the data size, and the error correction code.

Specifically, FIGS. 227 to 229 collectively show the above-mentioned relationships in each process shown in FIGS. 216 to 226. Further, as described above, the process of dividing the original data into three is divided into versions 1 to 3, and the process of dividing the original data into 6, 7 or 8 is divided into version 1 and version 2. FIG. 227 shows the above relationship in version 1 of the plurality of versions, if there are a plurality of versions with respect to the number of divisions. Similarly, FIG. 228 shows the above relationship in version 2 of the plurality of versions, if there are a plurality of versions for the number of divisions. Similarly, FIG. 229 shows the above relationship in version 3 of the plurality of versions, if there are a plurality of versions for the number of divisions.

Further, in this modification, there are a short mode and a full mode. In the sheet mode, the sub data in the packet is 0, and all the bits in the third bit string shown in FIG. 215 are 0. In this case, the numerical values W1 to W4 indicated by the symbols w1 to w4 are suppressed to 3 or less by the above-mentioned "b1 × 2 ⁰ + b2 × 2 ¹ + b3 × 2 ² ". As a result, as shown in FIG. 212, the time lengths DR1 to DR4 in the data _R are determined by DRi = 120 + 30 × wi ( _i ∈ 1 to ₄ , wi ∈ 0 to 7), and thus are shortened. That is, in the case of the short mode, the visible light signal per packet can be shortened. By shortening the visible light signal per packet, the receiver can receive the packet even from a distance, and the communication distance can be lengthened.

On the other hand, in the full mode, any bit of the third bit string shown in FIG. 215 is 1. In this case, the visible light signal is not as short as in the short mode.

In this modification, as shown in FIGS. 227 to 229, a short mode visible light signal can be generated if the number of divisions is small. The short mode data size in FIGS. 227 to 229 indicates the number of bits of the main data (Data), and the full mode data size indicates the number of bits of the original data (Data).

(Summary of Embodiment 20)
FIG. 230A is a flowchart showing a method of generating a visible light signal according to the present embodiment.

The method for generating a visible light signal in the present embodiment is a method for generating a visible light signal transmitted by a change in the luminance of a light source included in the transmitter, and includes steps SD1 to SD3.

In step SD1, each of the first and second luminance values, which are different luminance values, generate a preamble which is data that appears alternately along the time axis for a predetermined time length.

In step SD2, in the data in which the first and second luminance values appear alternately along the time axis, the time length in which each of the first and second luminance values continues is determined according to the signal to be transmitted. The first data is generated by determining according to the first method.

Finally, in step SD3, a visible light signal is generated by combining the preamble and the first data.

For example, as shown in FIG. 188, the first and second luminance values are High and Low, and the first data is data R or data L. By transmitting the visible light signal generated in this way, as shown in FIGS. 191 to 193, the number of received packets can be increased and the reliability can be increased. As a result, communication between various devices can be enabled.

Further, in the method of generating the visible light signal, further, in the data in which the first and second brightness values appear alternately along the time axis, the time during which each of the first and second brightness values continues. By determining the length according to the second method according to the signal to be transmitted, the second data having a complementary relationship with the brightness expressed by the first data is generated, and the visible light signal is generated. Then, the visible light signal may be generated by combining the preamble with the first and second data in the order of the first data, the preamble, and the second data.

For example, as shown in FIG. 188, the first and second luminance values are High and Low, and the first and second data are data R and data L.

Further, when a and b are constants, the numerical value included in the signal to be transmitted is n, and the constant which is the maximum value that the numerical value n can take is m, the first method is based on a + b × n. , The method for determining the length of time for the first or second luminance value to continue in the first data, and the second method is the second data by a + b × (mn). In, the method may be used to determine the length of time that the first or second luminance value continues.

For example, as shown in FIG. 188, a is 120 μs, b is 20 μs, n is an integer value from 0 to 15 (the numerical value indicated by the signal x _i ), and m is 15. ..

Further, in the complementary relationship, the sum of the time length of the first data as a whole and the time length of the second data as a whole may be constant.

Further, the method of generating the visible light signal further generates a dimming unit which is data for adjusting the brightness expressed by the visible light signal, and further, in the generation of the visible light signal, the dimming is further performed. The visible light signal may be generated by combining the portions.

The dimming unit is, for example, a signal (Dimming) in FIG. 188 showing a high luminance value only for a time length C ₁ and a low luminance value only for a time length C ₂ . This makes it possible to arbitrarily adjust the brightness of the visible light signal.

FIG. 230B is a block diagram showing the configuration of the signal generator according to the present embodiment.

The signal generation device D10 in the present embodiment is a signal generation device that generates a visible light signal transmitted by a change in the luminance of a light source included in the transmitter, and is a preamble generation unit D11, a data generation unit D12, and a coupling unit. It is equipped with D13.

The preamble generation unit D11 generates a preamble which is data in which the first and second luminance values, which are different luminance values, appear alternately along the time axis for a predetermined time length.

In the data in which the first and second luminance values appear alternately along the time axis, the data generation unit D12 determines the duration of each of the first and second luminance values according to the signal to be transmitted. The first data is generated by determining according to the first method.

The coupling unit D13 generates a visible light signal by coupling the preamble and the first data.

By transmitting the visible light signal generated in this way, as shown in FIGS. 191 to 193, the number of received packets can be increased and the reliability can be increased. As a result, communication between various devices can be enabled.

(Summary of Modification 1 of Embodiment 20)
Further, as in the first modification of the twenty embodiment, the visible light signal generation method further determines whether or not to divide the original data according to the bit length of the original data, and the result of the determination. At least one packet may be generated from the original data by performing the processing according to the above. Then, each of the at least one packet may be converted into a signal to be transmitted.

In the conversion to the signal to be transmitted, as shown in FIG. 215, for each target packet included in at least one packet, the data contained in the target packet is set to 3 from the first bit to the third bit, respectively. By allocating to any bit of the code w1, w2, w3, and w4 consisting of bits, the target packet is converted into a signal to be transmitted including the numerical values indicated by the codes w1, w2, w3, and w4, respectively.

In the data allocation, at least a part of the main data included in the target packet is allocated to the first bit string consisting of the first bits of the codes w1 to w4. The value of the stop bit included in the target packet is assigned to the second bit of the code w1. A part of the main data included in the target packet or at least a part of the address data included in the target packet is assigned to the second bit string consisting of the second bits of the codes w2 to w4, of the reference numerals w1 to w4. Sub-data included in the target packet is assigned to the third bit string consisting of each third bit.

Here, the stop bit indicates whether or not the target packet is at the end of at least one generated packet. The address data indicates the order of the target packets among at least one generated packet as an address. Each of the main data and the sub data is data for restoring the original data.

Further, when a and b are constants and the numerical values indicated by the reference numerals w1, w2, w3 and w4 are W1, W2, W3 and W4, for example, as shown in FIG. 212, the above-mentioned first method is used. Is a method of determining the duration of the first or second luminance value in the first data by a + b × W1, a + b × W2, a + b × W3, and a + b × W4.

For example, in each of the reference numerals w1 to w4, the value of the first bit is b1, the value of the second bit is b2, and the value of the third bit is b3. In this case, each of the values W1 to W4 indicated by the reference numerals w1 to w4 is, for example, b1 × 2 ⁰ + b2 × 2 ¹ + b3 × 2 ² . Therefore, in the codes w1 to w4, the values W1 to W4 indicated by the codes w1 to w4 are larger when the second bit is set to 1 than when the first bit is set to 1. Further, the values W1 to W4 indicated by the codes w1 to w4 are larger when the third bit is set to 1 than when the second bit is set to 1. When the values W1 to W4 indicated by these reference numerals w1 to w4 are large, the duration of each of the above-mentioned first and second luminance values (for example, _DRi ) becomes longer, so that the luminance of the visible light signal becomes longer. False positives can be suppressed and reception errors can be reduced. On the contrary, when the values W1 to W4 indicated by these reference numerals w1 to w4 are small, the duration of each of the above-mentioned first and second luminance values is shortened, so that the luminance of the visible light signal is erroneously detected. Is relatively easy to occur.

Therefore, in the first modification of the twenty embodiment, the stop bit and the address, which are important for receiving the original data, are preferentially assigned to the second bits of the codes w1 to w4 to reduce the reception error. Can be planned. Further, the reference numeral w1 defines the length of time during which the High or Low luminance value closest to the preamble continues. That is, since the code w1 is closer to the preamble than the other codes w2 to w4, it is more likely to be properly received than these other codes. Therefore, in the first modification of the twenty embodiment, the reduction of the reception error can be further suppressed by allocating the stop bit to the second bit of the reference numeral w1.

Further, in the first modification of the twenty embodiment, the main data is preferentially assigned to the first bit string in which false positives are relatively likely to occur. However, if an error correction code (parity) is inserted in the main data, the reception error of the main data can be suppressed.

Further, in the first modification of the twenty embodiment, the sub data is assigned to the third bit string composed of the third bits of the reference numerals w1 to w4. Therefore, if the sub data is set to 0, the duration of each of the high and low luminance values defined by the reference numerals w1 to w4 can be significantly shortened. As a result, it is possible to realize a so-called short mode in which the transmission time of the visible light signal per packet can be significantly shortened. In this short mode, since the transmission time is short as described above, the packet can be easily received even from a distance. Therefore, the communication distance of visible light communication can be lengthened.

Further, in the first modification of the twenty embodiment, as shown in FIG. 217, in the generation of at least one packet, two packets are generated by dividing the original data into two packets, and in the data allocation, the original data is generated. When converting a packet that is not at the end of two packets into a signal to be transmitted as a target packet, the second bit string is mainly included in the packet that is not at the end without allocating at least a part of the address data. Allocate a part of the data.

For example, the non-terminating packet (Packet1) shown in FIG. 217 does not contain address data. Then, the packet not at the end has 7 bits of the main data Da (1). Therefore, as shown in FIG. 215, the data Da1 to Da4 included in the 7-bit main data Da (1) are assigned to the first bit string, and the data Da5 to Da7 are assigned to the second bit string.

In this way, when the original data is divided into two packets, if the packet not at the end, that is, the first packet has a start bit (S = 0), the address data is unnecessary. Therefore, all the bits of the second bit string can be used for the main data, and the amount of data contained in the packet can be increased.

Further, in the data allocation in the first modification of the twenty embodiment, among the three bits included in the second bit string, the first bit in the arrangement order is preferentially used for the address data allocation, and the second bit is used. When all of the address data is allocated to the first one or two bits of the bit string, the main data is assigned to the one or two bits to which the address data is not assigned in the second bit string. Allocate a part of. For example, in Packet 1 in FIG. 218, one bit of address data A1 is assigned to one bit (the second bit of reference numeral w2) on the head side of the second bit string. In this case, the main data Da6 and Da5 are assigned to the two bits (the second bits of the reference numerals w3 and w4, respectively) to which the address data is not assigned in the second bit string.

As a result, the second bit string can be shared by the address data and a part of the main data, and the degree of freedom in packet configuration can be increased.

Further, in the data allocation in the first modification of the twenty embodiment, if it is not possible to allocate all of the address data to the second bit string, the address data is assigned to any bit of the third bit string. Allocate the rest except the part allocated to the second bit string. For example, in Packet 3 in FIG. 218, all of the 4-bit address data A1 to A4 cannot be assigned to the second bit string. In this case, the remaining portion A4 excluding the portions A1 to A3 assigned to the second bit string of the address data A1 to A4 in the last bit (third bit of the reference numeral w4) in the third bit string. To assign.

Thereby, the address data can be appropriately assigned to the reference numerals w1 to w4.

Further, in the data allocation in the first modification of the twenty embodiment, when the terminal packet of at least one packet is converted into the transmission target signal as the target packet, the second bit string and the third bit string are used. Address data is assigned to any one bit included in. For example, the number of bits of the address data of the terminal packet in FIGS. 217 to 226 is 4. In this case, 4-bit address data A1 to A4 are assigned to the second bit string and the last bit of the third bit string (third bit of reference numeral w4).

Thereby, the address data can be appropriately assigned to the reference numerals w1 to w4.

Further, in the generation of at least one packet in the first modification of the twenty embodiment, the two division source data are generated by dividing the original data into two, and the error of each of the two division source data is obtained. Generate a correction code. Then, two or more packets are generated by using the two division source data and the error correction code generated for each of the two division source data. In the generation of the error correction code for each of the two division source data, when the number of bits of the division source data of either of the two division source data is less than the number of bits required to generate the error correction code. Is padded with respect to the divided source data, and an error correction code of the padded divided source data is generated. For example, as shown in FIG. 223, when parity is generated by a Reed-Solomon code for Datab which is a division source data, if the Datab has only 15 bits and less than 16 bits, the Datab. Is padded, and parity is generated for the padded division source data (16 bits) by a Reed-Solomon code.

As a result, an appropriate error correction code can be generated even if the number of bits of the division source data is less than the number of bits required for generating the error correction code.

Further, in the data allocation in the first modification of the twenty embodiment, when the sub data indicates 0, 0 is assigned to all the bits included in the third bit string. As a result, the above-mentioned short mode can be realized, and the communication distance of visible light communication can be lengthened.

(Embodiment 21)
FIG. 231 is a diagram showing a method of receiving a high frequency visible light signal according to the present embodiment.

When the receiver receives the high-frequency visible light signal, for example, as shown in FIG. 231 (a), the receiver provides a guard time (guard interval) at the rising and falling edges of the visible light signal. Then, the receiver supplements the high frequency signal in the guard time by copying the high frequency signal received immediately before the guard time without using the high frequency signal in the guard time. The high frequency signal superimposed on the visible light signal may be modulated by OFDM (Orthogonal Frequency Division Multiplexing).

Further, when the receiver separates the high frequency signal indicating the high brightness value and the high frequency signal indicating the low brightness value from the high frequency visible light signal, the receiver automatically adjusts the gain of those high frequency signals (Automatic Gain Control). As a result, the gain (luminance value) of the high frequency signal is unified.

FIG. 232A is a diagram showing another method of receiving a high frequency visible light signal according to the present embodiment.

The receiver that receives the high-frequency visible light signal includes an image sensor as in each of the above embodiments, and further includes a DMD (Digital Mirror Device) element and a photo sensor. Photosensors are photodiodes or avalanche photodiodes.

The receiver captures a transmitter (light source) that transmits a high-frequency visible light signal by an image sensor. As a result, the receiver acquires the emission line image including the stripe pattern of the emission line. The striped pattern of the emission line appears by a signal other than the high frequency signal in the high frequency visible light signal, that is, a change in the luminance of the visible light signal shown in FIG. 188. The receiver identifies the positions (x1, y1) and (x2, y2) of the striped pattern of the emission lines in the emission line image. Then, the receiver identifies the micromirror corresponding to each of the positions (x1, y1) and (x2, y2) in the DMD element. These micromirrors receive the light of a high frequency visible light signal that reveals a striped pattern of emission lines. Therefore, the receiver adjusts the angle of each micromirror so that only the light reflected by the specified micromirror among the plurality of micromirrors included in the DMD element is received by the photo sensor. That is, the receiver turns on the micromirror so that only the light reflected by the micromirror corresponding to the position (x1, y1) is received by the photo sensor 1. Further, the receiver turns on the micromirror so that only the light reflected by the micromirror corresponding to the position (x2, y2) is received by the photo sensor 2. Then, the receiver turns off each micromirror other than those specified micromirrors. The light reflected by the micromirror turned off by this is absorbed by the light absorber (black body). Further, the high-frequency visible light signal is appropriately received by the photosensor by the turned-on micromirror. The tilt angle (+ 0 ° or −0 °) of each micromirror of the DMD element can be switched by switching between ON and OFF. When the micromirror is ON, the micromirror outputs the reflected light toward the photo sensor, and when the micromirror is OFF, the micromirror outputs the reflected light toward the light absorbing unit.

Further, the receiver may be provided with a half mirror and a light emitting element as shown in FIG. 232A. The light emitting element 1 transmits a visible light signal (or a high-frequency visible light signal) by emitting light and changing the brightness. The light output from the light emitting element 1 is reflected by the half mirror, and further reflected by the ON micromirror corresponding to the position (x1, y1) in the DMD element. As a result, the visible light signal from the light emitting element 1 is transmitted to the transmitter corresponding to the striped pattern of the emission line at the position (x1, y1). As a result, bidirectional communication can be performed between the receiver and the transmitter corresponding to the striped pattern of the emission line at the position (x1, y1). Similarly, the light output from the light emitting element 2 is reflected by the half mirror, and further reflected by the ON micromirror corresponding to the position (x2, y2) in the DMD element. As a result, the visible light signal from the light emitting element 2 is transmitted to the transmitter corresponding to the striped pattern of the emission line at the position (x2, y2). As a result, the receiver and the transmitter corresponding to the striped pattern of the emission line at the position (x2, y2) can perform bidirectional communication.

As a result, even if there are a plurality of transmitters (light sources) photographed by the image sensor, the receiver can perform two-way communication at high speed at the same time as these transmitters. For example, when the receiver is provided with 100 photosensors capable of receiving at 10 Gbps and the receiver communicates with 100 transmitters, a communication speed of 1 Tbps can be realized.

FIG. 232B is a diagram showing still another method of receiving the high frequency visible light signal according to the present embodiment.

The receiver includes, for example, lenses L1 and L2, a plurality of half mirrors, a DMD element, an image sensor, a light absorption unit (blackbody), a processing unit, a DMD control unit, and photosensors 1 and 2. , The light emitting elements 1 and 2.

Such a receiver communicates bidirectionally with two vehicles on the same principle as the example shown in FIG. 232A. The two cars transmit a high-frequency visible light signal by outputting light from the headlights and changing the brightness of the headlights. In addition, one car outputs normal light (light whose brightness does not change) from the headlights.

The image sensor receives their high frequency visible light signal and normal light through the lens L1. As a result, similar to the example shown in FIG. 232A, a bright line image including a striped pattern of bright lines generated by those high-frequency visible light signals can be obtained. The processing unit identifies the position of those striped patterns in the emission line image. The DMD control unit identifies the micromirrors corresponding to the positions of the specified striped patterns from the plurality of micromirrors included in the DMD element, and turns on those micromirrors.

As a result, the high-frequency visible light signal transmitted from each of the two cars through the lens L1 and the half mirror is reflected by the micromirror of the DMD element and directed toward the lens L2. Further, since the normal light of the headlight of one car does not generate a striped pattern of emission lines due to the light, even if it passes through the lens L1 and the half mirror, it is reflected by the OFF micromirror of the DMD element. The light reflected by the OFF micromirror is absorbed by the light absorption unit (blackbody).

The high-frequency visible light signal that has passed through the lens L2 passes through the half mirror and is received by the photo sensor 1 or 2. This makes it possible to receive high-frequency visible light signals from each vehicle. Further, if the light emitting elements 1 and 2 output a visible light signal (or a high-frequency visible light signal) to the half mirror, the visible light signal is reflected by the half mirror, passes through the lens L2, and is further DMD. It is reflected by the ON micromirror in the element. As a result, the visible light signal from the light emitting elements 1 and 2 is transmitted to the vehicle that has transmitted the high frequency visible light signal via the half mirror and the lens L1. That is, the receiver can perform bidirectional communication with a plurality of vehicles that transmit high-frequency visible light signals.

As described above, the receiver in the present embodiment acquires the emission line image by the image sensor and identifies the position of the emission line striped pattern in the emission line image. Then, the receiver identifies the micromirror corresponding to the position of the striped pattern among the plurality of micromirrors included in the DMD element. Then, the receiver receives the high-frequency visible light signal by the photo sensor by turning on the micromirror. Further, the receiver can transmit the visible light signal to the transmitter by outputting a visible light signal from the light emitting element and reflecting it on the turned-on micromirror.

In the examples shown in FIGS. 232A and 232B, a half mirror and a lens are used as the optical equipment, but any optical equipment may be used as long as it has the same functions as these. Further, the arrangement of the DMD element, the half mirror, the lens and the like is an example, and may be arranged in any way. Further, in the example shown in FIGS. 232A and 232B, the receiver includes two sets of the photo sensor and the light emitting element, but may include only one set or three or more sets. Further, one light emitting element may transmit a visible light signal to a plurality of ON microlenses. As a result, the receiver can simultaneously transmit the same visible light signal to a plurality of transmitters. Further, the receiver may not include all the components shown in FIGS. 232A and 232B, but may include only a part of those components.

FIG. 233 is a diagram showing a method of outputting a high frequency signal according to the present embodiment.

A signal output device that outputs a high-frequency signal superimposed on the visible light signal shown in FIG. 188 includes, for example, a blue laser and a phosphor. That is, as in the example shown in FIG. 114A, the signal output device irradiates the phosphor with high frequency blue laser light from the blue laser. As a result, the signal output device outputs high-frequency natural light as a high-frequency signal.

(Embodiment 22)
In this embodiment, an autonomous flight device (also referred to as a drone) using visible light communication in each of the above embodiments will be described.

FIG. 234 is a diagram for explaining an autonomous flight device according to the present embodiment.

The autonomous flight device 1921 in the present embodiment is housed inside the surveillance camera 1922. For example, when the surveillance camera 1922 captures an image of a suspicious person, the door of the surveillance camera 1922 opens, and the autonomous flight device 1921 housed inside takes off from the surveillance camera 1922 to track the suspicious person. Start. The autonomous flight device 1921 is equipped with a small camera, and tracks an image of a suspicious person captured by the surveillance camera 1922 so that the image of the suspicious person is also captured by the small camera. Further, when the autonomous flight device 1921 detects that the electric power for performing flight or the like is insufficient, the autonomous flight device 1921 returns to the surveillance camera 1922 and is housed inside the surveillance camera 1922. At this time, if another autonomous flight device 1921 is housed in the surveillance camera 1922, the other autonomous flight device 1921 starts tracking the suspicious person in place of the power-deficient autonomous flight device 1921. Further, the power-deficient autonomous flight device 1921 is powered by the wireless power feeding device 1921a provided in the surveillance camera 1922. The power supply by the wireless power supply device 1921a is performed according to, for example, the standard Qi.

The small camera and the surveillance camera 1922 of the autonomous flight device 1921 can receive the visible light signal in each of the above-described embodiments, and can perform an operation according to the received visible light signal. Further, if at least one of the autonomous flight device 1921 and the surveillance camera 1922 is provided with a visible light signal transmitter, visible light communication can be performed between the autonomous flight device 1921 and the surveillance camera 1922. As a result, suspicious persons can be tracked more efficiently.

(Embodiment 23)
In this embodiment, a display method for realizing AR (Augmented Reality) using an optical ID will be described.

FIG. 235 is a diagram showing an example in which the receiver in the present embodiment displays an AR image.

The receiver 200 in the present embodiment is a receiver provided with an image sensor and a display 201 in any one of the above embodiments 1 to 22, and is configured as, for example, a smartphone. Such a receiver 200 acquires the above-mentioned captured display image Pa, which is the normal captured image, and the above-mentioned image for decoding, which is the visible light communication image or the emission line image, by capturing the subject by the image sensor.

Specifically, the image sensor of the receiver 200 captures an image of the transmitter 100 configured as a station name sign. The transmitter 100 is a transmitter according to any one of the above embodiments 1 to 22, and includes one or a plurality of light emitting elements (for example, LEDs). The transmitter 100 changes its brightness by blinking one or more light emitting elements, and transmits an optical ID (light identification information) according to the change in brightness. This optical ID is the above-mentioned visible light signal.

The receiver 200 acquires an image capture display image Pa on which the transmitter 100 is projected by taking an image of the transmitter 100 at a normal exposure time, and at the same time, the transmitter 100 has a communication exposure time shorter than the normal exposure time. An image for decoding is acquired by taking an image of. The normal exposure time is the exposure time in the above-mentioned normal shooting mode, and the communication exposure time is the exposure time in the above-mentioned visible light communication mode.

The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P1 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes a region corresponding to the recognition information in the captured display image Pa as a target region. For example, the receiver 200 recognizes the area where the station name mark, which is the transmitter 100, is projected as the target area. Then, the receiver 200 superimposes the AR image P1 on the target area, and displays the captured display image Pa on which the AR image P1 is superposed on the display 201. For example, when "Kyoto Station" is described in Japanese as the station name on the station name mark of the transmitter 100, the receiver 200 is described as AR image P1 in which the station name is described in English, that is, "Kyoto Station". The AR image P1 that has been created is acquired. In this case, since the AR image P1 is superimposed on the target area of the captured display image Pa, the captured display image Pa can be displayed so that the station name mark on which the station name is written in English actually exists. As a result, a user who can understand English can easily understand the station name written on the station name sign of the transmitter 100 by looking at the captured image Pa even if he / she cannot read Japanese.

For example, the recognition information may be an image to be recognized (for example, an image of the above-mentioned station name sign), or may be a feature point and a feature amount of the image. The feature points and feature quantities are obtained by, for example, SIFT (Scale-invariant feature transition), SURF (Speed-Upped Robot Faceture), ORB (Oriented-BRIEF), AKAZE (Accelerated KAZ), and the like. Alternatively, the recognition information may be an image of a white square similar to the image to be recognized, and may further indicate the aspect ratio of the square. Alternatively, the identification information may be random dots appearing in the image to be recognized. Further, the recognition information may indicate an orientation with respect to a predetermined direction, such as the above-mentioned white rectangle or random dot. The predetermined direction is, for example, the direction of gravity.

The receiver 200 recognizes a region corresponding to such recognition information as a target region from the captured display image Pa. Specifically, if the recognition information is an image, the receiver 200 recognizes a region similar to the image, which is the recognition information, as a target region. Further, if the recognition information is a feature point and a feature amount obtained by image processing, the receiver 200 performs the feature point detection and feature amount extraction by performing the image processing on the captured display image Pa. .. Then, the receiver 200 recognizes the region having the feature points and the feature amount similar to the feature points and the feature amounts as the recognition information as the target area in the captured display image Pa. Further, if the recognition information indicates a white quadrangle and its direction, the receiver 200 first detects the direction of gravity by the acceleration sensor provided therein. Then, the receiver 200 recognizes a region similar to a white quadrangle oriented in the direction indicated by the recognition information as a target region from the captured display image Pa arranged with respect to the direction of gravity.

Here, the recognition information may include reference information for specifying a reference region in the captured display image Pa and target information indicating a relative position of the target region with respect to the reference region. The reference information is an image to be recognized, a feature point and a feature amount, a white square image, a random dot, or the like as described above. In this case, when the receiver 200 recognizes the target area, the receiver 200 first identifies the reference area from the captured display image Pa based on the reference information. Then, the receiver 200 recognizes the region of the captured display image Pa at the relative position indicated by the target information with respect to the position of the reference region as the target region. The target information may indicate that the target area is at the same position as the reference area. In this way, by including the reference information and the target information in the recognition information, the target area can be recognized in a wide range. Further, the server can freely set the place where the AR image is superimposed and teach the receiver 200.

Further, the reference information may indicate that the reference region in the captured display image Pa is a region in which the display is projected in the captured display image. In this case, if the transmitter 100 is configured as a display such as a television, the target area can be recognized with reference to the area on which the display is displayed.

In other words, the receiver 200 in the present embodiment specifies a reference image and an image recognition method based on the optical ID. The image recognition method is a method of recognizing the captured display image Pa, for example, geometric feature amount extraction, spectral feature amount extraction, texture feature amount extraction, and the like. The reference image is data showing a reference feature amount. The feature amount may be, for example, the feature amount of the white outer frame of the image, and specifically, may be data in which the feature of the image is expressed by a vector. The receiver 200 extracts a feature amount from the captured display image Pa according to an image recognition method, and compares the feature amount with the feature amount of the reference image to compare the feature amount with the feature amount of the reference image to obtain the above-mentioned reference region or target area from the captured display image Pa. Find out.

Further, the image recognition method may include, for example, a location utilization method, a marker utilization method, and a markerless method. The location utilization method is a method utilizing the position information of GPS (that is, the position of the receiver 200), and the target area is recognized from the captured display image Pa based on the position information. The marker utilization method is a method of using a marker composed of white and black figures such as a two-dimensional bar code as a mark for target identification. That is, in this marker utilization method, the target area is recognized based on the marker displayed on the captured display image Pa. In the markerless method, feature points or feature quantities are extracted from the captured and displayed image Pa by image analysis on the captured and displayed image Pa, and the position and region of the target are specified based on the extracted feature points or feature quantities. The method. That is, when the image recognition method is a markerless method, the image recognition method is the above-mentioned geometric feature amount extraction, spectral feature amount extraction, texture feature amount extraction, or the like.

Such a receiver 200 receives an optical ID from the transmitter 100, and obtains a reference image and an image recognition method associated with the optical ID (hereinafter referred to as a received optical ID) from a server to obtain the reference image. Images and image recognition methods may be specified. That is, a plurality of sets including the reference image and the image recognition method are stored in the server, and each of the plurality of sets is associated with a different optical ID. Thereby, one set associated with the received optical ID can be specified from the plurality of sets stored in the server. Therefore, the speed of image processing for superimposing AR images can be improved. Further, the receiver 200 may acquire a reference image or the like associated with the received optical ID by inquiring to the server, and the received optical ID may be obtained from a plurality of reference images held in advance by the receiver 200. The reference image associated with may be acquired.

Further, the server may hold the relative position information associated with the optical ID for each optical ID together with the reference image, the image recognition method, and the AR image. The relative position information is, for example, information indicating the relative positional relationship between the above-mentioned reference area and the target area. As a result, when the receiver 200 transmits the received optical ID to the server and inquires about it, the receiver 200 acquires the reference image, the image recognition method, the AR image, and the relative position information associated with the received optical ID. In this case, the receiver 200 specifies the above-mentioned reference region from the captured display image Pa based on the reference image and the image recognition method. Then, the receiver 200 recognizes a region in the direction and distance indicated by the above-mentioned relative position information from the position of the reference region as the above-mentioned target region, and superimposes the AR image on the target region. Further, the receiver 200 may recognize the above-mentioned reference area as a target area and superimpose the AR image on the reference area if there is no relative position information. That is, instead of acquiring the relative position information, the receiver 200 may hold a program for displaying the AR image based on the reference image in advance, and display the AR image in the white frame which is the reference area, for example. .. In this case, relative position information is unnecessary.

There are the following four variations (1) to (4) in holding or acquiring the reference image, the relative position information, the AR image, and the image recognition method.

(1) The server holds a plurality of sets including a reference image, relative position information, an AR image, and an image recognition method. The receiver 200 acquires one set associated with the received optical ID from those sets.

(2) The server holds a plurality of sets consisting of a reference image and an AR image. The receiver 200 uses predetermined relative position information and an image recognition method, and acquires one set associated with the received optical ID from those sets. Alternatively, the receiver 200 may hold a plurality of sets consisting of relative position information and an image recognition method in advance, and select one set associated with the received optical ID from the plurality of sets. In this case, the receiver 200 may transmit the received optical ID to the server to inquire, and acquire the relative position information corresponding to the received optical ID and the information for specifying the image recognition method from the server. Then, the receiver 200 selects one set based on the information acquired from the server from the plurality of sets each of the relative position information and the image recognition method held in advance. Alternatively, the receiver 200 selects one set associated with the received optical ID from a plurality of sets each consisting of relative position information and an image recognition method, which are held in advance without inquiring about the server. You may.

(3) The receiver 200 holds a plurality of sets including a reference image, relative position information, an AR image, and an image recognition method, and selects one set from those sets. Similar to the above (2), the receiver 200 may select one set by inquiring to the server, or may select one set associated with the receiver optical ID.

(4) The receiver 200 holds a plurality of sets consisting of a reference image and an AR image, and selects one set associated with the received optical ID. The receiver 200 is. A predetermined image recognition method and relative position information are used.

FIG. 236 is a diagram showing an example of a display system according to the present embodiment.

The display system in the present embodiment includes, for example, a transmitter 100, which is the above-mentioned station name sign, a receiver 200, and a server 300.

The receiver 200 first receives an optical ID from the transmitter 100 in order to display the captured display image on which the AR image is superimposed as described above. Next, the receiver 200 transmits the optical ID to the server 300.

The server 300 holds an AR image and recognition information associated with the optical ID for each optical ID. Therefore, when the server 300 receives the optical ID from the receiver 200, the server 300 selects the AR image and recognition information associated with the received optical ID, and transfers the selected AR image and recognition information to the receiver 200. Send. As a result, the receiver 200 receives the AR image and the recognition information transmitted from the server 300, and displays the captured display image on which the AR image is superimposed.

FIG. 237 is a diagram showing another example of the display system according to the present embodiment.

The display system in the present embodiment includes, for example, the transmitter 100, which is the above-mentioned station name mark, the receiver 200, the first server 301, and the second server 302.

The receiver 200 first receives an optical ID from the transmitter 100 in order to display the captured display image on which the AR image is superimposed as described above. Next, the receiver 200 transmits the optical ID to the first server 301.

When the first server 301 receives the optical ID from the receiver 200, the first server 301 notifies the receiver 200 of the URL (Uniform Resource Locator) and the key associated with the received optical ID. Upon receiving such a notification, the receiver 200 accesses the second server 302 based on the URL and hands the Key to the second server 302.

The second server 302 holds an AR image and recognition information associated with each key for each key. Therefore, when the second server 302 receives the key from the receiver 200, it selects the AR image and recognition information associated with the key, and transmits the selected AR image and recognition information to the receiver 200. .. As a result, the receiver 200 receives the AR image and the recognition information transmitted from the second server 302, and displays the captured display image on which the AR image is superimposed.

FIG. 238 is a diagram showing another example of the display system according to the present embodiment.

The display system in the present embodiment includes, for example, the transmitter 100, which is the above-mentioned station name mark, the receiver 200, the first server 301, and the second server 302.

The receiver 200 first receives an optical ID from the transmitter 100 in order to display the captured display image on which the AR image is superimposed as described above. Next, the receiver 200 transmits the optical ID to the first server 301.

When the first server 301 receives the optical ID from the receiver 200, the first server 301 notifies the second server 302 of the key associated with the received optical ID.

The second server 302 holds an AR image and recognition information associated with each key for each key. Therefore, when the second server 302 receives the key from the first server 301, the second server 302 selects the AR image and the recognition information associated with the key, and the selected AR image and the recognition information are used in the first. Send to server 301. When the first server 301 receives the AR image and the recognition information from the second server 302, the first server 301 transmits the AR image and the recognition information to the receiver 200. As a result, the receiver 200 receives the AR image and the recognition information transmitted from the first server 301, and displays the captured display image on which the AR image is superimposed.

In the above example, the second server 302 transmits the AR image and the recognition information to the first server 301, but the AR image and the recognition information may be transmitted to the receiver 200 without being transmitted to the first server 301. ..

FIG. 239 is a flowchart showing an example of the processing operation of the receiver 200 in the present embodiment.

First, the receiver 200 starts imaging with the above-mentioned normal exposure time and communication exposure time (step S101). Then, the receiver 200 acquires the optical ID by decoding the decoding image obtained by imaging at the communication exposure time (step S102). Next, the receiver 200 transmits the optical ID to the server (step S103).

The receiver 200 acquires the AR image corresponding to the transmitted optical ID and the recognition information from the server (step S104). Next, the receiver 200 recognizes the region corresponding to the recognition information in the captured display image obtained by the imaging of the normal exposure time as the target region (step S105). Then, the receiver 200 superimposes the AR image on the target area and displays the captured display image on which the AR image is superposed (step S106).

Next, the receiver 200 determines whether or not to end the imaging and the display of the captured image display image (step S107). Here, when the receiver 200 determines that it should not be terminated (N in step S107), it further determines whether or not the acceleration of the receiver 200 is equal to or greater than the threshold value (step S108). This acceleration is measured by an acceleration sensor provided in the receiver 200. When the receiver 200 determines that the acceleration is less than the threshold value (N in step S108), the receiver 200 executes the process from step S105. As a result, even if the captured display image displayed on the display 201 of the receiver 200 shifts, the AR image can be made to follow the target area of the captured display image. Further, when the receiver 200 determines that the acceleration is equal to or higher than the threshold value (Y in step S108), the receiver 200 executes the process from step S102. As a result, when the transmitter 100 does not appear in the captured display image, it is possible to prevent the region in which a subject different from the transmitter 100 is projected from being mistakenly recognized as the target region.

As described above, in the present embodiment, since the AR image is superimposed on the captured display image and displayed, it is possible to display an image useful to the user. Further, it is possible to suppress the processing load and superimpose the AR image on an appropriate target area.

That is, in general augmented reality (that is, AR), by comparing a huge number of recognition target images stored in advance with the captured display image, the captured display image includes any of the recognition target images. It is determined whether or not it is. Then, if it is determined that the recognition target image is included, the AR image corresponding to the recognition target image is superimposed on the captured display image. At this time, the AR image is aligned with respect to the recognition target image. As described above, in general augmented reality, since a huge number of recognition target images are compared with the captured display image, and further, position detection of the recognition target image in the captured display image is required for alignment. There is a problem that the amount of calculation is large and the processing load is high.

However, in the display method according to the present embodiment, the optical ID is acquired by decoding the decoding image obtained by imaging the subject. That is, the optical ID transmitted from the transmitter that is the subject is received. Further, the AR image corresponding to this optical ID and the recognition information are acquired from the server. Therefore, the server does not need to compare a huge number of recognition target images with the captured display image, and can select an AR image previously associated with the optical ID and transmit it to the display device. As a result, the amount of calculation can be reduced and the processing load can be significantly reduced. Further, the display process of the AR image can be speeded up.

Further, in the present embodiment, the recognition information corresponding to this optical ID is acquired from the server. The recognition information is information for recognizing a target area, which is an area on which an AR image is superimposed in a captured display image. This recognition information may be, for example, information indicating that the white quadrangle is the target area. In this case, the target area can be easily recognized, and the processing load can be further suppressed. That is, the processing load can be further suppressed according to the content of the recognition information. Further, since the content of the recognition information can be arbitrarily set in the server according to the optical ID, the balance between the processing load and the recognition accuracy can be appropriately maintained.

In the present embodiment, after the receiver 200 transmits the optical ID to the server, the receiver 200 acquires the AR image and the recognition information corresponding to the optical ID from the server, but among the AR images and the recognition information. At least one of the above may be acquired in advance. That is, the receiver 200 collectively acquires and stores a plurality of AR images and a plurality of recognition information corresponding to a plurality of optical IDs that may be received from the server. After that, when the receiver 200 receives the optical ID, it selects the AR image and the recognition information corresponding to the optical ID from the plurality of AR images and the plurality of recognition information stored in the receiver 200. This makes it possible to further speed up the display process of the AR image.

FIG. 240 is a diagram showing another example in which the receiver 200 in the present embodiment displays an AR image.

As shown in FIG. 240, for example, the transmitter 100 is configured as a lighting device, and transmits an optical ID by changing the brightness while illuminating the guide plate 101 of the facility. Since the guide plate 101 is illuminated by the light from the transmitter 100, the luminance changes like the transmitter 100, and the optical ID is transmitted.

The receiver 200 acquires the captured display image Pb and the decoding image in the same manner as described above by imaging the guide plate 101 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the guide plate 101. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P2 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes a region of the captured display image Pb corresponding to the recognition information as a target region. For example, the receiver 200 recognizes the area where the frame 102 on the guide plate 101 is projected as the target area. This frame 102 is a frame for showing the waiting time of the facility. Then, the receiver 200 superimposes the AR image P2 on the target area, and displays the captured display image Pb on which the AR image P2 is superposed on the display 201. For example, the AR image P2 is an image including the character string "30 minutes". In this case, since the AR image P2 is superimposed on the target area of the captured display image Pb, the receiver 200 receives the captured display image so that the guide plate 101 on which the waiting time “30 minutes” is described actually exists. Pb can be displayed. As a result, the user of the receiver 200 can be notified of the waiting time easily and in an easy-to-understand manner without providing the guide plate 101 with a special display device.

FIG. 241 is a diagram showing another example in which the receiver 200 in the present embodiment displays an AR image.

The transmitter 100 comprises two lighting devices, for example, as shown in FIG. 241. The transmitter 100 transmits the optical ID by changing the brightness while illuminating the guide plate 104 of the facility. Since the guide plate 104 is illuminated by the light from the transmitter 100, the luminance changes like the transmitter 100, and the optical ID is transmitted. Further, the guide plate 104 indicates the names of a plurality of facilities such as "ABC Land" and "Adventure Land".

The receiver 200 acquires the captured display image Pc and the decoding image in the same manner as described above by imaging the guide plate 104 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the guide plate 104. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P3 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes a region corresponding to the recognition information in the captured display image Pc as a target region. For example, the receiver 200 recognizes the area on which the guide plate 104 is projected as the target area. Then, the receiver 200 superimposes the AR image P3 on the target area, and displays the captured display image Pc on which the AR image P3 is superposed on the display 201. For example, the AR image P3 is an image showing the names of a plurality of facilities. In this AR image P3, the longer the waiting time of the facility, the smaller the name of the facility is displayed, and conversely, the shorter the waiting time of the facility, the larger the name of the facility is displayed. In this case, since the AR image P3 is superimposed on the target area of the captured display image Pc, the receiver 200 actually has a guide plate 104 on which the name of each facility having a size corresponding to the waiting time is described. The captured display image Pc can be displayed on the screen. As a result, the user of the receiver 200 can be notified of the waiting time of each facility in a simple and easy-to-understand manner without providing the guide plate 104 with a special display device.

FIG. 242 is a diagram showing another example in which the receiver 200 in the present embodiment displays an AR image.

The transmitter 100 comprises two lighting devices, for example, as shown in FIG. 242. The transmitter 100 transmits an optical ID by changing the brightness while illuminating the castle wall 105. Since the castle wall 105 is illuminated by the light from the transmitter 100, the brightness changes like the transmitter 100, and the optical ID is transmitted. Further, on the castle wall 105, for example, a small mark imitating the face of a character is engraved as a hidden character 106.

The receiver 200 acquires the captured display image Pd and the decoding image in the same manner as described above by imaging the castle wall 105 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the castle wall 105. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P4 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes a region corresponding to the recognition information of the captured display image Pd as a target region. For example, the receiver 200 recognizes the area of the castle wall 105 in which the area including the hidden character 106 is projected as the target area. Then, the receiver 200 superimposes the AR image P4 on the target area, and displays the captured display image Pd on which the AR image P4 is superposed on the display 201. For example, the AR image P4 is an image that imitates the face of a character. This AR image P4 is an image sufficiently larger than the hidden character 106 displayed on the captured display image Pd. In this case, since the AR image P4 is superimposed on the target area of the captured display image Pd, the receiver 200 captures the image so that the castle wall 105 engraved with a large mark imitating the face of the character actually exists. The display image Pd can be displayed. As a result, the user of the receiver 200 can be informed of the position of the hidden character 106 in an easy-to-understand manner.

FIG. 243 is a diagram showing another example in which the receiver 200 in the present embodiment displays an AR image.

The transmitter 100 comprises two lighting devices, for example, as shown in FIG. 243. The transmitter 100 transmits the optical ID by changing the brightness while illuminating the guide plate 107 of the facility. Since the guide plate 107 is illuminated by the light from the transmitter 100, the brightness changes like the transmitter 100, and the optical ID is transmitted. Further, the infrared blocking paint 108 is applied to a plurality of corners of the guide plate 107.

The receiver 200 acquires the captured display image Pe and the decoding image in the same manner as described above by imaging the guide plate 107 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the guide plate 107. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P5 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes a region corresponding to the recognition information in the captured display image Pe as a target region. For example, the receiver 200 recognizes the area on which the guide plate 107 is projected as the target area.

Specifically, the recognition information indicates that the target area is a rectangle circumscribing the infrared blocking paint 108 at a plurality of locations. Further, the infrared blocking paint 108 blocks infrared rays contained in the light emitted from the transmitter 100. Therefore, the image sensor of the receiver 200 recognizes the infrared blocking paint 108 as a darker image than its surroundings. The receiver 200 recognizes a rectangle circumscribing the infrared blocking paint 108 at a plurality of locations, each of which appears as a dark image, as a target area.

Then, the receiver 200 superimposes the AR image P5 on the target area, and displays the captured display image Pe on which the AR image P5 is superposed on the display 201. For example, the AR image P5 shows the schedule of the event to be held at the facility of the guide plate 107. In this case, since the AR image P5 is superimposed on the target area of the captured display image Pe, the receiver 200 displays the captured display image Pe so that the guide plate 107 on which the event schedule is described actually exists. can do. As a result, the user of the receiver 200 can be informed of the schedule of the event of the facility in an easy-to-understand manner without providing the guide plate 107 with a special display device.

The guide plate 107 may be coated with an infrared reflective paint instead of the infrared blocking paint 108. The infrared reflective paint reflects infrared rays contained in the light emitted from the transmitter 100. Therefore, the image sensor of the receiver 200 recognizes the infrared reflective paint as a brighter image than its surroundings. That is, in this case, the receiver 200 recognizes the rectangles circumscribing the infrared reflective paints at a plurality of locations, which appear as bright images, as the target area.

FIG. 244 is a diagram showing another example in which the receiver 200 in the present embodiment displays an AR image.

The transmitter 100 is configured as a station name sign and is arranged near the station exit information board 110. The station exit guide plate 110 is provided with a light source and emits light, but unlike the transmitter 100, it does not transmit an optical ID.

The receiver 200 acquires the captured display image Ppre and the decoding image Pdec by capturing the transmitter 100 and the station exit guide plate 110. Since the transmitter 100 changes its brightness and the station exit guide plate 110 emits light, the decoding image Pdec includes a bright line pattern region Pdec1 corresponding to the transmitter 100 and a bright region corresponding to the station exit guide plate 110. Pdec2 and the like appear. The bright line pattern region Pdec1 is a region composed of a plurality of bright line patterns appearing by exposure of a plurality of exposure lines included in the image sensor of the receiver 200 at the communication exposure time.

Here, as described above, the identification information includes the reference information for specifying the reference region Pbas in the captured display image Ppre and the target information indicating the relative position of the target region Ptar with respect to the reference region Pbas. There is. For example, the reference information indicates that the position of the reference region Pbas in the captured display image Ppre is the same as the position of the emission line pattern region Pdec1 in the decoding image Pdec. Further, the target information indicates that the position of the target area is the position of the reference area.

Therefore, the receiver 200 identifies the reference region Pbas from the captured display image Ppre based on the reference information. That is, the receiver 200 specifies a region in the captured display image Ppre at the same position as the position of the emission line pattern region Pdec1 in the decoding image Pdec as the reference region Pbas. Further, the receiver 200 recognizes a region of the captured display image Ppre at a relative position indicated by the target information with respect to the position of the reference region Pbas as the target region Ptra. In the above example, since the target information indicates that the position of the target region Ptar is the position of the reference region Pbas, the receiver 200 recognizes the reference region Pbas in the captured display image Ppre as the target region Pbas.

Then, the receiver 200 superimposes the AR image P1 on the target region Ptar in the captured display image Ppre.

As described above, in the above example, the emission line pattern region Pdec1 is used in order to recognize the target region Ptar. On the other hand, if the region on which the transmitter 100 is projected is to be recognized as the target region Ptar only from the captured display image Ppre without using the emission line pattern region Pdec1, erroneous recognition may occur. .. That is, there is a possibility that the region of the captured display image Ppre on which the station exit guide plate 110 is projected, not the region on which the transmitter 100 is projected, is erroneously recognized as the target region Ptar. This is because the image of the transmitter 100 and the image of the station exit guide plate 110 in the captured display image Ppre are similar. However, as in the above example, when the emission line pattern region Pdec1 is used, it is possible to suppress the occurrence of erroneous recognition and accurately recognize the target region Ptar.

FIG. 245 is a diagram showing another example in which the receiver 200 in the present embodiment displays an AR image.

In the example shown in FIG. 244, the transmitter 100 transmits an optical ID by changing the brightness of the entire station name sign, and the target information indicates that the position of the target area is the position of the reference area. However, in the present embodiment, the transmitter 100 transmits the optical ID by changing the brightness of the light emitting element arranged in a part of the outer frame of the station name mark without changing the brightness of the entire station name mark. May be good. Further, the target information may indicate the relative position of the target region Ptar with respect to the reference region Pbas. For example, the position of the target region Pbas is above the reference region Pbas (specifically, upward in the vertical direction). It may be shown.

In the example shown in FIG. 245, the transmitter 100 transmits an optical ID by changing the brightness of a plurality of light emitting elements arranged along the horizontal direction in the lower part of the outer frame of the station name sign. Further, the target information indicates that the position of the target region Ptar is on the reference region Pbas.

In such a case, the receiver 200 specifies the reference region Pbas from the captured display image Ppre based on the reference information. That is, the receiver 200 specifies a region in the captured display image Ppre at the same position as the position of the emission line pattern region Pdec1 in the decoding image Pdec as the reference region Pbas. Specifically, the receiver 200 specifies a rectangular reference region Pbas that is long in the horizontal direction and short in the vertical direction. Further, the receiver 200 recognizes a region of the captured display image Ppre at a relative position indicated by the target information with respect to the position of the reference region Pbas as the target region Ptra. That is, the receiver 200 recognizes the region above the reference region Pbas in the captured display image Ppre as the target region Ptar. At this time, the receiver 200 specifies the direction above the reference region Pbas based on the gravity direction measured by the acceleration sensor provided in the receiver 200.

The target information may indicate not only the relative position of the target area Ptar but also the size, shape, and aspect ratio of the target area Ptar. In this case, the receiver 200 recognizes the target area Ptar of the size, shape and aspect ratio indicated by the target information. Further, the receiver 200 may determine the size of the target region Ptar based on the size of the reference region Pbas.

FIG. 246 is a flowchart showing another example of the processing operation of the receiver 200 in the present embodiment.

The receiver 200 executes the processes of steps S101 to S104 in the same manner as in the example shown in FIG. 239.

Next, the receiver 200 identifies the emission line pattern region Pdec1 from the decoding image Pdec (step S111). Next, the receiver 200 identifies the reference region Pbas corresponding to the emission line pattern region Pdec1 from the captured display image Ppre (step S112). Then, the receiver 200 recognizes the target region Ptar from the captured display image Ppre based on the recognition information (specifically, the target information) and the reference region Pbas (step S113).

Next, the receiver 200 superimposes an AR image on the target area Ptar of the captured display image Ppre and displays the captured display image Ppre on which the AR image is superimposed, as in the example shown in FIG. 239 (step S106). .. Then, the receiver 200 determines whether or not to end the imaging and the display of the captured image display image Ppre (step S107). Here, when the receiver 200 determines that it should not be terminated (N in step S107), it further determines whether or not the acceleration of the receiver 200 is equal to or greater than the threshold value (step S114). This acceleration is measured by an acceleration sensor provided in the receiver 200. When the receiver 200 determines that the acceleration is less than the threshold value (N in step S114), the receiver 200 executes the process from step S113. As a result, even if the captured display image Ppre displayed on the display 201 of the receiver 200 is deviated, the AR image can be made to follow the target region Ptar of the captured display image Ppre. Further, when the receiver 200 determines that the acceleration is equal to or higher than the threshold value (Y in step S114), the receiver 200 executes the process from step S111 or step S102. As a result, it is possible to prevent the region on which a subject different from the transmitter 100 (for example, the station exit guide plate 110) is projected from being mistakenly recognized as the target region Ptar.

FIG. 247 is a diagram showing another example in which the receiver 200 in the present embodiment displays an AR image.

When the AR image P1 in the displayed captured image Ppre is tapped, the receiver 200 enlarges and displays the AR image P1. Alternatively, the receiver 200 may, when tapped, display a new AR image showing more detailed content than the content shown in the AR image P1 instead of the AR image P1. Further, when the AR image P1 shows information for one page of an information magazine composed of a plurality of pages, the receiver 200 uses a new AR image showing the information on the next page of the page of the AR image P1. It may be displayed instead of the AR image P1. Alternatively, when the receiver 200 is tapped, the moving image related to the AR image P1 may be displayed as a new AR image in place of the AR image P1. At this time, the receiver 200 may display a moving image in which an object (autumn leaves in the example of FIG. 247) comes out from the target area Ptar as an AR image.

FIG. 248 is a diagram showing an image capture display image Ppre and a decoding image Pdec acquired by the image pickup of the receiver 200 in the present embodiment.

When the receiver 200 is capturing images, for example, as shown in FIG. 248 (a1), the receiver 200 acquires captured images such as the captured image Ppre and the decoding image Pdec at a frame rate of 30 fps. Specifically, the receiver 200 acquires the captured display image Ppre "A" at time t1, acquires the decoding image Pdec at time t2, and acquires the captured display image Ppre "B" at time t3. , The captured display image Ppre and the decoding image Pdec are acquired alternately.

Further, when the receiver 200 is displaying the captured image, only the captured display image Ppre is displayed among the captured images, and the decoding image Pdec is not displayed. That is, as shown in FIG. 248 (a2), when the receiver 200 acquires the decoding image Pdec, the receiver 200 displays the captured image Ppre acquired immediately before instead of the decoding image Pdec. Specifically, the receiver 200 displays the acquired image capture display image Ppre "A" at time t1 and displays the image capture display image Ppre "A" acquired at time t1 again at time t2. do. As a result, the receiver 200 displays the captured display image Ppre at a frame rate of 15 fps.

Here, in the example shown in FIG. 248 (a1), the receiver 200 alternately acquires the captured display image Ppre and the decoding image Pdec, but the acquisition form of these images in the present embodiment is this. It is not limited to such a form. That is, the receiver 200 continuously acquires N (N is an integer of 1 or more) decoding image Pdec, and then continuously acquires M (M is an integer of 1 or more) image capture display images Ppre. You may repeat the acquisition.

Further, the receiver 200 needs to switch the acquired captured image between the captured image Ppre and the decoding image Pdec, and this switching may take time. Therefore, as shown in FIG. 248 (b1), the receiver 200 may provide a switching period at the time of switching between the acquisition of the captured display image Ppre and the acquisition of the decoding image Pdec. Specifically, when the receiver 200 acquires the decoding image Pdec at time t3, the receiver 200 executes a process for switching the captured image during the switching period from time t3 to t5, and at time t5, the captured image display image Ppre ". A "is acquired. After that, the receiver 200 executes a process for switching the captured image during the switching period from time t5 to t7, and acquires the decoding image Pdec at time t7.

When the switching period is provided in this way, the receiver 200 displays the captured display image Ppre acquired immediately before in the switching period, as shown in (b2) of FIG. 248. Therefore, in this case, the frame rate of the display of the captured display image Ppre in the receiver 200 is low, for example, 3 fps. When the frame rate is low as described above, even if the user moves the receiver 200, the displayed image capture image Ppre may not move according to the movement of the receiver 200. That is, the captured display image Ppre is not displayed as a live view. Therefore, the receiver 200 may move the captured display image Ppre according to the movement of the receiver 200.

FIG. 249 is a diagram showing an example of the captured display image Ppre displayed on the receiver 200 in the present embodiment.

The receiver 200 displays the captured display image Ppre obtained by imaging on the display 201, for example, as shown in FIG. 249 (a). Here, the user moves the receiver 200 to the left. At this time, if a new image pickup display image Ppre is not acquired by the image pickup by the receiver 200, the receiver 200 moves the displayed image pickup display image Ppre to the right side as shown in FIG. 249 (b). That is, the receiver 200 includes an acceleration sensor, and moves the displayed image capture image Ppre so as to match the movement of the receiver 200 according to the acceleration measured by the acceleration sensor. As a result, the receiver 200 can display the captured display image Ppre as a pseudo live view.

FIG. 250 is a flowchart showing another example of the processing operation of the receiver 200 in the present embodiment.

First, the receiver 200 superimposes an AR image on the target area Ptar of the captured display image Ppre and causes the AR image to follow the target area Ptar (step S122). That is, an AR image that moves together with the target region Ptar in the captured display image Ppre is displayed. Then, the receiver 200 determines whether or not to maintain the display of the AR image (step S122). Here, if it is determined that the display of the AR image is not maintained (N in step S122), if the receiver 200 acquires a new optical ID by imaging, the receiver 200 captures and displays a new AR image corresponding to the optical ID. It is superimposed on Ppre and displayed (step S123).

On the other hand, if it is determined that the display of the AR image is to be maintained (Y in step S122), the receiver 200 repeatedly executes the process from step S121. At this time, the receiver 200 does not display the other AR image even if the other AR image is acquired. Alternatively, even if the receiver 200 has acquired a new decoding image Pdec, the receiver 200 does not acquire the optical ID by decoding the decoding image Pdec. At this time, the power consumption required for decoding can be suppressed.

By maintaining the display of the AR image in this way, it is possible to prevent the displayed AR image from being erased or becoming difficult to see due to the display of another AR image. That is, the displayed AR image can be easily seen by the user.

For example, in step S122, the receiver 200 determines that the display of the AR image is maintained until a predetermined period (fixed period) elapses after the AR image is displayed. That is, when displaying the captured display image Ppre, the receiver 200 is predetermined while suppressing the display of the second AR image different from the first AR image which is the AR image superimposed in step S121. The first AR image is displayed only for the display period. The receiver 200 may prohibit decoding of the newly acquired image Pdec for decoding during this display period.

As a result, when the user is viewing the first AR image once displayed, it is possible to prevent the first AR image from being immediately replaced with a second AR image different from the first AR image. Further, since the decoding of the newly acquired image Pdec for decoding is a useless process when the display of the second AR image is suppressed, the power consumption can be suppressed by prohibiting the decoding. ..

Alternatively, in step S122, the receiver 200 may be provided with a face camera, and if it detects that the user's face is approaching based on the image pickup result by the face camera, it may be determined that the display of the AR image is maintained. .. That is, when displaying the captured display image Ppre, the receiver 200 further determines whether or not the user's face is approaching the receiver 200 by imaging with the face camera provided in the receiver 200. Then, when the receiver 200 determines that the face is approaching, the receiver 200 suppresses the display of the second AR image different from the first AR image, which is the AR image superimposed in step S121, while suppressing the display of the first AR image. Display the AR image.

Alternatively, in step S122, the receiver 200 may be provided with an acceleration sensor, and if it detects that the user's face is approaching based on the measurement result of the acceleration sensor, it may be determined that the display of the AR image is maintained. .. That is, when displaying the captured display image Ppre, the receiver 200 further determines whether or not the user's face is approaching the receiver 200 by the acceleration of the receiver 200 measured by the acceleration sensor. For example, when the acceleration of the receiver 200 measured by the accelerometer shows a positive value in the vertical outward direction with respect to the display 201 of the receiver 200, the receiver 200 is close to the user's face. judge. Then, when the receiver 200 determines that the face is approaching, the receiver 200 suppresses the display of the second AR image different from the first augmented reality image which is the AR image superimposed in step S121, and the first thereof. Display the AR image of.

As a result, when the user is approaching the receiver 200 to see the first AR image, it is possible to prevent the first AR image from being replaced with a second AR image different from the first AR image. ..

Alternatively, in step S122, the receiver 200 may determine that the display of the AR image is maintained when the lock button provided on the receiver 200 is pressed.

Further, in step S122, the receiver 200 determines that the display of the AR image is not maintained after the above-mentioned fixed period (that is, the display period) has elapsed. Further, the receiver 200 determines that the display of the AR image is not maintained when the acceleration equal to or higher than the threshold value is measured by the acceleration sensor even when the above-mentioned fixed period has not elapsed. That is, when displaying the captured display image Ppre, the receiver 200 further measures the acceleration of the receiver 200 by the acceleration sensor during the above-mentioned display period, and determines whether or not the measured acceleration is equal to or higher than the threshold value. Then, when the receiver 200 determines that it is equal to or higher than the threshold value, the receiver 200 displays the second AR image in place of the first AR image in step S123 by releasing the suppression of the display of the second AR image.

As a result, when the acceleration of the display device equal to or higher than the threshold value is measured, the suppression of the display of the second AR image is released. Therefore, for example, when the user moves the receiver 200 greatly in order to point the image sensor at another subject, the second AR image can be displayed immediately.

FIG. 251 is a diagram showing another example in which the receiver 200 in the present embodiment displays an AR image.

As shown in FIG. 251 for example, the transmitter 100 is configured as a lighting device, and transmits an optical ID by changing the brightness while illuminating the stage 111 for a small doll. Since the stage 111 is illuminated by the light from the transmitter 100, the brightness changes like the transmitter 100, and the optical ID is transmitted.

The two receivers 200 image the stage 111 illuminated by the transmitter 100 from the left and right.

The receiver 200 on the left side of the two receivers 200 acquires the captured display image Pf and the decoding image in the same manner as described above by imaging the stage 111 illuminated by the transmitter 100 from the left side. The receiver 200 on the left side acquires an optical ID by decoding the image for decoding. That is, the receiver 200 on the left side receives the optical ID from the stage 111. The receiver 200 on the left side transmits the optical ID to the server. Then, the receiver 200 on the left side acquires the three-dimensional AR image corresponding to the optical ID and the recognition information from the server. This three-dimensional AR image is, for example, an image for displaying a doll in three dimensions. The receiver 200 on the left side recognizes a region corresponding to the recognition information in the captured display image Pf as a target region. For example, the receiver 200 on the left side recognizes the area above the center of the stage 111 as the target area.

Next, the receiver 200 on the left side generates a two-dimensional AR image P6a corresponding to the orientation of the stage 111 projected on the captured display image Pf from the three-dimensional AR image. Then, the receiver 200 on the left side superimposes the two-dimensional AR image P6a on the target area, and displays the captured display image Pf on which the AR image P6a is superposed on the display 201. In this case, since the two-dimensional AR image P6a is superimposed on the target area of the captured display image Pf, the receiver 200 on the left side displays the captured display image Pf so that the doll actually exists on the stage 111. can do.

Similarly, the receiver 200 on the right side of the two receivers 200 captures the captured display image Pg and the decoding image in the same manner as described above by imaging the stage 111 illuminated by the transmitter 100 from the right side. get. The receiver 200 on the right side acquires an optical ID by decoding the image for decoding. That is, the receiver 200 on the right side receives the optical ID from the stage 111. The receiver 200 on the right side transmits the optical ID to the server. Then, the receiver 200 on the right side acquires the three-dimensional AR image corresponding to the optical ID and the recognition information from the server. The receiver 200 on the right side recognizes a region corresponding to the recognition information in the captured display image Pg as a target region. For example, the receiver 200 on the right side recognizes the area above the center of the stage 111 as the target area.

Next, the receiver 200 on the right side generates a two-dimensional AR image P6b corresponding to the orientation of the stage 111 projected on the captured display image Pg from the three-dimensional AR image. Then, the receiver 200 on the right side superimposes the two-dimensional AR image P6b on the target area, and displays the captured display image Pg on which the AR image P6b is superposed on the display 201. In this case, since the two-dimensional AR image P6b is superimposed on the target area of the captured display image Pg, the receiver 200 on the right side displays the captured display image Pg so that the doll actually exists on the stage 111. can do.

In this way, the two receivers 200 display the AR images P6a and P6b at the same position on the stage 111. Further, these AR images P6a and P6b are generated according to the orientation of the receiver 200 so that the virtual doll is actually oriented in a predetermined direction. Therefore, regardless of the direction of the stage 111, the captured display image can be displayed so that the doll actually exists on the stage 111.

In the above example, the receiver 200 generates a two-dimensional AR image according to the positional relationship between the receiver 200 and the stage 111 from the three-dimensional AR image, but the two-dimensional AR image is generated. May be obtained from the server. That is, the receiver 200 transmits the information indicating the positional relationship together with the optical ID to the server, and acquires the two-dimensional AR image from the server instead of the three-dimensional AR image. As a result, the burden on the receiver 200 can be reduced.

FIG. 252 is a diagram showing another example in which the receiver 200 in the present embodiment displays an AR image.

As shown in FIG. 252, for example, the transmitter 100 is configured as a lighting device, and transmits an optical ID by changing the brightness while illuminating the columnar structure 112. Since the structure 112 is illuminated by the light from the transmitter 100, the luminance changes like the transmitter 100, and the optical ID is transmitted.

The receiver 200 acquires the captured display image Ph and the decoding image in the same manner as described above by imaging the structure 112 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the structure 112. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P7 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes a region of the captured display image Ph corresponding to the recognition information as a target region. For example, the receiver 200 recognizes the area where the central portion of the structure 112 is projected as the target area. Then, the receiver 200 superimposes the AR image P7 on the target area, and displays the captured display image Ph on which the AR image P7 is superposed on the display 201. For example, the AR image P7 is an image including the character string “ABCD”, and the character string is distorted according to the curved surface at the center of the structure 112. In this case, since the AR image P2 including the distorted character string is superimposed on the target area of the captured display image Ph, the receiver 200 makes the character string drawn for the structure 112 actually exist. , The captured display image Ph can be displayed.

FIG. 253 is a diagram showing another example in which the receiver 200 in the present embodiment displays an AR image.

As shown in FIG. 253, for example, the transmitter 100 transmits the optical ID by changing the brightness while illuminating the menu 113 of the restaurant. Since the menu 113 is illuminated by the light from the transmitter 100, the brightness changes like the transmitter 100, and the optical ID is transmitted. Further, the menu 113 indicates the names of a plurality of dishes such as "ABC soup", "XYZ salad" and "KLM lunch".

The receiver 200 acquires the captured display image Pi and the decoding image in the same manner as described above by capturing the menu 113 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the menu 113. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P8 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes a region corresponding to the recognition information of the captured display image Pi as a target region. For example, the receiver 200 recognizes the area where the menu 113 is projected as the target area. Then, the receiver 200 superimposes the AR image P8 on the target area, and displays the captured display image Pi on which the AR image P8 is superposed on the display 201. For example, the AR image P8 is an image showing the ingredients used in each of the plurality of dishes with marks. For example, the AR image P8 shows a mark imitating an egg for the dish "XYZ salad" that uses eggs, and a pork for the dish "KLM lunch" that uses pork. Show the imitated mark. In this case, since the AR image P8 is superimposed on the target area of the captured display image Pi, the receiver 200 displays the captured display image Pi so that the menu 113 with the food ingredient mark actually exists. be able to. As a result, the user of the receiver 200 can be informed of the ingredients of each dish in an easy-to-understand manner without providing the menu 113 with a special display device.

Further, the receiver 200 acquires a plurality of AR images, selects an AR image suitable for the user from the plurality of AR images based on the user information set by the user, and superimposes the AR images. You may. For example, if the user information indicates that the user has an allergic reaction to the egg, the receiver 200 selects an AR image marked with an egg for the dish in which the egg is used. Further, if the user information indicates that the intake of pork is prohibited, the receiver 200 selects an AR image in which the pork mark is attached to the dish in which pork is used. Alternatively, the receiver 200 may transmit the user information together with the optical ID to the server, and acquire the AR image corresponding to the optical ID and the user information from the server. As a result, it is possible to display a menu for each user to urge the user to arouse.

FIG. 254 is a diagram showing another example in which the receiver 200 in the present embodiment displays an AR image.

As shown in FIG. 254, for example, the transmitter 100 is configured as a television and transmits an optical ID by changing the brightness while displaying an image on a display. Further, a normal television 114 is arranged in the vicinity of the transmitter 100. The television 114 displays an image on the display, but does not transmit an optical ID.

The receiver 200 acquires the captured display image Pj and the decoding image in the same manner as described above by, for example, capturing the television 114 together with the transmitter 100. The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P9 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes a region corresponding to the recognition information in the captured display image Pj as a target region.

For example, the receiver 200 uses the emission line pattern region of the decoding image so that the lower portion of the region in which the transmitter 100 transmitting the optical ID is projected in the captured display image Pj is the first target region. Recognize as. At this time, the reference information included in the recognition information indicates that the position of the reference region in the captured display image Pj is the same as the position of the emission line pattern region in the decoding image. Further, the target information included in the recognition information indicates that the target area is below the reference area. The receiver 200 recognizes the above-mentioned first target area by using such recognition information.

Further, the receiver 200 recognizes a region whose position is fixed in advance in the lower part of the captured display image Pj as a second target region. The second target area is larger than the first target area. The target information included in the recognition information further indicates not only the position of the first target area but also the position and size of the second target area as described above. The receiver 200 recognizes the above-mentioned second target area by using such recognition information.

Then, the receiver 200 superimposes the AR image P9 on the first target area and the second target area, and displays the captured display image Pj on which the AR image P8 is superposed on the display 201. In the superimposition of the AR image P9, the receiver 200 adjusts the size of the AR image P9 to the size of the first target area, and superimposes the size-adjusted AR image P9 on the first target area. Further, the receiver 200 adjusts the size of the AR image P9 to the size of the second target area, and superimposes the size-adjusted AR image P9 on the second target area.

For example, the AR image P9 shows subtitles for the video of the transmitter 100. Further, the language of the subtitles of the AR image P9 is a language corresponding to the user information set and registered in the receiver 200. That is, when the receiver 200 transmits the optical ID to the server, the receiver 200 also transmits the user information (for example, information indicating the nationality or the language used by the user) to the server. Then, the receiver 200 acquires the AR image P9 showing the subtitles in the language corresponding to the user information. Alternatively, the receiver 200 acquires a plurality of AR images P9 showing subtitles in different languages, and depending on the user information registered in the settings, the AR images used for superimposition from the plurality of AR images P9. P9 may be selected.

In other words, in the example shown in FIG. 254, the receiver 200 acquires the captured display image Pj and the decoding image by capturing a plurality of displays displaying the images as subjects. Then, when the receiver 200 recognizes the target area, the area in which the transmission display (that is, the transmitter 100), which is the display transmitting the optical ID among the plurality of displays, appears in the captured display image Pj. Is recognized as the target area. Next, the receiver 200 superimposes the first subtitle corresponding to the image displayed on the transmission display on the target area as an AR image. Further, the receiver 200 superimposes the second subtitle, which is the enlarged subtitle of the first subtitle, on the area larger than the target area of the captured display image Pj.

As a result, the receiver 200 can display the captured display image Pj so that the subtitles actually exist in the video of the transmitter 100. Further, since the receiver 200 superimposes a large subtitle on the lower part of the captured display image Pj, the subtitle can be easily seen even if the subtitle attached to the image of the transmitter 100 is small. If there is no subtitle attached to the image of the transmitter 100 and only a large subtitle is superimposed on the lower part of the captured display image Pj, the superimposed subtitle is the subtitle for the image of the transmitter 100 or the television. It is difficult to determine whether the subtitles are for the 114 images. However, in the present embodiment, since subtitles are also attached to the video of the transmitter 100 that transmits the optical ID, the user can easily determine which video the superimposed subtitle is for. Can be done.

Further, the receiver 200 may further determine whether or not the information acquired from the server includes audio information in the display of the captured display image Pj. Then, when it is determined that the voice information is included, the receiver 200 preferentially outputs the voice indicated by the voice information over the first and second subtitles. As a result, the voice is preferentially output, so that the burden on the user to read the subtitles can be reduced.

Further, in the above example, the language of the subtitles is different according to the user information (that is, the attribute of the user), but the video (that is, the content) itself displayed on the transmitter 100 may be different. For example, when the image displayed on the transmitter 100 is a news image and the user information indicates that the user is Japanese, the receiver 200 is a news image being broadcast in Japan. Is acquired as an AR image. Then, the receiver 200 superimposes the news image on the area where the display of the transmitter 100 is projected (that is, the target area). On the other hand, if the user information indicates that the user is an American, the receiver 200 acquires the news image broadcast in the United States as an AR image. Then, the receiver 200 superimposes the news image on the area where the display of the transmitter 100 is projected (that is, the target area). This makes it possible to display an image suitable for the user. The user information indicates, for example, nationality or language used as an attribute of the user, and the receiver 200 acquires the above-mentioned AR image based on the attribute.

FIG. 255 is a diagram showing an example of recognition information in the present embodiment.

Even if the recognition information is, for example, a feature point and a feature amount as described above, erroneous recognition may occur. For example, the transmitters 100a and 100b are configured as station name signs like the transmitter 100, respectively. Even if the transmitters 100a and 100b are different from each other, if they are located close to each other, they may be misrecognized because they are similar to each other.

Therefore, the recognition information of the transmitters 100a and 100b does not indicate each feature point and each feature amount of the entire image of the transmitter 100a or 100b, but each feature point and each feature point of only a characteristic part of the image. Each feature amount may be shown.

For example, the portion a1 of the transmitter 100a and the portion b1 of the transmitter 100b are significantly different from each other, and the portion a2 of the transmitter 100a and the portion b2 of the transmitter 100b are significantly different from each other. Therefore, if the transmitters 100a and 100b are installed within a predetermined range (that is, at a short distance), the server can be used as recognition information corresponding to the transmitter 100a, which is a feature of each image of the portion a1 and the portion a2. Holds points and features. Similarly, the server holds the feature points and feature quantities of the respective images of the portion b1 and the portion b2 as the identification information corresponding to the transmitter 100b.

Thereby, the receiver 200 can appropriately use the identification information even when the transmitters 100a and 100b which are similar to each other are close to each other (when they are within the predetermined range described above). The target area can be recognized.

FIG. 256 is a flowchart showing another example of the processing operation of the receiver 200 in the present embodiment.

First, the receiver 200 determines whether or not the user has a visual impairment based on the user information set and registered in the receiver 200 (step S131). Here, when the receiver 200 determines that there is a visual impairment (Y in step S131), the receiver 200 outputs the characters of the AR image to be superimposed and displayed by voice (step S132). On the other hand, when the receiver 200 determines that there is no visual impairment (N in step S131), it further determines whether or not the user has a hearing impairment based on the user information (step S133). Here, when the receiver 200 determines that there is a hearing impairment (Y in step S133), the receiver 200 stops the audio output (step S134). At this time, the receiver 200 stops the output of audio by all the functions.

The receiver 200 may perform the process of step S133 when it is determined in step S131 that there is a visual impairment (Y in step S131). That is, the receiver 200 may output the characters of the AR image superimposed and displayed by voice when it is determined that there is a visual impairment and no hearing impairment.

FIG. 257 is a diagram showing an example in which the receiver 200 in the present embodiment identifies the emission line pattern region.

The receiver 200 first acquires a decoding image by imaging two transmitters each transmitting an optical ID, and by decoding the decoding image, the optical ID is as shown in FIG. 257 (e). To get. At this time, since the decoding image includes two emission line pattern areas X and Y, the receiver 200 receives the optical ID of the transmitter corresponding to the emission line pattern area X and the transmission corresponding to the emission line pattern area Y. Obtain the optical ID of the machine. The optical ID of the transmitter corresponding to the emission line pattern region X is, for example, a numerical value (that is, data) corresponding to each of the addresses 0 to 9, and is "5,2,8,4,3,6,1,9, 4, 3 "is shown. Similarly, the optical ID of the transmitter corresponding to the emission line pattern region X also consists of numerical values corresponding to each of the addresses 0 to 9, for example, "5,2,7,7,1,5,3,2,7, 4 ”is shown.

Even if the receiver 200 acquires these optical IDs once, that is, even if these optical IDs are known, from which emission line pattern region each optical ID is obtained during imaging. Sometimes it's a situation you don't understand. In such a case, the receiver 200 can easily determine from which emission line pattern region each known optical ID is obtained by performing the processes shown in FIGS. 257 (a) to (d). , Can be determined quickly.

Specifically, as shown in FIG. 257 (a), the receiver 200 first acquires the decoding image Pdec11, and by decoding the decoding image Pdec11, each of the emission line pattern regions X and Y is obtained. The numerical value of the address 0 of the optical ID is acquired. For example, the numerical value of the address 0 of the optical ID of the emission line pattern region X is “5”, and the numerical value of the address 0 of the optical ID of the emission line pattern region Y is also “5”. Since the numerical value of the address 0 of each optical ID is "5", at this time, it is not possible to determine from which emission line pattern region the known optical ID is obtained.

Therefore, as shown in FIG. 257 (b), the receiver 200 acquires the decoding image Pdec12 and decodes the decoding image Pdec12 to address 1 of the respective optical IDs of the emission line pattern regions X and Y. Get the number of. For example, the numerical value of the address 1 of the optical ID of the emission line pattern region X is "2", and the numerical value of the address 1 of the optical ID of the emission line pattern region Y is also "2". Since the numerical value of the address 1 of each optical ID is "2", it cannot be determined from which emission line pattern region the known optical ID is obtained.

Therefore, further, as shown in FIG. 257 (c), the receiver 200 acquires the decoding image Pdec13, and by decoding the decoding image Pdec13, the optical IDs of the emission line pattern regions X and Y are obtained. Get the numerical value of address 2. For example, the numerical value of the address 2 of the optical ID of the emission line pattern region X is "8", and the numerical value of the address 2 of the optical ID of the emission line pattern region Y is "7". At this time, it can be determined that the known optical ID "5, 2, 8, 4, 3, 6, 1, 9, 4, 3" is obtained from the emission line pattern region X, and the known optical ID "5," can be determined. It can be determined that "2,7,7,1,5,3,2,7,4" was obtained from the emission line pattern region Y.

However, in order to increase the reliability, the receiver 200 may further acquire the numerical value of the address 3 of each optical ID as shown in FIG. 257 (d). That is, the receiver 200 acquires the decoding image Pdec14, and acquires the numerical value of the address 3 of each optical ID of the emission line pattern regions X and Y by decoding the decoding image Pdec14. For example, the numerical value of the address 3 of the optical ID of the emission line pattern region X is "4", and the numerical value of the address 3 of the optical ID of the emission line pattern region Y is "7". At this time, it can be determined that the known optical ID "5, 2, 8, 4, 3, 6, 1, 9, 4, 3" is obtained from the emission line pattern region X, and the known optical ID "5," can be determined. It can be determined that "2,7,7,1,5,3,2,7,4" was obtained from the emission line pattern region Y. That is, since the optical IDs of the emission line pattern regions X and Y can be identified not only by the address 2 but also by the address 3, the reliability can be improved.

As described above, in the present embodiment, the numerical values of at least one address are reacquired without reacquiring the numerical values (that is, data) of all the addresses of the optical ID. Thereby, it is possible to easily and quickly determine from which emission line pattern region the known optical ID is obtained.

In the example shown in FIGS. 257 (c) and 257 above, the numerical value acquired for the predetermined address matches the known optical ID numerical value, but even if they do not match. good. For example, in the example shown in FIG. 257 (d), the receiver 200 acquires "6" as the numerical value of the address 3 of the optical ID of the emission line pattern region Y. The numerical value "6" of the address 3 is different from the numerical value "7" of the address 3 of the known optical ID "5,2,7,7,1,5,3,2,7,4". However, since the numerical value "6" is a numerical value close to the numerical value "7", the known optical ID "5,2,7,7,1,5,3,2,7,4" is a bright line in the receiver 200. It may be determined that it is obtained from the pattern region Y. In the receiver, the numerical value "6" is close to the numerical value "7" depending on whether or not the numerical value "6" is within the range of the numerical value "7" ± n (n is, for example, a number of 1 or more). It may be determined whether or not.

FIG. 258 is a diagram showing another example of the receiver 200 in this embodiment.

Although the receiver 200 is configured as a smartphone in the above example, it may be configured as a head-mounted display (also referred to as glass) provided with an image sensor, as in the examples shown in FIGS. 19 to 21.

Such a receiver 200 detects a predetermined signal because power consumption increases if the processing circuit for displaying the AR image (hereinafter referred to as AR processing circuit) as described above is always activated. At that time, the AR processing circuit may be started.

For example, the receiver 200 includes a touch sensor 202. The touch sensor 202 outputs a touch signal when it touches a user's finger or the like. The receiver 200 activates the AR processing circuit when the touch signal is detected.

Alternatively, the receiver 200 may activate the AR processing circuit when it detects a radio signal such as Bluetooth (registered trademark) or Wi-Fi (registered trademark).

Alternatively, the receiver 200 may include an acceleration sensor and activate the AR processing circuit when the acceleration sensor measures an acceleration equal to or greater than a threshold value in the direction opposite to the direction of gravity. That is, the receiver 200 activates the AR processing circuit when it detects the signal indicating the acceleration. For example, when the user pushes up the nose pad portion of the receiver 200 configured as a glass upward with a fingertip from below, the receiver 200 detects the signal indicating the acceleration and activates the AR processing circuit.

Alternatively, the receiver 200 may activate the AR processing circuit when it detects that the image sensor is aimed at the transmitter 100 by GPS, a 9-axis sensor, or the like. That is, the receiver 200 activates the AR processing circuit when it detects a signal indicating that the receiver 200 is directed in a predetermined direction. In this case, if the transmitter 100 is the above-mentioned Japanese station name sign or the like, the receiver 200 superimposes and displays an AR image indicating an English station name on the station name sign.

FIG. 259 is a flowchart showing another example of the processing operation of the receiver 200 in the present embodiment.

When the receiver 200 acquires the optical ID from the transmitter 100 (step S141), the receiver 200 switches the noise canceling mode by receiving the mode designation information corresponding to the optical ID (step S142). Then, the receiver 200 determines whether or not the mode switching process should be terminated (step S143), and if it is determined that the mode switching process should not be terminated (N in step S143), the process from step S141 is repeatedly executed. The switching of the noise canceling mode is, for example, a mode (ON) for eliminating noise from an engine or the like in an airplane and a mode (OFF) for eliminating the noise. Specifically, a user carrying a receiver 200 puts an earphone connected to the receiver 200 on his / her ear and listens to voice such as music output from the receiver 200. When such a user boarded an airplane, the receiver 200 acquires an optical ID. As a result, the receiver 200 switches the noise canceling mode from OFF to ON. As a result, the user can hear the voice without noise such as engine noise even in the cabin. Also, when the user leaves the airplane, the receiver 200 acquires the optical ID. The receiver 200 that has acquired this optical ID switches the noise canceling mode from ON to OFF. The noise to be canceled may be any noise such as a human voice as well as the noise of the engine.

FIG. 260 is a diagram showing an example of a transmission system including a plurality of transmitters according to the present embodiment.

The transmitter system comprises a plurality of transmitters 120 arranged in a predetermined order. Similar to the transmitter 100, these transmitters 120 are transmitters according to any one of the above embodiments 1 to 22, and include one or a plurality of light emitting elements (for example, LEDs). The first transmitter 120 transmits an optical ID by changing the brightness of one or more light emitting elements according to a predetermined frequency (carrier frequency). Further, the first transmitter 120 outputs a signal indicating the change in luminance to the subsequent transmitter 120 as a synchronization signal. Upon receiving the synchronization signal, the subsequent transmitter 120 transmits the optical ID by changing the luminance of one or more light emitting elements according to the synchronization signal. Further, the succeeding transmitter 120 outputs a signal indicating the change in the luminance to the next succeeding transmitter 120 as a synchronization signal. As a result, all the transmitters 120 included in the transmission system transmit the optical ID in synchronization.

Here, the synchronization signal is passed from the first transmitter 120 to the succeeding transmitter 120, from the succeeding transmitter 120 to the next succeeding transmitter 120, and to the last transmitter 120. reach. It takes, for example, about 1 μsec to transfer the synchronization signal. Therefore, if the transmitting system is provided with N (N is an integer of 2 or more) transmitters 120, it takes 1 × Nμ seconds for the synchronization signal to reach the last transmitter 120 from the first transmitter 120. become. As a result, the timing of transmitting the optical ID is shifted by a maximum of Nμ seconds. For example, even if N transmitters 120 transmit optical IDs according to a frequency of 9.6 kHz and receiver 200 attempts to receive optical IDs at a frequency of 9.6 kHz, the receiver 200 is offset by N μ seconds. Since the ID is received, the optical ID may not be received correctly.

Therefore, in the present embodiment, the first transmitter 120 transmits the optical ID at high speed according to the number of transmitters 120 included in the transmission system. For example, the first transmitter 120 transmits an optical ID according to a frequency of 9.605 kHz. On the other hand, the receiver 200 receives the optical ID at a frequency of 9.6 kHz. At this time, even if the receiver 200 receives the optical ID shifted by Nμ seconds, the frequency of the leading transmitter 120 is 0.005 kHz higher than the frequency of the receiver 200, so that the reception error due to the deviation of the optical ID occurs. The occurrence can be suppressed.

Further, the first transmitter 120 may control the frequency adjustment amount by having the synchronization signal fed back from the last transmitter 120. For example, the first transmitter 120 measures the time from outputting the synchronization signal by itself to receiving the synchronization signal fed back from the last transmitter 120. Then, the leading transmitter 120 transmits the optical ID according to a frequency higher than the reference frequency (for example, 9.6 kHz) as the time becomes longer.

FIG. 261 is a diagram showing an example of a transmission system including a plurality of transmitters and receivers according to the present embodiment.

This transmission system includes, for example, two transmitters 120 and a receiver 200. One of the two transmitters 120, the transmitter 120, transmits an optical ID according to a frequency of 9.599 kHz. The other transmitter 120 transmits an optical ID according to a frequency of 9.601 kHz. In such a case, each of the two transmitters 120 notifies the receiver 200 of the frequency of its own optical ID by a radio signal.

Upon receiving the notification of those frequencies, the receiver 200 attempts to decode according to each of the notified frequencies. That is, the receiver 200 attempts to decode the image for decoding according to the frequency of 9.599 kHz, and if the optical ID cannot be received by this, attempts to decode the image for decoding according to the frequency of 9.601 kHz. In this way, the receiver 200 attempts to decode the image for decoding according to each of the notified frequencies. In other words, the receiver 200 brute force for each notified frequency. Alternatively, the receiver 200 may attempt decoding according to the average frequency of all notified frequencies. That is, the receiver 200 attempts to decode according to 9.6 kHz, which is the average frequency of 9.599 kHz and 9.601 kHz.

As a result, it is possible to reduce the occurrence rate of reception errors due to the difference in frequency between the receiver 200 and the transmitter 120.

FIG. 262A is a flowchart showing an example of the processing operation of the receiver 200 in the present embodiment.

First, the receiver 200 starts imaging (step S151) and initializes the parameter N to 1 (step S152). Next, the receiver 200 decodes the decoding image obtained by the imaging according to the frequency corresponding to the parameter N, and calculates the evaluation value for the decoding result (step S153). For example, frequencies such as 9.6 kHz, 9.601 kHz, 9.599 kHz, and 9.602 kHz are associated with each of the parameters N = 1, 2, 3, 4, and 5 in advance. The evaluation value shows a higher value as the decoding result resembles the correct optical ID.

Next, the receiver 200 determines whether or not the numerical value of the parameter N is equal to Nmax, which is a predetermined integer or more (step S154). Here, if the receiver 200 determines that it is not equal to Nmax (N in step S154), the parameter N is incremented (step S155), and the process from step S153 is repeatedly executed. On the other hand, when the receiver 200 determines that it is equal to Nmax (Y in step S154), the frequency in which the maximum evaluation value is calculated is set as the optimum frequency, and is registered in the server in association with the location information indicating the location of the receiver 200. do. The optimum frequency and location information registered in this way are used for receiving the optical ID by the receiver 200 that has moved to the location indicated by the location information after registration. Further, the location information may be, for example, information indicating a position measured by GPS, or may be access point identification information (for example, SSID: Service Set Idea) in a wireless LAN (Local Area Network).

Further, the receiver 200 registered in the server displays, for example, an AR image as described above according to the optical ID obtained by decoding with its optimum frequency.

FIG. 262B is a flowchart showing an example of the processing operation of the receiver 200 in the present embodiment.

After the registration to the server shown in FIG. 262A is performed, the receiver 200 transmits the location information indicating the location where the receiver 200 exists to the server (step S161). Next, the receiver 200 acquires the optimum frequency registered in association with the location information from the server (step S162).

Next, the receiver 200 starts imaging (step S163), and decodes the decoding image obtained by the imaging according to the optimum frequency acquired in step S162 (step S164). The receiver 200 displays, for example, an AR image as described above according to the optical ID obtained by this decoding.

As described above, after the registration to the server is performed, the receiver 200 can acquire the optimum frequency and receive the optical ID without executing the process shown in FIG. 262A. When the receiver 200 cannot acquire the optimum frequency in step S162, the receiver 200 may acquire the optimum frequency by executing the process shown in FIG. 262A.

[Summary of Embodiment 23]
FIG. 263A is a flowchart showing a display method according to the present embodiment.

The display method in the present embodiment is a display method in which the display device, which is the receiver 200 described above, displays an image, and includes steps SL11 to SL16.

In step SL11, the image sensor captures the subject to acquire the captured display image and the decoding image. In step SL12, the optical ID is acquired by decoding the image for decoding. In step SL13, the optical ID is transmitted to the server. In step SL14, the AR image corresponding to the optical ID and the recognition information are acquired from the server. In step SL15, the region corresponding to the recognition information in the captured display image is recognized as the target region. In step SL16, the captured display image in which the AR image is superimposed on the target area is displayed.

As a result, the AR image is superimposed on the captured display image and displayed, so that an image useful to the user can be displayed. Further, it is possible to suppress the processing load and superimpose the AR image on an appropriate target area.

That is, in general augmented reality (that is, AR), by comparing a huge number of recognition target images stored in advance with the captured display image, the captured display image includes any of the recognition target images. It is determined whether or not it is. Then, if it is determined that the recognition target image is included, the AR image corresponding to the recognition target image is superimposed on the captured display image. At this time, the AR image is aligned with respect to the recognition target image. As described above, in general augmented reality, since a huge number of recognition target images are compared with the captured display image, and further, position detection of the recognition target image in the captured display image is required for alignment. There is a problem that the amount of calculation is large and the processing load is high.

However, in the display method according to the present embodiment, as shown in FIGS. 235 to 262B, the optical ID is acquired by decoding the decoding image obtained by imaging the subject. That is, the optical ID transmitted from the transmitter that is the subject is received. Further, the AR image corresponding to this optical ID and the recognition information are acquired from the server. Therefore, the server does not need to compare a huge number of recognition target images with the captured display image, and can select an AR image previously associated with the optical ID and transmit it to the display device. As a result, the amount of calculation can be reduced and the processing load can be significantly reduced.

Further, in the display method in the present embodiment, the recognition information corresponding to this optical ID is acquired from the server. The recognition information is information for recognizing a target area, which is an area on which an AR image is superimposed in a captured display image. This recognition information may be, for example, information indicating that the white quadrangle is the target area. In this case, the target area can be easily recognized, and the processing load can be further suppressed. That is, the processing load can be further suppressed according to the content of the recognition information. Further, since the content of the recognition information can be arbitrarily set in the server according to the optical ID, the balance between the processing load and the recognition accuracy can be appropriately maintained.

Here, the recognition information is reference information for specifying a reference region in the captured display image, and in recognition of the target area, a reference region is specified from the captured display image based on the reference information, and the captured display image is displayed. Of these, the target area may be recognized based on the position of the reference area.

Alternatively, the recognition information may include reference information for specifying a reference region in the captured display image and target information indicating the relative position of the target region with respect to the reference region. In this case, in the recognition of the target area, the reference area is specified from the captured display image based on the reference information, and the region of the captured display image at the relative position indicated by the target information with respect to the position of the reference region is determined. Recognize as a target area.

As a result, as shown in FIGS. 244 and 245, the degree of freedom in the position of the target region recognized in the captured display image can be increased.

Further, the reference information is that the position of the reference region in the captured display image is the same as the position of the emission line pattern region consisting of a plurality of emission line patterns appearing by the exposure of the plurality of exposure lines possessed by the image sensor in the decoding image. May be shown.

As a result, as shown in FIGS. 244 and 245, the target region can be recognized with reference to the region corresponding to the emission line pattern region in the captured display image.

Further, the reference information may indicate that the reference region in the captured display image is a region in which the display is projected in the captured display image.

As a result, as shown in FIG. 235, if, for example, a station name sign is used as a display, the target area can be recognized with reference to the area on which the display is displayed.

Further, in the display of the captured display image, the first AR image is displayed only for a predetermined display period while suppressing the display of the second AR image different from the first AR image which is the above-mentioned AR image. You may.

As a result, as shown in FIG. 250, when the user is viewing the first AR image once displayed, the first AR image is immediately replaced with a second AR image different from the first AR image. Can be suppressed.

Further, in the display of the captured display image, the decoding of the newly acquired image for decoding may be prohibited during the display period.

As a result, as shown in FIG. 250, the decoding of the newly acquired image for decoding is a useless process when the display of the second AR image is suppressed. Therefore, by prohibiting the decoding, the decoding is prohibited. Power consumption can be suppressed.

Further, in the display of the captured display image, the acceleration of the display device may be further measured by the acceleration sensor during the display period, and it may be determined whether or not the measured acceleration is equal to or higher than the threshold value. Then, when it is determined that the threshold value is equal to or higher than the threshold value, the second AR image may be displayed instead of the first AR image by releasing the suppression of the display of the second AR image.

As a result, as shown in FIG. 250, when the acceleration of the display device equal to or higher than the threshold value is measured, the suppression of the display of the second AR image is released. Therefore, for example, when the user moves the display device greatly to point the image sensor at another subject, the second AR image can be displayed immediately.

Further, in the display of the captured display image, it may be determined whether or not the user's face is approaching the display device by imaging with the face camera provided in the display device. Then, when it is determined that the face is approaching, the first AR image may be displayed while suppressing the display of the second AR image different from the first AR image. Alternatively, in displaying the captured display image, it may be determined whether or not the user's face is approaching the display device based on the acceleration of the display device measured by the acceleration sensor. Then, when it is determined that the face is approaching, the first AR image may be displayed while suppressing the display of the second AR image different from the first AR image.

As a result, as shown in FIG. 250, when the user brings his / her face close to the display device to see the first AR image, the first AR image is replaced with a different second AR image. It can be suppressed.

Further, as shown in FIG. 254, in the acquisition of the captured display image and the decoding image, the captured display image and the decoding image are acquired by capturing a plurality of displays displaying the images as subjects. May be good. At this time, in the recognition of the target area, the area in which the transmission display, which is the display transmitting the optical ID among the plurality of displays, appears in the captured display image is recognized as the target area. Further, in the display of the captured display image, the first subtitle corresponding to the image displayed on the transmission display is superimposed on the target area as an AR image, and further, the area larger than the target area of the captured display image is displayed. The second subtitle, which is an enlarged subtitle of the first subtitle, is superimposed.

As a result, the first subtitle is superimposed on the image of the transmission display, so that the user can easily grasp which of the plurality of displays the first subtitle is the subtitle for the image of the plurality of displays. can. Further, since the second subtitle, which is an enlarged subtitle of the first subtitle, is also displayed, even if the first subtitle is small and difficult to read, the display of the second subtitle makes it easier to read the subtitle. can do.

Further, in the display of the captured display image, it is further determined whether or not the information acquired from the above-mentioned server contains audio information, and when it is determined that the information is included, the first and second subtitles are subtitled. The voice indicated by the voice information may be preferentially output.

As a result, the voice is preferentially output, so that the burden on the user to read the subtitles can be reduced.

FIG. 263B is a block diagram showing a configuration of a display device according to the present embodiment.

The display device 10 in the present embodiment is a display device that displays an image, and includes an image sensor 11, a decoding unit 12, a transmission unit 13, an acquisition unit 14, a recognition unit 15, and a display unit 16. Be prepared. The display device 10 corresponds to the above-mentioned receiver 200.

The image sensor 11 acquires a captured display image and a decoding image by capturing a subject. The decoding unit 12 acquires an optical ID by decoding the image for decoding. The transmission unit 13 transmits the optical ID to the server. The acquisition unit 14 acquires the AR image corresponding to the optical ID and the recognition information from the server. The recognition unit 15 recognizes a region corresponding to the recognition information in the captured display image as a target region. The display unit 16 displays a captured display image in which an AR image is superimposed on the target area.

As a result, the AR image is superimposed on the captured display image and displayed, so that an image useful to the user can be displayed. Further, it is possible to suppress the processing load and superimpose the AR image on an appropriate target area.

In the present embodiment, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software that realizes the receiver 200 or the display device 10 of the present embodiment is included in the flowcharts shown in FIGS. 239, 246, 250, 256, 259, and 262A to 263A. A program that causes a computer to perform steps.

[Modification 1 of Embodiment 23]
Hereinafter, a modification 1 of the embodiment 23, that is, a modification 1 of a display method for realizing AR using an optical ID will be described.

FIG. 264 is a diagram showing an example in which the receiver in the first modification of the 23rd embodiment displays an AR image.

The receiver 200 acquires a captured display image Pk, which is the above-mentioned normal captured image, and a decoding image, which is the above-mentioned visible light communication image or emission line image, by capturing the subject by the image sensor.

Specifically, the image sensor of the receiver 200 images the transmitter 100c configured as a robot and the person 21 next to the transmitter 100c. The transmitter 100c is a transmitter according to any one of the above embodiments 1 to 22, and includes one or a plurality of light emitting elements (for example, LEDs) 131. The transmitter 100c changes its brightness by blinking one or more light emitting elements 131, and transmits an optical ID (light identification information) according to the change in brightness. This optical ID is the above-mentioned visible light signal.

The receiver 200 acquires an image capture display image Pk on which the transmitter 100c and the person 21 are imaged at a normal exposure time. Further, the receiver 200 acquires a decoding image by imaging the transmitter 100c and the person 21 with a communication exposure time shorter than the normal exposure time.

The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the transmitter 100c. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P10 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes a region corresponding to the recognition information in the captured display image Pk as a target region. For example, the receiver 200 recognizes a region on the right side of the region on which the robot, which is the transmitter 100c, is projected as a target region. Specifically, the receiver 200 specifies the distance between the two markers 132a and 132b of the transmitter 100c displayed on the captured display image Pk. Then, the receiver 200 recognizes a region having a width and a height corresponding to the distance as a target region. That is, the recognition information indicates the shape of the markers 132a and 132b and the position and size of the target area with respect to the markers 132a and 132b.

Then, the receiver 200 superimposes the AR image P10 on the target area, and displays the captured display image Pk on which the AR image P10 is superposed on the display 201. For example, the receiver 200 acquires an AR image P10 showing another robot different from the transmitter 100c. In this case, since the AR image P10 is superimposed on the target area of the captured display image Pk, the captured display image Pk can be displayed so that another robot actually exists next to the transmitter 100c. As a result, the person 21 can be photographed together with the other robot together with the transmitter 100c even if the other robot does not actually exist.

FIG. 265 is a diagram showing another example in which the receiver 200 in the first modification of the second embodiment 23 displays an AR image.

As shown in FIG. 265, for example, the transmitter 100 is configured as an image display device having a display panel, and transmits an optical ID by changing the brightness while displaying a still image PS on the display panel. The display panel is, for example, a liquid crystal display or an organic EL (electroluminescence) display.

By imaging the transmitter 100, the receiver 200 acquires the captured display image Pm and the decoding image in the same manner as described above. The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P11 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes a region corresponding to the recognition information in the captured display image Pm as a target region. For example, the receiver 200 recognizes the area on which the display panel of the transmitter 100 is projected as the target area. Then, the receiver 200 superimposes the AR image P11 on the target area, and displays the captured display image Pm on which the AR image P11 is superposed on the display 201. For example, the AR image P11 is a moving image having the same or substantially the same picture as the still image PS displayed on the display panel of the transmitter 100 as the first picture in the display order. That is, the AR image P11 is a moving image that starts to move from the still image PS.

In this case, since the AR image P11 is superimposed on the target area of the captured display image Pm, the receiver 200 displays the captured display image Pm so that an image display device for displaying a moving image actually exists. Can be done.

FIG. 266 is a diagram showing another example in which the receiver 200 in the first modification of the second embodiment 23 displays an AR image.

The transmitter 100 is configured as a station name sign, for example, as shown in FIG. 266, and transmits an optical ID by changing the luminance.

As shown in FIG. 266 (a), the receiver 200 takes an image of the transmitter 100 from a position away from the transmitter 100. As a result, the receiver 200 acquires the captured display image Pn and the decoding image in the same manner as described above. The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR images P12 to P14 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes two regions of the captured display image Pn according to the recognition information as the first and second target regions. For example, the receiver 200 recognizes the area around the transmitter 100 as the first target area. Then, the receiver 200 superimposes the AR image P12 on the first target area, and displays the captured display image Pn on which the AR image P12 is superposed on the display 201. For example, the AR image P12 is an arrow that urges the user of the receiver 200 to approach the transmitter 100.

In this case, since the AR image P12 is superimposed and displayed on the first target area of the captured display image Pn, the user approaches the transmitter 100 with the receiver 200 facing the transmitter 100. Due to the approach of the receiver 200 to the transmitter 100, the area of the transmitter 100 (corresponding to the above-mentioned reference area) displayed on the captured display image Pn becomes large. When the size of the region becomes equal to or larger than the first threshold value, the receiver 200 further becomes a second target region, which is a region in which the transmitter 100 is projected, as shown in (b) of FIG. 266, for example. The AR image P13 is superimposed. That is, the receiver 200 displays the captured display image Pn on which the AR images P12 and P13 are superimposed on the display 201. For example, the AR image P13 is a message informing the user of an outline of the area around the station shown on the station name sign. Further, the AR image P13 is equal to the size of the area of the transmitter 100 projected on the captured display image Pn.

Further, also in this case, since the AR image P12, which is an arrow, is superimposed and displayed on the first target area of the captured display image Pn, the user can point the receiver 200 at the transmitter 100 and display the transmitter. Approaching 100. Due to the approach of the receiver 200 to the transmitter 100, the area of the transmitter 100 (corresponding to the above-mentioned reference area) displayed on the captured display image Pn becomes even larger. When the size of the region becomes equal to or larger than the second threshold value, the receiver 200 changes the AR image P13 superimposed on the second target region to the AR image P14, for example, as shown in FIG. 266 (c). do. Further, the receiver 200 deletes the AR image P12 superimposed on the first target area.

That is, the receiver 200 displays the captured display image Pn on which the AR image P14 is superimposed on the display 201. For example, the AR image P14 is a message informing the user of the details of the area around the station shown on the station name sign. Further, the AR image P14 is equal to the size of the area of the transmitter 100 projected on the captured display image Pn. The area of the transmitter 100 is larger as the receiver 200 is closer to the transmitter 100. Therefore, the AR image P14 is larger than the AR image P13.

As described above, the receiver 200 enlarges the AR image and displays a large amount of information as it approaches the transmitter 100. Further, since an arrow prompting the user to approach the AR image P12 is displayed, the user can easily understand that a large amount of information is displayed when the transmitter 100 is approached.

FIG. 267 is a diagram showing another example in which the receiver 200 in the first modification of the second embodiment 23 displays an AR image.

In the example shown in FIG. 266, the receiver 200 displays a lot of information when approaching the transmitter 100, but displays a lot of information in the form of a balloon, for example, regardless of the distance from the transmitter 100. May be good.

Specifically, as shown in FIG. 267, the receiver 200 acquires the captured display image Po and the decoding image in the same manner as described above by imaging the transmitter 100. The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P15 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes a region corresponding to the recognition information in the captured display image Po as a target region. For example, the receiver 200 recognizes the area around the transmitter 100 as the target area. Then, the receiver 200 superimposes the AR image P15 on the target area, and displays the captured display image Po on which the AR image P15 is superposed on the display 201. For example, the AR image P15 is a message informing the user of the details of the area around the station shown on the station name sign in the form of a balloon.

In this case, since the AR image P15 is superimposed on the target area of the captured display image Po, the user of the receiver 200 can display a large amount of information on the receiver 200 without approaching the transmitter 100.

FIG. 268 is a diagram showing another example of the receiver 200 in the first modification of the 23rd embodiment.

Although the receiver 200 is configured as a smartphone in the above example, it is configured as a head-mounted display (also referred to as a glass) provided with an image sensor as in the examples shown in FIGS. 19 to 21 and 258. May be good.

Such a receiver 200 acquires an optical ID by decoding only a part of the decoding target area of the image for decoding. For example, the receiver 200 includes a line-of-sight detection camera 203 as shown in FIG. 268 (a). The line-of-sight detection camera 203 captures the eyes of a user wearing a head-mounted display, which is a receiver 200. The receiver 200 detects the line of sight of the user based on the image of the eye obtained by the image pickup by the line-of-sight detection camera 203.

As shown in FIG. 268 (b), the receiver 200 displays the line-of-sight frame 204 so that the line-of-sight frame 204 appears in the region of the user's field of view to which the detected line-of-sight is directed. .. Therefore, the line-of-sight frame 204 moves according to the movement of the user's line of sight. The receiver 200 treats the area corresponding to the line-of-sight frame 204 of the image for decoding as the decoding target area. That is, even if there is a bright line pattern area outside the decoding target area of the decoding image, the receiver 200 does not decode the bright line pattern area, but decodes only the bright line pattern area in the decoding target area. .. As a result, even if the image for decoding has a plurality of bright line pattern areas, all of the bright line pattern areas are not decoded, so that the processing load can be reduced and the display of an unnecessary AR image can be suppressed. Can be done.

Further, when the receiver 200 includes a plurality of emission line pattern areas for outputting audio, only the emission line pattern area in the decoding target area is decoded, and the emission line pattern area is decoded. Only the audio corresponding to may be output. Alternatively, the receiver 200 decodes each of the plurality of emission line pattern areas included in the decoding image, outputs a large amount of sound corresponding to the emission line pattern area in the decoding target area, and outputs the sound corresponding to the emission line pattern area in the decoding target area to the emission line pattern area outside the decoding target area. The corresponding voice may be output in a small size. Further, when there are a plurality of emission line pattern areas outside the decoding target area, the receiver 200 may output the sound corresponding to the emission line pattern area larger as the emission line pattern area is closer to the decoding target area.

FIG. 269 is a diagram showing another example in which the receiver 200 in the first modification of the second embodiment 23 displays an AR image.

As shown in FIG. 269, for example, the transmitter 100 is configured as an image display device having a display panel, and transmits an optical ID by changing the brightness while displaying an image on the display panel.

By imaging the transmitter 100, the receiver 200 acquires the captured display image Pp and the decoding image in the same manner as described above.

At this time, the receiver 200 specifies a region that is at the same position as the emission line pattern region in the decoding image and has the same size as the emission line pattern region from the captured display image Pp. Then, the receiver 200 may display the scanning line P100 that repeatedly moves from one end to the other end of the region.

While the scanning line P100 is displayed, the receiver 200 acquires an optical ID by decoding the image for decoding and transmits the optical ID to the server. Then, the receiver 200 acquires the AR image corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes a region corresponding to the recognition information in the captured display image Pp as a target region.

Upon recognizing such a target area, the receiver 200 ends the display of the scanning line P100, superimposes an AR image on the target area, and displays the captured display image Pp on which the AR image is superimposed on the display 201. ..

As a result, since the moving scanning line P100 is displayed from the time when the transmitter 100 is imaged until the AR image is displayed, processing such as reading the optical ID and the AR image is performed. Can be notified to the user.

FIG. 270 is a diagram showing another example in which the receiver 200 in the first modification of the second embodiment 23 displays an AR image.

Each of the two transmitters 100 is configured as an image display device having a display panel, for example, as shown in FIG. 270, and transmits an optical ID by changing the brightness while displaying the same still image PS on the display panel. is doing. Here, the two transmitters 100 transmit different optical IDs (for example, optical IDs "01" and "02") by changing the luminance in different modes from each other.

Similar to the example shown in FIG. 265, the receiver 200 acquires the captured display image Pq and the decoding image by imaging the two transmitters 100. The receiver 200 acquires the optical IDs "01" and "02" by decoding the image for decoding. That is, the receiver 200 receives the optical ID "01" from one of the two transmitters 100 and the optical ID "02" from the other. The receiver 200 transmits those optical IDs to the server. Then, the receiver 200 acquires the AR image P16 corresponding to the optical ID "01" and the recognition information from the server. Further, the receiver 200 acquires the AR image P17 corresponding to the optical ID “02” and the recognition information from the server.

The receiver 200 recognizes a region of the captured display image Pq corresponding to the recognition information as a target region. For example, the receiver 200 recognizes the area on which the display panels of the two transmitters 100 are projected as the target area. Then, the receiver 200 superimposes the AR image P16 on the target area corresponding to the optical ID "01" and superimposes the AR image P17 on the target area corresponding to the optical ID "02". Then, the receiver 200 displays the captured display image Pq on which the AR images P16 and P17 are superimposed on the display 201. For example, the AR image P16 is a moving image having the same or substantially the same picture as the still image PS displayed on the display panel of the transmitter 100 corresponding to the optical ID "01" as the first picture in the display order. be. Further, the AR image P17 is a moving image having the same or substantially the same picture as the still image PS displayed on the display panel of the transmitter 100 corresponding to the optical ID "02" as the first picture in the display order. be. That is, the first pictures of the AR image P16 and the AR image P17, which are moving images, are the same. However, the AR image P16 and the AR image P17 are different moving images from each other, and the pictures other than the head of each are different.

Therefore, since such AR image P16 and AR image P17 are superimposed on the captured display image Pq, the receiver 200 may actually have an image display device that displays different moving images reproduced from the same picture. The captured display image Pq can be displayed.

FIG. 271 is a flowchart showing an example of the processing operation of the receiver 200 in the modification 1 of the embodiment 23. The processing operation shown by the flowchart of FIG. 271 is specifically an example of the processing operation of the receiver 200 that individually images the transmitters 100 when there are two transmitters 100 shown in FIG. 265. be.

First, the receiver 200 acquires the first optical ID by taking an image of the first transmitter 100 as the first subject (step S201). Next, the receiver 200 recognizes the first subject from the captured display image (step S202). That is, the receiver 200 acquires the first AR image and the first recognition information corresponding to the first optical ID from the server, and recognizes the first subject based on the first recognition information. Then, the receiver 200 starts reproduction of the first moving image, which is the first AR image, from the beginning (step S203). That is, the receiver 200 starts reproduction from the first picture of the first moving image.

Here, the receiver 200 determines whether or not the first subject deviates from the captured display image (step S204). That is, the receiver 200 determines whether or not the first subject cannot be recognized from the captured display image. Here, when it is determined that the first subject deviates from the captured display image (Y in step S204), the receiver 200 interrupts the reproduction of the first moving image which is the first AR image (step S205).

Next, the receiver 200 has a second optical ID different from the first optical ID acquired in step S201 by taking an image of the second transmitter 100 different from the first transmitter 100 as the second subject. It is determined whether or not the optical ID of the above is acquired (step S206). Here, when the receiver 200 determines that the second optical ID has been acquired (Y in step S206), the receiver 200 performs the same processing as in steps S202 to S203 after the acquisition of the first optical ID. That is, the receiver 200 recognizes the second subject from the captured display image (step S207). Then, the receiver 200 starts reproduction of the second moving image, which is the second AR image corresponding to the second optical ID, from the beginning (step S208). That is, the receiver 200 starts reproduction from the first picture of the second moving image.

On the other hand, if the receiver 200 determines in step S206 that the second optical ID has not been acquired (N in step S206), it determines whether or not the first subject has entered the captured display image again (step). S209). That is, the receiver 200 determines whether or not the first subject is recognized again from the captured display image. Here, when the receiver 200 determines that the first subject has entered the captured display image (Y in step S209), it further determines whether or not a predetermined time (that is, a predetermined time) has elapsed (that is, whether or not a predetermined time has elapsed). Step S210). That is, the receiver 200 determines whether or not a predetermined time has elapsed from the time when the first subject is removed from the captured display image to the time when the first subject is re-entered. Here, if it is determined that the predetermined time has not elapsed (Y in step S210), the receiver 200 starts reproducing the interrupted first moving image from the middle (step S211). It should be noted that the playback restart first picture, which is the first moving image picture displayed at the start of playback from the middle of the process, is next to the last displayed picture when the playback of the first moving image is interrupted. It may be a picture in the display order of. Alternatively, the reproduction restart first picture may be the previous picture in the display order by n (n is an integer of 1 or more) from the last displayed picture.

On the other hand, when it is determined that the predetermined time has elapsed (N in step S210), the receiver 200 starts reproducing the interrupted first moving image from the beginning (step S212).

Further, in the above example, the receiver 200 superimposes the AR image on the target area of the captured display image, but at this time, the brightness of the AR image may be adjusted. That is, the receiver 200 determines whether or not the brightness of the AR image acquired from the server matches the brightness of the target area of the captured display image. Then, when the receiver 200 determines that they do not match, the receiver 200 adjusts the brightness of the AR image to match the brightness of the AR image with the brightness of the target region. Then, the receiver 200 superimposes the brightness-adjusted AR image on the target area of the captured display image. As a result, the superimposed AR image can be made closer to the image of an existing object, and the user's discomfort with the AR image can be suppressed. The brightness of the AR image is the spatial average brightness of the AR image, and the brightness of the target area is also the spatial average brightness of the target area.

Further, as shown in FIG. 247, when the AR image is tapped, the receiver 200 may enlarge the AR image and display it on the entire display 201. Further, in the example shown in FIG. 247, the receiver 200 switches the AR image to which the AR image is tapped to another AR image, but the AR image may be automatically switched regardless of the tap. For example, when the time during which the AR image is displayed elapses for a predetermined time, the receiver 200 switches the AR image to another AR image and displays the AR image. Further, when the current time reaches a predetermined time, the receiver 200 switches the AR image displayed up to that point to another AR image and displays the AR image. As a result, the user can easily view the new AR image without performing any operation.

[Modification 2 of Embodiment 23]
Hereinafter, a modification 2 of the embodiment 23, that is, a modification 2 of a display method for realizing AR using an optical ID will be described.

FIG. 272 is a diagram showing an example of a problem when displaying an AR image assumed in the receiver 200 in the 23rd embodiment or the first modification thereof.

For example, the receiver 200 in the 23rd embodiment or the 1st modification thereof takes an image of the subject at time t1. The above-mentioned subject is a transmitter such as a television that transmits an optical ID due to a change in brightness, or a poster, a guide plate, a signboard, or the like illuminated by light from the transmitter. As a result, the receiver 200 displays the entire image obtained by the effective pixel region of the image sensor (hereinafter referred to as the total captured image) on the display 201 as an captured display image. At this time, the receiver 200 recognizes the region corresponding to the recognition information acquired based on the optical ID in the captured display image as the target region on which the AR image is superimposed. The target area is an area showing an image of a transmitter such as a television or an image of a poster. Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201. The AR image may be a still image or a moving image, or may be a character string composed of one or more characters or symbols.

Here, when the user of the receiver 200 approaches the subject in order to display the AR image in a large size, the region corresponding to the target region in the image sensor (hereinafter referred to as the recognition region) protrudes from the effective pixel region at time t2. The recognition area is an area in which the image of the target area in the captured display image is projected in the effective pixel area of the image sensor. That is, the effective pixel area and the recognition area in the image sensor correspond to the captured display image and the target area in the display 201, respectively.

When the recognition area protrudes from the effective pixel area, the receiver 200 cannot recognize the target area from the captured display image and cannot display the AR image.

Therefore, the receiver 200 in this modification acquires an image having a wider angle of view than the captured display image displayed on the entire display 201 as a total captured image.

FIG. 273 is a diagram showing an example in which the receiver 200 in the second modification of the second embodiment displays an AR image.

The angle of view of the entire captured image of the receiver 200 according to this modification, that is, the angle of view of the effective pixel region of the image sensor is wider than the angle of view of the captured display image displayed on the entire display 201. In the image sensor, the area corresponding to the image range displayed on the display 201 is hereinafter referred to as a display area.

For example, the receiver 200 takes an image of the subject at time t1. As a result, the receiver 200 displays only the image obtained by the display area narrower than the effective pixel area among all the captured images obtained by the effective pixel area of the image sensor on the display 201 as the captured display image. At this time, the receiver 200 recognizes the area corresponding to the recognition information acquired based on the optical ID among all the captured images as the target area on which the AR image is superimposed, as described above. Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201.

Here, when the user of the receiver 200 approaches the subject in order to display the AR image in a large size, the recognition area in the image sensor expands. Then, at time t2, the recognition area extends beyond the display area of the image sensor. That is, the image of the target area (for example, the image of a poster) protrudes from the captured display image displayed on the display 201. However, the recognition area in the image sensor does not extend beyond the effective pixel area. That is, the receiver 200 has acquired the entire captured image including the target region even at time t2. As a result, the receiver 200 can recognize the target area from all the captured images, and superimposes a part of the AR image corresponding to that area only on a part of the target area in the captured display image. Is displayed on the display 201.

As a result, even if the user approaches the subject in order to display the AR image in a large size and the target area extends beyond the captured display image, the display of the AR image can be continued.

FIG. 274 is a flowchart showing an example of the processing operation of the receiver 200 in the second modification of the second embodiment.

The receiver 200 acquires the entire captured image and the decoding image by the image sensor capturing the subject (step S301). Next, the receiver 200 acquires the optical ID by decoding the image for decoding (step S302). Next, the receiver 200 transmits the optical ID to the server (step S303). Next, the receiver 200 acquires the AR image corresponding to the optical ID and the recognition information from the server (step S304). Next, the receiver 200 recognizes a region corresponding to the recognition information as a target region among all the captured images (step S305).

Here, the receiver 200 determines whether or not the recognition area in the effective pixel area of the image sensor, which is the area corresponding to the image in the target area, extends beyond the display area (step S306). Here, if it is determined that the image is out of the range (Yes in step S306), the receiver 200 selects only a part of the AR image corresponding to the part of the target area in the captured display image. Display (step S307). On the other hand, when the receiver 200 determines that the AR image does not protrude (No in step S306), the receiver 200 superimposes the AR image on the target area of the captured display image and displays the captured display image on which the AR image is superimposed. (Step S308).

Then, the receiver 200 determines whether or not the AR image display process should be terminated (step S309), and if it is determined that the AR image display process should not be terminated (No in step S309), the receiver 200 repeatedly executes the process from step S305.

FIG. 275 is a diagram showing another example in which the receiver 200 in the second modification of the second embodiment displays an AR image.

The receiver 200 may switch the screen display of the AR image according to the ratio of the size of the recognition area to the display area described above.

When the horizontal width of the display area of the image sensor is w1, the vertical width is h1, the horizontal width of the recognition area is w2, and the vertical width is h2, the receiver has a ratio (h2 / h1). ) And (w2 / w1), whichever is greater, is compared with the threshold.

For example, when the receiver 200 is displaying the captured display image in which the AR image is superimposed on the target area as shown in FIG. 275 (screen display 1), the larger ratio is set as the first threshold value (the first threshold value (screen display 1). For example, compare with 0.9). Then, when the larger ratio becomes 0.9 or more, the receiver 200 enlarges and displays the AR image on the entire display 201 as shown in FIG. 275 (screen display 2). Even when the recognition area becomes larger than the display area and further becomes larger than the effective pixel area, the receiver 200 continues to enlarge and display the AR image on the entire display 201.

Further, when the receiver 200 enlarges and displays the AR image on the entire display 201, for example, as shown in FIG. 275 (screen display 2), the larger ratio is set to the second threshold value (the second threshold value (screen display 2). For example, compare with 0.7). The second threshold is smaller than the first threshold. Then, when the larger ratio becomes 0.7 or less, the receiver 200 displays the captured display image in which the AR image is superimposed on the target area as shown in FIG. 275 (screen display 1).

FIG. 276 is a flowchart showing another example of the processing operation of the receiver 200 in the second modification of the second embodiment.

The receiver 200 first performs optical ID processing (step S301a). This optical ID process is a process including steps S301 to S304 shown in FIG. 274. Next, the receiver 200 recognizes the region corresponding to the recognition information in the captured display image as the target region (step S311). Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed (step S312).

Next, in the receiver 200, the ratio of the recognition region, that is, the larger ratio of the ratios (h2 / h1) and (w2 / w1) is equal to or higher than the first threshold value K (for example, K = 0.9). It is determined whether or not (step S313). Here, if it is determined that the threshold value is not equal to or higher than the first threshold value K (No in step S313), the receiver 200 repeatedly executes the process from step S311. On the other hand, if it is determined that the threshold value is K or higher (Yes in step S313), the receiver 200 enlarges and displays the AR image on the entire display 201 (step S314). At this time, the receiver 200 periodically switches the power supply of the image sensor on and off. By periodically turning off the power of the image sensor, it is possible to save power in the receiver 200.

Next, the receiver 200 determines whether or not the ratio of the recognition region is equal to or less than the second threshold value L (for example, L = 0.7) when the power of the image sensor is periodically turned on. do. Here, if it is determined that the threshold value is not equal to or less than the second threshold value (No in step S315), the receiver 200 repeatedly executes the process from step S314. On the other hand, if it is determined that the threshold value is L or less (Yes in step S315), the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed. (Step S316).

Then, the receiver 200 determines whether or not the AR image display process should be terminated (step S317), and if it is determined that the AR image display process should not be terminated (No in step S317), the receiver 200 repeatedly executes the process from step S313.

By setting the second threshold value L to a value smaller than the first threshold value K in this way, the screen display of the receiver 200 can be frequently switched between (screen display 1) and (screen display 2). Can be prevented and the state of the screen display can be stabilized.

In the examples shown in FIGS. 275 and 276, the display area and the effective pixel area may be the same or different. Further, in that example, the ratio of the size of the recognition area to the display area was used, but when the display area and the effective pixel area are different, the size of the recognition area with respect to the effective pixel area is used instead of the display area. You may use the ratio.

FIG. 277 is a diagram showing another example in which the receiver 200 in the second modification of the second embodiment displays an AR image.

In the example shown in FIG. 277, as in the example shown in FIG. 273, the image sensor of the receiver 200 has an effective pixel area wider than the display area.

For example, the receiver 200 takes an image of the subject at time t1. As a result, the receiver 200 displays only the image obtained by the display area narrower than the effective pixel area among all the captured images obtained by the effective pixel area of the image sensor on the display 201 as the captured display image. At this time, the receiver 200 recognizes the area corresponding to the recognition information acquired based on the optical ID among all the captured images as the target area on which the AR image is superimposed, as described above. Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201.

Here, when the user changes the direction of the receiver 200 (specifically, the image sensor), the recognition area in the image sensor moves, for example, in the upper left direction in FIG. 277, and at time t2, the recognition area protrudes from the display area. That is, the image of the target area (for example, the image of a poster) protrudes from the captured display image displayed on the display 201. However, the recognition area in the image sensor does not extend beyond the effective pixel area. That is, the receiver 200 has acquired the entire captured image including the target region even at time t2. As a result, the receiver 200 can recognize the target area from all the captured images, and superimposes a part of the AR image corresponding to that area only on a part of the target area in the captured display image. And display it on the display 201. Further, the receiver 200 changes the size and position of a part of the displayed AR image according to the movement of the recognition area in the image sensor, that is, the movement of the target area in the entire captured image.

Further, when the recognition area extends beyond the display area as described above, the receiver 200 has the number of pixels corresponding to the distance between the edge of the effective pixel area and the edge of the display area (hereinafter referred to as the inter-regional distance). Is compared with the threshold.

For example, the shorter of the distance between the upper side of the effective pixel area and the upper side of the display area and the distance between the lower side of the effective pixel area and the lower side of the display area (hereinafter, the first). Let dh be the number of pixels corresponding to (referred to as distance). Further, the shorter of the distance between the left side of the effective pixel area and the left side of the display area and the distance between the right side of the effective pixel area and the right side of the display area (hereinafter referred to as the second second). Let dw be the number of pixels corresponding to (referred to as distance). At this time, the above-mentioned inter-regional distance is the shorter of the first and second distances.

That is, the receiver 200 compares the number of pixels, whichever is smaller of the number of pixels dw and dh, with the threshold value N. Then, for example, at time t2, when the smaller number of pixels becomes the threshold value N or less, the receiver 200 changes the size and position of a part of the AR image according to the position of the recognition area in the image sensor. Fix without. That is, the receiver 200 switches the screen display of the AR image. For example, the receiver 200 sets the size and position of a part of the displayed AR image to the size of a part of the AR image displayed on the display 201 when the smaller number of pixels reaches the threshold value N. Fix in position and position.

Therefore, even if the recognition area moves further at time t3 and extends beyond the effective pixel area, the receiver 200 continues to display a part of the AR image as at time t2. That is, as long as the smaller number of pixels among the number of pixels dw and dh is equal to or less than the threshold value N, the receiver 200 captures a part of the AR image having a fixed size and position as at time t2. Continue to display by superimposing on the display image.

In the example shown in FIG. 277, the receiver 200 changes the size and position of a part of the displayed AR image according to the movement of the recognition area in the image sensor, but the display magnification and position of the entire AR image are changed. You may change it.

FIG. 278 is a diagram showing another example in which the receiver 200 in the second modification of the second embodiment displays an AR image. Specifically, FIG. 278 shows an example in which the display magnification of the AR image is changed.

For example, as in the example shown in FIG. 277, when the user changes the direction of the receiver 200 (specifically, the image sensor) from the state at time t1, the recognition area in the image sensor is, for example, in the upper left direction in FIG. 278. It moves, and at time t2, it goes out of the display area. That is, the image of the target area (for example, the image of a poster) protrudes from the captured display image displayed on the display 201. However, the recognition area in the image sensor does not extend beyond the effective pixel area. That is, the receiver 200 has acquired the entire captured image including the target region even at time t2. As a result, the receiver 200 can recognize the target area from all the captured images.

Therefore, in the example shown in FIG. 278, the receiver 200 has a display magnification of the AR image so that the size of the entire AR image matches the size of a part of the target area in the captured display image. To change. That is, the receiver 200 reduces the AR image. Then, the receiver 200 superimposes the AR image whose display magnification has been changed (that is, reduced) on the area and displays it on the display 201. Further, the receiver 200 changes the display magnification and the position of the displayed AR image according to the movement of the recognition area in the image sensor, that is, the movement of the target area in all the captured images.

Further, when the recognition area extends beyond the display area as described above, the receiver 200 compares the number of pixels, whichever is smaller of the number of pixels dw and dh, with the threshold value N. Then, for example, at time t2, if the smaller number of pixels is equal to or less than the threshold value N, the receiver 200 is fixed without changing the display magnification and position of the AR image according to the position of the recognition area in the image sensor. do. That is, the receiver 200 switches the screen display of the AR image. For example, the receiver 200 fixes the display magnification and position of the displayed AR image to the display magnification and position of the AR image displayed on the display 201 when the smaller number of pixels reaches the threshold value N. ..

Therefore, even if the recognition area moves further at time t3 and extends beyond the effective pixel area, the receiver 200 continues to display the AR image in the same manner as at time t2. That is, as long as the smaller number of pixels among the number of pixels dw and dh is equal to or less than the threshold value N, the receiver 200 uses the AR image having a fixed display magnification and position as the captured display image as in the case of time t2. Continue to display in superimposition.

In the above example, the smaller of the pixel numbers dw and dh is compared with the threshold value, but the ratio of the smaller number of pixels and the threshold value may be compared. The ratio of the number of pixels dw is, for example, the ratio of the number of pixels dw (dw / w0) to the number of pixels w0 in the horizontal direction in the effective pixel area. Similarly, the ratio of the number of pixels dh is, for example, the ratio of the number of pixels dh (dh / h0) to the number h0 of pixels in the vertical direction of the effective pixel area. Alternatively, instead of the number of pixels in the horizontal or vertical direction of the effective pixel area, the number of pixels in the horizontal or vertical direction of the display area may be used to express the respective ratios of the number of pixels dw and dh. The threshold value compared with the ratio of the number of pixels dw and dh is, for example, 0.05.

Further, the smaller angle of view of the number of pixels dw and dh may be compared with the threshold value. When the number of pixels on the diagonal line of the effective pixel area is m and the angle of view corresponding to the diagonal line is θ (for example, 55 °), the angle of view corresponding to the number of pixels dw is θ × dw / m. The angle of view corresponding to the number of pixels dh is θ × dh / m.

Further, in the example shown in FIGS. 277 and 278, the receiver 200 switches the screen display of the AR image based on the distance between the effective pixel areas and the recognition area, but the display area and the recognition area are used. The screen display of the AR image may be switched based on the relationship of.

FIG. 279 is a diagram showing another example in which the receiver 200 in the second modification of the second embodiment displays an AR image. Specifically, FIG. 279 shows an example of switching the screen display of the AR image based on the relationship between the display area and the recognition area. Further, in the example shown in FIG. 279, the image sensor of the receiver 200 has an effective pixel area wider than the display area, as in the example shown in FIG. 273.

For example, the receiver 200 takes an image of the subject at time t1. As a result, the receiver 200 displays only the image obtained by the display area narrower than the effective pixel area among all the captured images obtained by the effective pixel area of the image sensor on the display 201 as the captured display image. At this time, the receiver 200 recognizes the area corresponding to the recognition information acquired based on the optical ID among all the captured images as the target area on which the AR image is superimposed, as described above. Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201.

Here, when the user changes the orientation of the receiver 200, the receiver 200 changes the position of the displayed AR image according to the movement of the recognition area in the image sensor. Then, for example, the recognition area in the image sensor moves toward the upper left in FIG. 279, and at time t2, a part of the edge of the recognition area and a part of the edge of the display area coincide with each other. That is, an image of the target area (for example, an image such as a poster) is arranged in the corner of the captured display image displayed on the display 201. As a result, the receiver 200 superimposes the AR image on the target area in the corner of the captured display image and displays it on the display 201.

Then, when the recognition area moves further and extends beyond the display area, the receiver 200 fixes the size and position of the AR image displayed at time t2 without changing the size and position. That is, the receiver 200 switches the screen display of the AR image.

Therefore, even if the recognition area moves further at time t3 and extends beyond the effective pixel area, the receiver 200 continues to display the AR image in the same manner as at time t2. That is, as long as the recognition area extends beyond the display area, the receiver 200 superimposes the AR image of the same size as at time t2 on the same position as at time t2 in the captured display image. Continue to display.

As described above, in the example shown in FIG. 279, the receiver 200 switches the screen display of the AR image depending on whether or not the recognition area protrudes from the display area. Further, the receiver 200 may include a display area and use a determination area larger than the display area and smaller than the effective pixel area instead of the display area. In this case, the receiver 200 switches the screen display of the AR image depending on whether or not the recognition area extends beyond the determination area.

Although the screen display of the AR image has been described above with reference to FIGS. 273 to 279, when the receiver 200 cannot recognize the target area from all the captured images, the target recognized until immediately before that. An AR image of the size of the region may be superimposed on the captured display image and displayed.

FIG. 280 is a diagram showing another example in which the receiver 200 in the second modification of the second embodiment displays an AR image.

In the example shown in FIG. 243, the receiver 200 acquires the captured display image Pe and the decoding image in the same manner as described above by imaging the guide plate 107 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the guide plate 107. However, if the entire surface of the guide plate 107 is a color that absorbs light (for example, a dark color), the surface is dark even when illuminated by the transmitter 100, so that the receiver 200 correctly receives the optical ID. It may not be possible. Alternatively, even if the entire surface of the guide plate 107 has a striped pattern such as a decoding image (that is, a bright line image), the receiver 200 may not be able to correctly receive the optical ID.

Therefore, as shown in FIG. 280, the reflector 109 may be arranged near the guide plate 107. As a result, the receiver 200 can receive the light reflected from the transmitter 100 by the reflector 109, that is, the visible light (specifically, the optical ID) transmitted from the transmitter 100. As a result, the receiver 200 can appropriately receive the optical ID and display the AR image P5.

[Summary of Modifications 1 and 2 of Embodiment 23]
FIG. 281A is a flowchart showing a display method according to one aspect of the present invention.

The display method according to one aspect of the present invention includes steps S41 to S43.

In step S41, an captured image is acquired by taking an image of an object lit up by a transmitter that transmits a signal due to a change in the brightness of light as a subject by an image pickup sensor. In step S42, the signal is decoded from the captured image. In step S43, the moving image corresponding to the decoded signal is read out from the memory, and the moving image is superimposed on the target area corresponding to the subject in the captured image and displayed on the display. Here, in step S43, a predetermined image that is any one of the plurality of images included in the moving image and is before and after the display time of the image including the object and the image including the object. The moving image is displayed from any of the plurality of images. For example, the predetermined number is 10 frames. Alternatively, the object is a still image, and in step S43, the moving image is displayed from the same image as the still image. The image from which the moving image is started to be displayed is not limited to the same image as the still image, but is before or after a predetermined number of frames in the display order from the same image as the still image, that is, the image including the object. It may be an image. Further, the object is not limited to a still image, but may be a doll or the like.

The image sensor and the captured image are, for example, the image sensor and all captured images in the 23rd embodiment. Further, the still image to be lit up may be a still image displayed on the display panel of the image display device, or may be a poster, a guide plate, a signboard, or the like illuminated by the light from the transmitter.

Further, such a display method may further include a transmission step of transmitting a signal to the server and a reception step of receiving a moving image corresponding to the signal from the server.

As a result, as shown in FIG. 265, for example, a moving image can be displayed in a virtual reality so that a still image starts to move, and an image useful to the user can be displayed.

Further, the still image has an outer frame of a predetermined color, and the display method according to one aspect of the present invention may further include a recognition step of recognizing a target area from a captured image by the predetermined color. In this case, in step S43, the moving image is resized so as to be the same as the size of the recognized target area, and the resized moving image is superimposed on the target area in the captured image and displayed on the display. good. For example, the outer frame of a predetermined color is a white or black rectangular frame surrounding a still image, and is indicated by the recognition information in the 23rd embodiment. Then, the AR image in the 23rd embodiment is resized and superimposed as a moving image.

As a result, the moving image can be displayed more realistically so that the moving image actually exists as a subject.

Further, of the image pickup area of the image pickup sensor, only the image projected on the display area which is smaller than the image pickup area is displayed on the display. In this case, in step S43, when the projection area in which the subject is projected in the image pickup area is larger than the display area, the image obtained by the portion of the projection area beyond the display area is displayed on the display. You don't have to. Here, for example, as shown in FIG. 273, the imaging region and the projection region are the effective pixel region and the recognition region of the image sensor.

As a result, for example, as shown in FIG. 273, even if a part of the image obtained by the projection area (recognition area of FIG. 273) is not displayed on the display due to the image sensor approaching the still image which is the subject, the subject. The entire still image may be projected onto the imaging region. Therefore, in this case, the still image as the subject can be appropriately recognized, and the moving image can be appropriately superimposed on the target area corresponding to the subject in the captured image.

Further, for example, the horizontal and vertical widths of the display area are w1 and h1, and the horizontal and vertical widths of the projection area are w2 and h2, respectively. In this case, in step S43, if the larger value of h2 / h1 or w2 / w1 is greater than or equal to the predetermined value, the moving image is displayed on the full screen of the display, and either h2 / h1 or w2 / w1 is displayed. When the larger value is smaller than the predetermined value, the moving image may be superimposed on the target area in the captured image and displayed on the display.

As a result, as shown in FIG. 275, for example, when the image pickup sensor approaches the still image as the subject, the moving image is displayed on the full screen. Therefore, the user moves the image pickup sensor closer to the still image to enlarge the moving image. There is no need to display it. Therefore, it is possible to prevent the signal from being unable to be decoded because the image pickup sensor is brought too close to the still image and the projection area (recognition area in FIG. 275) protrudes from the image pickup area (effective pixel area). ..

Further, the display method according to one aspect of the present invention may further include a control step of turning off the operation of the image pickup sensor when displaying a moving image on the full screen of the display.

Thereby, for example, as shown in step S314 of FIG. 276, the power consumption of the image pickup sensor can be suppressed by turning off the operation of the image pickup sensor.

Further, in step S43, when the target area cannot be recognized from the captured image due to the movement of the image pickup sensor, the moving image is displayed in the same size as the size of the target area recognized immediately before the image cannot be recognized. You may. Note that the target area cannot be recognized from the captured image is, for example, a situation in which at least a part of the target area corresponding to the still image as the subject is not included in the captured image. In this way, when the target area cannot be recognized, a moving image having the same size as the size of the target area recognized immediately before is displayed, for example, at time t3 in FIG. 279. Therefore, it is possible to prevent at least a part of the moving image from being displayed because the image pickup sensor has been moved.

Further, in step S43, when the image pickup sensor moves and only a part of the target area is included in the area displayed on the display in the captured image, it corresponds to a part of the target area. A part of the spatial area of the moving image may be superimposed on a part of the target area and displayed on the display. A part of the spatial area of the moving image is a part of each picture constituting the moving image.

As a result, only a part of the spatial region of the moving image (AR image of FIG. 277) is displayed on the display, for example, at the time t2 of FIG. 277. As a result, it is possible to inform the user that the image pickup sensor is not properly aimed at the still image as the subject.

Further, in step S43, when the target area cannot be recognized from the captured image due to the movement of the image pickup sensor, the space of the moving image corresponding to a part of the target area displayed immediately before the target area cannot be recognized. A part of the area may be continuously displayed.

As a result, even when the user points the image sensor in a direction different from the still image as the subject, as in the case of time t3 in FIG. 277, one of the spatial regions of the moving image (AR image in FIG. 277). The part is displayed continuously. As a result, it is possible for the user to easily understand how to point the image sensor to display the entire moving image.

Further, in step S43, the horizontal and vertical widths of the image pickup area of the image pickup sensor are w0 and h0, respectively, and the horizontal direction between the projection area on which the subject is projected in the image pickup area and the image pickup area thereof. When the distances in the direction and the vertical direction are dh and dw, and the smaller value of dw / w0 or dh / h0 is equal to or less than a predetermined value, it is determined that the target area cannot be recognized. good. The projection area is, for example, a recognition area shown in FIG. 277. Alternatively, in step S43, the angle of view corresponding to the shorter of the horizontal and vertical distances between the projection area on which the subject is projected in the image pickup area of the image pickup sensor and the image pickup area is predetermined. If it is less than or equal to the value, it may be determined that the target area cannot be recognized.

This makes it possible to appropriately determine whether or not the target area can be recognized.

FIG. 281B is a block diagram showing a configuration of a display device according to an aspect of the present invention.

The display device A10 according to one aspect of the present invention includes an image pickup sensor A11, a decoding unit A12, and a display control unit A13.

The image pickup sensor A11 acquires a captured image by capturing a still image lit up by a transmitter that transmits a signal due to a change in the brightness of light as a subject.

The decoding unit A12 decodes the signal from the captured image.

The display control unit A13 reads out the moving image corresponding to the decoded signal from the memory, superimposes the moving image on the target area corresponding to the subject in the captured image, and displays the moving image on the display. Here, the display control unit A13 displays the plurality of images in order from the first image, which is the same image as the still image, among the plurality of images included in the moving image.

Thereby, the same effect as the above-mentioned display method can be obtained.

Further, the image pickup sensor A11 may have a plurality of micromirrors and a photo sensor, and the display device A10 may further include an image pickup control unit that controls the image pickup sensor. In this case, the image pickup control unit specifies a region including a signal in the captured image as a signal region, and controls the angle of the micromirror corresponding to the specified signal region among the plurality of micromirrors. Then, the image pickup control unit causes the above-mentioned photo sensor to receive only the light reflected by the micromirror whose angle is controlled among the plurality of micromirrors.

Thereby, for example, as shown in FIG. 232A, even if the visible light signal, which is a signal represented by the change in the luminance of light, contains a high frequency component, the high frequency component can be correctly decoded.

In each of the above embodiments and modifications, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. For example, the program causes the computer to execute the display method shown by the flowcharts of FIGS. 271, 274, 276 and 281A.

Although the display method according to one or more embodiments has been described above based on each of the above embodiments and modifications, the present invention is not limited to this embodiment. As long as the gist of the present invention is not deviated, various modifications that can be conceived by those skilled in the art are applied to the present embodiment, and forms constructed by combining components in different embodiments and modifications are also within the scope of the present invention. May be included in.

[Modification 3 of Embodiment 23]
Hereinafter, a modification 3 of the embodiment 23, that is, a modification 3 of the display method for realizing AR using an optical ID will be described.

FIG. 282 is a diagram showing an example of enlargement and movement of an AR image.

As shown in FIG. 282 (a), the receiver 200 superimposes the AR image P21 on the target area of the captured display image Ppre, as in the above embodiment 23 or its modified example 1 or 2. Then, the receiver 200 displays the captured display image Ppre on which the AR image P21 is superimposed on the display 201. For example, the AR image P21 is a moving image.

Here, as shown in FIG. 282 (b), when the receiver 200 receives the size change instruction, the receiver 200 changes the size of the AR image P21 according to the instruction. For example, when the receiver 200 receives an instruction for enlargement, the receiver 200 enlarges the AR image P21 in response to the instruction. The instruction to change the size is given by, for example, a pinch operation, a double tap, or a long press on the AR image P21 by the user. Specifically, when the receiver 200 receives an instruction for enlargement performed by pinch-out, the receiver 200 enlarges the AR image P21 in response to the instruction. On the contrary, when the receiver 200 receives the reduction instruction performed by the pinch-in, the receiver 200 reduces the AR image P21 in response to the instruction.

Further, as shown in FIG. 282 (c), when the receiver 200 receives an instruction to change the position, the receiver 200 changes the position of the AR image P21 according to the instruction. The position change instruction is given by the user, for example, by swiping the AR image. Specifically, when the receiver 200 receives an instruction for changing the position performed by swiping, the receiver 200 changes the position of the AR image P21 according to the instruction. That is, the AR image P21 moves.

As a result, the AR image can be made easier to see by enlarging the AR image which is a moving image, and the area of the captured display image Ppre hidden in the AR image can be obtained by reducing or moving the AR image which is a moving image. It can be displayed to the user.

FIG. 283 is a diagram showing an example of enlargement of an AR image.

As shown in FIG. 283 (a), the receiver 200 superimposes the AR image P22 on the target area of the captured display image Ppre, as in the above embodiment 23 or its modification 1 or 2. Then, the receiver 200 displays the captured display image Ppre on which the AR image P22 is superimposed on the display 201. For example, the AR image P22 is a still image in which a character string is described.

Here, as shown in FIG. 283 (b), when the receiver 200 receives the size change instruction, the receiver 200 changes the size of the AR image P22 according to the instruction. For example, when the receiver 200 receives an instruction for enlargement, the receiver 200 enlarges the AR image P22 in response to the instruction. Similar to the above, the size change instruction is given by the user, for example, by pinching, double-tapping, or long-pressing the AR image P22. Specifically, when the receiver 200 receives an instruction for enlargement performed by pinch-out, the receiver 200 enlarges the AR image P22 in response to the instruction. By enlarging the AR image P22, the character string described in the AR image P22 can be made easier for the user to read.

Further, as shown in FIG. 283 (c), when the receiver 200 further receives the size change instruction, the receiver 200 changes the size of the AR image P22 according to the instruction. For example, when the receiver 200 receives an instruction for further enlargement, the receiver 200 further enlarges the AR image P22 in response to the instruction. By enlarging the AR image P22, the character string described in the AR image P22 can be made easier to read by the user.

The receiver 200 may acquire a high-resolution AR image when the enlargement ratio of the AR image in response to the instruction is equal to or higher than the threshold value when the instruction for enlargement is received. In this case, the receiver 200 may enlarge the high-resolution AR image to the above-mentioned enlargement ratio and display it instead of the original AR image already displayed. For example, the receiver 200 displays an AR image of 1920 × 1080 pixels instead of an AR image of 640 × 480 pixels. As a result, the AR image can be magnified and a high-resolution image that cannot be obtained by the optical zoom can be displayed so that the AR image is actually captured as a subject.

FIG. 284 is a flowchart showing an example of a processing operation related to enlargement and movement of the AR image by the receiver 200.

First, the receiver 200 starts imaging with the normal exposure time and the communication exposure time in the same manner as in step S101 shown in the flowchart of FIG. 239 (step S401). When this imaging is started, the captured display image Ppre with the normal exposure time and the decoding image (that is, the emission line image) Pdec with the communication exposure time are periodically obtained. Then, the receiver 200 acquires the optical ID by decoding the decoding image Pdec.

Next, the receiver 200 performs AR image superimposition processing including the processing of steps S102 to S106 shown in the flowchart of FIG. 239 (step S402). When this AR image superimposition processing is performed, the AR image is superimposed and displayed on the captured display image Ppre. At this time, the receiver 200 lowers the optical ID acquisition rate (step S403). The optical ID acquisition rate is the ratio of the number of decoding images (that is, bright line images) Pdec to the number of captured images per unit time obtained by the imaging started in step S401. For example, as the optical ID acquisition rate decreases, the number of decoding image Pdecs obtained per unit time becomes smaller than the number of captured display image Ppres obtained per unit time.

Next, the receiver 200 determines whether or not the size change instruction has been received (step S404). Here, if it is determined that the size change instruction has been accepted (Yes in step S404), the receiver 200 further determines whether or not the size change instruction is an expansion instruction (step S405). When it is determined that the size change instruction is an enlargement instruction (Yes in step S405), the receiver 200 further determines whether or not the AR image needs to be reacquired (step S406). For example, when the receiver 200 determines that the enlargement ratio of the AR image in response to the enlargement instruction becomes equal to or higher than the threshold value, the receiver 200 determines that the AR image needs to be reacquired. Here, when the receiver 200 determines that re-acquisition is necessary (Yes in step S406), the receiver 200 acquires a high-resolution AR image from, for example, a server, and superimposes the displayed AR image on the high-resolution AR image. Replace with an AR image (step S407).

Then, the receiver 200 changes the size of the AR image according to the received size change instruction (step S408). That is, when the high-resolution AR image is acquired in step S407, the receiver 200 enlarges the high-resolution AR image. If it is determined in step S406 that reacquisition of the AR image is unnecessary (No in step S406), the receiver 200 enlarges the superimposed AR image. Further, if it is determined in step S405 that the size change instruction is a reduction instruction (No in step S405), the receiver 200 superimposes according to the accepted size change instruction, that is, the reduction instruction. Reduce the displayed AR image.

On the other hand, if the receiver 200 determines in step S404 that the size change instruction has not been received (No in step S404), it determines whether or not the position change instruction has been received (step S409). Here, if it is determined that the position change instruction has been accepted (Yes in step S409), the receiver 200 changes the position of the AR image superimposed and displayed in response to the position change instruction (step S410). ). That is, the receiver 200 moves the AR image. Further, if it is determined that the position change instruction is not accepted (No in step S409), the receiver 200 repeatedly executes the process from step S404.

When the size of the AR image is changed in step S408 or the position of the AR image is changed in step S410, the receiver 200 does not acquire the optical ID periodically acquired from step S401. It is determined whether or not the image is present (step S411). Here, when it is determined that the optical ID is no longer acquired (Yes in step S411), the receiver 200 ends the processing operation related to the enlargement and movement of the AR image. On the other hand, if it is determined that the optical ID is still acquired (No in step S411), the receiver 200 repeatedly executes the process from step S404.

FIG. 285 is a diagram showing an example of superimposition of AR images by the receiver 200.

As described above, the receiver 200 superimposes the AR image P23 on the target area in the captured display image Ppre. Here, as shown in FIG. 285, the AR image P23 is configured so that the closer each part of the AR image P23 is to the end of the AR image P23, the higher the transmittance at that part. The transmittance is the degree to which the superimposed image is displayed through. For example, when the total transmittance of the AR image is 100%, even if the AR image is superimposed on the target area of the captured display image, the AR image is not displayed on the display 201 and only the target area is displayed. It means that. On the contrary, when the total transmittance of the AR image is 0%, it means that the target area of the captured display image is not displayed on the display 201, and only the AR image superimposed on the target area is displayed. ..

For example, when the AR image P23 has a rectangular shape, the transmittance of each portion in the AR image P23 is higher as the portion is closer to the upper end, the lower end, the left end, or the right end of the rectangle. More specifically, the transmittance at their edges is 100%. Further, in the central portion of the AR image P23, there is a rectangular area having a transmittance of 0%, which is smaller than that of the AR image P23, and the rectangular area is described in English as, for example, "Kyoto Station". That is, at the peripheral portion of the AR image P23, the transmittance changes stepwise from 0% to 100% like a gradation.

The receiver 200 superimposes such an AR image P23 on a target region in the captured display image Ppre as shown in FIG. 285. At this time, the receiver 200 adjusts the size of the AR image P23 to the size of the target area, and superimposes the resized AR image P23 on the target area. For example, in the target area, an image of a station name marker having the same background color as the rectangular area in the central portion of the AR image P23 appears. In addition, "Kyoto" is written in Japanese on the station name sign.

Here, as described above, the transmittance of each part of the AR image P23 is higher as the part is closer to the end of the AR image P23. Therefore, when the AR image P23 is superimposed on the target area, even if the rectangular area in the central portion of the AR image P23 is displayed, the end of the AR image P23 is not displayed, and the end of the target area, that is, the station name mark. The edges of the image are displayed.

As a result, the deviation between the AR image P23 and the target area can be made inconspicuous. That is, even if the AR image P23 is superimposed on the target area, a gap may occur between the AR image P23 and the target area due to the movement of the receiver 200 or the like. In this case, if the total transmittance of the AR image P23 is 0%, the edge of the AR image P23 and the edge of the target area are displayed, and the deviation becomes conspicuous. However, in the AR image P23 in this modification, the closer the portion is to the end, the higher the transmittance of that portion is, so that the end of the AR image P23 can be made difficult to be displayed. It is possible to make the gap between them inconspicuous. Further, since the transmittance changes like a gradation at the peripheral portion of the AR image P23, it is possible to make it less noticeable that the AR image P23 is superimposed on the target region.

FIG. 286 is a diagram showing an example of superimposition of AR images by the receiver 200.

As described above, the receiver 200 superimposes the AR image P24 on the target area in the captured display image Ppre. Here, as shown in FIG. 286, the subject to be imaged is, for example, a menu of a restaurant. This menu is surrounded by a white frame, and the white frame is also surrounded by a black frame. That is, the subject includes a menu, a white frame surrounding the menu, and a black frame surrounding the white frame.

The receiver 200 recognizes a region of the captured display image Ppre that is larger than the image of the white frame and smaller than the image of the black frame as the target region. Then, the receiver 200 adjusts the size of the AR image P24 to the size of the target area, and superimposes the resized AR image P24 on the target area.

As a result, even if the superimposed AR image P24 is displaced from the target area due to the movement of the receiver 200 or the like, the AR image P24 can be continuously displayed in a state surrounded by a black frame. Therefore, the deviation between the AR image P24 and the target area can be made inconspicuous.

In the example shown in FIG. 286, the color of the frame is black or white, but the color is not limited to these colors and may be any color.

FIG. 287 is a diagram showing an example of superimposition of AR images by the receiver 200.

For example, the receiver 200 takes an image of a poster on which a castle illuminated in the night sky is drawn as a subject. For example, this poster is illuminated by the above-mentioned transmitter 100 configured as a backlight, and the backlight transmits a visible light signal (that is, an optical ID). The receiver 200 acquires the captured display image Ppre including the image of the subject as the poster and the AR image P25 corresponding to the optical ID by the imaging. Here, the AR image P25 has the same shape as the image of the poster from which the above-mentioned area where the castle is drawn is extracted. That is, the area corresponding to the castle of the poster image in the AR image P25 is masked. Further, the AR image P25 is configured so that, like the AR image P23 described above, the closer each part of the AR image P25 is to the end of the AR image P25, the higher the transmittance at that part. Further, in the central portion of the AR image P25 where the transmittance is 0%, the fireworks launched in the night sky are displayed as a moving image.

The receiver 200 superimposes the resized AR image P25 on the target area in accordance with the size of the target area which is an image of the subject. As a result, the castle drawn on the poster is displayed as an image of the subject, not as an AR image, and a moving image of fireworks is displayed as an AR image.

This makes it possible to display the captured display image Ppre as if the fireworks were actually launched in the poster. Further, the transmittance of each part of the AR image P25 is higher as the part is closer to the end of the AR image P25. Therefore, when the AR image P25 is superimposed on the target area, even if the central portion of the AR image P25 is displayed, the end of the AR image P25 is not displayed, and the end of the target area is displayed. As a result, the deviation between the AR image P25 and the target area can be made inconspicuous. Further, since the transmittance changes like a gradation at the peripheral portion of the AR image P25, it is possible to make it less noticeable that the AR image P25 is superimposed on the target region.

FIG. 288 is a diagram showing an example of superimposition of AR images by the receiver 200.

For example, the receiver 200 takes an image of the transmitter 100 configured as a television as a subject. Specifically, the transmitter 100 displays a castle lit up in the night sky on a display and transmits a visible light signal (that is, an optical ID). The receiver 200 acquires an image capture display image Ppre on which the transmitter 100 is projected and an AR image P26 corresponding to the optical ID by the image pickup. Here, the receiver 200 first displays the captured display image Ppre on the display 201. At this time, the receiver 200 also displays a message m prompting the user to turn off the light on the display 201. Specifically, the message m is, for example, "Turn off the lights in the room to dim the room."

When the user is turned off by the display of the message m and the room in which the transmitter 100 is installed becomes dark, the receiver 200 superimposes the AR image P26 on the captured display image Ppre and displays it. Here, the AR image P26 has the same size as the captured display image Ppre, and the region corresponding to the castle of the captured display image Ppre in the AR image P26 is hollowed out. That is, in the AR image P26, the region corresponding to the castle of the captured display image Ppre is masked. Therefore, the castle of the captured display image Ppre can be shown to the user from that area. Further, in the peripheral portion of the region in the AR image P26, the transmittance may be gradually changed from 0% to 100% like a gradation, as described above. In this case, the deviation between the captured display image Ppre and the AR image P26 can be made inconspicuous.

In the above example, the AR image having a high transmittance at the peripheral portion is superimposed on the target area of the captured display image Ppre to make the deviation between the AR image and the target area less noticeable. However, instead of such an AR image, an AR image having the same size as the captured display image Ppre and being entirely translucent (that is, having a transmittance of 50%) may be superimposed on the captured display image Ppre. Even in this case, the deviation between the AR image and the target area can be made inconspicuous. Further, when the captured display image Ppre is generally bright, the AR image having low transparency is uniformly superimposed on the captured display image Ppre, and conversely, when the captured display image Ppre is totally dark, the transparency is uniformly uniform. The AR image with a high value may be superimposed on the captured display image Ppre.

Objects such as fireworks of AR image P25 and AR image P26 may be represented by CG (computer graphics). In this case, masking can be eliminated. Further, in the example shown in FIG. 288, the receiver 200 displays the message m prompting the user to turn off the lights, but the receiver 200 may automatically turn off the lights without displaying such a message. For example, the receiver 200 outputs an extinguishing signal to a lighting device in which the transmitter 100, which is a television, is set by Bluetooth (registered trademark), ZigBee, a specified low power radio station, or the like. As a result, the lighting device is automatically turned off.

FIG. 289A is a diagram showing an example of an image pickup display image Ppre obtained by image pickup by the receiver 200.

For example, the transmitter 100 is configured as a large display installed in a stadium. Then, the transmitter 100 displays a message indicating that, for example, an order for fast food and a drink can be ordered by an optical ID, and transmits a visible light signal (that is, an optical ID). When such a message is displayed, the user points the receiver 200 toward the transmitter 100 and performs an image pickup. That is, the receiver 200 takes an image of the transmitter 100 configured as a large display installed in the stadium as a subject.

The receiver 200 acquires the captured display image Ppre and the decoding image Pdec by the imaging thereof. Then, the receiver 200 acquires the optical ID by decoding the decoding image Pdec, and transmits the optical ID and the captured display image Ppre to the server.

For each optical ID, the server specifies the installation information of the large imaged display associated with the optical ID transmitted from the receiver 200 from the installation information associated with the optical ID. For example, the installation information indicates the position and orientation in which the large display is installed, the size of the large display, and the like. Further, the server identifies the number of the seat where the captured display image Ppre was imaged in the stadium based on the size and orientation of the large display displayed on the captured display image Ppre and the installation information thereof. do. Then, the server causes the receiver 200 to display a menu screen including the seat number.

FIG. 289B is a diagram showing an example of a menu screen displayed on the display 201 of the receiver 200.

The menu screen m1 includes, for example, an input field ma1 in which the number of orders for the product is input, a seat field mb1 in which the seat number of the stadium specified by the server is described, and an order button mc1 for each product. .. By operating the receiver 200, the user inputs the number of orders for the product in the input field ma1 corresponding to the desired product, and selects the order button mc1. As a result, the order is confirmed, and the receiver 200 transmits the order contents according to the input result to the server.

Upon receiving the order details, the server instructs the stadium staff to deliver the ordered number of products according to the order details to the seat with the specified number as described above.

FIG. 290 is a flowchart showing an example of the processing operation between the receiver 200 and the server.

The receiver 200 first takes an image of the transmitter 100 configured as a large display in the stadium (step S421). The receiver 200 acquires the optical ID transmitted from the transmitter 100 by decoding the decoding image Pdec obtained by the imaging (step S422). The receiver 200 transmits the optical ID acquired in step S422 and the captured display image Ppre obtained by the imaging in step S421 to the server (step S423).

When the server receives the optical ID and the captured display image Ppre (step S424), the server identifies the installation information of the large display installed in the stadium based on the optical ID (step S425). For example, the server holds a table showing the installation information of the large display associated with the optical ID for each optical ID, and the installation information associated with the optical ID transmitted from the receiver 200 is obtained from the table. Identify the installation information by searching.

Next, the server acquires (ie, captures) the captured display image Ppre at the stadium based on the specified installation information and the size and orientation of the large display shown in the captured display image Ppre. Identify the number of the seat that was lost (step S426). Then, the server transmits the URL (Uniform Resource Locator) of the menu screen m1 including the specified seat number to the receiver 200 (step S427).

When the receiver 200 receives the URL of the menu screen m1 transmitted from the server (step S428), the receiver 200 accesses the URL and displays the menu screen m1 (step S429). Here, the user inputs the order contents into the menu screen m1 by operating the receiver 200, and confirms the order by selecting the order button mc1. As a result, the receiver 200 transmits the order contents to the server (step S430).

When the server receives the order content transmitted from the receiver 200, the server performs order processing according to the order content (step S431). At this time, the server, for example, instructs the stadium staff to deliver the ordered number of products according to the order contents to the seats with the numbers specified in step S426.

In this way, since the seat number is specified based on the captured display image Ppre obtained by the image pickup by the receiver 200, the user of the receiver 200 bother to input the seat number when ordering the product. You don't have to. Therefore, the user can easily place an order for a product without inputting a seat number.

In the above example, the server specifies the seat number, but the receiver 200 may specify the seat number. In this case, the receiver 200 acquires the installation information from the server and specifies the seat number based on the installation information and the size and orientation of the large display displayed in the captured display image Ppre.

FIG. 291 is a diagram for explaining the volume of the sound reproduced by the receiver 1800a.

Similar to the example shown in FIG. 123, the receiver 1800a receives an optical ID (visible light signal) transmitted from the transmitter 1800b configured as, for example, a street digital signage. Then, the receiver 1800a reproduces the sound at the same timing as the image reproduction by the transmitter 1800b. That is, the receiver 1800a reproduces the sound in synchronization with the image reproduced by the transmitter 1800b. The receiver 1800a may reproduce the same image as the image (reproduced image) reproduced by the transmitter 1800b, or an AR image (AR moving image) related to the reproduced image together with the sound.

Here, when the receiver 1800a reproduces the sound as described above, the receiver 1800a adjusts the volume of the sound according to the distance to the transmitter 1800b. Specifically, the receiver 1800a adjusts the volume to be smaller as the distance to the transmitter 1800b is longer, and conversely, the volume is adjusted to be larger as the distance to the transmitter 1800b is shorter.

The receiver 1800a may specify the distance to the transmitter 1800b by using GPS (Global Positioning System) or the like. Specifically, the receiver 1800a acquires the position information of the transmitter 1800b associated with the optical ID from a server or the like, and further specifies the position of the receiver 1800a by GPS. Then, the receiver 1800a specifies the distance between the position of the transmitter 1800b indicated by the position information acquired from the server and the position of the specified receiver 1800a as the distance to the transmitter 1800b described above. .. The receiver 1800a may specify the distance to the transmitter 1800b by using Bluetooth (registered trademark) or the like instead of GPS.

Further, the receiver 1800a may specify the distance to the transmitter 1800b based on the size of the emission line pattern region of the above-mentioned decoding image Pdec obtained by imaging. The emission line pattern region is a region consisting of a plurality of emission line patterns appearing by exposure of a plurality of exposure lines included in the image sensor of the receiver 1800a at the communication exposure time, as in the examples shown in FIGS. 245 and 246. This emission line pattern region corresponds to the region of the display of the transmitter 1800b displayed on the captured display image Ppre. Specifically, the receiver 1800a specifies a shorter distance as the distance to the transmitter 1800b as the emission line pattern region is larger, and conversely, the longer distance is specified as the distance to the transmitter 1800b as the emission line pattern region is smaller. .. Further, the receiver 1800a uses distance data indicating the relationship between the size of the emission line pattern region and the distance, and in the distance data, the distance associated with the size of the emission line pattern region in the captured display image Ppre is set. It may be specified as the distance to the transmitter 1800b. The receiver 1800a may transmit the received optical ID as described above to the server and acquire the distance data associated with the optical ID from the server.

In this way, since the volume is adjusted according to the distance to the transmitter 1800b, the user of the receiver 1800a can make the sound reproduced by the receiver 1800a like the sound actually reproduced by the transmitter 1800b. Can be heard.

FIG. 292 is a diagram showing the relationship between the distance from the receiver 1800a to the transmitter 1800b and the volume.

For example, when the distance to the transmitter 1800b is between L1 and L2 [m], the volume increases or decreases in proportion to the distance in the range from Vmin to Vmax [dB]. Specifically, the receiver 1800a linearly reduces the volume from Vmax [dB] to Vmin [dB] when the distance to the transmitter 1800b increases from L1 [m] to L2 [m]. Further, even if the distance to the transmitter 1800b is shorter than L1 [m], the receiver 1800a maintains the volume at Vmax [dB], and the distance to the transmitter 1800b is longer than L2 [m]. However, the volume is maintained at Vmin [dB].

As described above, the receiver 1800a stores the maximum volume Vmax, the longest distance L1 in which the sound of the maximum volume Vmax is output, the minimum volume Vmin, and the shortest distance L2 in which the sound of the minimum volume Vmin is output. is doing. Further, the receiver 1800a may change its maximum volume Vmax, minimum volume Vmin, longest distance L1 and shortest distance L2 according to the attributes set by itself. For example, when the attribute is the age of the user and the age indicates old age, the receiver 1800a sets the maximum volume Vmax to be larger than the reference maximum volume and the minimum volume Vmin to be larger than the reference minimum volume. May be good. Further, the attribute may be information indicating whether the sound output is performed from the speaker or the earphone.

As described above, since the minimum volume Vmin is set in the receiver 1800a, it is possible to suppress the inability to hear the sound because the receiver 1800a is too far from the transmitter 1800b. Further, since the maximum volume Vmax is set in the receiver 1800a, it is possible to prevent the receiver 1800a from being output with an unnecessarily loud sound because the receiver 1800a is too close to the transmitter 1800b.

FIG. 293 is a diagram showing an example of superimposition of AR images by the receiver 200.

The receiver 200 captures the illuminated signboard. Here, the signboard is lit up by the lighting device, which is the above-mentioned transmitter 100 that transmits the optical ID. Therefore, the receiver 200 acquires the captured display image Ppre and the decoding image Pdec by the imaging thereof. Then, the receiver 200 acquires the optical ID by decoding the decoding image Pdec, and acquires the plurality of AR images P27a to P27c associated with the optical ID and the recognition information from the server. Based on the recognition information, the receiver 200 recognizes the periphery of the area m2 on which the signboard of the captured display image Ppre is projected as the target area.

Specifically, as shown in FIG. 293 (a), the receiver 200 recognizes the region in contact with the left side of the region m2 as the first target region, and superimposes the AR image P27a on the first target region. do.

Next, as shown in FIG. 293 (b), the receiver 200 recognizes the region including the lower side of the region m2 as the second target region, and superimposes the AR image P27b on the second target region. ..

Next, as shown in FIG. 293 (c), the receiver 200 recognizes the region in contact with the upper side of the region m2 as the third target region, and superimposes the AR image P27c on the third target region.

Here, each of the AR images P27a to P27c is, for example, an image of a character of Yukio, and may be a moving image.

Further, the receiver 200 switches the recognized target area to any one of the first to third target areas in a predetermined order and timing while continuously and repeatedly acquiring the optical ID. You may. That is, the receiver 200 may switch the recognized target area in the order of the first target area, the second target area, and the third target area. Alternatively, the receiver 200 may switch the recognized target area to any one of the first to third target areas in a predetermined order each time the above-mentioned optical ID is acquired. That is, the receiver 200 recognizes the first target area as shown in FIG. 293 (a) while first acquiring the optical ID and continuously and repeatedly acquiring the optical ID. Then, the AR image P27a is superimposed on the first target area. Then, the receiver 200 hides the AR image P27a when the optical ID cannot be acquired. Next, when the receiver 200 acquires the optical ID again, the receiver 200 recognizes the second target area as shown in FIG. 293 (b) while continuously and repeatedly acquiring the optical ID. Then, the AR image P27b is superimposed on the second target area. Then, the receiver 200 hides the AR image P27b when the optical ID cannot be acquired again. Next, when the receiver 200 acquires the optical ID again, the receiver 200 recognizes the third target area as shown in FIG. 293 (c) while continuously and repeatedly acquiring the optical ID. Then, the AR image P27c is superimposed on the third target area.

When the recognized target area is switched each time the optical ID is acquired in this way, the receiver 200 receives the AR image displayed once every N (N is an integer of 2 or more) times. You may change the color. N times is the number of times the AR image is displayed, and may be, for example, 200 times. That is, the AR images P27a to P27c are all images of the same white character, but an AR image of a pink character, for example, is displayed once every 200 times. When the receiver 200 receives an operation on the AR image by the user when the AR image of the pink character is displayed, the receiver 200 may give points to the user.

In this way, by switching the target area on which the AR image is superimposed or changing the color of the AR image at a predetermined frequency, the user can be interested in the image of the signboard lit up by the transmitter 100. , The user can repeatedly acquire the optical ID.

FIG. 294 is a diagram showing an example of superimposition of AR images by the receiver 200.

The receiver 200 functions as a so-called way finder (Way Finder) that presents a course to be taken by a user by, for example, capturing a mark M4 drawn on a floor surface at a position where a plurality of passages intersect in a building. Has. The building is, for example, a hotel, and the presented route is the route for the user who checked in to his / her room.

The mark M4 is lit up by a lighting device, which is the above-mentioned transmitter 100 that transmits an optical ID according to a change in luminance. Therefore, the receiver 200 acquires the captured display image Ppre and the decoding image Pdec by imaging the mark M4. Then, the receiver 200 acquires the optical ID by decoding the decoding image Pdec, and transmits the optical ID and the terminal information of the receiver 200 to the server. The receiver 200 acquires a plurality of AR images P28 associated with the optical ID and terminal information and recognition information from the server. The optical ID and terminal information are stored in the server in association with a plurality of AR images P28 and recognition information at the time of user check-in.

The receiver 200 recognizes a plurality of target areas around the area m4 on which the mark M4 of the captured display image Ppre is projected based on the recognition information. Then, as shown in FIG. 294, the receiver 200 superimposes and displays an AR image P28 such as an animal footprint on each of the plurality of target areas.

Specifically, the recognition information indicates a course that turns to the right at the position of the mark M4. Based on such recognition information, the receiver 200 identifies a path in the captured display image Ppre and recognizes a plurality of target regions arranged along the path. This route is a route that goes from the lower side of the display 201 toward the region m4 and turns right in the region m4. The receiver 200 arranges the AR image P28 in each of the plurality of recognized target areas as if the animal walked along the path.

Here, the receiver 200 may use the geomagnetism detected by the 9-axis sensor provided in the receiver 200 when specifying the path in the captured display image Ppre. In this case, the recognition information indicates the direction to be advanced at the position of the mark M4 with reference to the direction of the geomagnetism. For example, the recognition information indicates west as the direction to go at the position of the mark M4. Based on such recognition information, the receiver 200 identifies a route from the lower side of the display 201 toward the region m4 and westward in the region m4 in the captured display image Ppre. Then, the receiver 200 recognizes a plurality of target areas arranged along the path. The receiver 200 identifies the lower side of the display 201 by detecting the gravitational acceleration by the 9-axis sensor.

In this way, since the receiver 200 presents the user's course, the user can easily reach the destination if he / she proceeds according to the course. Further, since the course is displayed as an AR image in the captured display image Ppre, the course can be presented to the user in an easy-to-understand manner.

The lighting device, which is the transmitter 100, can appropriately transmit the optical ID while suppressing the brightness by illuminating the mark M4 with the light of a short pulse. Further, although the receiver 200 has imaged the mark M4, the lighting device may be imaged by using a camera (so-called self-portrait camera) arranged on the display 201 side. Further, the receiver 200 may capture both the mark M4 and the lighting device.

FIG. 295 is a diagram for explaining an example of how to obtain the line scan time by the receiver 200.

When the receiver 200 decodes the decoding image Pdec, the receiver 200 performs the decoding using the line scan time. This line scan time is the time from the start of exposure of one exposure line included in the image sensor to the start of exposure of the next exposure line. If the line scan time is known, the receiver 200 decodes the decoding image Pdec using the known line scan time. However, if the line scan time is not known, the receiver 200 obtains the line scan time from the decoding image Pdec.

For example, as shown in FIG. 295, the receiver 200 finds a line having the minimum width from a plurality of bright lines and a plurality of dark lines constituting the emission line pattern in the decoding image Pdec. The bright line is a line on the decoding image Pdec generated by exposing each of one or a plurality of continuous exposure lines when the brightness of the transmitter 100 is high. Further, the dark line is a line on the decoding image Pdec generated by exposure of each of one or a plurality of continuous exposure lines when the brightness of the transmitter 100 is low.

When the receiver 200 finds the line having the minimum width, the receiver 200 specifies the number of lines of the exposure line corresponding to the line having the minimum width, that is, the number of pixels. When the carrier frequency whose brightness changes for the transmitter 100 to transmit the optical ID is 9.6 kHz, the time when the brightness of the transmitter 100 is high or low is 104 μs at the shortest. Therefore, the receiver 200 calculates the line scan time by dividing 104 μs by the number of pixels of the specified minimum width.

FIG. 296 is a diagram for explaining an example of how to obtain the line scan time by the receiver 200.

The receiver 200 may perform a Fourier transform on the emission line pattern of the decoding image Pdec, and obtain the line scan time based on the spatial frequency obtained by the Fourier transform.

For example, as shown in FIG. 296, the receiver 200 derives a spectrum showing the relationship between the spatial frequency and the intensity of the component of the spatial frequency in the decoding image Pdec by the above-mentioned Fourier transform. Next, the receiver 200 sequentially selects each of the plurality of peaks shown in the spectrum. Then, each time the receiver 200 selects a peak, the receiver 200 sets a line scan time such that the spatial frequency of the selected peak (for example, the spatial frequency f2 in FIG. 296) is obtained by a time frequency of 9.6 kHz. Calculated as a can time candidate. 9.6 kHz is the carrier frequency of the luminance change of the transmitter 100 as described above. As a result, a plurality of line scan time candidates are calculated. The receiver 200 selects the maximum likelihood candidate among these plurality of line scan time candidates as the line scan time.

In order to select the maximum likelihood candidate, the receiver 200 calculates an allowable range of line scan time based on the frame rate in imaging and the number of exposure lines included in the image sensor. That is, the receiver 200 calculates the maximum value of the line scan time by 1 × 10 ⁶ [μs] / {(frame rate) × (number of exposure lines)}. Then, the receiver 200 determines the maximum value × the constant K (K <1) to the maximum value as the allowable range of the line scan time. The constant K is, for example, 0.9 or 0.8.

The receiver 200 selects the candidate within this allowable range from the plurality of line scan time candidates as the maximum likelihood candidate, that is, the line scan time.

The receiver 200 may evaluate the reliability of the calculated line scan time depending on whether or not the line scan time calculated according to the example shown in FIG. 295 is within the above-mentioned allowable range.

FIG. 297 is a flowchart showing an example of how to obtain the line scan time by the receiver 200.

The receiver 200 may determine the line scan time by attempting to decode the decoding image Pdec. Specifically, first, the receiver 200 starts imaging (step S441). Next, the receiver 200 determines whether or not the line scan time is known (step S442). For example, the receiver 200 may determine whether or not the line scan time is known by notifying the server of its type and model and inquiring about the line scan time according to the type and model. .. Here, if it is determined that it is known (Yes in step S442), the receiver 200 sets the reference acquisition count of the optical ID to n (n is an integer of 2 or more, for example, 4) (step S443). ). Next, the receiver 200 acquires the optical ID by decoding the decoding image Pdec using the known line scan time (step S444). At this time, the receiver 200 acquires a plurality of optical IDs by decoding each of the plurality of decoding image Pdecs sequentially obtained by the imaging started in step S441. Here, the receiver 200 determines whether or not the same optical ID has been acquired a reference number of times (that is, n times) (step S445). When it is determined that the image has been acquired n times (Yes in step S445), the receiver 200 trusts the optical ID and starts processing using the optical ID (for example, superimposing an AR image) (step S446). On the other hand, if it is determined that the optical ID has not been acquired n times (No in step S445), the receiver 200 does not trust the optical ID and ends the process.

If it is determined in step S442 that the line scan time is not known (No in step S442), the receiver 200 sets the reference acquisition number of optical IDs to n + k (k is an integer of 1 or more) (step S447). .. That is, when the line scan time is not known, the receiver 200 sets a larger number of reference acquisition times than when the line scan time is known. Next, the receiver 200 determines a temporary line scan time (step S448). Then, the receiver 200 acquires the optical ID by decoding the decoding image Pdec using the tentatively determined line scan time (step S449). At this time, similarly to the above, the receiver 200 acquires a plurality of optical IDs by decoding each of the plurality of decoding image Pdecs sequentially obtained by the imaging started in step S441. Here, the receiver 200 determines whether or not the same optical ID has been acquired a reference number of times (that is, (n + k) times) (step S450).

When it is determined that the acquisition is performed (n + k) times (Yes in step S450), the receiver 200 determines that the provisionally determined line scan time is the correct line scan time. Then, the receiver 200 notifies the server of the type and model of the receiver 200 and the line scan time thereof (step S451). As a result, in the server, the type and model of the receiver and the line scan time suitable for the receiver are stored in association with each other. Therefore, if another receiver of the same type and type initiates imaging, the other receiver can identify its line scan time by querying the server. That is, the other receiver can determine that the line scan time is known in the determination in step S442.

Then, the receiver 200 trusts the optical ID acquired (n + k) times, and starts processing using the optical ID (for example, superimposing an AR image) (step S446).

Further, if it is determined in step S450 that the same optical ID has not been acquired (n + k) times (No in step S450), the receiver 200 further determines whether or not the termination condition is satisfied (step S452). .. The end condition is, for example, that a predetermined time has elapsed from the start of imaging, or that the optical ID has been acquired more than the maximum number of acquisitions. When it is determined that such an end condition is satisfied (Yes in step S452), the receiver 200 ends the process. On the other hand, if it is determined that the end condition is not satisfied (No in step S452), the receiver 200 changes the provisionally determined line scan time (step S453). Then, the receiver 200 repeatedly executes the process from step S449 using the changed provisionally determined line scan time.

In this way, even if the line scan time is not known, the receiver 200 can obtain the line scan time as in the example shown in FIGS. 295 to 297. Thereby, regardless of the type and model of the receiver 200, the receiver 200 can appropriately decode the decoding image Pdec and acquire the optical ID.

FIG. 298 is a diagram showing an example of superimposition of AR images by the receiver 200.

The receiver 200 takes an image of the transmitter 100 configured as a television. The transmitter 100 periodically transmits an optical ID and a time code by changing the brightness while displaying a television program, for example. The time code is information indicating the time at the time of transmission each time it is transmitted, and may be, for example, a time packet shown in FIG. 126.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec by the above-mentioned imaging. Then, the receiver 200 acquires the above-mentioned optical ID and time code by decoding the decoding image Pdec while displaying the periodically acquired captured display image Ppre on the display 201. Next, the receiver 200 transmits the optical ID to the server 300. When the server 300 receives the optical ID, the server 300 transmits the voice data associated with the optical ID, the AR start time information, the AR image P29, and the recognition information to the receiver 200.

When the receiver 200 acquires the audio data, the receiver 200 synchronizes the audio data with the video of the television program displayed on the transmitter 100 and reproduces the audio data. That is, the voice data is composed of a plurality of voice unit data, and the plurality of voice unit data includes a time code. The receiver 200 starts reproducing a plurality of voice unit data from the voice unit data including the time code indicating the same time as the time code acquired from the transmitter 100 together with the optical ID among the voice data. As a result, the reproduction of the audio data is synchronized with the video of the television program. It should be noted that such synchronization between the audio and the video may be performed by the same method as the audio synchronous reproduction shown in each figure after FIG. 123.

When the receiver 200 acquires the AR image P29 and the recognition information, the receiver 200 recognizes the area corresponding to the recognition information in the captured display image Ppre as the target area, and superimposes the AR image P29 on the target area. For example, the AR image P29 is an image showing a crack in the display 201 of the receiver 200, and the target area is a region crossing the image of the transmitter 100 in the captured display image Ppre.

Here, the receiver 200 displays the captured display image Ppre on which the AR image P29 as described above is superimposed at the timing corresponding to the AR start time information. The AR start time information is information indicating the time when the AR image P29 is displayed. That is, the receiver 200 receives the time code indicating the same time as the AR start time information among the time codes transmitted from the transmitter 100 at any time, and the above-mentioned AR image P29 is superimposed on the captured display image. Display Ppre. For example, the time indicated by the AR start time information is the time when a scene in which a witch girl casts ice magic appears in a TV program. Further, at this time, the receiver 200 may output the sound in which the AR image P29 is cracked due to the reproduction of the audio data from the speaker of the receiver 200.

As a result, the user can watch the scene of the TV program with a more realistic feeling.

Further, the receiver 200 may vibrate the vibrator provided in the receiver 200 at the time indicated by the AR start time information, or may make the light source emit light like a flash, and the display 201 may be instantaneously emitted. It may be brightened or blinked. Further, the AR image P29 may include not only an image showing a crack but also an image showing a state in which the dew condensation on the display 201 is frozen.

FIG. 299 is a diagram showing an example of superimposition of AR images by the receiver 200.

The receiver 200 images a transmitter 100 configured as, for example, a toy wand. The transmitter 100 includes a light source, and the light source changes its brightness to transmit an optical ID.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec by the above-mentioned imaging. Then, the receiver 200 acquires the above-mentioned optical ID by decoding the decoding image Pdec while displaying the image captured image Ppre that is periodically acquired on the display 201. Next, the receiver 200 transmits the optical ID to the server 300. When the server 300 receives the optical ID, the server 300 transmits the AR image P30 associated with the optical ID and the recognition information to the receiver 200.

Here, the recognition information further includes gesture information indicating a gesture (that is, an operation) by a person holding the transmitter 100. Gesture information indicates, for example, a gesture in which a person moves the transmitter 100 from right to left. The receiver 200 compares the gesture by the person holding the transmitter 100, which is projected on each captured display image Ppre, with the gesture indicated by the gesture information. Then, when the gestures match, the receiver 200 arranges, for example, many star-shaped AR images P30 along the trajectory of the transmitter 100 moved by the gestures. Is superimposed on the captured display image Ppre.

FIG. 300 is a diagram showing an example of superimposition of AR images by the receiver 200.

Similar to the above, the receiver 200 images the transmitter 100 configured as, for example, a toy wand.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec by the imaging thereof. Then, the receiver 200 acquires the above-mentioned optical ID by decoding the decoding image Pdec while displaying the image captured image Ppre that is periodically acquired on the display 201. Next, the receiver 200 transmits the optical ID to the server 300. When the server 300 receives the optical ID, the server 300 transmits the AR image P31 associated with the optical ID and the recognition information to the receiver 200.

Here, the recognition information includes gesture information indicating a gesture by a person holding the transmitter 100, as described above. Gesture information indicates, for example, a gesture in which a person moves the transmitter 100 from right to left. The receiver 200 compares the gesture by the person holding the transmitter 100, which is projected on each captured display image Ppre, with the gesture indicated by the gesture information. Then, when the gestures of the receiver 200 match, for example, in the captured display image Ppre, the AR image P30 showing the costume of the dress is in the target area which is the area where the person holding the transmitter 100 is projected. To superimpose.

As described above, in the display method in this modification, the gesture information corresponding to the optical ID is acquired from the server. Next, it is determined whether or not the movement of the subject indicated by the captured and displayed image acquired periodically matches the movement indicated by the gesture information acquired from the server. Then, when it is determined that they match, the captured display image Ppre on which the AR image is superimposed is displayed.

This makes it possible to display an AR image according to the movement of a subject such as a person. That is, the AR image can be displayed at an appropriate timing.

FIG. 301 is a diagram showing an example of a decoding image Pdec acquired according to the posture of the receiver 200.

For example, as shown in FIG. 301 (a), the receiver 200 takes an image of the transmitter 100 that transmits an optical ID by changing the brightness in a sideways posture. The sideways posture is a posture in which the longitudinal direction of the display 201 of the receiver 200 is along the horizontal direction. Further, each exposure line of the image sensor provided in the receiver 200 is orthogonal to the longitudinal direction of the display 201. By the image pickup as described above, the decoding image Pdec including the emission line pattern region X having a small number of emission lines is acquired. The number of bright lines is small in the bright line pattern region X of the decoding image Pdec. That is, there are few parts where the brightness changes to High or Low. Therefore, the receiver 200 may not be able to properly acquire the optical ID by decoding the decoding image Pdec.

Therefore, for example, as shown in FIG. 301 (b), the user changes the posture of the receiver 200 from the horizontal direction to the vertical direction. The vertical posture is a posture in which the longitudinal direction of the display 201 of the receiver 200 is along the vertical direction. When the receiver 200 in such a posture takes an image of the transmitter 100 that transmits the optical ID, it can acquire a decoding image Pdec including the emission line pattern region Y having a large number of emission lines.

As described above, it may not be possible to appropriately acquire the optical ID depending on the posture of the receiver 200. Therefore, when the receiver 200 is made to acquire the optical ID, the posture of the receiver 200 being imaged is appropriately taken. You should change it. When the posture is changed, the receiver 200 can appropriately acquire the optical ID at the timing when the posture is such that the optical ID can be easily acquired.

FIG. 302 is a diagram showing another example of the decoding image Pdec acquired according to the posture of the receiver 200.

For example, the transmitter 100 is configured as a digital signage of a coffee shop, displays a video related to an advertisement of a coffee shop during a video display period, and transmits an optical ID by changing the brightness during an optical ID transmission period. That is, the transmitter 100 alternately and repeatedly executes the display of the video in the video display period and the transmission of the optical ID in the optical ID transmission period.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec by the imaging of the transmitter 100. At this time, by synchronizing the repetition cycle of the image display period and the optical ID transmission period of the transmitter 100 with the repetition cycle of the acquisition of the captured display image Ppre and the decoding image Pdec by the receiver 200, the decoding including the emission line pattern region is performed. It may not be possible to acquire the image Pdec. Further, depending on the posture of the receiver 200, it may not be possible to acquire the decoding image Pdec including the emission line pattern region.

For example, the receiver 200 takes an image of the transmitter 100 in the posture shown in FIG. 302 (a). That is, the receiver 200 approaches the transmitter 100 and images the transmitter 100 so that the image of the transmitter 100 is projected on the entire image sensor of the receiver 200.

Here, if the timing for the receiver 200 to acquire the captured display image Ppre is within the video display period of the transmitter 100, the receiver 200 appropriately acquires the captured display image Ppre projected by the transmitter 100. ..

Even if the timing at which the receiver 200 acquires the decoding image Pdec spans the video display period of the transmitter 100 and the optical ID transmission period, the receiver 200 is for decoding including the emission line pattern region Z1. The image Pdec can be acquired.

That is, the exposure of each exposure line included in the image sensor is started in order from the exposure line at the upper end in the vertical direction to the lower side. Therefore, even if the receiver 200 starts the exposure of the image sensor to acquire the decoding image Pdec during the video display period, the emission line pattern region cannot be obtained. However, when the video display period is switched to the optical ID transmission period, it is possible to obtain a bright line pattern region corresponding to each exposure line exposed during the optical ID transmission period.

Here, the receiver 200 takes an image of the transmitter 100 in the posture as shown in FIG. 302 (b). That is, the receiver 200 is separated from the transmitter 100 and images the transmitter 100 so that the image of the transmitter 100 is projected only on the area above the image sensor of the receiver 200. At this time, similarly to the above, if the timing for the receiver 200 to acquire the captured display image Ppre is within the video display period of the transmitter 100, the receiver 200 receives the captured display image Ppre on which the transmitter 100 is projected. Get it properly. However, when the timing for the receiver 200 to acquire the decoding image Pdec spans the video display period of the transmitter 100 and the optical ID transmission period, the receiver 200 receives the decoding image Pdec including the emission line pattern region. It may not be possible to obtain it. That is, even if the video display period of the transmitter 100 is switched to the optical ID transmission period, an image of the transmitter 100 whose brightness changes is displayed on each exposure line under the image sensor where exposure is performed during the optical ID transmission period. May not be projected. Therefore, it is not possible to acquire a decoding image Pdec having a bright line pattern region.

On the other hand, as shown in FIG. 302 (c), the receiver 200 projects the image of the transmitter 100 only on the lower region of the image sensor of the receiver 200 in a state of being away from the transmitter 100. In addition, the transmitter 100 is imaged. At this time, similarly to the above, if the timing for the receiver 200 to acquire the captured display image Ppre is within the video display period of the transmitter 100, the receiver 200 receives the captured display image Ppre on which the transmitter 100 is projected. Get it properly. Further, even when the timing for the receiver 200 to acquire the decoding image Pdec spans the video display period of the transmitter 100 and the optical ID transmission period, the receiver 200 acquires the decoding image Pdec including the emission line pattern region. There are things that can be done. That is, when the video display period of the transmitter 100 is switched to the optical ID transmission period, an image of the transmitter 100 whose brightness changes is displayed on each exposure line under the image sensor where exposure is performed during the optical ID transmission period. Be projected. Therefore, it is possible to acquire the decoding image Pdec having the emission line pattern region Z2.

As described above, since the optical ID may not be appropriately acquired depending on the posture of the receiver 200, the receiver 200 may change the posture of the receiver 200 when acquiring the optical ID. May be urged to. That is, when the image pickup is started, the receiver 200 displays, for example, a message "Please move" or "Please shake" or outputs a voice so that the posture of the receiver 200 changes. As a result, the receiver 200 takes an image while changing its posture, so that the optical ID can be appropriately acquired.

FIG. 303 is a flowchart showing an example of the processing operation of the receiver 200.

For example, the receiver 200 determines whether or not the receiver 200 is shaken during image pickup (step S461). Specifically, the receiver 200 determines whether or not the receiver 200 is shaken based on the output of the 9-axis sensor provided in the receiver 200. Here, when the receiver 200 determines that the image is being shaken during imaging (Yes in step S461), the receiver 200 increases the optical ID acquisition rate described above (step S462). Specifically, the receiver 200 acquires all captured images per unit time obtained during imaging as decoding images (that is, emission line images) Pdec, and decodes each of the acquired decoded images. .. Alternatively, the receiver 200 starts the acquisition and decoding when all the captured images are acquired as the captured display image Ppre, that is, when the acquisition and decoding of the decoding image Pdec are stopped.

On the other hand, when the receiver 200 determines that the image is not shaken during imaging (No in step S461), the receiver 200 acquires the decoding image Pdec at a low optical ID acquisition rate (step S463). Specifically, if the optical ID acquisition rate is increased in step S462 and is still a high optical ID acquisition rate, the receiver 200 has a high current optical ID acquisition rate, so that the optical ID acquisition rate is set. Lower. As a result, the frequency of the decoding process of the decoding image Pdec by the receiver 200 is reduced, so that the power consumption can be suppressed.

Then, the receiver 200 determines whether or not the termination condition for terminating the optical ID acquisition rate adjustment process is satisfied (step S464), and if it is determined that the termination condition is not satisfied (No in step S464), the step. The process from S461 is repeatedly executed. On the other hand, when it is determined that the end condition is satisfied (Yes in step S464), the receiver 200 ends the optical ID acquisition rate adjustment process.

FIG. 304 is a diagram showing an example of a camera lens switching process by the receiver 200.

The receiver 200 may include a wide-angle lens 211 and a telephoto lens 212 as camera lenses, respectively. The captured image obtained by imaging with the wide-angle lens 211 is an image having a wide angle of view, and the subject is projected small in the image. On the other hand, the captured image obtained by imaging with the telephoto lens 212 is an image having a narrow angle of view, and the subject is greatly projected on the image.

When the receiver 200 as described above performs imaging, the camera lens used for imaging may be switched by any of the methods A to E shown in FIG. 304.

In the method A, the receiver 200 always uses the telephoto lens 212 when taking an image, regardless of whether the image is taken normally or when the optical ID is received. Here, the case of normal imaging is a case where all captured images are acquired as captured display image Ppre by imaging. Further, the case of receiving the optical ID is a case of periodically acquiring the captured display image Ppre and the decoding image Pdec by imaging.

In method B, the receiver 200 uses a wide-angle lens 211 in the case of normal imaging. On the other hand, when receiving the optical ID, the receiver 200 first uses the wide-angle lens 211. Then, if the decoding image Pdec acquired while using the wide-angle lens 211 includes a bright line pattern region, the receiver 200 switches the camera lens from the wide-angle lens 211 to the telephoto lens 212. After this switching, the receiver 200 can acquire a decoding image Pdec having a narrow angle of view, that is, a large emission line pattern region.

In method C, the receiver 200 uses a wide-angle lens 211 in the case of normal imaging. On the other hand, when receiving the optical ID, the receiver 200 switches the camera lens between the wide-angle lens 211 and the telephoto lens 212. That is, the receiver 200 acquires the captured display image Ppre using the wide-angle lens 211, and acquires the decoding image Pdec using the telephoto lens 212.

In the method D, the receiver 200 switches the camera lens between the wide-angle lens 211 and the telephoto lens 212 according to the operation by the user regardless of whether the image is normally captured or the optical ID is received.

In the method E, when the receiver 200 receives the optical ID, the receiver 200 decodes the decoding image Pdec acquired by using the wide-angle lens 211, and if it cannot be decoded correctly, the camera lens is changed from the wide-angle lens 211 to the telephoto lens 212. Switch. Alternatively, the receiver 200 decodes the decoding image Pdec acquired by using the telephoto lens 212, and if it cannot be decoded correctly, the camera lens is switched from the telephoto lens 212 to the wide-angle lens 211. When determining whether or not the decoding image Pdec can be correctly decoded, the receiver 200 first transmits the optical ID obtained by decoding the decoding image Pdec to the server. If the optical ID matches the optical ID registered in itself, the server notifies the receiver 200 of the matching information indicating that the optical IDs match, and if they do not match, the server does not match. Notify the receiver 200 of the discrepancy information indicating. If the information notified from the server is matching information, the receiver 200 determines that the decoding image Pdec has been correctly decoded, and if the information notified from the server is non-matching information, the decoding image Pdec is correctly decoded. Judge that it could not be done. Alternatively, the receiver 200 determines that the decoding image Pdec has been correctly decoded when the optical ID obtained by decoding the decoding image Pdec satisfies a predetermined condition. On the other hand, if the condition is not satisfied, the receiver 200 determines that the decoding image Pdec could not be correctly decoded.

By switching the camera lens in this way, an appropriate decoding image Pdec can be obtained.

FIG. 305 is a diagram showing an example of camera switching processing by the receiver 200.

For example, the receiver 200 includes an in-camera 213 and an out-camera (not shown in FIG. 305) as cameras. The in-camera 213, also referred to as a face camera or a selfie camera, is a camera arranged on the same surface as the display 201 in the receiver 200. The out-camera is a camera arranged on the surface of the receiver 200 opposite to the surface of the display 201.

Such a receiver 200 takes an image of the transmitter 100 configured as a lighting device by the in-camera 213 with the in-camera 213 facing up. By this imaging, the receiver 200 acquires the decoding image Pdec, and acquires the optical ID transmitted from the transmitter 100 by decoding the decoding image Pdec.

Next, the receiver 200 acquires the AR image and recognition information associated with the optical ID from the server by transmitting the acquired optical ID to the server. The receiver 200 starts a process of recognizing a target area corresponding to the recognition information from each captured display image Ppre obtained by each of the out-camera and the in-camera 213. Here, the receiver 200 urges the user to move the receiver 200 when the target area cannot be recognized from either the captured display image Ppre obtained by the out-camera and the in-camera 213. The user prompted by the receiver 200 moves the receiver 200. Specifically, the user moves the receiver 200 so that the in-camera 213 and the out-camera face the user in the front-rear direction. As a result, the receiver 200 recognizes the target area from the captured display image Ppre acquired by the out-camera. That is, the receiver 200 recognizes the area on which the person is projected as the target area, superimposes the AR image on the target area of the captured display image Ppre, and displays the captured display image Ppre on which the AR image is superimposed. do.

FIG. 306 is a flowchart showing an example of the processing operation between the receiver 200 and the server.

The receiver 200 acquires the optical ID transmitted from the transmitter 100 by taking an image of the transmitter 100, which is a lighting device, with the in-camera 213, and transmits the optical ID to the server (step S471). The server receives the optical ID from the receiver 200 (step S472), and estimates the position of the receiver 200 based on the optical ID (step S473). For example, for each optical ID, the server stores a table indicating a room, a building, a space, or the like in which the transmitter 100 that transmits the optical ID is arranged. Then, in the table, the server estimates the room or the like associated with the optical ID transmitted from the receiver 200 as the position of the receiver 200. Further, the server transmits the AR image and recognition information associated with the estimated position to the receiver 200 (step S474).

The receiver 200 acquires the AR image and recognition information transmitted from the server (step S475). Here, the receiver 200 starts the process of recognizing the target area corresponding to the recognition information from each of the captured and displayed image Ppres obtained by the out-camera and the in-camera 213. Then, the receiver 200 recognizes the target area from, for example, the captured display image Ppre acquired by the out-camera (step S476). The receiver 200 superimposes an AR image on the target area of the captured display image Ppre, and displays the captured display image Ppre on which the AR image is superimposed (step S477).

In the above example, when the receiver 200 acquires the AR image and the recognition information transmitted from the server, in step S476, from each of the captured display image Ppres obtained by the out-camera and the in-camera 213, respectively. The process of recognizing the target area has started. However, in step S476, the receiver 200 may start the process of recognizing the target region from the captured display image Ppre obtained only by the out-camera. That is, if the camera for acquiring the optical ID (in-camera 213 in the above example) and the camera for acquiring the captured display image Ppre on which the AR image is superimposed (out-camera in the above example) are always different. You may let me.

Further, in the above example, the receiver 200 has captured the transmitter 100, which is a lighting device, with the in-camera 213, but the floor surface illuminated by the transmitter 100 may be captured with the out-camera. Even in such an image pickup by the out-camera, the receiver 200 can acquire the optical ID transmitted from the transmitter 100.

FIG. 307 is a diagram showing an example of superimposition of AR images by the receiver 200.

The receiver 200 takes an image of a transmitter 100 configured as a microwave oven installed in a store such as a convenience store. The transmitter 100 includes a camera for taking an image of the inside of the microwave oven and a lighting device for illuminating the inside of the microwave oven. Then, the transmitter 100 recognizes the food and drink (that is, the object to be warmed) stored in the refrigerator by imaging with the camera. Further, when the transmitter 100 warms the food or drink, the above-mentioned lighting device is made to emit light, and the lighting device is changed in brightness to transmit an optical ID indicating the recognized food or drink. The lighting device illuminates the inside of the microwave oven, and the light of the lighting device is emitted to the outside through the window portion having the transparency of the microwave oven. Therefore, the optical ID is transmitted from the lighting device to the outside of the microwave oven through the window portion of the microwave oven.

Here, the user purchases food and drink at a convenience store, and puts the food and drink in the transmitter 100, which is a microwave oven, in order to heat the food and drink. At this time, the transmitter 100 recognizes the food and drink by the camera, and starts warming the food and drink while transmitting an optical ID indicating the recognized food and drink.

The receiver 200 acquires the optical ID transmitted from the transmitter 100 by taking an image of the transmitter 100 that has started the warming, and transmits the optical ID to the server. Next, the receiver 200 acquires the AR image, voice data, and recognition information associated with the optical ID from the server.

The above-mentioned AR image is an AR image P32a which is a moving image showing a virtual state of the inside of the transmitter 100, an AR image P32b which shows the food and drink stored in the refrigerator in detail, and steam is emitted from the transmitter 100. It includes an AR image P32c showing the state of being moving with a moving image, and an AR image P32d showing the remaining time until the completion of warming the food and drink with a moving image.

For example, in the AR image P32a, if the food and drink stored in the microwave oven is pizza, the turntable on which the pizza is placed is rotating, and a plurality of dwarfs are dancing around the pizza. Is. The AR image P32b is, for example, an image showing the trade name “pizza” and the ingredients of the pizza if the food or drink stored in the refrigerator is pizza.

Based on the recognition information, the receiver 200 recognizes the area of the captured display image Ppre on which the window portion of the transmitter 100 is projected as the target area of the AR image P32a, and sets the AR image P32a in the target area. Superimpose. Further, the receiver 200 recognizes, based on the recognition information, a region of the captured display image Ppre above the region on which the transmitter 100 is projected as a target region of the AR image P32b, and the target thereof. The AR image P32b is superimposed on the area. Further, the receiver 200 recognizes the area between the target area of the AR image P32a and the target area of the AR image P32b in the captured display image Ppre as the target area of the AR image P32c based on the recognition information. Then, the AR image P32c is superimposed on the target area. Further, the receiver 200 recognizes the area of the captured display image Ppre below the area where the transmitter 100 is projected as the target area of the AR image P32d based on the recognition information, and sets the target area in the target area. The AR image P32d is superimposed.

Further, the receiver 200 outputs the sound generated when the food or drink is heated by reproducing the audio data.

The AR images P32a to P32d as described above are displayed by the receiver 200, and further, by outputting the sound, the user's interest can be attracted to the receiver 200 until the warming of the food and drink is completed. .. As a result, the burden on the user waiting for the completion of warming can be reduced. Further, the AR image P32c showing steam or the like is displayed, and the sound generated when the food or drink is heated is output, so that the user can be given a sizzle feeling. Further, by displaying the AR image P32d, the user can easily know the remaining time until the warming of the food and drink is completed. Therefore, the user can read, for example, a book displayed in the store away from the transmitter 100, which is a microwave oven, until the warming is completed. Further, the receiver 200 may notify the user that the warming is completed when the remaining time becomes zero.

In the above example, the AR image P32a is a moving image in which a turntable on which a pizza is placed is rotating and a plurality of dwarfs are dancing around the pizza. For example, the temperature distribution in the refrigerator is used. It may be an image that virtually shows. Further, the AR image P32b is an image showing the trade name and material of the food and drink stored in the refrigerator, but may be an image showing nutritional components or calories. Alternatively, the AR image P32b may be an image showing a discount ticket.

As described above, in the display method in this modification, the subject is a microwave oven equipped with a lighting device, and the lighting device illuminates the inside of the microwave oven and changes the brightness of the light ID of the microwave oven. Send to the outside. Then, in the acquisition of the captured display image Ppre and the decoding image Pdec, the captured display image Ppre and the decoding image Pdec are acquired by capturing the microwave oven transmitting the optical ID. In the recognition of the target area, the window portion of the microwave oven displayed on the captured display image Ppre is recognized as the target area. In the display of the captured display image Ppre, the captured display image Ppre on which the AR image showing the state change in the refrigerator is superimposed is displayed.

As a result, the change in the state inside the microwave oven is displayed as an AR image, so that the user of the microwave oven can be informed of the state inside the microwave oven in an easy-to-understand manner.

FIG. 308 is a sequence diagram showing a processing operation of a system including a receiver 200, a microwave oven, a relay server, and an electronic payment server. As described above, the microwave oven includes a camera and a lighting device, and transmits an optical ID by changing the brightness of the lighting device. That is, the microwave oven has a function as a transmitter 100.

First, the microwave oven recognizes the food and drink stored in the refrigerator by the camera (step S481). Next, the microwave oven transmits an optical ID indicating the recognized food and drink to the receiver 200 by changing the brightness of the lighting device.

By imaging the microwave oven, the receiver 200 receives the optical ID transmitted from the microwave oven (step S483), and transmits the optical ID and the card information to the relay server. The card information is information such as a credit card stored in advance in the receiver 200, and is information necessary for electronic payment.

The relay server holds a table showing AR images, recognition information, and product information corresponding to the optical ID for each optical ID. This product information indicates the price of food and drink indicated by the optical ID. When such a relay server receives the optical ID and the card information transmitted from the receiver 200 (step S486), the relay server finds the product information associated with the optical ID from the above table. Then, the relay server transmits the product information and the card information to the electronic payment server (step S486). When the electronic payment server receives the product information and the card information transmitted from the relay server (step S487), the electronic payment server processes the electronic payment based on the product information and the card information (step S488). Then, when the electronic payment processing is completed, the electronic payment server notifies the relay server of the completion (step S489).

When the relay server confirms the payment completion notification from the electronic payment server (step S490), the relay server instructs the microwave oven to start warming the food and drink (step S491). Further, the relay server transmits the AR image and recognition information associated with the optical ID received in step S485 to the receiver 200 in the above table (step S493).

When the microwave oven receives an instruction to start heating from the relay server, it starts heating the food and drink stored in the microwave oven (step S492). Further, when the receiver 200 receives the AR image and the recognition information transmitted from the relay server, the receiver 200 receives the recognition information from the image pickup display image Ppre which is periodically acquired by the image pickup started from step S483. Recognize the area. Then, the receiver 200 superimposes the AR image on the target area (step S494).

As a result, the user of the receiver 200 can easily complete the payment and start warming the food and drink by putting the food and drink in the microwave oven and taking an image. In addition, if payment cannot be made, it is possible to prohibit the user from warming food and drink. Further, when the warming is started, the AR image P32a and the like shown in FIG. 307 can be displayed, and the user can be informed of the state of the inside of the refrigerator.

FIG. 309 is a sequence diagram showing the processing operation of a system including a POS terminal, a server, a receiver 200, and a microwave oven. As described above, the microwave oven includes a camera and a lighting device, and transmits an optical ID by changing the brightness of the lighting device. That is, the microwave oven has a function as a transmitter 100. Further, the POS (point-of-sale) terminal is a terminal installed in a store such as a convenience store which is the same as the telegraph range.

First, the user of the receiver 200 selects a food or drink that is a product at the store, and heads to the place where the POS terminal is installed in order to purchase the food or drink. The store clerk operates the POS terminal and receives the price of food and drink from the user. By the operation of the POS terminal by the clerk, the POS terminal acquires the operation input data and the sales information (step S501). The sales information indicates, for example, the name, quantity and price of the product, the place of sale, and the date and time of sale. The operation input data indicates, for example, the gender and age of the user input by the clerk. The POS terminal transmits the operation input data and the sales information to the server (step S502). The server receives the operation input data and the sales information transmitted from the POS terminal (step S503).

On the other hand, when the user of the receiver 200 pays the clerk for the food and drink, the user puts the food and drink in the refrigerator of the electric potential range in order to heat the food and drink. The microwave oven recognizes the food and drink stored in the refrigerator by the camera (step S504). Next, the microwave oven transmits an optical ID indicating the recognized food and drink to the receiver 200 by changing the brightness of the lighting device (step S505). Then, the microwave oven starts warming the food and drink (step S507).

By imaging the microwave oven, the receiver 200 receives the optical ID transmitted from the microwave oven (step S508), and transmits the optical ID and the terminal information to the server (step S509). The terminal information is information stored in advance in the receiver 200, and indicates, for example, the type of language (for example, English or Japanese) displayed on the display 201 of the receiver 200.

When the server is accessed from the receiver 200 and receives the optical ID and the terminal information transmitted from the receiver 200, the server determines whether or not the access from the receiver 200 is the first access (step S510). The first access is the first access within a predetermined time from the time when the process of step S503 is performed. Here, when the server determines that the access from the receiver 200 is the first access (Yes in step S510), the server associates and saves the operation input data with the terminal information (step S511).

Although the server determines whether or not the access from the receiver 200 is the first access, it may determine whether or not the product indicated by the sales information matches the food or drink indicated by the optical ID. .. Further, in step S511, the server may not only associate the operation input data with the terminal information but also store the sales information in association with them.

(Use indoors)
FIG. 310 is a diagram showing a state of indoor use such as an underground mall.

The receiver 200 receives the optical ID transmitted by the transmitter 100 configured as a lighting device, and estimates its current position. In addition, the receiver 200 displays the current position on the map to provide directions, and displays information on nearby stores.

In an emergency, the transmitter 100 transmits disaster information and evacuation information, so that communication is congested, the communication base station breaks down, or the radio wave from the communication base station does not reach. However, this information can be obtained. This is effective for hearing-impaired people who miss the emergency broadcast or cannot hear the emergency broadcast.

That is, the receiver 200 acquires the optical ID transmitted from the transmitter 100 by taking an image, and further acquires the AR image P33 and the recognition information associated with the optical ID from the server. Then, the receiver 200 recognizes the target area corresponding to the recognition information from the captured display image Ppre obtained by the above-mentioned imaging, and superimposes the AR image P33 in the shape of an arrow on the target area. Thereby, the receiver 200 can be used as the above-mentioned way finder (see FIG. 294).

(Display of augmented reality object)
FIG. 311 is a diagram showing how an augmented reality object is displayed.

The stage 2718e for displaying augmented reality is configured as the transmitter 100 described above, and is a reference position for displaying information on an augmented reality object or an augmented reality object with light emission patterns and position patterns of light emitting units 2718a, 2718b, 2718c, and 2718d. To send.

The receiver 200 superimposes and displays the augmented reality object 2718f, which is an AR image, on the captured image based on the received information.

It should be noted that these comprehensive or specific embodiments may be realized in a recording medium such as a device, system, method, integrated circuit, computer program or computer readable CD-ROM, and the device, system, method, integrated. It may be realized by any combination of circuits, computer programs or recording media. Further, a computer program for executing the method according to the embodiment may be stored in a recording medium of the server, and may be realized in a mode of distribution from the server to the terminal in response to a request of the terminal.

[Modification 4 of Embodiment 23]
FIG. 312 is a diagram showing a configuration of a display system according to a modification 4 of the 23rd embodiment.

The display system 500 performs object recognition using a visible light signal and augmented reality / mixed reality display.

The receiver 200 performs imaging, receives a visible light signal, and extracts a feature amount for object recognition or space recognition. The feature amount extraction is the extraction of the image feature amount from the captured image obtained by the imaging. The visible light signal may be a visible light adjacent carrier signal such as infrared rays or ultraviolet rays. Further, in this modification, the receiver 200 is configured as a recognition device that recognizes an object on which an augmented reality image (that is, an AR image) is displayed. In the example shown in FIG. 312, the object is, for example, an AR object 501.

The transmitter 100 transmits information such as an ID for identifying itself or the AR object 501 as a visible light signal or a radio wave signal. The ID is identification information such as the above-mentioned optical ID, and the AR object 501 is the above-mentioned target area. The visible light signal is a signal transmitted by a change in the luminance of the light source included in the transmitter 100.

The receiver 200 or the server 300 holds the identification information transmitted by the transmitter 100 in association with the AR recognition information and the AR display information. The association may be one-to-one or one-to-many. The AR recognition information is the above-mentioned recognition information, and is information for recognizing an AR object 501 for performing AR display. Specifically, the AR recognition information includes the image feature amount (SIFT feature amount, SURF feature amount, ORB feature amount, etc.), color, shape, size, reflectance, transmittance, or three-dimensionality of the AR object 501. It is a model etc. Further, the AR recognition information may include identification information or a recognition algorithm indicating which recognition method is used for recognition. The AR display information is information for performing AR display, and is an image (that is, the above-mentioned AR image), video, audio, three-dimensional model, motion data, display coordinates, display size, transmittance, and the like. Further, the AR display information may be an absolute value or a change rate of each of hue, saturation and lightness.

The transmitter 100 may also serve as a server 300. That is, the transmitter 100 holds the AR recognition information and the AR display information, and may transmit the information by wired or wireless communication.

The receiver 200 captures an image with a camera (specifically, an image sensor). Further, the receiver 200 receives a visible light signal or a radio signal such as WiFi or Bluetooth (registered trademark). Further, the receiver 200 acquires information such as position information obtained by GPS or the like, information obtained by a gyro sensor or an acceleration sensor, and voice from a microphone, and integrates all or part of these information. You may recognize the AR object existing in the vicinity. Further, the receiver 200 may recognize the AR object by using only one of the information without integrating the information.

FIG. 313 is a flowchart showing the processing operation of the display system according to the modified example 4 of the embodiment 23.

First, the receiver 200 determines whether or not the visible light signal has already been received (step S521). That is, the receiver 200 determines whether or not the visible light signal indicating the identification information is acquired, for example, by photographing the transmitter 100 that transmits the visible light signal by changing the brightness of the light source. At this time, the captured image of the transmitter 100 is acquired by the photographing.

Here, when the receiver 200 determines that the visible light signal has already been received (Y in step S521), the receiver 200 is in the AR object (object, reference point, spatial coordinates, or space) from the received information. (Position and orientation of receiver 200) is specified. Further, the receiver 200 recognizes the relative position of the AR object. This relative position is represented by the distance and direction from the receiver 200 to the AR object. For example, the receiver 200 identifies an AR object (that is, a target area that is a bright line pattern region) based on the size and position of the bright line pattern region shown in FIG. 244, and recognizes the relative position of the AR target. do.

Then, the receiver 200 transmits information such as an ID included in the visible light signal and the relative position to the server 300, and uses the information and the relative position as a key to obtain AR recognition information registered in the server 300. And the AR display information (step S522). At this time, the receiver 200 simultaneously acquires not only the recognized AR object information but also the information of other AR objects existing in the vicinity of the AR object (that is, AR recognition information and AR display information). Is also good. As a result, when another AR object in the vicinity is imaged by the receiver 200, the receiver 200 recognizes the other AR object in the vicinity quickly and without error. Can be done. For example, other AR objects in the vicinity are different from the initially recognized AR object.

The receiver 200 may acquire these information from the database in the receiver 200 instead of accessing the server 300. The receiver 200 receives this information after a certain period of time from the time of acquisition, or a specific process (for example, turning off the screen, pressing a button, terminating or stopping the application, displaying an AR image, or another AR object. It may be discarded after recognition of). Alternatively, the receiver 200 may lower the reliability of each of the plurality of acquired information at regular time intervals from the acquisition of the information, and may use the highly reliable information among the plurality of information. good.

Here, the receiver 200 may preferentially acquire the AR recognition information of the AR object that is valid in relation to the relative position based on the relative position with each AR object. For example, the receiver 200 acquires a plurality of visible light signals (that is, identification information) by photographing a plurality of transmitters 100 in step S521, and corresponds to the plurality of visible light signals in step S522. Acquire a plurality of AR recognition information (that is, image feature amount). At this time, in step S522, the receiver 200 selects the image feature amount of the AR object closest to the receiver 200 that shoots the transmitter 100 among the plurality of AR objects. That is, this selected image feature amount is used to identify one AR object (that is, a first object) specified by using a visible light signal. As a result, even if a plurality of image feature amounts are acquired, an appropriate image feature amount can be used to specify the first object.

On the other hand, when the receiver 200 determines that the visible light signal has not been received (N in step S521), the receiver 200 further determines whether or not the AR recognition information has already been acquired (step S523). When it is determined that the AR recognition information has not been acquired (N in step S523), the receiver 200 does not use the identification information such as the ID indicated by the visible light signal, but by image processing, or the position information or the radio wave information. Recognize candidates for AR objects using other information such as (step S524). This process may be performed only by the receiver 200. Alternatively, the receiver 200 may transmit information such as the captured image or the image feature amount of the captured image to the server 300, and the server 300 may recognize the candidate of the AR object. As a result, the receiver 200 acquires the AR recognition information and the AR display information corresponding to the recognized candidates from the server 300 or its own database.

After step S522, the receiver 200 determines whether or not the AR object is detected by another method such as image recognition that does not use identification information such as an ID indicated by a visible light signal (step S525). ). That is, the receiver 200 determines whether or not the AR object has been recognized by a plurality of methods. Specifically, the receiver 200 identifies the AR object (that is, the first object) from the captured image by using the image feature amount acquired based on the identification information indicated by the visible light signal. Then, the receiver 200 determines whether or not the AR object (that is, the second object) is specified from the captured image by image processing without using such identification information.

Here, when the receiver 200 determines that the AR object has been recognized by a plurality of methods (Y in step S525), the recognition result by the visible light signal is prioritized. That is, the receiver 200 confirms whether or not the AR objects recognized by each method match. Then, if they do not match, the receiver 200 converts one AR object on which the AR image is superimposed in the captured image into the AR object recognized by the visible light signal from among those AR objects. Determine (step S526). That is, when the first object is different from the second object, the receiver 200 gives priority to the first object and recognizes it as an object on which the AR image is displayed. The object on which the AR image is displayed is the object on which the AR image is superimposed.

Alternatively, the receiver 200 may prioritize the method given the higher priority based on the priority order given to each of the plurality of methods. That is, the receiver 200 recognizes one AR object on which the AR image is superimposed in the captured image from the AR objects recognized by each method, for example, by the method to which the highest priority is given. Determine the AR object. Alternatively, the receiver 200 may determine one AR object on which the AR image is superimposed in the captured image by a majority vote or a priority majority vote. If the recognition result up to that point is overturned by this process, the receiver 200 performs an error handling process.

Next, the receiver 200 is based on the acquired AR recognition information, and the state of the AR object in the captured image (specifically, the absolute position, the relative position from the receiver 200, the size, the angle, and the lighting condition). , Or occlusion, etc.) (step S527). Then, the receiver 200 superimposes and displays the AR display information (that is, the AR image) on the captured image according to the recognition result (step S528). That is, the receiver 200 superimposes the AR display information on the recognized AR object in the captured image. Alternatively, the receiver 200 displays only the AR display information.

These enable recognition or detection that is difficult with image processing alone. The difficult recognition or detection is, for example, identification of image-like AR objects (for example, only the character content is different), detection of AR objects with few patterns, and AR objects with high reflectance or transmittance. Detection of an object, detection of an AR object whose shape or pattern changes (for example, an animal), or detection of an AR object from a wide angle (various directions). That is, in this modification, it is possible to recognize these AR objects and display them in AR. Further, in the image processing that does not use the visible light signal, as the number of AR objects to be recognized increases, it takes time to search the vicinity of the image feature amount, the recognition processing takes time, and the recognition rate also deteriorates. .. However, in this modification, the influence of the increase in the recognition time and the deterioration of the recognition rate due to the increase in the recognition target is completely absent or extremely small, and effective recognition of the AR object becomes possible. Further, by using the relative position of the AR object, efficient recognition becomes possible. For example, by using the approximate distance to the AR object, it is possible to omit the processing for making the image feature amount independent of the size of the AR object when calculating the image feature amount, or to use the feature that depends on the size. Can be done. In addition, using the angle of the AR object, it is usually necessary to evaluate the image feature amount for many angles, but only the image feature amount corresponding to the angle of the AR object is retained and calculated. However, the calculation speed or the memory efficiency can be improved.

[Summary of Modification 4 of Embodiment 23]
FIG. 314 is a flowchart showing a recognition method according to one aspect of the present invention.

The display method according to one aspect of the present invention is a method for recognizing an object on which an augmented reality image (AR image) is displayed, and includes steps S531 to 535.

In step S531, the receiver 200 acquires identification information by photographing the transmitter 100 that transmits a visible light signal by changing the brightness of the light source. The identification information is, for example, an optical ID. In step S532, the receiver 200 transmits the identification information to the server 300, and acquires the image feature amount corresponding to the identification information from the server 300. The image feature amount is shown as AR recognition information or recognition information.

In step S533, the receiver 200 identifies the first object from the captured image of the transmitter 100 by using the image feature amount. In step S534, the receiver 200 identifies the second object from the captured image of the transmitter 100 by image processing without using the identification information (that is, the optical ID).

In step S535, when the first object specified in step S533 is different from the second object specified in step S534, the receiver 200 gives priority to the first object and augmented reality. Recognize the impression image as an object to be displayed.

For example, the augmented reality image, the captured image, and the object correspond to the AR image, the captured display image, and the target area in the 23rd embodiment and each modification thereof, respectively.

As a result, as shown in FIG. 313, the first object specified by using the identification information indicated by the visible light signal and the second object specified by image processing without using the identification information. Even if they are different, the first object is preferentially recognized as the object on which the augmented reality image is displayed. Therefore, it is possible to appropriately recognize the object on which the augmented reality image is displayed from the captured image.

Further, the image feature amount includes not only the image feature amount of the first object but also the image feature amount of the third object located in the vicinity of the first object and different from the first object. You may.

As a result, as shown in step S522 of FIG. 313, not only the image feature amount of the first object but also the image feature amount of the third object is acquired. When appearing in a captured image, the third object can be quickly identified or recognized.

Further, the receiver 200 may acquire a plurality of identification information by photographing a plurality of transmitters in step S531, and may acquire a plurality of image feature quantities corresponding to the plurality of identification information in step S532. be. In such a case, in step S533, the receiver 200 sets the image feature amount of the object closest to the receiver 200 that captures images of the plurality of transmitters as the first object among the plurality of objects. It may be used to specify.

As a result, as shown in step S522 of FIG. 313, even if a plurality of image feature amounts are acquired, an appropriate image feature amount can be used for specifying the first object.

The recognition device in this modification is, for example, a device provided in the above-mentioned receiver 200, and includes a processor and a recording medium. A program for causing the processor to execute the recognition method shown in FIG. 314 is recorded on this recording medium. Further, the program in this modification is a program that causes a computer to execute the recognition method shown in FIG. 314.

(Embodiment 24)
FIG. 315 is a diagram showing an example of an operation mode of a visible light signal according to the present embodiment. In addition, this embodiment corresponds to the modification of Embodiment 20.

As shown in FIG. 315, there are two modes of operation of the physical (PHY) layer of the visible light signal. The first operation mode is a mode in which packet PWM (Pulse Width Modulation) is performed, and the second operation mode is a mode in which packet PPM (Pulse-Position Modulation) is performed. The transmitter according to each of the above embodiments or a modification thereof generates and transmits a visible light signal by modulating the signal to be transmitted according to any of the operation modes.

In the packet PWM operation mode, RLL (Run-Length Limited) coding is not performed, the optical clock rate is 100 kHz, forward error correction (FEC) is repeatedly encoded, and the typical data rate is 5.5 kbps. Is.

In this packet PWM, the pulse width is modulated and the pulse is represented by two brightness states. The two brightness states are a bright state (Bright or High) and a dark state (Dark or Low), but typically light on and off. A chunk of physical layer signals called a packet (also called a PHY packet) corresponds to a MAC (medium access control) frame. The transmitter can repeatedly transmit the PHY packets and transmit a plurality of sets of PHY packets in any particular order.

Note that this packet PWM is, for example, the modulation shown in FIGS. 188, 189A (b), 197, and the like described above. In addition, packet PWM is used to generate a visible light signal transmitted from a normal transmitter.

In the packet PPM operating mode, RLL coding is not performed, the optical clock rate is 100 kHz, forward error correction (FEC) is repeatedly coded, and the typical data rate is 8 kbps.

In this packet PPM, the position of the pulse with a short time length is modulated. That is, this pulse is a bright pulse of a bright pulse (High) and a dark pulse (Low), and the position of this pulse is modulated. The position of this pulse is also indicated by the interval between the pulse and the next pulse.

Packet PPM realizes deep dimming. The formats, waveforms and features of the packet PPM not described in each embodiment and its variants are similar to the packet PWM. Note that this packet PPM is, for example, the modulation shown in FIGS. 189B, 199, 213, and the like described above. The packet PPM is also used to generate a visible light signal transmitted from a transmitter having a light source that emits very bright light.

Further, in each of the packet PWM and the packet PPM, the dimming in the physical layer of the visible light signal is controlled by the average luminance of the optional field.

<Packet PWM PPDU format>
Here, the format of PPDU (physical-layer data unit) will be described.

FIG. 316 is a diagram showing an example of the PPDU format in the mode 1 of the packet PWM. FIG. 317 is a diagram showing an example of the PPDU format in the packet PWM mode 2. FIG. 318 is a diagram showing an example of the PPDU format in the packet PWM mode 3.

The packet modulated by the packet PWM includes a PHY payload A, a SHR (synchronization header), a PHY payload B, and an optional field in modes 1 and 2, as shown in FIGS. 316 and 317. The SHR is a header for the PHY payload A and the PHY payload B. In addition, each of PHY payload A and PHY payload B is collectively referred to as PHY payload.

Further, in mode 3, the packet modulated by the packet PWM includes an SHR, a PHY payload, an SFT (synchronization footer), and an optional field, as shown in FIG. 318. SHT is the header for the PHY payload and SFT is the footer for the PHY payload.

In each of the modes 1 to 3, in the PHY payload A, SHR, PHY payload B and SFT, the first and second luminance values, which are different luminance values from each other, appear alternately along the time axis. The first luminance value is Bright or High, and the second luminance value is Dark or Low.

Here, the SHR of the packet PWM includes two or four pulses. Those pulses are Bright or Dark brightness pulses.

FIG. 319 is a diagram showing an example of a pulse width pattern in each SHR of the packet PWM modes 1 to 3.

As shown in FIG. 319, in packet PWM mode 1, the SHR contains two pulses. Of these two pulses, the pulse width H1 of the first pulse in the transmission order is 100 μsec, and the pulse width H2 of the second pulse is 90 μsec. In packet PWM mode 2, the SHR contains four pulses. Of these four pulses, the pulse width H1 of the first pulse in the transmission order is 100 μsec, the pulse width H2 of the second pulse is 90 μsec, and the pulse width H3 of the third pulse is. It is 90 μsec, and the pulse width H4 of the fourth pulse is 100 μsec. In packet PWM mode 3, the SHR contains four pulses. Of these four pulses, the pulse width H1 of the first pulse in the transmission order is 50 μsec, the pulse width H2 of the second pulse is 40 μsec, and the pulse width H3 of the third pulse is. It is 40 μsec, and the pulse width H4 of the fourth pulse is 50 μsec.

The PHY payload contains 6-bit data (ie x ₀ -x ₅ ) as the signal to be transmitted in mode 1 and 12-bit data (ie x ₀ -x ₁₁ ) as the signal to be transmitted in mode 2. include. Further, in the mode 3, the PHY payload includes data having a variable number of bits (that is, x ₀ − x _n ) as a signal to be transmitted. n is an integer of 1 or more, but more specifically, it is an integer obtained by subtracting 1 from a multiple of 3.

Here, the parameter yk is defined as y _k = y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4. In mode 1, k is 0 or 1, and in mode 2, k is 0, 1, 2 or 3. In mode 3, k is an integer from 0 to {(n + 1) / 3-1}.

In each of mode 1 and mode 2, the signal to be transmitted contained in the PHY payload A has two pulse widths _PA1 and _PA2 , depending on the pulse width _PAk = 120 + 30 × (7− _yk ) [μsec]. Alternatively, it is modulated to four pulse widths _PA1 to _PA4 . The signal to be transmitted included in the PHY payload B is modulated into two pulse widths P _B1 and P _B2 or four pulse widths P _B1 to P _B4 by the pulse width P _Bk = 120 + 30 × y _k [μsec]. To.

Further, in the mode 3, the signal to be transmitted included in the PHY payload is modulated to (n + 1) / 3 pulse widths P1, P2, ... By the pulse width P _k = 100 + 20 × y _k [μsec]. Will be done.

In modes 1 and 2, half of the total payload, including the PHY payload A and the PHY payload B, is optional. That is, the transmitter may transmit the PHY payload A and the PHY payload B, or may transmit only one of them. Further, the transmitter may transmit only a part of the PHY payload A and only a part of the PHY payload B. Specifically, in mode 2, the transmitter includes a pulse having a pulse width PA3 and a pulse width _PA4 in the PHY payload _A , and a pulse having a pulse width _PB1 and a pulse having a pulse width _PB2 in the PHY payload B. May be sent.

The SFT in mode 3 includes four pulses having pulse widths F1 to F4 of 40 μsec, 50 μsec, 60 μsec, and 40 μsec, respectively. Also, SFT is optional. Therefore, the transmitter may transmit the next SHR instead of the SFT.

The transmitter may transmit any kind of signal as a signal included in the optional field. However, the signal must not contain a pattern of SHR. Such an optional field is used for DC current compensation or dimming control.

<PPDU format of packet PPM>
FIG. 320 is a diagram showing an example of the PPDU format in mode 1 of the packet PPM. FIG. 321 is a diagram showing an example of the PPDU format in mode 2 of the packet PPM. FIG. 322 is a diagram showing an example of the PPDU format in the mode 3 of the packet PPM.

Packets modulated by the packet PPM in mode 1 and mode 2 include an SHR, a PHY payload, and optional fields, as shown in FIGS. 320 and 321. SHR is a header for the PHY payload.

Further, in mode 3, the packet modulated by the packet PPM includes an SHR, a PHY payload, an SFT, and an optional field, as shown in FIG. 322. SFT is a footer for the PHY payload.

In each of the modes 1 to 3, in the SHR, the PHY payload and the SFT, the first and second luminance values, which are different luminance values from each other, appear alternately along the time axis. The first luminance value is Bright or High, and the second luminance value is Dark or Low.

The time length of the short and bright pulse in the packet PPM (L in FIGS. 320-322) is shorter than 10 μsec. As a result, the average brightness of the visible light signal can be suppressed and darkened.

The SHR time length of the packet PPM includes three intervals H1 to H3. Each of the three intervals H1 to H3 is an interval of four consecutive pulses (specifically, the bright pulse described above).

FIG. 323 is a diagram showing an example of an interval pattern in each SHR of modes 1 to 3 of the packet PPM.

As shown in FIG. 323, in mode 1 of the packet PPM, each of the three intervals H1 to H3 is 160 μsec. In the packet PWM mode 2, the first interval H1 of the three intervals H1 to H3 is 160 μsec, the second interval H2 is 180 μsec, and the third interval H3 is 160 μsec. In mode 3 of the packet PPM, the first interval H1 of the three intervals H1 to H3 is 80 μsec, the second interval H2 is 90 μsec, and the third interval H3 is 80 μsec.

The PHY payload contains 6-bit data (ie x ₀ -x ₅ ) as the signal to be transmitted in mode 1 and 12-bit data (ie x ₀ -x ₁₁ ) as the signal to be transmitted in mode 2. include. Further, in the mode 3, the PHY payload includes data having a variable number of bits (that is, x ₀ − x _n ) as a signal to be transmitted. n is an integer of 5 or more, but more specifically, it is an integer obtained by subtracting 1 from a multiple of 3.

Here, the parameter yk is defined as y _k = y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4. In mode 1, k is 0 or 1, and in mode 2, k is 0, 1, 2 or 3. In mode 3, k is an integer from 0 to {(n + 1) / 3-1}.

In each of mode 1 and mode 2, the signal to be transmitted contained in the PHY payload is modulated into two intervals P1 and P2 or four intervals P1 to P4 by the interval P _k = 180 + 30 × y _k [μsec]. Will be done.

Further, in the mode 3, the signal to be transmitted included in the PHY payload is modulated into (n + 1) / 3 intervals P1, P2, ... By the interval P _k = 100 + 20 × y _k [μsec]. .. In mode 3, the PHY payload that continues until the SFT or the next SHR is transmitted.

Further, the SFT in mode 3 includes three intervals F1 to F3, and the intervals F1 to F3 are 90 μs, 80 μs, and 90 μs, respectively. Also, SFT is optional. Therefore, the transmitter may transmit the next SHR instead of the SFT.

The transmitter may transmit any kind of signal as a signal included in the optional field. However, the signal must not contain a pattern of SHR. Such an optional field is used for DC current compensation or dimming control.

<PHY frame format>
Hereinafter, the PHY frame in each mode 1 of the packet PWM and the packet PPM will be described.

The PHY payload contains 6 bits of data (ie x ₀ -x ₅ ) as described above. The packet address A (a ₀ , a ₁ ) of the packet containing the data is indicated by (x ₁ , x ₄ ). Then, the packet data D (d ₀ , d ₁ , d ₂ , d ₃ ) is indicated by (x ₀ , x ₂ , x ₃ , x ₅ ). The PHY frame, which is the MAC frame described above, consists of 16 bits including packet data D ₀₀ , D ₀₁ , D ₁₀ and D ₁₁ of four packets. Here, the packet data Dk is the packet data D of the packet having the address A indicating k.

Here, as described above, 2 bits (x ₁ , x ₄ ) out of 6 bits (x ₀ − x ₅ ) are used for the packet address A (a ₀ , a ₁ ). As a result, the time length of the 6-bit PHY payload can be shortened, and as a result, the visible light signal can be transmitted over a long distance. That is, since each of 2 bits (x ₂ , x ₅ ) out of 6 bits (x ₀ − x ₅ ) is not used for the packet address A, it can be set to 0. Further, the 2 bits (x ₂ , x ₅ ) are multiplied by a large coefficient 4 by the above-mentioned y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4, and based on the multiplication result. The pulse width or interval is determined. Therefore, when each of the _two bits (x2, _x5 ) is 0, the time length of the PHY payload can be shortened, and as a result, the transmission distance of the visible light signal can be extended.

Further, since each of 2 bits (x ₀ , x ₃ ) out of 6 bits (x ₀ − x ₅ ) is not used for the packet address A, a reception error can be suppressed. That is, the influence of 2 bits (x ₀ , x ₃ ) out of 6 bits (x ₀ − x ₅ ) on the above-mentioned parameter y _k (x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4) is small. Therefore, if these 2 bits (x ₀ , x ₃ ) are used for the packet address A, a numerical value of the same parameter y _k , that is, a similar pulse width or interval is determined for different packet addresses A. There is a possibility. As a result, the receiver may mistake the packet address A. The PHY frame reception error rate is higher when the packet address A is mistaken than when a part of the packet data is mistaken. Therefore, reception error can be suppressed by using each of (x ₁ , x ₄ ) instead of 2 bits (x ₀ , x ₃ ) out of 6 bits (x ₀ − x ₅ ) for the packet address A. ..

By the way, MPDU (medium-access-control protocol-data unit) has a very large overhead for PHY frames, and most of the fields are shortly repeated MSDU (medium-access-control service-data unit). Not necessary for. Therefore, the PHY frame does not have an MHR (medium-access-control header), and the MFR (medium-access-control footer) is optional.

Next, the PHY frame in each mode 2 of the packet PWM and the packet PPM will be described.

FIG. 324 is a diagram showing an example of 12-bit data included in the PHY payload.

The PHY payload contains 12 bits of data (ie x ₀ -x ₁₁ ) as described above. This data includes packet address A (all or part of _a0 -a3), packet data Da (all or part of _{da0 - daa6} ), and packet data Db (all or part of _db0 - _db3 ). A part) and a stop bit S (s).

That is, as shown in FIG. 324, 3 bits (x ₀ , x ₁ , x ₂ ) indicate ( _{da 0} , s, _{db 0} ), and 3 bits (x ₃ , x ₄ , x ₅ ) indicate ( _d a 1). , A ₀ or _{da a6} , _db1 ). Further, 3 bits (x ₆ , x ₇ , x ₈ ) indicate (da ₂ , a ₁ or da a 5, db ₂ ), and 3 bits (x ₉ , x ₁₀ , x ₁₁ ) indicate ( _d a ₃ , a ₂ ). Or d _{a4 ,} a3 or _db3 ).

The 12-bit data shown in FIG. 324 is the same as the data shown in FIG. 215. That is, the reference numerals w1, w2, w3 and w4 shown in FIG. 215 are 3 bits (x ₀ , x ₁ , x ₂ ), (x ₃ , x ₄ , x ₅ ), (x ₆ , x ₇ , x ₈ ), respectively. ) And (x ₉ , x ₁₀ , x ₁₁ ).

Bits x ₄ , x ₇ , x ₁₀ and x ₁₁ are used for either the packet address or the packet data according to the packet division rule.

FIGS. 325 to 332 are diagrams showing a process of dividing a PHY frame into packets. The process shown in FIGS. 325 to 332 is the same as the process of generating the packet shown in FIGS. 216 to 226, but the packet generated by the division does not include parity. It is different from the processing shown in. Further, the numerical value in the second row from the top in each box shown in FIGS. 325 to 332 indicates the bit size, and the numerical value in the third row from the top indicates the bit value (0 or 1).

FIG. 325 is a diagram showing a process of accommodating a PHY frame in one packet. That is, FIG. 325 shows a process of accommodating the 7-bit data included in the PHY frame into one packet without dividing the PHY frame.

Specifically, of the 7 bits of the PHY frame, the packet data Da (0) consisting of 4 bits and the packet data Db (0) consisting of 3 bits are a 1-bit stop bit and a 4-bit packet address. It is stored in packet 0 together with. The stop bit indicates "1" and the packet address indicates "0000".

FIG. 326 is a diagram showing a process of dividing the PHY frame into two packets.

Of the 18 bits of the PHY frame, the packet data Da (0) consisting of 7 bits and the packet data Db (0) consisting of 4 bits are stored in the packet 0 together with the 1-bit stop bit. The stop bit indicates "0". Further, among the 18 bits of the PHY frame, the packet data Da (1) consisting of 4 bits and the packet data Db (1) consisting of 3 bits are packetized together with a 1-bit stop bit and a 4-bit packet address. It is stored in 1. The stop bit indicates "1" and the packet address indicates "1000".

FIG. 327 is a diagram showing a process of dividing a PHY frame into three packets.

Of the 27 bits of the PHY frame, the packet data Da (0) consisting of 6 bits and the packet data Db (0) consisting of 4 bits are combined with a 1-bit stop bit and a 1-bit packet address into packet 0. Can be stored. The stop bit indicates "0" and the packet address indicates "0". Further, of the 27 bits of the PHY frame, the packet data Da (1) consisting of 6 bits and the packet data Db (1) consisting of 4 bits are packetized together with a 1-bit stop bit and a 1-bit packet address. It is stored in 1. The stop bit indicates "0" and the packet address indicates "1". Further, of the 27 bits of the PHY frame, the packet data Da (2) consisting of 4 bits and the packet data Db (2) consisting of 3 bits are packetized together with a 1-bit stop bit and a 4-bit packet address. It fits in 2. The stop bit indicates "1" and the packet address indicates "0100".

FIG. 328 is a diagram showing a process of dividing the PHY frame into four packets.

Of the 34 bits of the PHY frame, the packet data Da (0) consisting of 5 bits and the packet data Db (0) consisting of 4 bits are combined with a 1-bit stop bit and a 2-bit packet address into packet 0. Can be stored. The stop bit indicates "0" and the packet address indicates "00". Further, of the 34 bits of the PHY frame, the packet data Da (1) consisting of 5 bits and the packet data Db (1) consisting of 4 bits are packetized together with a 1-bit stop bit and a 2-bit packet address. It is stored in 1. The stop bit indicates "0" and the packet address indicates "10". Further, of the 34 bits of the PHY frame, the packet data Da (2) consisting of 5 bits and the packet data Db (2) consisting of 4 bits are packetized together with a 1-bit stop bit and a 2-bit packet address. It fits in 2. The stop bit indicates "0" and the packet address indicates "01". Further, of the 34 bits of the PHY frame, the packet data Da (3) consisting of 4 bits and the packet data Db (3) consisting of 3 bits are packetized together with a 1-bit stop bit and a 4-bit packet address. It is stored in 3. The stop bit indicates "1" and the packet address indicates "1100".

FIG. 329 is a diagram showing a process of dividing the PHY frame into 5 packets.

Of the 43 bits of the PHY frame, the packet data Da (0) consisting of 5 bits and the packet data Db (0) consisting of 4 bits are combined with a 1-bit stop bit and a 2-bit packet address into packet 0. Can be stored. The stop bit indicates "0" and the packet address indicates "00". Similarly, in packets 1 to 3, packet data Da consisting of 5 bits and packet data Db consisting of 4 bits are stored together with a 1-bit stop bit and a 2-bit packet address. The stop bit of those packets indicates "0". Further, of the 34 bits of the PHY frame, the packet data Da (4) consisting of 4 bits and the packet data Db (4) consisting of 3 bits are packetized together with a 1-bit stop bit and a 4-bit packet address. It fits in 4. The stop bit indicates "1" and the packet address indicates "0010".

FIG. 330 is a diagram showing a process of dividing a PHY frame into N (N = 6, 7 or 8) packets.

Of the (8N-1) bits of the PHY frame, the packet data Da (0) consisting of 4 bits and the packet data Db (0) consisting of 4 bits are a 1-bit stop bit and a 3-bit packet address. It is stored in packet 0 together with. The stop bit indicates "0" and the packet address indicates "000". Similarly, the packet 1 to the packet (N-2) also include the packet data Da consisting of 4 bits and the packet data Db consisting of 4 bits together with the 1-bit stop bit and the 3-bit packet address. The stop bit of those packets indicates "0". Further, among the (8N-1) bits of the PHY frame, the packet data Da (N-1) consisting of 4 bits and the packet data Db (N-1) consisting of 3 bits are 1-bit stop bits. It is stored in the packet (N-1) together with the 4-bit packet address. The stop bit indicates "1".

FIG. 331 is a diagram showing a process of dividing a PHY frame into 9 packets.

Of the 71 bits of the PHY frame, the packet data Da (0) consisting of 4 bits and the packet data Db (0) consisting of 4 bits are combined with a 1-bit stop bit and a 3-bit packet address into packet 0. Can be stored. The stop bit indicates "0" and the packet address indicates "000". Similarly, in packets 1 to 7, packet data Da consisting of 4 bits and packet data Db consisting of 4 bits are stored together with a 1-bit stop bit and a 3-bit packet address. The stop bit of those packets indicates "0". Further, of the 71 bits of the PHY frame, the packet data Da (8) consisting of 4 bits and the packet data Db (8) consisting of 3 bits are packetized together with a 1-bit stop bit and a 4-bit packet address. It fits in 8. The stop bit indicates "1" and the packet address indicates "0001".

FIG. 332 is a diagram showing a process of dividing a PHY frame into N (N = 10 to 16) packets.

Of the 7N bits of the PHY frame, the packet data Da (0) consisting of 4 bits and the packet data Db (0) consisting of 3 bits are combined with a 1-bit stop bit and a 4-bit packet address into packet 0. Can be stored. The stop bit indicates "0" and the packet address indicates "0000". Similarly, in the packets 1 to (N-2), the packet data Da consisting of 4 bits and the packet data Db consisting of 3 bits are stored together with the 1-bit stop bit and the 4-bit packet address. The stop bit of those packets indicates "0". Further, of the 7N bits of the PHY frame, the packet data Da (N-1) consisting of 4 bits and the packet data Db (N-1) consisting of 3 bits are a 1-bit stop bit and a 4-bit stop bit. It is stored in the packet (N-1) together with the packet address. The stop bit indicates "1".

Further, when transmitting a large amount of data such as data exceeding 112 bits (PHY frame) or stream data, the transmitter sets the stop bit of the packet 15 to "0" without setting it to "1". Then, the transmitter stores and transmits the data that could not be included in the packets 0 to 15 among the above-mentioned large amount of data in each packet newly arranged from the packet 0. In other words, the transmitter stores the data that could not be included in the packets 0 to 15 again in each packet having a packet address starting with "0000" and transmits the data.

The PHY frame in mode 2 has no MHR and the MFR is optional, like the PHY frame in mode 1.

(Summary of Embodiment 24)
The method of generating a visible light signal according to the twenty-fourth embodiment is shown by the flowchart of FIG. 230A.

That is, this method of generating a visible light signal is a method of generating a visible light signal transmitted by a change in the luminance of a light source included in the transmitter, and includes steps SD1 to SD3.

In step SD1, each of the first and second luminance values, which are different luminance values, generate a preamble which is data that appears alternately along the time axis.

In step SD2, in the data in which the first and second luminance values appear alternately along the time axis, the time length in which each of the first and second luminance values continues is set according to the signal to be transmitted. The first payload is generated by determining according to the method of 1.

Finally, in step SD3, a visible light signal is generated by combining the preamble with the first payload.

For example, as shown in FIGS. 316-318, the first and second luminance values are Bright (High) and Dark (Low), and the first data is the PHY payload (PHY payload A or PHY payload B). ). By transmitting the visible light signal generated in this way, as shown in FIGS. 191 to 193, the number of received packets can be increased and the reliability can be increased. As a result, communication between various devices can be enabled.

Further, in this visible light signal generation method, further, in the data in which the first and second luminance values appear alternately along the time axis, the time length in which each of the first and second luminance values continues is determined. , A second payload having a complementary relationship with the brightness represented by the first payload may be generated by determining according to the second method according to the signal to be transmitted. In this case, in the generation of the visible light signal, the visible light signal is generated by combining the preamble and the first and second payloads in the order of the first payload, the preamble, and the second payload.

For example, as shown in FIGS. 316 and 317, the first and second luminance values are Bright (High) and Dark (Low), and the first and second payloads are the PHY payload A and the PHY payload B. Is.

As a result, since the brightness of the first payload and the brightness of the second payload have a complementary relationship, the brightness can be kept constant regardless of the signal to be transmitted. Further, since the first payload and the second payload are data obtained by modulating the same signal to be transmitted according to different methods, if the receiver receives only one of the payloads, the payload is to be transmitted. It can be demodulated into a signal. Further, a header (SHR) which is a preamble is arranged between the first payload and the second payload. Therefore, if the receiver receives only a part of the rear side of the first payload, a header, and a part of the head side of the second payload, it can demodulate them to the signal to be transmitted. can. Therefore, the reception efficiency of the visible light signal can be improved.

For example, the preamble is a header for the first and second payloads, in which the first luminance value of the first time length, the second luminance value of the second time length, respectively, respectively. The brightness value appears. Here, the first time length is 100 μsec, and the second time length is 90 μsec. That is, as shown in FIG. 319, the pattern of the time length (pulse width) of each pulse included in the header (SHR) in the mode 1 of the packet PWM is defined.

Further, the preamble is a header for the first and second payloads, in which the first luminance value of the first time length, the second luminance value of the second time length, and the third luminance value are used. The respective luminance values appear in the order of the first luminance value and the second luminance value of the fourth time length. Here, the first time length is 100 μsec, the second time length is 90 μsec, the third time length is 90 μsec, and the fourth time length is 100 μsec. be. That is, as shown in FIG. 319, the pattern of the time length (pulse width) of each pulse included in the header (SHR) in the mode 2 of the packet PWM is defined.

In this way, since the patterns of the headers of the mode 1 and the mode 2 of the packet PWM are defined, the receiver can appropriately receive the first and second payloads in the visible light signal.

Further, the signal to be transmitted consists of 6 bits from the first bit x ₀ to the sixth bit x ₅ , and each of the first and second payloads has a first luminance value having a third time length. , The respective luminance values appear in the order of the second luminance value of the fourth time length. Here, when the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4 (k is 0 or 1), in the generation of the first payload, in the first payload. Each of the third and fourth time lengths is determined according to the first method, time length P _k = 120 + 30 × (7−y _k ) [μsec]. Further, in the generation of the second payload, each of the third and fourth time lengths in the second payload is determined according to the second method, the time length P _k = 120 + 30 × y _k [μsec]. .. That is, as shown in FIG. 316, in the packet PWM mode 1, the time of each pulse in which the signal to be transmitted is included in each of the first payload (PHY payload A) and the second payload (PHY payload B). Modulated as length (pulse width).

Further, the signal to be transmitted consists of 12 bits from the first bit x ₀ to the twelfth bit x ₁₁ , and each of the first and second payloads has a first luminance value having a fifth time length. , The second luminance value of the sixth time length, the first luminance value of the seventh time length, and the second luminance value of the eighth time length appear in this order. Here, when the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4 (k is 0, 1, 2 or 3), in the generation of the first payload, the first payload is generated. Each of the fifth to eighth time lengths in the payload of 1 is determined according to the time length P _k = 120 + 30 × (7 −y _k ) [μsec], which is the first method. Further, in the generation of the second payload, each of the fifth to eighth time lengths in the second payload is determined according to the second method, the time length P _k = 120 + 30 × y _k [μsec]. .. That is, as shown in FIG. 317, in the packet PWM mode 2, the time of each pulse in which the signal to be transmitted is included in each of the first payload (PHY payload A) and the second payload (PHY payload B). Modulated as length (pulse width).

As described above, in the packet PWM modes 1 and 2, the signal to be transmitted is modulated as the pulse width of each pulse, so that the receiver can appropriately transmit the visible light signal based on the pulse width. It can be demodulated into a signal.

Further, the preamble is a header for the first payload, and in the header, the first luminance value of the first time length, the second luminance value of the second time length, and the first of the third time length. The respective luminance values appear in the order of the luminance value of the above and the second luminance value of the fourth time length. Here, the first time length is 50 μsec, the second time length is 40 μsec, the third time length is 40 μsec, and the fourth time length is 50 μsec. be. That is, as shown in FIG. 319, the pattern of the time length (pulse width) of each pulse included in the header (SHR) in the mode 3 of the packet PWM is defined.

In this way, since the pattern of the header of the mode 3 of the packet PWM is defined, the receiver can appropriately receive the first payload in the visible light signal.

Further, the signal to be transmitted consists of 3n bits from the first bit x ₀ to the third n bit x _3n-1 (n is an integer of 2 or more), and the time length of the first payload is the first. Alternatively, it consists of a first to nth time length in which the second luminance value continues. Here, when the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4 (k is an integer from 0 to (n-1)), the first payload is generated. Then, each of the first to nth time lengths in the first payload is determined according to the time length P _k = 100 + 20 × y _k [μsec] which is the first method. That is, as shown in FIG. 318, in the packet PWM mode 3, the signal to be transmitted is modulated as the time length (pulse width) of each pulse included in the first payload (PHY payload).

As described above, in the packet PWM mode 3, the signal to be transmitted is modulated as the pulse width of each pulse, so that the receiver appropriately demodulates the visible light signal to the signal to be transmitted based on the pulse width. can do.

FIG. 333A is a flowchart showing another method of generating a visible light signal according to the twenty-fourth embodiment. This visible light signal generation method is a method for generating a visible light signal transmitted by a change in the luminance of a light source included in the transmitter, and includes steps SE1 to SE3.

In step SE1, each of the first and second luminance values, which are different luminance values, generate a preamble which is data that appears alternately along the time axis.

In step SE2, in the data in which the first and second luminance values appear alternately along the time axis, the interval from the appearance of the first luminance value to the appearance of the next first luminance value is the transmission target. The first payload is generated by determining according to the method corresponding to the signal of.

In step SE3, a visible light signal is generated by combining the preamble and the first payload.

FIG. 333B is a block diagram showing the configuration of another signal generation device according to the twenty-fourth embodiment. This signal generation device E10 is a signal generation device that generates a visible light signal transmitted by a change in the luminance of a light source included in the transmitter, and includes a preamble generation unit E11, a payload generation unit E12, and a coupling unit E13. .. Further, the signal generation device E10 executes the processing of the flowchart shown in FIG. 333A.

That is, the preamble generation unit E11 generates a preamble which is data in which the first and second luminance values, which are different luminance values, appear alternately along the time axis.

The payload generation unit E12 sets the interval from the appearance of the first luminance value to the appearance of the next first luminance value in the data in which the first and second luminance values appear alternately along the time axis. The first payload is generated by determining according to the method according to the signal to be transmitted.

The coupling portion E13 generates a visible light signal by coupling the preamble and the first payload.

For example, as shown in FIGS. 320-322, the first and second luminance values are Bright (High) and Dark (Low), and the first payload is the PHY payload. By transmitting the visible light signal generated in this way, as shown in FIGS. 191 to 193, the number of received packets can be increased and the reliability can be increased. As a result, communication between various devices can be enabled.

For example, the time length of the first luminance value in each of the preamble and the first payload is 10 μsec or less.

As a result, the average brightness of the light source can be suppressed while performing visible light communication.

Further, the preamble is a header for the first payload, and the time length of the header includes three intervals from the appearance of the first luminance value to the appearance of the next first luminance value. Here, each of the three intervals is 160 μsec. That is, as shown in FIG. 323, the pattern of the interval between each pulse included in the header (SHR) in the mode 1 of the packet PPM is defined. Each of the above pulses is, for example, a pulse having a first luminance value.

Further, the preamble is a header for the first payload, and the time length of the header includes three intervals from the appearance of the first luminance value to the appearance of the next first luminance value. Here, the first interval of the three intervals is 160 μsec, the second interval is 180 μsec, and the third interval is 160 μsec. That is, as shown in FIG. 323, the pattern of the interval between each pulse included in the header (SHR) in the mode 2 of the packet PPM is defined.

Further, the preamble is a header for the first payload, and the time length of the header includes three intervals from the appearance of the first luminance value to the appearance of the next first luminance value. Here, the first interval of the three intervals is 80 μsec, the second interval is 90 μsec, and the third interval is 80 μsec. That is, as shown in FIG. 323, the pattern of the interval between each pulse included in the header (SHR) in the mode 3 of the packet PPM is defined.

In this way, since the pattern of the header of each of the mode 1, mode 2 and mode 3 of the packet PPM is defined, the receiver can appropriately receive the first payload in the visible light signal.

Further, the signal to be transmitted consists of 6 bits from the first bit x ₀ to the sixth bit x ₅ , and the time length of the first payload is the next first after the first luminance value appears. Includes two intervals until the brightness value of. Here, when the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4 (k is 0 or 1), in the generation of the first payload, in the first payload. Each of the two intervals is determined according to the above-mentioned method, interval P _k = 180 + 30 × y _k [μsec]. That is, as shown in FIG. 320, in mode 1 of the packet PPM, the signal to be transmitted is modulated as an interval between each pulse included in the first payload (PHY payload).

Further, the signal to be transmitted consists of 12 bits from the first bit x ₀ to the twelfth bit x ₁₁ , and the time length of the first payload is the next first after the first luminance value appears. Includes four intervals until the brightness value of. Here, when the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4 (k is 0, 1, 2 or 3), in the generation of the first payload, the first payload is generated. Each of the four intervals in one payload is determined according to the above-mentioned method, interval P _k = 180 + 30 × y _k [μsec]. That is, as shown in FIG. 321, in the mode 2 of the packet PPM, the signal to be transmitted is modulated as an interval between each pulse included in the first payload (PHY payload).

Further, the signal to be transmitted consists of 3n bits from the first bit x ₀ to the third n bit x _3n-1 (n is an integer of 2 or more), and the time length of the first payload is the first. It includes n intervals from the appearance of the luminance value to the appearance of the next first luminance value. Here, when the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4 (k is an integer from 0 to (n-1)), the first payload is generated. Then, each of the n intervals in the first payload is determined according to the interval P _k = 100 + 20 × y _k [μsec], which is the above-mentioned method. That is, as shown in FIG. 322, in the mode 3 of the packet PPM, the signal to be transmitted is modulated as an interval between each pulse included in the first payload (PHY payload).

As described above, in the mode 1, mode 2 and mode 3 of the packet PPM, the signal to be transmitted is modulated as an interval between each pulse, so that the receiver appropriately transmits the visible light signal based on the interval. It can be demodulated to the signal of interest.

Further, in the visible light signal generation method, a footer for the first payload may be further generated, and in the visible light signal generation, the footer may be coupled to the first payload. That is, as shown in FIGS. 318 and 322, in mode 3 of packet PWM and packet PPM, a footer (SFT) is transmitted following the first payload (PHY payload). As a result, the end of the first payload can be clearly specified by the footer, so that visible light communication can be efficiently performed.

Further, in the generation of the visible light signal, if the footer is not transmitted, a header for the next signal of the signal to be transmitted may be combined instead of the footer. That is, in mode 3 of packet PWM and packet PPM, instead of the footer (SFT) shown in FIGS. 318 and 322, a first payload (PHY payload) is followed by a header (SHR) for the next first payload. ) Is sent. As a result, the end of the first payload can be clearly specified by the header for the next first payload, and the footer is not transmitted, so that visible light communication can be performed more efficiently.

The configuration of the signal generator according to the 24th embodiment is shown by the block diagram of FIG. 230B.

That is, the signal generation device D10 according to the twenty-fourth embodiment is a signal generation device that generates a visible light signal transmitted by a change in the luminance of a light source included in the transmitter, and includes a preamble generation unit D11 and a data generation unit D12. , With a coupling portion D13.

The preamble generation unit D11 generates a preamble which is data in which the first and second luminance values, which are different luminance values, appear alternately along the time axis.

In the data in which the first and second luminance values appear alternately along the time axis, the data generation unit D12 determines the duration of each of the first and second luminance values according to the signal to be transmitted. The first payload is generated by determining according to the first method.

The coupling unit D13 generates a visible light signal by coupling the preamble and the first payload.

By transmitting the visible light signal generated by the signal generation device D10, as shown in FIGS. 191 to 193, the number of received packets can be increased and the reliability can be improved. As a result, communication between various devices can be enabled.

In each of the above embodiments and modifications, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. For example, the program causes a computer to execute the method of generating a visible light signal shown by the flowcharts of FIGS. 230A and 333A.

Although the method for generating a visible light signal according to one or a plurality of embodiments has been described above based on each of the above embodiments and modifications, the present invention is not limited to this embodiment. As long as the gist of the present invention is not deviated, various modifications that can be conceived by those skilled in the art are applied to the present embodiment, and forms constructed by combining components in different embodiments and modifications are also within the scope of the present invention. May be included in.

(Embodiment 25)
In this embodiment, a method for decoding a visible light signal, a method for encoding, and the like will be described.

FIG. 334 is a diagram showing the format of the MAC frame in the MPM.

The format of the MAC (medium access control) frame in MPM (Mirror Pulse Modulation) is composed of MHR (medium access control header) and MSDU (medium access control service-data unit). The MHR field includes a sequence number subfield. The MSDU includes a frame payload and has a variable length. The bit length of the MPDU (medium access control protocol-data unit) including the MHR and the MSDU is set as macMpmMpduLength.

The MPM is a modulation method according to the 20th embodiment and the 24th embodiment, and is, for example, information to be transmitted as shown in FIGS. 188 to 189B, 197 to 230B, and 315 to 332. Alternatively, it is a method of modulating a signal.

FIG. 335 is a flowchart showing the processing operation of the coding device that generates the MAC frame in the MPM. Specifically, FIG. 335 is a diagram showing how to determine the bit length of the sequence number subfield. The coding device is provided in, for example, the above-mentioned transmitter or transmission device that transmits a visible light signal.

The sequence number subfield contains a frame sequence number (also referred to as a sequence number). The bit length of the sequence number subfield is set as macMpmSnLength. If the bit length of the sequence number subfield is set to variable length, the first bit in the sequence number subfield is used as the last frame flag. That is, in this case, the sequence number subfield contains the last frame flag and a bit string indicating the sequence number. The final frame flag is set to 1 for the final frame and 0 for the other frames. That is, this final frame flag indicates whether or not the processing target frame is the final frame. This final frame flag corresponds to the above-mentioned stop bit. The sequence number corresponds to the above-mentioned address.

First, the coding apparatus determines whether or not the SN is set to a variable length (step S101a). Note that SN is the bit length of the sequence number subfield. That is, the coding device determines whether or not macMpmSnLength indicates 0xf. When macMpmSnLength indicates 0xf, SN has a variable length, and when macMpmSnLength indicates other than 0xf, SN has a fixed length. When the encoding device determines that the SN is not set to variable length, that is, the SN is set to a fixed length (N in step S101a), it determines the SN to the value indicated by macMpmSnLength (step S102a). .. At this time, the coding device does not use the final frame flag (that is, LFF).

On the other hand, when the coding apparatus determines that the SN is set to the variable length (Y in step S101a), the coding apparatus determines whether or not the processing target frame is the final frame (step S103a). Here, when the coding apparatus determines that the processing target frame is the final frame (Y in step S103a), the coding apparatus determines the SN to be 5 bits (step S104a). At this time, the coding apparatus determines the final frame flag indicating 1 as the first bit in the sequence number subfield.

Further, when the coding apparatus determines that the processing target frame is not the final frame (N in step S103a), the coding apparatus determines that the value of the sequence number of the final frame is any one of 1-15 (step S105a). The sequence number is an integer assigned to each frame in ascending order from 0. Further, in the case of N in step S103a, the number of frames is 2 or more. Therefore, in this case, the value of the sequence number of the final frame can be any one of 1-15 excluding 0.

When the coding apparatus determines in step S105a that the value of the sequence number of the final frame is 1, the SN is determined to be 1 bit (step S106a). At this time, the coding apparatus determines the value of the final frame flag, which is the first bit in the sequence number subfield, to 0.

For example, if the value of the sequence number of the last frame is 1, the sequence number subfield of the last frame is represented as (1,1) including the last frame flag (1) and the value of the sequence number (1). .. At this time, the coding apparatus determines the bit length of the sequence number subfield of the processing target frame to be 1 bit. That is, the coding device determines a sequence number subfield containing only the final frame flag (0).

When the coding apparatus determines in step S105a that the value of the sequence number of the final frame is 2, the encoding device determines the SN to 2 bits (step S107a). Also at this time, the coding device determines the value of the final frame flag to 0.

For example, if the value of the sequence number of the last frame is 2, the sequence number subfield of the last frame is listed as (1, 0, 1) including the last frame flag (1) and the value of the sequence number (2). Will be done. The sequence number is indicated by a bit string, in which the leftmost bit is the LSB (least significant bit) and the rightmost bit is the MSB (most significant bit). Therefore, the value (2) of the sequence number is expressed as a bit string (0,1). As described above, when the value of the sequence number of the final frame is 2, the encoding device determines the bit length of the sequence number subfield of the processing target frame to be 2 bits. That is, the encoding device determines a sequence number subfield containing the final frame flag (0) and the bit (0) or (1) indicating the sequence number.

When the coding apparatus determines in step S105a that the value of the sequence number of the final frame is 3 or 4, the SN is determined to be 3 bits (step S108a). Also at this time, the coding device determines the value of the final frame flag to 0.

When the coding apparatus determines in step S105a that the value of the sequence number of the final frame is any integer of 5-8, the coding apparatus determines the SN to be 4 bits (step S109a). Also at this time, the coding device determines the value of the final frame flag to 0.

When the coding apparatus determines in step S105a that the value of the sequence number of the final frame is any integer of 9-15, the coding apparatus determines the SN to be 5 bits (step S110a). Also at this time, the coding device determines the value of the final frame flag to 0.

FIG. 336 is a flowchart showing the processing operation of the decoding device that decodes the MAC frame in the MPM. Specifically, FIG. 336 is a diagram showing how to determine the bit length of the sequence number subfield. The decoding device is provided in, for example, the above-mentioned receiver or receiving device that receives a visible light signal.

Here, the decoding device determines whether or not the SN is set to a variable length (step S201a). That is, the decoding device determines whether or not macMpmSnLength indicates 0xf. When the decoder determines that the SN is not set to variable length, that is, the SN is set to a fixed length (N in step S201a), the decoder determines the SN to the value indicated by macMpmSnLength (step S202a). At this time, the decoding device does not use the final frame flag (that is, LFF).

On the other hand, when the decoding device determines that the SN is set to the variable length (Y in step S201a), the decoding device determines whether the value of the final frame flag of the frame to be decoded is 1 or 0 (step S203a). .. That is, the decoding device determines whether or not the decoding target frame is the final frame. Here, when the decoding device determines that the value of the final frame flag is 1 (1 in step S203a), the decoding device determines the SN to be 5 bits (step S204a).

Further, when the decoding device determines that the value of the final frame flag is 0 (0 in step S203a), the value indicated by the bit string from the second bit to the fifth bit in the sequence number subfield of the final frame is 1. It is determined which of -15 is (step S205a). The final frame has a final frame flag indicating 1, and is a frame generated from the same source as the frame to be decoded. Also, each source is identified by its position in the captured image. The source is divided into a plurality of frames (that is, packets) as shown in FIGS. 325 to 332, for example. That is, the final frame is the last frame among the plurality of frames generated by the division of one source. Further, the value indicated by the bit string from the second bit to the fifth bit in the sequence number subfield is the value of the sequence number.

When the decoding device determines in step S205a that the value indicated by the bit string is 1, the decoding device determines the SN to be 1 bit (step S206a). For example, if the sequence number subfield of the final frame is 2 bits of (1,1), the final frame flag is 1, and the sequence number of the final frame, that is, the value indicated by the bit string is 1. At this time, the decoding device determines the bit length of the sequence number subfield of the frame to be decoded to 1 bit. That is, the decoding device determines the sequence number subfield of the frame to be decoded to (0).

When the decoding device determines in step S205a that the value indicated by the bit string is 2, the decoding device determines the SN to be 2 bits (step S207a). For example, if the sequence number subfield of the final frame is 3 bits of (1,0,1), the final frame flag is 1, and the sequence number of the final frame, that is, the value indicated by the bit string (0,1). Is 2. In the above bit string, the leftmost bit is the LSB (least significant bit) and the rightmost bit is the MSB (most significant bit).

At this time, the decoding device determines the bit length of the sequence number subfield of the frame to be decoded to be 2 bits. That is, the decoding device determines the sequence number subfield of the frame to be decoded to (0,0) or (0,1).

When the decoding device determines in step S205a that the value indicated by the bit string is 3 or 4, the decoding device determines the SN to be 3 bits (step S208a).

When the decoding device determines in step S205a that the value indicated by the bit string is any integer of 5-8, the decoding device determines the SN to be 4 bits (step S209a).

When the decoding device determines in step S205a that the value indicated by the bit string is any integer of 9-15, the decoding device determines the SN to be 5 bits (step S210a).

FIG. 337 is a diagram showing the attributes of the PIB of the MAC.

The attributes of the MAC PIB (physical-layer personal-area-network information base) include macMpmSnLength and macMpmMpduLength. macMpmSnLength is any integer value in the range 0x0-0xf and indicates the bit length of the sequence number subfield. Specifically, macMpmSnLength, when it is any integer value in the range of 0x0-0xe, indicates the integer value as a fixed bit length of the sequence number subfield. Further, when macMpmSnLength is 0xf, it indicates that the bit length of the sequence number subfield is variable.

macMpmMpduLength is any integer value in the range of 0x00-0xff and indicates the bit length of MPDU.

FIG. 338 is a diagram for explaining a dimming method of MPM.

The MPM has a dimming function. The MPM dimming method includes, for example, (a) analog dimming method, (b) PWM dimming method, (c) VPPM dimming method, and (d) field insertion dimming method shown in FIG. 338. ..

In the analog dimming method, for example, as shown in (a2), a visible light signal is transmitted by changing the luminance. Here, when the visible light signal is darkened, for example, as shown in (a1), the overall brightness of the visible light signal is lowered. On the contrary, when the visible light signal is brightened, for example, as shown in (a3), the overall brightness of the visible light signal is increased.

In the PWM dimming method, for example, as shown in (b2), a visible light signal is transmitted by changing the brightness. Here, when the visible light signal is darkened, for example, as shown in (b1), the brightness is lowered for a short period during the period in which the high-luminance light shown in (b2) is output. On the contrary, when the visible light signal is brightened, for example, as shown in (b3), the brightness is increased only for a short period during the period in which the low-luminance light shown in (b2) is output. It should be noted that the short period described above should be less than 1/3 of the original pulse width and less than 50 μsec.

In the VPPM dimming method, for example, as shown in (c2), a visible light signal is transmitted by changing the luminance. Here, when the visible light signal is darkened, for example, as shown in (c1), the timing of the fall of the luminance is advanced. On the contrary, when the visible light signal is brightened, for example, as shown in (c3), the timing of the fall of the luminance is delayed. The VPPM modulation method can be used only for the PPM mode of PHY in MPM.

In the field insertion dimming method, for example, as shown in (d2), a visible light signal including a plurality of PPDUs (physical-layer data units) is transmitted. Here, when dimming the visible light signal, for example, as shown in (d1), a dimming field having a brightness lower than the brightness of the PPDU is inserted between the PPDUs. On the contrary, when brightening the visible light signal, for example, as shown in (d3), a dimming field having a brightness higher than that of the PPDU is inserted between the PPDUs.

FIG. 339 is a diagram showing the attributes of the PHY PIB.

PHY (physical layer) PIB attributes include phyMpmMode, phyMpmPlcpHeaderMode, phyMpmPlcpCenterMode, phyMpmSymbolSize, phyMpmOddSymbolBit, phyMpmEvenSymbolBit, phyMpmSymbolOffset, and phyMpmSymbolUnit.

The phyMpmMode is 0 or 1 and indicates the PHY mode of the MPM. Specifically, when phyMpmMode is 0, it indicates that the PHY mode is the PWM mode, and when it is 1, it indicates that the PHY mode is the PWM mode.

phyMpmPlcpHeaderMode is any integer value in the range of 0x0-0xf and indicates a PLCP (Physical Layer Conversion Protocol) header subfield mode and a PLCP footer subfield mode.

phyMpmPlcpCenterMode is any integer value in the range 0x0-0xf and indicates the PLCP center subfield mode.

phyMpmSymbolSize is any integer value in the range 0x0-0xf and indicates the number of symbols in the payload subfield. Specifically, phyMpmSymbolSize indicates that the number of symbols is variable when it is 0x0, and is referred to as N.

phyMpmOddSymbolBit is any integer value in the range of 0x0-0xf, indicates the bit length included in each _odd symbol of the payload subfield, and is referred to as Mod.

phyMpmEvenSymbolBit is any integer value in the range _0x0-0xf , indicates the bit length contained in each even symbol of the payload subfield, and is referred to as Moven.

phyMpmSymbolOffset is any integer value in the range _0x00-0xff , indicates the offset value of the symbol of the payload subfield, and is referred to as W1.

phyMpmSymbolUnit is any integer value in the range _0x00-0xff , indicates the unit value of the symbol of the payload subfield, and is referred to as W2.

FIG. 340 is a diagram for explaining MPM.

The MPM consists only of a PSDU (PHY service data unit) field. Also, the PSDU field contains the MPDU converted by the PLCP of the MPM.

The PLCP of the MPM converts the MPDU into five subfields, as shown in FIG. 340. The five subfields are the PLCP header subfield, the front payload subfield, the PLCP center subfield, the back payload subfield, and the PLCP footer subfield. The MPM PHY mode is set as phyMpmMode.

As shown in FIG. 340, the PLCP of the MPM includes a bit rearrangement unit 301a, a duplication unit 302a, a front conversion unit 303a, and a back conversion unit 304a.

Here, (x ₀ , x ₁ , x ₂ , ...) Is each bit contained in the MPDU, L _SN is the bit length of the sequence number subfield, and N is the bit length of each payload subfield. The number of symbols. The bit rearrangement unit 301a rearranges (x ₀ , x ₁ , x ₂ , ...) To (y ₀ , y ₁ , y ₂ , ...) According to the following (Equation 1).

By this rearrangement, each bit contained in the sequence number subfield at the beginning of the MPDU is moved to the rear side by L _SN . The duplication unit 302a duplicates the MPDU after the bit rearrangement.

The front payload subfield and the back payload subfield each consist of N symbols. Here, Mod is the bit length included in the odd-numbered symbol, _Meven is the bit length included in the _even -numbered symbol, and W ₁ is the symbol value offset (the offset value described above). , W ₂ are symbol value units (unit values described above). Note that N, _Modd , _Moven , W ₁ and W ₂ are set by the PHY PIB shown in FIG. 339.

The front conversion unit 303a and the back conversion unit 304a convert the relocated MPDU payload bits (y0, y1, y2, ...) To _zi by the following (Equation 2) to (Equation 5).

The front conversion unit 303a uses _zi to calculate the i-th symbol (that is, the symbol value) of the front payload subfield by the following (Equation 6).

The back conversion unit 304a uses _zi to calculate the i-th symbol (that is, the symbol value) of the back payload subfield by the following (Equation 7).

The symbol values calculated by (Equation 6) and (Equation 7) correspond to, for example, the time lengths _DR1 to _{DR4 and DL1 to DL4 shown in} FIG. 188.

FIG. 341 is a diagram showing PLCP header subfields.

As shown in FIG. 341, the PCCP header subfield is composed of four symbols in PWM mode and three symbols in PPM mode.

FIG. 342 is a diagram showing a PLCP center subfield.

As shown in FIG. 342, the subfield of the PLCP center is composed of four symbols in the PWM mode and three symbols in the PPM mode.

FIG. 343 is a diagram showing a PLCP footer subfield.

As shown in FIG. 343, the PLCP footer subfield is composed of four symbols in PWM mode and three symbols in PPM mode.

FIG. 344 is a diagram showing a waveform of the PWM mode of PHY in MPM.

In PWM mode, the symbol must be transmitted as one of two states of light intensity, the bright or dark state. In the PHY PWM mode in MPM, the symbol value corresponds to a continuous time in microseconds. For example, as shown in FIG. 344, the first symbol value corresponds to the continuous time of the first bright state, and the second symbol value corresponds to the continuous time of the next dark state. In the example shown in FIG. 344, the initial state of each subfield is a bright state, but it may be a dark state.

FIG. 345 is a diagram showing a waveform of the PPM mode of PHY in MPM.

In PPM mode, as shown in FIG. 345, the symbol value represents the time from the start of a bright state to the start of the next bright state in microseconds. The bright time should be less than 90% of the symbol value.

For both modes, the transmitter can only transmit some of the multiple symbols. However, the transmitter must transmit all the symbols in the PLCP center subfield and at least N symbols. Each of the at least N symbols is a symbol contained in either the front payload subfield or the back payload subfield.

(Summary of Embodiment 25)
FIG. 346 is a flowchart showing an example of the decoding method of the 25th embodiment. The flowchart shown in FIG. 346 corresponds to the flowchart shown in FIG. 336.

This decoding method is a method for decoding a visible light signal composed of a plurality of frames, and includes step S310b, step S320b, and step S330b as shown in FIG. 346. Also, each of these plurality of frames includes a sequence number and a frame payload.

In step S310b, variable length determination is performed to determine whether or not the bit length of the subfield is variable length based on macSnLength, which is information for determining the bit length of the subfield in which the sequence number is stored in the frame to be decoded. Perform processing.

In step S320b, the bit length of the subfield is determined based on the result of the variable length determination process. Then, in step S330b, the decoding target frame is decoded based on the bit length of the determined subfield.

Here, the determination of the bit length of the subfield in step S320b includes steps S321b to S324b.

That is, when it is determined in the variable length determination process of step S310b that the bit length of the subfield is not variable length, the bit length of the subfield is determined to the value indicated by the above-mentioned macSnLength (step S321b). ).

On the other hand, when it is determined in the variable length determination process of step S310b that the bit length of the subfield has a variable length, it is determined whether or not the decoding target frame is the final frame among the plurality of frames. The final determination process is performed (step S322b). Here, if it is determined that the frame is the final frame (Y in step S322b), the bit length of the subfield is determined to a predetermined value (step S323b). On the other hand, if it is determined that it is not the final frame (N in step S322b), the bit length of the subfield is determined based on the value of the sequence number of the final frame (step S324b).

As a result, as shown in FIG. 346, regardless of whether the bit length of the subfield in which the sequence number is stored (specifically, the sequence number subfield) is fixed length or variable length, the subfield The bit length can be determined appropriately.

Here, in the final determination process of step S322b, it may be determined whether or not the decoding target frame is the final frame based on the final frame flag indicating whether or not the decoding target frame is the final frame. Specifically, in the final determination process of step S322b, when the final frame flag indicates 1, it is determined that the decoding target frame is the final frame, and when the final frame flag indicates 0, the decoding target frame is determined. May not be the final frame. For example, the final frame flag may be included in the first bit of the subfield.

As a result, as shown in step S203a of FIG. 336, it is possible to appropriately determine whether or not the decoding target frame is the final frame.

More specifically, in the determination of the bit length of the subfield in step S320b, when it is determined in the final determination process of step S322b that the frame to be decoded is the final frame, the bit length of the subfield is determined as described above. It may be determined to be 5 bits which is a predetermined value of. That is, as shown in step S204a of FIG. 336, the bit length SN of the subfield is determined to be 5 bits.

Further, in the determination of the bit length of the subfield in step S320b, when it is determined in the final determination process of step S322b that the frame to be decoded is not the final frame, and the value of the sequence number of the final frame is 1, the value of the sequence number of the final frame is 1. The bit length of the subfield may be determined to be 1 bit. Further, when the value of the sequence number of the final frame is 2, the bit length of the subfield may be determined to be 2 bits. Further, when the value of the sequence number of the final frame is 3 or 4, the bit length of the subfield may be determined to be 3 bits. Further, when the value of the sequence number of the final frame is an integer of any of 5 to 8, the bit length of the subfield may be determined to be 4 bits. Further, when the value of the sequence number of the final frame is an integer of any of 9 to 15, the bit length of the subfield may be determined to be 5 bits. That is, as shown in steps S206a to S210a of FIG. 336, the bit length SN of the subfield is determined to be any of 1 to 5 bits.

FIG. 347 is a flowchart showing an example of the coding method of the 25th embodiment. The flowchart shown in FIG. 347 corresponds to the flowchart shown in FIG. 335.

This coding method is a method of coding the information to be coded into a visible light signal composed of a plurality of frames, and as shown in FIG. 347, step S410a, step S420a, and step S430a. including. Also, each of these plurality of frames includes a sequence number and a frame payload.

In step S410a, a variable length determination for determining whether or not the bit length of the subfield is variable length based on macSnLength, which is information for determining the bit length of the subfield in which the sequence number is stored in the processing target frame. Perform processing.

In step S420a, the bit length of the subfield is determined based on the result of the variable length determination process. Then, in step S430a, a part of the information to be encoded is encoded in the processing target frame based on the bit length of the determined subfield.

Here, in determining the bit length of the subfield in step S420a, steps S421a to S424a are included.

That is, when it is determined in the variable length determination process of step S410a that the bit length of the subfield is not variable length, the bit length of the subfield is determined to the value indicated by the above-mentioned macSnLength (step S421a). ).

On the other hand, when it is determined in the variable length determination process of step S410a that the bit length of the subfield is variable length, it is determined whether or not the processing target frame is the final frame among the plurality of frames. The final determination process is performed (step S422a). Here, if it is determined that the frame is the final frame (Y in step S422a), the bit length of the subfield is determined to a predetermined value (step S423a). On the other hand, if it is determined that it is not the final frame (N in step S422a), the bit length of the subfield is determined based on the value of the sequence number of the final frame (step S424a).

As a result, as shown in FIG. 347, regardless of whether the bit length of the subfield in which the sequence number is stored (specifically, the sequence number subfield) is fixed length or variable length, the subfield The bit length can be determined appropriately.

The decoding device according to the present embodiment includes a processor and a memory, and a program for causing the processor to execute the decoding method shown in FIG. 346 is recorded in the memory. The coding apparatus according to the present embodiment includes a processor and a memory, and a program for causing the processor to execute the coding method shown in FIG. 347 is recorded in the memory. Further, the program in the present embodiment is a program that causes a computer to execute the decoding method shown in FIG. 346 or the coding method shown in FIG. 347.

(Embodiment 26)
In this embodiment, a transmission method for transmitting an optical ID by a visible light signal will be described. The transmitter and receiver in this embodiment may have the same functions and configurations as the transmitter (or transmitter) and receiver (or receiver) in each of the above embodiments.

FIG. 348 is a diagram showing an example in which the receiver in the present embodiment displays an AR image.

The receiver 200 in the present embodiment is a receiver including an image sensor and a display 201, and is configured as, for example, a smartphone. Such a receiver 200 acquires the above-mentioned captured display image Pa, which is the normal captured image, and the above-mentioned image for decoding, which is the visible light communication image or the emission line image, by capturing the subject by the image sensor.

Specifically, the image sensor of the receiver 200 captures the transmitter 100. The transmitter 100 has a form like, for example, a light bulb, and includes a glass sphere 141 and a light emitting unit 142 that shines and flutters like a flame inside the glass sphere 141. The light emitting unit 142 illuminates by lighting one or a plurality of light emitting elements (for example, LEDs) provided in the transmitter 100. The transmitter 100 changes its brightness by blinking its light emitting unit 142, and transmits an optical ID (light identification information) according to the change in brightness. This optical ID is the above-mentioned visible light signal.

The receiver 200 acquires an image capture display image Pa on which the transmitter 100 is projected by taking an image of the transmitter 100 at a normal exposure time, and at the same time, the transmitter 100 has a communication exposure time shorter than the normal exposure time. An image for decoding is acquired by taking an image of. The normal exposure time is the exposure time in the above-mentioned normal shooting mode, and the communication exposure time is the exposure time in the above-mentioned visible light communication mode.

The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P42 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes a region corresponding to the recognition information in the captured display image Pa as a target region. Then, the receiver 200 superimposes the AR image P42 on the target area, and displays the captured display image Pa on which the AR image P42 is superposed on the display 201.

For example, the receiver 200 recognizes the area on the upper left of the area where the transmitter 100 is projected as the target area according to the recognition information, as in the example shown in FIG. 245. As a result, for example, the AR image P42 showing a fairy is displayed as if it is flying around the transmitter 100.

FIG. 349 is a diagram showing another example of the captured display image Pa on which the AR image P42 is superimposed.

As shown in FIG. 349, the receiver 200 displays the captured display image Pa on which the AR image P42 is superimposed on the display 201.

Here, the above-mentioned recognition information indicates that the range having the brightness equal to or higher than the threshold value in the captured display image Pa is the reference region. Further, the recognition information indicates that the target area is in a predetermined direction with respect to the reference area and that the target area is separated from the center (or the center of gravity) of the reference area by a predetermined distance. show.

Therefore, when the light emitting unit 142 of the transmitter 100 imaged by the receiver 200 fluctuates, as shown in FIG. 349, the AR image p42 superimposed on the target area of the captured display image Pa also moves the light emitting unit 142. Works to synchronize with. That is, when the light emitting unit 142 fluctuates, the image 142a of the light emitting unit 142 projected on the captured display image Pa also fluctuates. The image 142a is a range having a brightness equal to or higher than the above-mentioned threshold value and is a reference region. That is, since the reference area moves, the receiver 200 moves the target area so that the distance between the reference area and the target area is maintained at a predetermined distance, and moves the target area to the moving target area. The AR image P42 is superimposed. As a result, when the light emitting unit 142 fluctuates, the AR image P42 superimposed on the target area of the captured display image Pa also moves in synchronization with the movement of the light emitting unit 142. The center position of the reference region may also move due to the deformation of the light emitting unit 142. Therefore, even when the light emitting unit 142 is deformed, the AR image 42 may move so that the distance from the center position of the moving reference region is maintained at a predetermined distance.

Further, in the above example, the receiver 200 recognizes the target area based on the recognition information and superimposes the AR image P42 on the target area, but the AR image P42 may be vibrated around the target area. .. That is, the receiver 200 vibrates the AR image P42, for example, in the vertical direction according to a function indicating a change in amplitude with respect to time. The function is a trigonometric function such as a sine wave.

Further, the receiver 200 may change the size of the AR image P42 according to the size of the range having the brightness equal to or higher than the above-mentioned threshold value. That is, the receiver 200 increases the size of the AR image P42 as the area of the bright region in the captured display image Pa increases, and conversely, decreases the size of the AR image P42 as the area of the bright region decreases.

Alternatively, the receiver 200 increases the size of the AR image P42 as the average brightness in the range having the brightness equal to or higher than the above threshold value increases, and conversely, decreases the size of the AR image P42 as the average brightness decreases. You may. Instead of the size of the AR image P42, the transparency of the AR image P42 may be changed according to the average brightness thereof.

Further, in the example shown in FIG. 349, in the image 142a of the light emitting unit 142, all the pixels have the luminance equal to or higher than the threshold value, but any pixel may be less than the threshold value. That is, the range having brightness equal to or higher than the threshold value corresponding to the image 142a may be annular. Also in this case, the range having the brightness equal to or higher than the threshold value is specified as the reference region, and the AR image P42 is superimposed on the target region separated from the center (or the center of gravity) of the reference region by a predetermined distance.

FIG. 350 is a diagram showing another example in which the receiver 200 in the present embodiment displays an AR image.

As shown in FIG. 350, for example, the transmitter 100 is configured as a lighting device, and transmits an optical ID by changing the brightness while illuminating, for example, a graphic 143 composed of three circles drawn on a wall. Since the figure 143 is illuminated by the light from the transmitter 100, the luminance changes like the transmitter 100, and the optical ID is transmitted.

The receiver 200 acquires the captured display image Pa and the decoding image in the same manner as described above by capturing the graphic 143 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the figure 143. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P43 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes a region corresponding to the recognition information in the captured display image Pa as a target region. For example, the receiver 200 recognizes the area on which the figure 143 is projected as the target area. Then, the receiver 200 superimposes the AR image P43 on the target area, and displays the captured display image Pa on which the AR image P43 is superposed on the display 201. For example, the AR image P43 is a face image of a character.

Here, the figure 143 is composed of three circles as described above, but the figure 143 has few geometric features. Therefore, it is difficult to appropriately select and acquire an AR image corresponding to the figure 143 from many images stored in the server only from the captured image obtained by capturing the figure 143. However, in the present embodiment, the receiver 200 acquires the optical ID and acquires the AR image P43 corresponding to the optical ID from the server. Therefore, even if many images are stored in the server, the AR image P43 corresponding to the optical ID can be appropriately selected and acquired from the many images as the AR image corresponding to the figure 143. can.

FIG. 351 is a flowchart showing the operation of the receiver 200 in the present embodiment.

The receiver 200 in the present embodiment first acquires a plurality of AR image candidates (step S541). For example, the receiver 200 acquires a plurality of AR image candidates from the server by wireless communication (BTLE, Wi-Fi, etc.) different from visible light communication. Next, the receiver 200 takes an image of the subject (step S542). As described above, the receiver 200 acquires the captured display image Pa and the decoding image by this imaging. However, when the subject is a photograph of the transmitter 100, the optical ID is not transmitted from the subject, so that the receiver 200 cannot acquire the optical ID even if the decoding image is decoded. Can not.

Therefore, the receiver 200 determines whether or not the optical ID could be acquired, that is, whether or not the optical ID was received from the subject (step S543).

Here, if it is determined that the optical ID has not been received (No in step S543), the receiver 200 determines whether or not the AR display flag set for itself is 1 (step S544). The AR display flag is a flag indicating whether or not the AR image may be displayed only from the captured display image Pa even if the optical ID is not acquired. When the AR display flag is 1, the AR display flag indicates that the AR image may be displayed only from the captured display image Pa, and when the AR display flag is 0, the AR display flag is set. , Indicates that the AR image should not be displayed only from the captured display image Pa.

When it is determined that the AR display flag is 1 (Yes in step S544), the receiver 200 selects a candidate corresponding to the captured display image Pa as the AR image from the plurality of AR image candidates acquired in step S541. (Step S545). That is, the receiver 200 extracts the feature amount included in the captured display image Pa, and selects the candidate associated with the extracted feature amount as the AR image.

Then, the receiver 200 superimposes and displays the selected candidate AR image on the image pickup display image Pa (step S546).

On the other hand, if it is determined that the AR display flag is 0 (No in step S544), the receiver 200 does not display the AR image.

Further, when it is determined that the optical ID has been received in step S543 (Yes in step S543), the receiver 200 selects a candidate associated with the optical ID from the plurality of AR image candidates acquired in step S541. Select as an AR image (step S547). Then, the receiver 200 superimposes and displays the selected candidate AR image on the image pickup display image Pa (step S546).

In the above example, the AR display flag is set in the receiver 200, but it may be set in the server. In this case, the receiver 200 inquires the server whether the AR display flag is 1 or 0 in step S544.

Thereby, when the receiver 200 does not receive the optical ID even if the image is taken, it is possible to control whether or not to display the AR image to the receiver 200 by the AR display flag.

FIG. 352 is a diagram for explaining the operation of the transmitter 100 in the present embodiment.

For example, the transmitter 100 is configured as a projector. Here, the intensity of the light emitted from the projector and reflected on the screen changes depending on each factor such as the aged deterioration of the light source of the projector or the distance from the light source to the screen. When the light intensity is low, it becomes difficult for the receiver 200 to receive the optical ID transmitted from the transmitter 100.

Therefore, the transmitter 100 in the present embodiment adjusts the parameters for causing the light source to emit light in order to suppress the change in the light intensity according to each factor. This parameter is at least one of the value of the current input to the light source to cause the light source to emit light and the light emission time (more specifically, the light emission time per unit time). For example, the larger the current value and the longer the light emission time, the higher the light intensity of the light source.

That is, the transmitter 100 adjusts the parameters so that the light source of the light source becomes stronger as the light source deteriorates over time. Specifically, the transmitter 100 includes a timer, and the parameter is adjusted so that the longer the usage time of the light source measured by the timer is, the stronger the light of the light source is. That is, the longer the usage time of the transmitter 100, the higher the value of the current of the light source and the longer the light emission time. Alternatively, the transmitter 100 detects the intensity of the light emitted from the light source and adjusts the parameters so that the intensity of the detected light does not decrease. That is, the transmitter 100 adjusts the parameters so that the smaller the detected light intensity, the stronger the light.

Further, the transmitter 100 adjusts the parameters so that the longer the irradiation distance from the light source to the screen, the stronger the light of the light source. Specifically, the transmitter 100 detects the intensity of the light irradiated and reflected on the screen, and adjusts the parameters so that the smaller the intensity of the detected light is, the stronger the light of the light source is. That is, the smaller the intensity of the detected light, the higher the current value of the light source and the longer the light emission time of the transmitter 100. As a result, the parameters are adjusted so that the intensity of the reflected light is constant regardless of the irradiation distance. Alternatively, the transmitter 100 detects the irradiation distance from the light source to the screen by the ranging sensor, and adjusts the parameter so that the longer the detected irradiation distance, the stronger the light of the light source.

Further, the transmitter 100 adjusts the parameters so that the darker the color of the screen, the stronger the light of the light source. Specifically, the transmitter 100 detects the color of the screen by imaging the screen, and adjusts the parameters so that the darker the detected color, the stronger the light of the light source. That is, the blacker the detected color, the higher the current value of the light source and the longer the light emission time of the transmitter 100. This adjusts the parameters so that the intensity of the reflected light is constant regardless of the color of the screen.

Further, the transmitter 100 adjusts the parameters so that the stronger the external light, the stronger the light of the light source. Specifically, the transmitter 100 detects the difference between the brightness of the screen when the light source is turned on and irradiated with light and the brightness of the screen when the light source is turned off and not irradiated with light. Then, the transmitter 100 adjusts the parameters so that the smaller the difference in brightness, the stronger the light of the light source. That is, the smaller the difference in brightness, the higher the current value of the light source of the transmitter 100 and the longer the light emission time. As a result, the parameters are adjusted so that the S / N ratio of the optical ID becomes constant regardless of the external light. Alternatively, when the transmitter 100 is configured as an LED display, for example, the intensity of sunlight may be detected, and the parameter may be adjusted so that the higher the intensity of the sunlight, the stronger the light of the light source. ..

It should be noted that the parameter adjustment as described above may be performed when an operation is performed by the user. For example, the transmitter 100 includes a calibration button, and when the calibration button is pressed by the user, the above-mentioned parameter adjustment is performed. Alternatively, the transmitter 100 may periodically adjust the above-mentioned parameters.

FIG. 353 is a diagram for explaining other operations of the transmitter 100 in the present embodiment.

For example, the transmitter 100 is configured as a projector and irradiates a screen with light from a light source through a front member. When the projector is a liquid crystal projector, its front member is a liquid crystal panel, and when the projector is a DLP (registered trademark) projector, its front member is a DMD (Digital Mirror Device). That is, the front member is a member that adjusts the brightness of the image for each pixel. Further, the light source irradiates light toward the front member, and the intensity of the light is switched between High and Low. Further, the light source adjusts the time average brightness by adjusting the High time per unit time.

Here, when the transmittance of the front member is, for example, 100%, the light source is darkened so that the image projected from the projector to the screen is not too bright. That is, the light source shortens the High time per unit time.

At this time, when the light source transmits the optical ID by changing the luminance, the light source widens the pulse width of the optical ID.

On the other hand, when the transmittance of the front member is, for example, 20%, the light source becomes bright so that the image projected from the projector to the screen is not too dark. That is, the light source increases the High time per unit time.

At this time, when the light source transmits the optical ID by changing the luminance, the light source narrows the pulse width of the optical ID.

As described above, when the light source is dark, the pulse width of the optical ID becomes wide, and conversely, when the light source is bright, the pulse width of the optical ID becomes narrow. Therefore, the light from the light source is transmitted by transmitting the optical ID. It is possible to prevent the intensity of the light from becoming too weak or too bright.

In the above example, the transmitter 100 is a projector, but it may be configured as a large LED display. The large LED display includes a pixel switch and a common switch as shown in FIGS. 173, 175 and 180B. An image is expressed by turning on and off the pixel switch, and an optical ID is transmitted by turning on and off the common switch. In this case, functionally, the pixel switch corresponds to the front member and the common switch corresponds to the light source. When the average brightness by the pixel switch is high, the pulse width of the optical ID by the common switch may be shortened.

FIG. 354 is a diagram for explaining other operations of the transmitter 100 in the present embodiment. Specifically, FIG. 354 shows the dimming degree of the transmitter 100 configured as a spotlight with a dimming function and the current (specifically, the value of the peak current) input to the light source of the transmitter 100. Show the relationship.

The transmitter 100 receives a light intensity designated for the light source provided in the transmitter 100, and causes the light source to emit light at the designated luminous intensity. The luminous intensity is the ratio of the average brightness of the light source to the maximum average brightness. The average brightness is not the instantaneous brightness but the brightness in the time average. Further, the adjustment of the dimming degree is realized by adjusting the value of the current input to the light source and adjusting the time when the brightness of the light source becomes Low. The time when the brightness of the light source becomes Low may be the time when the light source is turned off.

Here, when the transmitter 100 transmits the transmission target signal as an optical ID, the transmitter 100 generates a coded signal by encoding the transmission target signal in a predetermined mode. Then, the transmitter 100 transmits the coded signal as an optical ID (that is, a visible light signal) by changing the luminance of the light source provided therein according to the coded signal.

For example, when the specified dimming intensity is 0% or more and x3 (%) or less, the transmitter 100 generates a coded signal by encoding the transmission target signal in the PWM mode having a duty ratio of 35%. .. x3 (%) is, for example, 50%. In the present embodiment, the PWM mode having a duty ratio of 35% is also referred to as a first mode, and the above-mentioned x3 is also referred to as a first value.

That is, when the specified luminous intensity is 0% or more and x3 (%) or less, the transmitter 100 sets the luminous intensity of the light source to the peak current value while maintaining the duty ratio of the visible light signal at 35%. Adjust by.

Further, when the specified dimming intensity is larger than x3 (%) and 100% or less, the transmitter 100 encodes the coded signal by encoding the transmission target signal in the PWM mode having a duty ratio of 65%. Generate. In the present embodiment, the PWM mode having a duty ratio of 65% is also referred to as a second mode.

That is, when the specified luminous intensity is larger than x3 (%) and 100% or less, the transmitter 100 keeps the duty ratio of the visible light signal at 65% and peaks the luminous intensity of the light source. Adjust according to the value of.

As described above, the transmitter 100 in the present embodiment accepts the dimming intensity designated for the light source as the designated dimming intensity. Then, when the designated luminous intensity is equal to or less than the first value, the transmitter 100 transmits the signal encoded in the first mode by changing the brightness while causing the light source to emit light at the designated luminous intensity. Further, when the value of the designated luminous intensity is larger than the first value, the transmitter 100 transmits the signal encoded in the second mode by the change in brightness while causing the light source to emit light at the designated luminous intensity. do. Specifically, the duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode.

Here, since the duty ratio of the second mode is larger than the duty ratio of the first mode, the rate of change of the peak current with respect to the dimming degree in the second mode is the change of the peak current with respect to the dimming degree in the first mode. It can be smaller than the rate.

Further, when the designated luminous intensity exceeds x3 (%), the mode is switched from the first mode to the second mode. Therefore, at this time, the peak current can be instantaneously reduced. That is, when the specified luminous intensity is x3 (%), the peak current is y3 (mA), but when the specified luminous intensity exceeds x3 (%) even a little, the peak current is y2 (mA). Can be suppressed to. In addition, y3 (mA) is, for example, 143 mA, and y2 (mA) is, for example, 100 mA. As a result, in order to increase the dimming degree, it is possible to suppress the peak current from becoming larger than y3 (mA), and it is possible to suppress the deterioration of the light source due to the flow of a large current.

Further, when the designated luminous intensity exceeds x4 (%), the peak current becomes larger than y3 (mA) even if the mode is the second mode. However, when the designated luminous intensity exceeds x4 (%) infrequently, deterioration of the light source can be suppressed. In this embodiment, the above-mentioned x4 is also referred to as a second value. Further, in the example shown in FIG. 354, x4 (%) is less than 100%, but may be 100%.

That is, in the transmitter 100 in the present embodiment, when the designated luminous intensity is larger than the first value and equal to or less than the second value, the signal encoded in the second mode is transmitted by the change in brightness. The value of the peak current of the light source is smaller than the value of the peak current of the light source for transmitting the signal encoded in the first mode by the luminance change when the designated luminous intensity is the first value.

As a result, the peak current value of the light source when the specified dimming degree is larger than the first value and equal to or less than the second value by switching the mode for encoding the signal is such that the specified dimming degree is the first value. It is smaller than the peak current value of the light source in some cases. Therefore, the larger the designated luminous intensity, the more the large peak current can be suppressed from flowing to the light source. As a result, deterioration of the light source can be suppressed.

Further, in the transmitter 100 of the present embodiment, when the designated luminous intensity is x1 (%) or more and smaller than x2 (%), the transmitter 100 emits light at the designated luminous intensity while emitting light in the first mode. The signal encoded by is transmitted by the change in brightness, and the value of the peak current is maintained at a constant value with respect to the change in the specified dimming intensity. x2 (%) is smaller than x3 (%). In this embodiment, the above-mentioned x2 is also referred to as a third value.

That is, when the designated luminous intensity is smaller than x2 (%), the transmitter 100 decreases the designated luminous intensity by increasing the time for turning off the light source as the designated luminous intensity decreases. The light source is made to emit light, and the peak current value is maintained at a constant value. Specifically, the transmitter 100 lengthens the period for transmitting each of the plurality of coded signals while maintaining the duty ratio of the coded signals at 35%. This increases the time for turning off the light source, that is, the period for turning off the light. As a result, the dimming degree can be reduced while maintaining the peak current value constant. Further, even when the designated luminous intensity becomes small, the value of the peak current is maintained constant, so that the receiver 200 can easily receive the visible light signal (that is, the optical ID) which is a signal transmitted by the change in luminance. be able to.

Here, the transmitter 100 determines the time for turning off the light source so that one cycle, which is the sum of the time for transmitting the coded signal due to the change in luminance and the time for turning off the light source, does not exceed 10 milliseconds. .. For example, if the time for turning off the light source is too long and one cycle exceeds 10 milliseconds, the change in the brightness of the light source for transmitting the coded signal may be perceived by the human eye as flickering. .. Therefore, in the present embodiment, the time for turning off the light source is determined so that one cycle does not exceed 10 milliseconds, so that it is possible to suppress the recognition of flicker by a person.

Further, even when the designated luminous intensity is smaller than x1 (%), the transmitter 100 transmits the signal encoded in the first mode by the luminance change while causing the light source to emit light at the designated luminous intensity. At this time, the transmitter 100 causes the light source to emit light at the designated dimming intensity by reducing the value of the peak current as the designated dimming intensity becomes smaller. x1 (%) is smaller than x2 (%). In this embodiment, the above-mentioned x1 is also referred to as a fourth value.

As a result, even if the designated luminous intensity is smaller, the light source can be appropriately made to emit light at the designated luminous intensity.

Here, in the example shown in FIG. 354, the value of the maximum peak current in the first mode (that is, y3 (mA)) is larger than the value of the maximum peak current in the second mode (that is, y4 (mA)). Small but may be the same. That is, the transmitter 100 encodes the transmission target signal in the first mode up to x3a (%) whose designated luminous intensity is larger than x3 (%). Then, when the specified luminous intensity is x3a (%), the transmitter 100 sets the light source at the same peak current value as the maximum peak current value (that is, y4 (mA)) in the second mode. Make it emit light. In this case, x3a is the first value. The value of the maximum peak current in the second mode is the value of the peak current when the designated dimming intensity is the maximum value, that is, 100%.

That is, in the present embodiment, the value of the peak current of the light source when the designated luminous intensity is the first value and the value of the peak current of the light source when the designated luminous intensity is the maximum value are the same. There may be. In this case, since the range of the luminous intensity for causing the light source to emit light with a peak current of y3 (mA) or more is widened, it is possible to facilitate the receiver 200 to receive the optical ID in the wide range of the luminous intensity. In other words, even in the first mode, a large peak current can be passed through the light source, so that the signal transmitted by the change in the luminance of the light source can be easily received by the receiver. In this case, the period during which the large peak current flows becomes long, so that the light source tends to deteriorate.

FIG. 355 is a diagram showing a comparative example for explaining the ease of receiving the optical ID in the present embodiment.

In this embodiment, as shown in FIG. 354, the first mode is used when the dimming degree is small, and the second mode is used when the dimming degree is large. The first mode is a mode in which the increase in peak current is large even if the increase in dimming intensity is small, and the second mode is a mode in which the increase in peak current is suppressed even if the increase in dimming intensity is large. Therefore, the second mode suppresses the flow of a large peak current to the light source, so that deterioration of the light source can be suppressed. Further, in the first mode, a large peak current flows through the light source even if the dimming degree is small, so that the receiver 200 can easily receive the optical ID.

On the other hand, when the second mode is used even when the dimming degree is small, as shown in FIG. 355, when the dimming degree is small, the peak current value is also small, so that the optical ID is sent to the receiver 200. It becomes difficult to receive.

Therefore, in the transmitter 100 in the present embodiment, it is possible to achieve both suppression of deterioration of the light source and ease of receiving the optical ID.

Further, when the value of the peak current of the light source exceeds the fifth value, the transmitter 100 may stop the transmission of the signal due to the change in the luminance of the light source. The fifth value may be, for example, y3 (mA).

As a result, deterioration of the light source can be further suppressed.

Further, the transmitter 100 may measure the usage time of the light source as in the example shown in FIG. 352. Then, when the usage time is longer than a predetermined time, the transmitter 100 may transmit a signal by changing the brightness by using the value of the parameter for causing the light source to emit light at a dimming intensity larger than the designated dimming intensity. .. In this case, the value of the parameter may be the value of the peak current or the time to turn off the light source. As a result, it is possible to prevent the optical ID from being difficult to be received by the receiver 200 due to deterioration of the light source over time.

Alternatively, the transmitter 100 may measure the usage time of the light source, and when the usage time is longer than the predetermined time, the pulse width of the current of the light source may be larger than when the usage time is less than the predetermined time. .. Thereby, as described above, it is possible to suppress the difficulty in receiving the optical ID due to the deterioration of the light source.

In the above embodiment, the transmitter 100 switches between the first mode and the second mode according to the designated luminous intensity, but the mode may be switched according to the operation by the user. good. That is, when the switch is operated by the user, the transmitter 100 switches the first mode to the second mode, and conversely, switches the second mode to the first mode. Further, the transmitter 100 may notify the user when the mode is switched. For example, the transmitter 100 may notify the user of the mode change by making a sound, blinking the light source at a cycle recognizable by a person, or turning on the notification LED. Further, the transmitter 100 may notify the user that the relationship changes not only when the mode is switched but also when the relationship between the peak current and the luminous intensity changes. The time point is, for example, a time point when the dimming degree shown in FIG. 354 changes from x1 (%), or a time point when the dimming degree changes from x2 (%).

FIG. 356A is a flowchart showing the operation of the transmitter 100 in the present embodiment.

First, the transmitter 100 accepts the dimming intensity designated for the light source as the designated dimming intensity (step S551). Next, the transmitter 100 transmits the signal according to the change in the brightness of the light source (step S552). Specifically, when the designated luminous intensity is equal to or less than the first value, the transmitter 100 emits a light source at the designated luminous intensity and changes the brightness of the signal encoded in the first mode. Send. Further, when the designated luminous intensity is larger than the first value, the transmitter 100 transmits the signal encoded in the second mode by the luminance change while causing the light source to emit light at the designated luminous intensity. Here, when the designated luminous intensity is larger than the first value and is equal to or less than the second value, the value of the peak current of the light source for transmitting the signal encoded in the second mode by the luminance change is set. When the designated luminous intensity is the first value, it is smaller than the value of the peak current of the light source for transmitting the signal encoded in the first mode by the luminance change.

FIG. 356B is a block diagram showing the configuration of the transmitter 100 according to the present embodiment.

The transmitter 100 includes a reception unit 551 and a transmitter unit 552. The reception unit 551 receives the dimming intensity designated for the light source as the designated dimming intensity (step S551). The transmission unit 552 transmits a signal by changing the brightness of the light source. Specifically, when the designated luminous intensity is equal to or less than the first value, the transmission unit 552 emits a light source at the designated luminous intensity and changes the luminance of the signal encoded in the first mode. Send. Further, when the designated luminous intensity is larger than the first value, the transmission unit 552 transmits the signal encoded in the second mode by the luminance change while causing the light source to emit light at the designated luminous intensity. Here, when the designated luminous intensity is larger than the first value and is equal to or less than the second value, the value of the peak current of the light source for transmitting the signal encoded in the second mode by the luminance change is set. When the designated luminous intensity is the first value, it is smaller than the value of the peak current of the light source for transmitting the signal encoded in the first mode by the luminance change.

As a result, as shown in FIG. 354, the value of the peak current of the light source when the specified dimming intensity is larger than the first value and equal to or less than the second value by switching the mode for encoding the signal is set to the specified adjustment. It is smaller than the peak current value of the light source when the luminous intensity is the first value. Therefore, the larger the designated luminous intensity, the more the large peak current can be suppressed from flowing to the light source. As a result, deterioration of the light source can be suppressed.

FIG. 357 is a diagram showing another example in which the receiver 200 in the present embodiment displays an AR image.

The receiver 200 acquires a captured display image Pk, which is the above-mentioned normal captured image, and a decoding image, which is the above-mentioned visible light communication image or emission line image, by capturing the subject by the image sensor.

Specifically, the image sensor of the receiver 200 images a transmitter 100 configured as a signage and a person 21 next to the transmitter 100. The transmitter 100 is the transmitter in each of the above embodiments, and includes one or a plurality of light emitting elements (for example, LEDs) and a translucent plate 144 having translucency such as frosted glass. The one or more light emitting elements emit light inside the transmitter 100, and the light from the one or more light emitting elements passes through the translucent plate 144 and is irradiated to the outside. As a result, the translucent plate 144 of the transmitter 100 is in a brightly shining state. Such a transmitter 100 changes the brightness by blinking one or a plurality of light emitting elements, and transmits an optical ID (light identification information) according to the change in brightness. This optical ID is the above-mentioned visible light signal.

Here, the message "Please hold your smartphone over here" is written on the translucent plate 144. Therefore, the user of the receiver 200 causes the person 21 to stand next to the transmitter 100 and instruct the person 21 to put his arm on the transmitter 100. Then, the user points the camera (that is, the image sensor) of the receiver 200 toward the person 21 and the transmitter 100 to take an image. The receiver 200 acquires an image capture display image Pk on which the transmitter 100 and the person 21 are imaged at a normal exposure time. Further, the receiver 200 acquires a decoding image by taking an image of the transmitter 100 and the person 21 with a communication exposure time shorter than the normal exposure time.

The receiver 200 acquires the optical ID by decoding the image for decoding. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P44 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes a region corresponding to the recognition information in the captured display image Pk as a target region. For example, the receiver 200 recognizes the area where the signage, which is the transmitter 100, is projected as the target area.

Then, the receiver 200 superimposes the AR image P44 on the image pickup display image Pk so that the target area is covered by the AR image P44, and displays the image pickup display image Pk on the display 201. For example, the receiver 200 acquires an AR image P44 showing a soccer player. In this case, since the AR image P44 is superimposed so as to cover the target area of the captured display image Pk, the captured display image Pk can be displayed so that the soccer player actually exists next to the person 21. .. As a result, the person 21 can be photographed together with the soccer player even if the soccer player is not next to him. More specifically, the person 21 can be photographed together with the soccer player with the arm of the person 21 resting on the shoulder of the soccer player.

(Embodiment 27)
In this embodiment, a transmission method for transmitting an optical ID by a visible light signal will be described. The transmitter and receiver in this embodiment may have the same functions and configurations as the transmitter (or transmitter) and receiver (or receiver) in each of the above embodiments.

FIG. 358 is a diagram for explaining the operation of the transmitter 100 in the present embodiment. Specifically, FIG. 358 shows the dimming degree of the transmitter 100 configured as a spotlight with a dimming function and the current (specifically, the value of the peak current) input to the light source of the transmitter 100. Show the relationship.

When the designated luminous intensity is 0% or more and x14 (%) or less, the transmitter 100 in the present embodiment encodes a transmission target signal in a PWM mode having a duty ratio of 35%, thereby encoding a signal. To generate. That is, the transmitter 100 increases the peak current value while maintaining the duty ratio of the visible light signal at 35% when the specified dimming intensity changes from 0% to x14 (%). , Makes the light source emit light at the specified dimming intensity. The PWM mode having a duty ratio of 35% is also referred to as a first mode as in the 26th embodiment, and the above-mentioned x14 is also referred to as a first value. For example, x14 (%) is a value in the range of 50-60%.

Further, when the designated luminous intensity is x13 (%) or more and 100% or less, the transmitter 100 generates a coded signal by encoding the transmission target signal in the PWM mode having a duty ratio of 65%. .. That is, when the specified luminous intensity changes from 100% to x13 (%), the transmitter 100 suppresses the peak current value while maintaining the duty ratio of the visible light signal at 65%. The light source is made to emit light at the specified dimming intensity. The PWM mode having a duty ratio of 65% is also referred to as a second mode as in the 26th embodiment, and the above-mentioned x13 is also referred to as a second value. Here, x13 (%) is a value smaller than x14 (%), and is, for example, a value in the range of 40 to 50%.

As described above, in the present embodiment, when the designated dimming degree increases, the PWM mode changes from the PWM mode having a duty ratio of 35% to the PWM mode having a duty ratio of 65% at the dimming degree x14 (%). Can be switched. On the other hand, when the specified dimming degree decreases, the PWM mode changes from the PWM mode with a duty ratio of 65% to the PWM mode with a duty ratio of 35% at a dimming degree x13 (%) smaller than the dimming degree x14 (%). Can be switched to. That is, in the present embodiment, the dimming degree at which the PWM mode can be switched differs depending on whether the designated dimming degree increases or the designated dimming degree decreases. Hereinafter, the dimming degree at which the PWM mode can be switched is referred to as a switching point.

Therefore, in the present embodiment, frequent switching of the PWM mode can be suppressed. In the example shown in FIG. 354 of the twenty-sixth embodiment, the switching point of the PWM mode is 50%, which is the same when the specified dimming degree increases and when the designated dimming degree decreases. .. As a result, in the example shown in FIG. 354, when the specified dimming intensity is repeatedly increased or decreased around 50%, the PWM mode frequently changes to the PWM mode having a duty ratio of 35% and the PWM mode having a duty ratio of 65%. Can be switched. However, in the present embodiment, since the switching point of the PWM mode is different depending on whether the designated dimming degree increases or the designated dimming degree decreases, such frequent switching of the PWM mode is performed. It can be suppressed.

Further, in the present embodiment, as in the example shown in FIG. 354 of the 26th embodiment, when the specified dimming degree is small, the PWM mode having a small duty ratio is used, and conversely, the designated dimming degree is used. When is large, a PWM mode with a large duty ratio is used.

Therefore, when the specified dimming intensity is large, the PWM mode with a large duty ratio is used, so that the rate of change of the peak current with respect to the dimming intensity can be reduced, and the light source emits light at a large dimming intensity due to the small peak current. Can be made to. For example, in the PWM mode with a small duty ratio such as a duty ratio of 35%, the light source cannot be made to emit light at 100% dimming unless the peak current is set to 250 mA. However, in the present embodiment, for a large luminous intensity, a PWM mode having a large duty ratio such as a duty ratio of 65% is used. Therefore, for example, the light source can be changed to 100 by simply setting the peak current to a smaller value of 154 mA. It can emit light with a dimming intensity of%. That is, it is possible to prevent the life of the light source from being shortened by passing an overcurrent through the light source.

Further, when the designated dimming degree is small, the PWM mode having a small duty ratio is used, so that the rate of change of the peak current with respect to the dimming degree can be increased. As a result, it is possible to transmit a visible light signal with a large peak current while causing the light source to emit light with a small dimming intensity. The light source emits brighter as the input current is larger. Therefore, when the visible light signal is transmitted by a large peak current, it is possible to facilitate the receiver 200 to receive the visible light signal. In other words, the range of dimming degree at which the visible light signal that can be received to the receiver 200 can be transmitted can be expanded to a smaller dimming degree. For example, as shown in FIG. 385, if the peak current is Ia (mA) or more, the receiver 200 can receive the visible light signal transmitted by the peak current. In this case, in the PWM mode having a large duty ratio such as a duty ratio of 65%, the range of dimming degree at which a receivable visible light signal can be transmitted is x12 (%) or more. However, in the PWM mode with a duty ratio as small as 35%, the range of dimming degree at which a receivable visible light signal can be transmitted is set to x11 (%) or more, which is smaller than x12 (%). Can be done.

By switching the PWM mode in this way, the life of the light source can be extended and the visible light signal can be transmitted in a wide dimming range.

FIG. 359A is a flowchart showing a transmission method according to the present embodiment.

The transmission method in the present embodiment is a transmission method for transmitting a signal by changing the brightness of a light source, and includes a reception step S561 and a transmission step S562. In the reception step S561, the transmitter 100 receives the dimming intensity designated for the light source as the designated dimming intensity. In the transmission step S562, the transmitter 100 transmits the signal encoded in the first mode or the second mode by the change in luminance while causing the light source to emit light at the designated luminous intensity. Here, the duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode. Further, in the transmission step S562, when the designated luminous intensity is changed from a small value to a large value, the transmitter 100 sets the mode used for coding the signal when the designated luminous intensity is the first value. Switch from the first mode to the second mode. Further, when the designated luminous intensity is changed from a large value to a small value, the transmitter 100 sets the mode used for coding the signal from the second mode when the designated luminous intensity is the second value. Switch to the first mode. Here, the second value is smaller than the first value.

For example, the first mode and the second mode are a PWM mode having a duty ratio of 35% and a PWM mode having a duty ratio of 65%, respectively, as shown in FIG. 358. The first value and the second value are x14 (%) and x15 (%) shown in FIG. 358, respectively.

As a result, the designated luminous intensity (that is, the switching point) at which the first mode and the second mode are switched differs depending on whether the designated luminous intensity increases or decreases. Therefore, frequent switching between these modes can be suppressed. That is, the occurrence of so-called chattering can be suppressed. As a result, the operation of the transmitter 100 that transmits a signal can be stabilized. Also, the duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode. Therefore, as in the transmission method shown in FIG. 354, the larger the designated luminous intensity, the more the large peak current can be suppressed from flowing to the light source. As a result, deterioration of the light source can be suppressed. In addition, since deterioration of the light source is suppressed, communication between various devices can be performed for a long period of time. When the designated luminous intensity is small, the first mode with a small duty ratio is used. Therefore, the above-mentioned peak current can be increased, and a signal easily received by the receiver 200 can be transmitted as a visible light signal.

Further, in the transmission step S562, when the mode is switched from the first mode to the second mode, the transmitter 100 sets the peak current of the light source for transmitting the encoded signal by the change in luminance. The current value of 1 is changed to a second current value smaller than the first current value. Further, when switching from the second mode to the first mode, the transmitter 100 changes the peak current from the third current value to the fourth current value larger than the third current value. change. Here, the first current value is larger than the fourth current value, and the second current value is larger than the third current value.

For example, the first current value, the second current value, the third current value, and the fourth current value are the current values Ie, the current value Ic, the current value Ib, and the current value Id shown in FIG. 358, respectively. be.

This makes it possible to appropriately switch between the first mode and the second mode.

FIG. 359B is a block diagram showing the configuration of the transmitter 100 according to the present embodiment.

The transmitter 100 in the present embodiment is a transmitter that transmits a signal by changing the brightness of a light source, and includes a reception unit 561 and a transmission unit 562. The reception unit 561 receives the dimming intensity designated for the light source as the designated dimming intensity. The transmission unit 562 transmits a signal encoded in the first mode or the second mode by changing the luminance while causing the light source to emit light at the designated luminous intensity. Here, the duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode. Further, when the designated luminous intensity is changed from a small value to a large value, the transmission unit 562 sets the mode used for coding the signal from the first mode when the designated luminous intensity is the first value. Switch to the second mode. Further, when the designated luminous intensity is changed from a large value to a small value, the transmission unit 562 sets the mode used for coding the signal from the second mode when the designated luminous intensity is the second value. Switch to the first mode. Here, the second value is smaller than the first value.

With such a transmitter 100, the transmission method of the flowchart shown in FIG. 359A is realized.

FIG. 360 is a diagram showing an example of a detailed configuration of a visible light signal according to the present embodiment.

Such a visible light signal is a PWM mode signal as in FIG. 188, FIG. 189A (b), FIG. 197, FIG. 212, FIG. 316, and FIG. 317.

The visible light signal packet includes an L data unit, a preamble, and an R data unit. The L data unit and the R data unit each correspond to a payload.

The preamble corresponds to the preamble of FIGS. 188, 189A (b), 197 and 212, and corresponds to the SHR of FIGS. 316 and 317. Specifically, the preamble alternately indicates the high and low luminance values along the time axis. That is, the preamble shows the high brightness value only for the time length C ₀ , shows the low brightness value only for the next time length C ₁ , shows the high brightness value only for the next time length C ₂ , and shows the high brightness value only for the next time length C 2. Only ₃ shows the brightness value of Low. The time lengths C ₀ and C ₃ are, for example, 100 μs. Further, the time lengths C ₁ and C ₂ are 90 μs shorter than, for example, the time lengths C ₁ and C ₃ by 10 μs.

The L data unit corresponds to the data L in FIGS. 188 and 189A (b), 197 and 212, and corresponds to the PHY payload A in FIGS. 316 and 317. Specifically, the L data unit alternately shows the luminance values of High and Low along the time axis, and is arranged immediately before the preamble. That is, the L data unit shows the high brightness value only for the time length D' ₀ , shows the low brightness value only for the next time length D' ₁ , and shows the high brightness value only for the next time length D' ₂ . The luminance value of Low is shown only for the next time length D' ₃ . The time lengths D' ₀ to D' ₃ are determined according to a mathematical formula according to the signal to be transmitted. This formula is D' ₀ = W ₀ + W ₁ x (3-y ₀ ), D' ₁ = W ₀ + W ₁ x (7-y ₁ ), D' ₂ = W ₀ + W ₁ x (3-y ₂ ). ), And D' ₃ = W ₀ + W ₁ × (7-y ₃ ). Here, the constant W ₀ is, for example, 110 μs, and the constant W ₁ is, for example, 30 μs. The variables y ₀ and y ₂ are any integers of 0 to 3 represented by 2 bits, and the variables y ₁ and y ₃ are any integers of 0 to 7 represented by 3 bits. The variables y ₀ to y ₃ are signals to be transmitted. In FIGS. 360 to 363, "*" is used as a symbol meaning multiplication.

The R data unit corresponds to the data R of FIGS. 188 and 189A (b), 197 and 212, and corresponds to the PHY payload B of FIGS. 316 and 317. Specifically, the R data unit alternately shows the luminance values of High and Low along the time axis, and is arranged immediately after the preamble. That is, the R data unit shows the high brightness value only for the time length D ₀ , shows the low brightness value only for the next time length D ₁ , and shows the high brightness value only for the next time length D ₂ , and then shows the next time. Only the length _D3 shows the low brightness value. The time lengths D ₀ to D ₃ are determined according to a mathematical formula according to the signal to be transmitted. This formula is D ₀ = W ₀ + W ₁ × y ₀ , D ₁ = W ₀ + W ₁ × y ₁ , D ₂ = W ₀ + W ₁ × y ₂ , and D ₃ = W ₀ + W ₁ × y ₃ . ..

Here, the L data unit and the R data unit have a complementary relationship with respect to the brightness. That is, if the brightness of the L data section is bright, the brightness of the R data section is dark, and conversely, if the brightness of the L data section is dark, the brightness of the R data section is bright. That is, the sum of the time length of the L data unit and the time length of the R data unit is constant regardless of the signal to be transmitted. In other words, the time-average brightness of the visible light signal transmitted from the light source can be made constant regardless of the signal to be transmitted.

In addition, D' ₀ = W ₀ + W ₁ x (3-y ₀ ), D' ₁ = W ₀ + W ₁ x (7-y ₁ ), D' ₂ = W ₀ + W ₁ x (3-y ₂ ), And the duty ratio of the PWM mode can be changed by changing the ratio of 3 and 7 in D' ₃ = W ₀ + W ₁ × (7−y ₃ ). The ratio of 3 and 7 corresponds to the ratio between the maximum value of the variables y ₀ and y ₂ and the maximum value of the variables y ₁ and y ₃ . For example, when the ratio is 3: 7, the PWM mode having a small duty ratio is selected, and conversely, when the ratio is 7: 3, the PWM mode having a large duty ratio is selected. Therefore, by adjusting the ratio, the PWM mode can be switched between the PWM mode having a duty ratio of 35% and the PWM mode having a duty ratio of 65% shown in FIGS. 354 and 358. Further, a preamble may be used to notify the receiver 200 of which PWM mode has been switched to. For example, the transmitter 100 notifies the receiver 200 of the switched PWM mode by including a preamble of the pattern associated with the switched PWM mode in the packet. The preamble pattern is changed by the time lengths C ₀ , C ₁ , C ₂ and C ₃ .

However, in the visible light signal shown in FIG. 360, since the packet contains two data units, it takes time to transmit the packet. For example, when the transmitter 100 is a DLP projector, the transmitter 100 projects red, green, and blue images in a time-division manner. Here, it is desirable that the transmitter 100 transmits a visible light signal when projecting a red image. This is because the visible light signal transmitted at this time has a red wavelength and is easily received by the receiver 200. The period during which the red image is continuously projected is, for example, 1.5 ms. In addition, this period is hereinafter referred to as a red projection period. It is difficult to transmit the packet including the above-mentioned L data unit, preamble, and R data unit in such a short red projection period.

Therefore, a packet having only the R data unit out of the two data units is recalled.

FIG. 361 is a diagram showing another example of the detailed configuration of the visible light signal in the present embodiment.

The visible light signal packet shown in FIG. 361 does not include the L data unit, unlike the example shown in FIG. 360. Instead, the visible light signal packet shown in FIG. 361 includes invalid data and an average luminance adjuster.

The invalid data alternately indicate the high and low luminance values along the time axis. That is, the invalid data shows the high luminance value only for the time length A ₀ , and shows the low luminance value only for the next time length A ₁ . The time length A ₀ is, for example, 100 μs, and the time length A ₁ is indicated by, for example, A ₁ = W ₀ − W ₁ . Such invalid data indicates that the packet does not include the L data part.

The average luminance adjusting unit alternately shows the luminance values of High and Low along the time axis. That is, the invalid data shows the high luminance value only for the time length B ₀ , and shows the low luminance value only for the next time length B ₁ . The time length B ₀ is indicated by, for example, B ₀ = 100 + W ₁ × ((3-y ₀ ) + (3-y ₂ )), and the time length B ₁ is, for example, B ₁ = W ₁ × ((7-y)). ₁ ) + (7-y ₃ )).

By such an average luminance adjusting unit, the average luminance in the packet can be made constant regardless of the signals y ₀ to y ₃ to be transmitted. That is, the total sum of the time lengths whose luminance value is High in the packet (that is, the total ON time) can be set to A ₀ + C ₀ + C ₂ + D ₀ + D ₂ + B ₀ = 790. Further, the sum of the time lengths whose luminance value is Low in the packet (that is, the total OFF time) can be set to A ₁ + C ₁ + C ₃ + D ₁ + D ₃ + B ₁ = 910.

However, even with such a visible light signal configuration, the effective time length E ₁ which is a part of the total time length E ₀ in the packet cannot be shortened. The effective time length E ₁ is the time from the first appearance of the high luminance value in the packet to the end of the last appearance of the high luminance value, and the receiver 200 demodulates or decodes the visible light signal packet. It's the time needed to do it. Specifically, the effective time length E ₁ is E ₁ = A ₀ + A ₁ + C ₀ + C ₁ + C ₂ + C ₃ + D ₀ + D ₁ + D ₂ + D ₃ + B ₀ . The total time length E ₀ is E ₀ = E ₁ + B ₁ .

That is, since the effective time length E ₁ is a maximum of 1700 μs even for the visible light signal having the configuration shown in FIG. 361, the transmitter 100 continues for the effective time length E ₁ during the above-mentioned red projection period. It is difficult to send one packet.

Therefore, in order to shorten the effective time length E ₁ and make the average luminance of the packet constant regardless of the signal to be transmitted, not only the time length of each luminance value of High and Low, but also the luminance value of High. Is also recalled to be adjusted.

FIG. 362 is a diagram showing another example of the detailed configuration of the visible light signal in the present embodiment.

In the visible light signal packet shown in FIG. 362, unlike the example shown in FIG. 361, in order to shorten the effective time length E1, the time length B ₀ of the high luminance _value of the average luminance adjustment unit is the signal to be transmitted. Regardless, it is fixed at the shortest 100 μs. Instead, in the visible light signal packet shown in FIG. 362, the high luminance value is set according to the variables y ₀ and y ₂ contained in the signal to be transmitted, that is, according to the time lengths D ₀ and D ₂ . It will be adjusted. For example, when the time lengths D ₀ and D ₂ are short, the transmitter 100 adjusts the high luminance value to a large value as shown in FIG. 362 (a). Further, when the time lengths D ₀ and D ₂ are long, the transmitter 100 adjusts the high luminance value to a small value as shown in FIG. 362 (b). Specifically, when the time lengths D ₀ and D ₂ are the shortest W ₀ (for example, 110 μs), the brightness value of High is 100% brightness. Further, when the time lengths D ₀ and D ₂ are the maximum “W ₀ + 3 W ₁ ” (for example, 200 μs), the brightness value of High is 77.2%.

In such a visible light signal packet, the sum of the time lengths whose luminance value is High (that is, the total ON time) can be set to, for example, A ₀ + C ₀ + C ₂ + D ₀ + D ₂ + B ₀ = 610 to 790. .. Further, the total sum of the time lengths whose luminance value is Low (that is, the total OFF time) can be set to A ₁ + C ₁ + C ₃ + D ₁ + D ₃ + B ₁ = 910.

However, in the visible light signal shown in FIG. 362, the shortest time lengths of the total time length E ₀ and the effective time length E ₁ in the packet can be made shorter than the example shown in FIG. 361, but the maximum time can be set. The length cannot be shortened.

Therefore, in order to shorten the effective time length E ₁ and keep the average luminance of the packet constant regardless of the signal to be transmitted, the L data unit is used as the data unit included in the packet according to the signal to be transmitted. It is recalled that it can be understood by using the R data part.

FIG. 363 is a diagram showing another example of the detailed configuration of the visible light signal in the present embodiment.

In the visible light signal shown in FIG. 363, unlike the examples shown in FIGS. 360 to 362, in order to shorten the effective time length, the L data unit corresponds to the sum of the variables y ₀ to y ₃ which are the signals to be transmitted. A packet including the R data unit and a packet including the R data unit are used properly.

That is, when the sum of the variables y ₀ to y ₃ is 7 or more, the transmitter 100 generates a packet containing only the L data unit of the two data units, as shown in FIG. 363 (a). .. Hereinafter, this packet is referred to as an L packet. Further, when the sum of the variables y ₀ to y ₃ is 6 or less, the transmitter 100 generates a packet containing only the R data unit out of the two data units, as shown in FIG. 363 (b). .. Hereinafter, this packet is referred to as an R packet.

As shown in FIG. 363 (a), the L-packet includes an average luminance adjusting unit, an L data unit, a preamble, and invalid data.

The average luminance adjusting unit of the L-packet shows the luminance value of Low by the time length _B'0 without indicating the luminance value of High. The time length B'0 is indicated by, for example, _{B'0 = 100 + W 1 × (y 0 + y 1 +} y ₂ + y _3-7 ).

The invalid data of the L-packet alternately shows the luminance values of High and Low along the time axis. That is, the invalid data shows the high luminance value only for the time length _A'0 , and shows the low luminance value _only for the next time length A'1. The time length _A'0 is indicated by _{A'0 = W 0-W 1 , for example} 80 μs, and the time length A'1 is, for example, 150 μs. Such invalid data indicates that the packet having the invalid data does not include the R data unit.

In such an L-packet, the total time length E'0 is _{E'0 = 5W 0 + 12W 1 + 4b +} 230 = 1540 μs regardless of the signal to be transmitted. Further, the effective time length _E'1 is a time length corresponding to the signal to be transmitted, and is in the range of 900 to 1290 μs. Further, while the total time length _E'0 is a constant 1540 μs, the total sum of the time lengths whose luminance value is High (that is, the total ON time) changes depending on the signal to be transmitted in the range of 490 to 670 μs. do. Therefore, even in this L-packet, the transmitter 100 sets the high luminance value from 100% to 73 according to the total ON time, that is, according to the time lengths D ₀ and D ₂ , as in the example shown in FIG. 362. .Variable in the range of 1%.

Similar to the example shown in FIG. 361, the R packet includes invalid data, a preamble, an R data unit, and an average luminance adjusting unit, as shown in FIG. 363 (b).

Here, in the R packet shown in FIG. 363 (b), in order to shorten the effective time length E ₁ , the time length B ₀ of the high luminance value in the average luminance adjusting unit is the shortest regardless of the signal to be transmitted. It is fixed at 100 μs. Further, the time length B ₁ of the brightness value of Low in the average brightness adjusting unit is, for example, B ₁ = W ₁ × (6- (y ₀ + y ₁ + y ₂ + y ₃ )) in order to make the total time length E ₁ constant. Further, also in the R packet shown in FIG. 363 (b), High according to the variables y ₀ and y ₂ included in the signal to be transmitted, that is, according to the time lengths D ₀ and D ₂ . The brightness value of is adjusted.

In such an R packet, the total time length E ₀ is E ₀ = 4W ₀ + 6W ₁ + 4b + 260 = 1280 μs regardless of the signal to be transmitted. Further, the effective time length E ₁ is a time length corresponding to the signal to be transmitted, and is in the range of 1100 to 1280 μs. Further, while the total time length E ₀ is a constant 1280 μs, the total sum of the time lengths whose luminance value is High (that is, the total ON time) changes depending on the signal to be transmitted in the range of 610 to 790 μs. .. Therefore, in this L-packet as well, the transmitter 100 sets the high luminance value to 80.3% according to the total ON time, that is, according to the time lengths D ₀ and D ₂ , as in the example shown in FIG. 362. Change in the range of ~ 62.1%.

As described above, in the visible light signal shown in FIG. 363, the maximum value of the effective time length in the packet can be shortened. Therefore, the transmitter ₁₀₀ can continuously transmit _one packet for the effective time length E1 or E'1 during the above-mentioned red projection period.

Here, in the example shown in FIG. 363, the transmitter 100 generates an L-packet when the total sum of the variables y ₀ to y ₃ is 7 or more, and when the total sum of the variables y ₀ to y ₃ is 6 or less. , Generates an R packet. In other words, since the sum of the variables y ₀ to y ₃ is an integer, the transmitter 100 generates an L-packet when the sum of the variables y ₀ to y ₃ is larger than 6, and the variables y ₀ to y ₃ are generated. When the sum of is 6 or less, an R packet is generated. That is, in this example, the threshold for switching the packet type is 6. However, the threshold for switching such packet types is not limited to 6, and may be any value of 3 to 10.

FIG. 364 is a diagram showing the relationship between the sum of the variables y ₀ to y ₃ and the total time length and the effective time length. The total time length shown in FIG. 364 is the larger of the total time length E _{0 of the R packet and the total time length E ′ 0 of} the L packet. _The effective time length shown in FIG. 364 is the larger of the maximum value of the effective time length E1 of the R packet and the maximum value of the effective time length _E'1 of the L packet. In the example shown in FIG. 364, the constants W ₀ , W ₁ , and b are W ₀ = 110 μs, W ₁ = 15 μs, and b = 100 μs, respectively.

As shown in FIG. 364, the total time length varies depending on the sum of the variables y ₀ to y ₃ , but the sum is the minimum at about 10. Further, as shown in FIG. 364, the effective time length changes according to the sum of the variables y ₀ to y ₃ , but the sum is the minimum at about 3.

Therefore, the threshold for switching the packet type may be set in the range of 3 to 10, depending on whether the total time length or the effective time length is shortened.

FIG. 365A is a flowchart showing a transmission method according to the present embodiment.

The transmission method in the present embodiment is a transmission method for transmitting a visible light signal by changing the brightness of a light emitter, and includes a determination step S571 and a transmission step S572. In the determination step S571, the transmitter 100 determines the pattern of luminance change by modulating the signal. In the transmission step S572, the transmitter 100 transmits a visible light signal by changing the brightness of the red color represented by the light source included in the light emitter according to a determined pattern. Here, the visible light signal includes data, a preamble, and a payload. In the data, a first luminance value and a second luminance value smaller than the first luminance value appear along the time axis, and at least one of the first luminance value and the second luminance value. The length of time that one continues is less than or equal to the first predetermined value. In the preamble, each of the first and second luminance values appears alternately along the time axis. In the payload, the first and second luminance values appear alternately along the time axis, and the duration of each of the first and second luminance values is greater than the first predetermined value, and It is determined according to the above-mentioned signal and a predetermined method.

For example, the data, preamble and payload are the invalid data, preamble, and L or R data sections shown in FIGS. 363 (a) and 363, respectively. Further, for example, the first predetermined value is 100 μs.

As a result, as shown in FIGS. 363 (a) and 363, the visible light signal has one payload (that is, an L data unit or an R data unit) having a waveform determined according to the signal to be modulated. Includes, does not include two payloads. Therefore, the visible light signal, that is, the packet of the visible light signal can be shortened. As a result, for example, even if the emission period of the red light represented by the light source included in the light emitting body is short, the packet of the visible light signal can be transmitted during the emission period.

Further, in the payload, the first luminance value of the first time length, the second luminance value of the second time length, the first luminance value of the third time length, and the second luminance value of the fourth time length. Each brightness value may appear in the order of the brightness value. In this case, in the transmission step S572, when the sum of the first time length and the third time length is smaller than the second predetermined value, the transmitter 100 has the first time length and the third time length. The value of the current flowing through the light source is made larger than the case where the sum of is larger than the second predetermined value. Here, the second predetermined value is larger than the first predetermined value. The second predetermined value is, for example, a value larger than 220 μs.

As a result, as shown in FIGS. 362 and 363, when the sum of the first time length and the third time length is small, the current value of the light source is increased, and the first time length and the third time length are increased. When the sum of the lengths is large, the current value of the light source is reduced. Therefore, the average luminance of a packet consisting of data, preamble and payload can be kept constant regardless of the signal.

Further, in the payload, the first luminance value of the first time length D ₀ , the second luminance value of the second time length D ₁ , the first luminance value of the third time length D ₂ , and the fourth luminance value. Each luminance value may appear in the order of the _second luminance value of the time length D3. In this case, when the sum of the four parameters y _k (k = 0, 1, 2, 3) obtained from the signal is equal to or less than the third predetermined value, the time lengths D ₀ to D ₃ of the first to fourth are set. Each is determined according to D _k = W ₀ + W ₁ × y _k (W ₀ and W ₁ are integers of 0 or more). For example, as shown in FIG. 363 (b), the third predetermined value is 3.

As a result, as shown in FIG. 363 (b), it is possible to generate a payload having a short waveform according to the signal while setting each of the first to fourth time lengths D ₀ to D ₃ to W ₀ or more.

Further, when the sum of the four parameters y _k (k = 0, 1, 2, 3) is equal to or less than the third predetermined value, in the transmission step S572, the data, the preamble, and the payload are replaced with the data, the preamble, and the payload. It may be sent in order. In the case of the example shown in FIG. 363 (b), the payload is the R data unit.

As a result, as shown in FIG. 363 (b), the receiver 200 that receives the packet by the data that the packet of the visible light signal containing the data (that is, invalid data) does not include the L data unit. Can be informed.

Further, when the sum of the four parameters y _k (k = 0, 1, 2, 3) is larger than the third predetermined value, each of the first to fourth time lengths D ₀ to D ₃ is D ₀ . = W ₀ + W ₁ x (A-y ₀ ), D ₁ = W ₀ + W ₁ x (By ₁ ), D ₂ = W ₀ + W ₁ x (A-y ₂ ), and D ₃ = W ₀ + W It may be determined according to ₁ × (By ₃ ) (A and B are integers of 0 or more, respectively).

As a result, as shown in FIG. 363 (a), each of the first to fourth time lengths D ₀ to D ₃ (that is, the first to fourth time lengths D' ₀ to D' ₃ ) is W ₀ or more. However, even if the sum is large, it is possible to generate a payload having a short waveform depending on the signal.

Further, when the sum of the four parameters y _k (k = 0, 1, 2, 3) is larger than the third predetermined value, in the transmission step S572, the data, the preamble and the payload are transferred to the payload, the preamble and the data. It may be sent in order. In the case of the example shown in FIG. 363 (a), the payload is the L data unit.

As a result, as shown in FIG. 363 (a), the receiving device that receives the packet by the data indicates that the packet of the visible light signal containing the data (that is, invalid data) does not include the R data unit. I can tell you.

Further, the light emitter has a plurality of light sources including a red light source, a blue light source, and a green light source, and in the transmission step S572, a visible light signal is transmitted by using only the red light source among the plurality of light sources. You may send it.

As a result, the light emitter can display an image using a red light source, a blue light source, and a green light source, and can transmit a visible light signal having a wavelength that is easy to receive to the receiver 200.

The light emitter may be, for example, a DLP projector. As described above, the DLP projector may have a plurality of light sources including a red light source, a blue light source, and a green light source, but may have only one light source. That is, the DLP projector may include one light source, a DMD (Digital Micromirror Device), and a color wheel arranged between the light source and the DMD. In this case, the DLP projector is visible during the period when the red light is output from the red light, the blue light, and the green light, which are output from the light source to the DMD via the color wheel in a time-divided manner. Send a packet of optical signals.

FIG. 365B is a block diagram showing the configuration of the transmitter 100 according to the present embodiment.

The transmitter 100 in the present embodiment includes a transmitter, a determination unit 571, and a transmission unit 572 that transmit a visible light signal according to a change in the brightness of the light emitter. The determination unit 571 determines the pattern of luminance change by modulating the signal. The transmission unit 572 transmits a visible light signal by changing the brightness of red represented by the light source included in the light emitter according to a determined pattern. Here, the visible light signal includes data, a preamble, and a payload. In the data, a first luminance value and a second luminance value smaller than the first luminance value appear along the time axis, and at least one of the first luminance value and the second luminance value. The length of time that one continues is less than or equal to the first predetermined value. In the preamble, each of the first and second luminance values appears alternately along the time axis. In the payload, the first and second luminance values appear alternately along the time axis, and the duration of each of the first and second luminance values is greater than the first predetermined value, and It is determined according to the above-mentioned signal and a predetermined method.

With such a transmitter 100, the transmission method of the flowchart shown in FIG. 365A is realized.

The transmission method of the present invention can be used, for example, in a transmission device that transmits a visible light signal from a display or lighting, and in particular, can be used in a transmission device that transmits a visible light signal from, for example, a spotlight.

100 Transmitter 551 Receptionist 552 Transmitter

Claims (15)

It is a transmission method that transmits a signal by changing the brightness of a light source.
A reception step that accepts the dimming intensity designated for the light source as the designated dimming intensity, and
When the designated luminous intensity is equal to or less than the first value,
While emitting the light source at the designated luminous intensity, the signal encoded in the first mode is transmitted by the change in brightness.
When the designated luminous intensity is larger than the first value,
The present invention includes a transmission step of transmitting the signal encoded in the second mode by a change in brightness while causing the light source to emit light at the specified dimming intensity.
The value of the peak current of the light source for transmitting the signal encoded in the second mode by the luminance change when the designated luminous intensity is larger than the first value and is equal to or less than the second value. teeth,
When the designated luminous intensity is the first value, it is smaller than the peak current value of the light source for transmitting the signal encoded in the first mode by the luminance change.
Sending method. When the designated luminous intensity is smaller than the third value,
While causing the light source to emit light at the designated luminous intensity, the signal encoded in the first mode is transmitted by a change in brightness, and at the same time.
The value of the peak current is maintained at a constant value with respect to the change in the designated dimming intensity, and the value is maintained.
The third value is smaller than the first value.
The transmission method according to claim 1. When the designated luminous intensity is smaller than the third value,
By lengthening the time for turning off the light source as the designated dimming intensity becomes smaller, the light source is made to emit light at the designated dimming intensity that becomes smaller, and the value of the peak current is maintained at a constant value. ,
The transmission method according to claim 2. When the designated luminous intensity is smaller than the fourth value,
While causing the light source to emit light at the designated luminous intensity, the signal encoded in the first mode is transmitted by a change in brightness, and at the same time.
By reducing the value of the peak current as the designated dimming intensity becomes smaller, the light source is made to emit light at the designated dimming intensity that becomes smaller.
The fourth value is smaller than the second value.
The transmission method according to claim 1. The time for turning off the light source is determined so that one cycle obtained by adding the time for transmitting the signal due to the change in luminance and the time for turning off the light source does not exceed 10 milliseconds.
The transmission method according to claim 3. The value of the peak current of the light source when the designated luminous intensity is the first value, and
It is the same as the value of the peak current of the light source when the designated luminous intensity is the maximum value.
The transmission method according to claim 1. The transmission method according to claim 1, wherein the duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode. When the peak current value of the light source exceeds the fifth value,
Stops the transmission of the signal due to the change in the brightness of the light source.
The transmission method according to claim 1. Measure the usage time of the light source and
If the usage time is longer than the specified time,
The signal is transmitted by a change in brightness using the value of the parameter for causing the light source to emit light at a dimming intensity larger than the designated dimming intensity.
The transmission method according to claim 1. Measure the usage time of the light source and
If the usage time is longer than the specified time,
Than when the usage time is less than the predetermined time
Increasing the pulse width of the current of the light source,
The transmission method according to claim 1. It is a transmission method that transmits a signal by changing the brightness of a light source.
A reception step that accepts the dimming intensity designated for the light source as the designated dimming intensity, and
The present invention includes a transmission step of transmitting the signal encoded in the first mode or the second mode by a change in brightness while causing the light source to emit light at the specified dimming intensity.
The duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode.
In the transmission step,
When the designated dimming intensity is changed from a small value to a large value, when the designated dimming intensity is the first value, the mode used for coding the signal is changed from the first mode to the second mode. Switch to the mode,
When the designated dimming intensity is changed from a large value to a small value, when the designated dimming intensity is a second value, the mode used for coding the signal is changed from the second mode to the first mode. Switch to the mode,
The second value is smaller than the first value.
Sending method. In the transmission step,
When the switching from the first mode to the second mode is performed, the peak current of the light source for transmitting the encoded signal by the luminance change is obtained from the first current value. Change to a second current value that is smaller than the current value of 1.
When switching from the second mode to the first mode, the peak current is changed from the third current value to a fourth current value larger than the third current value.
The transmission method according to claim 11, wherein the first current value is larger than the fourth current value, and the second current value is larger than the third current value. A program that causes a computer to execute the transmission method according to claim 1 or 11. A transmitter that transmits a signal by changing the brightness of a light source.
A reception unit that accepts the dimming intensity specified for the light source as the designated dimming intensity, and
When the designated luminous intensity is equal to or less than the first value,
While emitting the light source at the designated luminous intensity, the signal encoded in the first mode is transmitted by the change in brightness.
When the dimming intensity is larger than the first value,
It is provided with a transmission unit that transmits the signal encoded in the second mode by changing the luminance while causing the light source to emit light at the designated luminous intensity.
The value of the peak current of the light source for transmitting the signal encoded in the second mode by the luminance change when the designated luminous intensity is larger than the first value and is equal to or less than the second value. teeth,
When the designated luminous intensity is the first value, it is smaller than the peak current value of the light source for transmitting the signal encoded in the first mode by the luminance change.
Transmitter. A transmitter that transmits a signal by changing the brightness of a light source.
A reception unit that accepts the dimming intensity specified for the light source as the designated dimming intensity, and
It is provided with a transmission unit that transmits the signal encoded in the first mode or the second mode by changing the luminance while emitting the light source at the designated luminous intensity.
The duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode.
The transmitter is
When the designated dimming intensity is changed from a small value to a large value, when the designated dimming intensity is the first value, the mode used for coding the signal is changed from the first mode to the second mode. Switch to the mode,
When the designated dimming intensity is changed from a large value to a small value, when the designated dimming intensity is a second value, the mode used for coding the signal is changed from the second mode to the first mode. Switch to the mode,
The second value is smaller than the first value.
Transmitter.

Images