WO2018088380A1 - Transmission method, transmission device, and program - Google Patents

Abstract

This transmission method includes a step (S551) for receiving, as a designated light adjustment degree, a light adjustment degree designated with respect to a light source, and a step (S552) in which, when the designated light adjustment degree is no greater than a first value, a light source emits light at the designated light adjustment degree while a signal encoded in a first mode is transmitted via changes in luminance, and, when the designated light adjustment degree is greater than the first value, the light source emits light at the designated light adjustment degree while a signal encoded in a second mode is transmitted via changes in luminance. The peak current value of the light source when the designated light adjustment degree is greater than the first value but no more than a second value is smaller than the peak current value of the light source when the designated light adjustment value is the first value.

Description

Transmission method, transmission device, and program

The present invention relates to a visible light signal transmission method, a transmission device, a program, and the like.

In recent home networks, in addition to cooperation of AV home appliances by IP (Internet Protocol) connection in Ethernet (registered trademark) and wireless LAN (Local Area Network), management of power consumption corresponding to environmental problems, and from outside the home Introduction of a home appliance linkage function in which various home appliances are connected to a network by a home energy management system (HEMS) having a function of turning on / off the power of the home appliance. However, there are home appliances that do not have sufficient computing power to have a communication function, and home appliances that are difficult to install a communication function in terms of cost.

In order to solve such a problem, in Patent Literature 1, in an optical space transmission device that transmits information to free space using light, limited transmission is performed by performing communication using a plurality of monochromatic light sources of illumination light. A technology for efficiently realizing communication between devices is described in the apparatus.

JP 2002-290335 A

However, the conventional method is limited to a case where a device to be applied has a three-color light source such as illumination.

The present invention solves such a problem and provides a transmission method and the like that enables communication between various devices including devices other than lighting having a three-color light source.

A transmission method according to an aspect of the present invention is a transmission method for transmitting a signal according to a change in luminance of a light source, the step of accepting a dimming degree designated for the light source as a designated dimming degree, and the designated dimming degree Is less than or equal to the first value, the signal encoded in the first mode is transmitted by a luminance change while causing the light source to emit light at the specified dimming level, and the specified dimming level is the first dimming level. A transmission step of transmitting the signal encoded in the second mode by a luminance change while causing the light source to emit light at the specified dimming level, and the specified dimming level is The value of the peak current of the light source for transmitting the signal encoded in the second mode by a change in luminance when it is greater than the first value and less than or equal to the second value is the specified dimming degree Is the first If it is smaller than the value of the peak current of the light source for transmitting the first said signal encoded in the mode of the luminance change.

Note that these comprehensive or specific modes may be realized by a system, a method, an integrated circuit, a computer program, or a recording medium such as a computer-readable CD-ROM, and the system, method, integrated circuit, and computer program. And any combination of recording media. Moreover, the computer program which performs the method concerning one Embodiment is preserve | saved at the recording medium of the server, and may be implement | achieved in the aspect delivered to a terminal from a server according to the request | requirement of a terminal.

According to the present invention, it is possible to realize a transmission method that enables communication between devices including a device other than a lighting device having a three-color light source.

FIG. 1 is a diagram illustrating an example of an observation method of luminance of a light emitting unit in the first embodiment. FIG. 2 is a diagram illustrating an example of a method of observing the luminance of the light emitting unit in the first embodiment. FIG. 3 is a diagram illustrating an example of a method of observing the luminance of the light emitting unit in the first embodiment. FIG. 4 is a diagram illustrating an example of a method of observing the luminance of the light emitting unit in the first embodiment. FIG. 5A is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 5B is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 5C is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 5D is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 5E is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 5F is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 5G is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 5H is a diagram illustrating an example of an observation method of luminance of a light emitting unit in Embodiment 1. FIG. 6A is a flowchart of the information communication method in Embodiment 1. FIG. 6B is a block diagram of the information communication apparatus according to Embodiment 1. FIG. 7 is a diagram illustrating an example of a photographing operation of the receiver in the second embodiment. FIG. 8 is a diagram illustrating another example of the photographing operation of the receiver in the second embodiment. FIG. 9 is a diagram illustrating another example of the photographing operation of the receiver in the second embodiment. FIG. 10 is a diagram illustrating an example of display operation of the receiver in Embodiment 2. FIG. 11 is a diagram illustrating an example of display operation of the receiver in Embodiment 2. FIG. 12 is a diagram illustrating an example of operation of a receiver in Embodiment 2. FIG. 13 is a diagram illustrating another example of operation of a receiver in Embodiment 2. FIG. 14 is a diagram illustrating another example of operation of a receiver in Embodiment 2. FIG. 15 is a diagram illustrating another example of operation of a receiver in Embodiment 2. FIG. 16 is a diagram illustrating another example of operation of a receiver in Embodiment 2. FIG. 17 is a diagram illustrating another example of operation of a receiver in Embodiment 2. FIG. 18 is a diagram illustrating an example of operations of the receiver, the transmitter, and the server in the second embodiment. FIG. 19 is a diagram illustrating another example of operation of a receiver in Embodiment 2. FIG. 20 is a diagram illustrating another example of operation of a receiver in Embodiment 2. FIG. 21 is a diagram illustrating another example of operation of a receiver in Embodiment 2. FIG. 22 is a diagram illustrating an example of operation of a transmitter in Embodiment 2. FIG. 23 is a diagram illustrating another example of operation of a transmitter in Embodiment 2. FIG. 24 is a diagram illustrating an example of application of a receiver in Embodiment 2. FIG. 25 is a diagram illustrating another example of operation of a receiver in Embodiment 2. FIG. 26 is a diagram illustrating an example of processing operations of the receiver, the transmitter, and the server in Embodiment 3. FIG. 27 is a diagram illustrating an example of operations of a transmitter and a receiver in Embodiment 3. FIG. 28 is a diagram illustrating an example of operations of a transmitter, a receiver, and a server in Embodiment 3. FIG. 29 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 3. FIG. 30 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 4. FIG. 31 is a diagram illustrating an example of operations of a transmitter and a receiver in Embodiment 4. FIG. 32 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 4. FIG. 33 is a diagram illustrating an example of operations of a transmitter and a receiver in Embodiment 4. FIG. 34 is a diagram illustrating an example of operations of a transmitter and a receiver in Embodiment 4. FIG. 35 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 4. FIG. 36 is a diagram illustrating an example of operations of a transmitter and a receiver in Embodiment 4. FIG. 37 is a diagram for describing notification of visible light communication to a human in the fifth embodiment. FIG. 38 is a diagram for explaining an application example to the route guidance in the fifth embodiment. FIG. 39 is a diagram for explaining an application example to use log accumulation and analysis in the fifth embodiment. FIG. 40 is a diagram for explaining an application example to screen sharing in the fifth embodiment. FIG. 41 is a diagram illustrating an application example of the information communication method according to the fifth embodiment. FIG. 42 is a diagram illustrating an example of application of the transmitter and the receiver in the sixth embodiment. FIG. 43 is a diagram illustrating an example of application of the transmitter and the receiver in the sixth embodiment. FIG. 44 is a diagram illustrating an example of a receiver in Embodiment 7. FIG. 45 is a diagram illustrating an example of a reception system in the seventh embodiment. FIG. 46 is a diagram illustrating an example of a signal transmission / reception system according to the seventh embodiment. FIG. 47 is a flowchart showing a reception method in which interference is eliminated in the seventh embodiment. FIG. 48 is a flowchart showing a method for estimating the orientation of a transmitter in the seventh embodiment. FIG. 49 is a flowchart showing a reception start method according to the seventh embodiment. FIG. 50 is a flowchart showing an ID generation method using information of another medium together in the seventh embodiment. FIG. 51 is a flowchart showing a reception method selection method based on frequency separation in the seventh embodiment. FIG. 52 is a flowchart showing a signal reception method when the exposure time is long in the seventh embodiment. FIG. 53 is a diagram illustrating an example of a transmitter dimming (adjusting brightness) method in Embodiment 7. FIG. 54 is a diagram illustrating an example of a method for configuring a dimming function of a transmitter in the seventh embodiment. FIG. 55 is a diagram for explaining the EX zoom. FIG. 56 is a diagram illustrating an example of a signal reception method in Embodiment 9. FIG. 57 is a diagram illustrating an example of a signal reception method in Embodiment 9. FIG. 58 is a diagram illustrating an example of a signal reception method in Embodiment 9. FIG. 59 is a diagram illustrating an example of a screen display method of a receiver in Embodiment 9. FIG. 60 is a diagram illustrating an example of a signal reception method in Embodiment 9. FIG. 61 is a diagram illustrating an example of a signal reception method according to the ninth embodiment. FIG. 62 is a flowchart illustrating an example of a signal reception method in the ninth embodiment. FIG. 63 is a diagram illustrating an example of a signal reception method in the ninth embodiment. FIG. 64 is a flowchart showing processing of the reception program in the ninth embodiment. FIG. 65 is a block diagram of a receiving apparatus according to the ninth embodiment. FIG. 66 is a diagram illustrating an example of display on the receiver when a visible light signal is received. FIG. 67 is a diagram illustrating an example of display on the receiver when a visible light signal is received. FIG. 68 is a diagram illustrating an example of the display of the acquired data image. FIG. 69 is a diagram illustrating an example of an operation when saving or discarding acquired data. FIG. 70 is a diagram illustrating a display example when browsing acquired data. 71 is a diagram illustrating an example of a transmitter in Embodiment 9. FIG. FIG. 72 is a diagram illustrating an example of a reception method in Embodiment 9. FIG. 73 is a flowchart illustrating an example of a reception method in Embodiment 10. FIG. 74 is a flowchart illustrating an example of a reception method in Embodiment 10. FIG. 75 is a flowchart illustrating an example of a reception method in Embodiment 10. FIG. 76 is a diagram for describing a reception method in which the receiver according to Embodiment 10 uses an exposure time longer than the period of the modulation frequency (modulation period). FIG. 77 is a diagram for describing a reception method in which the receiver according to Embodiment 10 uses an exposure time longer than the period of the modulation frequency (modulation period). FIG. 78 is a diagram showing an efficient division number with respect to the size of transmission data in the tenth embodiment. FIG. 79A is a diagram illustrating an example of a setting method in Embodiment 10. FIG. 79B is a diagram illustrating another example of the setting method according to the tenth embodiment. FIG. 80 is a flowchart showing processing of the information processing program in the tenth embodiment. FIG. 81 is a diagram for describing an application example of the transmission and reception system in the tenth embodiment. FIG. 82 is a flowchart showing processing operations of the transmission / reception system in the tenth embodiment. FIG. 83 is a diagram for describing an example of application of the transmission and reception system in the tenth embodiment. FIG. 84 is a flowchart showing processing operations of the transmission / reception system in the tenth embodiment. FIG. 85 is a diagram for describing an application example of the transmission and reception system in the tenth embodiment. FIG. 86 is a flowchart showing processing operations of the transmission / reception system in the tenth embodiment. FIG. 87 is a diagram for describing an example of application of a transmitter in Embodiment 10. FIG. 88 is a diagram for describing an application example of the transmission and reception system in the eleventh embodiment. FIG. 89 is a diagram for explaining an application example of the transmission and reception system in the eleventh embodiment. 90 is a diagram for describing an example of application of a transmission and reception system in Embodiment 11. FIG. FIG. 91 is a diagram for describing an application example of the transmission and reception system in the eleventh embodiment. FIG. 92 is a diagram for describing an application example of the transmission and reception system in the eleventh embodiment. FIG. 93 is a diagram for describing an application example of the transmission / reception system in Embodiment 11. In FIG. FIG. 94 is a diagram for describing an application example of the transmission and reception system in the eleventh embodiment. FIG. 95 is a diagram for describing an application example of the transmission / reception system in Embodiment 11. In FIG. FIG. 96 is a diagram for describing an application example of the transmission / reception system in Embodiment 11. In FIG. FIG. 97 is a diagram for describing an application example of the transmission / reception system in Embodiment 11. In FIG. FIG. 98 is a diagram for describing an application example of the transmission / reception system in Embodiment 11. In FIG. 99 is a diagram for describing an application example of the transmission and reception system in Embodiment 11. FIG. FIG. 100 is a diagram for describing an example of application of the transmission and reception system in Embodiment 11. FIG. 101 is a diagram for explaining an application example of the transmission and reception system in the eleventh embodiment. FIG. 102 is a diagram for describing operation of a receiver in Embodiment 12. 103A is a diagram for describing another operation of the receiver in Embodiment 12. FIG. FIG. 103B is a diagram illustrating an example of an indicator displayed by the output unit 1215 in the twelfth embodiment. FIG. 103C is a diagram illustrating a display example of AR in the twelfth embodiment. 104A is a diagram for describing an example of a transmitter in Embodiment 12. FIG. FIG. 104B is a diagram for describing another example of the transmitter in Embodiment 12. 105A is a diagram for describing an example of synchronous transmission by a plurality of transmitters in Embodiment 12. FIG. 105B is a diagram for describing another example of synchronous transmission by a plurality of transmitters in Embodiment 12. FIG. FIG. 106 is a diagram for describing another example of synchronous transmission by a plurality of transmitters in Embodiment 12. FIG. 107 is a diagram for describing signal processing by a transmitter in Embodiment 12. FIG. 108 is a flowchart illustrating an example of a reception method in Embodiment 12. FIG. 109 is an explanatory diagram for describing an example of a reception method in Embodiment 12. FIG. 110 is a flowchart illustrating another example of a reception method in Embodiment 12. 111 is a diagram illustrating an example of a transmission signal in Embodiment 13. FIG. 112 is a diagram illustrating another example of a transmission signal in Embodiment 13. FIG. FIG. 113 is a diagram illustrating another example of a transmission signal in Embodiment 13. 114A is a diagram for illustrating a transmitter in Embodiment 14. FIG. FIG. 114B is a diagram showing each luminance change of RGB in the fourteenth embodiment. FIG. 115 is a diagram illustrating afterglow characteristics of the green fluorescent component and the red fluorescent component in the fourteenth embodiment. FIG. 116 is a diagram for describing a problem newly generated in order to suppress occurrence of a barcode reading error in the fourteenth embodiment. 117 is a diagram for explaining downsampling performed by a receiver in Embodiment 14. FIG. FIG. 118 is a flowchart illustrating a processing operation of the receiver in Embodiment 14. FIG. 119 is a diagram illustrating processing operations of the reception device (imaging device) in Embodiment 15. FIG. 120 is a diagram illustrating processing operation of a reception device (imaging device) in Embodiment 15. FIG. 121 is a diagram illustrating processing operation of a reception device (imaging device) in Embodiment 15. FIG. 122 is a diagram illustrating processing operation of the reception device (imaging device) in Embodiment 15. FIG. 123 is a diagram illustrating an example of an application according to the sixteenth embodiment. FIG. 124 is a diagram illustrating an example of an application according to the sixteenth embodiment. FIG. 125 is a diagram illustrating an example of the transmission signal and an example of the audio synchronization method in Embodiment 16. 126 is a diagram illustrating an example of a transmission signal in Embodiment 16. FIG. FIG. 127 is a diagram illustrating an example of processing flow of a receiver in Embodiment 16. FIG. 128 is a diagram illustrating an example of a user interface of the receiver in Embodiment 16. 129 is a diagram illustrating an example of a process flow of a receiver in Embodiment 16. FIG. FIG. 130 is a diagram illustrating another example of processing flow of a receiver in Embodiment 16. FIG. 131A is a diagram for explaining a specific method of synchronized playback in the sixteenth embodiment. FIG. 131B is a block diagram showing a configuration of a playback device (receiver) that performs synchronized playback in the sixteenth embodiment. FIG. 131C is a flowchart illustrating a processing operation of a playback device (receiver) that performs synchronized playback in the sixteenth embodiment. FIG. 132 is a diagram for describing preparation for synchronized playback in the sixteenth embodiment. 133 is a diagram illustrating an example of application of a receiver in Embodiment 16. FIG. FIG. 134A is a front view of a receiver held by a holder in Embodiment 16. FIG. FIG. 134B is a rear view of a receiver held by a holder in Embodiment 16. FIG. 135 is a diagram for describing a use case of a receiver held by a holder in Embodiment 16. 136 is a flowchart illustrating processing operation of a receiver held by a holder in Embodiment 16. FIG. FIG. 137 is a diagram illustrating an example of an image displayed by the receiver in Embodiment 16. In FIG. FIG. 138 is a diagram showing another example of the holder according to the sixteenth embodiment. 139A is a diagram illustrating an example of a visible light signal in Embodiment 17. FIG. FIG. 139B is a diagram illustrating an example of a visible light signal in Embodiment 17. 139C is a diagram illustrating an example of a visible light signal in Embodiment 17. FIG. 139D is a diagram illustrating an example of a visible light signal in Embodiment 17. FIG. FIG. 140 is a diagram illustrating a configuration of a visible light signal according to the seventeenth embodiment. FIG. 141 is a diagram illustrating an example of bright line images obtained by imaging of the receiver in Embodiment 17. FIG. 142 is a diagram illustrating another example of bright line images obtained by imaging by the receiver in Embodiment 17. FIG. 143 is a diagram illustrating another example of the bright line image obtained by imaging by the receiver in Embodiment 17. 144 is a diagram for describing adaptation of the receiver in Embodiment 17 to a camera system that performs HDR synthesis. FIG. FIG. 145 is a diagram for explaining the processing operation of the visible light communication system in the seventeenth embodiment. 146A is a diagram illustrating an example of vehicle-to-vehicle communication using visible light in Embodiment 17. FIG. 146B is a diagram illustrating another example of vehicle-to-vehicle communication using visible light in Embodiment 17. FIG. 147 is a diagram illustrating an example of a method for determining the positions of a plurality of LEDs in Embodiment 17. FIG. FIG. 148 is a diagram illustrating an example of bright line images obtained by capturing an image of the vehicle in the seventeenth embodiment. FIG. 149 is a diagram illustrating an example of application of the receiver and the transmitter in Embodiment 17. FIG. 149 is a view of the automobile from the back. FIG. 150 is a flowchart illustrating an example of processing operations of a receiver and a transmitter in Embodiment 17. FIG. 151 is a diagram illustrating an example of application of the receiver and the transmitter in Embodiment 17. FIG. 152 is a flowchart illustrating an example of processing operations of the receiver 7007a and the transmitter 7007b in Embodiment 17. FIG. 153 is a diagram illustrating a configuration of a visible light communication system applied to the inside of a train in Embodiment 17. FIG. 154 is a diagram illustrating a configuration of a visible light communication system applied to a facility such as an amusement park in Embodiment 17. FIG. 155 is a diagram illustrating an example of a visible light communication system including a playground device and a smartphone according to Embodiment 17. 156 is a diagram illustrating an example of a transmission signal in Embodiment 18. FIG. 157 is a diagram illustrating an example of a transmission signal in Embodiment 18. FIG. 158 is a diagram illustrating an example of a transmission signal in Embodiment 19. FIG. 159 is a diagram illustrating an example of a transmission signal in Embodiment 19. FIG. 160 is a diagram illustrating an example of a transmission signal in Embodiment 19. FIG. 161 is a diagram illustrating an example of a transmission signal in Embodiment 19. FIG. 162 is a diagram illustrating an example of a transmission signal in Embodiment 19. FIG. 163 is a diagram illustrating an example of a transmission signal in Embodiment 19. FIG. 164 is a diagram illustrating an example of a transmission and reception system in Embodiment 19. FIG. FIG. 165 is a flowchart illustrating an example of processing of the transmission / reception system in the nineteenth embodiment. FIG. 166 is a flowchart showing the operation of the server in the nineteenth embodiment. FIG. 167 is a flowchart illustrating an example of operation of a receiver in Embodiment 19. FIG. 168 is a flowchart illustrating a method of calculating the progress status in the simple mode according to the nineteenth embodiment. FIG. 169 is a flowchart illustrating a method for calculating the progress in the maximum likelihood estimation mode according to the nineteenth embodiment. FIG. 170 is a flowchart showing a display method in which the progress status does not decrease in the nineteenth embodiment. FIG. 171 is a flowchart illustrating a progress status display method when there are a plurality of packet lengths according to the nineteenth embodiment. 172 is a diagram illustrating an example of an operation state of a receiver in Embodiment 19. FIG. 173 is a diagram illustrating an example of a transmission signal in Embodiment 19. FIG. 174 is a diagram illustrating an example of a transmission signal in Embodiment 19. FIG. 175 is a diagram illustrating an example of a transmission signal in Embodiment 19. FIG. 176 is a block diagram illustrating an example of a transmitter in Embodiment 19. FIG. FIG. 177 is a timing chart when the LED display in Embodiment 19 is driven with the optical ID modulation signal of the present invention. FIG. 178 is a timing chart when the LED display in Embodiment 19 is driven with the optical ID modulation signal of the present invention. FIG. 179 is a timing chart when the LED display in Embodiment 19 is driven with the optical ID modulation signal of the present invention. FIG. 180A is a flowchart illustrating a transmission method according to one embodiment of the present invention. FIG. 180B is a block diagram illustrating a functional configuration of the transmission device according to one embodiment of the present invention. FIG. 181 is a diagram illustrating an example of a transmission signal in Embodiment 19. In FIG. 182 is a diagram illustrating an example of a transmission signal in Embodiment 19. FIG. 183 is a diagram illustrating an example of a transmission signal in Embodiment 19. FIG. 184 is a diagram illustrating an example of a transmission signal in Embodiment 19. FIG. 185 is a diagram illustrating an example of a transmission signal in Embodiment 19. FIG. 186 is a diagram illustrating an example of a transmission signal in Embodiment 19. FIG. 187 is a diagram illustrating an example of a structure of a visible light signal in Embodiment 20. FIG. 188 is a diagram illustrating an example of a detailed configuration of a visible light signal in Embodiment 20. FIG. FIG. 189A is a diagram illustrating another example of a visible light signal in Embodiment 20. FIG. 189B is a diagram illustrating another example of a visible light signal in Embodiment 20. FIG. 189C is a diagram illustrating the signal length of a visible light signal in Embodiment 20. FIG. FIG. 190 is a diagram illustrating a comparison result of luminance values between the visible light signal and the standard IEC visible light signal according to the twentieth embodiment. FIG. 191 is a diagram illustrating a comparison result of the number of received packets and the reliability with respect to the angle of view between the visible light signal and the standard IEC visible light signal according to the twentieth embodiment. FIG. 192 is a diagram illustrating comparison results of the number of received packets and reliability with respect to noise between the visible light signal and the standard IEC visible light signal according to the twentieth embodiment. FIG. 193 is a diagram illustrating a comparison result of the number of received packets and the reliability with respect to the reception-side clock error between the visible light signal and the standard IEC visible light signal according to the twentieth embodiment. FIG. 194 is a diagram illustrating a structure of a transmission target signal in the twentieth embodiment. 195A is a diagram illustrating a visible light signal receiving method in Embodiment 20. FIG. FIG. 195B is a diagram illustrating rearrangement of visible light signals in the twentieth embodiment. 196 is a diagram illustrating another example of a visible light signal in Embodiment 20. FIG. FIG. 197 is a diagram illustrating another example of a detailed configuration of a visible light signal in the twentieth embodiment. FIG. 198 is a diagram illustrating another example of a detailed configuration of a visible light signal in the twentieth embodiment. FIG. 199 is a diagram illustrating another example of a detailed configuration of a visible light signal in Embodiment 20. In FIG. 200 is a diagram illustrating another example of a detailed configuration of a visible light signal in Embodiment 20. FIG. FIG. 201 is a diagram illustrating another example of a detailed configuration of a visible light signal according to the twentieth embodiment. FIG. 202 is a diagram illustrating another example of a detailed configuration of a visible light signal in Embodiment 20. FIG. 203 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. 197. FIG. 204 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. 197. FIG. 205 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. 197. FIG. 206 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. FIG. 207 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. 197. FIG. 208 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. 197. FIG. 209 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. FIG. 210 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. FIG. 211 is a diagram for explaining a method of determining the values of x1 to x4 in FIG. 197. FIG. 212 is a diagram illustrating an example of a detailed configuration of a visible light signal according to the first modification of the twentieth embodiment. 213 is a diagram illustrating another example of a visible light signal according to Modification 1 of Embodiment 20. FIG. 214 is a diagram showing still another example of a visible light signal according to Modification 1 of Embodiment 20. FIG. 215 is a diagram illustrating an example of packet modulation according to Modification 1 of Embodiment 20. FIG. FIG. 216 is a diagram illustrating processing for dividing the original data into one according to the first modification of the twentieth embodiment. FIG. 217 is a diagram illustrating a process of dividing the original data into two according to the first modification of the twentieth embodiment. FIG. 218 is a diagram illustrating processing of dividing original data into three according to Modification 1 of Embodiment 20. FIG. 219 is a diagram illustrating another example of the process of dividing the original data into three according to the first modification of the twentieth embodiment. FIG. 220 is a diagram illustrating another example of the process of dividing the original data into three according to the first modification of the twentieth embodiment. FIG. 221 is a diagram illustrating a process of dividing the original data into four according to the first modification of the twentieth embodiment. FIG. 222 is a diagram showing processing for dividing original data into five parts according to Modification 1 of Embodiment 20. 223 is a diagram illustrating processing of dividing original data into 6, 7, or 8 portions according to Modification Example 1 of Embodiment 20. FIG. FIG. 224 is a diagram illustrating another example of the process of dividing the original data into 6, 7 or 8 according to the first modification of the twentieth embodiment. FIG. 225 is a diagram illustrating a process of dividing the original data into nine according to the first modification of the twentieth embodiment. FIG. 226 is a diagram illustrating processing of dividing original data into any number from 10 to 16 according to Modification 1 of Embodiment 20. FIG. 227 is a diagram illustrating an example of a relationship among the number of original data divisions, a data size, and an error correction code according to the first modification of the twentieth embodiment. FIG. 228 is a diagram illustrating another example of the relationship between the number of original data divisions, the data size, and the error correction code according to the first modification of the twentieth embodiment. 229 is a diagram illustrating still another example of the relationship among the number of original data divisions, the data size, and the error correction code according to Modification 1 of Embodiment 20. FIG. 230A is a flowchart illustrating a visible light signal generation method according to Embodiment 20. FIG. FIG. 230B is a block diagram illustrating a configuration of the signal generation device according to Embodiment 20. FIG. 231 is a diagram illustrating a method of receiving a high-frequency visible light signal in Embodiment 21. 232A is a diagram illustrating another method of receiving a high-frequency visible light signal in Embodiment 21. FIG. FIG. 232B is a diagram illustrating another method of receiving a high-frequency visible light signal in Embodiment 21. 233 is a diagram illustrating a method of outputting a high-frequency signal in Embodiment 21. FIG. FIG. 234 is a diagram for describing the autonomous flight apparatus according to the twenty-second embodiment. FIG. 235 is a diagram illustrating an example in which the receiver in Embodiment 23 displays an AR image. FIG. 236 is a diagram illustrating an example of a display system in Embodiment 23. FIG. 237 is a diagram illustrating another example of the display system in Embodiment 23. In FIG. FIG. 238 is a diagram illustrating another example of the display system in Embodiment 23. In FIG. FIG. 239 is a flowchart illustrating an example of process operations of a receiver in Embodiment 23. FIG. 240 is a diagram illustrating another example in which the receiver in Embodiment 23 displays an AR image. FIG. 241 is a diagram illustrating another example in which the receiver in Embodiment 23 displays an AR image. FIG. 242 is a diagram illustrating another example in which the receiver in Embodiment 23 displays an AR image. FIG. 243 is a diagram illustrating another example in which the receiver in Embodiment 23 displays an AR image. FIG. 244 is a diagram illustrating another example in which the receiver in Embodiment 23 displays an AR image. FIG. 245 is a diagram illustrating another example in which the receiver in Embodiment 23 displays an AR image. FIG. 246 is a flowchart illustrating another example of processing operations of a receiver in Embodiment 23. FIG. 247 is a diagram illustrating another example in which the receiver in Embodiment 23 displays an AR image. 248 is a diagram illustrating a captured display image Ppre and a decoding image Pdec acquired by capturing by the receiver in Embodiment 23. FIG. FIG. 249 is a diagram illustrating an example of a captured display image Ppre displayed on the receiver in Embodiment 23. FIG. FIG. 250 is a flowchart illustrating another example of processing operations of a receiver in Embodiment 23. FIG. 251 is a diagram illustrating another example in which the receiver in Embodiment 23 displays an AR image. FIG. 252 is a diagram illustrating another example in which the receiver in Embodiment 23 displays an AR image. FIG. 253 is a diagram illustrating another example in which the receiver in Embodiment 23 displays an AR image. FIG. 254 is a diagram illustrating another example in which the receiver in Embodiment 23 displays an AR image. FIG. 255 is a diagram illustrating an example of recognition information according to the twenty-third embodiment. FIG. 256 is a flowchart illustrating another example of processing operations of a receiver in Embodiment 23. FIG. 257 is a diagram illustrating an example in which the receiver in Embodiment 23 identifies bright line pattern regions. 258 is a diagram illustrating another example of a receiver in Embodiment 23. FIG. FIG. 259 is a flowchart illustrating another example of processing operations of a receiver in Embodiment 23. 260 is a diagram illustrating an example of a transmission system including a plurality of transmitters in Embodiment 23. FIG. FIG. 261 is a diagram illustrating an example of a transmission system including a plurality of transmitters and receivers in Embodiment 23. FIG. 262A is a flowchart illustrating an example of process operations of the receiver in Embodiment 23. FIG. 262B is a flowchart illustrating an example of process operations of the receiver in Embodiment 23. FIG. 263A is a flowchart illustrating a display method according to Embodiment 23. FIG. 263B is a block diagram illustrating a structure of the display device in Embodiment 23. FIG. 264 is a diagram illustrating an example in which the receiver in Modification 1 of Embodiment 23 displays an AR image. FIG. 265 is a diagram illustrating another example in which the receiver 200 in the first modification of the twenty-third embodiment displays an AR image. FIG. 266 is a diagram illustrating another example in which the receiver 200 in the first modification of the twenty-third embodiment displays an AR image. FIG. 267 is a diagram illustrating another example in which the receiver 200 in the first modification of the twenty-third embodiment displays an AR image. FIG. 268 is a diagram illustrating another example of the receiver 200 in the first modification of the twenty-third embodiment. FIG. 269 is a diagram illustrating another example in which the receiver 200 in the first modification of the twenty-third embodiment displays an AR image. FIG. 270 is a diagram illustrating another example in which the receiver 200 in the first modification of the twenty-third embodiment displays an AR image. FIG. 271 is a flowchart illustrating an example of processing operations of the receiver 200 in the first modification of the twenty-third embodiment. FIG. 272 is a diagram illustrating an example of a problem when an AR image assumed in the receiver in Embodiment 23 or the modification 1 thereof is displayed. FIG. 273 is a diagram illustrating an example in which the receiver in Modification 2 of Embodiment 23 displays the AR image. FIG. 274 is a flowchart illustrating an example of processing operations of a receiver in Modification 2 of Embodiment 23. FIG. 275 is a diagram illustrating another example in which the receiver in the second modification of the twenty-third embodiment displays an AR image. FIG. 276 is a flowchart illustrating another example of processing operations of a receiver in Modification 2 of Embodiment 23. 277 is a diagram illustrating another example in which a receiver in Modification 2 of Embodiment 23 displays an AR image. FIG. FIG. 278 is a diagram illustrating another example in which the receiver in the second modification of the twenty-third embodiment displays an AR image. FIG. 279 is a diagram illustrating another example in which the receiver in Modification 2 of Embodiment 23 displays an AR image. 280 is a diagram illustrating another example in which a receiver in Modification 2 of Embodiment 23 displays an AR image. FIG. FIG. 281A is a flowchart illustrating a display method according to one embodiment of the present invention. FIG. 281B is a block diagram illustrating a structure of a display device according to one embodiment of the present invention. FIG. 282 is a diagram illustrating an example of expansion and movement of the AR image in the third modification of the twenty-third embodiment. FIG. 283 is a diagram illustrating an example of expansion of an AR image in the third modification of the twenty-third embodiment. FIG. 284 is a flowchart illustrating an example of processing operations regarding expansion and movement of an AR image by a receiver in Modification 3 of Embodiment 23. FIG. 285 is a diagram illustrating an example of superimposition of AR images in the third modification of the twenty-third embodiment. FIG. 286 is a diagram illustrating an example of superimposition of AR images in the third modification of the twenty-third embodiment. FIG. 287 is a diagram illustrating an example of superimposition of AR images in the third modification of the twenty-third embodiment. FIG. 288 is a diagram illustrating an example of superimposition of AR images in the third modification of the twenty-third embodiment. FIG. 289A is a diagram illustrating an example of a captured display image obtained by imaging by the receiver in the third modification of the twenty-third embodiment. FIG. 289B is a diagram illustrating an example of a menu screen displayed on the display of the receiver in Modification 3 of Embodiment 23. FIG. 290 is a flowchart illustrating an example of processing operations of the receiver and the server in the third modification of the twenty-third embodiment. FIG. 291 is a diagram for describing the volume of audio reproduced by the receiver in the third modification of the twenty-third embodiment. FIG. 292 is a diagram illustrating a relationship between the distance from the receiver to the transmitter and the sound volume in the third modification of the twenty-third embodiment. 293 is a diagram illustrating an example of superimposition of AR images by a receiver in Modification 3 of Embodiment 23. FIG. 294 is a diagram illustrating an example of superimposition of AR images by a receiver in Modification 3 of Embodiment 23. FIG. 295 is a diagram for describing an example of how to obtain a line scan time by a receiver in Modification 3 of Embodiment 23. FIG. 296 is a diagram for describing an example of how to obtain a line scan time by a receiver in Modification 3 of Embodiment 23. FIG. FIG. 297 is a flowchart illustrating an example of how to obtain a line scan time by a receiver in the third modification of the twenty-third embodiment. 298 is a diagram illustrating an example of superimposition of AR images by a receiver in Modification 3 of Embodiment 23. FIG. 299 is a diagram illustrating an example of superimposition of AR images by a receiver in Modification 3 of Embodiment 23. FIG. 300 is a diagram illustrating an example of superimposition of AR images by a receiver in Modification 3 of Embodiment 23. FIG. 301 is a diagram illustrating an example of a decoding image obtained according to the attitude of the receiver in Modification 3 of Embodiment 23. FIG. 302 is a diagram illustrating another example of a decoding image acquired in accordance with the attitude of a receiver in Modification 3 of Embodiment 23. FIG. FIG. 303 is a flowchart illustrating an example of processing operation of a receiver in Modification 3 of Embodiment 23. 304 is a diagram illustrating an example of camera lens switching processing by a receiver in Modification 3 of Embodiment 23. FIG. 305 is a diagram illustrating an example of camera switching processing by a receiver in Modification 3 of Embodiment 23. FIG. FIG. 306 is a flowchart illustrating an example of processing operations of the receiver and the server in Modification 3 of Embodiment 23. 307 is a diagram illustrating an example of superimposition of AR images by a receiver in Modification 3 of Embodiment 23. FIG. FIG. 308 is a sequence diagram illustrating processing operations of a system including a receiver, a microwave oven, a relay server, and an electronic payment server in Modification 3 of Embodiment 23. FIG. 309 is a sequence diagram illustrating processing operations of a system including a POS terminal, a server, a receiver 200, and a microwave oven in Modification 3 of Embodiment 23. FIG. 310 is a diagram illustrating an example of indoor use in Modification 3 of Embodiment 23. FIG. 311 is a diagram illustrating an example of an augmented reality object display in the third modification of the twenty-third embodiment. FIG. 312 is a diagram showing a configuration of a display system in Modification 4 of Embodiment 23. In FIG. FIG. 313 is a flowchart showing processing operations of the display system in Modification 4 of Embodiment 23. FIG. 314 is a flowchart illustrating a recognition method according to an aspect of the present invention. FIG. 315 is a diagram illustrating an example of operation modes of visible light signals according to the twenty-fourth embodiment. FIG. 316 is a diagram illustrating an example of a PPDU format in mode 1 of the packet PWM according to the twenty-fourth embodiment. FIG. 317 is a diagram illustrating an example of a PPDU format in mode 2 of the packet PWM according to the twenty-fourth embodiment. FIG. 318 is a diagram illustrating an example of a PPDU format in mode 3 of the packet PWM according to the twenty-fourth embodiment. FIG. 319 is a diagram illustrating an example of a pulse width pattern in each SHR of modes 1 to 3 of the packet PWM according to the twenty-fourth embodiment. FIG. 320 is a diagram showing an example of a PPDU format in mode 1 of the packet PPM according to the twenty-fourth embodiment. FIG. 321 is a diagram illustrating an example of a PPDU format in mode 2 of the packet PPM according to the twenty-fourth embodiment. FIG. 322 is a diagram illustrating an example of a PPDU format in mode 3 of the packet PPM according to the twenty-fourth embodiment. FIG. 323 is a diagram illustrating an example of an interval pattern in each SHR of modes 1 to 3 of the packet PPM according to the twenty-fourth embodiment. FIG. 324 is a diagram illustrating an example of 12-bit data included in the PHY payload according to the twenty-fourth embodiment. FIG. 325 is a diagram illustrating processing of storing PHY frames in one packet according to Embodiment 24. FIG. 326 is a diagram illustrating processing of dividing the PHY frame into 2 packets according to Embodiment 24. FIG. 327 is a diagram illustrating processing of dividing a PHY frame into 3 packets according to Embodiment 24. FIG. 328 is a diagram illustrating processing of dividing the PHY frame into 4 packets according to Embodiment 24. FIG. 329 is a diagram illustrating processing of dividing the PHY frame into 5 packets according to Embodiment 24. FIG. 330 is a diagram illustrating processing of dividing the PHY frame into N (N = 6, 7 or 8) packets according to the twenty-fourth embodiment. FIG. 331 is a diagram illustrating processing of dividing the PHY frame into 9 packets according to Embodiment 24. FIG. 332 is a diagram illustrating processing of dividing the PHY frame into N (N = 10 to 16) packets according to the twenty-fourth embodiment. FIG. 333A is a flowchart illustrating a visible light signal generation method according to Embodiment 24. FIG. 333B is a block diagram showing a configuration of the signal generation apparatus according to Embodiment 24. FIG. 334 is a diagram illustrating a format of an MPM MAC frame in the twenty-fifth embodiment. FIG. 335 is a flowchart illustrating processing operations of the encoding device for generating an MPM MAC frame according to the twenty-fifth embodiment. FIG. 336 is a flowchart illustrating processing operation of the decoding device for decoding the MAC frame of MPM in the twenty-fifth embodiment. FIG. 337 is a diagram illustrating MAC PIB attributes according to the twenty-fifth embodiment. FIG. 338 is a diagram for describing an MPM light control method according to the twenty-fifth embodiment. FIG. 339 is a diagram illustrating attributes of a PHY PIB according to the twenty-fifth embodiment. FIG. 340 is a diagram for describing MPM in the twenty-fifth embodiment. FIG. 341 is a diagram illustrating PLCP header subfields according to Embodiment 25. FIG. 342 is a diagram illustrating PLCP center subfields according to the twenty-fifth embodiment. FIG. 343 is a diagram illustrating PLCP footer subfields according to the twenty-fifth embodiment. FIG. 344 is a diagram illustrating a waveform in a PHY PWM mode in the MPM according to the twenty-fifth embodiment. FIG. 345 is a diagram illustrating a PHY PPM mode waveform in the MPM according to the twenty-fifth embodiment. FIG. 346 is a flowchart illustrating an example of the decoding method according to the twenty-fifth embodiment. FIG. 347 is a flowchart illustrating an example of the coding method according to the twenty-fifth embodiment. FIG. 348 is a diagram illustrating an example in which the receiver in Embodiment 26 displays an AR image. 349 is a diagram illustrating an example of a captured display image on which an AR image is superimposed according to Embodiment 26. FIG. FIG. 350 is a diagram illustrating another example in which the receiver in Embodiment 26 displays an AR image. FIG. 351 is a flowchart illustrating operation of the receiver in Embodiment 26. FIG. 352 is a diagram for describing an operation of a transmitter in Embodiment 26. FIG. 353 is a diagram for describing another operation of the transmitter in Embodiment 26. FIG. 354 is a diagram for describing another operation of the transmitter in Embodiment 26. FIG. 355 is a diagram illustrating a comparative example for describing ease of reception of the optical ID in the twenty-sixth embodiment. FIG. 356A is a flowchart illustrating operation of the transmitter in Embodiment 26. FIG. 356B is a block diagram illustrating a configuration of a transmitter in Embodiment 26. FIG. 357 is a diagram illustrating another example in which the receiver in Embodiment 26 displays an AR image. 358 is a diagram for describing an operation of a transmitter in Embodiment 27. FIG. FIG. 359A is a flowchart illustrating a transmission method according to Embodiment 27. FIG. 359B is a block diagram illustrating a structure of a transmitter in Embodiment 27. FIG. 360 is a diagram illustrating an example of a detailed configuration of a visible light signal in Embodiment 27. 361 is a diagram illustrating another example of a detailed configuration of a visible light signal in Embodiment 27. FIG. 362 is a diagram illustrating another example of a detailed configuration of a visible light signal in Embodiment 27. FIG. FIG. 363 is a diagram illustrating another example of a detailed configuration of a visible light signal in Embodiment 27. FIG. 364 is a diagram illustrating a relationship between the total sum of the variables y ₀ to y ₃ , the total time length, and the effective time length in the twenty-seventh embodiment. FIG. 365A is a flowchart illustrating a transmission method according to Embodiment 27. FIG. 365B is a block diagram illustrating a structure of a transmitter in Embodiment 27.

A transmission method according to an aspect of the present invention is a transmission method for transmitting a signal according to a change in luminance of a light source, wherein a reception step of accepting a dimming degree designated for the light source as a designated dimming degree, and the designated dimming degree Is less than or equal to the first value, the signal encoded in the first mode is transmitted by a luminance change while causing the light source to emit light at the specified dimming level, and the specified dimming level is the first dimming level. A transmission step of transmitting the signal encoded in the second mode by a luminance change while causing the light source to emit light at the specified dimming level, and the specified dimming level is The value of the peak current of the light source for transmitting the signal encoded in the second mode by a change in luminance when it is greater than the first value and less than or equal to the second value is the specified dimming degree Is the first If it is smaller than the value of the peak current of the light source for transmitting the first said signal encoded in the mode of the luminance change.

As a result, as shown in FIG. 354, the peak current value of the light source when the designated dimming degree is greater than the first value and equal to or less than the second value by switching the signal encoding mode is It becomes smaller than the value of the peak current of the light source when the luminous intensity is the first value. Therefore, as the designated dimming degree is increased, a large peak current can be suppressed from flowing to the light source. As a result, deterioration of the light source can be suppressed. Further, since deterioration of the light source can be suppressed, communication between various devices can be performed for a long time.

In addition, when the designated dimming degree is smaller than a third value, the signal encoded in the first mode is transmitted by a luminance change while causing the light source to emit light at the designated dimming degree. The peak current value may be maintained constant with respect to the change in the designated dimming degree, and the third value may be smaller than the first value. Specifically, when the specified dimming level is smaller than the third value, the specified dimming level is decreased by increasing the time during which the light source is turned off as the specified dimming level decreases. The light source may be made to emit light at a luminous intensity, and the peak current value may be maintained at a constant value.

As a result, even when the specified dimming degree is small, the value of the peak current is kept constant, so that the visible light signal (that is, the light ID) that is transmitted by the luminance change is easily received by the receiver. be able to.

Further, the time for turning off the light source may be determined so that one cycle obtained by adding the time for transmitting the signal due to luminance change and the time for turning off the light source does not exceed 10 milliseconds.

For example, if one cycle exceeds 10 milliseconds, the luminance change of the light source for transmitting the encoded signal may be perceived by the human eye as flickering. Therefore, in the present disclosure, since the time for turning off the light source is determined so that one period does not exceed 10 milliseconds, it is possible to suppress flickering from being recognized by a person.

Further, when the designated dimming degree is smaller than a fourth value, the signal encoded in the first mode is transmitted by a luminance change while causing the light source to emit light at the designated dimming degree. As the specified dimming degree decreases, the peak current value is decreased to cause the light source to emit light at the specified dimming level, and the fourth value is smaller than the second value. Also good.

Thereby, even if the designated dimming degree is smaller, the light source can be appropriately emitted with the designated dimming degree.

The peak current value of the light source when the designated dimming level is the first value is the same as the peak current value of the light source when the designated dimming level is the maximum value. May be. For example, the maximum value of the specified dimming degree is 100%.

Thereby, even in the first mode, since a large peak current can be passed through the light source, it is possible to make it easier for the receiver to receive a signal transmitted by a change in luminance of the light source.

Also, the duty ratio of the signal encoded in the second mode may be larger than the duty ratio of the signal encoded in the first mode.

The first mode is a mode in which the increase in peak current is increased even if the increase in dimming degree is small, and the second mode is a mode in which the increase in peak current is suppressed even if the increase in dimming degree is large. Therefore, since the second mode suppresses a large peak current from flowing to the light source, deterioration of the light source can be suppressed. Further, since the first mode causes a large peak current to flow through the light source even when the dimming degree is small, the signal transmitted by the luminance change of the light source can be easily received by the receiver. Therefore, in the present disclosure, it is possible to achieve both suppression of deterioration of the light source and ease of signal reception.

Further, when the value of the peak current of the light source exceeds the fifth value, the transmission of the signal due to a change in luminance of the light source may be stopped.

Thereby, the deterioration of the light source can be further suppressed.

In addition, when the usage time of the light source is measured and the usage time is equal to or longer than a predetermined time, the brightness of the signal is set using a parameter value for causing the light source to emit light with a dimming degree greater than the specified dimming degree. You may send by change.

This makes it possible to prevent the signal transmitted due to the luminance change from becoming difficult to be received by the receiver due to the deterioration of the light source over time.

In addition, when the usage time of the light source is measured and the usage time is equal to or longer than the predetermined time, the pulse width of the current of the light source may be made larger than when the usage time is less than the predetermined time.

Thereby, even if the light source is deteriorated with time, the pulse width of the current of the light source is increased, so that it is possible to prevent the signal transmitted due to the luminance change of the light source from becoming difficult to be received by the receiver.

Moreover, the transmission method according to another aspect of the present invention is a transmission method for transmitting a signal according to a change in luminance of a light source, and accepting a dimming degree designated for the light source as a designated dimming degree; A step of transmitting the signal encoded in the first mode or the second mode by a luminance change while causing the light source to emit light at a specified dimming level, and the encoded in the second mode When the duty ratio of the signal is larger than the duty ratio of the signal encoded in the first mode, and the designated dimming level is changed from a small value to a large value in the transmission step, the designated dimming level When is the first value, the mode used for encoding the signal is switched from the first mode to the second mode, and the designated dimming level is small from a large value When the designated dimming level is the second value, the mode used for encoding the signal is switched from the second mode to the first mode, and the second value is changed. Is smaller than the first value.

Thereby, as shown in FIG. 358, the designated dimming degree (that is, the switching point) at which switching between the first mode and the second mode is performed differs depending on whether the designated luminous intensity increases or decreases. Therefore, frequent switching between these modes can be suppressed. That is, the occurrence of so-called chattering can be suppressed. As a result, the operation of the transmission device that transmits a signal can be stabilized. Further, the duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode. Therefore, similarly to the transmission method according to one embodiment of the present invention, it is possible to suppress a large peak current from flowing to the light source as the designated dimming degree is increased. As a result, deterioration of the light source can be suppressed. Further, since deterioration of the light source can be suppressed, communication between various devices can be performed for a long time. Further, when the designated dimming degree is small, the first mode with a small duty ratio is used. Therefore, the above-described peak current can be increased, and a signal that is easily received by the receiver can be transmitted as a visible light signal.

In the transmission method according to another aspect of the present invention, in the transmission step, the encoded signal is transmitted by a change in luminance when switching from the first mode to the second mode is performed. Changing the peak current of the light source for changing from the first current value to a second current value smaller than the first current value, and switching from the second mode to the first mode. When performed, the peak current is changed from a third current value to a fourth current value that is larger than the third current value, and the first current value is greater than the fourth current value. The second current value is larger than the third current value.

This makes it possible to appropriately switch between the first mode and the second mode.

A transmission method according to still another aspect of the present invention is a transmission method for transmitting a visible light signal according to a luminance change of a light emitter, and determining a pattern of the luminance change by modulating the signal; Transmitting the visible light signal by changing the luminance of red expressed by the light source included in the light emitter according to the determined pattern, wherein the visible light signal includes data, In the data, a first luminance value and a second luminance value smaller than the first luminance value appear on the time axis, and the first luminance value includes a preamble and a payload. And the length of time during which at least one of the second luminance values continues is not more than a first predetermined value, and in the preamble, that of the first and second luminance values. However, the first and second luminance values appear alternately along the time axis in the payload, and each of the first and second luminance values continues. The time length to be performed is greater than the first predetermined value and is determined according to the signal and a predetermined method.

Thereby, as shown in FIG. 363, the visible light signal includes one payload having a waveform determined according to the signal to be modulated (that is, the L data portion or the R data portion), and includes two payloads. Not in. Therefore, the visible light signal, that is, the packet of the visible light signal can be shortened. That is, a visible light signal can be transmitted in a short time, and communication between various devices can be performed in a short time. As a result, for example, even if the light emission period of red light expressed by the light source included in the light emitter is short, a packet of visible light signals can be transmitted during the light emission period.

In the payload, the first luminance value having a first time length, the second luminance value having a second time length, the first luminance value having a third time length, and a fourth time length When the respective luminance values appear in the order of the second luminance values, and in the transmission step, the sum of the first time length and the third time length is smaller than a second predetermined value The value of the current flowing through the light source is larger than when the sum of the first time length and the third time length is greater than the second predetermined value, and the second predetermined value is , Greater than the first predetermined value.

Accordingly, as shown in FIGS. 362 and 363, when the sum of the first time length and the third time length is small, the current value of the light source is increased, and the first time length and the third time length are increased. When the sum of the lengths is large, the current value of the light source is reduced. Therefore, the average luminance of the packet including data, preamble, and payload can be kept constant regardless of the signal.

In the payload, the first luminance value having the first time length D ₀ , the second luminance value having the second time length D ₁ , and the first luminance value having the third time length D ₂ are used. , The respective luminance values appear in the order of the second luminance value of the fourth time length D ₃ , and the sum of the four parameters y _k (k = 0, 1, 2, 3) obtained from the signal is If it is less than or equal to the third predetermined value, each of the first to fourth time lengths D ₀ to D ₃ is D _k = W ₀ + W ₁ × y _k (W ₀ and W ₁ are each greater than or equal to 0 It may be determined according to an integer).

As a result, as shown in FIG. 363 (b), it is possible to generate a payload having a short waveform according to the signal while setting each of the first to fourth time lengths D ₀ to D ₃ to be W ₀ or more.

When the sum of the four parameters y _k (k = 0, 1, 2, 3) is less than or equal to the third predetermined value, the transmission step includes the data, the preamble, and the payload, Data, the preamble, and the payload may be transmitted in this order.

As a result, as shown in FIG. 363 (b), the fact that the packet of the visible light signal including the data (that is, invalid data) does not include the L data portion is indicated to the receiving apparatus that receives the packet by the data. I can inform you.

When the sum of the four parameters y _k (k = 0, 1, 2, 3) is larger than the third predetermined value, each of the first to fourth time lengths D ₀ to D ₃ is D ₀ = W ₀ + W ₁ × (A−y ₀ ), D ₁ = W ₀ + W ₁ × (By ₁ ), D ₂ = W ₀ + W ₁ × (A−y ₂ ), and D ₃ = W ₀ + W ₁ × (B−y ₃ ) (A and B are each integers of 0 or more) may be determined.

Thereby, as shown in FIG. 363 (a), each of the first to fourth time lengths D ₀ to D ₃ (that is, the first to fourth time lengths D ′ ₀ to D ′ ₃ ) is set to W ₀ or more. However, even if the total sum is large, a short waveform payload can be generated according to the signal.

Further, when the sum of the four parameters y _k (k = 0, 1, 2, 3) is larger than the third predetermined value, in the transmission step, the data, the preamble, and the payload are The payload, the preamble, and the data may be transmitted in this order.

As a result, as shown in FIG. 363 (a), the fact that the packet of the visible light signal including data (that is, invalid data) does not include the R data portion is indicated to the receiving device that receives the packet by the data. I can inform you.

The light emitter has a plurality of light sources including a red light source, a blue light source, and a green light source, and in the transmitting step, the visible light source is formed using only the red light source among the plurality of light sources. An optical signal may be transmitted.

Thereby, the light emitter can display an image using a red light source, a blue light source, and a green light source, and can transmit a visible light signal having a wavelength that can be easily received to the receiving device.

Note that these comprehensive or specific modes may be realized by a recording medium such as an apparatus, a system, a method, an integrated circuit, a computer program, or a computer-readable CD-ROM. You may implement | achieve in arbitrary combinations of a circuit, a computer program, or a recording medium.

Hereinafter, embodiments will be specifically described with reference to the drawings.

It should be noted that each of the embodiments described below shows a comprehensive or specific example. The numerical values, shapes, materials, constituent elements, arrangement positions and connecting forms of the constituent elements, steps, order of steps, and the like shown in the following embodiments are merely examples, and are not intended to limit the present invention. In addition, among the constituent elements in the following embodiments, constituent elements that are not described in the independent claims indicating the highest concept are described as optional constituent elements.

(Embodiment 1)
The first embodiment will be described below.

(Observation of luminance of light emitting part)
We propose an imaging method that starts and ends the exposure at different times for each image sensor, rather than exposing all the image sensors at the same timing when capturing one image. FIG. 1 shows an example in which imaging devices arranged in one row are exposed simultaneously, and imaging is performed by shifting the exposure start time in the order of closer rows. Here, the exposure line of the image sensor that is exposed simultaneously is referred to as an exposure line, and the pixel line on the image corresponding to the image sensor is referred to as a bright line.

When this image capturing method is used to capture an image of a blinking light source on the entire surface of the image sensor, bright lines (light and dark lines of pixel values) along the exposure line appear on the captured image as shown in FIG. . By recognizing the bright line pattern, it is possible to estimate the light source luminance change at a speed exceeding the imaging frame rate. Thereby, by transmitting a signal as a change in light source luminance, communication at a speed higher than the imaging frame rate can be performed. When a signal is expressed by the light source taking two types of luminance values, the lower luminance value is called low (LO), and the higher luminance value is called high (HI). Low may be in a state where the light source is not shining, or may be shining weaker than high.

∙ By this method, information is transmitted at a speed exceeding the imaging frame rate.

When there are 20 exposure lines where the exposure times do not overlap in one captured image, and the imaging frame rate is 30 fps, a change in luminance with a period of 1.67 milliseconds can be recognized. When there are 1000 exposure lines whose exposure times do not overlap, it is possible to recognize a luminance change with a period of 1 / 30,000 second (about 33 microseconds). The exposure time is set shorter than 10 milliseconds, for example.

FIG. 2 shows a case where the exposure of the next exposure line is started after the exposure of one exposure line is completed.

In this case, when the number of frames per second (frame rate) is f and the number of exposure lines constituting one image is 1, if information is transmitted depending on whether or not each exposure line receives a certain amount of light, the maximum Can transmit information at a rate of fl bits per second.

It should be noted that communication is possible at higher speeds when exposure is performed with a time difference for each pixel, not for each line.

At this time, when the number of pixels per exposure line is m pixels and information is transmitted depending on whether each pixel receives light above a certain level, the transmission speed is a maximum of flm bits per second.

As shown in FIG. 3, if the exposure state of each exposure line by the light emission of the light emitting unit can be recognized at a plurality of levels, the light emission time of the light emitting unit is controlled by a unit time shorter than the exposure time of each exposure line. More information can be transmitted.

If the exposure state can be recognized in the Elv stage, information can be transmitted at a maximum rate of flElv bits per second.

Also, the basic period of transmission can be recognized by causing the light emitting unit to emit light at a timing slightly different from the exposure timing of each exposure line.

FIG. 4 shows a case where the exposure of the next exposure line is started before the exposure of one exposure line is completed. That is, the exposure times of adjacent exposure lines are partially overlapped in time. With such a configuration, (1) it is possible to increase the number of samples within a predetermined time as compared with the case where the exposure of the next exposure line is started after waiting for the end of the exposure time of one exposure line. By increasing the number of samples in a predetermined time, it becomes possible to more appropriately detect the optical signal generated by the optical transmitter that is the subject. That is, it is possible to reduce the error rate when detecting an optical signal. Further, (2) the exposure time of each exposure line can be made longer than when the exposure time of the next exposure line is started after waiting for the end of the exposure time of one exposure line. However, a brighter image can be acquired. That is, the S / N ratio can be improved. In all exposure lines, it is not necessary that the exposure time of adjacent exposure lines has a partial overlap in time, and a configuration in which some exposure lines have no partial overlap. It is also possible. By configuring a part of the exposure lines so as not to partially overlap in time, it is possible to suppress the generation of intermediate colors due to the overlap of exposure times on the imaging screen, and to detect bright lines more appropriately. .

In this case, the exposure time is calculated from the brightness of each exposure line, and the light emission state of the light emitting unit is recognized.

When the brightness of each exposure line is determined by a binary value indicating whether the luminance is equal to or higher than a threshold value, in order to recognize the state where no light is emitted, the state where the light emitting unit does not emit light is indicated for each line. It must last longer than the exposure time.

FIG. 5A shows the influence of the difference in exposure time when the exposure start times of the exposure lines are equal. 7500a is the case where the exposure end time of the previous exposure line is equal to the exposure start time of the next exposure line, and 7500b is the case where the exposure time is longer than that. As in the case of 7500b, the exposure time of adjacent exposure lines is partially overlapped in time, so that the exposure time can be increased. That is, the light incident on the image sensor increases and a bright image can be obtained. In addition, since the imaging sensitivity for capturing images with the same brightness can be suppressed to a low level, an image with less noise can be obtained, so that communication errors are suppressed.

FIG. 5B shows the influence of the difference in the exposure start time of each exposure line when the exposure times are equal. 7501a is the case where the exposure end time of the previous exposure line is equal to the exposure start time of the next exposure line, and 7501b is the case where the exposure of the next exposure line is started earlier than the end of exposure of the previous exposure line. By adopting a configuration in which the exposure times of adjacent exposure lines partially overlap in time as in 7501b, the number of lines that can be exposed per time can be increased. Thereby, the resolution becomes higher and a large amount of information can be obtained. Since the sample interval (= difference in exposure start time) becomes dense, the change in the light source luminance can be estimated more accurately, the error rate can be reduced, and the change in the light source luminance in a shorter time is recognized. be able to. By making the exposure time overlap, it is possible to recognize blinking of the light source that is shorter than the exposure time by using the difference in exposure amount between adjacent exposure lines.

Further, when the number of samples is small, that is, when the sample interval (time difference t _D shown in FIG. 5B) is long, there is a high possibility that a change in light source luminance cannot be detected accurately. In this case, the possibility can be suppressed by shortening the exposure time. That is, it is possible to accurately detect a change in light source luminance. Further, it is desirable that the exposure time satisfy the exposure time> (sample interval−pulse width). The pulse width is a pulse width of light that is a period during which the luminance of the light source is High. Thereby, the High brightness can be detected appropriately.

As described with reference to FIGS. 5A and 5B, in the configuration in which each exposure line is sequentially exposed so that the exposure times of adjacent exposure lines partially overlap in time, the exposure time is set to be longer than that in the normal shooting mode. By using the bright line pattern generated by setting it short for signal transmission, the communication speed can be dramatically improved. Here, it is possible to generate an appropriate bright line pattern by setting the exposure time during visible light communication to 1/480 seconds or less. Here, when the frame frequency = f, the exposure time needs to be set as exposure time <1/8 × f. Blanking that occurs during shooting is at most half the size of one frame. That is, since the blanking time is less than half of the shooting time, the actual shooting time is 1 / 2f at the shortest time. Furthermore, since it is necessary to receive quaternary information within a time of 1 / 2f, at least the exposure time needs to be shorter than 1 / (2f × 4). Since the normal frame rate is 60 frames / second or less, it is possible to generate an appropriate bright line pattern in the image data and perform high-speed signal transmission by setting the exposure time to 1/480 seconds or less. Become.

FIG. 5C shows an advantage when the exposure times are short when the exposure times of the exposure lines do not overlap. When the exposure time is long, even if the light source has a binary luminance change as in 7502a, the captured image has an intermediate color portion as in 7502e, and it becomes difficult to recognize the luminance change of the light source. There is. However, as the 7502D, after completion exposure of one exposure line, by a configuration in which the free time (predetermined waiting time) t _D2 not predetermined exposure start exposure of the next exposure line, the luminance variation of the light source Can be easily recognized. That is, a more appropriate bright line pattern such as 7502f can be detected. As in 7502D, be provided with a free time without predetermined exposure becomes an exposure time t _E can be realized to be smaller than the time difference t _D of the exposure start time of each exposure line. When the normal shooting mode has a configuration in which the exposure times of adjacent exposure lines partially overlap in time, the exposure time is set shorter than the normal shooting mode until a predetermined idle time occurs. This can be realized. Further, even when the normal photographing mode is the case where the exposure end time of the previous exposure line and the exposure start time of the next exposure line are equal, by setting the exposure time short until a predetermined non-exposure time occurs, Can be realized. Further, as 7502G, also by increasing the distance t _D of the exposure start time of each exposure line, after the exposure of one exposure line, following exposure line exposure start until a predetermined exposure was not free time (predetermined Waiting time) t _D2 can be provided. In this configuration, since the exposure time can be extended, a bright image can be taken, and noise is reduced, so that error tolerance is high. On the other hand, in this configuration, since the number of exposure lines that can be exposed within a certain time is reduced, there is a disadvantage that the number of samples is reduced as in 7502h. For example, when the imaging target is bright, the former configuration is used, and when the imaging target is dark, the latter configuration can be used to reduce the estimation error of the light source luminance change.

In all exposure lines, it is not necessary that the exposure time of adjacent exposure lines has a partial overlap in time, and a configuration in which some exposure lines have no partial overlap. It is also possible. Further, in all exposure lines, it is not necessary to provide a configuration in which an idle time (predetermined waiting time) in which a predetermined exposure is not performed is provided after the exposure of one exposure line is completed until the exposure of the next exposure line is started. It is also possible to have a configuration in which the lines partially overlap in time. With such a configuration, it is possible to take advantage of the advantages of each configuration. Further, the same readout method is used in the normal shooting mode in which shooting is performed at a normal frame rate (30 fps, 60 fps) and in the visible light communication mode in which shooting is performed with an exposure time of 1/480 second or less in which visible light communication is performed. Alternatively, a signal may be read by a circuit. By reading out signals with the same reading method or circuit, it is not necessary to use different circuits for the normal imaging mode and the visible light communication mode, and the circuit scale can be reduced.

FIG. 5D shows the relationship between the minimum change time t _S of the light source luminance, the exposure time t _E , the time difference t _D of the exposure start time of each exposure line, and the captured image. When t _E + t _D <t _S , since one or more exposure lines are always imaged in a state where the light source does not change from the start to the end of exposure, an image with clear brightness as in 7503d is obtained. It is easy to recognize the luminance change of the light source. When 2t _E > t _S , a bright line having a pattern different from the luminance change of the light source may be obtained, and it becomes difficult to recognize the luminance change of the light source from the captured image.

Figure 5E, the transition and time t _T of the light source luminance, which shows the relationship between the time difference t _D of the exposure start time of each exposure line. as t _D is larger than the t _T, exposure lines to be neutral is reduced, it is easy to estimate the light source luminance. When t _D > t _T , the exposure line of the intermediate color is continuously 2 or less, which is desirable. t _T, the light source is less than 1 microsecond in the case of LED, light source for an approximately 5 microseconds in the case of organic EL, a t _D by 5 or more microseconds, to facilitate estimation of the light source luminance be able to.

Figure 5F shows a high frequency noise _{t HT} of light source luminance, the relationship between the exposure time _{t E.} As t _E is larger than t _HT , the captured image is less affected by high frequency noise, and light source luminance is easily estimated. When t _E is an integral multiple of t _HT , the influence of high frequency noise is eliminated, and the light source luminance is most easily estimated. For estimation of the light source luminance, it is desirable that t _E > t _HT . The main cause of high frequency noise derived from the switching power supply circuit, since many of the t _HT in the switching power supply for the lamp is less than 20 microseconds, by the t _E and 20 micro-seconds or more, the estimation of the light source luminance It can be done easily.

Figure 5G is the case t _HT is 20 microseconds, which is a graph showing the relationship between the size of the exposure time t _E and the high frequency noise. When t _HT is considered that there is variation by the light source, from the graph, t _E is the value becomes equal to the value when the amount of noise takes a maximum, 15 microseconds or more, or, 35 microseconds or more, or, It can be confirmed that the efficiency is good when it is set to 54 microseconds or more, or 74 microseconds or more. From the viewpoint of reducing high-frequency noise, it is desirable that t _E be large. However, as described above, there is a property that light source luminance can be easily estimated in that the smaller the t _E , the more difficult the intermediate color portion is generated. Therefore, when the light source luminance change period is 15 to 35 microseconds, t _E is 15 microseconds or more, and when the light source luminance change period is 35 to 54 microseconds, t _E is 35 microseconds or more. t _E is 54 microseconds or more when the cycle is 54 to 74 microseconds of change, t _E when the period of the change in light source luminance is 74 microseconds or more may be set as 74 microseconds or more.

Figure 5H shows the relationship between the exposure time t _E and the recognition success rate. Since the exposure time t _E has a relative meaning with respect to the time when the luminance of the light source is constant, the value (relative exposure time) obtained by dividing the period t _S where the luminance of the light source changes by the exposure time t _E is taken as the horizontal axis. Yes. From the graph, it can be seen that if the recognition success rate is desired to be almost 100%, the relative exposure time should be 1.2 or less. For example, when the transmission signal is 1 kHz, the exposure time may be about 0.83 milliseconds or less. Similarly, when it is desired to set the recognition success rate to 95% or more, the relative exposure time may be set to 1.25 or less, and when the recognition success rate is set to 80% or more, the relative exposure time may be set to 1.4 or less. Recognize. Also, the recognition success rate drops sharply when the relative exposure time is around 1.5, and becomes almost 0% at 1.6, so it can be seen that the relative exposure time should not be set to exceed 1.5. . It can also be seen that after the recognition rate becomes 0 at 7507c, it rises again at 7507d, 7507e, and 7507f. Therefore, when it is desired to take a bright image by extending the exposure time, use an exposure time in which the relative exposure time is 1.9 to 2.2, 2.4 to 2.6, and 2.8 to 3.0. Just do it. For example, these exposure times may be used as the intermediate mode.

FIG. 6A is a flowchart of the information communication method in the present embodiment.

The information communication method in the present embodiment is an information communication method for acquiring information from a subject, and includes steps SK91 to SK93.

That is, in this information communication method, a plurality of bright lines corresponding to a plurality of exposure lines included in the image sensor are generated in an image obtained by photographing the subject by an image sensor according to a change in luminance of the subject. A first exposure time setting step SK91 for setting a first exposure time of the image sensor; and the image sensor shoots the subject whose luminance changes with the set first exposure time, First image acquisition step SK92 for acquiring a bright line image including a plurality of bright lines, and information acquisition for acquiring information by demodulating data specified by the patterns of the plurality of bright lines included in the acquired bright line image Step SK93, and in the first image acquisition step SK92, the plurality of exposure lines Re starts exposure at successively different times, and the exposure of the adjacent exposure line after a predetermined idle time from the end, the exposure is started adjacent to the exposure line.

FIG. 6B is a block diagram of the information communication apparatus according to the present embodiment.

The information communication device K90 in the present embodiment is an information communication device that acquires information from a subject, and includes constituent elements K91 to K93.

That is, the information communication apparatus K90 causes a plurality of bright lines corresponding to a plurality of exposure lines included in the image sensor to be generated in response to a change in luminance of the subject in an image obtained by photographing the subject by an image sensor. And an exposure time setting unit K91 for setting an exposure time of the image sensor, and the image sensor for acquiring a bright line image including the plurality of bright lines by photographing the subject whose luminance changes with the set exposure time. And an information acquisition unit K93 that acquires information by demodulating data specified by the patterns of the plurality of bright lines included in the acquired bright line image, and the plurality of exposure lines. Each of these starts exposure at different times sequentially and is adjacent to the exposure line. From the exposure of the down is completed after a predetermined idle time has elapsed, exposure is started.

In the information communication method and the information communication apparatus K90 shown in FIGS. 6A and 6B as described above, for example, as shown in FIG. 5C, each of the plurality of exposure lines is exposed to the adjacent exposure line adjacent to the exposure line. Since exposure is started after a lapse of a predetermined idle time after the end, it is possible to easily recognize a change in luminance of the subject. As a result, information can be appropriately acquired from the subject.

In the above embodiment, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. For example, the program causes the computer to execute the information communication method shown by the flowchart of FIG. 6A.

(Embodiment 2)
In the present embodiment, each application example using a receiver such as a smartphone that is the information communication device K90 in the first embodiment and a transmitter that transmits information as a blinking pattern of a light source such as an LED or an organic EL. explain.

In the following description, shooting in the normal shooting mode or normal shooting mode is referred to as normal shooting, and shooting in the visible light communication mode or visible light communication mode is referred to as visible light shooting (visible light communication). Further, instead of normal shooting and visible light shooting, shooting in an intermediate mode may be used, and an intermediate image may be used instead of a composite image described later.

FIG. 7 is a diagram illustrating an example of the photographing operation of the receiver in this embodiment.

The receiver 8000 switches the shooting mode to normal shooting, visible light communication, normal shooting, and so on. Then, the receiver 8000 generates a composite image in which the bright line pattern, the subject, and the surrounding area are clearly displayed by combining the normal captured image and the visible light communication image, and displays the composite image on the display. . This composite image is an image generated by superimposing the bright line pattern of the visible light communication image on the portion where the signal in the normal captured image is transmitted. Further, the bright line pattern, the subject, and the surroundings displayed by the composite image are clear and have a sharpness sufficiently recognized by the user. By displaying such a composite image, the user can more clearly know from where or from where the signal is transmitted.

FIG. 8 is a diagram illustrating another example of the photographing operation of the receiver in this embodiment.

The receiver 8000 includes a camera Ca1 and a camera Ca2. In such a receiver 8000, the camera Ca1 performs normal photographing, and the camera Ca2 performs visible light photographing. Thereby, the camera Ca1 acquires the normal captured image as described above, and the camera Ca2 acquires the visible light communication image as described above. Then, the receiver 8000 generates the above-described combined image by combining the normal captured image and the visible light communication image, and displays the combined image on the display.

FIG. 9 is a diagram illustrating another example of the photographing operation of the receiver in this embodiment.

In the receiver 8000 having two cameras, the camera Ca1 switches the shooting mode to normal shooting, visible light communication, normal shooting, and so on. On the other hand, the camera Ca2 continuously performs normal shooting. When the normal shooting is performed simultaneously with the cameras Ca1 and Ca2, the receiver 8000 receives from the normal shooting images acquired by these cameras using stereo vision (the principle of triangulation). The distance from the machine 8000 to the subject (hereinafter referred to as subject distance) is estimated. By using the subject distance estimated in this way, the receiver 8000 can superimpose the bright line pattern of the visible light communication image on an appropriate position of the normal captured image. That is, an appropriate composite image can be generated.

FIG. 10 is a diagram illustrating an example of the display operation of the receiver in this embodiment.

As described above, the receiver 8000 switches the photographing mode to visible light communication, normal photographing, visible light communication, and so on. Here, the receiver 8000 activates an application program when performing visible light communication for the first time. Then, the receiver 8000 estimates its own position based on the signal received by visible light communication. Next, when performing normal shooting, the receiver 8000 displays AR (Augmented Reality) information on the normal shot image acquired by the normal shooting. This AR information is acquired based on the position estimated as described above. Further, the receiver 8000 estimates the movement and direction change of the receiver 8000 based on the detection result of the 9-axis sensor and the motion detection of the normal captured image, and matches the estimated movement and direction change. To move the display position of the AR information. Thereby, the AR information can be made to follow the subject image of the normal captured image.

Further, when the receiver 8000 switches the shooting mode from the normal shooting to the visible light communication, the AR information is superimposed on the latest normal shooting image acquired at the time of the normal shooting immediately before the visible light communication. The receiver 8000 displays a normal captured image on which the AR information is superimposed. Similarly to the normal shooting, the receiver 8000 estimates the movement and direction change of the receiver 8000 on the basis of the detection result by the 9-axis sensor, and AR in accordance with the estimated movement and direction change. Move information and normal captured images. Thereby, AR information can be made to follow the subject image of the normal captured image in accordance with the movement of the receiver 8000 or the like in the case of visible light communication as in the case of normal imaging. Further, the normal image can be enlarged and reduced in accordance with the movement of the receiver 8000 or the like.

FIG. 11 is a diagram showing an example of the display operation of the receiver in this embodiment.

For example, the receiver 8000 may display the composite image on which the bright line pattern is projected, as shown in FIG. In addition, as shown in FIG. 11B, the receiver 8000 normally captures a signal explicit object that is an image having a predetermined color for notifying that a signal is transmitted instead of the bright line pattern. A composite image may be generated by superimposing on the image, and the composite image may be displayed.

Further, as shown in FIG. 11 (c), the receiver 8000 normally has a location where a signal is transmitted indicated by a dotted frame and an identifier (for example, ID: 101, ID: 102, etc.). The captured image may be displayed as a composite image. Further, as shown in FIG. 11D, the receiver 8000 recognizes a signal that is an image having a predetermined color for notifying that a specific type of signal is transmitted instead of the bright line pattern. A composite image may be generated by superimposing an object on a normal captured image, and the composite image may be displayed. In this case, the color of the signal identification object differs depending on the type of signal output from the transmitter. For example, when the signal output from the transmitter is position information, a red signal identification object is superimposed, and when the signal output from the transmitter is a coupon, the green signal identification object is Superimposed.

FIG. 12 is a diagram illustrating an example of the operation of the receiver in this embodiment.

For example, when receiving a signal through visible light communication, the receiver 8000 may display a normal captured image and output a sound for notifying the user that the transmitter has been found. In this case, the receiver 8000 varies the type of output sound, the number of outputs, or the output time depending on the number of transmitters found, the type of received signal, or the type of information specified by the signal. It may be allowed.

FIG. 13 is a diagram illustrating another example of the operation of the receiver in this embodiment.

For example, when the user touches the bright line pattern displayed in the composite image, the receiver 8000 generates an information notification image based on a signal transmitted from the subject corresponding to the touched bright line pattern, and the information notification Display an image. This information notification image indicates, for example, a store coupon or a place. The bright line pattern may be a signal explicit object, a signal identification object, a dotted line frame, or the like shown in FIG. The same applies to the bright line patterns described below.

FIG. 14 is a diagram illustrating another example of the operation of the receiver in this embodiment.

For example, when the user touches the bright line pattern displayed in the composite image, the receiver 8000 generates an information notification image based on a signal transmitted from the subject corresponding to the touched bright line pattern, and the information notification Display an image. For example, the information notification image indicates the current location of the receiver 8000 by a map or the like.

FIG. 15 is a diagram illustrating another example of the operation of the receiver in this embodiment.

For example, when the user performs a swipe on the receiver 8000 on which the composite image is displayed, the receiver 8000 performs normal shooting having a dotted frame and an identifier, similar to the normal shot image illustrated in FIG. An image is displayed and a list of information is displayed so as to follow the swipe operation. In this list, information specified by a signal transmitted from a location (transmitter) indicated by each identifier is shown. The swipe may be, for example, an operation of moving a finger from outside the right side of the display in the receiver 8000. The swipe may be an operation of moving a finger from the upper side, the lower side, or the left side of the display.

In addition, when information included in the list is tapped by the user, the receiver 8000 may display an information notification image (for example, an image showing a coupon) showing the information in more detail.

FIG. 16 is a diagram illustrating another example of the operation of the receiver in this embodiment.

For example, when the user swipes the receiver 8000 on which the composite image is displayed, the receiver 8000 displays the information notification image superimposed on the composite image so as to follow the swipe operation. This information notification image shows the subject distance with an arrow in an easy-to-understand manner for the user. The swipe may be, for example, an operation of moving a finger from outside the lower side of the display in the receiver 8000. The swipe may be an operation of moving a finger from the left side of the display, from the upper side, or from the right side.

FIG. 17 is a diagram illustrating another example of the operation of the receiver in this embodiment.

For example, the receiver 8000 images a transmitter, which is a signage indicating a plurality of stores, as a subject, and displays a normal captured image acquired by the imaging. Here, when the user taps the signage image of one store included in the subject displayed in the normal captured image, the receiver 8000 generates an information notification image based on a signal transmitted from the signage of the store Then, the information notification image 8001 is displayed. This information notification image 8001 is an image showing, for example, a vacant seat situation in a store.

FIG. 18 is a diagram illustrating an example of operations of the receiver, the transmitter, and the server in the present embodiment.

First, the transmitter 8012 configured as a television transmits a signal to the receiver 8011 by a luminance change. This signal includes, for example, information for prompting the user to purchase content related to the program being viewed. When the receiver 8011 receives the signal through visible light communication, the receiver 8011 displays an information notification image that prompts the user to purchase content based on the signal. When the user performs an operation to purchase the content, the receiver 8011 receives information included in a SIM (Subscriber Identity Module) card inserted into the receiver 8011, user ID, terminal ID, credit card information, billing At least one of the information, the password, and the transmitter ID is transmitted to the server 8013. The server 8013 manages a user ID and payment information in association with each user. Then, the server 8013 identifies the user ID based on the information transmitted from the receiver 8011, and confirms the payment information associated with the user ID. By this confirmation, the server 8013 determines whether or not to allow the user to purchase content. If the server 8013 determines to permit, the server 8013 transmits permission information to the receiver 8011. When receiving the permission information, the receiver 8011 transmits the permission information to the transmitter 8012. The transmitter 8012 that has received the permission information acquires and reproduces the content via a network, for example.

Further, the transmitter 8012 may transmit information including the ID of the transmitter 8012 to the receiver 8011 by changing the luminance. In this case, the receiver 8011 transmits the information to the server 8013. When the server 8013 obtains the information, the server 8013 can determine that, for example, a television program is being viewed by the transmitter 8012, and can perform a viewing rate survey of the television program.

Further, the receiver 8011 includes the content operated by the user (voting or the like) in the above information and transmits it to the server 8013, so that the server 8013 can reflect the content in the television program. That is, a viewer participation type program can be realized. Further, when the receiver 8011 accepts writing by the user, the contents of the writing are included in the above-described information and transmitted to the server 8013 so that the server 8013 can write the writing to the TV program or a bulletin board on the network. Etc. can be reflected.

Furthermore, when the transmitter 8012 transmits information as described above, the server 8013 can charge for viewing of a television program by pay broadcasting or an on-demand program. Further, the server 8013 displays an advertisement on the receiver 8011, displays detailed information of a TV program displayed on the transmitter 8012, and displays a URL of a site indicating the detailed information. Can do. Further, the server 8013 obtains the number of times the advertisement is displayed by the receiver 8011 or the amount of the product purchased by the advertisement, and thereby charges the advertiser according to the number or amount. be able to. Such billing can be made even if the user who saw the advertisement does not purchase the product immediately. Further, when the server 8013 acquires information indicating the manufacturer of the transmitter 8012 from the transmitter 8012 via the receiver 8011, the server 8013 provides a service (for example, a reward for sales of the above-described product to the manufacturer indicated by the information). Payment).

FIG. 19 is a diagram illustrating another example of the operation of the receiver in this embodiment.

The receiver 8030 is configured as a head mounted display including a camera, for example. The receiver 8030 starts photographing in the visible light communication mode, that is, visible light communication when the start button is pressed. When a signal is received through visible light communication, the receiver 8030 notifies the user of information corresponding to the received signal. This notification is performed, for example, by outputting sound from a speaker provided in the receiver 8030 or by displaying an image. In addition, when the start button is pressed, the visible light communication is received by the receiver 8030 when an input of a voice instructing the start is received by the receiver 8030 or a signal instructing the start is received by wireless communication. It may be started when it is done. In addition, visible light communication is started when the change width of the value obtained by the 9-axis sensor provided in the receiver 8030 exceeds a predetermined range or when a bright line pattern appears even in a normal photographed image. May be.

FIG. 20 is a diagram illustrating another example of the operation of the receiver in this embodiment.

The receiver 8030 displays the composite image 8034 as described above. Here, the user performs an operation of moving the fingertip so as to surround the bright line pattern in the composite image 8034. Upon receiving this operation, the receiver 8030 identifies the bright line pattern that is the target of the operation, and displays an information notification image 8032 based on a signal transmitted from a location corresponding to the bright line pattern.

FIG. 21 is a diagram illustrating another example of the operation of the receiver in this embodiment.

The receiver 8030 displays the composite image 8034 as described above. Here, the user performs an operation of placing the fingertip on the bright line pattern in the composite image 8034 for a predetermined time or more. Upon receiving this operation, the receiver 8030 identifies the bright line pattern that is the target of the operation, and displays an information notification image 8032 based on a signal transmitted from a location corresponding to the bright line pattern.

FIG. 22 is a diagram illustrating an example of the operation of the transmitter according to the present embodiment.

The transmitter transmits the signal 1 and the signal 2 alternately at a predetermined cycle, for example. Transmission of the signal 1 and transmission of the signal 2 are performed by luminance changes such as blinking of visible light. Further, the luminance change pattern for transmitting the signal 1 and the luminance change pattern for transmitting the signal 2 are different from each other.

FIG. 23 is a diagram illustrating another example of the operation of the transmitter according to the present embodiment.

As described above, when the transmitter repeatedly transmits the signal sequence of the structural unit including the block 1, the block 2, and the block 3, for each signal sequence, the transmitter changes the arrangement of the blocks included in the signal sequence. Also good. For example, each block is arranged in the order of block 1, block 2, and block 3 in the first signal sequence, and each block is arranged in the order of block 3, block 1, and block 2 in the next signal sequence. Thereby, it can be avoided that only the same block is acquired by a receiver that requires a periodic blanking period.

FIG. 24 is a diagram illustrating an application example of the receiver in this embodiment.

For example, the receiver 7510a configured as a smartphone images the light source 7510b with a back camera (out camera) 7510c, receives a signal transmitted from the light source 7510b, and acquires the position and orientation of the light source 7510b from the received signal. The receiver 7510a estimates the position and orientation of the receiver 7510a itself from how the light source 7510b is captured in the captured image and the sensor values of the 9-axis sensor provided in the receiver 7510a. The receiver 7510a captures the user 7510e with a front camera (face camera, in-camera) 7510f, and estimates the position and orientation of the head of the 7510e and the line-of-sight direction (eyeball position and orientation) by image processing. . The receiver 7510a transmits the estimation result to the server. The receiver 7510a changes the behavior (display content and playback sound) according to the viewing direction of the user 7510e. The imaging by the back camera 7510c and the imaging by the front camera 7510f may be performed simultaneously or alternately.

FIG. 25 is a diagram illustrating another example of the operation of the receiver in this embodiment.

The receiver displays the bright line pattern by the composite image or the intermediate image as described above. At this time, the receiver may not be able to receive a signal from the transmitter corresponding to the bright line pattern. Here, when the bright line pattern is selected by the user performing an operation (for example, tapping) on the bright line pattern, the receiver performs an optical zoom to enlarge the composite line image or the intermediate image in which the bright line pattern is enlarged. Display an image. By performing such optical zoom, the receiver can appropriately receive a signal from the transmitter corresponding to the bright line pattern. That is, even if an image obtained by imaging is too small to acquire a signal, the signal can be appropriately received by performing optical zoom. Even when an image having a size capable of acquiring a signal is displayed, fast reception can be performed by performing optical zoom.

(Summary of this embodiment)
The information communication method according to the present embodiment is an information communication method for acquiring information from a subject, and an bright line corresponding to an exposure line included in the image sensor is included in an image obtained by photographing the subject with an image sensor. A first exposure time setting step for setting an exposure time of the image sensor so as to occur in accordance with a change in luminance of the subject; and the image sensor photographs the subject whose luminance changes with the set exposure time. The bright line image acquisition step of acquiring a bright line image that is an image including the bright line, and the spatial position of the portion where the bright line appears can be identified based on the bright line image, and the subject and the subject An image display step for displaying a display image in which the surroundings of the subject are projected, and the image included in the acquired bright line image Including an information acquisition step of acquiring transmission information by demodulating the data identified by the pattern of lines.

For example, a composite image or an intermediate image as shown in FIGS. 7, 8, and 11 is displayed as a display image. In the display image in which the subject and the surroundings of the subject are projected, the spatial position of the part where the bright line appears is identified by a bright line pattern, a signal explicit object, a signal identification object, a dotted line frame, or the like. Therefore, the user can easily find a subject that is transmitting a signal due to a change in luminance by viewing such a display image.

The information communication method further includes a second exposure time setting step for setting an exposure time longer than the exposure time, and the image sensor photographs the subject and the surroundings of the subject with the long exposure time. A normal image acquisition step of acquiring a normal photographed image, a part where the bright line appears in the normal photographed image is identified based on the bright line image, and a signal object which is an image indicating the part is designated as the normal image A composite step of generating a composite image by superimposing the captured image, and the composite image may be displayed as the display image in the image display step.

For example, the signal object is a bright line pattern, a signal explicit object, a signal identification object, a dotted line frame, or the like, and a composite image is displayed as a display image as shown in FIGS. Thus, the user can more easily find the subject that is transmitting the signal due to the luminance change.

In the first exposure time setting step, an exposure time is set to 1/3000 sec. In the bright line image acquisition step, the bright line image in which the periphery of the subject is projected is acquired, and in the image display step. The bright line image may be displayed as the display image.

For example, the bright line image is acquired and displayed as an intermediate image. Therefore, it is not necessary to perform processing such as acquiring and synthesizing the normal captured image and the visible light communication image, and the processing can be simplified.

The image sensor includes a first image sensor and a second image sensor. In the normal image acquisition step, the first image sensor captures the normal captured image, and the bright line is acquired. In the image acquisition step, the bright line image may be acquired by capturing the second image sensor simultaneously with the capturing of the first image sensor.

For example, as shown in FIG. 8, a normal photographed image and a visible light communication image that is a bright line image are acquired by each camera. Therefore, compared with the case where a normal captured image and a visible light communication image are acquired with one camera, those images can be acquired earlier, and the processing can be speeded up.

Further, in the information communication method, when the part where the bright line appears in the display image is designated by a user operation, the information communication method further includes the transmission information acquired from the pattern of the bright line of the designated part. An information presentation step of presenting presentation information based on the information may be included. For example, the operation by the user is shown in association with a tap, swipe, an operation in which a fingertip is continuously applied to the part for a predetermined time, an operation in which a line of sight is directed to the part for a predetermined time, or the like. An operation of moving a part of the user's body to an arrow, an operation of applying a pen tip that changes in luminance to the part, or a touch sensor is touched, and a pointer displayed on the display image is applied to the part It is an operation.

For example, as shown in FIGS. 13 to 17, 20 and 21, the presentation information is displayed as an information notification image. Thereby, desired information can be presented to the user.

Further, the image sensor may be provided in a head mounted display, and in the image display step, a projector mounted on the head mounted display may display the display image.

Thus, for example, as shown in FIGS. 19 to 21, information can be easily presented to the user.

In addition, in the information communication method for acquiring information from a subject, a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in luminance of the subject. As described above, the first exposure time setting step of setting the exposure time of the image sensor, and the image including the bright line by the image sensor photographing the subject whose luminance changes with the set exposure time. A bright line image acquiring step for acquiring the bright line image, and an information acquiring step for acquiring information by demodulating data specified by the pattern of the bright line included in the acquired bright line image. In the acquisition step, the plurality of subjects are photographed during the period in which the image sensor is moved. By acquiring the bright line image including a plurality of parts where the bright lines appear, and in the information acquisition step, for each part, by demodulating data specified by the pattern of the bright lines of the parts, The information communication method further estimates the position of the image sensor based on the acquired positions of the plurality of subjects and the movement state of the image sensor. A position estimation step may be included.

This makes it possible to accurately estimate the position of the receiver including the image sensor based on luminance changes caused by subjects such as a plurality of lights.

In addition, in the information communication method for acquiring information from a subject, a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in luminance of the subject. As described above, the first exposure time setting step of setting the exposure time of the image sensor, and the image including the bright line by the image sensor photographing the subject whose luminance changes with the set exposure time. A bright line image acquisition step of acquiring a bright line image, an information acquisition step of acquiring information by demodulating data specified by a pattern of the bright line included in the acquired bright line image, and the acquired information Information presenting step for presenting, in the information presenting step, the image sensor Against over The may present an image that prompts the gesture that has been predetermined as the information.

Thus, depending on whether or not the user performs the gesture as prompted, the user can be authenticated and the convenience can be improved.

In addition, in the information communication method for acquiring information from a subject, a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in luminance of the subject. As described above, the exposure time setting step for setting the exposure time of the image sensor, and the image sensor captures the bright line image including the bright line by photographing the subject whose luminance changes at the set exposure time. And an information acquisition step of acquiring information by demodulating data specified by the bright line pattern included in the acquired bright line image. In the image acquisition step, the image is reflected on the reflection surface. The bright line image is acquired by photographing a plurality of the subjects, and the information acquisition step is performed. The bright line is separated into bright lines corresponding to each of the plurality of subjects according to the intensity of the bright lines included in the bright line image, and each subject is specified by a bright line pattern corresponding to the subject. Information may be acquired by demodulating the data.

This makes it possible to acquire appropriate information from each of the subjects even when the subject such as a plurality of illuminations changes in luminance.

In addition, in the information communication method for acquiring information from a subject, a bright line corresponding to an exposure line included in the image sensor is generated in an image obtained by photographing the subject by an image sensor according to a change in luminance of the subject. As described above, the exposure time setting step for setting the exposure time of the image sensor, and the image sensor captures the bright line image including the bright line by photographing the subject whose luminance changes at the set exposure time. And an information acquisition step of acquiring information by demodulating data specified by the bright line pattern included in the acquired bright line image. In the image acquisition step, the image is reflected on the reflection surface. The bright line image is acquired by photographing the subject, and the information communication method includes: , Based on the luminance distribution in the emission line image may include position estimation step for estimating the position of the object.

This makes it possible to estimate an appropriate subject position based on the luminance distribution.

An information communication method for transmitting a signal according to a luminance change, wherein a first determination step of determining a first pattern of luminance change by modulating a first signal to be transmitted, A second determination step of determining a second pattern of luminance change by modulating the signal of 2; and a luminance change according to the first pattern determined by the light emitter; A transmission step of transmitting the first and second signals by alternately performing a luminance change according to the second pattern.

Thereby, for example, as shown in FIG. 22, the first signal and the second signal can be transmitted without delay.

Further, in the transmission step, when the luminance change is switched between the luminance change according to the first pattern and the luminance change according to the second pattern, it may be switched with a buffer time.

Thereby, interference between the first signal and the second signal can be suppressed.

An information communication method for transmitting a signal according to a luminance change, wherein a determination step of determining a luminance change pattern by modulating a signal to be transmitted, and the light emitter changes in luminance according to the determined pattern And transmitting the signal to be transmitted, the signal comprising a plurality of large blocks, each of the plurality of large blocks including first data and a preamble for the first data. And a check signal for the first data, wherein the first data is composed of a plurality of small blocks, and the small blocks include second data, a preamble for the second data, and the first data. And a check signal for the second data may be included.

This makes it possible to acquire data appropriately for both receivers that require a blanking period and receivers that do not require a blanking period.

An information communication method for transmitting a signal by luminance change, wherein a plurality of transmitters each modulate a signal to be transmitted to determine a luminance change pattern, and for each transmitter, And a transmitting step in which a light emitter provided in the transmitter changes the luminance according to the determined pattern and transmits the signal to be transmitted. In the transmitting step, signals having different frequencies or protocols are transmitted. May be.

This makes it possible to suppress signal interference from multiple transmitters.

An information communication method for transmitting a signal by luminance change, wherein a plurality of transmitters each modulate a signal to be transmitted to determine a luminance change pattern, and for each transmitter, A transmitter in which a light emitter provided in the transmitter transmits a signal to be transmitted by changing in luminance according to the determined pattern, wherein in the transmitting step, one of the plurality of transmitters One transmitter may receive a signal transmitted from the other transmitter and transmit another signal in a manner that does not interfere with the received signal.

This makes it possible to suppress signal interference from multiple transmitters.

(Embodiment 3)
In this embodiment, each application example using a receiver such as a smartphone in Embodiment 1 or 2 and a transmitter that transmits information as a blinking pattern such as an LED or an organic EL will be described.

FIG. 26 is a diagram illustrating an example of processing operations of the receiver, the transmitter, and the server in the third embodiment.

For example, the receiver 8142 configured as a smartphone acquires position information indicating its own position, and transmits the position information to the server 8141. Note that the receiver 8142 acquires position information when, for example, GPS is used or other signals are received. The server 8141 transmits the ID list associated with the position indicated by the position information to the receiver 8142. The ID list includes, for each ID such as “abcd”, the ID and information associated with the ID.

The receiver 8142 receives a signal from a transmitter 8143 configured as, for example, a lighting device. At this time, the receiver 8142 may receive only a part of the ID (for example, “b”) as the above signal. In this case, the receiver 8142 searches the ID list for an ID including a part of the ID. If the unique ID is not found, the receiver 8142 further receives a signal from the transmitter 8143 that includes another portion of that ID. As a result, the receiver 8142 acquires a larger part (for example, “bc”) of the ID. Then, the receiver 8142 searches the ID list again for an ID including a part of the ID (for example, “bc”). By performing such a search, the receiver 8142 can specify all of the IDs even if only a part of the IDs can be acquired. When receiving a signal from the transmitter 8143, the receiver 8142 receives not only a part of the ID but also a check part such as CRC (Cyclic Redundancy Check).

FIG. 27 is a diagram illustrating an example of operations of the transmitter and the receiver in the third embodiment.

For example, the transmitter 8165 configured as a television acquires an image and an ID (ID 1000) associated with the image from the control unit 8166. The transmitter 8165 displays the image and transmits the ID (ID 1000) to the receiver 8167 by changing the luminance. The receiver 8167 receives the ID (ID 1000) by imaging and displays information associated with the ID (ID 1000). Here, the control unit 8166 changes the image output to the transmitter 8165 to another image. At this time, the control unit 8166 also changes the ID output to the transmitter 8165. That is, the control unit 8166 outputs the other ID (ID 1001) associated with the other image to the transmitter 8165 together with the other image. Thus, the transmitter 8165 displays another image and transmits another ID (ID 1001) to the receiver 8167 by changing the luminance. The receiver 8167 receives the other ID (ID 1001) by imaging, and displays information associated with the other ID (ID 1001).

FIG. 28 is a diagram illustrating an example of operations of the transmitter, the receiver, and the server in the third embodiment.

For example, the transmitter 8185 configured as a smartphone displays information indicating “coupon 100 yen discount”, for example, by changing the luminance of the display 8185a excluding the barcode portion 8185b, that is, by visible light communication. Send. The transmitter 8185 displays the barcode on the barcode portion 8185b without changing the luminance of the barcode portion 8185b. This bar code indicates the same information as the information transmitted by the visible light communication described above. Further, the transmitter 8185 displays a character or a picture indicating information transmitted by visible light communication, for example, a character “coupon 100 yen discount” on a portion of the display 8185a excluding the barcode portion 8185b. By displaying such characters or pictures, the user of the transmitter 8185 can easily grasp what information is being transmitted.

The receiver 8186 acquires the information transmitted by visible light communication and the information indicated by the barcode by imaging, and transmits the information to the server 8187. The server 8187 determines whether or not these pieces of information match or relate to each other. When it determines that these pieces of information match or relate to each other, the server 8187 executes processing according to the pieces of information. Alternatively, the server 8187 transmits the determination result to the receiver 8186, and causes the receiver 8186 to execute processing according to the information.

Note that the transmitter 8185 may transmit a part of the information indicated by the barcode by visible light communication. The barcode may indicate the URL of the server 8187. Further, the transmitter 8185 may acquire information associated with the ID by acquiring the ID as a receiver and transmitting the ID to the server 8187. The information associated with this ID is the same as the information transmitted by the above visible light communication or the information indicated by the barcode. Further, the server 8187 may transmit an ID associated with information (visible light communication information or barcode information) transmitted from the transmitter 8185 via the receiver 8186 to the transmitter 8185.

FIG. 29 is a diagram illustrating an example of operations of the transmitter and the receiver in the third embodiment.

For example, the receiver 8183 images a subject including a plurality of persons 8197 and street lamps 8195. The streetlight 8195 includes a transmitter 8195a that transmits information according to a change in luminance. By this imaging, the receiver 8183 acquires an image in which the image of the transmitter 8195a appears as the bright line pattern described above. Furthermore, the receiver 8183 acquires the AR object 8196a associated with the ID indicated by the bright line pattern from, for example, a server. Then, the receiver 8183 superimposes the AR object 8196a on a normal captured image 8196 obtained by normal imaging, and displays a normal captured image 8196 on which the AR object 8196a is superimposed.

(Summary of this embodiment)
The information communication method according to the present embodiment is an information communication method for transmitting a signal according to a luminance change, wherein a determination step for determining a luminance change pattern by modulating a signal to be transmitted and a light emitter are determined. A transmission step of transmitting the signal to be transmitted by changing the luminance according to the pattern, wherein the luminance change pattern includes two different luminances at arbitrary positions in a predetermined time width. In the pattern in which one of the values appears, in the determination step, the luminance change position that is the rising or falling position of the luminance in the time width is different for each of the different signals to be transmitted. In addition, the integrated value of the luminance of the luminous body in the time width is the same according to the preset brightness As a value to determine the pattern of the luminance change.

For example, for each of the signals “00”, “01”, “10”, and “11” that are different from each other, the rising positions of the luminance (luminance change positions) are different from each other, and a predetermined time width is set. The luminance change pattern is determined so that the integrated value of the luminance of the light emitter in (unit time width) becomes the same value according to a predetermined brightness (for example, 99% or 1%). As a result, the brightness of the illuminant can be kept constant for each signal to be transmitted, flicker can be suppressed, and the receiver that images the illuminant is based on the luminance change position. The brightness change pattern can be demodulated appropriately. In addition, the luminance change pattern is a pattern in which one of two different luminance values (luminance H (High) or luminance L (Low)) appears at any position in the unit time width. The brightness of the body can be changed continuously.

The information communication method further includes an image display step of sequentially switching and displaying each of the plurality of images. In the determination step, the displayed image is displayed each time the image is displayed in the image display step. The luminance change pattern for the identification information is determined by modulating the identification information corresponding to the signal to be transmitted. In the transmission step, the image is displayed every time the image is displayed in the image display step. The identification information may be transmitted by the luminance change of the light emitter according to the luminance change pattern determined with respect to the identification information corresponding to the existing image.

As a result, for example, as shown in FIG. 27, each time an image is displayed, identification information corresponding to the displayed image is transmitted. Therefore, the user receives the received information on the receiver based on the displayed image. The identification information to be made can be easily selected.

In the transmission step, each time an image is displayed in the image display step, the light emitter is further radiated according to a luminance change pattern determined for identification information corresponding to an image displayed in the past. The identification information may be transmitted by changing.

Thus, even if the receiver cannot receive the identification signal transmitted before switching because the displayed image has been switched, the image displayed in the past together with the identification information corresponding to the currently displayed image is displayed. Since the identification information corresponding to is also transmitted, the identification information transmitted before switching can be properly received again by the receiver.

In the determination step, each time an image is displayed in the image display step, the identification information corresponding to the displayed image and the time when the image is displayed are modulated as the signal to be transmitted. Thus, a pattern of luminance change with respect to the identification information and the time is determined, and in the transmission step, each time an image is displayed in the image display step, the identification information and time corresponding to the displayed image are determined. The identification information and the time are transmitted when the luminous body changes in luminance according to the luminance change pattern determined in the above, and the identification information and the time corresponding to an image displayed in the past are further determined. The identification information and the time are transmitted when the luminous body changes in luminance according to a luminance change pattern. Good.

As a result, each time an image is displayed, a plurality of ID time information (information consisting of identification information and time) is transmitted, and therefore the receiver transmits in the past from the received plurality of ID time information. Thus, the identification signal that could not be received can be easily selected based on the time included in each of the ID time information.

In addition, each of the light emitters has a plurality of regions that emit light, light in regions adjacent to each other among the plurality of regions interfere with each other, and only one of the plurality of regions is determined. When the luminance changes according to the luminance change pattern, in the transmission step, only the region arranged at the end of the plurality of regions may change in luminance according to the determined luminance change pattern.

As a result, only the area (light emitting part) arranged at the edge changes in luminance, so that the luminance change due to light from other areas is different from the case where only the area arranged at the edge other than the luminance changes. Can be suppressed. As a result, the receiver can appropriately capture the luminance change pattern by photographing.

In addition, when only two of the plurality of regions change in luminance according to the determined luminance change pattern, the transmission step includes: a region disposed at an end of the plurality of regions; The area adjacent to the area arranged at the end may change in luminance according to the determined luminance change pattern.

As a result, the luminance of the region arranged at the end (light emitting unit) and the region adjacent to the region arranged at the end (light emitting unit) change in luminance, compared to the case where the luminance of regions separated from each other changes. It is possible to keep a wide area in which the luminance continuously changes in brightness. As a result, the receiver can appropriately capture the luminance change pattern by photographing.

The information communication method according to the present embodiment is an information communication method for acquiring information from a subject, a location information transmission step for transmitting location information indicating a location of an image sensor used for photographing the subject, and the location information. A list receiving step for receiving an ID list including a plurality of pieces of identification information associated with the position indicated by, and an image obtained by photographing the subject by the image sensor corresponding to an exposure line included in the image sensor An exposure time setting step for setting an exposure time of the image sensor so that a bright line to be generated is generated according to a change in luminance of the subject, and the subject in which the image sensor changes in luminance is photographed at the set exposure time. An image acquisition step of acquiring a bright line image including the bright line, and An information acquisition step of acquiring information by demodulating data specified by the bright line pattern included in the bright line image, and a search step of searching the ID list for identification information including the acquired information. Including.

As a result, for example, as shown in FIG. 26, since the ID list is received in advance, even if the acquired information “bc” is only a part of the identification information, the appropriate identification information “ abcd "can be specified.

If the identification information including the acquired information is not uniquely specified in the search step, new information is acquired by repeatedly performing the image acquisition step and the information acquisition step, and the information communication The method may further include a re-search step of searching the ID list for identification information including the acquired information and the new information.

As a result, for example, as shown in FIG. 26, even if the acquired information “b” is only a part of the identification information and the identification information is not uniquely specified by the information alone, the new information “ Since “c” is acquired, appropriate identification information “abcd” can be specified based on the new information and the ID list.

The information communication method according to the present embodiment is an information communication method for acquiring information from a subject, and an bright line corresponding to an exposure line included in the image sensor is included in an image obtained by photographing the subject with an image sensor. An exposure time setting step for setting an exposure time of the image sensor so as to occur according to a change in luminance of the subject, and the image sensor photographing the subject whose luminance changes with the set exposure time, An image acquisition step of acquiring a bright line image including the bright line, an information acquisition step of acquiring identification information by demodulating data specified by the pattern of the bright line included in the acquired bright line image, and A transmission step for transmitting the identification information and position information indicating the position of the image sensor. And an error that receives error notification information for notifying an error when there is no acquired identification information in an ID list that includes a plurality of identification information associated with the position indicated by the position information. Receiving step.

As a result, when the acquired identification information is not in the ID list, the error notification information is received. Therefore, the user of the receiver that has received the error notification information can change the information associated with the acquired identification information. It can be easily grasped that it cannot be obtained.

(Embodiment 4)
In this embodiment, an application example using a receiver such as a smartphone in Embodiments 1 to 4 and a transmitter that transmits information as a blinking pattern such as an LED or an organic EL will be described.

FIG. 30 is a diagram illustrating an example of operations of the transmitter and the receiver in the fourth embodiment.

The transmitter includes an ID storage unit 8361, a random number generation unit 8362, an addition unit 8363, an encryption unit 8364, and a transmission unit 8365. The ID storage unit 8361 stores the ID of the transmitter. The random number generation unit 8362 generates different random numbers every certain time. Adder 8363 combines the latest random number generated by random number generator 8362 with the ID stored in ID storage unit 8361, and outputs the result as an edit ID. The encryption unit 8364 generates an encrypted edit ID by encrypting the edit ID. The transmission unit 8365 transmits the encrypted edit ID to the receiver by changing the luminance.

The receiver includes a receiving unit 8366, a decoding unit 8367, and an ID acquisition unit 8368. The receiving unit 8366 receives the encrypted edit ID from the transmitter by imaging the transmitter (visible light imaging). The decryption unit 8367 restores the edit ID by decrypting the received encrypted edit ID. The ID acquisition unit 8368 acquires the ID by extracting the ID from the restored editing ID.

For example, the ID storage unit 8361 stores the ID “100”, and the random number generation unit 8362 generates the latest random number “817” (Example 1). In this case, the adding unit 8363 generates and outputs the edit ID “100817” by combining the random number “817” with the ID “100”. The encryption unit 8364 generates an encrypted edit ID “abced” by encrypting the edit ID “100817”. The decryption unit 8367 of the receiver restores the edit ID “100817” by decrypting the encrypted edit ID “abced”. Then, the ID acquisition unit 8368 extracts the ID “100” from the restored editing ID “100817”. In other words, the ID acquisition unit 8368 acquires the ID “100” by deleting the last three digits of the edit ID.

Next, the random number generation unit 8362 generates a new random number “619” (example 2). In this case, the adding unit 8363 generates and outputs the edit ID “100619” by combining the random number “619” with the ID “100”. The encryption unit 8364 generates an encrypted edit ID “diffia” by encrypting the edit ID “100619”. The decryption unit 8367 of the transmitter restores the edit ID “100619” by decrypting the encrypted edit ID “diffia”. Then, the ID acquisition unit 8368 extracts the ID “100” from the restored editing ID “100619”. In other words, the ID acquisition unit 8368 acquires the ID “100” by deleting the last three digits of the edit ID.

In this way, the transmitter simply encrypts a combination of random numbers that change every fixed time without simply encrypting the ID, so the ID can be easily decrypted from the signal transmitted from the transmitter 8365. Can be prevented. In other words, when a simple encrypted ID is transmitted from the transmitter to the receiver several times, even if the ID is encrypted, if the ID is the same, the transmitter to the receiver. Since the transmitted signal is the same, the ID may be deciphered. However, in the example shown in FIG. 30, a random number that is changed at regular intervals is combined with an ID, and the ID that is combined with the random number is encrypted. Therefore, even when the same ID is transmitted to the receiver several times, the signals transmitted from the transmitter to the receiver can be made different if the transmission timings of these IDs are different. As a result, it is possible to prevent the ID from being easily decoded.

Note that when the receiver shown in FIG. 30 acquires the encryption edit ID, the receiver may transmit the encryption edit ID to the server and acquire the ID from the server.

(Guidance at the station)
FIG. 31 shows an example of a form of use of the present invention in a train platform. The user holds the portable terminal over an electronic bulletin board or lighting, and obtains information displayed on the electronic bulletin board, train information of a station where the electronic bulletin board is installed, information on the premises of the station, or the like by visible light communication. Here, the information itself displayed on the electronic bulletin board may be transmitted to the portable terminal by visible light communication, or ID information corresponding to the electronic bulletin board is transmitted to the portable terminal, and the ID information acquired by the portable terminal The information displayed on the electronic bulletin board may be acquired by inquiring the server. When the ID information is transmitted from the mobile terminal, the server transmits the content displayed on the electronic bulletin board to the mobile terminal based on the ID information. The train ticket information stored in the memory of the mobile terminal is compared with the information displayed on the electronic bulletin board, and the ticket information corresponding to the user's ticket is displayed on the electronic bulletin board. An arrow indicating the destination to the home where the user's scheduled train arrives is displayed on the display. When getting off, the route to the vehicle near the exit or the transfer route may be displayed. If a seat has been designated, the route to that seat may be displayed. When the arrow is displayed, it can be displayed more easily by displaying the arrow using the same color as the color of the train route in the map or the train guide information. In addition to the arrow display, the user's reservation information (home number, vehicle number, departure time, seat number) can also be displayed. By displaying the user reservation information together, it is possible to prevent erroneous recognition. If the ticket information is stored in the server, query the server from the mobile terminal to obtain and compare the ticket information, or compare the ticket information with the information displayed on the electronic bulletin board on the server side. Thus, information related to the ticket information can be acquired. The target route may be estimated from the history of the user performing a transfer search, and the route may be displayed. Further, not only the contents displayed on the electronic bulletin board but also the train information / premises information of the station where the electronic bulletin board is installed may be acquired and compared. Information related to the user may be highlighted with respect to the display of the electronic bulletin board on the display, or may be rewritten and displayed. When the user's boarding schedule is unknown, an arrow for guiding to the boarding place on each route may be displayed. When station premises information is acquired, an arrow for guiding to a store or restroom may be displayed on the display. The user's behavior characteristics may be managed in advance by a server, and an arrow for guiding the user to a store / restroom may be displayed on the display when the user often stops at a store / restaurant in the station. Only users who have behavioral characteristics of stopping at a store / restroom will display an arrow that directs them to a store / restroom, etc., and will not be displayed to other users, so the amount of processing can be reduced. . The color of the arrow leading to a store / restroom may be different from the arrow guiding the destination to the home. When both arrows are displayed simultaneously, it is possible to prevent erroneous recognition by using different colors. Note that although an example of a train is shown in FIG. 31, it is possible to perform display with a similar configuration even on an airplane or a bus.

(Coupon pop-up)
FIG. 32 shows an example in which coupon information acquired by visible light communication is displayed or a popup is displayed on the display of the mobile terminal when the user approaches the store. A user acquires coupon information of a store from an electronic bulletin board etc. by visible light communication using a portable terminal. Next, when the user enters the predetermined range from the store, the coupon information of the store or a pop-up is displayed. Whether or not the user has entered the predetermined range from the store is determined using the GPS information of the mobile terminal and the store information included in the coupon information. Not only coupon information but also ticket information may be used. Since it automatically alerts when a store where coupons and tickets can be used approaches, the user can use coupons and tickets appropriately.

(Launch operation application)
FIG. 33 shows an example in which a user acquires information from a home appliance by visible light communication using a mobile terminal. When ID information or information related to the home appliance is acquired from the home appliance by visible light communication, an application for operating the home appliance is automatically started. FIG. 33 shows an example using a television. With such a configuration, it is possible to start an application for operating a home appliance simply by holding the portable terminal over the home appliance.

(Database)
FIG. 34 shows an example of the configuration of the database held by the server that manages the ID transmitted by the transmitter.

The database has an ID-data table that holds data to be passed in response to an inquiry using the ID as a key, and an access log table that stores a record of the inquiry using the ID as a key. In the ID-data table, the ID sent by the transmitter, the data passed in response to an inquiry using the ID as a key, the conditions for passing the data, the number of times the access was made using the ID as a key, and the data are passed with the conditions cleared Have the number of times. The conditions for passing data include the date and time, the number of accesses, the number of successful accesses, the information of the terminal of the inquiry source (terminal model, the application that made the inquiry, the current location of the terminal, etc.) Age, gender, occupation, nationality, language, religion, etc.). By setting the number of successful accesses as a condition, a service method such as “1 yen per access, but 100 yen as the upper limit and no data returned thereafter” becomes possible. When there is access using ID as a key, the log table clears the ID, requested user ID, time, other incidental information, whether or not the data is passed, and the passed data Record the contents of.

(Different communication protocols for each zone)
FIG. 35 is a diagram illustrating an example of operation of a transmitter and a receiver in Embodiment 4.

The receiver 8420a receives zone information from the base station 8420h, recognizes in which zone it is located, and selects a reception protocol. The base station 8420h is configured as, for example, a mobile phone communication base station, a Wi-Fi access point, an IMES transmitter, a speaker, or a wireless transmitter (Bluetooth (registered trademark), ZigBee, specific low power wireless station, etc.). Note that the receiver 8420a may specify a zone based on position information obtained from GPS or the like. As an example, it is assumed that communication is performed at a signal frequency of 9.6 kHz in zone A, and communication is performed at a signal frequency of 15 kHz for ceiling lighting and signage of 4.8 kHz in zone B. At the position 8420j, the receiver 8420a recognizes that the current location is zone A from the information of the base station 8420h, performs reception at a signal frequency of 9.6 kHz, and receives signals transmitted from the transmitters 8420b and 8420c. The receiver 8420a recognizes that the current location is zone B from the information of the base station 8420i at the position 8420l, and is further trying to receive a signal from the ceiling lighting because the in-camera is directed upward. Is received at a signal frequency of 15 kHz, and signals transmitted by the transmitters 8420e and 8420f are received. At the position 8420m, the receiver 8420a recognizes that the current location is the zone B from above the base station 8420i, and further estimates that it is trying to receive a signal transmitted by the signage from the movement of the out camera. Receive at a signal frequency of .8 kHz and receive the signal transmitted by transmitter 8420g. The receiver 8420a receives the signals of both the base station 8420h and the base station 8420i at the position 8420k, and cannot determine whether the current location is the zone A or the zone B. Therefore, the reception process is performed at both 9.6 kHz and 15 kHz. I do. It should be noted that the part where the protocol differs depending on the zone is not limited to the frequency, but may be the server that inquires about the modulation method, signal format, and ID of the transmission signal. The base stations 8420h and 8420i may transmit the protocol in the zone to the receiver, or may transmit only the ID indicating the zone, and the receiver may acquire the protocol information from the server using the zone ID as a key. Good.

The transmitters 8420b to 8420f receive the zone ID and protocol information transmitted by the base stations 8420h and 8420i, and determine the signal transmission protocol. A transmitter 8420d capable of receiving signals transmitted by both base station 8420h and base station 8420i utilizes a base station zone protocol with stronger signal strength, or alternately uses both protocols.

(Zone recognition and services for each zone)
FIG. 36 is a diagram illustrating an example of operation of a receiver and a transmitter in Embodiment 4.

The receiver 8421a recognizes the zone to which it belongs from the received signal. The receiver 8421a provides services (coupon distribution, point assignment, route guidance, etc.) determined for each zone. As an example, the receiver 8421a receives a signal transmitted from the left side of the transmitter 8421b and recognizes that it is in the zone A. Here, the transmitter 8421b may transmit different signals depending on the transmission direction. Further, the transmitter 8421b may transmit a signal such that a different signal is received according to the distance to the receiver by using a signal having a light emission pattern such as 2217a. In addition, the receiver 8421a may recognize the positional relationship with the transmitter 8421b from the direction and size in which the transmitter 8421b is imaged, and may recognize the zone in which the receiver 8421a is located.

一部 A part of the signal indicating that it is located in the same zone may be shared. For example, IDs representing zone A transmitted from the transmitter 8421b and the transmitter 8421c are common to the first half. Accordingly, the receiver 8421a can recognize the zone where the receiver 8421a is located only by receiving the first half of the signal.

(Summary of this embodiment)
The information communication method in the present embodiment is an information communication method for transmitting a signal by a luminance change, and a determination step for determining a plurality of luminance change patterns by modulating each of a plurality of transmission target signals; Each of the plurality of light emitters changes a luminance according to any one of the determined plurality of luminance change patterns, thereby transmitting a transmission target signal corresponding to any one of the patterns. A transmission step, wherein two or more of the plurality of light emitters each have two types of brightness different from each other for each time unit predetermined for the light emitter. The time unit predetermined for each of the two or more light emitters is set so that either one of the two lights is output. Differently, luminance changes at different frequencies to.

Thereby, since each of the two or more light emitters (for example, a transmitter configured as a lighting device) changes in luminance at a different frequency, a signal to be transmitted (for example, the ID of the light emitter) from the light emitters. Can easily distinguish and acquire these signals to be transmitted.

In the transmission step, each of the plurality of light emitters changes in luminance at any one of at least four frequencies, and two or more light emitters of the plurality of light emitters are: The luminance may be changed at the same frequency. For example, in the transmitting step, when the plurality of light emitters are projected onto a light receiving surface of an image sensor for receiving the plurality of transmission target signals, all the light emitters adjacent to each other on the light receiving surface. Each of the plurality of light emitters changes in luminance so that the frequency of the luminance change differs between them.

Thus, if there are at least four types of frequencies used for luminance change, even if there are two or more illuminants that change in luminance at the same frequency, that is, the number of frequency types is the number of illuminants. Even in the case where the number is smaller, the frequency of the luminance change can be surely made different among all the light emitters adjacent to each other on the light receiving surface of the image sensor based on the four-color problem or the four-color theorem. As a result, the receiver can easily distinguish and acquire each of the transmission target signals transmitted from the plurality of light emitters.

In the transmission step, each of the plurality of light emitters may transmit the signal to be transmitted by changing in luminance at a frequency specified by a hash value of the signal to be transmitted.

Thereby, since each of the plurality of light emitters changes in luminance at a frequency specified by a hash value of a signal to be transmitted (for example, the ID of the light emitter), when the receiver receives the signal to be transmitted, It can be determined whether the frequency specified from the actual luminance change matches the frequency specified by the hash value. That is, the receiver can determine whether or not there is an error in the received signal (for example, the ID of the light emitter).

The information communication method may further calculate a frequency corresponding to the transmission target signal as a first frequency from the transmission target signal stored in the signal storage unit according to a predetermined function. A calculation step; a frequency determination step for determining whether or not the second frequency stored in the frequency storage unit matches the calculated first frequency; and the first frequency and the second frequency A frequency error notification step of notifying an error when it is determined that the frequency does not match, and a determination step when it is determined that the first frequency and the second frequency match. Then, a luminance change pattern is determined by modulating the transmission target signal stored in the signal storage unit, and in the transmission step, the plurality of light emitters are detected. Of any one of the light emitters according to the determined the pattern, by changing the luminance in the first frequency may transmit a signal of the transmission target stored in the signal storage unit.

Thereby, it is determined whether or not the frequency stored in the frequency storage unit matches the frequency calculated from the signal to be transmitted stored in the signal storage unit (ID storage unit). When the determination is made, an error is notified, so that the abnormality detection of the signal transmission function by the light emitter can be easily performed.

The information communication method further includes a check value calculation step of calculating a first check value from a transmission target signal stored in the signal storage unit according to a predetermined function, and a check value storage unit. A check value determination step for determining whether or not the stored second check value matches the calculated first check value; and the first check value and the second check value. A check value error notification step of notifying an error when it is determined that they do not match, and a determination step when it is determined that the first check value and the second check value match Then, a luminance change pattern is determined by modulating the signal to be transmitted stored in the signal storage unit. In the transmission step, the plurality of light emitters are determined. Any one of the emitters of blood, by changing luminance in accordance with the determined the pattern, may transmit a signal of the transmission target stored in the signal storage unit.

Thereby, it is determined whether or not the check value stored in the check value storage unit matches the check value calculated from the transmission target signal stored in the signal storage unit (ID storage unit), If it is determined that they do not coincide with each other, an error is notified, so that it is possible to easily detect abnormality of the signal transmission function by the light emitter.

The information communication method according to the present embodiment is an information communication method for acquiring information from a subject, and corresponds to a plurality of exposure lines included in the image sensor in an image obtained by photographing the subject by an image sensor. An exposure time setting step for setting an exposure time of the image sensor so that a plurality of bright lines are generated according to a change in luminance of the subject, and the exposure time set for the subject whose luminance is changed by the image sensor. And acquiring information by demodulating data identified by the pattern of the plurality of bright lines included in the acquired bright line image, and an image acquisition step of acquiring a bright line image including the plurality of bright lines Information acquisition step, and a plurality of bright line patterns included in the acquired bright line image. There are, and a frequency specifying step of specifying the frequency of the luminance change of the subject. For example, in the frequency specifying step, a plurality of header patterns, which are a plurality of predetermined patterns for indicating a header, are included in the plurality of bright line patterns, and the number of pixels between the plurality of header patterns is determined. Is determined as the frequency of luminance change of the subject.

Thus, since the frequency of the luminance change of the subject is specified, when a plurality of subjects having different luminance change frequencies are photographed, information from these subjects can be easily distinguished and acquired.

Further, in the image acquisition step, the bright line image including a plurality of patterns each represented by a plurality of bright lines is acquired by photographing a plurality of subjects each changing in luminance, and the information acquisition step acquires the bright line image. When a part of each of the plurality of patterns included in the bright line image overlaps, by demodulating data specified by a portion excluding the part from each of the plurality of patterns, the plurality of patterns Information may be acquired from each of the patterns.

Thus, since data is not demodulated from a portion where a plurality of patterns (a plurality of bright line patterns) overlap, it is possible to prevent erroneous information from being acquired.

Further, in the image acquisition step, a plurality of bright line images are acquired by photographing the plurality of subjects a plurality of times at mutually different timings, and in the frequency specifying step, the bright line images are included in the bright line image. A frequency for each of a plurality of patterns is specified, and in the information acquisition step, a plurality of patterns in which the same frequency is specified is searched from a plurality of previous bright line images, and the searched plurality of patterns are combined and combined. The information may be obtained by demodulating data specified by the plurality of patterns.

Thereby, a plurality of patterns (plural emission line patterns) in which the same frequency is specified are searched from a plurality of emission line images, the plurality of searched patterns are combined, and information is acquired from the combined patterns. Therefore, even when a plurality of subjects are moving, information from the plurality of subjects can be easily distinguished and acquired.

The information communication method may further include: identifying information on the subject included in the information acquired in the information acquisition step; and the frequency specification for a server in which frequencies are registered for each of the identification information. A transmission step of transmitting specific frequency information indicating the frequency specified in the step; a related information acquisition step of acquiring related information associated with the identification information and the frequency indicated by the specific frequency information from the server; May be included.

Thereby, related information associated with the identification information (ID) acquired based on the luminance change of the subject (transmitter) and the frequency of the luminance change is acquired. Therefore, by changing the frequency of the luminance change of the subject and updating the frequency registered in the server to the frequency after the change, the receiver that acquired the identification information before the frequency change acquires the related information from the server. Can be prevented. That is, by changing the frequency registered in the server in accordance with the change in the luminance change frequency of the subject, the receiver that has acquired the subject identification information in the past can acquire the related information from the server indefinitely. It can prevent becoming a state.

The information communication method is further acquired in the identification information acquisition step of acquiring the identification information of the subject by extracting a part from the information acquired in the information acquisition step, and in the information acquisition step. In addition, a setting frequency specifying step of specifying a number indicated by the remaining part other than the part of the information as a setting frequency of the luminance change set for the subject may be included.

As a result, the information obtained from the plurality of bright line patterns can include the identification information of the subject and the set frequency of the luminance change set for the subject without depending on each other. Can increase the degree of freedom.

(Embodiment 5)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

(Well-known communication of visible light to humans)
FIG. 37 is a diagram illustrating an example of operation of a transmitter in Embodiment 5.

The light emitting unit of the transmitter 8921a repeats blinking and visible light communication that are visible to humans, as shown in FIG. By performing blinking that is visible to humans, it is possible to inform humans that visible light communication is possible. The user notices that visible light communication is possible because the transmitter 8921a is blinking, performs visible light communication with the receiver 8921b directed to the transmitter 8921a, and performs user registration of the transmitter 8921a.

That is, the transmitter in the present embodiment alternately and repeatedly performs a step in which the light emitter transmits a signal due to a change in luminance and a step in which the light emitter blinks so as to be visually recognized by human eyes.

The transmitter may separately provide a visible light communication unit and a blinking unit (communication status display unit) as shown in FIG.

When the transmitter operates as shown in FIG. 37 (c), it can be seen that the light emitting unit is blinking while performing visible light communication. That is, for example, the transmitter repeatedly performs high luminance visible light communication with a brightness of 75% and low luminance visible light communication with a brightness of 1% alternately. For example, when an abnormality occurs in the transmitter and a signal different from usual is transmitted, the operation shown in (c) of FIG. 37 is performed to alert the user without stopping the visible light communication. Can do.

(Example of application to directions)
FIG. 38 is a diagram illustrating an example of application of the transmission and reception system in the fifth embodiment.

The receiver 8955a receives, for example, the transmission ID of the transmitter 8955b configured as a guide plate, acquires the map data displayed on the guide plate from the server, and displays the map data. At this time, the server may transmit an advertisement suitable for the user of the receiver 8955a, and the receiver 8955a may also display this advertisement information. The receiver 8955a displays a route from the current location to a location designated by the user.

(Application log storage and application examples)
FIG. 39 is a diagram illustrating an example of application of the transmission / reception system in the fifth embodiment.

The receiver 8957a receives the ID transmitted from the transmitter 8957b configured as a signboard, for example, acquires coupon information from the server, and displays the coupon information. The receiver 8957a stores subsequent user actions such as saving a coupon, moving to a store displayed on the coupon, shopping at the store, and leaving without saving the coupon. Save to 8957c. As a result, the subsequent behavior of the user who has acquired information from the sign 8957b can be analyzed, and the advertising value of the sign 8957b can be estimated.

(Application example for screen sharing)
FIG. 40 is a diagram illustrating an example of application of the transmission and reception system in the fifth embodiment.

For example, the transmitter 8960b configured as a projector or a display transmits information (SSID, password for wireless connection, IP address, password for operating the transmitter) for wireless connection to itself. Alternatively, an ID serving as a key for accessing these pieces of information is transmitted. For example, the receiver 8960a configured as a smartphone, a tablet, a laptop computer, or a camera receives the signal transmitted from the transmitter 8960b, acquires the information, and establishes a wireless connection with the transmitter 8960b. This wireless connection may be connected via a router, or may be directly connected by Wi-Fi Direct, Bluetooth (registered trademark), Wireless Home Digital Interface, or the like. The receiver 8960a transmits a screen displayed by the transmitter 8960b. Thereby, the image of the receiver can be easily displayed on the transmitter.

When the transmitter 8960b is connected to the receiver 8960a, the transmitter 8960b notifies the receiver 8960a that a password is required in addition to the information transmitted by the transmitter in order to display the screen. If is not sent, the transmitted screen may not be displayed. At this time, the receiver 8960a displays a password input screen such as 8960d and allows the user to input the password.

As described above, the information communication method according to one or more aspects has been described based on the embodiment. However, the present invention is not limited to this embodiment. Unless it deviates from the gist of the present invention, various modifications conceived by those skilled in the art have been made in this embodiment, and forms constructed by combining components in different embodiments are also within the scope of one or more aspects. May be included.

Also, as shown in FIG. 41, an information communication method according to an aspect of the present invention may be applied.

FIG. 41 is a diagram showing an example of application of the transmission / reception system in the fifth embodiment.

A camera configured as a receiver for visible light communication performs imaging in a normal imaging mode (Step 1). By this imaging, the camera acquires an image file configured in a format such as EXIF (Exchangeable image file format). Next, the camera performs imaging in the visible light communication imaging mode (Step 2). Based on the bright line pattern in the image obtained by this imaging, the camera acquires a signal (visible light communication information) transmitted by the visible light communication from the transmitter that is the subject (Step 3). Furthermore, the camera obtains information corresponding to the key from the server by using the signal (reception information) as a key and accessing the server (Step 4). The camera transmits a signal (visible light reception data) transmitted from the subject by visible light communication, information acquired from the server, and data indicating a position where the transmitter as the subject is projected in the image indicated by the image file. And the data indicating the time when the signal transmitted by the visible light communication is received (the time in the moving image) are stored as metadata in the above-described image file. Note that when a plurality of transmitters are projected as subjects in an image (image file) obtained by imaging, the camera stores, for each transmitter, some metadata corresponding to the transmitter. Save to file.

When a display or projector configured as a transmitter for visible light communication displays an image indicated by the above-described image file, the display or projector transmits a signal corresponding to metadata included in the image file by visible light communication. For example, the display or the projector may transmit the metadata itself by visible light communication, or may transmit a signal associated with the transmitter displayed in the image as a key.

A portable terminal (smart phone) configured as a receiver for visible light communication receives a signal transmitted by visible light communication from the display or projector by capturing an image of the display or projector. If the received signal is the above-described key, the portable terminal uses the key to acquire the transmitter metadata associated with the key from the display, projector, or server. In addition, if the received signal is a signal (visible light reception data or visible light communication information) transmitted from an actual transmitter by visible light communication, the portable terminal receives the visible light from a display, a projector, or a server. Information corresponding to received light data or visible light communication information is acquired.

(Summary of this embodiment etc.)
The information communication method in the present embodiment is an information communication method for acquiring information from a subject, and each exposure included in the image sensor is included in an image obtained by photographing the first subject that is the subject by an image sensor. A first exposure time setting step for setting a first exposure time of the image sensor so that a plurality of bright lines corresponding to a line are generated according to a change in luminance of the first subject; A first bright line image acquisition step of acquiring a first bright line image that is an image including the plurality of bright lines by photographing the first subject that changes with the set first exposure time; The first transmission information is acquired by demodulating data specified by the pattern of the plurality of bright lines included in the acquired first bright line image. An information obtaining step of, after said first transmission information is obtained by sending a control signal, and a door control steps to open the door against the opening and closing devices of the door.

This makes it possible to use a receiver equipped with an image sensor like a door key and eliminate the need for a special electronic lock. As a result, it is possible to perform communication between various devices including a device having a small computing power.

The information communication method may further include a second image in which the image sensor includes a plurality of bright lines by photographing the second subject whose luminance changes with the set first exposure time. Second transmission information is acquired by demodulating data specified by a pattern of the plurality of bright lines included in the acquired second bright line image and a second bright line image acquiring step of acquiring a bright line image A second information acquisition step; and an approach determination step of determining whether or not a receiving device including the image sensor is approaching the door based on the acquired first and second transmission information. In the door control step, the control signal may be transmitted when it is determined that the receiving device is approaching the door.

Thereby, the door can be opened only when the receiving device (receiver) approaches the door, that is, only at an appropriate timing.

Further, in the information communication method, a second exposure time setting step for setting a second exposure time longer than the first exposure time, and the image sensor sets a third subject. A normal image acquisition step of acquiring a normal image on which the third subject is projected by photographing at the second exposure time, and the normal image acquisition step includes an area including optical black of the image sensor For each of the plurality of exposure lines, the charge is read after a predetermined time has elapsed from the time when the charge is read for the exposure line adjacent to the exposure line, and the first bright line image obtaining step is performed. Then, without using the optical black for the charge readout, the optical black in the image sensor is different from the optical black. For each of the plurality of exposure lines in the region, the charge is read after a time longer than the predetermined time from when the charge is read for the exposure line adjacent to the exposure line. Also good.

As a result, when the first bright line image is acquired, the readout (exposure) of charges from the optical black is not performed. Therefore, the readout (exposure) of charges from the effective pixel area, which is an area other than the optical black in the image sensor, is performed. This time can be lengthened. As a result, it is possible to increase the time for receiving a signal in the effective pixel region, and it is possible to acquire many signals.

Further, in the information communication method, the length in the direction perpendicular to each of the plurality of bright lines in the plurality of bright line patterns included in the first bright line image is less than a predetermined length. A length determination step for determining whether or not there is a frame rate of the image sensor when the pattern length is determined to be less than the predetermined length; A frame rate changing step of changing to a second frame rate that is slower than the first frame rate when the image is acquired; and the image sensor detects the first subject whose luminance changes at the second frame rate. And a third bright line image acquisition step of acquiring a third bright line image, which is an image including a plurality of bright lines, by taking an image with the set first exposure time. It may include a third information obtaining step of obtaining the first transmission information by demodulating the data identified by said plurality of emission lines pattern contained in has been the third bright line image.

As a result, when the signal length indicated by the bright line pattern (bright line region) included in the first bright line image is less than, for example, one block of the transmitted signal, the frame rate is reduced and the bright line is renewed. An image is acquired as a third bright line image. As a result, the length of the bright line pattern included in the third bright line image can be increased, and the transmitted signal can be acquired for one block.

The information communication method further includes a ratio setting step for setting a ratio between a vertical width and a horizontal width of an image obtained by the image sensor, and the first bright line image acquisition step includes the set ratio. A clipping determination step for determining whether or not an end in a direction perpendicular to each exposure line in the image is clipped, and when it is determined that the end is clipped, the ratio set in the ratio setting step Changing the ratio to a non-clipping ratio which is a ratio at which the edge is not clipped, and the image sensor captures the first bright line with the non-clipping ratio by photographing the first subject whose luminance changes. An acquisition step of acquiring an image.

Thereby, for example, when the ratio of the horizontal width to the vertical width of the effective pixel area of the image sensor is 4: 3, the ratio of the horizontal width to the vertical width of the image is set to 16: 9, and a bright line along the horizontal direction appears. That is, when the exposure line is along the horizontal direction, it is determined that the upper end and the lower end of the image are clipped. That is, it is determined that the end of the first bright line image is missing. In this case, the ratio of the image is changed to 4: 3, which is a ratio that is not clipped. As a result, it is possible to prevent the end of the first bright line image from being lost, and a large amount of information can be acquired from the first bright line image.

The information communication method further includes a compression step of generating a compressed image by compressing the first bright line image in a direction parallel to each of the plurality of bright lines included in the first bright line image. And a compressed image transmission step of transmitting the compressed image.

Thereby, it is possible to appropriately compress the first bright line image without losing information indicated by a plurality of bright lines.

The information communication method further determines that the receiving device including the image sensor has been moved in a predetermined manner, and a gesture determination step for determining whether or not the receiving device has been moved in a predetermined manner. In some cases, an activation step of activating the image sensor may be included.

This makes it possible to easily start the image sensor only when necessary, and to improve power consumption efficiency.

(Embodiment 6)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

FIG. 42 is a diagram illustrating an application example of the transmitter and the receiver in the sixth embodiment.

The robot 8970 has, for example, a function as a self-propelled cleaner and a function as a receiver in each of the above embodiments. The lighting devices 8971a and 8971b each have a function as a transmitter in each of the above embodiments.

For example, the robot 8970 performs cleaning while moving in the room and photographs the lighting device 8971a that illuminates the room. The lighting device 8971a transmits the ID of the lighting device 8971a by changing the luminance. As a result, the robot 8970 receives the ID from the lighting device 8971a and estimates its own position (self-position) based on the ID as in the above embodiments. That is, the robot 8970 moves itself based on the detection result by the 9-axis sensor, the relative position of the lighting device 8971a reflected in the image obtained by photographing, and the absolute position of the lighting device 8971a specified by the ID. Is estimated.

Furthermore, when the robot 8970 moves away from the lighting device 8971a by moving, the robot 8970 transmits a signal to turn off the lighting device 8971a (turn-off command). For example, when the robot 8970 leaves the lighting device 8971a by a predetermined distance, the robot 8970 transmits a turn-off command. Alternatively, the robot 8970 transmits a turn-off command to the lighting device 8971a when the lighting device 8971a does not appear in the image obtained by shooting or when another lighting device appears in the image. When the lighting device 8971a receives a turn-off command from the robot 8970, the lighting device 8971a turns off according to the turn-off command.

Next, the robot 8970 detects that it has approached the lighting device 8971b based on the estimated self-position while moving and cleaning. That is, the robot 8970 holds information indicating the position of the lighting device 8971b, and when the distance between the self position and the position of the lighting device 8971b is equal to or less than a predetermined distance, the lighting device 8971b. Detecting that you are approaching. Then, the robot 8970 transmits a signal (lighting command) for instructing lighting to the lighting device 8971b. When the lighting device 8971b receives the lighting command, the lighting device 8971b lights up in accordance with the lighting command.

Thereby, the robot 8970 can brighten only the surroundings while moving and can easily perform cleaning.

FIG. 43 is a diagram illustrating an application example of the transmitter and the receiver in the sixth embodiment.

The lighting device 8974 has a function as a transmitter in each of the above embodiments. The lighting device 8974 illuminates a route bulletin board 8975 at a railway station, for example, while changing in luminance. The receiver 8973 pointed to the route bulletin board 8975 by the user photographs the route bulletin board 8975. As a result, the receiver 8973 acquires the ID of the route bulletin board 8975, and acquires detailed information about each route described in the route bulletin board 8975, which is information associated with the ID. The receiver 8973 displays a guide image 8973a indicating the detailed information. For example, the guidance image 8973a indicates the distance to the route described on the route bulletin board 8975, the direction toward the route, and the time when the next train arrives on the route.

Here, when the guide image 8973a is touched by the user, the receiver 8973 displays a supplementary guide image 8973b. This supplementary guide image 8973b is, for example, a user's selection operation of any one of a railway timetable, information on another route different from the route indicated by the guide image 8973a, and detailed information on the station. It is an image for displaying accordingly.

(Embodiment 7)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

(Reception of signals from multiple directions by multiple light receiving units)
FIG. 44 is a diagram illustrating an example of a receiver in Embodiment 7.

For example, the receiver 9020a configured as a wristwatch includes a plurality of light receiving units. For example, as shown in FIG. 44, the receiver 9020a is arranged in the vicinity of the character indicating 12 o'clock in the light receiving portion 9020b arranged at the upper end portion of the rotating shaft that supports the long hand and the short hand of the watch. Light receiving portion 9020c. The light receiving unit 9020b receives light traveling toward the light receiving unit 9020b along the direction of the rotation axis described above, and the light receiving unit 9020c transmits light traveling toward the light receiving unit 9020c along the direction connecting the rotation axis and the character indicating 12:00. Receive. Accordingly, when the user holds the receiver 9020a in front of the chest as when checking the time, the light receiving unit 9020b can receive light from above. As a result, the receiver 9020a can receive a signal from the ceiling lighting. Further, when the user holds the receiver 9020a in front of the chest as when checking the time, the light receiving unit 9020c can receive light from the front direction. As a result, the receiver 9020a can receive a signal from a signage or the like at the front.

These light receiving units 9020b and 9020c have directivity so that signals can be received without interference even when there are a plurality of transmitters at close positions.

(Wayway guidance using a watch-type display)
FIG. 45 is a diagram illustrating an example of a reception system in the seventh embodiment.

For example, the receiver 9023b configured as a wristwatch is connected to the smartphone 9022a via wireless communication such as Bluetooth (registered trademark). The receiver 9023b has a dial made up of a display such as a liquid crystal display, and can display information other than the time. The smartphone 9022a recognizes the current location from the signal received by the receiver 9023b, and displays the route and distance to the destination on the display surface of the receiver 9023b.

FIG. 46 is a diagram illustrating an example of a signal transmission / reception system according to the seventh embodiment.

The signal transmission / reception system includes a smartphone (smartphone) that is a multi-function mobile phone, an LED light emitting device that is a lighting device, a home appliance such as a refrigerator, and a server. The LED light emitter performs communication using BTLE (Bluetooth (registered trademark) Low Energy) and visible light communication using LED (Light-Emitting-Diode). For example, the LED light emitter controls a refrigerator or communicates with an air conditioner by BTLE. In addition, the LED light emitter controls a power source of a microwave oven, an air purifier, a television (TV), or the like by visible light communication.

The television includes, for example, a solar power generation element, and uses the solar power generation element as an optical sensor. That is, when a signal is transmitted when the brightness of the LED light emitter changes, the television detects a change in the brightness of the LED light emitter based on a change in the power generated by the solar power generation element. Then, the television acquires the signal transmitted from the LED light emitter by demodulating the signal indicated by the detected luminance change. When the signal is a command indicating that the power is on, the television switches its main power source to ON, and when the signal is a command indicating that the power source is OFF, the television switches its main power source to OFF.

Also, the server can communicate with the air conditioner via a router and a specific low power radio station (extra small). Furthermore, since the air conditioner can communicate with the LED light emitter via BTLE, the server can communicate with the LED light emitter. Therefore, the server can switch the power of the TV ON and OFF via the LED light emitter. The smartphone can control the power supply of the TV via the server by communicating with the server via, for example, Wi-Fi (Wireless Fidelity).

As shown in FIG. 46, the information communication method according to the present embodiment allows the mobile terminal (smart phone) to transmit a control signal (transmission data string or user command) by wireless communication (such as BTLE or Wi-Fi) different from visible light communication. ) To a lighting device (light emitting device), a visible light communication step in which the lighting device performs visible light communication by changing the luminance according to the control signal, and a control target device (such as a microwave oven). Includes an execution step of detecting a luminance change of the lighting device, acquiring a control signal by demodulating a signal specified by the detected luminance change, and executing a process according to the control signal. Thereby, even if the portable terminal cannot change the luminance for visible light communication, the luminance of the lighting device can be changed instead of the portable terminal by wireless communication, and the control target device is appropriately controlled. can do. The mobile terminal may be a wristwatch instead of a smartphone.

(Reception without interference)
FIG. 47 is a flowchart showing a reception method in which interference is eliminated in the seventh embodiment.

First, start is made in step 9001a, and it is checked whether there is a periodic change in the intensity of light received in step 9001b. If YES, the process proceeds to step 9001c. In the case of NO, the process proceeds to Step 9001d to receive a wide range of light by setting the lens of the light receiving unit to a wide angle, and returns to Step 9001b. In step 9001c, it is confirmed whether the signal can be received. If YES, the process proceeds to step 9001e, the signal is received, and the process ends in step 9001g. In the case of NO, the process proceeds to Step 9001f, the lens of the light receiving unit is telephoto, and light in a narrow range is received, and the process returns to Step 9001c.

This method makes it possible to receive signals from transmitters in a wide direction while eliminating signal interference from a plurality of transmitters.

(Estimation of transmitter orientation)
FIG. 48 is a flowchart showing a method for estimating the orientation of a transmitter in the seventh embodiment.

First, start is performed in step 9002a, the lens of the light receiving unit is set to the maximum telephoto in step 9002b, and it is checked whether there is a periodic change in the intensity of light received in step 9002c. Proceed to 9002d. In the case of NO, the process proceeds to Step 9002e, where a wide range of light is received by setting the lens of the light receiving unit to a wide angle, and the process returns to Step 9002c. In step 9002d, the signal is received, in step 9002f, the lens of the light receiving unit is set to the maximum telephoto position, the light receiving direction is changed along the boundary of the light receiving range, and the direction in which the light receiving intensity is maximized is detected. Estimating that it is in the direction, the process ends at step 9002d.

This method makes it possible to estimate the direction in which the transmitter exists. The maximum wide angle may be set first, and the telephoto may be gradually increased.

(Start receiving)
FIG. 49 is a flowchart showing a reception start method according to the seventh embodiment.

First, start is performed in step 9003a, and in step 9003b, it is confirmed whether a signal from a base station such as Wi-Fi, Bluetooth (registered trademark), IMES or the like is received. If YES, the process proceeds to step 9003c. If NO, the process returns to step 9003b. In step 9003c, it is confirmed whether or not the base station is registered in the receiver or server as a trigger for starting reception. If YES, the process proceeds to step 9003d, starts receiving a signal, and ends in step 9003e. . If NO, the process returns to step 9003b.

This method can start reception without the user performing a reception start operation. Further, power consumption can be suppressed as compared with the case where reception is always performed.

(Generation of ID using information from other media)
FIG. 50 is a flowchart showing an ID generation method using information of another medium together in the seventh embodiment.

First, start in step 9004a, ID of carrier communication network, Wi-Fi, Bluetooth (registered trademark), etc. connected in step 9004b, or position information obtained from the ID, position information obtained from GPS, etc. Is transmitted to the upper bit ID index server. In step 9004c, the upper bits of the visible light ID are received from the upper bit ID index server, and in step 9004d, the signal from the transmitter is received as the lower bits of the visible light ID. In step 9004e, the upper and lower bits of the visible light ID are combined and transmitted to the ID resolution server, and the process ends in step 9004f.

This method makes it possible to obtain upper bits that are commonly used near the receiver and reduce the amount of data transmitted by the transmitter. In addition, the receiving speed of the receiver can be increased.

Note that the transmitter may transmit both the upper and lower bits. In this case, the receiver using this method can synthesize the ID when the lower bit is received, and the receiver not using this method obtains the ID by receiving the entire ID from the transmitter. .

(Selection of reception method by frequency separation)
FIG. 51 is a flowchart showing a reception method selection method based on frequency separation in the seventh embodiment.

First, start is performed in step 9005a, and the optical signal received in step 9005b is applied to a frequency filter circuit, or discrete Fourier series expansion is performed to perform frequency decomposition. In step 9005c, it is confirmed whether or not a low frequency component exists. If YES, the process proceeds to step 9005d, a signal expressed in a low frequency region such as frequency modulation is decoded, and the process proceeds to step 9005e. If NO, the process proceeds to step 9005e. In step 9005e, it is confirmed whether or not the base station is registered in the receiver or server as a trigger for starting reception. If YES, the process proceeds to step 9005f, and a signal expressed in a high frequency region such as pulse position modulation. And the process proceeds to Step 9005g. If NO, the process advances to step 9005g. In step 9005g, signal reception is started, and in step 9005h, the process ends.

This method makes it possible to receive signals modulated by a plurality of modulation schemes.

(Signal reception when exposure time is long)
FIG. 52 is a flowchart showing a signal reception method when the exposure time is long in the seventh embodiment.

First, start in step 9030a, and if the sensitivity can be set in step 9030b, set the sensitivity to the maximum. If the exposure time can be set in step 9030c, the exposure time is set shorter than that in the normal shooting mode. In step 9030d, two images are picked up and a difference in luminance is obtained. If the position or direction of the imaging unit changes during the imaging of two images, the change is canceled and an image as if it was captured from the same position and direction is generated to obtain the difference. In Step 9030e, a value obtained by averaging the luminance values in the direction parallel to the difference image or the exposure line of the captured image is obtained. The averaged values in step 9030f are arranged in the direction perpendicular to the exposure line, and discrete Fourier transform is performed. In step 9030g, it is recognized whether there is a peak near a predetermined frequency, and the process ends in step 9030h.

This method allows signals to be received even when the exposure time is long, such as when the exposure time cannot be set or when a normal image is captured simultaneously.

When the exposure time is set automatically, when the camera is directed to a transmitter configured as illumination, the exposure time is set to about 1/60 second to 1/480 second by the automatic exposure correction function. If the exposure time cannot be set, a signal is received under this condition. In the experiment, when the illumination is blinked periodically, if the time of one cycle is about 1/16 or more of the exposure time, stripes can be visually recognized in the direction perpendicular to the exposure line. I was able to recognize it. At this time, since the brightness is too high in the portion where the illumination is reflected and it is difficult to confirm the stripe, it is preferable to obtain the signal period from the portion where the illumination light is reflected.

When using a method of periodically turning on / off the light emitting unit, such as a frequency shift keying method or a frequency multiplexing modulation method, even if the modulation frequency is the same as that of the pulse position modulation method, It is difficult for the viewer to see the flicker, and it is difficult for the flicker to appear in the video shot with the video camera. Therefore, a low frequency can be used as the modulation frequency. Since the temporal resolution of human vision is about 60 Hz, a frequency higher than this frequency can be used as the modulation frequency.

When the modulation frequency is an integral multiple of the imaging frame rate of the receiver, pixels at the same position in the two images are imaged when the light pattern of the transmitter is in the same phase, so a bright line appears in the difference image. It is difficult to receive. Since the imaging frame rate of the receiver is usually 30 fps, reception is easy if the modulation frequency is set to a value other than an integral multiple of 30 Hz. Also, since there are various imaging frame rates of the receiver, two disjoint modulation frequencies are assigned to the same signal, and the transmitter receives signals by alternately using the two modulation frequencies for transmission. The machine can easily restore the signal by receiving at least one signal.

FIG. 53 is a diagram illustrating an example of a transmitter dimming (adjusting brightness) method.

By adjusting the ratio between the high luminance section and the low luminance section, the average luminance changes and the brightness can be adjusted. At this time, to keep the period T ₁ which repeats high and low brightness constant, it is possible to keep the frequency peak constant. For example, in any of (a), (b), and (c) of FIG. 53, transmission is performed while the time T1 between the first luminance change that becomes brighter than the average luminance and the second luminance change is kept constant. When dimming the machine darkly, the time for lighting brighter than the average brightness is shortened. On the other hand, when the transmitter is dimmed brightly, the illumination time is set longer than the average luminance. 53 (b) and 53 (c) are dimmed darker than (a), and FIG. 53 (c) is dimmed the darkest. Thereby, dimming can be performed while transmitting signals having the same meaning.

The average brightness may be changed by changing the brightness value of the section with high brightness, the brightness of the section with low brightness, or both brightness values.

FIG. 54 is a diagram showing an example of a method for configuring the dimming function of the transmitter.

Since the accuracy of the component parts is limited, even if the same dimming setting is performed, the brightness is slightly different from that of another transmitter. However, when transmitters are arranged side by side, if the brightness of adjacent transmitters is different, unnaturalness can be felt. Therefore, the user adjusts the brightness of the transmitter by operating the dimming correction operation unit. The dimming correction unit holds the correction value, and the dimming control unit controls the brightness of the light emitting unit according to the correction value. When the degree of dimming is changed by the user operating the dimming operation unit, the dimming control unit uses the changed dimming setting value and the correction value held in the dimming correction unit. In addition, the brightness of the light emitting unit is controlled. The dimming control unit transmits the dimming setting value to another transmitter through the interlocking dimming unit. When the dimming setting value is transmitted from another device through the linked dimming unit, the dimming control unit, based on the dimming setting value and the correction value held in the dimming correction unit, To control the brightness.

According to one embodiment of the present invention, there is provided a control method for controlling an information communication device that transmits a signal by changing luminance of a light emitter, and includes a plurality of different signals for a computer of the information communication device. A determination step for determining a luminance change pattern of a different frequency for each different signal by modulating a signal to be transmitted; and a luminance change obtained by modulating a single signal at a time corresponding to a single frequency. A control method may include a transmission step of transmitting a signal to be transmitted by changing the luminance of the light emitter so as to include only the pattern.

For example, when a pattern of luminance change obtained by modulating a plurality of signals is included in a time corresponding to a single frequency, the waveform of the luminance change over time becomes complicated and it is difficult to receive it appropriately. However, by performing control so that only a luminance change pattern obtained by modulating a single signal is included at a time corresponding to a single frequency, reception can be performed more appropriately.

According to one embodiment of the present invention, the determining step is configured such that the number of transmissions for transmitting one signal among a plurality of different signals is different from the number of transmissions for transmitting other signals within a predetermined time. In addition, the number of transmissions may be determined.

The flickering at the time of transmission can be prevented because the number of transmissions for transmitting one signal is different from the number of transmissions for transmitting other signals.

According to one embodiment of the present invention, the determining step may increase the number of transmissions of a signal corresponding to a high frequency within a predetermined time, compared to the number of transmissions of other signals.

When performing frequency conversion on the receiving side, a signal corresponding to a high frequency is reduced in luminance, but by increasing the number of transmissions, it is possible to increase the luminance value when performing frequency conversion.

According to one embodiment of the present invention, the luminance change pattern may be a pattern in which the waveform of the luminance change over time is any one of a rectangular wave, a triangular wave, and a sawtooth wave.

By using a rectangular wave or the like, it becomes possible to receive more appropriately.

According to one embodiment of the present invention, when increasing the value of the average luminance of the illuminant, the time during which the luminance of the illuminant is greater than a predetermined value at a time corresponding to a single frequency, You may lengthen with respect to the case where the value of the average luminance of the said light-emitting body is made small.

It is possible to transmit a signal and adjust the average luminance of the light emitter by adjusting the time during which the luminance of the light emitter becomes larger than a predetermined value at a time corresponding to a single frequency. For example, when the light emitter is used as illumination, it is possible to transmit a signal while making the overall brightness darker or brighter.

The receiver can set the exposure time to a predetermined value by using an API for setting the exposure time (which stands for application programming interface and indicates a means for using the function of the OS). The visible light signal can be received stably. Further, the receiver can set the sensitivity to a predetermined value by using an API for setting the sensitivity, and stably receives a visible light signal even when the brightness of the transmission signal is dark or bright. be able to.

(Embodiment 8)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

Here, the EX zoom will be described.

FIG. 55 is a diagram for explaining the EX zoom.

The zoom, that is, the method of obtaining a large image, includes an optical zoom that adjusts the focal length of the lens to change the size of the image captured on the image sensor, and a digital image that interpolates the image captured on the image sensor. There are a digital zoom for obtaining a zoom lens and an EX zoom for obtaining a large image by changing a plurality of image sensors used for imaging. The EX zoom can be used when the number of image sensors included in the image sensor is larger than the resolution of the captured image.

For example, in the image sensor 10080a shown in FIG. 55, 32 × 24 image sensors are arranged in a matrix. That is, 32 image sensors are arranged horizontally and 24 elements are arranged vertically. When an image having a horizontal 16 × vertical 12 resolution is obtained by imaging by the image sensor 10080a, as shown in FIG. 55A, among the 32 × 24 imaging elements included in the image sensor 10080a, the image sensor Only 16 × 12 image sensors (for example, image sensors indicated by black squares in the image sensor 1080a in FIG. 55A) arranged uniformly distributed throughout 10080a are used for imaging. That is, only an odd-numbered or even-numbered image sensor is used for imaging among a plurality of image sensors arranged in the vertical direction and the horizontal direction. As a result, an image 10080b having a desired resolution is obtained. In FIG. 55, a subject appears on the image sensor 1008a, in order to make it easy to understand the correspondence between each imaging element and an image obtained by imaging.

The receiver including the image sensor 10080a captures a wide range, and searches for a transmitter or receives information from many transmitters in the image sensor 10080a. An image is picked up using only a part of the image pickup elements arranged in a distributed manner.

When the receiver performs the EX zoom, as shown in FIG. 55 (b), the image sensor 10080a has a part of the image pickup elements arranged locally densely (for example, FIG. 55 (b)). Only 16 × 12 image sensors indicated by black squares in the image sensor 1080a in FIG. As a result, a part of the image 10080b corresponding to a part of the imaging elements is zoomed, and an image 10080d is obtained. By such an EX zoom, it is possible to receive a visible light signal for a long time by capturing a large image of the transmitter, the reception speed is improved, and a visible light signal can be received from a distance.

In digital zoom, the number of exposure lines that receive visible light signals cannot be increased, and the reception time of visible light signals does not increase. Therefore, it is better to use other zooms as much as possible. The optical zoom requires a physical movement time of the lens and the image sensor. However, since the EX zoom is performed only by electronic setting change, there is an advantage that the time required for the zoom is short. From this viewpoint, the priority order of each zoom is (1) EX zoom, (2) optical zoom, and (3) digital zoom. The receiver may select and use any one or a plurality of zooms according to the priority order and the necessity of the zoom magnification. In the imaging methods shown in FIGS. 55A and 55B, image noise can be suppressed by using an unused imaging device.

(Embodiment 9)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

In this embodiment, the exposure time is set for each exposure line or each image sensor.

56, 57, and 58 are diagrams illustrating an example of a signal reception method according to the ninth embodiment.

As shown in FIG. 56, in the image sensor 10010a which is an imaging unit provided in the receiver, an exposure time is set for each exposure line. That is, a long exposure time for normal imaging is set for a predetermined exposure line (white exposure line in FIG. 56), and a visible light imaging for other exposure lines (black exposure line in FIG. 56). A short exposure time is set. For example, a long exposure time and a short exposure time are alternately set for each exposure line arranged in the vertical direction. Thereby, when imaging the transmitter which transmits a visible light signal by a luminance change, normal imaging and visible light imaging (visible light communication) can be performed almost simultaneously. The two exposure times may be alternately set for each line, may be set for every several lines, or different exposure times may be set for the upper part and the lower part of the image sensor 10010a. By using two exposure times in this way, when data obtained by imaging a plurality of exposure lines set to the same exposure time are collected, a normal captured image 10010b and a bright line image showing a plurality of bright line patterns are obtained. A visible light captured image 10010c is obtained. In the normal captured image 10010b, since a portion that has not been captured with a long exposure time (that is, an image corresponding to a plurality of exposure lines set to a short exposure time) is missing, the preview image is obtained by interpolating that portion. 10010d can be displayed. Here, information obtained by visible light communication can be superimposed on the preview image 10010d. This information is information associated with a visible light signal obtained by decoding a plurality of bright line patterns included in the visible light captured image 10010c. The receiver stores the normal captured image 10010b or an image obtained by performing interpolation on the normal captured image 10010b as a captured image, and stores the received visible light signal or information associated with the visible light signal. As additional information, it can also be added to the stored captured image.

As shown in FIG. 57, an image sensor 10011a may be used instead of the image sensor 10010a. In the image sensor 1011a, the exposure time is set not for each exposure line but for each column (hereinafter, referred to as a vertical line) composed of a plurality of imaging elements arranged along a direction perpendicular to the exposure line. That is, a long exposure time for normal imaging is set for a predetermined vertical line (white vertical line in FIG. 57), and a visible light imaging for other vertical lines (black vertical line in FIG. 57). A short exposure time is set. In this case, in the image sensor 10011a, exposure is started at different timings for each exposure line, as in the image sensor 10010a. However, the exposure time of each image sensor included in the exposure line is different for each exposure line. . The receiver obtains a normal captured image 10011b and a visible light captured image 10011c by imaging with the image sensor 10011a. Further, the receiver generates and displays a preview image 10011d based on the normal captured image 10011b and information associated with the visible light signal obtained from the visible light captured image 10011c.

In this image sensor 10011a, unlike the image sensor 10010a, all exposure lines can be used for visible light imaging. As a result, since the visible light captured image 10011c obtained by the image sensor 10011a includes more bright lines than the visible light captured image 10010c, the reception accuracy of the visible light signal can be increased.

As shown in FIG. 58, an image sensor 10012a may be used instead of the image sensor 10010a. In the image sensor 10012a, the exposure time is set for each image sensor so that the same exposure time is not set continuously for each image sensor along the horizontal direction and the vertical direction. That is, the exposure time for each image sensor is such that a plurality of image sensors with a long exposure time set and a plurality of image sensors with a short exposure time are distributed like a grid or checkered pattern. Is set. Also in this case, similarly to the image sensor 10010a, the exposure is started at different timings for each exposure line, but the exposure time of each image sensor included in the exposure line is different for each exposure line. The receiver obtains a normal captured image 10012b and a visible light captured image 10012c by imaging with the image sensor 10012a. Further, the receiver generates and displays a preview image 10012d based on the normal captured image 10012b and information associated with the visible light signal obtained from the visible light captured image 10012c.

Since the normal captured image 10012b obtained by the image sensor 10012a has data of a plurality of imaging elements arranged in a grid pattern or uniformly, interpolation or more accurately than the normal captured image 10010b and the normal captured image 10011b is performed. You can resize. The visible light captured image 10012c is generated by imaging using all exposure lines of the image sensor 10012a. That is, in the image sensor 10012a, unlike the image sensor 10010a, all exposure lines can be used for visible light imaging. As a result, the visible light captured image 10012c obtained by the image sensor 10012a includes a larger number of bright lines than the visible light captured image 10010c, as in the visible light captured image 10011c. Can be done with precision.

Here, the interlaced display of the preview image will be described.

FIG. 59 is a diagram illustrating an example of a receiver screen display method according to the ninth embodiment.

The receiver including the image sensor 10010a shown in FIG. 56 is set to an exposure time set for an odd-numbered exposure line (hereinafter referred to as an odd-numbered line) and an even-numbered exposure line (hereinafter referred to as an even-numbered line). The exposure time is changed every predetermined time. For example, as shown in FIG. 59, at time t1, the receiver sets a long exposure time for each image sensor on the odd lines and sets a short exposure time on each image sensor on the even lines. Imaging is performed using the set exposure time. Further, at time t2, the receiver sets a short exposure time for each image sensor of the odd line, sets a long exposure time for each image sensor of the even line, and sets the set exposure time. The used imaging is performed. Then, at time t3, the receiver performs imaging using each exposure time set similarly to time t1, and uses each exposure time set similarly to time t2 at time t4. Take an image.

Further, at time t1, the receiver captures an image obtained from each of a plurality of odd lines by imaging (hereinafter referred to as an odd line image) and an image obtained from each of a plurality of even lines by imaging (hereinafter referred to as an even line image). Image1 including the above. At this time, since the exposure time is short in each of the plurality of even lines, the subject is not clearly displayed in each of the even line images. Therefore, the receiver generates a plurality of interpolated line images by interpolating pixel values for the plurality of odd line images. Then, the receiver displays a preview image including a plurality of interpolation line images instead of the plurality of even line images. That is, odd-numbered line images and interpolated line images are alternately arranged in the preview image.

The receiver acquires Image2 including a plurality of odd line images and even line images by imaging at time t2. At this time, since the exposure time is short in each of the plurality of odd lines, the subject is not clearly displayed in each of the odd line images. Therefore, the receiver displays a preview image including the odd line image of Image1 instead of the odd line image of Image2. That is, the odd line image of Image1 and the even line image of Image2 are alternately arranged in the preview image.

Further, at time t3, the receiver acquires Image3 including a plurality of odd line images and even line images by imaging. At this time, similarly to the time t1, since the exposure time is short in each of the plurality of even lines, the subject is not clearly displayed in each of the even line images. Therefore, the receiver displays a preview image including an even line image of Image2 instead of an even line image of Image3. That is, in the preview image, the even line image of Image2 and the odd line image of Image3 are alternately arranged. Then, at time t4, the receiver acquires Image4 including a plurality of odd line images and even line images by imaging. At this time, similarly to the time t2, since the exposure time is short in each of the plurality of odd lines, the subject is not clearly displayed in each of the odd line images. Therefore, the receiver displays a preview image including the odd line image of Image3 instead of the odd line image of Image4. That is, the odd line image of Image3 and the even line image of Image4 are alternately arranged in the preview image.

As described above, the receiver performs so-called interlaced display, in which an image including even line images and odd line images obtained at different timings is displayed.

Such a receiver can display a fine preview image while performing visible light imaging. Note that the plurality of image sensors set with the same exposure time may be a plurality of image sensors arranged along the horizontal direction of the exposure line as in the image sensor 10010a, or the exposure line as in the image sensor 10011a. A plurality of image sensors arranged along a direction perpendicular to the image sensor may be used, or a plurality of image sensors arranged according to a checkered pattern like the image sensor 10012a. Further, the receiver may save the preview image as imaging data.

Next, the spatial ratio between normal imaging and visible light imaging will be described.

FIG. 60 is a diagram illustrating an example of a signal reception method according to the ninth embodiment.

In the image sensor 10014b provided in the receiver, a long exposure time or a short exposure time is set for each exposure line as in the image sensor 10010a described above. In this image sensor 10014b, the ratio of the number of image sensors for which a long exposure time is set to the number of image sensors for which a short exposure time is set is 1: 1. This ratio is a ratio between normal imaging and visible light imaging, and is hereinafter referred to as a spatial ratio.

However, in this embodiment, the space ratio does not have to be 1: 1. For example, the receiver may include an image sensor 10014a. In this image sensor 10014a, the number of image sensors with a short exposure time is larger than that of images with a long exposure time, and the spatial ratio is 1: N (N> 1). The receiver may include an image sensor 10014c. In this image sensor 10014c, the image sensor with a short exposure time is smaller than the image sensor with a long exposure time, and the spatial ratio is N (N> 1): 1. In addition, instead of the image sensors 10014a to 10014c, the receiver sets the exposure time for each of the above-described vertical lines, and the image sensors 10015a to 10015c have a spatial ratio of 1: N, 1: 1, or N: 1, respectively. Any of these may be provided.

In such image sensors 10014a and 10015a, since there are many imaging elements with a short exposure time, the reception accuracy or reception speed of a visible light signal can be increased. On the other hand, since the image sensors 10014c and 10015c have many image sensors with a long exposure time, a fine preview image can be displayed.

Further, the receiver may perform interlaced display as shown in FIG. 59 using the image sensors 10014a, 10014c, 10015a, and 10015c.

Next, the time ratio between normal imaging and visible light imaging will be described.

FIG. 61 is a diagram illustrating an example of a signal reception method according to the ninth embodiment.

The receiver may switch the imaging mode between the normal imaging mode and the visible light imaging mode for each frame, as shown in FIG. The normal imaging mode is an imaging mode in which a long exposure time for normal imaging is set for all imaging elements of the image sensor of the receiver. The visible light imaging mode is an imaging mode in which a short exposure time for visible light imaging is set for all imaging elements of the image sensor of the receiver. In this way, by switching between long and short exposure times, a preview image can be displayed by imaging with a long exposure time while receiving a visible light signal by imaging with a short exposure time.

Note that when the long exposure time is determined by automatic exposure, the receiver ignores the image obtained by imaging with a short exposure time, and only determines the brightness of the image obtained by imaging with a long exposure time. Automatic exposure may be performed with reference. Thereby, a long exposure time can be determined as an appropriate time.

Also, as shown in FIG. 61 (b), the receiver may switch the imaging mode between the normal imaging mode and the visible light imaging mode for each set of a plurality of frames. When it takes time to switch the exposure time or when it takes time to stabilize the exposure time, as shown in FIG. 61 (b), by changing the exposure time for each set of a plurality of frames, Visible light imaging (reception of visible light signals) and normal imaging can be made compatible. Further, since the number of exposure time switching is reduced as the number of frames included in the set is increased, power consumption and heat generation in the receiver can be suppressed.

Here, the number of at least one frame continuously generated by imaging with a long exposure time in the normal imaging mode and the at least one frame continuously generated by imaging with a short exposure time in the visible light imaging mode The ratio (hereinafter referred to as the time ratio) with the number of s is not necessarily 1: 1. That is, in the cases shown in FIGS. 61A and 61B, the time ratio is 1: 1, but the time ratio may not be 1: 1.

For example, the receiver may increase the number of frames in the visible light imaging mode as compared with the frame in the normal imaging mode, as illustrated in FIG. Thereby, the receiving speed of the visible light signal can be increased. If the frame rate of the preview image is equal to or higher than a predetermined rate, the difference in the preview image depending on the frame rate is not recognized by human eyes. When the imaging frame rate is sufficiently high, for example, when the frame rate is 120 fps, the receiver sets the visible light imaging mode for three consecutive frames, and then the visible light imaging for one frame. Set the mode. Thereby, the receiver can receive a visible light signal at high speed while displaying a preview image at a frame rate of 30 fps, which is sufficiently higher than the above-described predetermined rate. Further, since the number of times of switching is reduced, the effect described with reference to FIG.

Further, as shown in FIG. 61 (d), the receiver may increase the number of frames in the normal imaging mode than that in the visible light imaging mode. Thus, the preview image can be smoothly displayed by increasing the number of frames in the normal imaging mode, that is, the frames obtained by imaging with a long exposure time. In addition, since the number of times of receiving the visible light signal is reduced, there is a power saving effect. Further, since the number of times of switching is reduced, the effect described with reference to FIG.

As shown in (e) of FIG. 61, the receiver first switches the imaging mode for each frame as in the case of (a) of FIG. 61, and then receives a visible light signal. When completed, the number of frames in the normal imaging mode may be increased as in the case shown in FIG. Thereby, after the reception of the visible light signal is completed, the search for a new visible light signal can be continued while the preview image is displayed smoothly. Further, since the number of times of switching is reduced, the effect described with reference to FIG.

FIG. 62 is a flowchart illustrating an example of a signal reception method according to the ninth embodiment.

The receiver starts visible light reception, which is a process of receiving a visible light signal (step S10017a), and sets the exposure time length setting ratio to a value designated by the user (step S10017b). The exposure time length short / high setting ratio is at least one of the above-described space ratio and time ratio. The user may specify only the space ratio, only the time ratio, or both the space ratio and the time ratio, or the receiver may automatically set regardless of the user's specification.

Next, the receiver determines whether or not the reception performance is equal to or less than a predetermined value (step S10017c). If it is determined that the value is equal to or less than the predetermined value (Y in step S10017c), the receiver sets a high ratio of the short exposure time (step S10017d). Thereby, reception performance can be improved. Note that the ratio of the short exposure time is the ratio of the number of image sensors set with a short exposure time to the number of image sensors set with a long exposure time in the case of a spatial ratio. It is a ratio of the number of frames generated continuously in the visible light imaging mode to the number of frames generated continuously in the mode.

Next, the receiver receives at least a part of the visible light signal and determines whether or not a priority is set for at least a part of the received visible light signal (hereinafter referred to as a received signal) ( Step S10017e). When priority is set, an identifier indicating the priority is included in the received signal. When the receiver determines that the priority is set (Y in step S10017e), the receiver sets the exposure time length / short ratio according to the priority (step S10017f). That is, if the priority is high, the receiver sets the ratio of the short exposure time high. For example, an emergency light configured as a transmitter emits an identifier indicating a high priority when the luminance changes. In this case, the receiver can increase the reception speed by increasing the ratio of the short exposure time, and can promptly display the evacuation route and the like.

Next, the receiver determines whether or not reception of all visible light signals has been completed (step S10017g). Here, when it is determined that it has not been completed (N in step S10017g), the receiver repeatedly executes the processing from step S10017c. On the other hand, when it is determined that the process is completed (Y in step S10017g), the receiver sets the ratio of the long exposure time to a high value and shifts to the power saving mode (step S10017h). Note that the ratio of the long exposure time is the ratio of the number of image sensors set with a long exposure time to the number of image sensors set with a short exposure time in the case of a spatial ratio. This is the ratio of the number of frames generated continuously in the normal imaging mode to the number of frames generated continuously in the imaging mode. As a result, the preview image can be displayed smoothly without receiving unnecessary visible light.

Next, the receiver determines whether another visible light signal has been found (step S10017i). Here, when it is determined that it has been found (Y in step S10017i), the receiver repeatedly executes the processing from step S10017b.

Next, simultaneous execution of visible light imaging and normal imaging will be described.

FIG. 63 is a diagram illustrating an example of a signal reception method according to the ninth embodiment.

The receiver may set two or more exposure times for the image sensor. That is, as shown in FIG. 63A, each of the exposure lines included in the image sensor is continuously exposed for the longest exposure time among two or more set exposure times. For each exposure line, the receiver reads out the imaging data obtained by exposure of the exposure line when the above-described two or more set exposure times have elapsed. Here, the receiver does not reset the read image data until the longest exposure time has elapsed. Therefore, the receiver can obtain image data for a plurality of exposure times only by exposure with the longest exposure time by recording the accumulated value of the read image data. Note that the image sensor may or may not record the cumulative value of the imaging data. When the image sensor is not used, the components of the receiver that read data from the image sensor perform accumulation calculation, that is, recording the accumulated value of the imaging data.

For example, when two exposure times are set, as shown in (a) of FIG. 63, the receiver captures visible light imaging data including a visible light signal generated by exposure with a short exposure time. Read out, followed by normal imaging data generated by exposure with a long exposure time.

Thereby, visible light imaging that is imaging for receiving a visible light signal and normal imaging can be performed simultaneously, and normal imaging can be performed while receiving a visible light signal. Further, by using data of a plurality of exposure times, a signal frequency higher than the sampling theorem can be recognized, and a high-frequency signal or a high-density modulation signal can be received.

Furthermore, when outputting the imaging data, the receiver outputs a data string including the imaging data as an imaging data body, as shown in FIG. 63 (b). That is, the receiver has an imaging mode identifier indicating an imaging mode (visible light imaging or normal imaging), an imaging element identifier for specifying an imaging element or an exposure line to which the imaging element belongs, and an exposure number of an imaging data body. By adding additional information including an imaging data number indicating whether it is time-based imaging data and an imaging data length indicating the size of the imaging data body to the imaging data body, the above-described data string is generated and output. . In the readout method of imaging data described with reference to FIG. 63A, the respective imaging data is not always output in the order of exposure lines. Therefore, by adding the additional information shown in (b) of FIG. 63, it is possible to specify which exposure line the imaging data is.

FIG. 64 is a flowchart showing processing of the reception program in the ninth embodiment.

This reception program is a program that causes a computer provided in the receiver to execute the processes shown in FIGS. 56 to 63, for example.

That is, this reception program is a reception program for receiving information from a light-emitting body that changes in luminance. Specifically, this reception program causes the computer to execute step SA31, step SA32, and step SA33. In step SA31, a first exposure time is set for some of the K image sensors (K is an integer of 4 or more) included in the image sensor, and the K image sensors are set. A second exposure time shorter than the first exposure time is set for the remaining plurality of image sensors. In step SA32, the image sensor captures the subject, which is a light-emitting body that changes in luminance, with the set first and second exposure times, and outputs from the plurality of image sensors having the first exposure time set. And acquiring bright lines corresponding to the plurality of exposure lines included in the image sensor, the images corresponding to the outputs from the plurality of imaging elements set with the second exposure time. A bright line image that is an image to be included is acquired. In step SA33, information is acquired by decoding a plurality of bright line patterns included in the acquired bright line image.

Thereby, since imaging is performed by the plurality of imaging elements in which the first exposure time is set and the plurality of imaging elements in which the second exposure time is set, a normal image can be obtained by one imaging by the image sensor. And bright line images can be acquired. That is, it is possible to simultaneously capture a normal image and acquire information by visible light communication.

In the exposure time setting step SA31, the first exposure time is set for a part of a plurality of image sensor rows in the L (L is an integer of 4 or more) image sensor rows included in the image sensor. Then, the second exposure time is set for the remaining plurality of image sensor rows in the L image sensor rows. Here, each of the L image sensor rows is composed of a plurality of image sensors arranged in a row included in the image sensor.

Accordingly, it is possible to set the exposure time for each image pickup device array which is a large unit without individually setting the exposure time for each image pickup device which is a small unit, and to reduce the processing load. .

For example, each of the L image sensor rows is an exposure line included in the image sensor as shown in FIG. Alternatively, as shown in FIG. 57, each of the L image pickup device arrays includes a plurality of image pickup devices arranged along a direction perpendicular to an exposure line included in the image sensor.

Further, as shown in FIG. 59, in the exposure time setting step SA31, the same exposure time is used for each of the odd-numbered image sensor rows of the L image sensor rows included in the image sensor. One of the first exposure time and the second exposure time is set, and the first exposure time and the second exposure time are the same exposure time for each of even-numbered image sensor rows of the L image sensor rows. You may set the other of time. When the exposure time setting step SA31, the image acquisition step SA32, and the information acquisition step SA33 are repeated, the repeated exposure time setting step SA31 is set for each of the odd-numbered image sensor rows in the previous exposure time setting step SA31. The exposure time that has been set may be interchanged with the exposure time that has been set for each even-numbered imaging element array.

Thus, each time a normal image is acquired, the plurality of image sensor rows used for the acquisition can be switched between an odd number of image sensor rows and an even number of image sensor rows. As a result, each of the sequentially acquired normal images can be displayed by interlace. Further, by complementing two consecutively acquired normal images with each other, a new normal image including an image by an odd-numbered plurality of imaging element arrays and an image by an even-numbered plurality of imaging element arrays is generated. be able to.

As shown in FIG. 60, in the exposure time setting step SA31, the first exposure time is set when the setting mode is switched between the normal priority mode and the visible light priority mode and can be switched to the normal priority mode. The number of image sensors may be larger than the number of image sensors for which the second exposure time is set. In addition, when the mode is switched to the visible light priority mode, the number of image sensors for which the first exposure time is set may be smaller than the number of image sensors for which the second exposure time is set.

As a result, when the setting mode is switched to the normal priority mode, the image quality of the normal image can be improved. When the setting mode is switched to the visible light priority mode, the reception efficiency of information from the light emitter is improved. can do.

As shown in FIG. 58, in the exposure time setting step SA31, a plurality of image sensors for which the first exposure time is set and a plurality of image sensors for which the second exposure time is set are in a checkered pattern ( The exposure time of the image sensor may be set for each image sensor included in the image sensor so as to be distributed like Checkered pattern.

As a result, the plurality of image pickup devices for which the first exposure time is set and the plurality of image pickup devices for which the second exposure time is set are uniformly distributed. Not normal images and bright line images can be acquired.

FIG. 65 is a block diagram of a receiving apparatus according to the ninth embodiment.

The receiving device A30 is the above-described receiver that executes the processing shown in FIGS. 56 to 63, for example.

That is, the receiving device A30 is a receiving device that receives information from a light-emitting body that changes in luminance, and includes a multiple exposure time setting unit A31, an imaging unit A32, and a decoding unit A33. The multiple exposure time setting unit A31 sets the first exposure time for some of the plurality of image pickup elements (K is an integer of 4 or more) included in the image sensor, and K pieces. A second exposure time shorter than the first exposure time is set for the remaining plurality of image sensors. The imaging unit A32 causes the image sensor to capture an image of a subject, which is a light-emitting body that changes in luminance, with the set first and second exposure times, so that a plurality of imaging elements with the first exposure time are set. A bright image corresponding to each of the plurality of exposure lines included in the image sensor, which is an image corresponding to the output from the plurality of imaging elements for which the second exposure time is set, while acquiring a normal image according to the output A bright line image that is an image including The decoding unit A33 acquires information by decoding a plurality of bright line patterns included in the acquired bright line image. Such a receiving apparatus A30 can achieve the same effects as the above-described receiving program.

Next, display of contents related to the received visible light signal will be described.

66 and 67 are diagrams showing an example of the display of the receiver when a visible light signal is received.

66 (a), when the receiver captures an image of the transmitter 10020d, the receiver displays an image 10020a on which the transmitter 10020d is projected. Further, the receiver generates and displays an image 10020b by superimposing the object 10020e on the image 10020a. The object 10020e is an image indicating that the image of the transmitter 10020d is present and that a visible light signal is received from the transmitter 10020d. The object 10020e may be an image that varies depending on the reception state of the visible light signal (the state of reception, the state of searching for a transmitter, the degree of progress of reception, the reception speed, or the error rate). For example, the receiver changes the color of the object 1020e, the thickness of the line, the type of line (single line, double line, dotted line, or the like), or the interval between dotted lines. Thereby, a user can be made to recognize a receiving state. Next, the receiver generates and displays an image 10020c by superimposing an image indicating the content of the acquired data on the image 10020a as an acquired data image 10020f. The acquired data is data associated with the received visible light signal or the ID indicated by the received visible light signal.

When the receiver displays the acquired data image 10020f, as shown in FIG. 66A, the receiver displays the acquired data image 10020f like a balloon from the transmitter 10020d or near the transmitter 10020d. The acquired data image 10020f is displayed. In addition, as illustrated in FIG. 66B, the receiver may display the acquired data image 10020f so that the acquired data image 10020f gradually approaches the receiver side from the transmitter 10020d. Thereby, the user can recognize which transmitter the received data image 10020f is based on the visible light signal received from. In addition, as shown in FIG. 67, the receiver may display the acquired data image 10020f so that the acquired data image 10020f gradually emerges from the end of the receiver display. This makes it possible for the user to easily recognize that the visible light signal has been acquired at that time.

Next, AR (Augmented Reality) will be described.

FIG. 68 is a diagram showing an example of display of the acquired data image 10020f.

When the transmitter image moves in the display, the receiver moves the acquired data image 10020f in accordance with the movement of the transmitter image. This allows the user to recognize that the acquired data image 10020f corresponds to the transmitter. The receiver may display the acquired data image 10020f in association with another image instead of the image of the transmitter. Thereby, AR display can be performed.

Next, saving and discarding of acquired data will be described.

FIG. 69 is a diagram illustrating an example of an operation when saving or discarding acquired data.

For example, as shown in FIG. 69A, when the user performs a swipe down on the acquired data image 10020f, the receiver stores the acquired data indicated by the acquired data image 10020f. . The receiver places an acquired data image 10020f indicating the stored acquired data at the end of the acquired data image indicating one or more other already stored acquired data. This allows the user to recognize that the acquisition data indicated by the acquisition data image 10020f is the acquisition data stored last. For example, as shown in FIG. 69A, the receiver arranges the acquired data image 10020f in the forefront among the plurality of acquired data images.

Further, as shown in FIG. 69 (b), when the user swipes the acquired data image 10020f to the right side, the receiver discards the acquired data indicated by the acquired data image 10020f. Alternatively, the receiver may discard the acquired data indicated by the acquired data image 10020f when the image of the transmitter is framed out of the display by the user moving the receiver. Note that the same effect as described above can be obtained regardless of whether the swipe direction is up, down, left, or right. The receiver may display the swipe direction corresponding to the save or discard. As a result, the user can recognize that the data can be saved or discarded by the operation.

Next, browsing of acquired data will be described.

FIG. 70 is a diagram showing a display example when browsing acquired data.

As shown in (a) of FIG. 70, the receiver displays the acquired data images of a plurality of stored acquired data in a small manner overlapping the lower end of the display. At this time, when the user taps a part of the acquired data image displayed, the receiver displays each of the plurality of acquired data images in a large size as shown in FIG. Thereby, only when it is necessary to view each piece of acquired data, those acquired data images are displayed in a large size, and when not necessary, the display can be used effectively for other displays.

When the user taps the acquired data image to be displayed in the state shown in (b) of FIG. 70, the receiver displays the tapped acquired data image in a larger size as shown in (c) of FIG. A lot of information is displayed in the acquired data image. When the user taps the back surface display button 10024a, the receiver displays the back surface of the acquired data image, and displays other data related to the acquired data.

Next, we will explain how to turn off image stabilization when estimating the accident location.

The receiver disables camera shake correction (off), or converts the captured image according to the correction direction and correction amount of camera shake correction, thereby acquiring the correct imaging direction and accurately performing self-position estimation. Can be done. The captured image is an image obtained by imaging by the imaging unit of the receiver. Self-position estimation means that the receiver estimates its own position. In the self-position estimation, specifically, the receiver specifies the position of the transmitter based on the received visible light signal, and receives based on the size, position, or shape of the transmitter reflected in the captured image. Identify the relative positional relationship between the transmitter and transmitter. Then, the receiver estimates the position of the receiver based on the position of the transmitter and the relative positional relationship between the receiver and the transmitter.

Also, at the time of partial readout where imaging is performed using only a part of the exposure lines shown in FIG. 56, that is, when imaging shown in FIG. 56 is performed, the transmitter is out of frame with a slight blurring of the receiver. End up. In such a case, the receiver can continuously receive the signal by enabling the camera shake correction.

Next, self-position estimation using an asymmetrical light emitting unit will be described.

FIG. 71 is a diagram illustrating an example of a transmitter in the ninth embodiment.

The above-described transmitter includes a light emitting unit, and transmits a visible light signal by changing the luminance of the light emitting unit. In the above-described self-position estimation, the receiver determines the relative positional relationship between the receiver and the transmitter based on the shape of the transmitter (specifically, the light emitting unit) in the captured image. Find the relative angle to the transmitter. Here, for example, as shown in FIG. 71, when the transmitter includes a light-emitting unit 10090a having a rotationally symmetric shape, as described above, based on the shape of the transmitter in the captured image, The relative angle with the receiver cannot be determined accurately. Therefore, it is desirable that the transmitter includes a light emitting unit having a shape that is not rotationally symmetric. Thereby, the receiver can obtain | require correctly the above-mentioned relative angle. That is, since the error of the measurement result is large in the azimuth sensor for acquiring the angle, the receiver can perform accurate self-position estimation by using the relative angle obtained by the above method.

Here, as shown in FIG. 71, the transmitter may include a light emitting unit 10090b that is not in a completely rotationally symmetric shape. The shape of the light emitting unit 10090b is symmetric with respect to 90 ° rotation, but is not completely rotationally symmetric. In this case, the receiver obtains a rough angle with the azimuth sensor, and further uses the shape of the transmitter in the captured image to uniquely limit the relative angle between the receiver and the transmitter. And accurate self-position estimation can be performed.

Further, the transmitter may include a light emitting unit 10090c shown in FIG. The shape of the light emitting unit 10090c is basically a rotationally symmetric shape. However, since a light guide plate or the like is provided in a part of the light emitting unit 10090c, the shape of the light emitting unit 10090c is not rotationally symmetric.

Further, the transmitter may include a light emitting unit 10090d shown in FIG. Each of the light emitting units 10090d includes rotationally symmetric illumination. However, the overall shape of the light emitting unit 10090d configured by combining them is not a rotationally symmetric shape. Therefore, the receiver can perform accurate self-position estimation by imaging the transmitter. In addition, it is not necessary for all illumination included in the light emitting unit 10090d to be illumination for visible light communication that changes in luminance in order to transmit a visible light signal, and only some illumination is illumination for visible light communication. May be.

Further, the transmitter may include a light emitting unit 10090e and an object 10090f shown in FIG. Here, the object 10090f is an object (for example, a fire alarm or a pipe) configured so that the positional relationship with the light emitting unit 10090e does not change. Since the shape of the combination of the light emitting unit 10090e and the object 10090f is not a rotationally symmetric shape, the receiver can accurately perform self-position estimation by imaging the light emitting unit 10090e and the object 10090f.

Next, the time series processing for self-position estimation will be described.

Every time the receiver takes an image, the receiver can perform self-position estimation from the position and shape of the transmitter in the captured image. As a result, the receiver can estimate the moving direction and distance of the receiver being imaged. Further, the receiver can perform more accurate self-position estimation by performing triangulation using a plurality of frames or images. By integrating estimation results using a plurality of images and estimation results using a plurality of different combinations of images, the receiver can perform self-position estimation more accurately. At this time, the receiver can perform self-position estimation more accurately by emphasizing and summing up the results estimated from recent captured images.

Next, optical black skipping will be described.

FIG. 72 is a diagram illustrating an example of a reception method according to the ninth embodiment. The horizontal axis of the graph shown in FIG. 72 indicates time, and the vertical axis indicates the position of each exposure line in the image sensor. Furthermore, a solid line arrow in the graph indicates a time (exposure timing) at which exposure of each exposure line in the image sensor is started.

During normal imaging, the receiver reads the horizontal optical black signal in the image sensor as shown in FIG. 72A, but skips the horizontal optical black signal as shown in FIG. 72B. May be. Thereby, a continuous visible light signal can be received.

Horizontal optical black is optical black in the horizontal direction on the exposure line. The vertical optical black is a portion of the optical black other than the horizontal optical black.

Since the receiver adjusts the black level according to the signal read from the optical black, the black level can be adjusted using the optical black at the start of the visible light imaging similarly to the normal imaging. When vertical optical black can be used, the receiver can perform continuous reception and black level adjustment by performing black level adjustment using only vertical optical black. When visible light imaging continues, the receiver may adjust the black level using horizontal optical black every predetermined time. In the case where the normal imaging and the visible light imaging are alternately performed, the receiver skips the horizontal optical black signal when continuously performing the visible light imaging, and reads the horizontal optical black signal otherwise. Then, the receiver can adjust the black level while continuously receiving the visible light signal by adjusting the black level based on the read signal. The receiver may adjust the black level by setting the darkest part of the visible light captured image as black.

In this way, continuous reception of visible light signals is possible by using only vertical optical black as the optical black from which signals are read out. In addition, by providing a mode for skipping horizontal optical black signals, black level adjustment can be performed during normal imaging, and continuous communication can be performed as necessary during visible light imaging. In addition, by skipping the horizontal optical black signal, the difference in timing of starting exposure between exposure lines is increased, so that a visible light signal from a transmitter that is only small can be received.

Next, an identifier indicating the type of transmitter will be described.

The transmitter may add a transmitter identifier indicating the type of transmitter to the visible light signal for transmission. In this case, when the receiver receives the transmitter identifier, the receiver can perform a receiving operation according to the type of the transmitter. For example, in the case where the transmitter identifier indicates digital signage, the transmitter displays a content ID indicating which content is currently displayed in addition to the transmitter ID for performing individual identification of the transmitter as a visible light signal. As sending. The receiver can display information according to the content currently displayed by the transmitter by separately handling these IDs based on the transmitter identifier. For example, when the transmitter identifier indicates digital signage or an emergency light, the receiver can reduce reception errors by imaging with increased sensitivity.

(Embodiment 10)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

Here, a reception method for comparing the data part of the same address will be described.

FIG. 73 is a flowchart showing an example of the reception method in the present embodiment.

The receiver receives the packet (step S10101) and performs error correction (step S10102). Then, the receiver determines whether or not a packet having the same address as that of the received packet has already been received (step S10103). If it is determined that the data is received (Y in step S10103), the receiver compares the data. That is, the receiver determines whether the data parts are equal (step S10104). Here, when it is determined that they are not equal (N in step S10104), the receiver further determines whether the difference in the plurality of data parts is equal to or greater than a predetermined number, specifically, the number of different bits or the luminance. It is determined whether or not the number of slots having different states is equal to or greater than a predetermined number (step S10105). If it is determined that the number is equal to or greater than the predetermined number (N in step S10105), the receiver discards the packet that has already been received (step S10106). Thereby, when a packet is started to be received from another transmitter, interference with a packet received from a previous transmitter can be avoided. On the other hand, if it is determined that the number is not equal to or greater than the predetermined number (N in step S10105), the receiver sets the data of the data part with the most packets having the same data part as the data of the address (step S10107). Alternatively, the receiver takes the most equal bit as the value of that bit at that address. Alternatively, the receiver sets the luminance state having the largest number of equal luminance states as the luminance state of the slot at the address, and demodulates the data at the address.

Thus, in the present embodiment, the receiver first acquires a first packet including a data portion and an address portion from a plurality of bright line patterns. Next, the receiver has at least one second packet which is a packet including the same address part as the address part of the first packet among at least one packet already acquired before the first packet. It is determined whether or not there is a packet. Next, when the receiver determines that the at least one second packet exists, the receiver determines whether the data parts of the at least one second packet and the first packet are all equal. judge. If it is determined that the respective data parts are not all equal, the receiver, in each of the at least one second packet, out of the parts included in the data part of the second packet, the first packet It is determined whether the number of parts different from each part included in the data part is greater than or equal to a predetermined number. Here, when there is a second packet in which the number of different portions is determined to be greater than or equal to a predetermined number among the at least one second packet, the receiver includes the at least one second packet. Is discarded. On the other hand, if there is no second packet in which the number of different portions of the at least one second packet is determined to be greater than or equal to a predetermined number, the receiver may include the first packet and the at least one second packet. Among these packets, a plurality of packets having the largest number of packets having the same data part are specified. The receiver then decodes at least a part of the visible light identifier (ID) by decoding the data part included in each of the plurality of packets as a data part corresponding to the address part included in the first packet. get.

Thereby, when a plurality of packets having the same address part are received, even if the data part of those packets is different, the appropriate data part can be decoded, and at least a part of the visible light identifier is You can get it correctly. That is, a plurality of packets having the same address part transmitted from the same transmitter basically have the same data part. However, when the receiver switches the transmitter that is the transmission source of the packet, the receiver may receive a plurality of packets having different data parts even though they have the same address part. In such a case, in the present embodiment, as in step S10106 in FIG. 73, the already received packet (second packet) is discarded, and the data portion of the latest packet (first packet). Can be decoded as a correct data portion corresponding to the address portion. Furthermore, even when there is no transmitter switching as described above, the data portions of a plurality of packets having the same address portion may be slightly different depending on the transmission / reception state of the visible light signal. In such a case, in the present embodiment, an appropriate data part can be decoded by so-called majority decision as in step S10107 of FIG.

Here, a reception method for demodulating data in a data portion from a plurality of packets will be described.

FIG. 74 is a flowchart showing an example of the reception method in this embodiment.

First, the receiver receives the packet (step S10111) and performs error correction of the address part (step S10112). At this time, the receiver does not demodulate the data part and holds the pixel value obtained by imaging as it is. Then, the receiver determines whether or not there are a predetermined number or more of packets having the same address among the plurality of packets that have already been received (step S10113). If it is determined that the packet exists (Y in step S10113), the receiver performs demodulation processing by combining pixel values of portions corresponding to data portions of a plurality of packets having the same address (step S10114).

As described above, in the reception method according to the present embodiment, the first packet including the data portion and the address portion is acquired from the plurality of bright line patterns. Then, among at least one packet that has already been acquired before the first packet, there are a predetermined number or more of second packets that include the same address part as the address part of the first packet. It is determined whether or not. If it is determined that there are more than the predetermined number of second packets, the pixel values of the partial areas of the bright line image corresponding to the respective data portions of the second packet more than the predetermined number, The pixel values of a part of the bright line image corresponding to the data portion of the packet are matched. That is, the pixel values are added. By the addition, a composite pixel value is calculated, and at least a part of the visible light identifier (ID) is obtained by decoding the data portion including the composite pixel value.

Since the timing at which a plurality of packets are received is different, the pixel values in the data portion are values that reflect the luminance of the transmitter at slightly different times. Therefore, the portion to be demodulated as described above includes a larger amount of data (number of samples) than the data portion of a single packet. Thereby, a data part can be demodulated more correctly. Further, by increasing the number of samples, a signal modulated with a higher modulation frequency can be demodulated.

The data part and its error correction code part are modulated at a higher frequency than the header part, the address part, and the error correction code part of the address part. By the above demodulation method, since the data portion and the subsequent part can be demodulated even if modulated at a high modulation frequency, the transmission time of the whole packet can be shortened by this configuration, and from a smaller light source even from a longer distance. However, a visible light signal can be received faster.

Next, a reception method for receiving variable length address data will be described.

FIG. 75 is a flowchart showing an example of a reception method in the present embodiment.

The receiver receives the packet (step S10121), and determines whether or not a packet in which all the bits of the data part are 0 (hereinafter referred to as a 0 termination packet) is received (step S10122). Here, if it is determined that it has been received, that is, if it is determined that there is a zero-termination packet (Y in step S10122), the receiver determines whether or not all packets having addresses below the zero-termination packet address are available, that is, It is determined whether or not it has been received (step S10123). Note that the address is set to a value that increases in accordance with the order of transmission of each packet generated by dividing the transmitted data. If it is determined that the receiver is complete (Y in step S10123), the receiver determines that the address of the 0-termination packet is the last address of the packet transmitted from the transmitter. Then, the receiver restores the data by connecting the data of the packets of each address up to the 0 terminal packet (step S10124). Further, the receiver performs an error check on the restored data (step S10125). This makes it possible to send and receive variable-length address data even when the data to be transmitted is not divided into several parts, that is, when the address is not fixed length but variable length. More IDs can be transmitted and received with high efficiency.

Thus, in the present embodiment, the receiver acquires a plurality of packets each including a data portion and an address portion from a plurality of bright line patterns. Then, the receiver determines whether or not there is a 0-termination packet that is a packet in which all bits included in the data part indicate 0 among the plurality of acquired packets. If it is determined that there is a zero-termination packet, the receiver, among a plurality of packets, N packets (N is an integer equal to or greater than one) including an address part associated with the address part of the zero-termination packet. It is determined whether or not all related packets are present. Next, when the receiver determines that all the N related packets exist, the receiver acquires a visible light identifier (ID) by arranging and decoding the data portions of the N related packets. Here, the address part associated with the address part of the 0-termination packet is an address part that indicates an address that is smaller than the address shown in the address part of the 0-termination packet and is 0 or more.

Next, a reception method using an exposure time longer than the modulation frequency period will be described.

76 and 77 are diagrams for explaining a reception method in which the receiver according to the present embodiment uses an exposure time longer than the period of the modulation frequency (modulation period).

For example, as shown in (a) of FIG. 76, when the exposure time is set to a time equal to the modulation period, the visible light signal may not be received correctly. The modulation period is the time of one slot described above. That is, in such a case, there are few exposure lines (exposure lines indicated by black in FIG. 76) reflecting the luminance state of a certain slot. As a result, it is difficult to estimate the luminance of the transmitter when a lot of noise is accidentally included in the pixel values of these exposure lines.

On the other hand, for example, as shown in FIG. 76 (b), when the exposure time is set longer than the modulation period, the visible light signal can be correctly received. That is, in such a case, since there are many exposure lines reflecting the brightness of a certain slot, the brightness of the transmitter can be estimated from the pixel values of many exposure lines, and it is resistant to noise.

Also, if the exposure time is too long, the visible light signal cannot be received correctly.

For example, as shown in FIG. 77 (a), when the exposure time is equal to the modulation period, the luminance change (that is, the change in the pixel value of each exposure line) received by the receiver is used for transmission. Follow changes in brightness. However, as shown in FIG. 77 (b), when the exposure time is three times the modulation period, the luminance change received by the receiver can sufficiently follow the luminance change used for transmission. Can not. As shown in FIG. 77 (c), when the exposure time is 10 times the modulation period, the luminance change received by the receiver cannot follow the luminance change used for transmission at all. That is, the longer the exposure time, the higher the noise resistance because the luminance can be estimated from many exposure lines. However, the longer the exposure time, the lower the identification margin or the smaller the identification margin. With these balances, the noise resistance can be maximized by setting the exposure time to about 2 to 5 times the modulation period.

Next, the number of packet divisions will be described.

FIG. 78 is a diagram showing an efficient number of divisions with respect to the size of transmission data.

When the transmitter transmits data due to a change in luminance, if all data to be transmitted (transmission data) is included in one packet, the data size of the packet is large. However, if the transmission data is divided into a plurality of partial data and the partial data is included in each packet, the data size of each packet is reduced. Here, the receiver receives the packet by imaging. However, the larger the data size of the packet, the more difficult it is for the receiver to receive the packet by one imaging, and it is necessary to repeat imaging.

Therefore, as shown in FIGS. 78 (a) and 78 (b), it is desirable for the transmitter to increase the number of divisions of the transmission data as the data size of the transmission data increases. However, if the number of divisions is too large, the transmission data cannot be restored unless all of the partial data is received.

Therefore, as shown in FIG. 78A, the data size of the address (address size) is variable, and the data size of the transmission data is 2-16 bits, 16-24 bits, 24-64 bits, 66- In the case of 78 bits, 78-128 bits, 128 bits or more, send data into 1-2 pieces, 2-4 pieces, 4 pieces, 4-6 pieces, 6-8 pieces, 7 pieces or more, respectively. Is divided, transmission data can be efficiently transmitted by a visible light signal. Further, as shown in FIG. 78 (b), the data size of the address (address size) is fixed to 4 bits, and the data size of the transmission data is 2-8 bits, 8-16 bits, 16-30 bits, For 30-64 bits, 66-80 bits, 80-96 bits, 96-132 bits, 132 bits or more, 1-2, 2-3, 2-4, 4-5, When the transmission data is divided into 4-7 pieces, 6 pieces, 6-8 pieces, 7 pieces or more partial data, the transmission data can be efficiently transmitted by a visible light signal.

In addition, the transmitter sequentially changes the luminance based on each packet including each of a plurality of partial data. For example, the transmitter changes the luminance based on the packets in the order of the addresses of the packets. Further, the transmitter may perform the luminance change based on the plurality of partial data again in an order different from the address order. Thereby, each partial data can be reliably received by the receiver.

Next, the method for setting the notification operation by the receiver will be described.

FIG. 79A is a diagram showing an example of a setting method in the present embodiment.

First, the receiver acquires a notification operation identifier for identifying the notification operation and a priority of the notification operation identifier (specifically, an identifier indicating the priority) from a server near the receiver. (Step S10131). Here, in the notification operation, when each packet including each of a plurality of partial data is transmitted due to a change in luminance and received by the receiver, reception is performed to notify the user of the receiver that the packets have been received. The operation of the machine. For example, the operation is sounding, vibration, or screen display.

Next, the receiver receives a packetized visible light signal, that is, each packet including each of a plurality of partial data (step S10132). Here, the receiver acquires the notification operation identifier and the priority of the notification operation identifier (specifically, an identifier indicating the priority) included in the visible light signal (step S10133).

Further, the receiver sets the current notification operation setting of the receiver, that is, the notification operation identifier preset in the receiver and the priority of the notification operation identifier (specifically, an identifier indicating the priority). ) Is read out (step S10134). Note that the notification operation identifier set in advance in the receiver is set by a user operation, for example.

Then, the receiver selects an identifier having the highest priority among the preset notification operation identifiers and the notification operation identifiers acquired in steps S10131 and S10133 (step S10135). Next, the receiver resets the selected notification operation identifier to itself, thereby performing the operation indicated by the selected notification operation identifier, and notifies the user of reception of the visible light signal (step S10136).

Note that the receiver may select a notification operation identifier having a higher priority from the two notification operation identifiers without performing any one of steps S10131 and S10133.

Note that the priority of notification action identifiers transmitted from servers installed in theaters or museums, or the priority of notification action identifiers included in visible light signals transmitted within those facilities are set high. Also good. Thereby, it is possible to prevent the sound for the reception notification from being sounded in the facility regardless of the setting of the user. In other facilities, by setting the priority of the notification operation identifier low, the receiver can notify the reception by the operation according to the user's setting.

FIG. 79B is a diagram showing another example of the setting method in the present embodiment.

First, the receiver acquires a notification operation identifier for identifying the notification operation and a priority of the notification operation identifier (specifically, an identifier indicating the priority) from a server near the receiver. (Step S10141). Next, the receiver receives a packetized visible light signal, that is, each packet including each of a plurality of partial data (step S10142). Here, the receiver acquires the notification operation identifier and the priority of the notification operation identifier (specifically, an identifier indicating the priority) included in the visible light signal (step S10143).

Further, the receiver sets the current notification operation setting of the receiver, that is, the notification operation identifier preset in the receiver and the priority of the notification operation identifier (specifically, an identifier indicating the priority). ) Is read (step S10144).

The receiver includes an operation notification identifier indicating an operation for prohibiting the generation of the notification sound in the notification operation identifier set in advance and the notification operation identifier acquired in each of steps S10141 and S10143. It is determined whether or not it is (step S10145). Here, if it is determined that it is included (Y in step S10145), the receiver sounds a notification sound for notifying completion of reception (step S10146). On the other hand, if it is determined that it is not included (N in step S10145), the receiver notifies the user of the completion of reception by, for example, vibration (step S10147).

Note that the receiver does not perform either one of steps S10141 and S10143, and determines whether or not the two notification operation identifiers include an operation notification identifier indicating an operation that prohibits the generation of the notification sound. May be.

Further, the receiver may perform self-position estimation based on an image obtained by imaging, and notify the user of reception by an operation associated with the estimated position or a facility at the position.

FIG. 80 is a flowchart showing processing of the information processing program in the tenth embodiment.

This information processing program is a program for changing the luminance of the light emitter of the transmitter described above according to the number of divisions shown in FIG.

That is, this information processing program is an information processing program for causing a computer to process information to be transmitted in order to transmit the information to be transmitted by a change in luminance. Specifically, the information processing program includes an encoding step SA41 that generates an encoded signal by encoding information to be transmitted, and the number of bits of the generated encoded signal is within a range of 24 to 64 bits. In some cases, the computer executes a division step SA42 for dividing the encoded signal into four partial signals and an output step SA43 for sequentially outputting the four partial signals. These partial signals are output as packets. The information processing program may cause the computer to specify the number of bits of the encoded signal and determine the number of partial signals based on the specified number of bits. In this case, the information processing program causes the computer to generate the determined number of partial signals by dividing the encoded signal.

Thus, when the number of bits of the encoded signal is in the range of 24 to 64 bits, the encoded signal is divided into four partial signals and output. As a result, when the luminance of the illuminant changes according to the four partial signals that are output, the four partial signals are transmitted as visible light signals and received by the receiver. Here, the larger the number of bits of the output signal, the more difficult it is for the receiver to properly receive the signal by imaging, and the reception efficiency decreases. Therefore, it is desirable to divide the signal into signals having a small number of bits, that is, small signals. However, if the signal is too finely divided into many small signals, the receiver cannot receive the original signal unless each small signal is individually received, resulting in a decrease in reception efficiency. Therefore, as described above, when the number of bits of the encoded signal is in the range of 24 to 64 bits, the encoded signal is divided into four partial signals and sequentially output to indicate the information to be transmitted. The encoded signal can be transmitted as a visible light signal with the best reception efficiency. As a result, communication between various devices can be enabled.

Also, in the output step SA43, four partial signals may be output according to the first order, and further, the four partial signals may be output again according to a second order different from the first order.

As a result, the four partial signals are repeatedly output in a different order. Therefore, when each output signal is transmitted to the receiver as a visible light signal, the reception efficiency of the four partial signals is increased. It can be further increased. That is, even if the four partial signals are repeatedly output in the same order, the same partial signal may not be received by the receiver. However, by changing the order, the occurrence of such a case can be suppressed.

Also, as shown in FIGS. 79A and 79B, in the output step SA43, a notification operation identifier may be further attached to the four partial signals. The notification operation identifier is used to identify the operation of the receiver notifying the user of the receiver that the four partial signals are received when the four partial signals are transmitted by the luminance change and received by the receiver. Identifier.

Thus, when the notification operation identifier is transmitted as a visible light signal and received by the receiver, the receiver receives the four partial signals to the user according to the operation identified by the notification operation identifier. You can be notified. That is, the notification operation by the receiver can be set on the side that transmits the information to be transmitted.

Further, as shown in FIGS. 79A and 79B, in the output step SA43, a priority identifier for identifying the priority of the notification operation identifier may be further output in association with the four partial signals.

Thus, when the priority identifier and the notification operation identifier are transmitted as a visible light signal and received by the receiver, the receiver handles the notification operation identifier according to the priority identified by the priority identifier. be able to. That is, when the receiver acquires another notification operation identifier, the receiver is identified by the notification operation identified by the notification operation identifier transmitted as the visible light signal and the other notification operation identifier. One of the notification operations can be selected based on the priority.

An information processing program according to an aspect of the present invention is an information processing program that causes a computer to process information on a transmission target in order to transmit the information on the transmission target with a luminance change, and encodes the information on the transmission target An encoding step for generating an encoded signal by dividing the encoded signal into four partial signals when the number of bits of the generated encoded signal is in a range of 24 to 64 bits; And causing the computer to execute an output step of sequentially outputting the four partial signals.

Thus, as shown in FIGS. 77 to 80, when the number of bits of the encoded signal is in the range of 24 to 64 bits, the encoded signal is divided into four partial signals and output. As a result, when the luminance of the illuminant changes according to the four partial signals that are output, the four partial signals are transmitted as visible light signals and received by the receiver. Here, the larger the number of bits of the output signal, the more difficult it is for the receiver to properly receive the signal by imaging, and the reception efficiency decreases. Therefore, it is desirable to divide the signal into signals having a small number of bits, that is, small signals. However, if the signal is too finely divided into many small signals, the receiver cannot receive the original signal unless each small signal is individually received, resulting in a decrease in reception efficiency. Therefore, as described above, when the number of bits of the encoded signal is in the range of 24 to 64 bits, the encoded signal is divided into four partial signals and sequentially output to indicate the information to be transmitted. The encoded signal can be transmitted as a visible light signal with the best reception efficiency. As a result, communication between various devices can be enabled.

In the output step, the four partial signals may be output according to a first order, and the four partial signals may be output again according to a second order different from the first order.

As a result, the four partial signals are repeatedly output in a different order. Therefore, when each output signal is transmitted to the receiver as a visible light signal, the reception efficiency of the four partial signals is increased. It can be further increased. That is, even if the four partial signals are repeatedly output in the same order, the same partial signal may not be received by the receiver. However, by changing the order, the occurrence of such a case can be suppressed.

In the output step, the four partial signals are further output with a notification operation identifier attached thereto, and the notification operation identifier is transmitted when the four partial signals are transmitted by a luminance change and received by a receiver. Furthermore, an identifier for identifying the operation of the receiver that notifies the user of the receiver that the four partial signals have been received may be used.

Thus, when the notification operation identifier is transmitted as a visible light signal and received by the receiver, the receiver receives the four partial signals to the user according to the operation identified by the notification operation identifier. You can be notified. That is, the notification operation by the receiver can be set on the side that transmits the information to be transmitted.

Further, in the output step, a priority identifier for identifying a priority of the notification operation identifier may be output along with the four partial signals.

Thus, when the priority identifier and the notification operation identifier are transmitted as a visible light signal and received by the receiver, the receiver handles the notification operation identifier according to the priority identified by the priority identifier. be able to. That is, when the receiver acquires another notification operation identifier, the receiver is identified by the notification operation identified by the notification operation identifier transmitted as the visible light signal and the other notification operation identifier. One of the notification operations can be selected based on the priority.

Next, registration of network connection of electronic devices will be described.

FIG. 81 is a diagram for explaining an application example of the transmission / reception system in the present embodiment.

This transmission / reception system includes a transmitter 10131b configured as an electronic device such as a washing machine, a receiver 10131a configured as a smartphone, and a communication device 10131c configured as an access point or a router.

FIG. 82 is a flowchart showing the processing operation of the transmission / reception system in the present embodiment.

When the start button is pressed (step S10165), the transmitter 10131b transmits Wi-Fi, Bluetooth (registered trademark) information such as an SSID, a password, an IP address, a MAC address, or an encryption key to connect to the transmitter 10131b. ) Or Ethernet (registered trademark) or the like (step S10166), and waits for connection. The transmitter 10131b may transmit these pieces of information directly or indirectly. When transmitting indirectly, the transmitter 10131b transmits an ID associated with the information. For example, the receiver 10131a that has received the ID downloads information associated with the ID from a server or the like.

The receiver 10131a receives the information (step S10151), connects to the transmitter 10131b, and connects to the communication device 10131c configured as an access point or router (SSID, password, IP address, MAC address, Alternatively, the encryption key or the like is transmitted to the transmitter 10131b (step S10152). The receiver 10131a registers information (MAC address, IP address, encryption key, etc.) for the transmitter 10131b to connect to the communication device 10131c in the communication device 10131c, and makes the communication device 10131c wait for connection. Further, the receiver 10131a notifies the transmitter 10131b that preparation for connection from the transmitter 10131b to the communication device 10131c is completed (step S10153).

The transmitter 10131b disconnects from the receiver 10131a (step S10168) and connects to the communication device 10131c (step S10169). If the connection is successful (Y in step S10170), the transmitter 10131b notifies the receiver 10131a of the connection success via the communication device 10131c, and notifies the user of the connection success with a screen display, LED status, voice, or the like. (Step S10171). If the connection fails (N in step S10170), the transmitter 10131b notifies the receiver 10131a of the connection failure through visible light communication, and notifies the user in the same way as when it is successful (step S10172). The connection success may be notified by visible light communication.

The receiver 10131a connects to the communication device 10131c (step S10154), and if there is no notification of a connection success or failure (N in step S10155 and N in step S10156), the transmitter 10131b can be accessed via the communication device 10131c. It is confirmed whether or not (step S10157). If not (N in step S10157), the receiver 10131a determines whether or not the connection to the transmitter 10131b using the information received from the transmitter 10131b has been made a predetermined number of times or more (step S10158). If it is determined that the predetermined number of times has not been performed (N in step S10158), the receiver 10131a repeats the process from step S10152. On the other hand, if it is determined that the predetermined number of times has been performed (Y in step S10158), the receiver 10131a notifies the user of the processing failure (step S10159). If the receiver 10131a determines in step S10156 that there has been a notification of a successful connection (Y in step S10156), the receiver 10131a notifies the user of the processing success (step S10160). That is, the receiver 10131a notifies the user whether or not the transmitter 10131b has been able to connect to the communication device 10131c by screen display, voice, or the like. Accordingly, the transmitter 10131b can be connected to the communication device 10131c without requiring complicated input by the user.

Next, registration of network connection of electronic devices (when connecting via another electronic device) will be described.

FIG. 83 is a diagram for explaining an application example of the transmission / reception system in the present embodiment.

This transmission / reception system includes an air conditioner 10133b, a transmitter 10133c configured as an electronic device such as a wireless adapter connected to the air conditioner 10133b, a receiver 10133a configured as, for example, a smartphone, and a communication device configured as an access point or a router. 10133d and another electronic device 10133e configured as, for example, a wireless adapter, a wireless access point, or a router.

FIG. 84 is a flowchart showing the processing operation of the transmission / reception system in the present embodiment. Hereinafter, the air conditioner 10133b or the transmitter 10133c is referred to as an electronic device A, and the electronic device 10133e is referred to as an electronic device B.

First, when the start button is pressed (step S10188), the electronic device A transmits information (individual ID, password, IP address, MAC address, encryption key, etc.) for connection to itself (step S10189). The connection is awaited (step S10190). The electronic device A may transmit these pieces of information directly or indirectly as described above.

The receiver 10133a receives the information from the electronic device A (step S10181) and transmits the information to the electronic device B (step S10182). When electronic device B receives the information (step S10196), it connects to electronic device A according to the received information (step S10197). Then, the electronic device B determines whether or not the connection with the electronic device A is established (step S10198), and notifies the receiver 10133a of the success or failure (step S10199 or step S10200).

If the electronic device A is connected to the electronic device B during a predetermined time (Y in step S10191), the electronic device A notifies the receiver 10133a of the connection success via the electronic device B (step S10192) and is not connected (step S10192). N in step S10191), the receiver 10133a is notified of the connection failure by visible light communication (step S10193). In addition, the electronic device A notifies the user of the success or failure of the connection through a screen display, a light emission state, sound, or the like. Accordingly, the electronic device A (transmitter 10133c) can be connected to the electronic device B (electronic device 10133e) without requiring complicated input by the user. Note that the air conditioner 10133b and the transmitter 10133c illustrated in FIG. 83 may be configured integrally, and similarly, the communication device 10133d and the electronic device 10133e may be configured integrally.

Next, transmission of appropriate imaging information will be described.

FIG. 85 is a diagram for explaining an application example of the transmission / reception system according to the present embodiment.

This transmission / reception system includes a receiver 10135a configured as, for example, a digital still camera or a digital video camera, and a transmitter 10135b configured as, for example, illumination.

FIG. 86 is a flowchart showing the processing operation of the transmission / reception system in the present embodiment.

First, the receiver 10135a sends an imaging information transmission command to the transmitter 10135b (step S10211). Next, the transmitter 10135b receives the imaging information transmission command, the imaging information transmission button is pressed, the imaging information transmission switch is turned on, or the power is turned on (in step S10221). Y), imaging information is transmitted (step S10222). The imaging information transmission command is a command for transmitting imaging information, and the imaging information indicates, for example, the color temperature of illumination, spectral distribution, illuminance, or light distribution. The transmitter 10135b may transmit the imaging information directly or indirectly as described above. When transmitting indirectly, the transmitter 10135b transmits an ID associated with the imaging information. The receiver 10135a that has received the ID downloads, for example, imaging information associated with the ID from a server or the like. At this time, the transmitter 10135b is a method for transmitting a transmission stop command to itself (frequency of radio waves, infrared rays, or sound waves for transmitting the transmission stop command, or an SSID, a password or an IP address for connection to self-confidence) ) May be sent.

When receiving the imaging information (step S10212), the receiver 10135a transmits a transmission stop command to the transmitter 10135b (step S10213). Here, when the transmitter 10135b receives a transmission stop command from the receiver 10135a (step S10223), the transmitter 10135b stops transmission of imaging information and emits light uniformly (step S10224).

Furthermore, the receiver 10135a sets the imaging parameter according to the imaging information received in step S10212 (step S10214), or notifies the user of the imaging information. The imaging parameter is, for example, white balance, exposure time, focal length, sensitivity, or scene mode. Thereby, it is possible to take an image with an optimum setting according to the illumination. Next, the receiver 10135a captures an image after the transmission of imaging information from the transmitter 10135b is stopped (Y in step S10215) (step S10216). Thereby, it is possible to perform imaging without changing the brightness of the subject due to signal transmission. Note that, after step S10216, the receiver 10135a may transmit a transmission start command for prompting the start of transmission of imaging information to the transmitter 10135b (step S10217).

Next, the display of the state of charge will be described.

FIG. 87 is a diagram for describing an application example of the transmitter in this embodiment.

For example, the transmitter 10137b configured as a charger includes a light emitting unit, and transmits a visible light signal indicating a charged state of the battery from the light emitting unit. Thus, the state of charge of the battery can be notified without an expensive display device. When a small LED is used as the light emitting unit, a visible light signal cannot be received unless the LED is imaged from nearby. In addition, in the transmitter 10137c having a protrusion near the LED, it is difficult to close-up the LED due to the protrusion. Therefore, the visible light signal from the transmitter 10137b having no protrusion near the LED can be received more easily than the visible light signal from the transmitter 10137c.

(Embodiment 11)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

First, transmission in demo mode and failure will be explained.

FIG. 88 is a diagram for explaining an example of the operation of the transmitter in this embodiment.

When an error has occurred, the transmitter transmits a signal indicating that the error has occurred, or a signal corresponding to the error code. Can tell error details. The receiver can correct the error or appropriately report the error content to the service center by indicating an appropriate response according to the error content.

When the transmitter is in demo mode, it will send a demo code. As a result, for example, when a demo of a transmitter that is a product is performed at a storefront, the store visitor can receive the demo code and acquire the product description associated with the demo code. Whether or not the demo mode is selected is determined by whether the transmitter operation setting is in the demo mode, the storefront CAS card is inserted, the CAS card is not inserted, or the recording medium is not inserted. Judging from the point.

Next, signal transmission from the remote controller will be described.

FIG. 89 is a diagram for explaining an example of the operation of the transmitter in this embodiment.

For example, when a transmitter configured as an air conditioner remote controller receives main body information, the transmitter transmits the main body information, so that the receiver receives information on a distant main body from a nearby transmitter. can do. The receiver can also receive information from a main body that exists in a place where visible light communication is impossible, such as over a network.

Next, a description will be given of processing for transmitting only when the subject is in a bright place.

FIG. 90 is a diagram for explaining an example of the operation of the transmitter in this embodiment.

The transmitter transmits if the ambient brightness is above a certain level, and stops transmitting if it is below a certain level. Thereby, for example, a transmitter configured as an advertisement for a train can automatically stop its operation when the vehicle enters the garage, and battery consumption can be suppressed.

Next, content distribution (association change / scheduling) in accordance with the transmitter display will be described.

FIG. 91 is a diagram for explaining an example of the operation of the transmitter in this embodiment.

The transmitter associates the content that the receiver wants to acquire with the transmission ID in accordance with the display timing of the content to be displayed. Each time the display content is changed, the association change is registered with the server.

When the display timing of the display content is known, the transmitter sets the server so that another content is delivered to the receiver in accordance with the change timing of the display content. When the request for the content associated with the transmission ID is received from the receiver, the server transmits the content according to the set schedule to the receiver.

Thus, for example, when the transmitter configured as digital signage changes display contents one after another, the receiver can acquire content that matches the content displayed by the transmitter.

Next, content distribution (synchronization by time) in accordance with the display of the transmitter will be described.

FIG. 92 is a diagram for explaining an example of the operation of the transmitter in this embodiment.

In response to a content acquisition request associated with a predetermined ID, it is registered in advance in the server so that different content is passed according to the time.

The transmitter synchronizes the time with the server, adjusts the timing so that a predetermined part is displayed at a predetermined time, and displays the content.

Thus, for example, when the transmitter configured as digital signage changes display contents one after another, the receiver can acquire content that matches the content displayed by the transmitter.

Next, content distribution (transmission of display time) that matches the display of the transmitter will be described.

FIG. 93 is a diagram for explaining an example of operations of the transmitter and the receiver in this embodiment.

The transmitter transmits the display time of the content being displayed in addition to the transmitter ID. The content display time is information that can identify the currently displayed content, and can be expressed by, for example, an elapsed time from the start time of the content.

The receiver acquires the content associated with the received ID from the server and displays the content according to the received display time. Thereby, for example, when a transmitter configured as digital signage changes display contents one after another, the receiver can acquire content that matches the content displayed by the transmitter.

Also, the receiver changes the content to be displayed over time. As a result, even if a signal is not received again when the display content of the transmitter changes, the content that matches the display content is displayed.

Next, data uploading according to the user's permission status will be described.

FIG. 94 is a diagram for explaining an example of the operation of the receiver in this embodiment.

If the user has registered for an account, the receiver is authorized to access information such as the location of the receiver, phone number, ID, installed application, and user (Age, sex, occupation, preference, etc.) are transmitted to the server together with the received ID.

If the account is not registered, if the user permits uploading of information as described above, it is similarly sent to the server. If not permitted, only the received ID is sent to the server. .

This allows the user to receive content tailored to the situation at the time of reception and his / her personality, and the server can be used for data analysis by obtaining user information.

Next, activation of the content playback application will be described.

FIG. 95 is a diagram for explaining an example of operation of the receiver in this embodiment.

The receiver acquires content associated with the received ID from the server. When the active application can handle (can display or play) the acquired content, the acquired content is displayed / reproduced by the active application. If it cannot be handled, it is confirmed whether or not an app that can be handled is installed in the receiver. If it is installed, the app is activated to display / play the acquired content. If it is not installed, it automatically installs, displays a message prompting installation, displays a download screen, and displays and plays the acquired content after installation.

Thereby, the acquired content can be handled appropriately (display / playback etc.).

Next, activation of the specified application will be described.

FIG. 96 is a diagram for explaining an example of operation of the receiver in this embodiment.

The receiver acquires the content associated with the received ID and information (application ID) for specifying the application to be activated from the server. When the running application is the designated application, the acquired content is displayed / reproduced. When the designated application is installed in the receiver, the designated application is activated to display / play the acquired content. If it is not installed, it automatically installs, displays a message prompting installation, displays a download screen, and displays and plays the acquired content after installation.

The receiver may acquire only the application ID from the server and start the designated application.

The receiver may perform specified settings. The receiver may set the designated parameter and activate the designated application.

Next, selection of streaming reception and normal reception will be described.

FIG. 97 is a diagram for explaining an example of operation of the receiver in this embodiment.

The receiver determines that the signal is being streamed when the value of the predetermined address of the received data is a predetermined value or the received data includes a predetermined identifier, and receives the streaming data. Receive with. Otherwise, it is received by a normal receiving method.

This makes it possible to receive a signal regardless of whether it is transmitted using streaming distribution or normal distribution.

Next, private data will be explained.

FIG. 98 is a diagram for explaining an example of operation of the receiver in this embodiment.

When the received ID value is within a predetermined range or includes a predetermined identifier, the receiver refers to the table in the application, and if the reception ID exists in the table, the receiver is designated in the table. Get the content. Otherwise, the content set as the reception ID is acquired from the server.

This makes it possible to receive content without registering with the server. Further, since communication with the server is not performed, a quick response can be obtained.

Next, the setting of the exposure time according to the frequency will be described.

FIG. 99 is a diagram for explaining an example of operation of the receiver in this embodiment.

The receiver detects the signal and recognizes the modulation frequency of the signal. The receiver sets the exposure time according to the period of the modulation frequency (modulation period). For example, the signal can be easily received by setting the exposure time to be approximately equal to the modulation period. Further, for example, by setting the exposure time to an integral multiple of the modulation period or a value close to it (approximately ± 30%), it is possible to easily receive a signal by convolutional decoding.

Next, the optimal parameter setting for the transmitter will be described.

FIG. 100 is a diagram for explaining an example of operation of the receiver in this embodiment.

In addition to the data received from the transmitter, the receiver transmits current location information and information related to the user (address, gender, age, preference, etc.) to the server. The server transmits parameters for optimal operation of the transmitter to the receiver in accordance with the received information. The receiver sets the received parameter if it can be set in the transmitter. If it cannot be set, the parameter is displayed and the user is prompted to set the parameter.

This allows, for example, operating the washing machine to optimize the water properties of the area where the transmitter is used, or operating the rice cooker to cook rice in a way that is optimal for the type of rice used by the user You can make it.

Next, an identifier indicating the data structure will be described.

FIG. 101 is a diagram for explaining an example of a configuration of transmission data in the present embodiment.

The information to be transmitted includes an identifier, and the receiver can know the configuration of the subsequent part by its value. For example, it is possible to specify the data length, the type and length of the error correction code, the data division point, and the like.

This allows the transmitter to change the type and length of the data body and error correction code according to the nature of the transmitter and the communication path. Also, the transmitter can cause the receiver to acquire an ID corresponding to the content ID by transmitting the content ID in addition to the ID of the transmitter.

(Embodiment 12)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

FIG. 102 is a diagram for explaining the operation of the receiver in this embodiment.

The receiver 1210a in the present embodiment switches the shutter speed between high speed and low speed, for example, in units of frames when performing continuous shooting by the image sensor. Furthermore, the receiver 1210a switches the process for the frame to a barcode recognition process and a visible light recognition process based on the frame obtained by the shooting. Here, the barcode recognition process is a process for decoding a barcode reflected in a frame obtained by a low shutter speed. The visible light recognition process is a process of decoding the above-described bright line pattern reflected in a frame obtained with a high shutter speed.

Such a receiver 1210a includes a video input unit 1211, a barcode / visible light identification unit 1212, a barcode recognition unit 1212a, a visible light recognition unit 1212b, and an output unit 1213.

The video input unit 1211 includes an image sensor, and switches the shutter speed for shooting by the image sensor. That is, the video input unit 1211 switches the shutter speed alternately between low speed and high speed, for example, in units of frames. More specifically, the video input unit 1211 switches the shutter speed to high speed for odd-numbered frames and switches the shutter speed to low speed for even-numbered frames. Shooting at a low shutter speed is shooting in the above-described normal shooting mode, and shooting at a high shutter speed is shooting in the above-described visible light communication mode. That is, when the shutter speed is low, the exposure time of each exposure line included in the image sensor is long, and a normal captured image on which the subject is projected is obtained as a frame. Further, when the shutter speed is high, the exposure time of each exposure line included in the image sensor is short, and a visible light communication image in which the above-described bright line is projected is obtained as a frame.

The barcode / visible light identifying unit 1212 switches processing for the image by determining whether a barcode appears or whether a bright line appears in the image obtained by the video input unit 1211. For example, if a barcode appears in a frame obtained by shooting at a low shutter speed, the barcode / visible light identification unit 1212 causes the barcode recognition unit 1212a to perform processing on the image. On the other hand, if a bright line appears in an image obtained by shooting at a high shutter speed, the barcode / visible light identifying unit 1212 causes the visible light recognizing unit 1212b to execute processing on the image.

The barcode recognition unit 1212a decodes a barcode appearing in a frame obtained by shooting at a low shutter speed. The barcode recognition unit 1212a acquires barcode data (for example, a barcode identifier) by decoding, and outputs the barcode identifier to the output unit 1213. The barcode may be a one-dimensional code or a two-dimensional code (for example, a QR code (registered trademark)).

The visible light recognizing unit 1212b decodes the bright line pattern appearing in the frame obtained by photographing at a high shutter speed. The visible light recognizing unit 1212b obtains visible light data (for example, a visible light identifier) by the decoding, and outputs the visible light identifier to the output unit 1213. The visible light data is the above-described visible light signal.

The output unit 1213 displays only frames obtained by shooting at a low shutter speed. Therefore, when the subject imaged by the video input unit 1211 is a barcode, the output unit 1213 displays the barcode. When the subject photographed by the video input unit 1211 is a digital signage that transmits a visible light signal, the output unit 1213 displays an image of the digital signage without displaying the bright line pattern. And when the output part 1213 acquires a barcode identifier, it acquires the information matched with the barcode identifier from a server etc., for example, and displays the information. Further, when the output unit 1213 acquires a visible light identifier, the output unit 1213 acquires information associated with the visible light identifier from, for example, a server and displays the information.

That is, the receiver 1210a as a terminal device includes an image sensor, and the image sensor is switched while alternately switching the shutter speed of the image sensor between a first speed and a second speed higher than the first speed. Perform continuous shooting with. (A) When the subject imaged by the image sensor is a barcode, the receiver 1210a obtains an image showing the barcode by photographing when the shutter speed is the first speed, A barcode identifier is obtained by decoding the barcode reflected in the image. In addition, (b) when the subject to be photographed by the image sensor is a light source (for example, digital signage), the receiver 1210a includes a plurality of images included in the image sensor by photographing when the shutter speed is the second speed. A bright line image that is an image including a bright line corresponding to each of the exposure lines is acquired. Then, the receiver 1210a acquires a visible light signal as a visible light identifier by decoding a plurality of bright line patterns included in the acquired bright line image. Further, the receiver 1210a displays an image obtained by photographing when the shutter speed is the first speed.

In such a receiver 1210a in this embodiment, the barcode is decoded and the visible light signal can be received by switching between the barcode recognition process and the visible light recognition process. Furthermore, power consumption can be suppressed by switching.

The receiver in this embodiment may perform image recognition processing simultaneously with visible light processing instead of barcode recognition processing.

FIG. 103A is a diagram for explaining another operation of the receiver in this embodiment.

The receiver 1210b in the present embodiment switches the shutter speed between high speed and low speed, for example, in units of frames when performing continuous shooting by the image sensor. Further, the receiver 1210b simultaneously performs the image recognition process and the above-described visible light recognition process on the image (frame) obtained by the photographing. The image recognition process is a process for recognizing a subject appearing in a frame obtained with a low shutter speed.

Such a receiver 1210b includes a video input unit 1211, an image recognition unit 1212c, a visible light recognition unit 1212b, and an output unit 1215.

The video input unit 1211 includes an image sensor, and switches the shutter speed for shooting by the image sensor. That is, the video input unit 1211 switches the shutter speed alternately between a low speed and a high speed, for example, in frame units. More specifically, the video input unit 1211 switches the shutter speed to high speed for odd-numbered frames and switches the shutter speed to low speed for even-numbered frames. Shooting at a low shutter speed is shooting in the above-described normal shooting mode, and shooting at a high shutter speed is shooting in the above-described visible light communication mode. That is, when the shutter speed is low, the exposure time of each exposure line included in the image sensor is long, and a normal captured image on which the subject is projected is obtained as a frame. Further, when the shutter speed is high, the exposure time of each exposure line included in the image sensor is short, and a visible light communication image in which the above-described bright line is projected is obtained as a frame.

The image recognition unit 1212c recognizes a subject appearing in a frame obtained by shooting at a low shutter speed and specifies the position of the subject in the frame. As a result of recognition, the image recognition unit 1212c determines whether or not the subject is an AR (Augmented Reality) target (hereinafter referred to as an AR target). When the image recognition unit 1212c determines that the subject is an AR object, the image recognition unit 1212c generates image recognition data that is data for displaying information about the subject (for example, the position of the subject and the AR marker). The AR marker is output to the output unit 1215.

The output unit 1215 displays only frames obtained by shooting at a low shutter speed, as with the output unit 1213 described above. Therefore, when the subject photographed by the video input unit 1211 is digital signage that transmits a visible light signal, the output unit 1213 displays the image of the digital signage without displaying the bright line pattern. Further, when acquiring the image recognition data from the image recognition unit 1212c, the output unit 1215 superimposes a white frame-shaped indicator surrounding the subject on the frame based on the position of the subject in the frame indicated by the image recognition data. .

FIG. 103B is a diagram illustrating an example of an indicator displayed by the output unit 1215.

The output unit 1215 superimposes a white frame-shaped indicator 1215b surrounding a subject image 1215a configured as digital signage on a frame, for example. That is, the output unit 1215 displays the indicator 1215b indicating the subject whose image has been recognized. Furthermore, when the output unit 1215 acquires the visible light identifier from the visible light recognition unit 1212b, the output unit 1215 changes the color of the indicator 1215b from white to red, for example.

FIG. 103C is a diagram showing a display example of AR.

The output unit 1215 further acquires information related to the subject associated with the visible light identifier as related information from, for example, a server. The output unit 1215 describes related information in the AR marker 1215c indicated by the image recognition data, and displays the AR marker 1215c in which the related information is described in association with the subject image 1215a in the frame.

In such a receiver 1210b in this embodiment, an AR using visible light communication can be realized by simultaneously performing an image recognition process and a visible light recognition process. Note that the receiver 1210a illustrated in FIG. 103A may also display the indicator 1215b illustrated in FIG. 103B, similarly to the receiver 1210b. In this case, when a barcode is recognized in a frame obtained by shooting at a low shutter speed, the receiver 1210a displays a white frame-shaped indicator 1215b surrounding the barcode. Then, when the barcode is decoded, the receiver 1210a changes the color of the indicator 1215b from white to red. Similarly, when a bright line pattern is recognized in a frame obtained by imaging at a high shutter speed, the receiver 1210a identifies a part in the low-speed frame corresponding to the part where the bright line pattern is located. For example, when the digital signage is transmitting a visible light signal, an image of the digital signage in the low-speed frame is specified. Note that the low speed frame is a frame obtained by shooting at a low shutter speed. Then, the receiver 1210a displays a white frame-shaped indicator 1215b surrounding a specified portion (for example, the above-mentioned digital signage image) in the low-speed frame so as to be superimposed on the low-speed frame. Then, when the bright line pattern is decoded, the receiver 1210a changes the color of the indicator 1215b from white to red.

FIG. 104A is a diagram for describing an example of a transmitter in this embodiment.

The transmitter 1220a in the present embodiment transmits a visible light signal in synchronization with the transmitter 1230. That is, the transmitter 1220a transmits the same visible light signal as the visible light signal at the timing when the transmitter 1230 transmits the visible light signal. Note that the transmitter 1230 includes a light emitting unit 1231, and transmits a visible light signal when the luminance of the light emitting unit 1231 changes.

Such a transmitter 1220 a includes a light receiving unit 1221, a signal analyzing unit 1222, a transmission clock adjusting unit 1223 a, and a light emitting unit 1224. The light emitting unit 1224 transmits a visible light signal that is the same as the visible light signal transmitted from the transmitter 1230 by changing the luminance. The light receiving unit 1221 receives a visible light signal from the transmitter 1230 by receiving visible light from the transmitter 1230. The signal analysis unit 1222 analyzes the visible light signal received by the light receiving unit 1221 and transmits the analysis result to the transmission clock adjustment unit 1223a. The transmission clock adjustment unit 1223a adjusts the timing of the visible light signal transmitted from the light emitting unit 1224 based on the analysis result. That is, the transmission clock adjustment unit 1223a uses the light emitting unit 1224 so that the timing at which the visible light signal is transmitted from the light emitting unit 1231 of the transmitter 1230 matches the timing at which the visible light signal is transmitted from the light emitting unit 1224. Adjust the brightness change timing.

Thereby, the waveform of the visible light signal transmitted by the transmitter 1220a and the waveform of the visible light signal transmitted by the transmitter 1230 can be matched in timing.

FIG. 104B is a diagram for describing another example of the transmitter in this embodiment.

The transmitter 1220b in the present embodiment transmits a visible light signal in synchronization with the transmitter 1230, similarly to the transmitter 1220a. That is, the transmitter 1200b transmits the same visible light signal as the visible light signal at the timing when the transmitter 1230 transmits the visible light signal.

Such a transmitter 1220b includes a first light receiving unit 1221a, a second light receiving unit 1221b, a comparison unit 1225, a transmission clock adjusting unit 1223b, and a light emitting unit 1224.

The first light receiving unit 1221a receives the visible light from the transmitter 1230 by receiving the visible light from the transmitter 1230, similarly to the light receiving unit 1221. The second light receiving unit 1221 b receives visible light from the light emitting unit 1224. The comparison unit 1225 compares the first timing at which visible light is received by the first light receiving unit 1221a and the second timing at which visible light is received by the second light receiving unit 1221b. Then, the comparison unit 1225 outputs the difference (that is, the delay time) between the first timing and the second timing to the transmission clock adjustment unit 1223b. The transmission clock adjusting unit 1223b adjusts the timing of the visible light signal transmitted from the light emitting unit 1224 so that the delay time is shortened.

Thereby, the waveform of the visible light signal transmitted by the transmitter 1220b and the waveform of the visible light signal transmitted by the transmitter 1230 can be matched more accurately in terms of timing.

In the example shown in FIGS. 104A and 104B, two transmitters transmit the same visible light signal, but different visible light signals may be transmitted. That is, when the two transmitters transmit the same visible light signal, they are transmitted in synchronization as described above. When two transmitters transmit different visible light signals, only one of the two transmitters transmits a visible light signal, while the other transmitter is uniformly lit or extinguished. To do. Thereafter, one transmitter is uniformly lit or extinguished, while only the other transmitter transmits a visible light signal. Two transmitters may transmit different visible light signals simultaneously.

FIG. 105A is a diagram for describing an example of synchronous transmission by a plurality of transmitters in the present embodiment.

The plurality of transmitters 1220 in the present embodiment are arranged, for example, in a line as shown in FIG. 105A. Note that these transmitters 1220 have the same configuration as the transmitter 1220a shown in FIG. 104A or the transmitter 1220b shown in FIG. 104B. Each of the plurality of transmitters 1220 transmits a visible light signal in synchronization with one of the transmitters 1220 on both sides.

This allows many transmitters to transmit visible light signals synchronously.

FIG. 105B is a diagram for describing an example of synchronous transmission by a plurality of transmitters in the present embodiment.

One transmitter 1220 among the plurality of transmitters 1220 in this embodiment is a reference for synchronizing the visible light signal, and the remaining plurality of transmitters 1220 are visible light signals so as to match the reference. Send.

This allows many transmitters to transmit visible light signals more accurately and synchronously.

FIG. 106 is a diagram for describing another example of synchronous transmission by a plurality of transmitters in the present embodiment.

Each of the plurality of transmitters 1240 in the present embodiment receives a synchronization signal and transmits a visible light signal in accordance with the synchronization signal. Thereby, a visible light signal is transmitted from each of the plurality of transmitters 1240 in synchronization.

Specifically, each of the plurality of transmitters 1240 includes a control unit 1241, a synchronization control unit 1242, a photocoupler 1243, an LED drive circuit 1244, an LED 1245, and a photodiode 1246.

The control unit 1241 receives the synchronization signal and outputs the synchronization signal to the synchronization control unit 1242.

The LED 1245 is a light source that emits visible light, and blinks (that is, changes in luminance) in accordance with control by the LED drive circuit 1244. Thus, a visible light signal is transmitted from the LED 1245 to the outside of the transmitter 1240.

The photocoupler 1243 transmits a signal between the synchronization control unit 1242 and the LED drive circuit 1244 while being electrically insulated. Specifically, the photocoupler 1243 transmits a transmission start signal described later transmitted from the synchronization control unit 1242 to the LED drive circuit 1244.

When the LED drive circuit 1244 receives the transmission start signal from the synchronization control unit 1242 via the photocoupler 1243, the LED drive circuit 1244 causes the LED 1245 to start transmitting the visible light signal at the timing when the transmission start signal is received.

The photodiode 1246 detects visible light emitted from the LED 1245, and outputs a detection signal indicating that the visible light has been detected to the synchronization control unit 1242.

When receiving the synchronization signal from the control unit 1241, the synchronization control unit 1242 transmits a transmission start signal to the LED drive circuit 1244 via the photocoupler 1243. By transmitting this transmission start signal, transmission of the visible light signal is started. When the synchronization control unit 1242 receives the detection signal from the photodiode 1246 by transmitting the visible light signal, the synchronization control unit 1242 is a delay that is a difference between the timing at which the detection signal is received and the timing at which the synchronization signal is received from the control unit 1241. Calculate time. When the synchronization control unit 1242 receives the next synchronization signal from the control unit 1241, the synchronization control unit 1242 adjusts the timing for transmitting the next transmission start signal based on the calculated delay time. That is, the synchronization control unit 1242 adjusts the timing of transmitting the next transmission start signal so that the delay time for the next synchronization signal becomes a predetermined set delay time. Thus, the synchronization control unit 1242 transmits the next transmission start signal at the adjusted timing.

FIG. 107 is a diagram for explaining signal processing in the transmitter 1240.

When receiving the synchronization signal, the synchronization control unit 1242 generates a delay time setting signal that generates a delay time setting pulse at a predetermined timing. Note that receiving the synchronization signal specifically means receiving a synchronization pulse. That is, the synchronization control unit 1242 generates the delay time setting signal so that the delay time setting pulse rises at the timing when the set delay time has passed since the falling edge of the synchronization pulse.

The synchronization control unit 1242 transmits a transmission start signal to the LED drive circuit 1244 via the photocoupler 1243 at a timing delayed by the correction value N obtained last time from the falling edge of the synchronization pulse. As a result, a visible light signal is transmitted from the LED 1245 by the LED drive circuit 1244. Here, the synchronization control unit 1242 receives the detection signal from the photodiode 1246 at a timing delayed by the sum of the intrinsic delay time and the correction value N from the falling edge of the synchronization pulse. That is, transmission of a visible light signal is started from that timing. Hereinafter, this timing is referred to as transmission start timing. Note that the above-described intrinsic delay time is a delay time caused by a circuit such as the photocoupler 1243, and is a delay time that occurs even when the synchronization control unit 1242 receives the synchronization signal and immediately transmits the transmission start signal. .

The synchronization control unit 1242 identifies the time difference from the transmission start timing to the rise of the delay time setting pulse as the correction correction value N. Then, the synchronization control unit 1242 calculates and holds the correction value (N + 1) by the correction value (N + 1) = the correction value N + the correction correction value N. Thus, when the synchronization control unit 1242 receives the next synchronization signal (synchronization pulse), it transmits a transmission start signal to the LED drive circuit 1244 at a timing delayed by the correction value (N + 1) from the falling edge of the synchronization pulse. To do. The correction correction value N can be a negative value as well as a positive value.

Accordingly, each of the plurality of transmitters 1240 transmits the visible light signal after the set delay time has elapsed after receiving the synchronization signal (synchronization pulse). it can. That is, even if there is a variation in the inherent delay time caused by a circuit such as the photocoupler 1243 in each of the plurality of transmitters 1240, the visible light from each of the plurality of transmitters 1240 is not affected by the variation. The transmission of the optical signal can be accurately synchronized.

Note that the LED drive circuit consumes a large amount of power, and is electrically insulated from the control circuit that handles the synchronization signal using a photocoupler or the like. Therefore, when such a photocoupler is used, it is difficult to synchronize the transmission of visible light signals from a plurality of transmitters due to the variation in the inherent delay time described above. However, in the plurality of transmitters 1240 in this embodiment, the light emission timing of the LED 1245 is detected by the photodiode 1246, the delay time from the synchronization signal is detected by the synchronization control unit 1242, and the delay time is set in advance. The time is adjusted to be the time (the set delay time described above). As a result, even if there are individual variations in the photocouplers provided in a plurality of transmitters each configured as, for example, LED lighting, a visible light signal (for example, visible light ID) is synchronized with high accuracy from the plurality of LED lighting. Can be sent.

Note that the LED illumination may be turned on or off outside the visible light signal transmission period. When lighting other than the visible light signal transmission period, the first falling edge of the visible light signal may be detected. In the case of turning off the light other than the visible light signal transmission period, the first rising edge of the visible light signal may be detected.

In the above-described example, the transmitter 1240 transmits a visible light signal every time a synchronization signal is received. However, the transmitter 1240 may transmit a visible light signal without receiving the synchronization signal. That is, if the transmitter 1240 transmits a visible light signal once in response to reception of the synchronization signal, the transmitter 1240 may sequentially transmit the visible light signal without receiving the synchronization signal. Specifically, the transmitter 1240 may sequentially transmit the visible light signal 2 to several thousand times with respect to one reception of the synchronization signal. The transmitter 1240 may transmit a visible light signal corresponding to the synchronization signal at a rate of once every 100 milliseconds or once every few seconds.

Further, when the visible light signal is repeatedly transmitted according to the synchronization signal, the continuity of light emission of the LED 1245 may be lost due to the above-described set delay time. That is, a slightly longer blanking period may occur. As a result, blinking of the LED 1245 is visually recognized by a person, and so-called flicker may occur. Therefore, the transmitter 1240 may transmit a visible light signal corresponding to the synchronization signal at a period of 60 Hz or more. Thereby, blinking is performed at high speed, and the blinking is difficult to be visually recognized by a person. As a result, the occurrence of flicker can be suppressed. Alternatively, the transmitter 1240 may transmit a visible light signal corresponding to the synchronization signal at a sufficiently long cycle such as once every few minutes. Thereby, although blinking is visually recognized by a person, it is possible to prevent the blinking from being viewed repeatedly and continuously, and to reduce the discomfort that flicker gives to the person.

(Preprocessing for receiving method)
FIG. 108 is a flowchart illustrating an example of a reception method in this embodiment. FIG. 109 is an explanatory diagram for describing an example of a reception method in this embodiment.

First, the receiver calculates the average value of the pixel values of a plurality of pixels arranged in a direction parallel to the exposure line (step S1211). If the pixel values of N pixels are averaged according to the central limit theorem, the expected value of the noise amount becomes N minus ½ power, and the SN ratio is improved.

Next, the receiver leaves only the part where the pixel value changes in the vertical direction in all the colors, and removes the change in the pixel value where the pixel value changes differently (step S1212). When the transmission signal (visible light signal) is expressed by the luminance of the light emitting unit provided in the transmitter, the luminance of the illumination of the transmitter or the backlight of the display changes. At this time, as in the part (b) of FIG. 109, the pixel values change in the same direction for all the colors. In the portions (a) and (c) of FIG. 109, the pixel values change differently for each color. In these portions, the pixel value fluctuates due to reception noise or a picture of the display or signage. Therefore, by removing these fluctuations, the SN ratio can be improved.

Next, the receiver obtains a luminance value (step S1213). Since the luminance is not easily changed by color, the influence of the display or signage picture can be eliminated, and the SN ratio can be improved.

Next, the receiver applies a low-pass filter to the luminance value (step S1214). In the receiving method according to the present embodiment, since a moving average filter based on the length of exposure time is applied, almost no signal is included in the high frequency region, and noise is dominant. Therefore, the S / N ratio can be improved by using a low-pass filter that cuts a high frequency region. Since there are many signal components up to the frequency up to the reciprocal of the exposure time, the effect of improving the S / N ratio can be increased by blocking the higher frequency. When the frequency component included in the signal is finite, the S / N ratio can be improved by blocking a frequency higher than that frequency. A filter (such as a Butterworth filter) that does not include a frequency vibration component is suitable for the low-pass filter.

(Receiving method by convolution maximum likelihood decoding)
FIG. 110 is a flowchart illustrating another example of the reception method in this embodiment. Hereinafter, the reception method when the exposure time is longer than the transmission cycle will be described with reference to FIG.

When the exposure time is an integral multiple of the transmission period, reception can be performed with the highest accuracy. Even if it is not an integral multiple, it can be received within a range of (N ± 0.33) times (N is an integer).

First, the receiver sets the transmission / reception offset to 0 (step S1221). The transmission / reception offset is a value for correcting a difference between the transmission timing and the reception timing. Since this deviation is unknown, the receiver gradually changes the value that is a candidate for the transmission / reception offset, and adopts the most suitable value as the transmission / reception offset.

Next, the receiver determines whether or not the transmission / reception offset is less than the transmission cycle (step S1222). Here, since the reception cycle and the transmission cycle are not synchronized with each other, a reception value matching the transmission cycle is not always obtained. For this reason, when the receiver determines in step S1222 that it is less than the transmission cycle (Y in step S1222), the reception value (for example, pixel value) matched to the transmission cycle is used for each transmission cycle using the reception value in the vicinity thereof. ) Is calculated by interpolation (step S1223). As the interpolation method, linear interpolation, nearest neighbor value, spline interpolation, or the like can be used. Next, the receiver obtains a difference between the received values obtained for each transmission cycle (step S1224).

The receiver adds a predetermined value to the transmission / reception offset (step S1226), and repeatedly executes the processing from step S1222. If the receiver determines in step S1222 that it is not less than the transmission cycle (N in step S1222), the receiver specifies the highest likelihood among the likelihoods of the received signals calculated for each transmission / reception offset. Then, the receiver determines whether or not the highest likelihood is greater than or equal to a predetermined value (step S1227). If it is determined that the value is equal to or greater than the predetermined value (Y in step S1227), the receiver uses the received signal with the highest likelihood as the final estimation result. Alternatively, the receiver uses, as a received signal candidate, a received signal having a likelihood equal to or higher than a value obtained by subtracting a predetermined value from the highest likelihood (step S1228). On the other hand, if it is determined in step S1227 that the highest likelihood is less than the predetermined value (N in step S1227), the receiver discards the estimation result (step S1229).

When there is too much noise, it is often impossible to estimate the received signal properly, and at the same time, the likelihood decreases. Therefore, when the likelihood is low, the reliability of the received signal can be improved by discarding the estimation result. In addition, even when a valid signal is not included in the input image, there is a problem in that the maximum likelihood decoding outputs a valid signal as an estimation result. However, since the likelihood also decreases in this case, this problem can be avoided by discarding the estimation result when the likelihood is low.

(Embodiment 13)
In this embodiment, a protocol transmission method for visible light communication will be described.

(Multi-value amplitude pulse signal)
111, 112, and 113 are diagrams illustrating an example of a transmission signal in the present embodiment.

By giving meaning to the amplitude of the pulse, more information can be expressed per unit time. For example, when the amplitude is classified into three stages, as shown in FIG. 111, three values can be expressed by the transmission time of two slots while maintaining the average luminance at 50%. However, if (c) in FIG. 111 is transmitted continuously, there is no change in luminance, so the presence of a signal is difficult to understand. Also, ternary values are a little difficult to handle in digital processing.

Therefore, by using the four types of symbols (a) to (d) in FIG. 112, four values can be expressed with an average transmission time of three slots while keeping the average luminance at 50%. Although the transmission time differs depending on the symbol, it is possible to recognize the end point of the symbol by setting the last state of the symbol to a low luminance state. The same effect can be obtained even when the high luminance state and the low luminance state are switched. (E) in FIG. 112 is not suitable because it cannot be distinguished from transmitting (a) twice. 112 (f) and (g) are somewhat difficult to recognize because the intermediate luminance is continuous, but can be used.

Consider using patterns (a) and (b) in FIG. 113 as headers. Since these patterns have strong specific frequency components in frequency analysis, signal detection can be performed by frequency analysis by using these patterns as headers.

As shown in (c) of FIG. 113, a transmission packet is configured using the patterns (a) and (b). Data can be delimited by using a pattern with a specific length as a header of the entire packet and using a pattern with a different length as a separator. Moreover, signal detection can be facilitated by including this pattern in the middle. Thus, even when one packet is longer than the image capturing time of one frame image, the receiver can connect and decode the data. This also makes it possible to make the packet length variable by adjusting the number of separators. The length of the entire packet may be expressed by the length of the packet header pattern. Moreover, the receiver can synthesize the partially received data by using the separator as a packet header and the length of the separator as the data address.

The transmitter repeatedly transmits the packet configured as described above. The contents of packets 1 to 4 in (c) of FIG. 113 may all be the same, or may be combined as different data on the receiving side.

(Embodiment 14)
In this embodiment, each application example using a receiver such as a smartphone in each of the above embodiments and a transmitter that transmits information as a blinking pattern of an LED or an organic EL will be described.

FIG. 114A is a diagram for describing a transmitter according to the present embodiment.

The transmitter in this embodiment is configured as a backlight of a liquid crystal display, for example, and includes a blue LED 2303 and a phosphor 2310 including a green fluorescent component 2304 and a red fluorescent component 2305.

Blue LED 2303 emits blue (B) light. The phosphor 2310 emits yellow (Y) light when receiving blue light emitted from the blue LED 2303 as excitation light. That is, the phosphor 2310 emits yellow light. Specifically, since the phosphor 2130 includes a green fluorescent component 2304 and a red fluorescent component 2305, yellow light is emitted by the emission of these fluorescent components. Of these two fluorescent components, the green fluorescent component 2304 emits green light when receiving blue light emitted from the blue LED 2303 as excitation light. That is, the green fluorescent component 2304 emits green (G) light. Of the two fluorescent components described above, the red fluorescent component 2305 emits red light when receiving blue light emitted from the blue LED 2303 as excitation light. That is, the red fluorescent component 2305 emits red (R) light. Thereby, since each light of RGB or Y (RG) B is emitted, a transmitter outputs white light as a backlight.

This transmitter transmits a visible light signal of white light by changing the luminance of the blue LED 2303 as in the above embodiments. At this time, a visible light signal having a predetermined carrier frequency is output as the luminance of white light changes.

Here, the barcode reader irradiates the barcode with the red laser beam, and reads the barcode based on the luminance change of the red laser beam reflected from the barcode. The barcode reading frequency of the red laser light may coincide with or approximate the carrier frequency of the visible light signal output from a general transmitter currently in practical use. Therefore, in such a case, when the barcode reader attempts to read a barcode illuminated with white light, which is a visible light signal from the general transmitter, the luminance of the red light contained in the white light Depending on the change, the reading may fail. In other words, a barcode reading error occurs due to interference between the carrier frequency of the visible light signal (particularly red light) and the barcode reading frequency.

Therefore, a fluorescent material having a longer afterglow duration than that of the green fluorescent component 2304 is used for the red fluorescent component 2305 in the present embodiment. That is, the red fluorescent component 2305 in the present embodiment changes in luminance at a frequency sufficiently lower than the luminance change frequency of the blue LED 2303 and the green fluorescent component 2304. In other words, the red fluorescent component 2305 smoothes the frequency of the red luminance change included in the visible light signal.

FIG. 114B is a diagram showing luminance changes of RGB.

Blue light from the blue LED 2303 is included in the visible light signal and output as shown in FIG. 114B (a). As shown in FIG. 114B (b), the green fluorescent component 2304 emits green light when receiving blue light from the blue LED 2303. The duration of afterglow in the green fluorescent component 2304 is short. Therefore, when the blue LED 2303 changes in luminance, the green fluorescent component 2304 emits green light whose luminance changes at substantially the same frequency as the luminance change frequency of the blue LED 2303 (that is, the visible light signal carrier frequency). .

The red fluorescent component 2305 emits red light when receiving blue light from the blue LED 2303 as shown in (c) of FIG. 114B. The duration of the afterglow in the red fluorescent component 2305 is long. Therefore, when the blue LED 2303 changes in luminance, the red fluorescent component 2305 emits red light whose luminance changes at a frequency lower than the frequency of luminance change of the blue LED 2303 (that is, the carrier frequency of the visible light signal). .

FIG. 115 is a diagram showing the afterglow characteristics of the green fluorescent component 2304 and the red fluorescent component 2305 in the present embodiment.

For example, when the blue LED 2303 is lit without a change in luminance, the green fluorescent component 2304 emits green light having an intensity I = I0 without changing the luminance (that is, light having a luminance change frequency f = 0). . Further, even if the blue LED 2303 changes in luminance at a low frequency, the green fluorescent component 2304 emits green light having an intensity I = I0 that changes in luminance at substantially the same frequency f as the low frequency. However, when the luminance of the blue LED 2303 changes at a high frequency, the intensity I of the green light emitted from the green fluorescent component 2304 that changes in luminance at substantially the same frequency f as the high frequency is influenced by the afterglow in the green fluorescent component 2304. Therefore, the intensity becomes smaller than the intensity I0. As a result, the intensity I of the green light emitted from the green fluorescent component 2304 is maintained at I = I0 when the frequency f of the luminance change of the light is less than the threshold value fb, as shown by the dotted line in FIG. However, when the frequency f exceeds the threshold value fb, it gradually decreases.

Further, the duration of afterglow of the red fluorescent component 2305 in this embodiment is longer than the duration of afterglow of the green fluorescent component 2304. Therefore, as shown by the solid line in FIG. 115, the intensity I of the red light emitted from the red fluorescent component 2305 is I = I until the frequency f of the luminance change of the light is less than the threshold fa lower than the threshold fb. Although it is kept at I0, it becomes gradually smaller when the frequency f becomes higher than the threshold value fb. In other words, the red light emitted from the red fluorescent component 2305 does not exist in the high frequency region of the frequency band of the green light emitted from the green fluorescent component 2304 and exists only in the low frequency region.

More specifically, for the red fluorescent component 2305 in the present embodiment, a fluorescent material is used in which the intensity I of red light emitted at the same frequency f as the carrier frequency f1 of the visible light signal is I = I1. It is done. The carrier frequency f1 is a carrier frequency of luminance change by the blue LED 2303 provided in the transmitter. Further, the above-described intensity I1 is 1/3 of the intensity I0 or −10 dB of the intensity I0. For example, the carrier frequency f1 is 10 kHz or 5 to 100 kHz.

That is, the transmitter according to the present embodiment is a transmitter that transmits a visible light signal, and receives a blue LED that emits blue light whose luminance changes as light included in the visible light signal, and the blue light. A green fluorescent component that emits green light as light included in the visible light signal, and a red fluorescent component that emits red light as light included in the visible light signal by receiving the blue light. The duration of afterglow in the red fluorescent component is longer than the duration of afterglow in the green fluorescent component. The green fluorescent component and the red fluorescent component may be included in a single phosphor that receives yellow light and emits yellow light as light included in the visible light signal. Alternatively, the green fluorescent component may be included in a green phosphor, and the red fluorescent component may be included in a red phosphor that is separate from the green phosphor.

Thereby, since the persistence time of the afterglow in the red fluorescent component is long, it is possible to change the luminance of the red light at a frequency lower than the frequency in the luminance change of the blue and green light. Therefore, even if the frequency of the luminance change of blue and green light included in the visible light signal of white light is the same as or close to the barcode reading frequency of the red laser light, it is included in the visible light signal of white light. The frequency of the red light can be significantly different from the barcode reading frequency. As a result, the occurrence of barcode reading errors can be suppressed.

Here, the red fluorescent component may emit red light whose luminance changes at a frequency lower than the frequency of luminance change of the light emitted from the blue LED.

The red fluorescent component may include a red fluorescent material that emits red light by receiving blue light and a low-pass filter that transmits only light in a predetermined frequency band. For example, the low-pass filter transmits only light in a low frequency band out of blue light emitted from the blue LED and applies the light to the red fluorescent material. The red fluorescent material may have the same afterglow characteristics as the green fluorescent component. Alternatively, the low-pass filter transmits only light in a low frequency band out of red light emitted from the red fluorescent material when blue light emitted from the blue LED hits the red fluorescent material. Let Even when such a low-pass filter is used, the occurrence of barcode reading errors can be suppressed as described above.

The red fluorescent component may be made of a fluorescent material having a predetermined afterglow characteristic. For example, predetermined afterglow characteristics are: (a) the intensity of the red light when the frequency f of the luminance change of the red light emitted from the red fluorescent component is 0 is I0, and (b) When the carrier frequency in the luminance change of the light emitted from the blue LED is f1, when the frequency f of the red light is f = f1, the intensity of the red light is 1/3 or less of the I0. Or -10 dB or less.

This ensures that the frequency of the red light contained in the visible light signal can be significantly different from the barcode reading frequency. As a result, the occurrence of barcode reading errors can be reliably suppressed.

The carrier frequency f1 may be approximately 10 kHz.

As a result, since the carrier frequency used for transmitting visible light signals, which is currently in practical use, is 9.6 kHz, it is effective to generate bar code reading errors in this practical transmission of visible light signals. Can be suppressed.

The carrier frequency f1 may be approximately 5 to 100 kHz.

It is assumed that carrier frequencies such as 20 kHz, 40 kHz, 80 kHz, and 100 kHz will be used in future visible light communication due to the advancement of image sensors (imaging devices) of receivers that receive visible light signals. Therefore, by setting the above-mentioned carrier frequency f1 to approximately 5 to 100 kHz, it is possible to effectively suppress the occurrence of barcode reading errors in future visible light communication.

In the present embodiment, regardless of whether the green fluorescent component and the red fluorescent component are contained in a single phosphor, or each of these two fluorescent components is contained in a separate phosphor, The above effects can be achieved. That is, even when a single phosphor is used, the afterglow characteristics, that is, the frequency characteristics, of the red light and the green light emitted from the phosphor are different. Therefore, the above-mentioned effects can also be achieved by using a single phosphor that is inferior in afterglow characteristics or frequency characteristics in red light and superior in afterglow characteristics or frequency characteristics in green light. Note that poor afterglow characteristics or frequency characteristics means that the duration of afterglow is long or the intensity of light in a high frequency band is weak, and that the afterglow characteristics or frequency characteristics are superior means that afterglow characteristics The duration of the light is short, or the light intensity in the high frequency band is high.

Here, in the example shown in FIGS. 114A to 115, the occurrence of barcode reading errors is suppressed by smoothing the frequency of the red luminance change included in the visible light signal, but the carrier frequency of the visible light signal is reduced. By making it high, the occurrence of the reading error may be suppressed.

FIG. 116 is a diagram for explaining a problem newly generated in order to suppress occurrence of a barcode reading error.

As shown in FIG. 116, when the carrier frequency fc of the visible light signal is about 10 kHz, the reading frequency of the red laser light used for reading the barcode is also about 10 to 20 kHz. A barcode reading error occurs.

Therefore, by increasing the carrier frequency fc of the visible light signal from about 10 kHz to, for example, 40 kHz, it is possible to suppress the occurrence of barcode reading errors.

However, if the carrier frequency fc of the visible light signal is about 40 kHz, the sampling frequency fs for the receiver to sample the visible light signal by photographing needs to be 80 kHz or more.

That is, since the sampling frequency fs required in the receiver is high, a new problem arises that the processing load of the receiver increases. In order to solve this new problem, the receiver in this embodiment performs downsampling.

FIG. 117 is a diagram for explaining the downsampling performed by the receiver in this embodiment.

The transmitter 2301 in the present embodiment is configured as, for example, a liquid crystal display, digital signage, or a lighting device. The transmitter 2301 then outputs a frequency-modulated visible light signal. At this time, the transmitter 2301 switches the carrier frequency fc of the visible light signal to, for example, 40 kHz and 45 kHz.

The receiver 2302 in this embodiment photographs the transmitter 2301 at a frame rate of 30 fps, for example. At this time, similarly to the receiver in each of the above embodiments, the receiver 2302 performs imaging with a short exposure time so that bright lines are generated in each image (specifically, each frame) obtained by imaging. The image sensor used for photographing by the receiver 2302 has, for example, 1000 exposure lines. Accordingly, in one-frame shooting, visible light signals are sampled by starting exposure at different timings for 1000 exposure lines. As a result, in 1 second, 30 fps × 1000 lines = 30000 samplings (30 ks / second) are performed. In other words, the sampling frequency fs of the visible light signal is 30 kHz.

According to a general sampling theorem, at a sampling frequency fs = 30 kHz, only a visible light signal having a carrier frequency of 15 kHz or less can be demodulated.

However, the receiver 2302 in this embodiment downsamples a visible light signal with a sampling frequency fs = 30 kHz and a carrier frequency fc = 40 kHz or 45 kHz. Although aliasing occurs in the frame by this downsampling, the receiver 2302 in this embodiment estimates the carrier frequency fc of the visible light signal by observing and analyzing the aliasing.

FIG. 118 is a flowchart showing a processing operation of the receiver 2302 in this embodiment.

First, the receiver 2302 performs down-sampling with a sampling frequency fs = 30 kHz on a visible light signal with a carrier frequency fc = 40 kHz or 45 kHz by photographing a subject (step S2310).

Next, the receiver 2302 observes and analyzes an alias generated in the frame obtained by the downsampling (step S2311). As a result, the receiver 2302 specifies the frequency of the alias as, for example, 5.1 kHz or 5.5 kHz.

Then, the receiver 2302 estimates the carrier frequency fc of the visible light signal based on the identified alias frequency (step S2311). That is, the receiver 2302 restores the original frequency from the alias. Thereby, the receiver 2302 estimates the carrier frequency fc of the visible light signal as 40 kHz or 45 kHz, for example.

As described above, the receiver 2302 in this embodiment can appropriately receive a visible light signal having a high carrier frequency by performing downsampling and restoration of a frequency based on an alias. For example, the receiver 2302 can receive a visible light signal having a carrier frequency of 30 kHz to 60 kHz even if the sampling frequency is fs = 30 kHz. Accordingly, the carrier frequency of the visible light signal can be increased from 30 kHz to 60 kHz from the frequency (about 10 kHz) currently in practical use. As a result, the carrier frequency of the visible light signal and the barcode reading frequency (10 to 20 kHz) can be greatly different, and interference between the frequencies can be suppressed. As a result, the occurrence of barcode reading errors can be suppressed.

Such a receiving method in the present embodiment is a receiving method for acquiring information from a subject, and corresponds to a plurality of exposure lines included in the image sensor in a frame obtained by photographing the subject by an image sensor. An exposure time setting step for setting an exposure time of the image sensor so that a plurality of bright lines are generated according to a change in luminance of the subject, and exposure of each of the plurality of exposure lines included in the image sensor at different times sequentially The image sensor causes the image sensor to shoot the subject whose luminance changes at a predetermined frame rate and at the set exposure time, and for each frame obtained by the shooting. , Specified by the plurality of bright line patterns included in the frame Including an information acquisition step of acquiring information by demodulating the over data. In the photographing step, each of the plurality of exposure lines sequentially repeats starting exposure at different times, so that the sampling frequency is lower than the carrier frequency of the visible light signal transmitted by the luminance change of the subject. The visible light signal is down-sampled, and in the information acquisition step, the frequency of the alias specified by the pattern of the plurality of bright lines included in the frame is specified and specified for each frame obtained by the imaging. The information is obtained by estimating the frequency of the visible light signal from the frequency of the alias and demodulating the estimated frequency of the visible light signal.

In such a receiving method, a visible light signal having a high carrier frequency can be appropriately received by performing downsampling and frequency restoration based on alias.

In the downsampling, a visible light signal having a carrier frequency higher than 30 kHz may be downsampled. Accordingly, interference between the carrier frequency of the visible light signal and the barcode reading frequency (10 to 20 kHz) can be avoided, and barcode reading errors can be more effectively suppressed.

(Embodiment 15)
FIG. 119 is a diagram illustrating processing operations of the reception device (imaging device). Specifically, FIG. 119 is a diagram for describing an example of switching processing between the normal imaging mode and the macro imaging mode when receiving visible light communication.

Here, the reception device 1610 receives visible light emitted from a transmission device including a plurality of light sources (four light sources in FIG. 119).

First, when the receiving device 1610 transitions to a mode for performing visible light communication, the receiving device 1610 activates the imaging unit in the normal imaging mode (S1601). Note that the receiving device 1610 displays a frame 1611 for imaging a light source on the screen when the mode is changed to a mode for performing visible light communication.

After a predetermined time, the receiving device 1610 switches the imaging mode of the imaging unit to the macro imaging mode (S1602). Note that the timing of switching from step S1601 to step S1602 may not be after a predetermined time from step S1601, but may be when the receiving device 1610 determines that the light source is captured within the frame 1611. By switching to the macro imaging mode in this way, the user can easily place the light source in the frame 1611 because the light source can be stored in the frame 1611 with a clear image in the normal imaging mode before the image is blurred in the macro imaging mode. Can fit in.

Next, the receiving device 1610 determines whether or not a signal from the light source has been received (S1603). If it is determined that the signal from the light source is received (Yes in S1603), the process returns to the normal imaging mode in Step S1601, and if it is determined that the signal from the light source is not received (No in S1603), the macro imaging in Step 1602 is performed. Continue mode. In the case of Yes in step S1603, processing based on the received signal (for example, processing for displaying an image indicated by the received signal) may be performed.

According to the receiving device 1610, the user can take an image in a blurred state by switching from the normal imaging mode to the macro imaging mode by touching the display unit of the light source 1611 of the smartphone with a finger. For this reason, the image captured in the macro imaging mode includes more bright areas than the image captured in the normal imaging mode. In particular, since light from two light sources overlaps between two adjacent light sources among a plurality of light sources, stripe-like images are separated as shown in the left diagram of FIG. The problem that it cannot be received as a signal can be demodulated as a continuous reception signal for forming a continuous stripe as shown in the right figure. Since a long code can be received at once, the response time is shortened. As shown in FIG. 119 (b), when a photographed image is first photographed with a normal shutter and a normal focus, a beautiful normal image is obtained. However, if the light source is far away, such as text, even if the shutter speed is increased, continuous data cannot be obtained and cannot be demodulated. Next, when the speed of the shutter is increased and the focus driving unit of the lens is set to a short distance (macro), the light source is blurred and spreads, and the four light sources are connected, so that data can be received. Next, when the focus is returned and the shutter speed is returned to normal, the original beautiful image can be obtained. As shown in (c), by recording and displaying a beautiful image in the memory on the display unit, there is an effect that only the beautiful image is displayed on the display unit. The image captured in the macro imaging mode includes more areas brighter than the predetermined brightness than the image captured in the normal imaging mode. Therefore, in the macro imaging mode, the number of exposure lines that can generate bright lines for the subject can be increased.

FIG. 120 is a diagram illustrating processing operations of the receiving device (imaging device). Specifically, FIG. 120 is a diagram for describing another example of the switching process between the normal imaging mode and the macro imaging mode when receiving visible light communication.

Here, the receiving device 1620 receives visible light emitted from a transmitting device including a plurality of light sources (four light sources in FIG. 120).

First, when the receiving device 1620 transitions to a mode for performing visible light communication, the imaging device is activated in the normal imaging mode, and an image 1623 having a wider range than the image 1622 displayed on the screen of the receiving device 1620 is captured. . The image data indicating the captured image 1623 and the posture information indicating the posture of the receiving device 1620 detected by the gyro sensor, the geomagnetic sensor, and the acceleration sensor of the receiving device 1620 when the image 1623 is captured are stored in the memory. (S1611). Note that the captured image 1623 is an image having a wide range by a predetermined width in the vertical direction and the horizontal direction with reference to the image 1622 displayed on the screen of the reception device 1620. In addition, when the receiving apparatus 1620 transitions to a mode for performing visible light communication, the receiving apparatus 1620 displays a frame 1621 for imaging a light source on the screen.

After a predetermined time, the receiving device 1620 switches the imaging mode of the imaging unit to the macro imaging mode (S1612). Note that the timing of switching from step S1611 to step S1612 is not after a predetermined time from step S1611, but when the image 1623 is captured and it is determined that the image data indicating the captured image 1623 is held in the memory. Good. At this time, the receiving device 1620 displays an image 1624 having a size corresponding to the screen size of the receiving device 1620 among the images 1623 based on the image data held in the memory.

Note that an image 1624 displayed on the receiving device 1620 at this time is a part of the image 1623, and the posture of the receiving device 1620 indicated by the posture information acquired in step S1611 (indicated by a white broken line). Position) and the current posture of the receiving device 1620. This is an image of an area predicted to be captured by the current receiving device 1620. That is, the image 1624 is a partial image of the image 1623 and is an image of an area corresponding to the imaging target of the image 1625 actually captured in the macro imaging mode. That is, in step S1612, the posture (imaging direction) changed from the time of step S1611 is acquired, and the imaging target that is presumed to be currently imaged from the acquired current posture (imaging direction) is identified and imaged in advance. An image 1624 corresponding to the current posture (imaging direction) is specified from the image 1623, and processing for displaying the image 1624 is performed. For this reason, as shown by an image 1623 in FIG. 120, when the receiving device 1620 moves from the position indicated by the white broken line in the direction of the white arrow, the receiving device 1620 cuts out the image 1623 according to the amount of movement. An area 1624 can be determined, and an image 1624 that is the image 1623 in the determined area can be displayed.

Accordingly, the reception device 1620 does not display the image 1625 captured in the macro imaging mode even when capturing in the macro imaging mode, and starts from the image 1623 captured in the clearer normal imaging mode. An image 1624 cut out in accordance with the current posture of the receiving device 1620 can be displayed. In the method of the present invention in which continuous visible light information is obtained from a plurality of light sources that are separated from a defocused image, the stored normal plane image is displayed on the display unit, and the user takes a picture using a smartphone. In this case, camera shake occurs, and the direction of the actual captured image and the direction of the still image displayed from the memory shifts, and a problem that the user cannot adjust the direction to the target light source is expected to occur. In this case, countermeasures are necessary because data from the light source cannot be received. However, according to the present invention, even if the camera shake is detected, the camera shake is detected by the image swing detection means or the swing gyro detection means, and the target image in the still image is shifted in a predetermined direction. The user can see the deviation from the direction. This display makes it possible for the user to point the camera at the target light source, so that multiple divided light sources can be photographed while displaying normal images, and signals are received continuously. can do. Thereby, since the normal image is displayed, the light source divided into a plurality of parts can be received. In this case, it is possible to easily adjust the posture of the reception device 1620 so that a plurality of light sources fits the frame 1621. Note that when the focal point is blurred, the light source is dispersed and the luminance is equivalently reduced. Therefore, there is an effect that the visible light data can be received more reliably by increasing the sensitivity of the ISO of the camera.

Next, the receiving device 1620 determines whether or not a signal from the light source has been received (S1613). If it is determined that the signal from the light source has been received (Yes in S1613), the process returns to the normal imaging mode in Step S1611. If it is determined that the signal from the light source has not been received (No in S1613), the macro imaging in Step 1612 is performed. Continue mode. In the case of Yes in step S1613, processing based on the received signal (for example, processing for displaying an image indicated by the received signal) may be performed.

As with the receiving device 1610, the receiving device 1620 can pick up an image including a brighter region in the macro image pickup mode. For this reason, in the macro imaging mode, the number of exposure lines that can generate bright lines for the subject can be increased.

121 is a diagram showing processing operations of the receiving device (imaging device).

Here, the transmission device 1630 is a display device such as a television, for example, and transmits different transmission IDs by visible light communication at a predetermined time interval Δ1630. Specifically, at times t1631, t1632, t1633, and t1634, ID1631, ID1632, ID1633, and ID1634, which are transmission IDs associated with the data corresponding to the displayed images 1631, 1632, 1633, and 1634, respectively, are transmitted. To do. That is, ID 1631 to ID 1634 are transmitted one after another at a predetermined time interval Δt 1630 from the transmission device 1630.

The receiving device 1640 requests the data associated with each transmission ID to the server 1650 based on the transmission ID received by visible light communication, receives the data from the server, and displays an image corresponding to the data. Specifically, images 1641, 1642, 1643, and 1644 respectively corresponding to ID1631, ID1632, ID1633, and ID1634 are displayed at times t1631, t1632, t1633, and t1634, respectively.

When the receiving apparatus 1640 acquires the ID 1631 received at time t1631, the receiving apparatus 1640 may acquire ID information indicating a transmission ID scheduled to be transmitted from the transmitting apparatus 1630 from the server 1650 at subsequent times t1632 to t1634. In this case, the receiving device 1640 uses the acquired ID information, so that the data associated with ID 1632 to ID 1634 at times t1632 to t1634 can be stored in the server 1650 without receiving a transmission ID from the transmitting device 1630 each time. The received data can be displayed at times t1632 to t1634.

Also, the receiving device 1640 may request data corresponding to the ID 1631 at time t1631 without acquiring information indicating the transmission ID scheduled to be transmitted from the transmitting device 1630 from time t1632 to t1634 from the server 1650. The data associated with the transmission ID corresponding to the subsequent times t1632 to t1634 may be received from the server 1650, and the received data may be displayed at each time t1632 to t1634. That is, when server 1650 receives a request for data associated with ID 1631 transmitted at time t1631 from reception device 1640, server 1650 receives data associated with a transmission ID corresponding to subsequent times t1632 to t1634. Even if there is no request from 1640, transmission is made to receiving apparatus 1640 at times t1632 to t1634. That is, in this case, the server 1650 holds association information in which each time t1631 to 1634 and data associated with the transmission ID corresponding to each time t1631 to 1634 are associated, and based on the association information The predetermined data associated with the predetermined time is transmitted at the predetermined time.

In this way, if receiving apparatus 1640 can obtain transmission ID 1631 by visible light communication at time t1631, data corresponding to each time t1632 to t1634 from server 1650 can be obtained at subsequent times t1632 to t1634 without performing visible light communication. Can be received. For this reason, the user does not need to keep the receiving device 1640 directed at the transmitting device 1630 in order to acquire the transmission ID by visible light communication, and can easily display the data acquired from the server 1650 on the receiving device 1640. In this case, when the receiving device 1640 obtains data corresponding to the ID from the server every time, the time delay from the server occurs and the response time becomes longer. Therefore, in order to speed up the response, data corresponding to the ID is stored in advance in the storage unit of the receiver from the server or the like, and the response time is displayed by displaying the data corresponding to the ID in the storage unit. Can be quick. In this method, if the time information for outputting the next ID is included in the transmission signal from the visible light transmitter, the receiver side can obtain the time even if the visible light signal is not continuously received. In this case, since it is possible to know the transmission time of the next ID, there is an effect that it is not necessary to keep the receiving device directed toward the light source. This system only synchronizes the time information (clock) on the transmitter side with the time information (clock) on the receiver side when receiving visible light. However, there is an effect that a screen synchronized with the transmitter can be continuously displayed.

In the above-described example, the receiving device 1640 displays images 1641, 1642, 1643, and 1644 corresponding to the transmission IDs ID1631, ID1632, ID1633, and ID1634, respectively, at times t1631, t1632, t1633, and t1634. Displayed respectively. Here, as illustrated in FIG. 122, the reception device 1640 may present not only the image but also other information at each time. That is, at time t1631, the receiving device 1640 displays the image 1641 corresponding to the ID 1631 and outputs sound or sound corresponding to the ID 1631. At this time, the receiving device 1640 may further display, for example, a purchase site for the product displayed in the image. Such sound output and purchase site display are performed in the same manner at times t1632, t1633, and t1634 other than time t1631.

Next, as shown in FIG. 119 (b), in the case of a smartphone equipped with two left and right cameras for stereoscopic display, an image with a normal image quality is displayed with a normal shutter speed and a normal focus for the left eye. At the same time, the right-eye camera uses a shutter that is faster than the left eye and / or is set to a focal point or macro at a short distance to obtain the stripe-like bright line of the present invention and demodulate the data. As a result, an image having normal image quality is displayed on the display unit, and the optical communication data of a plurality of light sources divided in distance can be received by the right eye camera.

(Embodiment 16)
Here, an application example of audio synchronized playback will be described below.

FIG. 123 is a diagram illustrating an example of an application according to the sixteenth embodiment.

For example, a receiver 1800a configured as a smartphone receives a signal (visible light signal) transmitted from a transmitter 1800b configured as, for example, a street digital signage. That is, the receiver 1800a receives the timing of image reproduction by the transmitter 1800b. The receiver 1800a reproduces sound at the same timing as the image reproduction. In other words, the receiver 1800a performs synchronized reproduction of the sound so that the image and sound reproduced by the transmitter 1800b are synchronized. Note that the receiver 1800a may reproduce the same image as the image (reproduced image) reproduced by the transmitter 1800b or a related image related to the reproduced image together with the sound. Further, the receiver 1800a may cause a device connected to the receiver 1800a to reproduce sound and the like. Further, after receiving the visible light signal, the receiver 1800a may download content such as sound or related images associated with the visible light signal from the server. The receiver 1800a performs synchronous reproduction after the download.

Thereby, even when the sound from the transmitter 1800b cannot be heard or when the sound from the transmitter 1800b is not reproduced because the street sound reproduction is prohibited, the user can select the sound that matches the display of the transmitter 1800b. Can hear. Further, even when there is a distance that takes time to reach the voice, it is possible to listen to the voice that matches the display.

Here, multilingual support by synchronized playback is described below.

FIG. 124 is a diagram illustrating an example of an application according to the sixteenth embodiment.

Each of the receiver 1800a and the receiver 1800c obtains and reproduces audio corresponding to a video such as a movie displayed on the transmitter 1800d from the server, in the language set in the receiver. To do. Specifically, the transmitter 1800d transmits a visible light signal indicating an ID for identifying the displayed video to the receiver. When the receiver receives the visible light signal, the receiver transmits a request signal including the ID indicated in the visible light signal and the language set in the receiver to the server. The receiver acquires the audio corresponding to the request signal from the server and reproduces it. Thereby, the user can enjoy the work displayed on the transmitter 1800d in the language set by the user.

Here, the audio synchronization method will be described below.

125 and 126 are diagrams showing an example of a transmission signal and an example of a voice synchronization method in the sixteenth embodiment.

Different data (for example, data shown in FIG. 125: 1 to 6 and the like) are associated with a time every fixed time (N seconds). These data may be, for example, an ID for identifying time, may be time, or may be audio data (for example, 64 Kbps data). The following description is based on the assumption that the data is an ID. Different IDs may have different additional information parts attached to the ID.

It is desirable that the packets that make up the ID are different. Therefore, it is desirable that IDs are not continuous. Alternatively, it is desirable to use a packetizing method in which the discontinuous portion is configured as one packet when the ID is packetized. Since error correction signals tend to have different patterns even with consecutive IDs, the error correction signals may be configured to be distributed in a plurality of packets rather than being combined into one packet.

The transmitter 1800d transmits the ID in accordance with the reproduction time of the displayed image, for example. The receiver can recognize the reproduction time (synchronization time) of the image of the transmitter 1800d by detecting the timing when the ID is changed.

In the case of (a), since the change time points of ID: 1 and ID: 2 are received, the synchronization time can be accurately recognized.

When the time N during which the ID is transmitted is long, there are few such opportunities, and the ID may be received as shown in (b). Even in this case, the synchronization time can be recognized by the following method.

(B1) It is assumed that the midpoint of the receiving section where the ID has changed is the ID changing point. Further, the time after an integer multiple of the time N from the ID change point estimated in the past is also estimated as the ID change point, and the midpoint of the plurality of ID change points is estimated as a more accurate ID change point. With such an estimation algorithm, an accurate ID change point can be gradually estimated.

(B2) In addition to the above, it is possible that the reception section where the ID has not changed and the time after an integer multiple of the time N are gradually included in the ID change point by estimating that no ID change point is included. There are fewer sections of the ID, and an accurate ID change point can be estimated.

Ｎ By setting N to 0.5 seconds or less, it can be synchronized accurately.

設定 By setting N to 2 seconds or less, it is possible to synchronize without causing the user to feel a delay.

¡By setting N to 10 seconds or less, it is possible to synchronize while suppressing wasted ID.

FIG. 126 is a diagram illustrating an example of a transmission signal in the sixteenth embodiment.

In FIG. 126, waste of ID can be avoided by performing synchronization by time packets. A time packet is a packet that holds the time of transmission. When it is necessary to express a long time, the time packet is divided into a time packet 1 representing a fine time and a time packet 2 representing a rough time. For example, time packet 2 indicates the hour and minute of the time, and time packet 1 indicates only the second of the time. A packet indicating the time may be divided into three or more time packets. Since the coarse time is less necessary, the receiver can recognize the synchronization time quickly and accurately by transmitting more fine time packets than coarse time packets.

That is, in the present embodiment, the visible light signal includes the second information (hour packet 2) indicating the hour and minute of the time, and the first information (time packet 1) indicating the second of the time. The time when the visible light signal is transmitted from the transmitter 1800d is indicated. The receiver 1800a receives the second information and receives the first information more times than the number of times of receiving the second information.

Here, the synchronization time adjustment will be described below.

FIG. 127 is a diagram illustrating an example of a process flow of the receiver 1800a according to the sixteenth embodiment.

Since it takes a certain amount of time for the audio or video to be played back after the signal is transmitted and processed by the receiver 1800a, it is possible to accurately reproduce the audio or video in anticipation of this processing time. Synchronous playback can be performed.

First, a processing delay time is designated for the receiver 1800a (step S1801). This may be stored in the processing program or specified by the user. When the user performs correction, it is possible to realize more accurate synchronization according to the individual receiver. This processing delay time can be synchronized more accurately by changing it depending on the receiver model, the temperature of the receiver, and the CPU usage rate.

The receiver 1800a determines whether or not a time packet has been received or whether or not an ID associated for voice synchronization has been received (step S1802). Here, when the receiver 1800a determines that it has been received (Y in step S1802), it further determines whether there is an image waiting for processing (step S1804). If it is determined that there is an image waiting for processing (Y in step S1804), the receiver 1800a discards the image waiting for processing or delays processing of the image waiting for processing to receive from the latest acquired image. Is performed (step S1805). Thereby, it is possible to avoid an unexpected delay due to the amount of waiting for processing.

The receiver 1800a measures the position in the image where the visible light signal (specifically the bright line) is located (step S1806). In other words, by measuring the position in the direction perpendicular to the exposure line from the first exposure line in the image sensor, the time difference (delay time in the image) from the image acquisition start time to the signal reception time is obtained. Can be calculated.

The receiver 1800a can accurately perform synchronized reproduction by reproducing the sound or moving image at the time obtained by adding the processing delay time and the in-image delay time to the recognized synchronization time (step S1807).

On the other hand, if it is determined in step S1802 that the receiver 1800a has not received the time packet or the voice synchronization ID, the receiver 1800a receives a signal from the image obtained by imaging (step S1803).

128 is a diagram illustrating an example of a user interface of the receiver 1800a in Embodiment 16.

The user can adjust the processing delay time described above by pressing any of the buttons Bt1 to Bt4 displayed on the receiver 1800a as shown in FIG. 128 (a). Alternatively, the processing delay time may be set by a swipe operation as shown in FIG. 128 (b). Thereby, synchronous reproduction can be performed more accurately based on the user's sense.

Here, the earphone-only playback will be described below.

FIG. 129 is a diagram illustrating an example of a process flow of the receiver 1800a according to the sixteenth embodiment.

The earphone-only playback shown by this processing flow enables audio playback without disturbing the surroundings.

The receiver 1800a checks whether or not the setting limited to the earphone is performed (step S1811). When the setting limited to the earphone is performed, for example, the setting limited to the earphone is set in the receiver 1800a. Alternatively, settings that are limited to earphones are made in the received signal (visible light signal). Alternatively, it is recorded in the server or the receiver 1800a in association with the received signal that it is limited to the earphone.

When it is confirmed that the receiver 1800a is limited to the earphone (Y in step S1811), it is determined whether or not the earphone is connected to the receiver 1800a (step S1813).

When the receiver 1800a confirms that the earphone is not limited (N in Step S1811) or determines that the earphone is connected (Y in Step S1813), the receiver 1800a reproduces the sound (Step S1812). When playing back audio, the receiver 1800a adjusts the volume so that the volume is within the set range. This setting range is set similarly to the setting limited to the earphone.

If the receiver 1800a determines that the earphone is not connected (N in step S1813), the receiver 1800a performs a notification prompting the user to connect the earphone (step S1814). This notification is performed by, for example, screen display, audio output, or vibration.

In addition, when the setting for prohibiting forced audio reproduction is not set, the receiver 1800a prepares an interface for forced reproduction and determines whether or not the user has performed an operation of forced reproduction ( Step S1815). If it is determined that the forced playback operation has been performed (Y in step S1815), the receiver 1800a plays back the audio even when the earphone is not connected (step S1812).

On the other hand, if it is determined that the forced regeneration operation is not performed (N in step S1815), the receiver 1800a retains the audio data received in advance and the analyzed synchronization time so that the earphone is connected. Quickly synchronize audio playback.

FIG. 130 is a diagram illustrating another example of the process flow of the receiver 1800a according to the sixteenth embodiment.

First, the receiver 1800a receives an ID from the transmitter 1800d (step S1821). That is, the receiver 1800a receives a visible light signal indicating the ID of the transmitter 1800d or the ID of the content displayed on the transmitter 1800d.

Next, the receiver 1800a downloads information (content) associated with the received ID from the server (step S1822). Alternatively, the receiver 1800a reads out the information from the data holding unit in the receiver 1800a. Hereinafter, this information is referred to as related information.

Next, the receiver 1800a determines whether or not the synchronous reproduction flag included in the related information indicates ON (step S1823). If it is determined that the synchronous reproduction flag does not indicate ON (N in step S1823), the receiver 1800a outputs the content indicated by the related information (step S1824). That is, when the content is an image, the receiver 1800a displays an image, and when the content is audio, the receiver 1800a outputs audio.

On the other hand, when receiver 1800a determines that the synchronous reproduction flag indicates ON (Y in step S1823), is the time adjustment mode included in the related information set to the transmitter reference mode? Then, it is determined whether or not the absolute time mode is set (step S1825). If it is determined that the absolute time mode is set, the receiver 1800a determines whether or not the last time adjustment has been performed within a certain time from the current time (step S1826). The time adjustment at this time is processing for obtaining time information by a predetermined method and using the time information to adjust the time of a clock provided in the receiver 1800a to the absolute time of the reference clock. The predetermined method is, for example, a method using a GPS (Global Positioning System) radio wave or an NTP (Network Time Protocol) radio wave. Note that the current time described above may be a time when the receiver 1800a, which is a terminal device, receives a visible light signal.

If the receiver 1800a determines that the last time adjustment has been performed within a certain time (Y in step S1826), the receiver 1800a outputs the related information based on the time of the clock of the receiver 1800a, and is displayed on the transmitter 1800d. Content and related information are synchronized (step S1827). When the content indicated by the related information is, for example, a moving image, the receiver 1800a displays the moving image so as to be synchronized with the content displayed on the transmitter 1800d. When the content indicated by the related information is, for example, audio, the receiver 1800a outputs the audio so as to be synchronized with the content displayed on the transmitter 1800d. For example, when the related information indicates sound, the related information includes each frame constituting the sound, and these frames are time stamped. The receiver 1800a outputs a sound synchronized with the content of the transmitter 1800d by playing back a frame with a type stamp corresponding to the time of its own clock.

If the receiver 1800a determines that the last time adjustment has not been performed within a certain time (N in step S1826), the receiver 1800a attempts to obtain the time information by a predetermined method, and whether or not the time information has been obtained. Is determined (step S1828). If it is determined that the time information has been obtained (Y in step S1828), the receiver 1800a updates the time of the clock of the receiver 1800a using the time information (step S1829). Then, the receiver 1800a executes the process of step S1827 described above.

If it is determined in step S1825 that the time adjustment mode is the transmitter reference mode, or if it is determined in step S1828 that time information could not be obtained (N in step S1828), the receiver 1800a The time information is acquired from the transmitter 1800d (step S1830). That is, the receiver 1800a acquires time information that is a synchronization signal from the transmitter 1800d through visible light communication. For example, the synchronization signals are time packet 1 and time packet 2 shown in FIG. Alternatively, the receiver 1800a acquires time information from the transmitter 1800d by radio waves such as Bluetooth (registered trademark) or Wi-Fi. Then, the receiver 1800a executes the processes of steps S1829 and S1827 described above.

In the present embodiment, as in steps S1829 and S1830, processing (time adjustment) is performed for synchronization between the clock of the terminal device that is the receiver 1800a and the reference clock by GPS radio waves or NTP radio waves. The time of the terminal device, the time of the terminal device, and the time of the transmitter according to the time indicated by the visible light signal transmitted from the transmitter 1800d. Synchronize between. Accordingly, the terminal device can reproduce the content (moving image or sound) at the timing synchronized with the transmitter-side content reproduced by the transmitter 1800d.

FIG. 131A is a diagram for explaining a specific method of synchronized playback in the sixteenth embodiment. As a method of synchronous reproduction, there are methods a to e shown in FIG. 131A.

(Method a)
In the method a, the transmitter 1800d outputs a visible light signal indicating the content ID and the content playback time by changing the luminance of the display, as in the above embodiments. The content playback time is the playback time of data that is part of the content that is being played back by the transmitter 1800d when the content ID is transmitted from the transmitter 1800d. The data is a picture or a sequence constituting the moving image if the content is a moving image, or a frame constituting the sound if the content is sound. The playback time indicates, for example, the playback time from the beginning of the content as the time. If the content is a moving image, the playback time is included in the content as a PTS (Presentation Time Stamp). That is, the content includes the reproduction time (display time) of the data for each data constituting the content.

The receiver 1800a receives the visible light signal by photographing the transmitter 1800d as in the above embodiments. Then, the receiver 1800a transmits a request signal including the content ID indicated by the visible light signal to the server 1800f. The server 1800f receives the request signal, and transmits the content associated with the content ID included in the request signal to the receiver 1800a.

When the receiver 1800a receives the content, the receiver 1800a plays the content from the time of (content playback time + elapsed time since ID reception). The elapsed time from the reception of the ID is an elapsed time from when the content ID is received by the receiver 1800a.

(Method b)
In the method b, the transmitter 1800d outputs a visible light signal indicating the content ID and the content playback time by changing the luminance of the display, as in the above embodiments. The receiver 1800a receives the visible light signal by photographing the transmitter 1800d as in the above embodiments. Then, the receiver 1800a transmits a request signal including the content ID indicated by the visible light signal and the content playback time to the server 1800f. The server 1800f receives the request signal, and transmits only a part of the content after the content playback time to the receiver 1800a among the content associated with the content ID included in the request signal.

When the receiver 1800a receives the part of the content, the receiver 1800a reproduces the part of the content from the time point (elapsed time since the ID reception).

(Method c)
In the method c, the transmitter 1800d outputs a visible light signal indicating the transmitter ID and the content reproduction time by changing the luminance of the display, as in the above embodiments. The transmitter ID is information for identifying the transmitter.

The receiver 1800a receives the visible light signal by photographing the transmitter 1800d as in the above embodiments. Then, the receiver 1800a transmits a request signal including the transmitter ID indicated by the visible light signal to the server 1800f.

The server 1800f holds, for each transmitter ID, a reproduction schedule that is a timetable of content reproduced by the transmitter with the transmitter ID. Further, the server 1800f includes a clock. When such a server 1800f receives the request signal, the content associated with the transmitter ID included in the request signal and the clock time (server time) of the server 1800f is the content being played back. Identify from the playback schedule. Then, the server 1800f transmits the content to the receiver 1800a.

When the receiver 1800a receives the content, the receiver 1800a plays the content from the time of (content playback time + elapsed time since ID reception).

(Method d)
In the method d, the transmitter 1800d outputs a visible light signal indicating the transmitter ID and the transmitter time by changing the luminance of the display as in the above embodiments. The transmitter time is a time indicated by a clock provided in the transmitter 1800d.

The receiver 1800a receives the visible light signal by photographing the transmitter 1800d as in the above embodiments. Then, the receiver 1800a transmits a request signal including the transmitter ID indicated by the visible light signal and the transmitter time to the server 1800f.

The server 1800f holds the above reproduction schedule. When such a server 1800f receives the request signal, the server 1800f identifies the content associated with the transmitter ID and the transmitter time included in the request signal as the content being reproduced from the reproduction schedule. Furthermore, the server 1800f specifies the content playback time from the transmitter time. That is, the server 1800f finds the playback start time of the specified content from the playback schedule, and specifies the time between the transmitter time and the playback start time as the content playback time. Then, the server 1800f transmits the content and the content playback time to the receiver 1800a.

Upon receiving the content and the content playback time, the receiver 1800a plays the content from the time of (content playback time + elapsed time since reception of ID).

Thus, in the present embodiment, the visible light signal indicates the time when the visible light signal is transmitted from the transmitter 1800d. Therefore, the receiver 1800a, which is a terminal device, can receive content associated with the time (transmitter time) at which the visible light signal is transmitted from the transmitter 1800d. For example, if the transmitter time is 5:43, content played back at 5:43 can be received.

In the present embodiment, the server 1800f has a plurality of contents each associated with a time. However, the content associated with the time indicated by the visible light signal may not exist in the server 1800f. In such a case, the receiver 1800a as the terminal device is closest to the time indicated by the visible light signal and is associated with the time after the time indicated by the visible light signal among the plurality of contents. Content may be received. Thereby, even if the content associated with the time indicated by the visible light signal does not exist in the server 1800f, it is possible to receive appropriate content from among the plurality of contents in the server 1800f.

In addition, the reproduction method according to the present embodiment includes a signal receiving step of receiving a visible light signal by a sensor of the receiver 1800a (terminal device) from a transmitter 1800d that transmits a visible light signal according to a luminance change of the light source, and a receiver 1800a transmits a request signal for requesting the content associated with the visible light signal to the server 1800f, the receiver 1800a receives the content from the server 1800f, and reproduces the content. A playback step. The visible light signal indicates a transmitter ID and a transmitter time. The transmitter ID is ID information. The transmitter time is the time indicated by the clock of the transmitter 1800d, and the time when the visible light signal is transmitted from the transmitter 1800d. In the content reception step, the receiver 1800a receives the content associated with the transmitter ID and the transmitter time indicated by the visible light signal. As a result, the receiver 1800a can reproduce appropriate content with respect to the transmitter ID and the transmitter time.

(Method e)
In the method e, the transmitter 1800d outputs a visible light signal indicating the transmitter ID by changing the luminance of the display as in the above embodiments.

The receiver 1800a receives the visible light signal by photographing the transmitter 1800d as in the above embodiments. Then, the receiver 1800a transmits a request signal including the transmitter ID indicated by the visible light signal to the server 1800f.

The server 1800f holds the above-described reproduction schedule and further includes a clock. When such a server 1800f receives the request signal, the server 1800f identifies the content associated with the transmitter ID and the server time included in the request signal from the reproduction schedule as content being reproduced. The server time is the time indicated by the clock of the server 1800f. Further, the server 1800f finds the reproduction start time of the specified content from the reproduction schedule table. Then, the server 1800f transmits the content and the content reproduction start time to the receiver 1800a.

When the receiver 1800a receives the content and the content playback start time, the receiver 1800a plays the content from the time of (receiver time-content playback start time). The receiver time is a time indicated by a clock provided in the receiver 1800a.

As described above, the reproduction method according to the present embodiment includes a signal receiving step of receiving a visible light signal by a sensor of the receiver 1800a (terminal device) from a transmitter 1800d that transmits a visible light signal due to a luminance change of the light source; The transmitting step of transmitting a request signal for requesting the content associated with the visible light signal from the receiver 1800a to the server 1800f, and the receiver 1800a include each time and data reproduced at each time A content receiving step of receiving content from the server 1800f and a playback step of playing back data corresponding to the time of the clock provided in the receiver 1800a among the content. Therefore, the receiver 1800a can appropriately reproduce the data in the content at the correct time indicated by the content without reproducing the data at the wrong time. Also, in the transmitter 1800d, if content related to the content (transmitter-side content) is reproduced, the receiver 1800a can reproduce the content in synchronization with the transmitter-side content appropriately. .

Note that even in the above methods c to e, like the method b, the server 1800f may transmit only a part of the content after the content playback time to the receiver 1800a.

In the above methods a to e, the receiver 1800a transmits a request signal to the server 1800f and receives necessary data from the server 1800f. However, the data in the server 1800f is transmitted in advance without performing such transmission / reception. You may keep it.

FIG. 131B is a block diagram showing a configuration of a playback apparatus that performs synchronized playback by the method e described above.

The playback device B10 is a receiver 1800a or a terminal device that performs synchronous playback by the method e described above, and includes a sensor B11, a request signal transmission unit B12, a content reception unit B13, a clock B14, and a playback unit B15. I have.

Sensor B11 is, for example, an image sensor, and receives the visible light signal from a transmitter 1800d that transmits a visible light signal according to a change in luminance of the light source. The request signal transmission unit B12 transmits a request signal for requesting content associated with the visible light signal to the server 1800f. The content receiving unit B13 receives content including each time and data reproduced at each time from the server 1800f. The reproduction unit B15 reproduces data corresponding to the time of the clock B14 in the content.

FIG. 131C is a flowchart showing the processing operation of the terminal device that performs synchronous reproduction by the method e described above.

The playback device B10 is a receiver 1800a or a terminal device that performs synchronized playback by the method e described above, and executes each process of steps SB11 to SB15.

In step SB11, the visible light signal is received from the transmitter 1800d that transmits the visible light signal according to the luminance change of the light source. In step SB12, a request signal for requesting content associated with the visible light signal is transmitted to server 1800f. In step SB13, content including each time and data reproduced at each time is received from server 1800f. In step SB15, data corresponding to the time of the clock B14 is reproduced from the content.

As described above, in the playback device B10 and the playback method in the present embodiment, the data in the content can be appropriately played back at the correct time indicated by the content without being played back at the wrong time.

In the present embodiment, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, the software that realizes the playback apparatus B10 and the like of the present embodiment is a program that causes a computer to execute each step included in the flowchart shown in FIG. 131C.

FIG. 132 is a diagram for explaining preparations for synchronized playback in the sixteenth embodiment.

The receiver 1800a adjusts the time of the clock provided in the receiver 1800a to the time of the reference clock in order to perform synchronized playback. For this time adjustment, the receiver 1800a performs the following processes (1) to (5).

(1) The receiver 1800a receives a signal. This signal may be a visible light signal transmitted by a change in luminance of the display of the transmitter 1800d, or a radio wave signal based on Wi-Fi or Bluetooth (registered trademark) from a wireless device. Alternatively, the receiver 1800a acquires position information indicating the position of the receiver 1800a by, for example, GPS instead of receiving such a signal. Then, the receiver 1800a recognizes that the receiver 1800a has entered a predetermined place or building based on the position information.

(2) When the receiver 1800a receives the above signal or recognizes that it has entered a predetermined location, it receives a request signal for requesting data (related information) associated with the signal or location. It transmits to the server (visible light ID resolution server) 1800f.

(3) The server 1800f transmits the above-described data and a time adjustment request for causing the receiver 1800a to adjust the time to the receiver 1800a.

(4) When receiving the data and the time adjustment request, the receiver 1800a transmits the time adjustment request to the GPS time server, the NTP server, or the base station of the telecommunications carrier (carrier).

(5) Upon receiving the time adjustment request, the server or the base station transmits time data (time information) indicating the current time (reference clock time or absolute time) to the receiver 1800a. The receiver 1800a adjusts the time by adjusting the time of the clock provided to the receiver 1800a to the current time indicated by the time data.

As described above, in this embodiment, a GPS (Global Positioning System) radio wave or an NTP (Network Time Protocol) radio wave is used between the clock provided in the receiver 1800a (terminal device) and the reference clock. Synchronized. Therefore, the receiver 1800a can reproduce the data corresponding to the time at an appropriate time according to the reference clock.

FIG. 133 is a diagram illustrating an example of application of the receiver 1800a according to the sixteenth embodiment.

The receiver 1800a is configured as a smartphone as described above, and is used by being held by a holder 1810 formed of, for example, a translucent resin or glass member. The holder 1810 includes a back plate portion 1810a and a locking portion 1810b provided upright on the back plate portion 1810a. The receiver 1800a is inserted between the back plate portion 1810a and the locking portion 1810b so as to be along the back plate portion 1810a.

FIG. 134A is a front view of receiver 1800a held by holder 1810 in Embodiment 16. FIG.

The receiver 1800a is held by the holder 1810 in the inserted state as described above. At this time, the locking portion 1810b locks with the lower portion of the receiver 1800a and sandwiches the lower portion with the back plate portion 1810a. In addition, the back surface of the receiver 1800a faces the back plate portion 1810a, and the display 1801 of the receiver 1800a is exposed.

FIG. 134B is a rear view of receiver 1800a held by holder 1810 in the sixteenth embodiment.

Further, a through hole 1811 is formed in the back plate portion 1810a, and a variable filter 1812 is attached in the vicinity of the through hole 1811. When receiver 1800a is held by holder 1810, camera 1802 of receiver 1800a is exposed through back hole 1811 from back plate portion 1810a. The flashlight 1803 of the receiver 1800a faces the variable filter 1812.

The variable filter 1812 is formed in a disk shape, for example, and has three color filters (a red filter, a yellow filter, and a green filter) each having a fan shape and the same size. The variable filter 1812 is attached to the back plate portion 1810a so as to be rotatable about the center of the variable filter 1812. The red filter is a filter having red translucency, the yellow filter is a filter having yellow translucency, and the green filter is a filter having green translucency.

Therefore, the variable filter 1812 is rotated, and, for example, the red filter is disposed at a position facing the flashlight 1803a. In this case, the light emitted from the flashlight 1803a is diffused inside the holder 1810 as red light by passing through the red filter. As a result, substantially the entire holder 1810 emits red light.

Similarly, the variable filter 1812 is rotated and, for example, the yellow filter is disposed at a position facing the flashlight 1803a. In this case, the light emitted from the flashlight 1803a is diffused inside the holder 1810 as yellow light by passing through the yellow filter. As a result, substantially the entire holder 1810 emits yellow light.

Similarly, the variable filter 1812 is rotated so that, for example, the green filter is disposed at a position facing the flashlight 1803a. In this case, the light emitted from the flashlight 1803a is diffused inside the holder 1810 as green light by passing through the green filter. As a result, substantially the entire holder 1810 emits green light.

That is, the holder 1810 lights in red, yellow or green like a penlight.

FIG. 135 is a diagram for describing a use case of the receiver 1800a held by the holder 1810 in the sixteenth embodiment.

For example, a receiver with a holder that is a receiver 1800a held by a holder 1810 is used in an amusement park or the like. That is, the plurality of receivers with holders that are directed to the float moving in the amusement park blink in synchronization with the music flowing from the float. That is, the float is configured as a transmitter in each of the above embodiments, and transmits a visible light signal by a change in luminance of a light source attached to the float. For example, the float transmits a visible light signal indicating the ID of the float. And the receiver with a holder receives the visible light signal, ie, ID, by imaging | photography with the camera 1802 of the receiver 1800a similarly to said each embodiment. The receiver 1800a that has received the ID acquires a program associated with the ID from, for example, a server. This program includes instructions for turning on the flashlight 1803 of the receiver 1800a at each predetermined time. Each predetermined time is set in accordance with the music flowing from the float (so as to be synchronized). Then, the receiver 1800a blinks the flashlight 1803a according to the program.

Thereby, the holder 1810 of each receiver 1800a that has received the ID repeats lighting at the same timing according to the music flowing from the float of the ID.

Here, each receiver 1800a blinks the flashlight 1803 in accordance with a set color filter (hereinafter referred to as a setting filter). The setting filter is a color filter that faces the flashlight 1803 of the receiver 1800a. Each receiver 1800a recognizes the current setting filter based on an operation by the user. Alternatively, each receiver 1800a recognizes the current setting filter based on the color of an image obtained by photographing with the camera 1802.

That is, among the plurality of receivers 1800a that have received the ID, only the holders 1810 of the plurality of receivers 1800a that recognize that the setting filter is a red filter are lit simultaneously at a predetermined time. At the next time, only the holders 1810 of the plurality of receivers 1800a that recognize that the setting filter is a green filter are lit simultaneously. Further, at the next time, only the holders 1810 of the plurality of receivers 1800a that recognize that the setting filter is a yellow filter are lit simultaneously.

As described above, the receiver 1800a held in the holder 1810 is synchronized with the float music and the receiver 1800a held in the other holder 1810 in the same manner as the synchronous playback shown in FIGS. 123 to 129 described above. Then, the flashlight 1803, that is, the holder 1810 is blinked.

FIG. 136 is a flowchart showing the processing operation of the receiver 1800a held by the holder 1810 in the sixteenth embodiment.

The receiver 1800a receives the float ID indicated by the visible light signal from the float (step S1831). Next, the receiver 1800a acquires a program associated with the ID from the server (step S1832). Next, the receiver 1800a executes the program to turn on the flashlight 1803 at each predetermined time according to the setting filter (step S1833).

Here, the receiver 1800a may cause the display 1801 to display an image corresponding to the received ID or the acquired program.

FIG. 137 is a diagram illustrating an example of an image displayed by the receiver 1800a according to the sixteenth embodiment.

For example, when the receiver 1800a receives an ID from a Santa Claus float, the receiver 1800a displays a Santa Claus image as shown in FIG. Further, as illustrated in FIG. 137 (b), the receiver 1800a may change the background color of the Santa Claus image to the color of the setting filter simultaneously with the lighting of the flashlight 1803. For example, when the color of the setting filter is red, the holder 1810 is lit red by turning on the flashlight 1803, and at the same time, a Santa Claus image having a red background color is displayed on the display 1801. That is, the blinking of the holder 1810 and the display on the display 1801 are synchronized.

FIG. 138 is a diagram showing another example of the holder according to the sixteenth embodiment.

The holder 1820 is configured in the same manner as the holder 1810 described above, but does not include the through hole 1811 and the variable filter 1812. Such a holder 1820 holds the receiver 1800a in a state where the display 1801 of the receiver 1800a is directed to the back plate portion 1820a. In this case, the receiver 1800a causes the display 1801 to emit light instead of the flashlight 1803. As a result, light from the display 1801 is diffused over substantially the entire holder 1820. Therefore, when the receiver 1800a causes the display 1801 to emit light with red light according to the above-described program, the holder 1820 is lit red. Similarly, when the receiver 1800a causes the display 1801 to emit light with yellow light according to the above-described program, the holder 1820 is lit in yellow. When the receiver 1800a causes the display 1801 to emit light with green light according to the above-described program, the holder 1820 lights up in green. If such a holder 1820 is used, the setting of the variable filter 1812 can be omitted.

(Embodiment 17)
(Visible light signal)
139A to 139D are diagrams illustrating an example of a visible light signal in Embodiment 17. FIG.

As described above, the transmitter generates a 4PPM visible light signal and changes the luminance in accordance with the visible light signal, for example, as shown in FIG. 139A. Specifically, the transmitter allocates 4 slots to one signal unit, and generates a visible light signal composed of a plurality of signal units. The signal unit indicates High (H) or Low (L) for each slot. The transmitter emits light brightly in the H slot and emits light darkly or extinguishes in the L slot. For example, one slot is a period corresponding to a time of 1/9600 seconds.

Further, for example, as shown in FIG. 139B, the transmitter may generate a visible light signal in which the number of slots allocated to one signal unit is variable. In this case, the signal unit includes a signal indicating H in one or more consecutive slots and a signal indicating L in one slot following the H signal. Since the number of slots of H is variable, the total number of slots in the signal unit is variable. For example, as shown in FIG. 139B, the transmitter generates a visible light signal including these signal units in the order of a signal unit of 3 slots, a signal unit of 4 slots, and a signal unit of 6 slots. Also in this case, the transmitter emits light brightly in the H slot and emits light darkly or extinguishes in the L slot.

Further, for example, as shown in FIG. 139C, the transmitter may assign an arbitrary period (signal unit period) to one signal unit without assigning a plurality of slots to one signal unit. The signal unit period includes an H period and an L period following the H period. The period of H is adjusted according to the signal before modulation. The period L may be fixed and may be a period corresponding to the slot. The H period and the L period are, for example, periods of 100 μs or more. For example, as shown in FIG. 139C, the transmitter transmits a visible light signal including signal units in the order of a signal unit having a signal unit period of 210 μs, a signal unit having a signal unit period of 220 μs, and a signal unit having a signal unit period of 230 μs. Send. In this case as well, the transmitter emits light brightly during the H period and emits light darkly or extinguishes during the L period.

Further, for example, as shown in FIG. 139D, the transmitter may generate a signal indicating L and H alternately as a visible light signal. In this case, the L period and the H period in the visible light signal are adjusted according to the signals before modulation. For example, as shown in FIG. 139D, the transmitter indicates H for a period of 100 μs, then indicates L for a period of 120 μs, then indicates H for a period of 110 μs, and further indicates L for a period of 200 μs. A visible light signal is transmitted. In this case as well, the transmitter emits light brightly during the H period and emits light darkly or extinguishes during the L period.

FIG. 140 is a diagram showing a configuration of a visible light signal in the seventeenth embodiment.

The visible light signal includes, for example, a signal 1, a brightness adjustment signal corresponding to the signal 1, a signal 2, and a brightness adjustment signal corresponding to the signal 2. When the transmitter generates the signal 1 and the signal 2 by modulating the signals before modulation, the transmitter generates a brightness adjustment signal for the signals and generates the above-described visible light signal.

The brightness adjustment signal corresponding to signal 1 is a signal that compensates for increase / decrease in brightness due to a luminance change according to signal 1. The brightness adjustment signal corresponding to the signal 2 is a signal that compensates for increase / decrease in brightness due to a luminance change according to the signal 2. Here, the brightness B1 is expressed by the luminance change according to the signal 1 and the brightness adjustment signal of the signal 1, and the brightness change according to the signal 2 and the brightness adjustment signal of the signal 2 Brightness B2 is expressed. The transmitter in the present embodiment generates the brightness adjustment signals of signal 1 and signal 2 as part of the visible light signal so that the brightness B1 and brightness B2 are equal. Thereby, the brightness is kept constant and flicker can be suppressed.

Further, when the transmitter 1 generates the signal 1, the transmitter 1 generates the signal 1 including the data 1, the preamble (header) following the data 1, and the data 1 following the preamble. Here, the preamble is a signal corresponding to data 1 arranged before and after the preamble. For example, this preamble is a signal serving as an identifier for reading data 1. Thus, since the signal 1 is composed of the two data 1 and the preamble arranged between them, even if the receiver reads the visible light signal from the middle of the preceding data 1, Data 1 (ie, signal 1) can be correctly demodulated.

(Bright line image)
FIG. 141 is a diagram illustrating an example of bright line images obtained by imaging of the receiver in Embodiment 17.

As described above, the receiver captures a bright line image including a visible light signal transmitted from the transmitter as a bright line pattern by capturing an image of the transmitter that changes in luminance. With such imaging, a visible light signal is received by the receiver.

For example, as shown in FIG. 141, the receiver uses N exposure lines included in the image sensor to capture an image at time t1, so that each line includes a region a and a region b where a bright line pattern appears. Get an image. Regions a and b are regions in which bright line patterns appear when the luminance of the transmitter, which is the subject, changes.

Here, the receiver demodulates the visible light signal from the bright line pattern of the region a and the region b. However, if the receiver determines that the demodulated visible light signal alone is not sufficient, only M (M <N) consecutive exposure lines corresponding to the area a are used among the N exposure lines. The image is taken at time t2. Thereby, the receiver acquires a bright line image including only the region a out of the regions a and b. The receiver repeatedly performs such imaging at times t3 to t5. As a result, a visible light signal having a sufficient amount of data from the subject corresponding to the region a can be received at high speed. Further, the receiver captures an image at time t6 using only L (L <N) consecutive exposure lines corresponding to the region b among the N exposure lines. Thereby, the receiver acquires a bright line image including only the region b out of the regions a and b. The receiver repeatedly performs such imaging at times t7 to t9. As a result, a visible light signal having a sufficient amount of data from the subject corresponding to the region b can be received at high speed.

Further, the receiver may acquire a bright line image including only the region a by performing the same imaging at the times t2 to t5 at the times t10 and t11. Further, the receiver may acquire a bright line image including only the region b by performing imaging similar to that at times t6 to t9 at times t12 and t13.

Further, in the above-described example, when the receiver determines that the visible light signal is insufficient, the receiver performs continuous shooting of the bright line image including only the region a from time t2 to t5. If bright lines appear in the image obtained by the above, the above-described continuous shooting may be performed. Similarly, when the receiver determines that the visible light signal is insufficient, the receiver performs continuous shooting of the bright line image including only the region b from time t6 to time t9, which is obtained by imaging at time t1. If bright lines appear in the image, the above-described continuous shooting may be performed. The receiver may alternately perform acquisition of a bright line image including only the region a and acquisition of a bright line image including only the region b.

Note that the M consecutive exposure lines corresponding to the area a are exposure lines that contribute to the generation of the area a, and the L consecutive exposure lines corresponding to the area b are the generation of the area b. Is an exposure line that contributes to

FIG. 142 is a diagram illustrating another example of the bright line image obtained by imaging of the receiver in the seventeenth embodiment.

For example, as shown in FIG. 142, the receiver captures an image at time t1 using N exposure lines included in the image sensor, so that each line includes a region a and a region b where a bright line pattern appears. Get an image. Each of the areas a and b is an area where a bright line pattern appears when the luminance of the transmitter, which is a subject, changes as described above. Further, each of the region a and the region b has a region that overlaps with each other along the direction of the bright line or the exposure line (hereinafter referred to as an overlapping region).

Here, when the receiver determines that the visible light signal demodulated from the bright line pattern of the area a and the area b is insufficient, P (P <N) corresponding to the overlapping area among the N exposure lines. An image is taken at time t2 using only the continuous exposure lines. As a result, the receiver acquires a bright line image including only the overlapping regions of the region a and the region b. The receiver repeatedly performs such imaging at times t3 and t4. As a result, a visible light signal having a sufficient amount of data from the subject corresponding to each of the region a and the region b can be received substantially simultaneously and at high speed.

FIG. 143 is a diagram illustrating another example of the bright line image obtained by imaging by the receiver in the seventeenth embodiment.

For example, as illustrated in FIG. 143, the receiver uses the N exposure lines included in the image sensor to capture an image at time t <b> 1, thereby clearly displaying a portion a in which the bright line pattern appears unclearly. A bright line image including an area including the appearing portion b is acquired. Similar to the above, this region is a region where a bright line pattern appears when the luminance of the transmitter that is the subject changes.

In such a case, when the receiver determines that the visible light signal demodulated from the bright line pattern in the above-described region is insufficient, Q (Q <N) corresponding to the portion b of the N exposure lines. The image is taken at time t2 using only the continuous exposure lines. Thereby, the receiver acquires the bright line image including only the part b in the above-described region. The receiver repeatedly performs such imaging at times t3 and t4. As a result, a visible light signal having a sufficient amount of data from the subject corresponding to the above-described region can be received at high speed.

Further, the receiver may perform continuous shooting of the bright line image including only the part a after the continuous shooting of the bright line image including only the part b.

As described above, when a plurality of regions (or portions) where the bright line pattern appears in the bright line image are included, the receiver assigns an order to each region, and according to the order, only that region is included. Performs continuous shooting of bright line images including. In this case, the order may be an order corresponding to the magnitude of the signal (area or area size) or an order corresponding to the clarity of the bright line. Further, the order may be an order corresponding to the color of light from the subject corresponding to these areas. For example, the first continuous shooting is performed on a region corresponding to red light, and the next continuous shooting is performed on a region corresponding to white light. Further, only continuous shooting of the area with respect to red light may be performed.

(HDR synthesis)
144 is a diagram for describing adaptation of the receiver in Embodiment 17 to a camera system that performs HDR synthesis. FIG.

The vehicle is equipped with a camera system to prevent collisions. This camera system performs HDR (High Dynamic Range) composition using an image obtained by imaging by a camera. By this HDR synthesis, an image with a wide dynamic range of luminance can be obtained. The camera system recognizes surrounding vehicles, obstacles, or people based on this wide dynamic range image.

For example, the camera system has a normal setting mode and a communication setting mode as setting modes. When the setting mode is the normal setting mode, for example, as shown in FIG. 144, the camera system performs four times of imaging at the same 1/100 second shutter speed and at different sensitivities at times t1 to t4. . The camera system performs HDR synthesis using the four images obtained by the four imaging operations.

On the other hand, when the setting mode is the communication setting mode, for example, as shown in FIG. 144, the camera system captures three times from time t5 to t7 at the same 1/100 second shutter speed and with different sensitivity. I do. Furthermore, the camera system captures an image with a maximum sensitivity (for example, ISO = 1600) at a shutter speed of 1/10000 seconds at time t8. The camera system performs HDR synthesis using the three images obtained by the first three images out of the four images. Further, the camera system receives a visible light signal by the last imaging among the above four imaging operations, and demodulates the bright line pattern appearing in the image obtained by the imaging.

In addition, when the setting mode is the communication setting mode, the camera system may not perform HDR synthesis. For example, as shown in FIG. 144, the camera system performs imaging at time t9 with a shutter speed of 1/100 second and with low sensitivity (for example, ISO = 200). Furthermore, the camera system performs imaging three times with a shutter speed of 1/10000 seconds and different sensitivities from time t10 to t12. The camera system recognizes a surrounding vehicle, an obstacle, a person, or the like from an image obtained by the first one of the four images. Further, the camera system receives a visible light signal by the last three imagings among the above four imagings, and demodulates the bright line pattern appearing in the image obtained by the imaging.

In the example shown in FIG. 144, imaging is performed with different sensitivities at times t10 to t12. However, imaging may be performed with the same sensitivity.

Such a camera system can perform HDR synthesis and can also receive a visible light signal.

(Security)
FIG. 145 is a diagram for explaining the processing operation of the visible light communication system in the seventeenth embodiment.

This visible light communication system includes, for example, a transmitter arranged at a cash register, a smartphone as a receiver, and a server. Note that the communication between the smartphone and the server and the communication between the transmitter and the server are each performed via a secure communication line. Communication between the transmitter and the smartphone is performed by visible light communication. The visible light communication system according to the present embodiment ensures security by determining whether a visible light signal from a transmitter is accurately received by a smartphone.

Specifically, the transmitter transmits, for example, a visible light signal indicating the value “100” to the smartphone by changing the luminance at time t1. When the smartphone receives the visible light signal at time t2, the smartphone transmits a radio signal indicating the value “100” to the server. The server receives the radio signal from the smartphone at time t3. At this time, the server performs a process for determining whether or not the value “100” indicated by the radio signal is the value of the visible light signal received by the smartphone from the transmitter. That is, the server transmits, for example, a radio signal indicating a value “200” to the transmitter. The transmitter that has received the radio wave signal transmits a visible light signal indicating the value “200” to the smartphone by changing the luminance at time t4. When the smartphone receives the visible light signal at time t5, the smartphone transmits a radio signal indicating the value “200” to the server. The server receives the radio signal from the smartphone at time t6. The server determines whether or not the value indicated by the received radio signal is the same as the value indicated by the radio signal transmitted at time t3. If they are the same, the server determines that the value “100” indicated by the visible light signal received at time t3 is the value of the visible light signal transmitted from the transmitter to the smartphone and received. On the other hand, if not the same, the server determines that the value “100” indicated by the visible light signal received at time t3 is suspicious as the value of the visible light signal transmitted from the transmitter to the smartphone and received.

This allows the server to determine whether the smartphone has surely received a visible light signal from the transmitter. That is, it is possible to prevent the smartphone from transmitting the signal to the server by making it appear as if it has received the visible light signal even though it has not received the visible light signal from the transmitter.

In the above example, communication using a radio wave signal is performed between the smartphone, the server, and the transmitter. However, communication using an optical signal other than a visible light signal or communication using an electrical signal may be performed. Good. The visible light signal transmitted from the transmitter to the smartphone indicates, for example, a charging value, a coupon value, a monster value, or a bingo value.

(Vehicle related)
146A is a diagram illustrating an example of vehicle-to-vehicle communication using visible light in Embodiment 17. FIG.

For example, the head vehicle recognizes that there is an accident in the direction of travel by a sensor (camera etc.) mounted on the vehicle. When an accident is recognized in this way, the leading vehicle transmits a visible light signal by changing the brightness of the tail lamp. For example, the leading vehicle transmits a visible light signal that prompts the subsequent vehicle to decelerate. When the succeeding vehicle receives the visible light signal by imaging with a camera mounted on the vehicle, the following vehicle decelerates according to the visible light signal and further transmits a visible light signal that prompts the subsequent vehicle to decelerate. To do.

Thus, the visible light signal that prompts deceleration is sequentially transmitted from the head to a plurality of vehicles traveling in a line, and the vehicle that receives the visible light signal decelerates. Since the transmission of the visible light signal to each vehicle is performed quickly, the plurality of vehicles can be decelerated in the same manner at substantially the same time. Therefore, it is possible to reduce traffic congestion due to accidents.

FIG. 146B is a diagram illustrating another example of vehicle-to-vehicle communication using visible light according to the seventeenth embodiment.

For example, the front vehicle may transmit a visible light signal indicating a message (for example, “thank you”) to the subsequent vehicle by changing the brightness of the tail lamp. This message is generated, for example, by a user operation on a smartphone. And a smart phone transmits the signal which shows the message to the above-mentioned previous vehicle. As a result, the preceding vehicle can transmit a visible light signal indicating the message to the subsequent vehicle.

FIG. 147 is a diagram illustrating an example of a method for determining the positions of a plurality of LEDs in the seventeenth embodiment.

For example, a vehicle headlight has a plurality of LEDs (Light Emitting Diodes). The transmitter of this vehicle transmits a visible light signal from each LED by individually changing the brightness of each of the plurality of LEDs of the headlight. Other vehicle receivers receive visible light signals from their LEDs by imaging the vehicle with the headlights.

At this time, the receiver determines the position of each of the plurality of LEDs from the image obtained by the imaging in order to recognize which LED the received visible light signal is transmitted from. . Specifically, the receiver uses an acceleration sensor attached to the same vehicle as the receiver, and uses a plurality of gravitational directions (for example, a downward arrow in FIG. 147) indicated by the acceleration sensor as a reference. Determine the position of each LED.

In the above example, an LED is used as an example of a light-emitting body that changes in luminance, but a light-emitting body other than the LED may be used.

FIG. 148 is a diagram illustrating an example of a bright line image obtained by imaging a vehicle in the seventeenth embodiment.

For example, a receiver mounted on a traveling vehicle acquires a bright line image shown in FIG. 148 by imaging a subsequent vehicle (following vehicle). The transmitter mounted on the following vehicle transmits a visible light signal to the preceding vehicle by changing the brightness of the two headlights of the vehicle. A camera for imaging the rear is attached to the rear part of the front vehicle or the side mirror. The receiver acquires a bright line image by imaging with the camera of the following vehicle as a subject, and demodulates a bright line pattern (visible light signal) included in the bright line image. Thereby, the visible light signal transmitted from the transmitter of the following vehicle is received by the receiver of the preceding vehicle.

Here, the receiver acquires the ID of the vehicle having the headlight, the speed of the vehicle, and the type of the vehicle from each of the visible light signals transmitted and demodulated from the two headlights. If the IDs of the two visible light signals are the same, the receiver determines that the two visible light signals are signals transmitted from the same vehicle. And a receiver specifies the length (distance between lights) between the two headlights which the vehicle has from the model of the vehicle. Further, the receiver measures a distance L1 between two regions where the bright line pattern appears, which is included in the bright line image. Then, the receiver calculates the distance (inter-vehicle distance) from the vehicle on which the receiver is mounted to the following vehicle by triangulation using the distance L1 and the inter-light distance. The receiver determines the risk of collision based on the inter-vehicle distance and the vehicle speed acquired from the visible light signal, and notifies the vehicle driver of a warning corresponding to the determination result. Thereby, the collision of a vehicle can be avoided.

In the above-described example, the receiver specifies the inter-light distance from the vehicle type included in the visible light signal, but may specify the inter-light distance from information other than the vehicle type. In the above-described example, the receiver issues a warning when it is determined that there is a risk of a collision. However, the receiver may output a control signal for causing the vehicle to perform an operation to avoid the risk. Good. For example, the control signal is a signal for accelerating the vehicle or a signal for causing the vehicle to change lanes.

In the above example, the camera images the following vehicle, but may image the oncoming vehicle. In addition, when the receiver determines from the image obtained by imaging by the camera that fog is in the vicinity of the receiver (that is, the vehicle equipped with the receiver), the receiver enters a mode for receiving the visible light signal as described above. May be. As a result, the receiver of the vehicle can identify the position and speed of the oncoming vehicle by receiving the visible light signal transmitted from the headlight of the oncoming vehicle, even if fog is in the vicinity.

FIG. 149 is a diagram illustrating an example of application of the receiver and the transmitter in the seventeenth embodiment. FIG. 149 is a view of the automobile from the back.

For example, a transmitter (car) 7006a having two tail lamps (light emitting unit or light) of a car transmits identification information (ID) of the transmitter 7006a to a receiver configured as a smartphone, for example. When the receiver receives the ID, the receiver acquires information associated with the ID from the server. For example, the information includes the ID of the car or transmitter, the distance between the light emitting parts, the size of the light emitting part, the size of the car, the shape of the car, the weight of the car, the car number, the appearance in front, or the danger. This is information indicating the presence or absence of. In addition, the receiver may acquire these pieces of information directly from the transmitter 7006a.

FIG. 150 is a flowchart illustrating an example of processing operations of the receiver and the transmitter 7006a in the seventeenth embodiment.

The ID of the transmitter 7006a and the information passed to the receiver that has received the ID are associated with each other and stored in the server (Step 7106a). The information to be passed to the receiver includes identification of the size of the light emitting unit that becomes the transmitter 7006a, the distance between the light emitting units, the shape of the object having the transmitter 7006a as a component, the weight, the body number, etc. Information such as numbers, places that are difficult to observe from the receiver, and presence or absence of danger may be included.

The transmitter 7006a transmits the ID (Step 7106b). The transmission content may include the URL of the server and information stored in the server.

The receiver receives information such as the transmitted ID (Step 7106c). The receiver acquires information associated with the received ID from the server (Step 7106d). The receiver displays the received information and the information acquired from the server (Step 7106e).

The receiver and the light emitting unit can be triangulated from the size information of the light emitting unit and the appearance size of the imaged light emitting unit, or from the distance information between the light emitting units and the distance between the imaged light emitting units. Is calculated (Step 7106f). The receiver issues a warning of danger based on information such as the state of the place that is difficult to observe from the receiver and the presence or absence of danger (Step 7106g).

FIG. 151 is a diagram illustrating an example of application of the receiver and the transmitter in the seventeenth embodiment.

For example, a transmitter (car) 7007b having two tail lamps (light emitting unit or light) of a car transmits information of the transmitter 7007b to a receiver 7007a configured as a transmission / reception device of a parking lot, for example. The information of the transmitter 7007b indicates identification information (ID) of the transmitter 7007b, a car number, a car size, a car shape, or a car weight. When receiving the information, the receiver 7007a transmits information indicating whether parking is possible, billing information, or a parking position. Note that the receiver 7007a may receive the ID and acquire information other than the ID from the server.

FIG. 152 is a flowchart illustrating an example of processing operations of the receiver 7007a and the transmitter 7007b according to the seventeenth embodiment. Note that the transmitter 7007b includes an in-vehicle transmitter and an in-vehicle receiver in order to perform not only transmission but also reception.

The ID of the transmitter 7007b and the information passed to the receiver 7007a that has received the ID are associated with each other and stored in the server (parking lot management server) (Step 7107a). Information to be passed to the receiver 7007a includes an identification number such as the shape, weight, and body number of the object having the transmitter 7007b as a component, the identification number of the user of the transmitter 7007b, and information for payment. May be included.

The transmitter 7007b (on-vehicle transmitter) transmits the ID (Step 7107b). The contents of transmission may include the URL of the server and information stored in the server. The parking lot receiver 7007a (parking lot transmission / reception device) transmits the received information to a server (parking lot management server) that manages the parking lot (Step 7107c). The parking lot management server acquires information associated with the ID using the ID of the transmitter 7007b as a key (Step 7107d). The parking lot management server investigates the parking lot availability (Step 7107e).

The parking lot receiver 7007a (parking lot transmission / reception device) transmits whether or not parking is possible, parking position information, or the address of a server holding these pieces of information (Step 7107f). Or a parking lot management server transmits such information to another server. The transmitter (on-vehicle receiver) 7007b receives the information transmitted above (Step 7107g). Or an in-vehicle system acquires these information from another server.

The parking lot management server controls the parking lot so as to facilitate parking (Step 7107h). For example, control of a multistory parking lot is performed. The transmission / reception device of the parking lot transmits the ID (Step 7107i). The in-vehicle receiver (transmitter 7007b) makes an inquiry to the parking lot management server based on the user information of the in-vehicle receiver and the received ID (Step 7107j).

The parking lot management server charges according to the parking time (7107k). The parking lot management server controls the parking lot so that the parked vehicle can be easily accessed (Step 7107m). For example, control of a multistory parking lot is performed. The in-vehicle receiver (transmitter 7007b) displays a map to the parking position and performs navigation from the current location (Step 7107n).

(On the train)
FIG. 153 is a diagram illustrating a configuration of a visible light communication system applied to the inside of a train in Embodiment 17.

The visible light communication system includes, for example, a plurality of lighting devices 1905 arranged in a train, a smartphone 1906 held by a user, a server 1904, and a camera 1903 arranged in the train.

Each of the plurality of lighting devices 1905 is configured as the above-described transmitter, illuminates the light, and transmits a visible light signal by changing the luminance. This visible light signal indicates the ID of the illumination device 1905 that transmits the visible light signal.

The smartphone 1906 is configured as the above-described receiver, and receives a visible light signal transmitted from the lighting device 1905 by imaging the lighting device 1905. For example, when a user is involved in a trouble (for example, a molester or a fight) on a train, the user causes the smartphone 1906 to receive the visible light signal. When the smartphone 1906 receives the visible light signal, the smartphone 1906 notifies the server 1904 of the ID indicated by the visible light signal.

Upon receiving the notification of the ID, the server 1904 specifies the camera 1903 whose imaging range is the range illuminated by the lighting device 1905 identified by the ID. Then, the server 1904 causes the identified camera 1903 to image the range illuminated by the lighting device 1905.

The camera 1903 captures an image in accordance with an instruction from the server 1904 and transmits an image obtained by the image capture to the server 1904.

This makes it possible to acquire an image showing the trouble situation on the train. This image can be used as evidence of trouble.

In addition, the user may cause the server 1904 to transmit an image obtained by imaging with the camera 1903 to the smartphone 1906 by operating the smartphone 1906.

Further, the smartphone 1906 may display an imaging button on the screen, and when the imaging button is touched by the user, a signal for prompting imaging may be transmitted to the server 1904. Thereby, the user can determine the timing of imaging himself.

FIG. 154 is a diagram illustrating a configuration of a visible light communication system applied to a facility such as an amusement park according to the seventeenth embodiment.

The visible light communication system includes, for example, a plurality of cameras 1903 arranged in a facility and a jewelry 1907 attached to a person.

The accessory 1907 is, for example, a headband having a ribbon to which a plurality of LEDs are attached. Moreover, this accessory 1907 is comprised as an above-mentioned transmitter, and transmits a visible light signal by changing the brightness | luminance of several LED.

Each of the plurality of cameras 1903 is configured as the above-described receiver and has a visible light communication mode and a normal imaging mode. In addition, each of the plurality of cameras 1903 is disposed at a different location on the path in the facility.

Specifically, when the accessory 1907 is imaged as a subject when the camera 1903 is set to the visible light communication mode, the camera 1903 receives a visible light signal from the accessory 1907. Upon receiving the visible light signal, the camera 1903 switches the set mode from the visible light communication mode to the normal imaging mode. As a result, the camera 1903 images a person wearing the accessory 1907 as a subject.

Therefore, when a person wearing the accessory 1907 is walking along the street in the facility, the camera 1903 near the person sequentially captures the person. As a result, it is possible to automatically acquire and save an image showing the person enjoying at the facility.

Note that the camera 1903 may perform imaging in the normal imaging mode instead of performing imaging in the normal imaging mode immediately after receiving a visible light signal, for example, when receiving an instruction to start imaging from a smartphone. Accordingly, the user can cause the camera 1903 to image itself at the timing when the user touches the imaging start button displayed on the screen of the smartphone.

FIG. 155 is a diagram illustrating an example of a visible light communication system including a playground equipment and a smartphone according to the seventeenth embodiment.

The play equipment 1901 is configured as the above-described transmitter having a plurality of LEDs, for example. That is, the playground equipment 1901 transmits a visible light signal by changing the luminance of the plurality of LEDs.

The smartphone 1902 receives the visible light signal transmitted from the play equipment 1901 by imaging the play equipment 1901. Then, as shown in FIG. 155 (a), when the smartphone 1902 receives the visible light signal for the first time, the smartphone 1902 displays the moving image 1 associated with the visible light signal and the first time from, for example, a server or the like. Download and play. On the other hand, when the smartphone 1902 receives the visible light signal for the second time, as shown in FIG. 155 (b), the smartphone 1902 displays the moving picture 2 associated with the visible light signal and the second time from, for example, a server or the like. Download and play.

That is, even if the smartphone 1902 receives the same visible light signal, the smartphone 1902 switches the reproduced video according to the number of times the visible light signal is received. The number of times the visible light signal is received may be counted by the smartphone 1902 or may be counted by the server. Or even if the smartphone 1902 receives the same visible light signal a plurality of times, the smartphone 1902 does not continuously reproduce the same moving image. Alternatively, the smartphone 1902 reduces the appearance probability of a video that has already been played among a plurality of videos associated with the same visible light signal, and preferentially downloads and plays a video with a high appearance probability. May be.

Further, the smartphone 1902 may receive a visible light signal transmitted from a touch panel provided in a guide of a facility having a plurality of stores, and may display an image corresponding to the visible light signal. For example, when the touch panel displays an initial screen showing the outline of the facility, the touch panel transmits a visible light signal indicating the outline of the facility by a change in luminance. Therefore, when the smartphone receives the visible light signal by capturing an image of the touch panel displaying the initial screen, the smartphone can display an image showing an outline of the facility on its display. Here, when the touch panel is operated by the user, the touch panel displays, for example, a store image indicating information on a specific store. At this time, the touch panel transmits a visible light signal indicating the information of the specific store. Therefore, the smart phone can display the store image which shows the information of a specific store, if the visible light signal is received by imaging the touch panel which displays the store image. Thus, the smartphone can display an image synchronized with the touch panel.

(Summary of the above embodiment)
The reproduction method according to an aspect of the present invention includes a signal receiving step of receiving the visible light signal by a sensor of a terminal device from a transmitter that transmits a visible light signal according to a luminance change of a light source, and the visible light signal from the terminal device. A transmission step of transmitting a request signal for requesting the content associated with the optical signal to the server, and the terminal device transmits content including each time and data reproduced at each time from the server. A content receiving step of receiving, and a playback step of playing back data corresponding to a time of a clock provided in the terminal device among the content.

Thereby, as shown in FIG. 131C, content including each time and data reproduced at each time is received by the terminal device, and data corresponding to the clock time of the terminal device is reproduced. Therefore, the terminal device can appropriately reproduce the data in the content at the correct time indicated by the content without reproducing the data at the wrong time. Specifically, as in the method e of FIG. 131A, the receiver as the terminal device reproduces the content from the time of (receiver time−content reproduction start time). The data corresponding to the clock time of the terminal device described above is data at the time of (receiver time−content reproduction start time) in the content. Also, in the transmitter, if content related to the content (transmitter-side content) is being played back, the terminal device can play back the content appropriately synchronized with the transmitter-side content. The content is sound or image.

Further, the clock provided in the terminal device and the reference clock may be synchronized with each other by GPS (Global Positioning System) radio waves or NTP (Network Time Protocol) radio waves.

As a result, as shown in FIGS. 130 and 132, since the clock of the terminal device (receiver) is synchronized with the reference clock, the time corresponds to the appropriate time according to the reference clock. Can be played back.

Further, the visible light signal may indicate a time when the visible light signal is transmitted from the transmitter.

Thereby, as shown in the method d in FIG. 131A, the terminal device (receiver) can receive the content associated with the time (transmitter time) at which the visible light signal is transmitted from the transmitter. For example, if the transmitter time is 5:43, content played back at 5:43 can be received.

Further, in the reproduction method, the time at which the process for synchronizing the clock of the terminal device and the reference clock is performed by the GPS radio wave or the NTP radio wave is determined by the terminal device as the visible signal. If it is before a predetermined time from the time when the optical signal is received, synchronization is performed between the clock of the terminal device and the clock of the transmitter according to the time indicated by the visible light signal transmitted from the transmitter. It may be taken.

For example, if a predetermined time elapses after the processing for synchronizing the clock of the terminal device and the reference clock is performed, the synchronization may not be properly maintained. In such a case, there is a possibility that the terminal device cannot reproduce the content at a time synchronized with the transmitter-side content reproduced by the transmitter. Therefore, in the playback method according to one aspect of the present invention, as shown in steps S1829 and S1830 of FIG. 130, when a predetermined time has elapsed, the clock of the terminal device (receiver) and the clock of the transmitter are set. Synchronized. Therefore, the terminal device can reproduce the content at a time synchronized with the transmitter-side content reproduced by the transmitter.

Further, the server has a plurality of contents each associated with a time, and in the contents receiving step, when the contents associated with the time indicated by the visible light signal does not exist in the server, Among the plurality of contents, content that is closest to the time indicated by the visible light signal and that is associated with a time after the time indicated by the visible light signal may be received.

As a result, as shown in the method d in FIG. 131A, even if the content associated with the time indicated by the visible light signal does not exist in the server, the appropriate content is received from the plurality of contents in the server. be able to.

In addition, a signal receiving step of receiving the visible light signal by a sensor of a terminal device from a transmitter that transmits a visible light signal due to a luminance change of the light source, and a content associated with the visible light signal from the terminal device. A transmission step of transmitting a request signal for requesting to the server, a content reception step in which the terminal device receives the content from the server, and a reproduction step of reproducing the content, wherein the visible light signal is: ID information and a time at which the visible light signal is transmitted from the transmitter are indicated. In the content receiving step, the ID information indicated by the visible light signal and the content associated with the time are received. Good.

Thus, as in method d in FIG. 131A, the visible light signal is associated with the time (transmitter time) at which the visible light signal is transmitted from the transmitter among the plurality of contents associated with the ID information (transmitter ID). The received content is received and played back. Therefore, it is possible to reproduce content appropriate for the transmitter ID and transmitter time.

The visible light signal includes second information indicating the hour and minute of the time and first information indicating the second of the time, so that the visible light signal is transmitted from the transmitter. In the signal receiving step, the second information may be received and the first information may be received more times than the number of times the second information is received.

Thereby, for example, when notifying the terminal device of the time at which each packet included in the visible light signal is transmitted in seconds, the packet indicating the current time expressed using all of the hour, minute, and second Can be saved in every second. That is, as shown in FIG. 126, if the hour and minute of the time when the packet is transmitted is not updated from the time and minute indicated in the previously transmitted packet, the packet indicating only the second (time packet 1 Only the first information is required to be transmitted. Therefore, by reducing the second information that is the packet indicating the hour and the minute (time packet 2) than the first information that is the packet indicating the second (time packet 1) transmitted by the transmitter, Transmission of packets containing redundant contents can be suppressed.

The sensor of the terminal device is an image sensor, and in the signal receiving step, the shutter speed of the image sensor is set to a first speed and a second speed higher than the first speed. While continuously switching, the image sensor performs continuous shooting, and (a) when the subject imaged by the image sensor is a barcode, by shooting when the shutter speed is the first speed, A barcode identifier is obtained by acquiring an image in which a barcode is reflected and decoding the barcode reflected in the image. (B) When the subject imaged by the image sensor is the light source The plurality of exposure lines included in the image sensor are photographed when the shutter speed is the second speed. The bright line image, which is an image including the bright line corresponding to each, is acquired, the visible light signal is acquired as a visible light identifier by decoding a plurality of bright line patterns included in the acquired bright line image, and the reproduction method Then, an image obtained by photographing when the shutter speed is the first speed may be displayed.

As a result, as shown in FIG. 102, an identifier corresponding to the barcode or the visible light signal can be appropriately acquired, and an image on which the barcode or light source as the subject is projected is displayed. Can be displayed.

In addition, in the acquisition of the visible light identifier, a first packet including a data part and an address part is acquired from the plurality of bright line patterns, and at least one already acquired before the first packet is obtained. It is determined whether or not there are a predetermined number or more of second packets, which are packets including the same address part as the address part of the first packet, and the second packet is the predetermined number If it is determined that there are more than the predetermined number, the pixel value of a part of the bright line image corresponding to each data portion of the second packet more than the predetermined number and the data portion of the first packet The visible light identification is performed by calculating a composite pixel value by combining the pixel values of a part of the region of the corresponding bright line image and decoding a data portion including the composite pixel value. At least a portion of the may be obtained.

As a result, as shown in FIG. 74, even if the data part is slightly different in each of a plurality of packets including the same address part, an appropriate data part is obtained by matching the pixel values of the data part of those packets. It can be decoded and at least a portion of the visible light identifier can be obtained correctly.

The first packet further includes a first error correction code for the data portion and a second error correction code for the address portion. In the signal receiving step, a second error correction code is sent from the transmitter. The data part transmitted by the luminance change according to the first frequency higher than the second frequency, receiving the address part and the second error correction code transmitted by the luminance change according to the frequency of The first error correction code may be received.

This makes it possible to prevent the address part from being received in error and to obtain a data part with a large amount of data quickly.

In addition, in the acquisition of the visible light identifier, a first packet including a data part and an address part is acquired from the plurality of bright line patterns, and at least one already acquired before the first packet is obtained. It is determined whether or not there is at least one second packet that is a packet including the same address part as the address part of the first packet, and if there is the at least one second packet. If it is determined, it is determined whether or not the data portions of the at least one second packet and the first packet are all equal, and when it is determined that the data portions are not all equal. In each of the at least one second packet, among the parts included in the data portion of the second packet, the first packet It is determined whether there are more than a predetermined number of parts different from each part included in the data portion of the packet, and the number of different parts of the at least one second packet is the predetermined number. If there is a second packet determined to exist, the at least one second packet is discarded, and the number of different parts of the at least one second packet is greater than or equal to the predetermined number Then, if there is no second packet determined, the plurality of packets having the largest number of packets having the same data part are identified from among the first packet and the at least one second packet, By decoding the data part included in each of the plurality of packets as the data part corresponding to the address part included in the first packet, the visible light identifier is reduced. The phrase may also get the part.

As a result, as shown in FIG. 73, when a plurality of packets having the same address part are received, even if the data parts of those packets are different, the appropriate data part can be decoded and visible. At least a part of the optical identifier can be acquired correctly. That is, a plurality of packets having the same address part transmitted from the same transmitter basically have the same data part. However, when the terminal device switches the transmitter that is the transmission source of the packet, the terminal device may receive a plurality of packets having different data portions even though they have the same address portion. In such a case, in the reproduction method according to one aspect of the present invention, the already received packet (second packet) is discarded as in step S10106 of FIG. Data portion of the packet) can be decoded as a correct data portion corresponding to the address portion. Furthermore, even when there is no transmitter switching as described above, the data portions of a plurality of packets having the same address portion may be slightly different depending on the transmission / reception state of the visible light signal. In such a case, in the reproduction method according to one aspect of the present invention, an appropriate data portion can be decoded by a so-called majority decision as in step S10107 of FIG.

Further, in the acquisition of the visible light identifier, a plurality of packets each including a data portion and an address portion are acquired from the plurality of bright line patterns, and all of the plurality of acquired packets included in the data portion are acquired. It is determined whether or not there is a 0-termination packet that is a packet whose bit indicates 0, and if it is determined that the 0-termination packet exists, the address part of the 0-termination packet is included in the plurality of packets. When it is determined whether or not all N related packets (N is an integer of 1 or more) that are packets including the associated address part exist, and when it is determined that all the N related packets exist The visible light identifier may be acquired by arranging and decoding the data portions of the N related packets. For example, the address part associated with the address part of the zero-termination packet is an address part that indicates an address that is smaller than or equal to the address indicated in the address part of the zero-termination packet.

Specifically, as shown in FIG. 75, it is determined whether or not all packets having addresses equal to or less than the address of the zero-termination packet are prepared as related packets. Each data portion of the packet is decoded. As a result, the terminal device does not know in advance how many related packets are necessary to obtain the visible light identifier, and further does not know the addresses of those related packets in advance. Also, when the zero-termination packet is acquired, it can be easily known. As a result, the terminal device can obtain an appropriate visible light identifier by arranging and decoding the data portions of the N related packets.

(Embodiment 18)
Hereinafter, the variable length / variable division number compatible protocol will be described.

FIG. 156 is a diagram illustrating an example of a transmission signal in the present embodiment.

The transmission packet is composed of a preamble, a TYPE, a payload, and a check unit. Packets may be transmitted continuously or intermittently. By providing a period during which no packet is transmitted, the state of the liquid crystal can be changed when the backlight is turned off, and the dynamic resolution of the liquid crystal display can be improved. Interference can be avoided by making the packet transmission interval random.

The pattern that does not appear in 4PPM is used for the preamble. The reception process can be simplified by using a short basic pattern.

By expressing the number of data divisions according to the type of preamble, the number of data divisions can be made variable without using extra transmission slots.

The transmission data can be made variable by changing the payload length according to the value of TYPE. In TYPE, the payload length may be expressed, or the data length before division may be expressed. By expressing the packet address by the value of TYPE, the receiver can arrange the received packets correctly. Further, the payload length (data length) represented by the value of TYPE may be changed depending on the type of preamble or the number of divisions.

Efficient error correction (detection) can be performed by changing the length of the check part according to the payload length. By setting the shortest length of the check part to 2 bits, it can be efficiently converted to 4PPM. Also, error correction (detection) can be performed efficiently by changing the type of error correction (detection) code depending on the payload length. The length of the check unit or the type of error correction (detection) code may be changed depending on the type of preamble or the value of TYPE.

There are combinations that have the same data length with different combinations of payload and number of divisions. In such a case, even if the data value is the same, more values can be expressed by giving different meanings to each combination.

Hereinafter, the high-speed transmission / luminance modulation protocol will be described.

FIG. 157 is a diagram illustrating an example of a transmission signal in the present embodiment.

The transmission packet is composed of a preamble part, a body part, and a brightness adjustment part. The body includes an address part, a data part, and an error correction (detection) code part. By allowing intermittent transmission, the same effect as described above can be obtained.

(Embodiment 19)
(Frame configuration of Single frame transmission)
FIG. 158 is a diagram illustrating an example of a transmission signal in this embodiment.

The transmission frame includes a preamble (PRE), a frame length (FLEN), an ID type (IDTYPE), a content (ID / DATA), and a check code (CRC), and may include a content type (CONTENTTYPE). The number of bits in each area is an example.

By specifying the ID / DATA length with FLEN, variable length content can be transmitted.

CRC is a check code that corrects or detects errors other than PRE. By changing the CRC length in accordance with the length of the inspection area, the inspection capability can be maintained above a certain level. Moreover, the inspection capability per CRC length can be improved by using different inspection codes depending on the length of the inspection region.

(Frame configuration of multiple frame transmission)
FIG. 159 is a diagram illustrating an example of a transmission signal in this embodiment.

The transmission frame is composed of a preamble (PRE), an address (ADDR), and a part of the divided data (DATAPART), and may include a division number (PARTNUM) and an address flag (ADDRFRAG). The number of bits in each area is an example.

・ Dividing content into multiple parts and transmitting them enables long-distance communications.

By dividing the size into equal parts, the maximum frame length can be reduced, and stable communication can be performed.

If equal division is not possible, data of just the right size can be transmitted by making some division parts smaller than other division parts.

* More information can be transmitted by making the division size different and giving meaning to combinations of division sizes. For example, even if the data is the same value of 32 bits, when 8 bits are transmitted 4 times, when 16 bits are transmitted 2 times, when 15 bits are transmitted once and 17 is transmitted once Then, it is possible to express a larger amount of information by treating it as different information.

By indicating the number of divisions with PARTNUM, the receiver can immediately know the number of divisions and can accurately display the progress of reception.

If ADDRFRAG is 0, it is not the last address, but if it is 1, the area indicating the number of divisions is not required, and transmission can be performed in a shorter time.

CRC is a check code for correcting or detecting errors other than PRE, as described above. By this inspection, interference can be detected when transmission frames from a plurality of transmission sources are received. By setting the CRC length to an integer multiple of the DATAPART length, it is possible to detect interference most efficiently.

A check code for checking a portion other than the PRE of each frame may be added to the end of the divided frame (the frame indicated by (a), (b), or (c) in FIG. 159).

The IDTYPE shown by (d) in FIG. 159 may be a fixed length such as 4 bits or 5 bits as in FIGS. 158 (a) to (d), or the IDTYPE length may be changed by the ID / DATA length. Good. Thereby, the same effect as described above can be obtained.

(Designation of ID / DATA length)
FIG. 160 is a diagram illustrating an example of a transmission signal in this embodiment.

In the case of (a) to (d) in FIG. 158, the ucode can be represented at 128 bits by setting as shown in the tables (a) and (b) shown in FIG.

(CRC length and generator polynomial)
FIG. 161 is a diagram illustrating an example of a transmission signal in this embodiment.

By setting the CRC length in this way, the inspection capability can be maintained regardless of the length of the inspection target.

The generator polynomial is an example, and another generator polynomial may be used. A check code other than CRC may be used. As a result, the inspection capability can be improved.

(Designation of DATAPART length by the type of preamble and designation of the last address)
FIG. 162 is a diagram illustrating an example of a transmission signal in this embodiment.

By indicating the DATAPART length as the type of preamble, an area indicating the DATAPART length is not necessary, and information can be transmitted in a shorter transmission time. Further, by indicating whether or not it is the last address, an area indicating the number of divisions is not necessary, and information can be transmitted in a shorter transmission time. In the case of (b) in FIG. 162, since the DATAPART length is not known in the case of the last address, it is estimated that it is the same as the DATAPART length of the frame that is not the last address received immediately before or after the frame reception. By performing the receiving process, it is possible to receive normally.

The address length may be different depending on the type of preamble. Thereby, the combination of the length of transmission information can be increased, or it can transmit in a short time.

In the case of (c) in FIG. 162, the number of divisions is defined by the preamble, and an area indicating the DATAPART length is added.

(Specify address)
FIG. 163 is a diagram illustrating an example of a transmission signal in this embodiment.

By indicating the address of the frame with the value of ADDR, the receiver can reconstruct the correctly transmitted information.

By indicating the number of divisions by the value of PARTNUM, the receiver can always know the number of divisions when the first frame is received, and can accurately display the progress of reception.

(Prevention of interference due to differences in the number of divisions)
FIGS. 164 and 165 are a diagram and a flowchart illustrating an example of the transmission / reception system in this embodiment.

When transmission information is divided equally and transmitted separately, the signals from transmitter A and transmitter B in FIG. 164 have different preambles, so even if these signals are received simultaneously, the receiver confuses the transmission source. Transmission information can be reconstructed without any problem.

Transmitters A and B are provided with a division number setting unit, so that the user can set the division number of transmitters installed nearby to be different, thereby preventing interference.

The receiver registers the number of divisions of the received signal with the server, so that the server can know the number of divisions set by the transmitter, and the other receivers obtain the information from the server, The progress of reception can be accurately displayed.

The receiver obtains from the server or from the storage unit of the receiver whether the signal from the nearby or corresponding transmitter is divided into equal lengths. When the acquired information is equal length division, a signal is restored only from a frame having the same DATAPART length. If this is not the case, or if a situation in which all addresses are not complete in a frame with the same DATAPART length continues for a predetermined time or longer, the signal is restored by combining frames with different DATAPART lengths.

(Prevention of interference due to differences in the number of divisions)
FIG. 166 is a flowchart showing the operation of the server in this embodiment.

The server receives from the receiver the ID received by the receiver and the divided configuration (what combination of DATAPART lengths the signal was received in). When the ID is an object to be expanded by a divided configuration, a numerical value of the divided configuration pattern is used as an auxiliary ID, and information associated with the extended ID obtained by combining the ID and the auxiliary ID as a key is received by the receiver. To pass.

If it is not the target of expansion by the split configuration, it is checked whether the split configuration associated with the ID exists in the storage unit, and whether it is the same as the received split configuration. If they are different, send a reconfirmation command to the receiver. Thereby, it is possible to prevent erroneous information from being presented due to a reception error of the receiver.

When the same division configuration is received with the same ID within a predetermined time after the reconfirmation command is transmitted, it is determined that the division configuration has been changed, and the division configuration associated with the ID is updated. Thereby, as described in the explanation of FIG. 164, it is possible to cope with a case where the division configuration is changed.

When the division configuration is not stored, when the received division configuration matches the stored division configuration, or when the division configuration is updated, the associated information is passed to the receiver using the ID as a key, The divided configuration is associated with the ID and stored in the storage unit.

(Display of reception progress)
167 to 172 are a flowchart and a diagram illustrating an example of operation of the receiver in this embodiment.

The receiver obtains from the storage area of the server or the receiver the type and ratio of the number of divisions of the transmitter supported by the receiver or the transmitter in the vicinity of the receiver. Further, when a part of the divided data has already been received, the type and ratio of the number of divisions of the transmitter that transmits the information that matches the part of the divided data is acquired.

The receiver receives the divided frame.

If the last address has already been received, if the obtained number of divisions is only one, or if the number of divisions supported by the receiving application being executed is only one, the number of divisions is Since it is known, the progress status is displayed based on the number of divisions.

Otherwise, if the available processing resources are few or the energy saving mode is set, the receiver calculates and displays the progress status in the simple mode. On the other hand, when there are many available processing resources or when the energy saving mode is not set, the progress status is calculated and displayed in the maximum likelihood estimation mode.

FIG. 168 is a flowchart showing a method for calculating the progress in the simple mode.

First, the receiver acquires the standard division number Ns from the server. Alternatively, the receiver reads the standard division number Ns from its own data holding unit. Note that the standard number of divisions is (a) the mode value or expected value of the number of transmitters to be transmitted with the number of divisions, (b) the number of divisions determined for each packet length, and (c) the number of divisions determined for each application. Or (d) the number of divisions determined for each identifiable range where the receiver is located.

Next, the receiver determines whether or not a packet indicating the final address has been received. If it is determined that the packet is received, the address of the last packet is set to N. On the other hand, if it is determined that it has not been received, Ne is set to a number obtained by adding 1 or 2 or more to the received maximum address Amax. Here, the receiver determines whether Ne> Ns. If it is determined that Ne> Ns, the receiver sets N = Ne. On the other hand, if it is determined that Ne> Ns is not satisfied, the receiver sets N = Ns.

Then, the receiver calculates the ratio of the number of received packets among the packets necessary for receiving the entire signal, assuming that the number of divisions of the signal being received is N.

In such a simple mode, the progress can be calculated with simpler calculation than the maximum likelihood estimation mode, which is advantageous in terms of processing time or energy consumption.

FIG. 169 is a flowchart showing a method of calculating the progress in the maximum likelihood estimation mode.

First, the receiver acquires the prior distribution of the number of divisions from the server. Alternatively, the receiver reads the prior distribution from its own data holding unit. The prior distribution is defined as (a) the distribution of the number of transmitters to be transmitted in the division number, (b) defined for each packet length, (c) defined for each application, or (D) The location where the receiver is located, and is determined for each identifiable range.

Next, the receiver receives the packet x, and calculates the probability P (x | y) of receiving the packet x when the division number is y. Then, when the receiver receives the packet x, the receiver obtains a probability P (y | x) that the number of transmission signal divisions is y as P (x | y) × P (y) ÷ A (note that A Is a normalized multiplier). Further, the receiver sets P (y) = P (y | x).

Here, the receiver determines whether the division number estimation mode is the maximum likelihood mode or the likelihood average mode. In the case of the maximum likelihood mode, the receiver calculates the ratio of the number of received packets, with y being the maximum P (y) as the division number. On the other hand, in the likelihood average mode, the receiver calculates the ratio of the number of received packets with the sum of y × P (y) as the number of divisions.

In such maximum likelihood estimation mode, it is possible to calculate a more accurate degree of progress than in the simple mode.

In addition, when the division number estimation mode is the maximum likelihood mode, the likelihood of what the last address is from the addresses received so far is calculated, and the maximum likelihood is estimated as the division number. Displays the progress of reception. This display method can display the progress status closest to the actual progress status.

FIG. 170 is a flowchart showing a display method in which the progress status does not decrease.

First, the receiver calculates the ratio of the number of received packets among the packets necessary for receiving the entire signal. Then, the receiver determines whether or not the calculated ratio is smaller than the ratio being displayed. If it is determined that the ratio is smaller than the display ratio, the receiver further determines whether the display ratio is a calculation result before a predetermined time or more. If it is determined that the calculation result is more than a predetermined time, the receiver displays the calculated ratio. On the other hand, if it is determined that the calculation result is not more than a predetermined time ago, the receiver continues to display the displayed ratio.

Further, when the receiver determines that the calculated ratio is equal to or higher than the ratio being displayed, Ne sets the number obtained by adding one or two or more to the received maximum address Amax. The receiver then displays the calculated percentage.

It is unnatural that the progress calculation result becomes smaller than before, that is, when the final packet is received, that is, the displayed progress (degree of progress) decreases. However, the above-described display method can suppress such unnatural display.

FIG. 171 is a flowchart showing a progress display method when there are a plurality of packet lengths.

First, the receiver calculates the ratio P of the number of received packets for each packet length. Here, the receiver determines whether the display mode is the maximum mode, the full display mode, or the latest mode. If it is determined that the mode is the maximum mode, the receiver displays the maximum ratio among the ratios P of the plurality of packet lengths. When it is determined that the display mode is the full display mode, the receiver displays all the ratios P. If it is determined that the mode is the latest mode, the receiver displays the packet length ratio P of the last received packet.

In FIG. 172, (a) is the progress status calculated as the simple mode, (b) is the progress status calculated as the maximum likelihood mode, and (c) is the minimum of the obtained number of divisions. Is the progress of the case. Since the progress status increases in the order of (a), (b), and (c), all the progress statuses can be displayed at the same time by displaying (a), (b), and (c) in this manner. .

(Light emission control by common switch and pixel switch)
In the transmission method according to the present embodiment, for example, each LED included in the LED display for video display is changed in luminance according to the switching of the common switch and the pixel switch, so that a visible light signal (also a visible light communication signal) is obtained. Send).

The LED display is configured as a large display disposed outdoors, for example. The LED display includes a plurality of LEDs arranged in a matrix, and displays an image by blinking these LEDs in accordance with a video signal. Such an LED display includes a plurality of common lines (COM lines) and a plurality of pixel lines (SEG lines). Each common line is composed of a plurality of LEDs arranged in a line in the horizontal direction, and each pixel line is composed of a plurality of LEDs arranged in a line in the vertical direction. Each of the plurality of common lines is connected to a common switch corresponding to the common line. The common switch is, for example, a transistor. Each of the plurality of pixel lines is connected to a pixel switch corresponding to the pixel line. A plurality of pixel switches corresponding to a plurality of pixel lines are provided in, for example, an LED driver circuit (constant current circuit). The LED driver circuit is configured as a pixel switch control unit that switches a plurality of pixel switches.

More specifically, one of the anode and the cathode of each LED included in the common line is connected to a terminal such as a collector of a transistor corresponding to the common line. The other of the anode and the cathode of each LED included in the pixel line is connected to a terminal (pixel switch) corresponding to the pixel line in the LED driver circuit.

When such an LED display displays an image, a common switch control unit that controls a plurality of common switches turns on those common switches in a time-sharing manner. For example, the common switch control unit turns on only the first common switch among the plurality of common switches during the first period, and the second common among the plurality of common switches during the next second period. Turn on only the switch. Then, the LED driver circuit turns on each pixel switch according to the video signal during a period when any one of the common switches is turned on. As a result, the LEDs corresponding to the common switch and the pixel switch are lit only during a period in which the common switch is on and the pixel switch is on. The luminance of the pixels in the video is expressed by this lighting period. That is, the luminance of the image pixels is PWM-controlled.

In the transmission method in the present embodiment, a visible light signal is transmitted using such an LED display, a common switch and a pixel switch, and a common switch control unit and a pixel switch control unit. In addition, the transmission device (also referred to as a transmitter) in the present embodiment that transmits a visible light signal by such a transmission method includes the common switch control unit and the pixel switch control unit.

FIG. 173 is a diagram illustrating an example of a transmission signal in this embodiment.

The transmitter transmits each symbol included in the visible light signal according to a predetermined symbol period. For example, when transmitting the symbol “00” by 4PPM, the transmitter switches the common switch according to the symbol (the luminance change pattern of “00”) in a symbol period of 4 slots. Then, the transmitter switches the pixel switch according to the average luminance indicated by the video signal or the like.

More specifically, when the average luminance in the symbol period is set to 75% (FIG. 173 (a)), the transmitter turns off the common switch during the first slot, and the second to fourth slots. The common switch is turned on during the period up to the slot. Further, the transmitter turns off the pixel switch during the period of the first slot, and turns on the pixel switch during the period from the second slot to the fourth slot. As a result, the LEDs corresponding to the common switch and the pixel switch are lit only during a period in which the common switch is on and the pixel switch is on. In other words, the luminance of the LED changes by lighting at the luminance of LO (Low), HI (High), HI, and HI in each of the four slots. As a result, the symbol “00” is transmitted.

When the average luminance in the symbol period is 25% ((e) in FIG. 173), the transmitter turns off the common switch during the first slot and during the period from the second slot to the fourth slot. Turn on the common switch. Further, the transmitter turns off the pixel switch during the first slot, the third slot, and the fourth slot, and turns on the pixel switch during the second slot. As a result, the LEDs corresponding to the common switch and the pixel switch are lit only during a period in which the common switch is on and the pixel switch is on. In other words, the luminance of the LED changes by being lit like LO (Low), HI (High), LO, and LO in each of the four slots. As a result, the symbol “00” is transmitted. In addition, since the transmitter in this Embodiment transmits the visible light signal close | similar to the above-mentioned V4PPM (variable 4PPM), even when transmitting the same symbol, it can make an average luminance variable. In other words, when transmitting the same symbol (for example, “00”) with different average luminance, the transmitter, as shown in FIGS. ) Is constant regardless of the average brightness. Thereby, the receiver can receive a visible light signal without being conscious of luminance.

Note that the common switch is switched by the above-described common switch control unit, and the pixel switch is switched by the above-described pixel switch control unit.

As described above, the transmission method in the present embodiment is a transmission method for transmitting a visible light signal by a change in luminance, and includes a determination step, a common switch control step, and a first pixel switch control step. In the determining step, the luminance change pattern is determined by modulating the visible light signal. In the common switch control step, a common switch for commonly lighting a plurality of light sources (LEDs) included in a light source group (common line) provided in the display and representing each pixel in the video is displayed with the brightness. Switching according to the change pattern. In the first pixel switch control step, the common switch is turned on by turning on the first pixel switch for turning on the first light source among the plurality of light sources included in the light source group, and The visible light signal is transmitted by turning on the first light source only during a period in which the first pixel switch is on.

Thereby, a visible light signal can be appropriately transmitted from a display including a plurality of LEDs as light sources. Therefore, the communication between the apparatuses of the aspect containing apparatuses other than illumination is enabled. When the display is a display for displaying an image by controlling the common switch and the first pixel switch, a visible light signal may be transmitted using the common switch and the first pixel switch. it can. Therefore, a visible light signal can be easily transmitted without making a significant change to the configuration for displaying an image on a display.

Also, by making the control timing of the pixel switch coincide with the transmission symbol (one 4PPM) and controlling as shown in FIG. 173, a visible light signal can be transmitted from the LED display without flickering. Image signals (that is, video signals) usually change at 1/30 second or 1/60 second cycles, but by changing the image signal according to the symbol transmission cycle (symbol cycle), this can be achieved without changing the circuit. can do.

Thus, in the determination step of the transmission method in the present embodiment, the luminance change pattern is determined for each symbol period. In the first pixel switch control step, the pixel switch is switched in synchronization with the symbol period. Thereby, even if a symbol period is 1/2400 second, a visible light signal can be appropriately transmitted according to the symbol period.

When the signal (symbol) is “10” and the average luminance is near 50%, the luminance change pattern is close to 0101, and there are two luminance rising points. However, in that case, the receiver can correctly receive the signal by giving priority to the subsequent rising point. In other words, the subsequent rising point is the timing at which the luminance rising inherent to the symbol “10” is obtained.

The higher the average brightness, the closer to the signal modulated with 4 PPM, the more the signal can be output. Therefore, when the luminance of the entire screen or the portion where the power supply line is common is low, the HI section can be lengthened and the error can be reduced by reducing the current and decreasing the instantaneous luminance value. . In this case, the maximum brightness of the screen is reduced, but when high brightness is not necessary in the first place, such as indoor use, or when priority is given to visible light communication, enabling the switch to enable this enables communication. The balance between quality and image quality can be set optimally.

In the first pixel switch control step of the transmission method according to the present embodiment, when an image is displayed on the display (LED display), the pixel value of the pixel in the image corresponding to the first light source is expressed. The first pixel switch is switched so that the lighting period is supplemented only during the period during which the first light source is turned off in order to transmit the visible light signal. That is, in the transmission method in the present embodiment, a visible light signal is transmitted when an image is displayed on the LED display. Therefore, the LED may be turned off for transmission of a visible light signal in a period in which the LED is to be turned on in order to express a pixel value (specifically, a luminance value) indicated by the video signal. In such a case, in the transmission method according to the present embodiment, the first pixel switch is switched so as to compensate for the lighting period only during the period when the LED is turned off.

For example, when displaying an image indicated by a video signal without transmitting a visible light signal, the common switch is turned on during one symbol period, and the pixel switch is an average value that is the pixel value indicated by the video signal. It is turned on only for the period according to the brightness. When the average luminance is 75%, the common switch is turned on in the first slot to the fourth slot of the symbol period. Further, the pixel switch is turned on in the first to third slots of the symbol period. As a result, during the symbol period, the LEDs are lit in the first slot to the third slot, so that the above-described pixel values can be expressed. However, in order to transmit the symbol “01”, the second slot is turned off. Therefore, in the transmission method according to the present embodiment, the pixel switch is switched so that only the second slot in which the LED is turned off compensates for the lighting period of the LED, that is, the LED is turned on in the fourth slot. .

In the transmission method according to the present embodiment, the lighting period is compensated by changing the pixel value of the pixel in the video. For example, in the above case, the pixel value with an average luminance of 75% is changed to a pixel value with an average luminance of 100%. When the average luminance is 100%, the LED tries to light up in the first slot to the fourth slot, but for transmission of the symbol “01”, the first slot is turned off. Therefore, even when a visible light signal is transmitted, the LED can be lit with the original pixel value (average luminance 75%).

This makes it possible to prevent the image from being corrupted by the transmission of the visible light signal.

(Light emission control shifted for each pixel)
FIG. 174 is a diagram illustrating an example of a transmission signal in this embodiment.

When the transmitter in the present embodiment transmits the same symbol (for example, “10”) from the pixel A and the pixels in the vicinity of the pixel A (for example, the pixel B and the pixel C) as shown in FIG. The light emission timing of the pixels is shifted. However, the transmitter causes the pixels to emit light without shifting the rise timing of the luminance specific to the symbol between the pixels. Note that each of the pixels A to C corresponds to a light source (specifically, an LED). In addition, when the symbol is “10”, the luminance rise timing specific to the symbol is the boundary timing between the third slot and the fourth slot. Such timing is hereinafter referred to as symbol specific timing. The receiver can receive a symbol corresponding to the timing by specifying the symbol specific timing.

By shifting the light emission timing in this manner, the waveform indicating the average luminance transition between pixels has a gradual rise or fall except for the rise at the symbol specific timing, as shown in FIG. That is, the rise at the symbol specific timing is steeper than the rise at other timings. Therefore, the receiver can identify an appropriate symbol specific timing by giving priority to receiving the steepest rise among a plurality of rises, and as a result, reception errors can be suppressed.

In other words, when the symbol “10” is transmitted from a predetermined pixel and the luminance of the predetermined pixel is an intermediate value of 25% to 75%, the transmitter opens the pixel switch corresponding to the predetermined pixel. Set the section short or long. Further, the transmitter reversely adjusts the open period of the pixel switch corresponding to the pixel in the vicinity of the predetermined pixel. Thus, errors can also be suppressed by setting the open interval of each pixel switch so that the overall luminance including the predetermined pixel and the neighboring pixels does not change. The open section is a section where the pixel switch is on.

Thus, the transmission method in the present embodiment further includes a second pixel switch control step. In this second pixel switch control step, by turning on the second pixel switch for turning on the second light source around the first light source included in the light source group (common line) described above. The visible light signal is transmitted by turning on the second light source only during the period when the common switch is on and the second pixel switch is on. Note that the second light source is, for example, a light source adjacent to the first light source.

In the first and second pixel switch control steps, when the same symbol included in the visible light signal is simultaneously transmitted from each of the first and second light sources, the first and second pixel switches are controlled. Among a plurality of timings each of which is turned on or off to transmit the same symbol, the timing at which a rise in luminance specific to the same symbol is obtained is made the same in each of the first and second pixel switches, Other timings are made different for each of the first and second pixel switches, and the overall average luminance of the first and second light sources in a period in which the same symbol is transmitted is set to a predetermined luminance. Match.

As a result, as shown in FIG. 174, the rising edge can be sharpened only at the timing when the rising edge inherent in the symbol is obtained in the spatially averaged brightness as in the transition between the average brightness of pixels. The occurrence of errors can be suppressed. That is, the reception error of the visible light signal by the receiver can be suppressed.

In addition, when the symbol “10” is transmitted from a predetermined pixel and the luminance of the predetermined pixel is an intermediate value of 25% to 75%, the transmitter transmits the symbol “10” to the predetermined pixel in the first period. The open section of the corresponding pixel switch is set to be short or long. Further, the transmitter reversely adjusts the open period of the pixel switch in a second period (for example, a frame) before or after the first period. As described above, the error can be suppressed by setting the open period of the pixel switch so that the entire time average luminance including the first period and the second period in the predetermined pixel does not change.

That is, in the first pixel switch control step in the transmission method according to the present embodiment, for example, it is included in the visible light signal in the first period and the second period following the first period. Send the same symbol. At this time, in each of the first and second periods, a rise in luminance specific to the same symbol among a plurality of timings at which the first pixel switch is turned on or off to transmit the same symbol. The obtained timing is made the same, and other timings are made different. Then, the average luminance of the first light source in the entire first and second periods is matched with a predetermined luminance. Each of the first period and the second period may be a period for displaying a frame and a period for displaying the next frame. Further, the first period and the second period may each be a symbol period. That is, each of the first period and the second period may be a period for transmitting one symbol and a period for transmitting the next symbol.

As a result, like the average luminance transition between pixels shown in FIG. 174, in the luminance averaged over time, the rising edge can be made steep only at the timing when the rising edge inherent in the symbol is obtained. The occurrence of reception errors can be suppressed. That is, the reception error of the visible light signal by the receiver can be suppressed.

(Light emission control when the pixel switch can be driven at double speed)
FIG. 175 is a diagram illustrating an example of a transmission signal in this embodiment.

When the pixel switch can be opened and closed at half the transmission symbol period, that is, when the pixel switch can be driven at double speed, the same light emission pattern as V4PPM can be obtained as shown in FIG.

In other words, when the symbol period (symbol transmission period) is composed of 4 slots, a pixel switch control unit such as an LED driver circuit that controls the pixel switch can control the pixel switch every two slots. That is, the pixel switch control unit can turn on the pixel switch for an arbitrary time in a period of two slots from the first time point of the symbol period. Further, the pixel switch control unit can turn on the pixel switch for an arbitrary time in a period of two slots from the first time point of the third slot of the symbol period.

That is, in the transmission method according to the present embodiment, the pixel value may be changed at a period that is ½ of the above-described symbol period.

In this case, the fineness per opening / closing of the pixel switch may be reduced (accuracy will be reduced). Therefore, by performing this only when the transmission priority switch is valid, the balance between image quality and transmission quality can be set optimally.

(Light emission control block by pixel value adjustment)
FIG. 176 is a block diagram illustrating an example of a transmitter in this embodiment.

FIG. 176 (a) is a block diagram showing a configuration of a device that does not transmit a visible light signal and only displays an image, that is, a display device that displays an image on the LED display described above. As shown in FIG. 176 (a), the display device includes an image / video input unit 1911, an N-times speed increasing unit 1912, a common switch control unit 1913, and a pixel switch control unit 1914.

The image / video input unit 1911 outputs a video signal indicating an image or video at a frame rate of 60 Hz, for example, to the N-times speed increasing unit 1912.

The N-times speed increasing unit 1912 increases the frame rate of the video signal input from the image / video input unit 1911 to N (N> 1) times and outputs the video signal. For example, the N-times speed increasing unit 1912 increases the frame rate to 10 times (N = 10), that is, to a frame rate of 600 Hz.

The common switch control unit 1913 switches the common switch based on the 600 Hz frame rate video. Similarly, the pixel switch control unit 1914 switches pixel switches based on the 600 Hz frame rate video. Thus, flickering due to opening and closing of a switch such as a common switch or a pixel switch can be avoided by increasing the frame rate by the N-times speed increasing unit 1912. Even when the LED display is imaged by the imaging device with a high-speed shutter, it is possible to cause the imaging device to capture an image without missing pixels or flicker.

FIG. 176 (b) is a block diagram showing a configuration of a display device that transmits not only video but also the above-described visible light signal, that is, a transmitter (transmitting device). This transmitter includes an image / video input unit 1911, a common switch control unit 1913, a pixel switch control unit 1914, a signal input unit 1915, and a pixel value adjustment unit 1916. The signal input unit 1915 outputs a visible light signal including a plurality of symbols to the pixel value adjustment unit 1916 at a symbol rate (frequency) of 2400 symbols / second.

The pixel value adjustment unit 1916 duplicates the image input from the image / video input unit 1911 according to the symbol rate of the visible light signal, and adjusts the pixel value according to the above-described method. Accordingly, the common switch control unit 1913 and the pixel switch control unit 1914 in the subsequent stage from the pixel value adjustment unit 1916 can output a visible light signal without changing the luminance of the image or video.

For example, in the case of the example shown in FIG. 176, if the symbol rate of the visible light signal is 2400 symbols / second, the pixel value adjustment unit 1916 is included in the video signal so that the frame rate 60 Hz of the video signal is 4800 Hz. Duplicate the image. For example, the value of the symbol included in the visible light signal is “00”, and the pixel value (luminance value) of the pixel included in the first image before duplication is 50%. In this case, the pixel value adjustment unit 1916 adjusts the pixel value to 100% for the first image after duplication and to 50% for the second image. As a result, the luminance is 50% due to the AND of the common switch and the pixel switch as in the luminance change in the case of the symbol “00” shown in FIG. As a result, a visible light signal can be transmitted while maintaining the same luminance as the original image. Note that the AND of the common switch and the pixel switch means that the light source (that is, the LED) corresponding to the common switch and the pixel switch is lit only when the common switch is on and the pixel switch is on. .

Further, in the transmission method according to the present embodiment, the video display and the visible light signal transmission may be performed separately in the signal transmission period and the video display time without simultaneously performing the video display and the visible light signal transmission.

That is, in the above-described first pixel switch control step in the present embodiment, the first pixel switch is turned on during the signal transmission period in which the common switch is switching according to the luminance change pattern. The transmission method in the present embodiment further turns on the common switch during a video display period different from the signal transmission period, and turns on the first pixel switch according to the video to be displayed in the video display period. Accordingly, a video display step of displaying the pixels in the video by turning on the first light source only during a period in which the common switch is on and the first pixel switch is on may be included. .

Thereby, since the display of the image and the transmission of the visible light signal are performed in different periods, the display and the transmission can be easily performed.

(Power change timing)
When the power supply line is changed, a signal off period occurs. However, even if the last part of 4PPM is not emitting light, it does not affect reception. By changing the power supply line according to the transmission period of the 4PPM symbol, The power line can be changed without affecting the reception quality.

Also, changing the power supply line during the 4PPM LO period can change the power supply line without affecting the reception quality. In this case, transmission can be performed while keeping the maximum luminance high.

(Drive timing)
In the present embodiment, the LED display may be driven at the timings shown in FIGS. 177 to 179.

FIGS. 177 to 179 are timing charts when the LED display is driven by the optical ID modulation signal of the present invention.

For example, as shown in FIG. 178, when the common switch (COM1) is turned off (period t1) in order to transmit the visible light signal (light ID), the LED is turned on with the luminance indicated by the video signal. Therefore, the LED is lit after the period t1. Thereby, the video can be appropriately displayed without breaking the video indicated by the video signal while appropriately transmitting the visible light signal.

(Summary)
FIG. 180A is a flowchart illustrating a transmission method according to one embodiment of the present invention.

The transmission method according to an aspect of the present invention is a transmission method for transmitting a visible light signal by a change in luminance, and includes steps SC11 to SC13.

In step SC11, the luminance change pattern is determined by modulating the visible light signal as in the above-described embodiments.

In Step SC12, a common switch for commonly lighting a plurality of light sources included in the light source group provided in the display for representing pixels in the video is switched according to the luminance change pattern.

In step S13, the common switch is turned on by turning on the first pixel switch (that is, the pixel switch) for turning on the first light source among the plurality of light sources included in the light source group, and The visible light signal is transmitted by turning on the first light source only during a period in which the first pixel switch is on.

FIG. 180B is a block diagram illustrating a functional configuration of the transmission device according to one embodiment of the present invention.

A transmission device C10 according to an aspect of the present invention is a transmission device (or transmitter) that transmits a visible light signal by a luminance change, and includes a determination unit C11, a common switch control unit C12, a pixel switch control unit C13, Is provided. The determination unit C11 determines the luminance change pattern by modulating the visible light signal, as in the above-described embodiments. In addition, this determination part C11 is provided in the signal input part 1915 shown in FIG. 176, for example.

The common switch control unit C12 switches the common switch according to the luminance change pattern. This common switch is a switch for commonly lighting a plurality of light sources included in a light source group provided in the display, each representing a pixel in an image.

The pixel switch control unit C13 turns on the pixel switch for turning on the light source to be controlled among the plurality of light sources included in the light source group, so that the common switch is on and the pixel switch is on. The visible light signal is transmitted by turning on the light source to be controlled only during a certain period. The light source to be controlled is the first light source described above.

Thereby, a visible light signal can be appropriately transmitted from a display including a plurality of LEDs as light sources. Therefore, the communication between the apparatuses of the aspect containing apparatuses other than illumination is enabled. Further, when the display is a display for displaying an image by controlling the common switch and the pixel switch, the visible light signal can be transmitted using the common switch and the pixel switch. Therefore, a visible light signal can be easily transmitted without making a significant change to the configuration for displaying an image on a display (that is, a display device).

(Frame configuration of Single frame transmission)
FIG. 181 is a diagram illustrating an example of a transmission signal in this embodiment.

The transmission frame includes a preamble (PRE), an ID length (IDLEN), an ID type (IDTYPE), content (ID / DATA), and a check code (CRC), as shown in FIG. The number of bits in each area is an example.

By using the preamble as shown in FIG. 181 (b), the receiver can distinguish from other parts encoded by 4PPM, I-4PPM or V4PPM, and can find a signal delimiter. .

As shown in FIG. 181 (c), variable length content can be transmitted by specifying the ID / DATA length with IDLEN.

CRC is a check code that corrects or detects errors other than PRE. By changing the CRC length in accordance with the length of the inspection area, the inspection capability can be maintained above a certain level. Moreover, the inspection capability per CRC length can be improved by using different inspection codes depending on the length of the inspection region.

(Frame configuration of multiple frame transmission)
182 and 183 are diagrams illustrating an example of a transmission signal in this embodiment.

The transmission data (BODY) is added with a partition type (PTYPE) and a check code (CRC), and becomes Joined data. Joined data is divided into several DATA PARTs, and transmitted with a preamble (PRE) and an address (ADDR) added.

PTYPE (or partition mode (PMODE)) indicates the division method or meaning of BODY. By using 2 bits as shown in FIG. 182 (a), it is possible to encode with 4PPM. By setting 1 bit as shown in FIG. 182 (b), the transmission time can be shortened.

CRC is an inspection code for inspecting PTYPE and BODY. As defined in FIG. 161, by changing the CRC code length according to the length of the portion to be inspected, the inspection capability can be maintained at a certain level or more.

By defining the preamble as shown in FIG. 162, it is possible to shorten the transmission time while ensuring variations of the division pattern.

If the address is determined as shown in FIG. 163, the receiver can restore the data regardless of the order in which the frames are received.

FIG. 183 shows a combination of possible Joined data length and the number of frames. The underlined combination is a combination used when a PTYPE described later is a single frame compatible.

(Configuration of BODY field)
FIG. 184 is a diagram illustrating an example of a transmission signal in this embodiment.

By setting BODY to a field configuration as shown in the figure, it is possible to transmit the same ID as in single frame transmission.

In the case of the same IDTYPE and the same ID, it is possible to flexibly signal when the continuous transmission / reception time is short by expressing the same meaning regardless of the combination of single frame transmission or multiframe transmission and packet transmission. Can be sent.

Specify the ID length with IDLEN, and send PADDING for the remaining part. This part may be all 0 or 1, may transmit data for extending the ID, or may be a check code. PADDING may be left-justified.

184 (b), (c), or (d) in FIG. 184 can make the transmission time shorter than (a) in FIG. 184. At this time, the length of the ID is assumed to be the maximum of the lengths that can be taken as the ID.

In the case of (b) or (c) in FIG. 184, the number of IDTYPE bits is an odd number, but when combined with the 1-bit PTYPE shown in FIG. Can do.

In FIG. 184 (c), a longer ID can be transmitted.

In FIG. 184 (d), more IDTYPEs can be expressed.

(PTYPE)
FIG. 185 is a diagram illustrating an example of a transmission signal in this embodiment.

When PTYPE is a predetermined bit, it indicates that BODY is in single frame compatible mode. Thereby, the same ID as in the case of single frame transmission can be transmitted.

For example, when PTYPE = 00, the ID or ID type corresponding to the PTYPE can be handled in the same manner as the ID or ID type transmitted by single frame transmission, and the management of the ID or ID type can be simplified. it can.

When PTYPE is a predetermined bit, BODY indicates a Data stream mode. At this time, all combinations of the number of transmission frames and the DATAPART length can be used, and different combinations of data can have different meanings. Depending on the PTYPE bits, the different combinations may have the same meaning or different meanings. Thereby, a transmission method can be selected flexibly.

For example, when PTYPE = 01, an ID having a size not defined for single frame transmission can be transmitted. Further, even if the ID corresponding to the PTYPE is the same as the ID for single frame transmission, the ID corresponding to the PTYPE can be handled as an ID different from the ID for the single frame transmission. As a result, the number of IDs that can be expressed can be increased.

(Field structure in Single frame compatible mode)
FIG. 186 is a diagram illustrating an example of a transmission signal in this embodiment.

When (a) in FIG. 184 is used, in the single frame compatible mode, it is most efficient to transmit in the combination of the tables shown in FIG. 186.

184 When (b), (c) or (d) in FIG. 184 is used, when the ID is 32 bits, a combination of 13 frames and a DATAPART length of 4 bits is efficient. When the ID is 64 bits, the combination of 11 frames and DATAPART length of 8 bits is efficient.

Suppose that only combinations of tables are transmitted, so that different combinations can be judged as reception errors, and the reception error rate can be lowered.

(Summary of Embodiment 19)
A transmission method according to an aspect of the present invention is a transmission method for transmitting a visible light signal by a luminance change, and includes a determination step of determining a luminance change pattern by modulating the visible light signal, and a display. A common switch control step of switching a common switch included in the light source group for commonly lighting a plurality of light sources for representing pixels in the video according to the luminance change pattern; and a plurality of light sources included in the light source group By turning on the first pixel switch for turning on the first light source among the light sources, the common switch is on and only during the period when the first pixel switch is on. And a first pixel switch control step of transmitting the visible light signal by turning on the first light source.

Thereby, for example, as shown in FIGS. 173 to 180B, a visible light signal can be appropriately transmitted from a display having a plurality of LEDs as light sources. Therefore, the communication between the apparatuses of the aspect containing apparatuses other than illumination is enabled. When the display is a display for displaying an image by controlling the common switch and the first pixel switch, a visible light signal may be transmitted using the common switch and the first pixel switch. it can. Therefore, a visible light signal can be easily transmitted without making a significant change to the configuration for displaying an image on a display.

In the determination step, the luminance change pattern may be determined for each symbol period, and in the first pixel switch control step, the first pixel switch may be switched in synchronization with the symbol period.

Thereby, as shown in FIG. 173, for example, even if the symbol period is 1/2400 seconds, a visible light signal can be appropriately transmitted according to the symbol period.

Further, in the first pixel switch control step, when displaying an image on the display, the visible period of the lighting period for expressing the pixel value of the pixel in the image corresponding to the first light source is displayed. The first pixel switch may be switched so as to compensate for the lighting period only during a period in which the first light source is turned off to transmit an optical signal. For example, the lighting period may be supplemented by changing pixel values of pixels in the video.

As a result, for example, as shown in FIGS. 173 and 175, even when the first light source is turned off for the transmission of a visible light signal, the lighting period is supplemented, so that the original image is displayed properly without being destroyed. can do.

Further, the pixel value may be changed at a cycle that is ½ of the symbol cycle.

Thereby, for example, as shown in FIG. 175, it is possible to appropriately display the image and transmit the visible light signal.

In the transmission method, the common switch is further turned on by turning on a second pixel switch included in the light source group for turning on a second light source around the first light source. A second pixel switch control step of transmitting the visible light signal by turning on the second light source only during a period in which the second pixel switch is on. In the first and second pixel switch control steps, when the same symbol included in the visible light signal is simultaneously transmitted from each of the first and second light sources, the first and second pixel switches Among a plurality of timings at which each is turned on or off to transmit the same symbol, a timing at which a rise in luminance specific to the same symbol is obtained. The same timing is used for each of the first and second pixel switches, and other timings are made different for each of the first and second pixel switches. You may make the average brightness | luminance of the whole 1st and 2nd light source correspond to a predetermined brightness | luminance.

As a result, for example, as shown in FIG. 174, in the spatially averaged luminance, the rising edge can be made sharp only at the timing when the rising edge inherent to the symbol is obtained, and the occurrence of a reception error can be prevented. Can be suppressed.

In the first pixel switch control step, when the same symbol included in the visible light signal is transmitted in the first period and the second period following the first period, In each of the second period and the second period, among the plurality of timings at which the first pixel switch is turned on or off to transmit the same symbol, a timing at which a rise in luminance specific to the same symbol is obtained. The same brightness may be set at different timings so that the average brightness of the first light source in the whole of the first and second periods matches the predetermined brightness.

As a result, for example, as shown in FIG. 174, in the luminance averaged over time, the rising edge can be made sharp only at the timing when the rising edge inherent in the symbol is obtained, and the occurrence of a reception error can be prevented. Can be suppressed.

In the first pixel switch control step, the first pixel switch is turned on during a signal transmission period in which the common switch is switched according to the luminance change pattern, and the transmission method further includes the signal During the video display period different from the transmission period, turning on the common switch, and turning on the first pixel switch according to the display target video in the video display period, and the common switch is on, and A video display step of displaying pixels in the video by turning on the first light source only during a period in which the first pixel switch is on may be included.

Thereby, since the display of the image and the transmission of the visible light signal are performed in different periods, the display and the transmission can be easily performed.

(Embodiment 20)
In this embodiment, details or modifications of the visible light signal in each of the above embodiments will be specifically described. The camera trend is to increase the resolution (4K) and increase the frame rate (60 fps). The frame scan time is reduced by increasing the frame rate. As a result, the reception distance decreases and the reception time increases. Therefore, in a transmitter that transmits a visible light signal, it is necessary to shorten the packet transmission time. In addition, the time resolution of reception increases due to the reduction of the line scan time. The exposure time is 1/8000 second. In 4PPM, signal expression and dimming are performed simultaneously, so signal density is low and efficiency is poor. Therefore, in the visible light signal in the present embodiment, the signal portion and the dimming portion are separated, and the portion necessary for reception is shortened.

FIG. 187 is a diagram illustrating an example of a configuration of a visible light signal in the present embodiment.

As shown in FIG. 187, the visible light signal includes a plurality of combinations of a signal part and a light control part. The time length of this combination is, for example, 2 ms or less (frequency is 500 Hz or more).

FIG. 188 is a diagram illustrating an example of a detailed configuration of a visible light signal in the present embodiment.

The visible light signal includes data L (Data L), preamble (Preamble), data R (Data R), and a dimming unit (Dimming). The signal portion is composed of the data L, the preamble, and the data R.

The preamble alternately indicates High and Low luminance values along the time axis. That is, the preamble indicates a luminance value of High for a time length _{P 1,} it indicates the luminance value of the next time length _{P 2} only Low, only the following time length _{P 3} represents a luminance value at High, the next time length P Only ₄ indicates a low luminance value. The time lengths P ₁ to P ₄ are, for example, 100 μs.

Data R alternately indicates High and Low luminance values along the time axis, and is arranged immediately after the preamble. That is, the data R indicates the luminance value of the High only time length _{D R1,} represents a luminance value at Low only the following time length _{D R2,} represents a luminance value at High only the following time length _{D R3,} the next time length only D _R4 indicates a luminance value of Low. Note that the time lengths D _R1 to D _R4 are determined according to a mathematical formula corresponding to the transmission target signal. This equation is D _Ri = 120 + 20x _i (iε1 to 4, x _i ε0 to 15). Numerical values such as 120 and 20 indicate time (μs). Moreover, these numerical values are examples.

Data L alternately indicates High and Low luminance values along the time axis, and is arranged immediately before the preamble. That is, the data L indicates a high luminance value for the time length D _L1 , indicates a low luminance value for the next time length D _L2 , indicates a high luminance value for the next time length D _L3, and displays the next time length. Only D _L4 indicates a low luminance value. Note that the time lengths D _L1 to D _L4 are determined according to a mathematical formula corresponding to the transmission target signal. This formula is D _Li = 120 + 20 × (15−x _i ). As described above, numerical values such as 120 and 20 indicate time (μs). Moreover, these numerical values are examples.

The signal to be transmitted consists of 4 × 4 = 16 bits, and _xi is a 4-bit signal among the signals to be transmitted. In the visible light signal, the numerical value of x _i (4-bit signal) is indicated by each of the time lengths D _R1 to D _R4 in the data R or each of the time lengths D _L1 to D _{L4 in} the data L. Of the 16 bits of the signal to be transmitted, 4 bits indicate an address, 8 bits indicate data, and 4 bits are used for error detection.

Here, the data R and the data L are complementary to the brightness. That is, if the brightness of the data R is bright, the brightness of the data L is dark. Conversely, if the brightness of the data R is dark, the brightness of the data L is bright. That is, the sum of the entire time length of the data R and the time length of the data L is constant regardless of the signal to be transmitted.

The dimming unit is a signal for adjusting the brightness (luminance) of the visible light signal, and indicates a high luminance value for the time length C ₁ and indicates a low signal for the next time length C ₂ . Time length _{C 1} and _{C 2} are adjusted arbitrarily. Note that the dimming unit may or may not be included in the visible light signal.

In the example illustrated in FIG. 188, the data R and the data L are included in the visible light signal, but only one of the data R and the data L may be included. When it is desired to brighten the visible light signal, only bright data of the data R and data L may be transmitted. Further, the arrangement of the data R and the data L may be reversed. Also, if it contains data R is tone duration C ₁ of the optical part, for example greater than 100 [mu] s, if the data L is included, the time length of the dimmer C _2, for example 100 [mu] s Longer than.

FIG. 189A is a diagram illustrating another example of the visible light signal in this embodiment.

In the visible light signal shown in FIG. 188, the signal to be transmitted is represented by each of the time length indicating the high luminance value and the time length indicating the low luminance value, as shown in FIG. 189A (a). The signal to be transmitted may be expressed only by the time length indicating the Low luminance value. Note that (b) in FIG. 189A shows the visible light signal in FIG. 188.

For example, as shown in (a) of FIG. 189A, in the preamble, the time lengths indicating the high luminance value are all equal and relatively short, and the time lengths P ₁ to P ₄ indicating the low luminance value are each 100 μs, for example. It is. In the data R, the time lengths indicating the high luminance value are all equal and relatively short, and the time lengths D _R1 to D _R4 indicating the low luminance value are respectively adjusted according to the signal x _i . In the preamble and data R, the time length indicating the high luminance value is, for example, 10 μs or less.

FIG. 189B is a diagram illustrating another example of the visible light signal in this embodiment.

For example, as shown in FIG. 189B, in the preamble, the time lengths indicating the high luminance value are all equal and relatively short, and the time lengths P ₁ to P ₃ indicating the low luminance value are, for example, 160 μs, 180 μs, and 160 μs, respectively. It is. In the data R, the time lengths indicating the high luminance value are all equal and relatively short, and the time lengths D _R1 to D _R4 indicating the low luminance value are respectively adjusted according to the signal x _i . In the preamble and data R, the time length indicating the high luminance value is, for example, 10 μs or less.

FIG. 189C is a diagram illustrating the signal length of the visible light signal in the present embodiment.

FIG. 190 is a diagram showing a comparison result of luminance values between the visible light signal in the present embodiment and the visible light signal of the standard IEC (International Electrotechnical Commission). The standard IEC is specifically “VISIBLEILIGHT BEACON SYSTEM FOR MULTIMEDIA APPLICATIONS”.

In the visible light signal in this embodiment (the method of the embodiment (Data one side)), a maximum luminance of 82% higher than the maximum luminance of the standard IEC visible light signal can be obtained. A minimum luminance of 18% lower than the minimum luminance can be obtained. Note that the maximum luminance of 82% and the minimum luminance of 18% are numerical values obtained by a visible light signal including only one of the data R and the data L in the present embodiment.

FIG. 191 is a diagram showing a comparison result of the number of received packets and the reliability with respect to the angle of view between the visible light signal and the standard IEC visible light signal in the present embodiment.

In the visible light signal in the present embodiment (method of the embodiment (both)), even if the angle of view decreases, that is, the distance from the transmitter that transmits the visible light signal to the receiver increases, The number of received packets is larger than that of a standard IEC visible light signal, and high reliability can be obtained. In addition, the numerical value of the system (both) of embodiment shown in FIG. 191 is a numerical value obtained by the visible light signal including both the data R and the data L.

FIG. 192 is a diagram illustrating a comparison result of the number of received packets and reliability with respect to noise between the visible light signal and the standard IEC visible light signal according to the present embodiment.

In the visible light signal (IEEE) in this embodiment, the number of received packets is larger than that of the standard IEC visible light signal regardless of noise (noise dispersion value), and high reliability can be obtained.

FIG. 193 is a diagram showing a comparison result of the number of received packets and the reliability of the visible light signal according to the present embodiment and the standard IEC visible light signal with respect to the reception-side clock error.

In the visible light signal (IEEE) in the present embodiment, the number of received packets is larger than that of the standard IEC visible light signal in a wide range of the receiving side clock error, and high reliability can be obtained. The reception-side clock error is an error in timing at which the exposure line in the image sensor of the receiver starts exposure.

FIG. 194 is a diagram illustrating a configuration of a transmission target signal in the present embodiment.

The signal to be transmitted includes four 4-bit signals (x _i ) (4 × 4 = 16 bits) as described above. For example, the signal to be transmitted includes signals x ₁ to x ₄ . The signal x ₁ is composed of bits x ₁₁ to x ₁₄ , and the signal x ₂ is composed of bits x ₂₁ to x ₂₄ . The signal x ₃ is composed of bits x ₃₁ to x ₃₄ , and the signal x ₄ is composed of bits x ₄₁ to x ₄₄ . Here, the bit x ₁₁ , the bit x ₂₁ , the bit x _31, and the bit x ₄₁ are easy to mistake, and other bits are hard to mistake. Therefore, each bit _{x 42} ~ bits _{x 44} included in the signal _{x 4,} the bit _{x 11} signals _{x 1,} bit _{x 21} signal _{x 2,} used for the parity bits _{x 31} signal _{x 3,} signal _{x 4} The bit x ₄₁ included in is not used and always indicates 0. Formulas shown in FIG. 194 are used to calculate each of the bits x ₄₂ , x ₄₃ , and x ₄₄ . According to this equation, bits x ₄₂ , x ₄₃ , and x ₄₄ are calculated such that bit x ₄₂ = bit x ₁₁ , bit x ₄₃ = bit x ₂₁ , and bit x ₄₄ = bit x ₃₁ .

FIG. 195A is a diagram illustrating a visible light signal receiving method according to the present embodiment.

The receiver sequentially acquires the signal part of the visible light signal described above. The signal part includes a 4-bit address (Addr) and 8-bit data (Data). The receiver combines the data of these signal parts, and generates an ID composed of a plurality of data and a parity composed of one or a plurality of data.

FIG. 195B is a diagram showing rearrangement of visible light signals in the present embodiment.

FIG. 196 is a diagram illustrating another example of the visible light signal according to the present embodiment.

The visible light signal shown in FIG. 196 is configured by superimposing a high-frequency signal on the visible light signal shown in FIG. The frequency of the high frequency signal is, for example, 1 to several Gbps. Thus, data can be transmitted at a higher speed than the visible light signal shown in FIG.

FIG. 197 is a diagram illustrating another example of the detailed configuration of the visible light signal according to the present embodiment. The configuration of the visible light signal shown in FIG. 197 is the same as the configuration shown in FIG. 188. However, the time lengths C1 and C2 of the light control unit in the visible light signal shown in FIG. 197 are the time length C1 shown in FIG. And different from C2.

FIG. 198 is a diagram illustrating another example of a detailed configuration of a visible light signal in this embodiment. In the visible light signal shown in FIG. 198, data R and data L include eight V4PPM symbols. The rising position or falling position of the symbol D _Li included in the data L is the same as the rising position or the falling position of the symbol D _Ri included in the data R. However, the average luminance of the symbol D _{Li and} the average luminance of the symbol D _Ri may be the same or different.

FIG. 199 is a diagram illustrating another example of the detailed configuration of the visible light signal according to the present embodiment. The visible light signal shown in FIG. 199 is a signal for ID communication or low average luminance use, and is the same as the visible light signal shown in FIG. 189B.

FIG. 200 is a diagram illustrating another example of a detailed configuration of a visible light signal in this embodiment. In the visible light signal shown in FIG. 200, the even-numbered time length D _2i and the odd-numbered time length D _{2i + 1} in the data (Data) are equal.

FIG. 201 is a diagram showing another example of the detailed configuration of the visible light signal in the present embodiment. The data (Data) in the visible light signal shown in FIG. 201 includes a plurality of symbols that are signals of pulse position modulation.

FIG. 202 is a diagram showing another example of the detailed configuration of the visible light signal in the present embodiment. The visible light signal shown in FIG. 202 is a signal for continuous communication and is the same as the visible light signal shown in FIG.

203 to 211 are diagrams for explaining a method of determining the values x1 to x4 in FIG. 197. Note that x1 to x4 shown in FIGS. 203 to 211 are determined by a method similar to the method of determining the values (W1 to W4) of the symbols w1 to w4 shown in the following modifications. However, each of x1 to x4 is a code consisting of 4 bits, and is different from the codes w1 to w4 shown in the following modified example in that the first bit includes a parity.

(Modification 1)
FIG. 212 is a diagram showing an example of a detailed configuration of a visible light signal according to the first modification of the present embodiment. The visible light signal according to Modification 1 is the same as the visible light signal shown in FIG. 188 in the above embodiment, but the time length indicating the high or low luminance value is different from the visible light signal shown in FIG. Yes. For example, in the visible light signal according to this modification, the preamble time lengths P ₂ and P ₃ are 90 μs. Further, in the visible light signal according to the present modification, the time lengths D _R1 to D _R4 in the data R are determined according to a mathematical expression corresponding to the transmission target signal, as in the above embodiment. However, the mathematical formula in this modification is D _Ri = 120 + 30 × wi (iε1 to 4, wiε0 to 7). Note that wi is a 3-bit code and is a transmission target signal indicating an integer value of 0 to 7. Further, in the visible light signal according to the present modification, the time lengths D _L1 to D _L4 in the data L are determined according to a mathematical expression corresponding to the transmission target signal, as in the above embodiment. However, the mathematical formula in this modification is D _Li = 120 + 30 × (7−wi).

In the example shown in FIG. 212, the data R and the data L are included in the visible light signal, but only one of the data R and the data L may be included in the visible light signal. When it is desired to brighten the visible light signal, only bright data of the data R and data L may be transmitted. Further, the arrangement of the data R and the data L may be reversed.

FIG. 213 is a diagram illustrating another example of the visible light signal according to the present modification.

The visible light signal according to the modified example 1 may represent the signal to be transmitted only by the time length indicating the Low luminance value, similarly to the example illustrated in FIG. 189A (a) and FIG. 189B.

For example, as shown in FIG. 213, in the preamble, the time length indicating the high luminance value is less than 10 μs, for example, and the time lengths P ₁ to P ₃ indicating the low luminance value are 160 μs, 180 μs, and 160 μs, respectively. It is. In the data (Data), the time length indicating the High luminance value is less than 10 μs, and the time lengths D ₁ to D ₃ indicating the Low luminance value are adjusted according to the signal wi. Specifically, the time length D _i indicating the Low luminance value is D _i = 180 + 30 × wi (i∈1 to 4, wi∈0 to 7).

FIG. 214 is a diagram showing still another example of the visible light signal according to the present modification.

The visible light signal according to this modification may include a preamble and data as shown in FIG. As in the preamble shown in FIG. 212, the preamble alternately indicates High and Low luminance values along the time axis. The time lengths P ₁ to P ₄ in the preamble are 50 μs, 40 μs, 40 μs, and 50 μs, respectively. The data (Data) alternately indicates high and low luminance values along the time axis. For example, the data L is a luminance value of High only time length _{D 1,} represents a luminance value at the next time length _{D 2} only Low, represents a luminance value at the next time length _{D 3} only High, the next time length Only D ₄ indicates a low luminance value.

Here, the time length D _2i-1 + D _2i is determined according to a mathematical expression corresponding to the signal to be transmitted. That is, the sum of the time length indicating the high luminance value and the time length indicating the low luminance value following the luminance value is determined according to the mathematical formula. This formula is, for example, _{D 2i-1 + D 2i =} 100 + 20 × x i (i∈1 ~ N, x i ∈0 ~ 7, D 2i> 50μs, D 2i + 1> 50μs).

FIG. 215 is a diagram illustrating an example of packet modulation.

The signal generation device generates a visible light signal by the visible light signal generation method according to this modification. In the visible light signal generation method according to this modification, the packet is modulated (ie, converted) into the transmission target signal wi described above. Note that the signal generation device described above may be provided in the transmitter in each of the above embodiments, or may not be provided in the transmitter.

For example, as shown in FIG. 215, the signal generation device converts the packet into a transmission target signal including numerical values indicated by the symbols w1, w2, w3, and w4. These codes w1, w2, w3 and w4 are codes each consisting of 3 bits from the first bit to the third bit, and indicate integer values of 0 to 7, as shown in FIG.

Here, in each of the codes w1 to w4, the value of the first bit is b1, the value of the second bit is b2, and the value of the third bit is b3. B1, b2 and b3 are 0 or 1. In this case, each of the numerical values W1 to W4 indicated by the symbols w1 to w4 is, for example, b1 × 2 ⁰ + b2 × 2 ¹ + b3 × 2 ² .

The packet is composed of 0 to 4 bits of address data (A1 to A4), 4 to 7 bits of main data Da (Da1 to Da7), 3 to 4 bits of sub data Db (Db1 to Db4), The stop bit value (S) is included as data. Each of Da1 to Da7, A1 to A4, Db1 to Db4, and S represents a bit value, that is, 0 or 1.

That is, when the signal generation apparatus modulates a packet into a signal to be transmitted, the signal generation apparatus assigns data included in the packet to any bit of the codes w1, w2, w3, and w4. As a result, the packet is converted into a signal to be transmitted that includes numerical values indicated by the symbols w1, w2, w3, and w4.

When the signal generator allocates the data included in the packet, specifically, at least one of the main data Da included in the packet is added to the first bit string composed of the first bits (bit 1) of the codes w1 to w4. Parts (Da1 to Da4) are allocated. Further, the signal generation device assigns the value (S) of the stop bit included in the packet to the second bit (bit2) of the code w1. Further, the signal generation device includes a part (Da5 to Da7) of the main data Da included in the packet or the packet in the second bit string including the second bits (bit2) of the codes w2 to w4. At least a part (A1 to A3) of the address data is allocated. Further, the signal generation device includes at least a part (Db1 to Db3) of the sub data Db included in the packet and the sub data Db in the third bit string including the third bits (bit 3) of the codes w1 to w4. A part (Db4) or a part (A4) of address data is allocated.

When the third bits (bit 3) of the codes w1 to w4 are all 0, the numerical values indicated by the codes are 3 by the above-mentioned “b1 × 2 ⁰ + b2 × 2 ¹ + b3 × 2 ² ”. It is suppressed to the following. Therefore, the time length D _Ri can be shortened by the formula D _Ri = 120 + 30 × w _i (iε1 to 4, w _i ε0 to 7) shown in FIG. As a result, the time for transmitting one packet can be shortened, and the packet can be received from a further distance.

216 to 226 are diagrams showing processing for generating a packet from original data.

The signal generation device according to this modification determines whether to divide the original data according to the bit length of the original data. Then, the signal generation device generates at least one packet from the original data by performing processing according to the determination result. That is, the signal generation device divides the original data into a larger number of packets as the bit length of the original data is longer. Conversely, if the bit length of the original data is shorter than the predetermined bit length, the signal generation device generates a packet without dividing the original data.

In this way, when the signal generating device generates at least one packet from the original data, each of the at least one packet is converted into the above-described transmission target signals, that is, the codes w1 to w4.

In FIGS. 216 to 226, Data indicates original data, Data indicates main original data included in the original data, and Data indicates sub-original data included in the original data. Da (k) indicates the main element data itself or the k-th part of a plurality of parts constituting the data including the main element data and the parity. Similarly, Db (k) indicates the sub-element data itself or the k-th part of a plurality of parts constituting the data including the sub-element data and the parity. For example, Da (2) indicates a second part of a plurality of parts constituting data including main element data and parity. S represents a start bit, and A represents address data.

Also, the top notation shown in each block is a label for identifying original data, main original data, sub original data, start bit, address data, and the like. Further, the numerical value at the center shown in each block is the bit size (number of bits), and the numerical value at the bottom is the value of each bit.

FIG. 216 is a diagram illustrating a process of dividing the original data into one.

For example, if the bit length of the original data (Data) is 7 bits, the signal generating device generates one packet without dividing the original data. Specifically, the original data includes 4-bit main original data Data (Da1 to Da4) and 3-bit suboriginal data Data (Db1 to Db3), respectively, as main data Da (1) and sub data Db ( Included as 1). In this case, the signal generator adds a packet by adding a start bit S (S = 1) and address data (A1 to A4) consisting of 4 bits and indicating “0000” to the original data. Generate. Note that the start bit S = 1 indicates that the packet including the start bit is a terminal packet.

By converting this packet, the signal generator converts the code w1 = (Da1, S = 1, Db1), the code w2 = (Da2, A1 = 0, Db2), and the code w3 = (Da3, A2 = 0, Db3). ), And a code w4 = (Da4, A3 = 0, A4 = 0). Further, the signal generation device generates a signal to be transmitted including numerical values W1, W2, W3, and W4 indicated by the symbols w1, w2, w3, and w4, respectively.

In this modification, wi is expressed as a 3-bit code and also as a decimal number. Therefore, in this modified example, to make the explanation easy to understand, wi (w1 to w4) used as a decimal number is expressed as a numerical value Wi (W1 to W4).

FIG. 217 is a diagram showing a process of dividing the original data into two.

For example, if the bit length of the original data (Data) is 16 bits, the signal generator generates two intermediate data by dividing the original data. Specifically, the original data includes 10-bit main original data Dataa and 6-bit secondary original data Datab. In this case, the signal generation device generates first intermediate data including main original data Dataa and 1-bit parity corresponding to the main original data Dataa, and corresponds to the sub original data Data and the sub original data Datab. Second intermediate data including 1-bit parity is generated.

Next, the signal generation device divides the first intermediate data into 7-bit main data Da (1) and 4-bit main data Da (2). Further, the signal generation device divides the second intermediate data into sub-data Db (1) consisting of 4 bits and sub-data Db (2) consisting of 3 bits. The main data is one part of a plurality of parts constituting data including main original data and parity. Similarly, the sub data is one part of a plurality of parts constituting data including the sub original data and the parity.

Next, the signal generation device generates a 12-bit first packet including a start bit S (S = 0), main data Da (1), and sub data Db (1). As a result, a first packet that does not include address data is generated.

Further, the signal generation device includes a 12-bit start bit S (S = 1), 4-bit address data indicating “1000”, main data Da (2), and sub-data Db (2). A second packet is generated. Note that the start bit S = 0 indicates that, among a plurality of generated packets, a packet including the start bit is a packet that does not end. Further, the start bit S = 1 indicates that a packet including the start bit among a plurality of generated packets is a packet at the end.

Thereby, the original data is divided into the first packet and the second packet.

The signal generator converts the first packet to generate a code w1 = (Da1, S = 0, Db1), a code w2 = (Da2, Da7, Db2), a code w3 = (Da3, Da6, Db3), and A code w4 = (Da4, Da5, Db4) is generated. Further, the signal generation device generates a signal to be transmitted including numerical values W1, W2, W3, and W4 indicated by the symbols w1, w2, w3, and w4, respectively.

Further, the signal generation device converts the second packet to generate a code w1 = (Da1, S = 1, Db1), a code w2 = (Da2, A1 = 1, Db2), a code w3 = (Da3, A2 = 0, Db3), and a code w4 = (Da4, A3 = 0, A4 = 0). Further, the signal generation device generates a signal to be transmitted including numerical values W1, W2, W3, and W4 indicated by the symbols w1, w2, w3, and w4, respectively.

FIG. 218 is a diagram showing processing for dividing the original data into three parts.

For example, if the bit length of the original data (Data) is 17 bits, the signal generator generates two intermediate data by dividing the original data. Specifically, the original data includes 10-bit main original data Dataa and 7-bit secondary original data Datab. In this case, the signal generation device generates first intermediate data including main original data Dataa and 6-bit parity corresponding to the main original data Dataa. Further, the signal generation device generates second intermediate data including the secondary original data Datab and 4-bit parity corresponding to the secondary original data Datab. For example, the signal generation device generates a parity by CRC (Cyclic Redundancy Check).

Next, the signal generating device converts the first intermediate data into main data Da (1) having 6-bit parity, main data Da (2) having 6 bits, and main data Da (3) having 4 bits. And split into Further, the signal generating device converts the second intermediate data into sub-data Db (1) having 4-bit parity, sub-data Db (2) having 4 bits, and sub-data Db (3) having 3 bits. Divide into

Next, the signal generating device has 12 bits including a start bit S (S = 0), address data including 1 bit and indicating “0”, main data Da (1), and sub data Db (1). The first packet is generated. Further, the signal generation device includes a 12-bit start bit S (S = 0), 1-bit address data indicating “1”, main data Da (2), and sub-data Db (2). A second packet is generated. Further, the signal generation device has 12 bits including start bit S (S = 1), 4-bit address data indicating “0100”, main data Da (3), and sub-data Db (3). A third packet is generated.

Thereby, the original data is divided into the first packet, the second packet, and the third packet.

By converting the first packet, the signal generator converts the code w1 = (Da1, S = 0, Db1), the code w2 = (Da2, A1 = 0, Db2), and the code w3 = (Da3, Da6, Db3). , And code w4 = (Da4, Da5, Db4). Further, the signal generation device generates a signal to be transmitted including numerical values W1, W2, W3, and W4 indicated by the symbols w1, w2, w3, and w4, respectively.

Similarly, by converting the second packet, the signal generation device converts the code w1 = (Da1, S = 0, Db1), the code w2 = (Da2, A1 = 1, Db2), and the code w3 = (Da3, Da6). , Db3) and w4 = (Da4, Da5, Db4). Further, the signal generation device generates a signal to be transmitted including numerical values W1, W2, W3, and W4 indicated by the symbols w1, w2, w3, and w4, respectively.

Similarly, by converting the third packet, the signal generation device converts the code w1 = (Da1, S = 1, Db1), the code w2 = (Da2, A1 = 0, Db2), and the code w3 = (Da3, A2). = 1, Db3), and a code w4 = (Da4, A3 = 0, A4 = 0). Further, the signal generation device generates a signal to be transmitted including numerical values W1, W2, W3, and W4 indicated by the symbols w1, w2, w3, and w4, respectively.

FIG. 219 is a diagram showing another example of the process of dividing the original data into three parts.

In the example shown in FIG. 218, 6-bit or 4-bit parity is generated by CRC, but 1-bit parity may be generated.

In this case, if the bit length of the original data (Data) is 25 bits, the signal generation device generates two intermediate data by dividing the original data. Specifically, the original data includes 15-bit main original data Dataa and 10-bit auxiliary original data Datab. In this case, the signal generation device generates first intermediate data including main original data Dataa and 1-bit parity corresponding to the main original data Dataa, and corresponds to the sub original data Data and the sub original data Datab. Second intermediate data including 1-bit parity is generated.

Next, the signal generation device converts the first intermediate data into 6-bit main data Da (1) including parity, 6-bit main data Da (2), and 4-bit main data Da (3 ) And split. Further, the signal generation device converts the second intermediate data into 4-bit sub data Db (1) including parity, 4-bit sub data Db (2), and 3-bit sub data Db (3). And split into

Next, the signal generation device generates a first packet, a second packet, and a third packet from the first intermediate data and the second intermediate data, as in the example shown in FIG.

FIG. 220 is a diagram illustrating another example of the process of dividing the original data into three parts.

In the example shown in FIG. 218, a 6-bit parity is generated by CRC for the main original data Dataa, and a 4-bit parity is generated by CRC for the secondary original data Datab. However, the parity may be generated by CRC for the entire main element data Dataa and sub element data Dataab.

In this case, if the bit length of the original data (Data) is 22 bits, the signal generating device generates two intermediate data by dividing the original data.

Specifically, the original data includes 15-bit main original data Dataa and 7-bit secondary original data Datab. The signal generation device generates first intermediate data including main original data Dataa and 1-bit parity corresponding to the main original data Dataa. Further, the signal generation device generates the 4-bit parity by CRC with respect to the whole of the main original data Dataa and the sub original data Datab. Then, the signal generation device generates second intermediate data including the sub-element data Datab and 4-bit parity.

Next, the signal generation device converts the first intermediate data into 6-bit main data Da (1) including parity, 6-bit main data Da (2), and 4-bit main data Da (3 ) And split. Further, the signal generating device converts the second intermediate data into sub data Db (1) consisting of 4 bits, sub data Db (2) consisting of 4 bits including a part of CRC parity, and the remainder of CRC parity. Are divided into sub-data Db (3) consisting of 3 bits.

Next, the signal generation device generates a first packet, a second packet, and a third packet from the first intermediate data and the second intermediate data, as in the example shown in FIG.

Of the specific examples of the process of dividing the original data into three, the process shown in FIG. 218 is referred to as version 1, the process shown in FIG. 219 is referred to as version 2, and the process shown in FIG. 220 is referred to as version 3.

FIG. 221 is a diagram showing processing for dividing the original data into four parts. FIG. 222 is a diagram showing processing for dividing the original data into five parts.

The signal generation device divides the original data into four or five as in the process of dividing the original data into three, that is, the processes shown in FIGS. 218 to 220.

FIG. 223 is a diagram showing a process of dividing the original data into 6, 7 or 8 parts.

For example, if the bit length of the original data (Data) is 31 bits, the signal generator generates two intermediate data by dividing the original data. Specifically, the original data includes 16-bit main original data Dataa and 15-bit auxiliary original data Datab. In this case, the signal generation device generates first intermediate data including main original data Dataa and 8-bit parity corresponding to the main original data Dataa. Further, the signal generation device generates second intermediate data including the secondary original data Datab and the 8-bit parity corresponding to the secondary original data Datab. For example, the signal generator generates parity using a Reed-Solomon code.

Here, when 4 bits are handled as one symbol in the Reed-Solomon code, the bit length of each of the main original data Dataa and the sub-original data Data must be an integer multiple of 4 bits. However, the secondary data Datab is 15 bits as described above, and is 1 bit less than 16 bits, which is an integer multiple of 4 bits.

Therefore, when generating the second intermediate data, the signal generation device performs padding on the secondary original data Datab, and reads the 8-bit parity corresponding to the padded 16-bit secondary data Data- Generated by Solomon code.

Next, the signal generation device divides each of the first intermediate data and the second intermediate data into six parts (4 bits or 3 bits) in the same manner as described above. Then, the signal generation device generates a first packet that includes a start bit, 3-bit or 4-bit address data, first main data, and first sub-data. Similarly, the signal generation device generates the second packet to the sixth packet.

FIG. 224 is a diagram showing another example of the process for dividing the original data into 6, 7 or 8 parts.

In the example shown in FIG. 223, the parity is generated by the Reed-Solomon code, but the parity may be generated by CRC.

For example, if the bit length of the original data (Data) is 39 bits, the signal generation device generates two intermediate data by dividing the original data. Specifically, the original data includes 20-bit main original data Dataa and 19-bit auxiliary original data Datab. In this case, the signal generation device generates first intermediate data including main original data Dataa and 4-bit parity corresponding to the main original data Dataa, and corresponds to the sub original data Data and the sub original data Datab. Second intermediate data including 4-bit parity is generated. For example, the signal generation device generates parity by CRC.

Next, the signal generation device divides each of the first intermediate data and the second intermediate data into six parts (4 bits or 3 bits) in the same manner as described above. Then, the signal generation device generates a first packet that includes a start bit, 3-bit or 4-bit address data, first main data, and first sub-data. Similarly, the signal generation device generates the second packet to the sixth packet.

Of the specific examples of the process of dividing the original data into 6, 7 or 8 parts, the process shown in FIG. 223 is referred to as version 1 and the process shown in FIG. 224 is referred to as version 2.

FIG. 225 is a diagram showing a process for dividing the original data into nine parts.

For example, if the bit length of the original data (Data) is 55 bits, the signal generator generates nine packets from the first packet to the ninth packet by dividing the original data. In FIG. 225, the first intermediate data and the second intermediate data are omitted.

Specifically, the bit length of the original data (Data) is 55 bits, which is 1 bit less than 56 bits, which is an integer multiple of 4 bits. Therefore, the signal generation device performs padding on the original data, and generates parity (16 bits) for the 56-bit original data subjected to the padding by the Reed-Solomon code.

Next, the signal generating apparatus divides the entire data including 16-bit parity and 55-bit original data into nine data DaDb (1) to DaDb (9).

Each of the data DaDb (k) includes a k-th 4 bits part included in the main original data Dataa and a k-th 4 bits part included in the sub-original data Datab. K is an integer from 1 to 8. The data DaDb (9) includes a ninth 4-bit portion included in the main original data Dataa and a ninth 3-bit portion included in the sub-original data Datab.

Next, the signal generator generates the first packet to the ninth packet by adding the start bit S and the address data to each of the nine data DaDb (1) to DaDb (9).

FIG. 226 is a diagram showing a process of dividing the original data into any number between 10 and 16.

For example, if the bit length of the original data (Data) is 7 × (N−2) bits, the signal generating apparatus divides the original data to obtain N packets from the first packet to the Nth packet. Is generated. N is an integer from 10 to 16. In FIG. 226, the first intermediate data and the second intermediate data are omitted.

Specifically, the signal generation device generates parity (14 bits) for the original data composed of 7 × (N−2) bits by the Reed-Solomon code. In this Reed-Solomon code, 7 bits are treated as one symbol.

Next, the signal generator divides the entire data including the 14-bit parity and the 7 × (N−2) -bit original data into N pieces of data DaDb (1) to DaDb (N).

Each of the data DaDb (k) includes a portion made up of the kth 4 bits included in the main original data Dataa and a portion made up of the kth 3 bits included in the sub original data Datab. K is an integer from 1 to (N−1).

Next, the signal generator generates the first packet to the Nth packet by adding the start bit S and the address data to each of the nine data DaDb (1) to DaDb (N).

FIG. 227 to FIG. 229 are diagrams showing an example of the relationship between the number of divisions of the original data, the data size, and the error correction code.

Specifically, FIGS. 227 to 229 collectively show the above relationships in the processes shown in FIGS. 216 to 226. As described above, there are versions 1 to 3 for the process of dividing the original data into three, and there are version 1 and version 2 for the process of dividing the original data into 6, 7 or 8 parts. FIG. 227 shows the above relationship in version 1 of a plurality of versions if there are a plurality of versions for the number of divisions. Similarly, FIG. 228 shows the above relationship in version 2 of a plurality of versions if there are a plurality of versions for the number of divisions. Similarly, FIG. 229 shows the above relationship in version 3 of a plurality of versions if there are a plurality of versions for the number of divisions.

In this modification, there are a short mode and a full mode. In the sheet mode, the sub data in the packet is 0, and all the bits of the third bit string shown in FIG. 215 are 0. In this case, the numerical values W1 to W4 indicated by the symbols w1 to w4 are suppressed to 3 or less by the above-described “b1 × 2 ⁰ + b2 × 2 ¹ + b3 × 2 ² ”. As a result, as shown in FIG. 212, the time lengths D _R1 to D _R4 in the data R become shorter because they are determined by D _Ri = 120 + 30 × wi (iε1 to 4, wiε0 to 7). That is, in the short mode, the visible light signal per packet can be shortened. By shortening the visible light signal per packet, the receiver can receive the packet even from a distance, and the communication distance can be increased.

On the other hand, in the full mode, one of the bits in the third bit string shown in FIG. In this case, the visible light signal is not as short as in the short mode.

In this modification, as shown in FIGS. 227 to 229, a short mode visible light signal can be generated if the number of divisions is small. In FIG. 227 to FIG. 229, the short mode data size indicates the number of bits of the main original data (Dataa), and the full mode data size indicates the number of bits of the original data (Data).

(Summary of Embodiment 20)
FIG. 230A is a flowchart illustrating a visible light signal generation method according to this embodiment.

The visible light signal generation method in the present embodiment is a method for generating a visible light signal transmitted by a change in luminance of a light source provided in a transmitter, and includes steps SD1 to SD3.

In step SD1, a preamble which is data in which each of the first and second luminance values, which are different luminance values, appears alternately along the time axis for a predetermined time length is generated.

In step SD2, in the data in which the first and second luminance values appear alternately along the time axis, the length of time that each of the first and second luminance values continues depends on the signal to be transmitted. The first data is generated by determining according to the first method.

Finally, in step SD3, a visible light signal is generated by combining the preamble and the first data.

For example, as shown in FIG. 188, the first and second luminance values are High and Low, and the first data is data R or data L. By transmitting the visible light signal generated in this way, as shown in FIGS. 191 to 193, the number of received packets can be increased and the reliability can be increased. As a result, communication between various devices can be enabled.

The visible light signal generation method may further include a time in which each of the first and second luminance values continues in the data in which the first and second luminance values appear alternately along the time axis. The second data having a complementary relationship with the brightness expressed by the first data is generated by determining the length according to the second method according to the signal to be transmitted, and the generation of the visible light signal Then, the visible light signal may be generated by combining the preamble and the first and second data in the order of the first data, the preamble, and the second data.

For example, as shown in FIG. 188, the first and second luminance values are High and Low, and the first and second data are data R and data L.

Further, when each of a and b is a constant, a numerical value included in the transmission target signal is n, and a constant that is the maximum value that the numerical value n can take is m, the first method is expressed by a + b × n. , A time length in which the first or second luminance value lasts in the first data is determined. The second method uses the second data by a + b × (mn). The time length in which the first or second luminance value lasts may be determined.

For example, as shown in FIG. 188, a is 120 μs, b is 20 μs, n is any integer value from 0 to 15 (numerical value indicated by signal x _i ), and m is 15. .

In the complementary relationship, the sum of the time length of the entire first data and the time length of the entire second data may be constant.

The visible light signal generation method further generates a dimming unit that is data for adjusting brightness expressed by the visible light signal, and the visible light signal generation further includes the dimming unit. The visible light signal may be generated by combining parts.

Light regulator, for example in FIG. 188, indicates the luminance values of High for a time length _{C 1,} a signal indicating the luminance value of the Low only time length _{C 2} (Dimming). Thereby, the brightness of the visible light signal can be arbitrarily adjusted.

FIG. 230B is a block diagram illustrating a configuration of the signal generation device according to the present embodiment.

The signal generation device D10 in the present embodiment is a signal generation device that generates a visible light signal transmitted by the luminance change of the light source provided in the transmitter, and includes a preamble generation unit D11, a data generation unit D12, and a combining unit D13.

The preamble generation unit D11 generates a preamble that is data in which the first and second luminance values, which are different from each other, appear alternately along the time axis for a predetermined time length.

In the data in which the first and second luminance values appear alternately along the time axis, the data generation unit D12 determines the time length that each of the first and second luminance values continues according to the transmission target signal. The first data is generated by determining according to the first method.

The combining unit D13 generates a visible light signal by combining the preamble and the first data.

By transmitting the visible light signal generated in this way, as shown in FIGS. 191 to 193, the number of received packets can be increased and the reliability can be increased. As a result, communication between various devices can be enabled.

(Summary of Modification 1 of Embodiment 20)
Further, as in Modification 1 of Embodiment 20, the visible light signal generation method further determines whether or not to divide the original data according to the bit length of the original data, and the result of the determination By performing processing according to the above, at least one packet may be generated from the original data. Then, each of the at least one packet may be converted into a signal to be transmitted.

In the conversion to the signal to be transmitted, as shown in FIG. 215, for each target packet included in the at least one packet, the data included in the target packet is changed to 3 from the first bit to the third bit. By assigning the bit to any one of the bits w1, w2, w3, and w4, the target packet is converted into a signal to be transmitted that includes a numerical value indicated by each of the symbols w1, w2, w3, and w4.

In the data allocation, at least a part of the main data included in the target packet is allocated to the first bit string including the first bits of the codes w1 to w4. The value of the stop bit included in the target packet is assigned to the second bit of the code w1. A part of the main data included in the target packet or at least a part of the address data included in the target packet is assigned to the second bit string composed of the second bits of each of the codes w2 to w4. The sub-data included in the target packet is allocated to the third bit string including the third bits.

Here, the stop bit indicates whether or not the target packet is at the end among the generated at least one packet. The address data indicates the order of the target packet among at least one generated packet as an address. Each of the main data and the sub data is data for restoring the original data.

Also, when a and b are constants and the numerical values indicated by the symbols w1, w2, w3 and w4 are W1, W2, W3 and W4, for example, as shown in FIG. Is a method of determining the length of time for which the first or second luminance value in the first data lasts by a + b × W1, a + b × W2, a + b × W3 and a + b × W4.

For example, in each of the codes w1 to w4, the value of the first bit is b1, the value of the second bit is b2, and the value of the third bit is b3. In this case, each of the values W1 to W4 indicated by the symbols w1 to w4 is, for example, b1 × 2 ⁰ + b2 × 2 ¹ + b3 × 2 ² . Accordingly, in the codes w1 to w4, the values W1 to W4 indicated by the codes w1 to w4 are larger when the second bit is set to 1 than the first bit is set to 1. Further, when the third bit is set to 1, the values W1 to W4 indicated by the symbols w1 to w4 are larger than when the second bit is set to 1. When the values W1 to W4 indicated by these codes w1 to w4 are large, the time length (for example, D _Ri ) in which each of the first and second luminance values described above continues is increased, and therefore the luminance of the visible light signal is increased. False detection can be suppressed and reception errors can be reduced. On the other hand, if the values W1 to W4 indicated by these symbols w1 to w4 are small, the time length for which each of the first and second luminance values described above lasts becomes short, so that the erroneous detection of the luminance of the visible light signal is detected. Is relatively easy to occur.

Therefore, in the first modification of the twentieth embodiment, the stop bits and addresses important for receiving the original data are preferentially assigned to the second bits of the codes w1 to w4, thereby reducing the reception error. Can be achieved. The code w1 defines the length of time that the High or Low luminance value closest to the preamble continues. That is, since the code w1 is closer to the preamble than the other codes w2 to w4, it is more likely to be received more appropriately than these other codes. Therefore, in the first modification of the twentieth embodiment, it is possible to further suppress a reduction in reception error by assigning the stop bit to the second bit of code w1.

Further, in the first modification of the twentieth embodiment, the main data is preferentially assigned to the first bit string in which erroneous detection is relatively likely to occur. However, if an error correction code (parity) is included in the main data, reception errors of the main data can be suppressed.

Furthermore, in the first modification of the twentieth embodiment, the sub data is assigned to the third bit string composed of the third bits of the codes w1 to w4. Therefore, if the sub data is set to 0, the length of time that each of the High and Low luminance values defined by the symbols w1 to w4 continues can be significantly shortened. As a result, a so-called short mode in which the transmission time of the visible light signal per packet can be significantly shortened can be realized. In this short mode, since the transmission time is short as described above, the packet can be easily received even from a distance. Therefore, the communication distance of visible light communication can be increased.

Further, in the first modification of the twentieth embodiment, as shown in FIG. 217, in the generation of at least one packet, the original data is divided into two packets to generate two packets. When a packet that is not at the end of the two packets is converted into a signal to be transmitted as a target packet, the second bit string includes a main bit included in the packet at the end without assigning at least a part of the address data. Allocate part of the data.

For example, the address data is not included in the packet (Packet 1) not at the end shown in FIG. A packet not at the end has 7 bits of main data Da (1). Therefore, as shown in FIG. 215, data Da1 to Da4 included in 7-bit main data Da (1) are assigned to the first bit string, and data Da5 to Da7 are assigned to the second bit string.

Thus, when the original data is divided into two packets, if there is a start bit (S = 0) in the packet not at the end, that is, the first packet, the address data is unnecessary. Therefore, all the bits of the second bit string can be used for the main data, and the amount of data included in the packet can be increased.

In the data allocation in the first modification of the twentieth embodiment, among the three bits included in the second bit string, the first bit in the arrangement order is preferentially used for address data allocation, and the second When all of the address data is assigned to one or two bits on the head side of the bit string, the main data is assigned to one or two bits to which no address data is assigned in the second bit string. Allocate a part of For example, in Packet 1 in FIG. 218, 1-bit address data A1 is assigned to one bit (second bit of code w2) on the head side of the second bit string. In this case, main data Da6 and Da5 are assigned to two bits (second bits of the symbols w3 and w4) to which no address data is assigned in the second bit string.

Thereby, the second bit string can be shared by the address data and a part of the main data, and the degree of freedom of the packet configuration can be increased.

In addition, in the data assignment in the first modification of the twentieth embodiment, when all of the address data cannot be assigned to the second bit string, the address data is assigned to any bit of the third bit string. The remaining part excluding the part assigned to the second bit string is assigned. For example, in Packet 3 in FIG. 218, all of the 4-bit address data A1 to A4 cannot be assigned to the second bit string. In this case, the remaining part A4 excluding the parts A1 to A3 assigned to the second bit string of the address data A1 to A4 is added to the last bit (third bit of the code w4) of the third bit string. Assign.

This makes it possible to appropriately assign the address data to the codes w1 to w4.

In addition, in the data allocation in the first modification of the twentieth embodiment, when the terminal packet of at least one packet is converted into the transmission target signal as the target packet, the second bit string and the third bit string The address data is assigned to any one of the bits included in. For example, the number of bits of the address data of the end packet in FIGS. 217 to 226 is 4. In this case, 4-bit address data A1 to A4 are allocated to the second bit string and the last bit (third bit of the code w4) in the third bit string.

This makes it possible to appropriately assign the address data to the codes w1 to w4.

Further, in the generation of at least one packet in the first modification of the twentieth embodiment, the original data is divided into two to generate two divided original data, and each error of the two divided original data A correction code is generated. Then, two or more packets are generated using the two divided original data and the error correction code generated for each of the two divided original data. When generating the error correction code for each of the two divided source data, the number of bits of any one of the two divided source data is less than the number of bits required for generating the error correcting code First, padding is performed on the divided original data, and an error correction code for the padded divided original data is generated. For example, as shown in FIG. 223, when generating parity by Reed-Solomon code for Data which is the division source data, if the Data is only 15 bits and less than 16 bits, the Data And the parity is generated by Reed-Solomon code for the padded divided original data (16 bits).

Thereby, even if the number of bits of the division source data is less than the number of bits required for generating the error correction code, an appropriate error correction code can be generated.

In addition, in the data assignment in the first modification of the twentieth embodiment, when the sub data indicates 0, 0 is assigned to all the bits included in the third bit string. Thereby, the short mode described above can be realized, and the communication distance of visible light communication can be increased.

(Embodiment 21)
FIG. 231 is a diagram illustrating a method of receiving a high-frequency visible light signal in this embodiment.

When the receiver receives a high-frequency visible light signal, for example, as shown in FIG. 231 (a), the receiver provides a guard time (guard interval) at the rise and fall of the visible light signal. The receiver compensates the high frequency signal at the guard time by copying the high frequency signal received immediately before the guard time without using the high frequency signal at the guard time. Note that the high-frequency signal superimposed on the visible light signal may be modulated by OFDM (Orthogonal Frequency Division Multiplexing).

In addition, when the high frequency signal indicating the high luminance value and the high frequency signal indicating the low luminance value are separated from the high frequency visible light signal, the receiver automatically adjusts the gain of these high frequency signals (Automatic Gain Control). Thereby, the gain (luminance value) of the high frequency signal is unified.

FIG. 232A is a diagram showing another method for receiving a high-frequency visible light signal in the present embodiment.

A receiver that receives a high-frequency visible light signal includes an image sensor as in the above embodiments, and further includes a DMD (Digital Mirror Device) element and a photosensor. The photosensor is a photodiode or an avalanche photodiode.

The receiver takes an image of a transmitter (light source) that transmits a high-frequency visible light signal. Thereby, the receiver acquires a bright line image including a stripe pattern of bright lines. This bright stripe pattern appears due to a change in luminance of the signal other than the high-frequency signal in the high-frequency visible light signal, that is, the visible light signal shown in FIG. The receiver specifies the positions (x1, y1) and (x2, y2) of the stripe pattern of the bright line in the bright line image. Then, the receiver specifies the micromirror corresponding to each of the positions (x1, y1) and (x2, y2) in the DMD element. These micromirrors receive light of a high-frequency visible light signal that shows a stripe pattern of bright lines. Therefore, the receiver adjusts the angle of each micromirror so that only the reflected light from the identified micromirror is received by the photosensor among the plurality of micromirrors included in the DMD element. That is, the receiver turns on the micromirror so that the photosensor 1 receives only the light reflected by the micromirror corresponding to the position (x1, y1). Further, the receiver turns on the micromirror so that only the light reflected by the micromirror corresponding to the position (x2, y2) is received by the photosensor 2. Then, the receiver turns off each micromirror other than the identified micromirrors. As a result, the reflected light from the micromirror turned off is absorbed by the light absorber (black body). Further, the high-frequency visible light signal is appropriately received by the photosensor by the micromirror that is turned on. Note that the tilt angle (+ 0 ° or −0 °) of each micromirror of the DMD element is switched by switching between ON and OFF. When the micromirror is ON, the micromirror outputs reflected light toward the photosensor, and when the micromirror is OFF, the micromirror outputs reflected light toward the light absorbing portion.

Further, as shown in FIG. 232A, the receiver may include a half mirror and a light emitting element. The light emitting element 1 transmits a visible light signal (or a high frequency visible light signal) by emitting light and changing the luminance. The light output from the light emitting element 1 is reflected by the half mirror, and further reflected by the ON micromirror corresponding to the position (x1, y1) in the DMD element. As a result, the visible light signal from the light emitting element 1 is transmitted to the transmitter corresponding to the stripe pattern of the bright line at the position (x1, y1). As a result, the receiver and the transmitter corresponding to the stripe pattern of the bright line at the position (x1, y1) can perform bidirectional communication. Similarly, the light output from the light emitting element 2 is reflected by the half mirror, and further reflected by the ON micromirror corresponding to the position (x2, y2) in the DMD element. As a result, the visible light signal from the light emitting element 2 is transmitted to the transmitter corresponding to the stripe pattern of the bright line at the position (x2, y2). Accordingly, the receiver and the transmitter corresponding to the bright stripe pattern at the position (x2, y2) can perform bidirectional communication.

Thus, even if there are a plurality of transmitters (light sources) photographed by the image sensor, the receiver can perform bidirectional communication simultaneously with these transmitters at high speed. For example, when the receiver includes 100 photosensors capable of receiving at 10 Gbps and these receivers communicate with 100 transmitters, a communication speed of 1 Tbps can be realized.

FIG. 232B is a diagram showing still another method for receiving a high-frequency visible light signal in the present embodiment.

The receiver includes, for example, lenses L1 and L2, a plurality of half mirrors, a DMD element, an image sensor, a light absorption unit (black body), a processing unit, a DMD control unit, and photosensors 1 and 2. And light emitting elements 1 and 2.

Such a receiver performs two-way communication with two cars on the same principle as the example shown in FIG. 232A. Two cars transmit high-frequency visible light signals by outputting light from the headlights and changing the brightness of the headlights. One car outputs normal light (light that does not change in luminance) from the headlight.

The image sensor receives these high-frequency visible light signals and normal light via the lens L1. Thereby, similarly to the example shown in FIG. 232A, a bright line image including a bright line stripe pattern generated by the high-frequency visible light signal is obtained. The processing unit specifies the positions of these striped patterns in the bright line image. The DMD control unit identifies micromirrors corresponding to the identified stripe pattern positions from among the plurality of micromirrors included in the DMD element, and turns on those micromirrors.

Thus, the high-frequency visible light signal transmitted from each of the two cars through the lens L1 and the half mirror is reflected by the micromirror of the DMD element and travels toward the lens L2. Further, normal light from a headlight of one car does not generate a stripe pattern of bright lines due to the light, so that even if it passes through the lens L1 and the half mirror, it is reflected by the OFF micromirror of the DMD element. The light reflected by the OFF micromirror is absorbed by the light absorbing portion (black body).

The high-frequency visible light signal that has passed through the lens L2 passes through the half mirror and is received by the photosensor 1 or 2. Thereby, the high frequency visible light signal from each vehicle can be received. If the light-emitting elements 1 and 2 output a visible light signal (or high-frequency visible light signal) to the half mirror, the visible light signal is reflected by the half mirror, passes through the lens L2, and further DMD. Reflected by the ON micromirror in the element. As a result, the visible light signals from the light emitting elements 1 and 2 are transmitted to the vehicle that has transmitted the high-frequency visible light signal through the half mirror and the lens L1. That is, the receiver can perform two-way communication with a plurality of vehicles that transmit high-frequency visible light signals.

As described above, the receiver in the present embodiment acquires the bright line image by the image sensor, and specifies the position of the bright line stripe pattern in the bright line image. And a receiver specifies the micromirror corresponding to the position of the striped pattern among the several micromirrors contained in a DMD element. And a receiver receives a high frequency visible light signal with a photo sensor by turning on the micromirror. Further, the receiver can transmit the visible light signal to the transmitter by outputting a visible light signal from the light emitting element and reflecting the signal to the micromirror which is turned on.

In the example shown in FIGS. 232A and 232B, a half mirror and a lens are used as the optical device, but any optical device may be used as long as it has the same function as these. Further, the arrangement of the DMD element, the half mirror, and the lens is an example and may be arranged in any manner. In the example illustrated in FIGS. 232A and 232B, the receiver includes two sets of photosensors and light-emitting elements, but may include only one set or three or more sets. One light emitting element may transmit a visible light signal to a plurality of ON microlenses. Thereby, the receiver can simultaneously transmit the same visible light signal to a plurality of transmitters. Further, the receiver may include only a part of the components without including all the components illustrated in FIGS. 232A and 232B.

FIG. 233 is a diagram showing a method for outputting a high-frequency signal in the present embodiment.

The signal output device that outputs a high-frequency signal superimposed on the visible light signal shown in FIG. 188 includes, for example, a blue laser and a phosphor. That is, similarly to the example shown in FIG. 114A, the signal output device irradiates the phosphor with high-frequency blue laser light from the blue laser. Thus, the signal output device outputs high-frequency natural light as a high-frequency signal.

(Embodiment 22)
In this embodiment, an autonomous flight device (also referred to as a drone) using visible light communication in each of the above embodiments will be described.

FIG. 234 is a diagram for explaining the autonomous flight device according to the present embodiment.

The autonomous flight device 1921 in the present embodiment is housed inside the surveillance camera 1922. For example, when an image of a suspicious person is captured by the monitoring camera 1922, the door of the monitoring camera 1922 opens, and the autonomous flight device 1921 housed inside takes off the monitoring camera 1922 and tracks the suspicious person. Start. The autonomous flight device 1921 includes a small camera, and performs tracking so that an image of a suspicious person captured by the surveillance camera 1922 can be captured by the small camera. Further, when the autonomous flying device 1921 detects that power for performing a flight or the like is insufficient, the autonomous flying device 1921 returns to the monitoring camera 1922 and is accommodated in the monitoring camera 1922. At this time, if another autonomous flying device 1921 is accommodated in the monitoring camera 1922, the other autonomous flying device 1921 starts tracking the suspicious person in place of the autonomous flying device 1921 having insufficient power. In addition, the autonomous flight device 1921 with insufficient power is supplied with power by the wireless power supply device 1921 a provided in the monitoring camera 1922. Note that the power supply by the wireless power supply apparatus 1921a is performed according to the standard Qi, for example.

The small camera and surveillance camera 1922 of the autonomous flight apparatus 1921 can receive the visible light signal in each of the above embodiments, and can perform an operation according to the received visible light signal. Further, if at least one of the autonomous flight device 1921 and the monitoring camera 1922 is provided with a visible light signal transmitter, visible light communication can be performed between the autonomous flight device 1921 and the monitoring camera 1922. As a result, the suspicious person can be tracked more efficiently.

(Embodiment 23)
In the present embodiment, a display method for realizing AR (Augmented Reality) using an optical ID will be described.

FIG. 235 is a diagram illustrating an example in which the receiver according to the present embodiment displays an AR image.

The receiver 200 according to the present embodiment is a receiver including the image sensor and the display 201 according to any one of the first to second embodiments, and is configured as a smartphone, for example. Such a receiver 200 acquires the above-described captured display image Pa, which is the normal captured image, and the above-described decoding image, which is the visible light communication image or the bright line image, by capturing the subject with the image sensor.

Specifically, the image sensor of the receiver 200 images the transmitter 100 configured as a station name sign. The transmitter 100 is the transmitter according to any one of Embodiments 1 to 22, and includes one or a plurality of light emitting elements (for example, LEDs). The transmitter 100 changes in luminance by blinking one or more light emitting elements, and transmits an optical ID (light identification information) by the change in luminance. This light ID is the above-mentioned visible light signal.

The receiver 200 captures the transmitter 100 with the normal exposure time, thereby acquiring the captured display image Pa projected by the transmitter 100, and the transmitter 100 with a communication exposure time shorter than the normal exposure time. A decoding image is acquired by imaging. The normal exposure time is the exposure time in the above-described normal photographing mode, and the communication exposure time is the exposure time in the above-described visible light communication mode.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P1 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pa as a target area. For example, the receiver 200 recognizes an area in which a station name sign that is the transmitter 100 is displayed as a target area. Then, the receiver 200 superimposes the AR image P1 on the target area, and displays the captured display image Pa on which the AR image P1 is superimposed on the display 201. For example, when “Kyoto Station” is written in Japanese as a station name in the station name that is the transmitter 100, the receiver 200 describes the AR image P1 in which the station name is written in English, that is, “Kyoto Station”. Obtained AR image P1. In this case, since the AR image P1 is superimposed on the target area of the captured display image Pa, the captured display image Pa can be displayed so that a station name with a station name written in English actually exists. As a result, a user who can understand English can easily understand the station name described in the station name sign that is the transmitter 100 by looking at the captured display image Pa even if the user cannot read Japanese.

For example, the recognition information may be an image to be recognized (for example, an image of the above-described station name sign), or may be a feature point and a feature amount of the image. The feature points and feature quantities are obtained by, for example, SIFT (Scale-invariant feature transform), SURF (Speed-Uploaded Robust Feature), ORB (Oriented-BREF), AKAZE (Accelerated) image, etc. Alternatively, the recognition information may be a white square image similar to the image to be recognized, and may further indicate the aspect ratio (aspect ratio) of the square. Alternatively, the identification information may be random dots that appear in the recognition target image. Furthermore, the recognition information may indicate a direction based on a predetermined direction, such as the above-described white square or random dot. The predetermined direction is, for example, the direction of gravity.

The receiver 200 recognizes an area corresponding to such recognition information as a target area from the captured display image Pa. Specifically, if the recognition information is an image, the receiver 200 recognizes a region similar to the image that is the recognition information as a target region. If the recognition information is a feature point and a feature amount obtained by image processing, the receiver 200 performs feature point detection and feature amount extraction by performing the image processing on the captured display image Pa. . Then, the receiver 200 recognizes, in the captured display image Pa, a region having feature points and feature amounts that are similar to the feature points and feature amounts that are recognition information as target regions. If the recognition information indicates a white square and its direction, the receiver 200 first detects the direction of gravity using an acceleration sensor provided in the receiver 200. Then, the receiver 200 recognizes, as a target area, an area similar to a white square directed in the direction indicated by the recognition information from the captured display image Pa arranged with reference to the direction of gravity.

Here, the recognition information may include reference information for specifying a reference area in the captured display image Pa and target information indicating a relative position of the target area with respect to the reference area. The reference information is an image to be recognized, a feature point and a feature amount, a white square image, or a random dot as described above. In this case, when recognizing the target area, the receiver 200 first specifies the reference area from the captured display image Pa based on the reference information. And the receiver 200 recognizes the area | region in the relative position shown by object information among the picked-up display images Pa as a reference | standard by using the position of a reference | standard area | region as a reference | standard. The target information may indicate that the target area is in the same position as the reference area. As described above, since the recognition information includes the reference information and the target information, the target region can be recognized in a wide range. Further, the server can freely set the location where the AR image is superimposed and can be instructed to the receiver 200.

Further, the reference information may indicate that the reference area in the captured display image Pa is an area where the display of the captured display image is displayed. In this case, if the transmitter 100 is configured as a display such as a television, for example, the target area can be recognized with reference to the area where the display is displayed.

In other words, the receiver 200 in the present embodiment specifies the reference image and the image recognition method based on the light ID. The image recognition method is a method for recognizing the captured display image Pa, for example, geometric feature amount extraction, spectral feature amount extraction, texture feature amount extraction, or the like. The reference image is data indicating a reference feature amount. The feature amount is, for example, the feature amount of the white outer frame of the image, and specifically, may be data expressing the feature of the image as a vector. The receiver 200 extracts a feature amount from the captured display image Pa in accordance with an image recognition method, and compares the feature amount with the feature amount of the reference image, so that the above-described reference region or target region is extracted from the captured display image Pa. Find out.

Also, the image recognition method may include, for example, a location use method, a marker use method, and a markerless method. The location utilization method is a method utilizing GPS position information (that is, the position of the receiver 200), and the target area is recognized from the captured display image Pa based on the position information. The marker utilization method is a method of using a marker composed of white and black graphics such as a two-dimensional barcode as a target specifying mark. That is, in this marker usage method, the target region is recognized based on the marker displayed in the captured display image Pa. In the markerless method, a feature point or feature amount is extracted from the captured display image Pa by image analysis on the captured display image Pa, and the position and region of the target are specified based on the extracted feature point or feature amount. Is the method. That is, when the image recognition method is a markerless method, the image recognition method is the above-described geometric feature amount extraction, spectral feature amount extraction, texture feature amount extraction, or the like.

Such a receiver 200 receives an optical ID from the transmitter 100 and acquires a reference image and an image recognition method associated with the optical ID (hereinafter referred to as a received optical ID) from the server, thereby obtaining the reference. Images and image recognition methods may be specified. That is, the server stores a plurality of sets including the reference image and the image recognition method, and each of the plurality of sets is associated with a different light ID. Thus, one set associated with the received light ID can be identified from among a plurality of sets stored in the server. Therefore, the speed of image processing for superimposing the AR image can be improved. Further, the receiver 200 may acquire a reference image associated with the received light ID by inquiring of the server, and the received light ID may be obtained from a plurality of reference images held by the receiver 200 in advance. You may acquire the reference | standard image matched with.

Further, the server may hold the relative position information associated with each light ID together with the reference image, the image recognition method, and the AR image for each light ID. The relative position information is information indicating the relative positional relationship between the reference area and the target area, for example. Thereby, when the receiver 200 sends an inquiry to the server by receiving the received light ID, the receiver 200 acquires the reference image, the image recognition method, the AR image, and the relative position information associated with the received light ID. In this case, the receiver 200 specifies the above-described reference region from the captured display image Pa based on the reference image and the image recognition method. Then, the receiver 200 recognizes the region in the direction and the distance indicated by the relative position information from the position of the reference region as the target region, and superimposes the AR image on the target region. Further, if there is no relative position information, the receiver 200 may recognize the above-described reference area as a target area and superimpose an AR image on the reference area. That is, the receiver 200 may store a program for displaying an AR image based on the reference image in advance, instead of acquiring the relative position information, and display the AR image in a white frame that is a reference region, for example. . In this case, relative position information is not necessary.

There are the following four variations (1) to (4) for holding or acquiring the reference image, relative position information, AR image, and image recognition method.

(1) The server holds a plurality of sets including a reference image, relative position information, an AR image, and an image recognition method. The receiver 200 acquires one set associated with the received light ID from these sets.

(2) The server holds a plurality of sets including a reference image and an AR image. The receiver 200 uses a predetermined relative position information and an image recognition method, and acquires one set associated with the received light ID from the set. Alternatively, the receiver 200 may hold a plurality of sets including the relative position information and the image recognition method in advance, and select one set associated with the received light ID from the plurality of sets. In this case, the receiver 200 may inquire by transmitting the received light ID to the server, and acquire relative position information corresponding to the received light ID and information for specifying the image recognition method from the server. Then, the receiver 200 selects one set based on the information acquired from the server from among a plurality of sets each having the relative position information and the image recognition method. Alternatively, the receiver 200 selects one set associated with the received light ID from a plurality of sets each including the relative position information and the image recognition method stored in advance without inquiring of the server. May be.

(3) The receiver 200 holds a plurality of sets including a reference image, relative position information, an AR image, and an image recognition method, and selects one set from these sets. Similarly to (2) above, the receiver 200 may select one set by making an inquiry to the server, or may select one set associated with the receiver optical ID.

(4) The receiver 200 holds a plurality of sets including the reference image and the AR image, and selects one set associated with the received light ID. Receiver 200. A predetermined image recognition method and relative position information are used.

FIG. 236 is a diagram illustrating an example of a display system in the present embodiment.

The display system in the present embodiment includes, for example, the transmitter 100, the receiver 200, and the server 300, which are the above-described station names.

The receiver 200 first receives an optical ID from the transmitter 100 in order to display the captured display image on which the AR image is superimposed as described above. Next, the receiver 200 transmits the optical ID to the server 300.

The server 300 holds an AR image and recognition information associated with each light ID for each light ID. Therefore, when the server 300 receives the optical ID from the receiver 200, the server 300 selects the AR image and the recognition information associated with the received optical ID, and sends the selected AR image and the recognition information to the receiver 200. Send. Thereby, the receiver 200 receives the AR image and the recognition information transmitted from the server 300, and displays the captured display image on which the AR image is superimposed.

FIG. 237 is a diagram showing another example of the display system in the present embodiment.

The display system according to the present embodiment includes, for example, the transmitter 100 that is the above-described station name sign, the receiver 200, the first server 301, and the second server 302.

The receiver 200 first receives an optical ID from the transmitter 100 in order to display the captured display image on which the AR image is superimposed as described above. Next, the receiver 200 transmits the optical ID to the first server 301.

When receiving the optical ID from the receiver 200, the first server 301 notifies the receiver 200 of a URL (Uniform Resource Locator) and Key associated with the received optical ID. Receiving such notification, the receiver 200 accesses the second server 302 based on the URL, and passes the key to the second server 302.

The second server 302 holds an AR image and recognition information associated with each key for each key. Therefore, when receiving a key from the receiver 200, the second server 302 selects an AR image and recognition information associated with the key, and transmits the selected AR image and recognition information to the receiver 200. . Thereby, the receiver 200 receives the AR image and the recognition information transmitted from the second server 302, and displays a captured display image on which the AR image is superimposed.

FIG. 238 is a diagram showing another example of the display system in the present embodiment.

The display system according to the present embodiment includes, for example, the transmitter 100 that is the above-described station name sign, the receiver 200, the first server 301, and the second server 302.

The receiver 200 first receives an optical ID from the transmitter 100 in order to display the captured display image on which the AR image is superimposed as described above. Next, the receiver 200 transmits the optical ID to the first server 301.

When receiving the optical ID from the receiver 200, the first server 301 notifies the second server 302 of the Key associated with the received optical ID.

The second server 302 holds an AR image and recognition information associated with each key for each key. Therefore, when the second server 302 receives the key from the first server 301, the second server 302 selects the AR image and the recognition information associated with the key, and the selected AR image and the recognition information are used as the first server. To the server 301. When receiving the AR image and the recognition information from the second server 302, the first server 301 transmits the AR image and the recognition information to the receiver 200. Thereby, the receiver 200 receives the AR image and the recognition information transmitted from the first server 301, and displays the captured display image on which the AR image is superimposed.

In the above-described example, the second server 302 transmits the AR image and the recognition information to the first server 301. However, the second server 302 may transmit the AR image and the recognition information to the receiver 200 without transmitting to the first server 301. .

FIG. 239 is a flowchart illustrating an example of processing operations of the receiver 200 in the present embodiment.

First, the receiver 200 starts imaging with the above-described normal exposure time and communication exposure time (step S101). Then, the receiver 200 acquires an optical ID by decoding the decoding image obtained by imaging with the communication exposure time (step S102). Next, the receiver 200 transmits the optical ID to the server (step S103).

The receiver 200 acquires the AR image corresponding to the transmitted optical ID and the recognition information from the server (step S104). Next, the receiver 200 recognizes, as a target area, an area corresponding to the recognition information in the captured display image obtained by imaging with the normal exposure time (step S105). Then, the receiver 200 superimposes the AR image on the target area, and displays the captured display image on which the AR image is superimposed (step S106).

Next, the receiver 200 determines whether or not the imaging and the display of the captured display image should be terminated (step S107). Here, when the receiver 200 determines that it should not be terminated (N in step S107), it further determines whether or not the acceleration of the receiver 200 is equal to or greater than a threshold value (step S108). This acceleration is measured by an acceleration sensor provided in the receiver 200. When the receiver 200 determines that the acceleration is less than the threshold value (N in step S108), the receiver 200 executes the processing from step S105. Thereby, even when the captured display image displayed on the display 201 of the receiver 200 is shifted, the AR image can follow the target area of the captured display image. If the receiver 200 determines that the acceleration is equal to or greater than the threshold (Y in step S108), the receiver 200 executes the processing from step S102. Thereby, when the transmitter 100 is no longer displayed in the captured display image, it is possible to suppress erroneously recognizing an area where a subject different from the transmitter 100 is displayed as the target area.

Thus, in the present embodiment, since the AR image is displayed superimposed on the captured display image, an image useful for the user can be displayed. Furthermore, it is possible to superimpose an AR image on an appropriate target area while suppressing the processing load.

That is, in general augmented reality (that is, AR), a huge number of recognition target images stored in advance are compared with the captured display image, and any of the recognition target images is included in the captured display image. It is determined whether or not If it is determined that the recognition target image is included, the AR image corresponding to the recognition target image is superimposed on the captured display image. At this time, the AR image is aligned based on the recognition target image. In this way, in general augmented reality, a huge number of recognition target images are compared with captured display images, and further, position detection of the recognition target images in the captured display image is necessary even in alignment. There is a problem that the calculation amount is large and the processing load is high.

However, in the display method according to the present embodiment, the light ID is acquired by decoding the decoding image obtained by imaging the subject. That is, the optical ID transmitted from the transmitter that is the subject is received. Furthermore, the AR image corresponding to this optical ID and the recognition information are acquired from the server. Therefore, the server does not need to compare a huge number of recognition target images and captured display images, and can select and transmit an AR image previously associated with the optical ID to the display device. Thereby, the amount of calculation can be reduced and the processing load can be significantly suppressed. Furthermore, the AR image display process can be speeded up.

In this embodiment, recognition information corresponding to this optical ID is acquired from the server. The recognition information is information for recognizing a target area that is an area in which an AR image is superimposed in a captured display image. This recognition information may be information indicating that a white square is the target area, for example. In this case, the target area can be easily recognized, and the processing load can be further suppressed. That is, the processing load can be further suppressed according to the content of the recognition information. Further, since the server can arbitrarily set the content of the recognition information according to the optical ID, the balance between the processing load and the recognition accuracy can be appropriately maintained.

In this embodiment, after the receiver 200 transmits the optical ID to the server, the receiver 200 acquires the AR image and the recognition information corresponding to the optical ID from the server. Of the AR image and the recognition information, At least one of these may be acquired in advance. That is, the receiver 200 collects a plurality of AR images and a plurality of recognition information corresponding to a plurality of optical IDs that may be received from the server and stores them. Thereafter, when receiving the optical ID, the receiver 200 selects an AR image and recognition information corresponding to the optical ID from a plurality of AR images and a plurality of recognition information stored in the receiver 200. Thereby, the display processing of the AR image can be further accelerated.

FIG. 240 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

For example, as shown in FIG. 240, the transmitter 100 is configured as an illumination device, and transmits a light ID by changing the luminance while illuminating the facility guide plate 101. Since the guide plate 101 is illuminated by the light from the transmitter 100, the luminance changes in the same manner as the transmitter 100, and the light ID is transmitted.

The receiver 200 acquires the captured display image Pb and the decoding image in the same manner as described above by imaging the guide plate 101 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the guide plate 101. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P2 and the recognition information corresponding to the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pb as a target area. For example, the receiver 200 recognizes the area where the frame 102 on the guide plate 101 is projected as the target area. This frame 102 is a frame for indicating the waiting time of the facility. Then, the receiver 200 superimposes the AR image P2 on the target area, and displays the captured display image Pb on which the AR image P2 is superimposed on the display 201. For example, the AR image P2 is an image including the character string “30 minutes”. In this case, since the AR image P2 is superimposed on the target area of the captured display image Pb, the receiver 200 captures the captured display image so that the guide plate 101 on which the waiting time “30 minutes” is described actually exists. Pb can be displayed. Thereby, it is possible to inform the user of the receiver 200 of the waiting time easily and easily without providing a special display device on the guide plate 101.

FIG. 241 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

The transmitter 100 includes two illumination devices as shown in FIG. The transmitter 100 transmits the light ID by changing the luminance while illuminating the facility guide plate 104. Since the guide plate 104 is illuminated by the light from the transmitter 100, the luminance changes in the same manner as the transmitter 100, and the light ID is transmitted. The guide plate 104 indicates names of a plurality of facilities such as “ABC land” and “adventure land”.

The receiver 200 acquires the captured display image Pc and the decoding image in the same manner as described above by imaging the guide plate 104 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the guide plate 104. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P3 and the recognition information corresponding to the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pc as a target area. For example, the receiver 200 recognizes an area where the guide plate 104 is projected as a target area. Then, the receiver 200 superimposes the AR image P3 on the target area, and displays the captured display image Pc on which the AR image P3 is superimposed on the display 201. For example, the AR image P3 is an image indicating names of a plurality of facilities. In this AR image P3, the longer the waiting time of the facility, the smaller the name of the facility is displayed. Conversely, the shorter the waiting time of the facility, the larger the name of the facility is displayed. In this case, since the AR image P3 is superimposed on the target area of the captured display image Pc, the receiver 200 seems to actually have a guide plate 104 on which each facility name having a size corresponding to the waiting time is described. The captured display image Pc can be displayed. Thereby, without providing a special display device on the guide plate 104, the user of the receiver 200 can be informed easily and easily of the waiting time of each facility.

FIG. 242 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

The transmitter 100 includes two illumination devices as shown in FIG. 242, for example. The transmitter 100 transmits the light ID by changing the luminance while illuminating the castle wall 105. Since the castle wall 105 is illuminated by the light from the transmitter 100, the luminance is changed in the same manner as the transmitter 100, and the light ID is transmitted. Further, on the castle wall 105, for example, a small mark imitating the character's face is engraved as a hidden character 106.

The receiver 200 acquires the captured display image Pd and the decoding image in the same manner as described above by capturing an image of the castle wall 105 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the castle wall 105. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P4 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pd as a target area. For example, the receiver 200 recognizes, as the target area, an area in which a range including the hidden character 106 in the castle wall 105 is projected. Then, the receiver 200 superimposes the AR image P4 on the target area, and displays the captured display image Pd on which the AR image P4 is superimposed on the display 201. For example, the AR image P4 is an image imitating a character's face. The AR image P4 is an image that is sufficiently larger than the hidden character 106 displayed in the captured display image Pd. In this case, since the AR image P4 is superimposed on the target area of the captured display image Pd, the receiver 200 captures the image so that the castle wall 105 engraved with a large mark imitating the character's face actually exists. The display image Pd can be displayed. Thereby, it is possible to inform the user of the receiver 200 of the position of the hidden character 106 in an easy-to-understand manner.

FIG. 243 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

The transmitter 100 includes two illumination devices as shown in FIG. 243, for example. The transmitter 100 transmits the light ID by changing the luminance while illuminating the facility guide plate 107. Since the guide plate 107 is illuminated by the light from the transmitter 100, the luminance changes in the same manner as the transmitter 100, and the light ID is transmitted. In addition, an infrared shielding paint 108 is applied to a plurality of corners of the guide plate 107.

The receiver 200 acquires the captured display image Pe and the decoding image in the same manner as described above by imaging the guide plate 107 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the guide plate 107. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P5 and the recognition information corresponding to the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pe as a target area. For example, the receiver 200 recognizes an area where the guide plate 107 is projected as a target area.

Specifically, the recognition information indicates that a rectangle circumscribing the plurality of infrared shielding paints 108 is the target region. The infrared blocking paint 108 blocks infrared rays included in the light emitted from the transmitter 100. Therefore, the image sensor of the receiver 200 recognizes the infrared shielding paint 108 as an image darker than the surrounding area. The receiver 200 recognizes rectangles circumscribing the plurality of infrared shielding paints 108 that appear as dark images as target regions.

Then, the receiver 200 superimposes the AR image P5 on the target area, and displays the captured display image Pe on which the AR image P5 is superimposed on the display 201. For example, the AR image P5 shows a schedule of events to be performed in the facility of the guide board 107. In this case, since the AR image P5 is superimposed on the target area of the captured display image Pe, the receiver 200 displays the captured display image Pe so that the guide plate 107 on which the event schedule is described actually exists. can do. Thereby, without providing a special display device on the guide plate 107, it is possible to inform the user of the receiver 200 of the schedule of the facility event in an easy-to-understand manner.

Note that an infrared reflecting paint may be applied to the guide plate 107 instead of the infrared shielding paint 108. The infrared reflecting paint reflects infrared rays included in the light irradiated from the transmitter 100. Therefore, the image sensor of the receiver 200 recognizes the infrared reflective paint as an image brighter than the surrounding area. That is, in this case, the receiver 200 recognizes rectangles circumscribing a plurality of infrared reflective paints that appear as bright images as target regions.

FIG. 244 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

The transmitter 100 is configured as a station name sign and is disposed near the station exit guide plate 110. The station exit guide plate 110 includes a light source and emits light, but, unlike the transmitter 100, does not transmit an optical ID.

When the receiver 200 images the transmitter 100 and the station exit guide plate 110, the captured display image Ppre and the decoding image Pdec are acquired. Since the transmitter 100 changes in luminance and the station exit guide plate 110 emits light, the decoding image Pdec includes a bright line pattern region Pdec1 corresponding to the transmitter 100 and a bright region corresponding to the station exit guide plate 110. Pdec2 appears. The bright line pattern region Pdec1 is a region composed of a plurality of bright line patterns that appear by exposure at a communication exposure time of a plurality of exposure lines of the image sensor of the receiver 200.

Here, as described above, the identification information includes reference information for specifying the reference region Pbas in the captured display image Ppre and target information indicating the relative position of the target region Ptar with respect to the reference region Pbas. Yes. For example, the reference information indicates that the position of the reference area Pbas in the captured display image Ppre is the same as the position of the bright line pattern area Pdec1 in the decoding image Pdec. Further, the target information indicates that the position of the target area is the position of the reference area.

Therefore, the receiver 200 specifies the reference region Pbas from the captured display image Ppre based on the reference information. That is, the receiver 200 specifies, in the captured display image Ppre, an area that is at the same position as the bright line pattern area Pdec1 in the decoding image Pdec as the reference area Pbas. Further, the receiver 200 recognizes, as the target region Ptar, a region in the relative position indicated by the target information with reference to the position of the reference region Pbas in the captured display image Ppre. In the above example, since the target information indicates that the position of the target area Ptar is the position of the reference area Pbas, the receiver 200 recognizes the reference area Pbas in the captured display image Ppre as the target area Ptar.

Then, the receiver 200 superimposes the AR image P1 on the target area Ptar in the captured display image Ppre.

Thus, in the above-described example, the bright line pattern region Pdec1 is used to recognize the target region Ptar. On the other hand, when the region where the transmitter 100 is projected is to be recognized as the target region Ptar from only the captured display image Ppre without using the bright line pattern region Pdec1, erroneous recognition may occur. . That is, in the captured display image Ppre, not the area where the transmitter 100 is projected but the area where the station exit guide plate 110 is projected may be erroneously recognized as the target area Ptar. This is because the image of the transmitter 100 and the image of the station exit guide plate 110 in the captured display image Ppre are similar. However, as in the above example, when the bright line pattern region Pdec1 is used, it is possible to accurately recognize the target region Ptar while suppressing the occurrence of erroneous recognition.

FIG. 245 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

In the example shown in FIG. 244, the transmitter 100 transmits the light ID by changing the luminance of the entire station name sign, and the target information indicates that the position of the target area is the position of the reference area. However, in this embodiment, the transmitter 100 transmits the light ID by changing the luminance of the light emitting elements arranged in a part of the outer frame of the station name sign without changing the brightness of the entire station name sign. Also good. The target information only needs to indicate the relative position of the target area Ptar with respect to the reference area Pbas. For example, the position of the target area Ptar is above the reference area Pbas (specifically, vertically upward). May be shown.

In the example shown in FIG. 245, the transmitter 100 transmits the light ID by changing the luminance of a plurality of light emitting elements arranged along the horizontal direction below the outer frame of the station name sign. The target information indicates that the position of the target area Ptar is above the reference area Pbas.

In such a case, the receiver 200 specifies the reference region Pbas from the captured display image Ppre based on the reference information. That is, the receiver 200 specifies, in the captured display image Ppre, an area that is at the same position as the bright line pattern area Pdec1 in the decoding image Pdec as the reference area Pbas. Specifically, the receiver 200 specifies a rectangular reference region Pbas that is long in the horizontal direction and short in the vertical direction. Further, the receiver 200 recognizes, as the target region Ptar, a region in the relative position indicated by the target information with reference to the position of the reference region Pbas in the captured display image Ppre. That is, the receiver 200 recognizes an area above the reference area Pbas in the captured display image Ppre as the target area Ptar. At this time, the receiver 200 specifies the direction above the reference region Pbas based on the direction of gravity measured by the acceleration sensor provided in the receiver 200.

Note that the target information may indicate not only the relative position of the target area Ptar but also the size, shape, and aspect ratio of the target area Ptar. In this case, the receiver 200 recognizes the target area Ptar having the size, shape, and aspect ratio indicated by the target information. In addition, the receiver 200 may determine the size of the target area Ptar based on the size of the reference area Pbas.

FIG. 246 is a flowchart illustrating another example of the processing operation of the receiver 200 in the present embodiment.

The receiver 200 executes the processing of steps S101 to S104, similarly to the example shown in FIG.

Next, the receiver 200 identifies the bright line pattern region Pdec1 from the decoding image Pdec (step S111). Next, the receiver 200 specifies a reference area Pbas corresponding to the bright line pattern area Pdec1 from the captured display image Ppre (step S112). The receiver 200 recognizes the target area Ptar from the captured display image Ppre based on the recognition information (specifically, target information) and the reference area Pbas (step S113).

Next, similarly to the example shown in FIG. 239, the receiver 200 superimposes the AR image on the target area Ptar of the captured display image Ppre, and displays the captured display image Ppre on which the AR image is superimposed (step S106). . Then, the receiver 200 determines whether or not the imaging and the display of the captured display image Pre are to be ended (Step S107). Here, if the receiver 200 determines that it should not be ended (N in step S107), it further determines whether or not the acceleration of the receiver 200 is equal to or greater than a threshold value (step S114). This acceleration is measured by an acceleration sensor provided in the receiver 200. When the receiver 200 determines that the acceleration is less than the threshold value (N in step S114), the receiver 200 executes the processing from step S113. Thereby, even when the captured display image Ppre displayed on the display 201 of the receiver 200 is shifted, the AR image can follow the target area Ptar of the captured display image Ppre. If the receiver 200 determines that the acceleration is equal to or greater than the threshold (Y in step S114), the receiver 200 executes the processing from step S111 or step S102. Thereby, it can suppress that the area | region where the to-be-photographed object (for example, station exit guide board 110) different from the transmitter 100 is reflected is mistakenly recognized as the object area Ptar.

FIG. 247 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

When the AR image P1 in the displayed captured display image Ppre is tapped, the receiver 200 enlarges and displays the AR image P1. Alternatively, when tapped, the receiver 200 may display a new AR image showing more detailed content than that shown in the AR image P1 instead of the AR image P1. In addition, when the AR image P1 indicates information for one page of an information magazine including a plurality of pages, the receiver 200 displays a new AR image indicating information on the next page of the page of the AR image P1. You may display instead of AR image P1. Alternatively, when tapped, the receiver 200 may display a moving image related to the AR image P1 as a new AR image instead of the AR image P1. At this time, the receiver 200 may display a moving image in which an object (autumn leaves in the example of FIG. 247) appears from the target area Ptar as an AR image.

FIG. 248 is a diagram illustrating a captured display image Ppre and a decoding image Pdec acquired by imaging of the receiver 200 in the present embodiment.

For example, as illustrated in FIG. 248 (a1), the receiver 200 acquires captured images such as a captured display image Ppre and a decoding image Pdec at a frame rate of 30 fps. Specifically, the receiver 200 acquires the captured display image Ppre “A” at time t1, acquires the decoding image Pdec at time t2, and acquires the captured display image Ppre “B” at time t3. The captured display image Ppre and the decoding image Pdec are obtained alternately.

In addition, when displaying the captured image, the receiver 200 displays only the captured display image Ppre among the captured images, and does not display the decoding image Pdec. That is, as shown in (a2) of FIG. 248, the receiver 200 displays the captured display image Ppre acquired immediately before, instead of the decoding image Pdec, when acquiring the decoding image Pdec. Specifically, the receiver 200 displays the acquired captured display image Ppre “A” at time t1, and again displays the captured display image Ppre “A” acquired at time t1 at time t2. To do. Thereby, the receiver 200 displays the captured display image Ppre at a frame rate of 15 fps.

Here, in the example illustrated in (a1) of FIG. 248, the receiver 200 alternately acquires the captured display image Ppre and the decoding image Pdec. However, the acquisition form of these images in the present embodiment is as follows. It is not restricted to such a form. That is, the receiver 200 continuously obtains N (N is an integer equal to or greater than 1) decoding images Pdec, and then continuously captures M (M is an integer equal to or greater than 1) captured display images Ppre. You may repeat acquiring.

In addition, the receiver 200 needs to switch the acquired captured image to the captured display image Ppre and the decoding image Pdec, and this switching may take time. Therefore, as illustrated in (b1) of FIG. 248, the receiver 200 may provide a switching period when switching between acquisition of the captured display image Ppre and acquisition of the decoding image Pdec. Specifically, when the receiver 200 obtains the decoding image Pdec at time t3, the receiver 200 executes processing for switching the captured image during the switching period from time t3 to t5, and at time t5, the captured display image Prep " A ”is acquired. Thereafter, the receiver 200 executes processing for switching the captured image in the switching period from time t5 to time t7, and acquires the decoding image Pdec at time t7.

When the switching period is provided in this way, the receiver 200 displays the captured display image Ppre acquired immediately before in the switching period, as shown in (b2) of FIG. 248. Therefore, in this case, the display frame rate of the captured display image Ppre in the receiver 200 is low, for example, 3 fps. Thus, when the frame rate is low, even if the user moves the receiver 200, the captured display image Ppre displayed may not move according to the movement of the receiver 200. That is, the captured display image Ppre is not displayed as a live view. Therefore, the receiver 200 may move the captured display image Ppre according to the movement of the receiver 200.

FIG. 249 is a diagram illustrating an example of a captured display image Pre displayed on the receiver 200 according to the present embodiment.

The receiver 200 displays a captured display image Ppre obtained by imaging on the display 201 as shown in FIG. Here, the user moves the receiver 200 to the left side. At this time, when a new captured display image Ppre is not acquired by imaging by the receiver 200, the receiver 200 moves the displayed captured display image Ppre to the right as illustrated in FIG. That is, the receiver 200 includes an acceleration sensor, and moves the displayed captured display image Ppre to match the movement of the receiver 200 according to the acceleration measured by the acceleration sensor. Thereby, the receiver 200 can display the captured display image Ppre as a live view in a pseudo manner.

FIG. 250 is a flowchart illustrating another example of the processing operation of the receiver 200 in the present embodiment.

First, similarly to the above, the receiver 200 superimposes the AR image on the target area Ptar of the captured display image Ppre and follows the target area Ptar (step S122). That is, an AR image that moves together with the target area Ptar in the captured display image Ppre is displayed. Then, the receiver 200 determines whether or not to maintain the display of the AR image (step S122). If it is determined that the display of the AR image is not maintained (N in step S122), if the receiver 200 acquires a new light ID by imaging, the new AR image corresponding to the light ID is captured and displayed. It is displayed superimposed on Pre (step S123).

On the other hand, if it is determined that the display of the AR image is to be maintained (Y in step S122), the receiver 200 repeatedly executes the processing from step S121. At this time, the receiver 200 does not display another AR image even if another AR image is acquired. Alternatively, even when the receiver 200 has acquired a new decoding image Pdec, the receiver 200 does not acquire an optical ID by decoding the decoding image Pdec. At this time, power consumption for decoding can be suppressed.

As described above, by maintaining the display of the AR image, it is possible to prevent the displayed AR image from being erased or being difficult to see due to the display of another AR image. That is, the displayed AR image can be easily seen by the user.

For example, in step S122, the receiver 200 determines to maintain the display of the AR image until a predetermined period (a certain period) elapses after the AR image is displayed. That is, when displaying the captured display image Ppre, the receiver 200 is determined in advance while suppressing the display of the second AR image different from the first AR image that is the AR image superimposed in step S121. The first AR image is displayed only during the display period. The receiver 200 may prohibit the decoding of the newly acquired decoding image Pdec during this display period.

Thereby, when the user is viewing the first AR image once displayed, it is possible to prevent the first AR image from being immediately replaced with a different second AR image. Furthermore, since decoding of the newly acquired decoding image Pdec is a wasteful process when the display of the second AR image is suppressed, power consumption can be suppressed by prohibiting the decoding. .

Alternatively, in step S122, the receiver 200 may include a face camera, and when detecting that the user's face is approaching based on the imaging result of the face camera, the receiver 200 may determine to maintain the display of the AR image. . That is, when displaying the captured display image Ppre, the receiver 200 further determines whether or not the user's face is approaching the receiver 200 by imaging with a face camera provided in the receiver 200. When the receiver 200 determines that the face is approaching, the first AR image is suppressed while suppressing the display of the second AR image that is different from the first AR image that is the AR image superimposed in step S121. An AR image is displayed.

Alternatively, in step S122, the receiver 200 may include an acceleration sensor, and when detecting that the user's face is approaching based on the measurement result of the acceleration sensor, the receiver 200 may determine to maintain the display of the AR image. . That is, when displaying the captured display image Ppre, the receiver 200 further determines whether or not the user's face is approaching the receiver 200 based on the acceleration of the receiver 200 measured by the acceleration sensor. For example, when the acceleration of the receiver 200 measured by the acceleration sensor shows a positive value in a direction perpendicular to the display 201 of the receiver 200, the receiver 200 is approaching the user's face. judge. When the receiver 200 determines that the face is approaching, the first 200 is performed while suppressing the display of the second AR image that is different from the first augmented reality image that is the AR image superimposed in step S121. The AR image is displayed.

Accordingly, when the user is approaching the receiver 200 to see the first AR image, the first AR image can be prevented from being replaced with a different second AR image. .

Alternatively, in step S122, the receiver 200 may determine that the display of the AR image is maintained when a lock button provided in the receiver 200 is pressed.

In step S122, the receiver 200 determines that the display of the AR image is not maintained when the above-described certain period (that is, the display period) has elapsed. Further, the receiver 200 determines that the display of the AR image is not maintained when acceleration equal to or greater than the threshold value is measured by the acceleration sensor even when the above-described certain period has not elapsed. That is, when displaying the captured display image Ppre, the receiver 200 further measures the acceleration of the receiver 200 with the acceleration sensor during the display period described above, and determines whether or not the measured acceleration is equal to or greater than a threshold value. When the receiver 200 determines that the second AR image is greater than or equal to the threshold, the receiver 200 cancels the suppression of the display of the second AR image, thereby displaying the second AR image instead of the first AR image in step S123.

Thereby, when the acceleration of the display device equal to or greater than the threshold is measured, the suppression of the display of the second AR image is released. Therefore, for example, when the user moves the receiver 200 greatly so as to point the image sensor at another subject, the second AR image can be displayed immediately.

FIG. 251 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

For example, as shown in FIG. 251, the transmitter 100 is configured as an illumination device, and transmits a light ID by changing the luminance while illuminating a stage 111 for a small doll. Since the stage 111 is illuminated by the light from the transmitter 100, the luminance changes similarly to the transmitter 100, and the optical ID is transmitted.

The two receivers 200 image the stage 111 illuminated by the transmitter 100 from the left and right.

The left receiver 200 of the two receivers 200 captures the captured display image Pf and the decoding image in the same manner as described above by capturing the stage 111 illuminated by the transmitter 100 from the left. The receiver 200 on the left side acquires the optical ID by decoding the decoding image. That is, the left receiver 200 receives the optical ID from the stage 111. The left receiver 200 transmits the optical ID to the server. Then, the left-side receiver 200 acquires a three-dimensional AR image and recognition information corresponding to the optical ID from the server. This three-dimensional AR image is an image for displaying a doll three-dimensionally, for example. The left receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pf as a target area. For example, the left receiver 200 recognizes the upper area at the center of the stage 111 as the target area.

Next, the left-side receiver 200 generates a two-dimensional AR image P6a corresponding to the orientation from the three-dimensional AR image based on the orientation of the stage 111 displayed in the captured display image Pf. Then, the receiver 200 on the left side superimposes the two-dimensional AR image P6a on the target area, and displays the captured display image Pf on which the AR image P6a is superimposed on the display 201. In this case, since the two-dimensional AR image P6a is superimposed on the target area of the captured display image Pf, the left receiver 200 displays the captured display image Pf so that the doll actually exists on the stage 111. can do.

Similarly, the right receiver 200 of the two receivers 200 captures the captured display image Pg and the decoding image in the same manner as described above by capturing the stage 111 illuminated by the transmitter 100 from the right. get. The right receiver 200 acquires the optical ID by decoding the decoding image. That is, the right receiver 200 receives the optical ID from the stage 111. The right receiver 200 transmits the optical ID to the server. Then, the right-side receiver 200 acquires a three-dimensional AR image and recognition information corresponding to the optical ID from the server. The right receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pg as a target area. For example, the right receiver 200 recognizes the upper area at the center of the stage 111 as the target area.

Next, the right-side receiver 200 generates a two-dimensional AR image P6b corresponding to the orientation from the three-dimensional AR image based on the orientation of the stage 111 displayed in the captured display image Pg. Then, the receiver 200 on the right side superimposes the two-dimensional AR image P6b on the target region, and displays the captured display image Pg on which the AR image P6b is superimposed on the display 201. In this case, since the two-dimensional AR image P6b is superimposed on the target region of the captured display image Pg, the right-side receiver 200 displays the captured display image Pg so that the doll actually exists on the stage 111. can do.

Thus, the two receivers 200 display the AR images P6a and P6b at the same position on the stage 111. The AR images P6a and P6b are generated according to the orientation of the receiver 200 so that the virtual doll is actually facing a predetermined direction. Therefore, the captured display image can be displayed so that the doll actually exists on the stage 111 no matter what direction the stage 111 is captured.

In the above example, the receiver 200 generates a two-dimensional AR image corresponding to the positional relationship between the receiver 200 and the stage 111 from the three-dimensional AR image. May be obtained from the server. That is, the receiver 200 transmits information indicating the positional relationship together with the optical ID to the server, and acquires the two-dimensional AR image from the server instead of the three-dimensional AR image. Thereby, the burden on the receiver 200 can be reduced.

FIG. 252 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

For example, as illustrated in FIG. 252, the transmitter 100 is configured as a lighting device, and transmits a light ID by changing luminance while illuminating a cylindrical structure 112. Since the structure 112 is illuminated by the light from the transmitter 100, the luminance is changed similarly to the transmitter 100, and the light ID is transmitted.

The receiver 200 acquires the captured display image Ph and the decoding image in the same manner as described above by imaging the structure 112 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the structure 112. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P7 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Ph as a target area. For example, the receiver 200 recognizes an area where the central portion of the structure 112 is projected as a target area. Then, the receiver 200 superimposes the AR image P7 on the target area, and displays the captured display image Ph on which the AR image P7 is superimposed on the display 201. For example, the AR image P <b> 7 is an image including a character string “ABCD”, and the character string is distorted in accordance with the curved surface at the center of the structure 112. In this case, since the AR image P2 including the distorted character string is superimposed on the target region of the captured display image Ph, the receiver 200 makes sure that the character string drawn on the structure 112 actually exists. The captured display image Ph can be displayed.

FIG. 253 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

For example, as shown in FIG. 253, the transmitter 100 transmits the light ID by changing the luminance while illuminating the restaurant menu 113. Since the menu 113 is illuminated by the light from the transmitter 100, the luminance changes in the same manner as the transmitter 100, and the light ID is transmitted. The menu 113 indicates names of a plurality of dishes such as “ABC soup”, “XYZ salad”, and “KLM lunch”.

The receiver 200 acquires the captured display image Pi and the decoding image in the same manner as described above by capturing an image of the menu 113 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the menu 113. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P8 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pi as a target area. For example, the receiver 200 recognizes an area where the menu 113 is displayed as the target area. Then, the receiver 200 superimposes the AR image P8 on the target region, and displays the captured display image Pi on which the AR image P8 is superimposed on the display 201. For example, the AR image P8 is an image that shows the ingredients used for each of a plurality of dishes by marks. For example, the AR image P8 shows a mark imitating an egg for a dish “XYZ salad” using eggs, and a pig for a dish “KLM lunch” using pork. The imitated mark is shown. In this case, since the AR image P8 is superimposed on the target area of the captured display image Pi, the receiver 200 displays the captured display image Pi so that the menu 113 with the food mark is actually present. be able to. Thereby, without providing a special display device in the menu 113, the user of the receiver 200 can be easily and easily informed of the ingredients of each dish.

In addition, the receiver 200 acquires a plurality of AR images, selects an AR image suitable for the user from the plurality of AR images based on the user information set by the user, and superimposes the AR images. May be. For example, if the user information indicates that the user shows an allergic reaction to the egg, the receiver 200 selects an AR image that is marked with an egg for a dish in which the egg is used. If the user information indicates that the intake of pork is prohibited, the receiver 200 selects an AR image in which a pork mark is attached to a dish in which pork is used. Alternatively, the receiver 200 may transmit the user information together with the optical ID to the server and acquire an AR image corresponding to the optical ID and the user information from the server. Thereby, for each user, a menu that prompts the user to call can be displayed.

FIG. 254 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

For example, as shown in FIG. 254, the transmitter 100 is configured as a television, and transmits an optical ID by changing the luminance while displaying an image on a display. In addition, a normal television 114 is disposed in the vicinity of the transmitter 100. The television 114 displays an image on the display, but does not transmit an optical ID.

The receiver 200 acquires the captured display image Pj and the decoding image in the same manner as described above, for example, by imaging the television 114 together with the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P9 and the recognition information corresponding to the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pj as a target area.

For example, the receiver 200 uses the bright line pattern area of the image for decoding, so that the lower part of the area where the transmitter 100 transmitting the optical ID is displayed in the captured display image Pj is the first target area. Recognize as At this time, the reference information included in the recognition information indicates that the position of the reference area in the captured display image Pj is the same as the position of the bright line pattern area in the decoding image. Furthermore, the target information included in the recognition information indicates that there is a target area below the reference area. The receiver 200 recognizes the first target area described above using such recognition information.

Furthermore, the receiver 200 recognizes an area whose position is fixed in advance below the captured display image Pj as the second target area. The second target area is larger than the first target area. The target information included in the recognition information further indicates not only the position of the first target area but also the position and size of the second target area as described above. The receiver 200 recognizes the second target area described above using such recognition information.

Then, the receiver 200 superimposes the AR image P9 on the first target area and the second target area, and displays the captured display image Pj on which the AR image P8 is superimposed on the display 201. In the superposition of the AR image P9, the receiver 200 matches the size of the AR image P9 with the size of the first target area, and superimposes the AR image P9 whose size has been adjusted on the first target area. Furthermore, the receiver 200 matches the size of the AR image P9 with the size of the second target area, and superimposes the AR image P9 whose size has been adjusted on the second target area.

For example, the AR image P9 indicates a caption for the video of the transmitter 100. Further, the language of the caption of the AR image P9 is a language according to the user information set and registered in the receiver 200. That is, when transmitting the optical ID to the server, the receiver 200 also transmits the user information (for example, information indicating the user's nationality or language used) to the server. Then, the receiver 200 acquires an AR image P9 indicating a language caption corresponding to the user information. Alternatively, the receiver 200 acquires a plurality of AR images P9 indicating subtitles in different languages, and uses the AR images used for superimposition from the plurality of AR images P9 according to the user information registered and registered. P9 may be selected.

In other words, in the example illustrated in FIG. 254, the receiver 200 acquires a captured display image Pj and a decoding image by capturing a plurality of displays each displaying an image as a subject. Then, when the receiver 200 recognizes the target area, an area in which the transmission display (that is, the transmitter 100) that is the display that transmits the light ID among the plurality of displays appears in the captured display image Pj. Is recognized as a target area. Next, the receiver 200 superimposes the first subtitle corresponding to the image displayed on the transmission display as an AR image on the target area. Furthermore, the receiver 200 superimposes a second subtitle, which is a subtitle obtained by enlarging the first subtitle, on a region larger than the target region in the captured display image Pj.

Thereby, the receiver 200 can display the captured display image Pj so that captions actually exist in the video of the transmitter 100. Furthermore, since the receiver 200 also superimposes a large caption on the lower part of the captured display image Pj, even if the caption attached to the video of the transmitter 100 is small, the caption can be easily viewed. In addition, when there is no caption attached to the video of the transmitter 100 and only a large caption is superimposed on the lower part of the captured display image Pj, whether the superimposed caption is a caption for the video of the transmitter 100 or a television It is difficult to determine whether it is a caption for 114 videos. However, in the present embodiment, since captions are attached to the video of the transmitter 100 that transmits the optical ID, the user can easily determine which video the superimposed caption is for. Can do.

In the display of the captured display image Pj, the receiver 200 may further determine whether or not audio information is included in the information acquired from the server. When the receiver 200 determines that the audio information is included, the receiver 200 outputs the audio indicated by the audio information with priority over the first and second subtitles. Thereby, since sound is preferentially output, it is possible to reduce the burden of the user reading subtitles.

In the above example, the subtitle language is changed according to the user information (that is, the user attribute), but the video (that is, the content) itself displayed on the transmitter 100 may be changed. For example, in the case where the video displayed on the transmitter 100 is a news video, if the user information indicates that the user is Japanese, the receiver 200 may update the news video broadcast in Japan. Is acquired as an AR image. Then, the receiver 200 superimposes the news video on an area where the display of the transmitter 100 is displayed (that is, the target area). On the other hand, if the user information indicates that the user is American, the receiver 200 acquires a news video broadcast in the United States as an AR image. Then, the receiver 200 superimposes the news video on an area where the display of the transmitter 100 is displayed (that is, the target area). Thereby, an image suitable for the user can be displayed. Note that the user information indicates, for example, nationality or language used as the user attribute, and the receiver 200 acquires the above-described AR image based on the attribute.

FIG. 255 is a diagram showing an example of recognition information in the present embodiment.

Even if the recognition information is, for example, the above-described feature points and feature amounts, misrecognition may occur. For example, the transmitters 100a and 100b are configured as station names as with the transmitter 100, respectively. Even if these transmitters 100a and 100b are station names different from each other, they may be misrecognized because they are similar if they are located close to each other.

Therefore, the recognition information of each of the transmitters 100a and 100b does not indicate each feature point and each feature amount of the entire image of the transmitter 100a or 100b, and each feature point of only a characteristic part of the image and Each feature amount may be indicated.

For example, the part a1 of the transmitter 100a and the part b1 of the transmitter 100b are greatly different from each other, and the part a2 of the transmitter 100a and the part b2 of the transmitter 100b are greatly different from each other. Therefore, if the transmitters 100a and 100b are installed in a predetermined range (that is, a short distance), the server uses image characteristics of the parts a1 and a2 as the recognition information corresponding to the transmitter 100a. Preserve points and features. Similarly, the server holds the feature points and feature amounts of the images of the parts b1 and b2 as the identification information corresponding to the transmitter 100b.

Thereby, even when the transmitters 100a and 100b similar to each other are close to each other (when they are within the predetermined range described above), the receiver 200 can appropriately use the identification information. The target area can be recognized.

FIG. 256 is a flowchart showing another example of the processing operation of the receiver 200 in the present embodiment.

The receiver 200 first determines whether or not the user has a visual impairment based on the user information set and registered in the receiver 200 (step S131). If the receiver 200 determines that there is a visual impairment (Y in step S131), the receiver 200 outputs the characters of the AR image displayed in a superimposed manner by voice (step S132). On the other hand, when the receiver 200 determines that there is no visual impairment (N in step S131), the receiver 200 further determines whether the user has a hearing impairment based on the user information (step S133). Here, if the receiver 200 determines that there is a hearing impairment (Y in step S133), the receiver 200 stops the sound output (step S134). At this time, the receiver 200 stops outputting sound by all functions.

The receiver 200 may perform the process of step S133 when it is determined in step S131 that there is a visual impairment (Y in step S131). That is, when it is determined that there is a visual impairment and there is no hearing impairment, the receiver 200 may output the AR image characters displayed in a superimposed manner by voice.

FIG. 257 is a diagram illustrating an example in which the receiver 200 according to the present embodiment identifies bright line pattern regions.

First, the receiver 200 obtains a decoding image by imaging two transmitters each transmitting an optical ID, and performs decoding on the decoding image to obtain an optical ID as shown in FIG. 257 (e). To get. At this time, since the decoding image includes two bright line pattern areas X and Y, the receiver 200 transmits the light ID of the transmitter corresponding to the bright line pattern area X and the transmission corresponding to the bright line pattern area Y. Get the machine's optical ID. The light ID of the transmitter corresponding to the bright line pattern region X is made up of numerical values (that is, data) corresponding to addresses 0 to 9, for example, “5, 2, 8, 4, 3, 6, 1, 9, 4,3 ". Similarly, the transmitter optical ID corresponding to the bright line pattern region X is also composed of numerical values corresponding to addresses 0 to 9, for example, “5, 2, 7, 7, 1, 5, 3, 2, 7, 4 ".

Even if the receiver 200 acquires these light IDs once, that is, even if these light IDs are known, from which bright line pattern area each light ID was obtained when taking an image. It may be a situation that you do not understand. In such a case, the receiver 200 can easily determine from which bright line pattern area each known light ID is obtained by performing the processes shown in FIGS. 257 (a) to (d). Can be determined quickly.

Specifically, first, as shown in FIG. 257 (a), the receiver 200 acquires the decoding image Pdec11 and decodes each of the bright line pattern regions X and Y by decoding the decoding image Pdec11. The numerical value of the address 0 of the optical ID is acquired. For example, the numerical value of the light ID address 0 of the bright line pattern region X is “5”, and the numerical value of the light ID address 0 of the bright line pattern region Y is also “5”. Since the numerical value of the address 0 of each light ID is “5”, at this time, it cannot be determined from which bright line pattern area the known light ID is obtained.

Therefore, as shown in FIG. 257 (b), the receiver 200 obtains the decoding image Pdec12, and decodes the decoding image Pdec12 to obtain the address 1 of each light ID of the bright line pattern regions X and Y. Get the number of. For example, the numerical value of the address 1 of the light ID in the bright line pattern region X is “2”, and the numerical value of the address 1 of the light ID in the bright line pattern region Y is also “2”. Since the numerical value of the address 1 of each light ID is “2”, it is impossible to determine from which bright line pattern region the known light ID is obtained.

Therefore, as shown in FIG. 257 (c), the receiver 200 obtains the decoding image Pdec13 and decodes the decoding image Pdec13 to obtain the respective light IDs of the bright line pattern regions X and Y. Get the numerical value of address 2. For example, the numerical value of the address 2 of the light ID in the bright line pattern region X is “8”, and the numerical value of the address 2 of the light ID in the bright line pattern region Y is “7”. At this time, it can be determined that the known light ID “5, 2, 8, 4, 3, 6, 1, 9, 4, 3” has been obtained from the bright line pattern region X, and the known light ID “5, 2, 7, 7, 1, 5, 3, 2, 7, 4 "can be determined to have been obtained from the bright line pattern region Y.

However, in order to increase the reliability, the receiver 200 may further acquire the numerical value of the address 3 of each optical ID as shown in FIG. 257 (d). That is, the receiver 200 acquires the decoding image Pdec14, and acquires the numerical value of the address 3 of the light ID of each of the bright line pattern areas X and Y by decoding the decoding image Pdec14. For example, the numerical value of the address 3 of the light ID in the bright line pattern region X is “4”, and the numerical value of the address 3 of the light ID in the bright line pattern region Y is “7”. At this time, it can be determined that the known light ID “5, 2, 8, 4, 3, 6, 1, 9, 4, 3” has been obtained from the bright line pattern region X, and the known light ID “5, 2, 7, 7, 1, 5, 3, 2, 7, 4 "can be determined to have been obtained from the bright line pattern region Y. That is, since the light IDs of the bright line pattern areas X and Y can be identified not only by the address 2 but also by the address 3, the reliability can be increased.

Thus, in this embodiment, the numerical value of at least one address is reacquired without acquiring the numerical values (that is, data) of all addresses of the optical ID again. This makes it possible to easily and quickly determine from which bright line pattern region the known light ID is obtained.

In the example shown in (c) and (d) of FIG. 257 described above, the numerical value acquired for the predetermined address matches the numerical value of the known optical ID. Good. For example, in the example illustrated in (d) of FIG. 257, the receiver 200 acquires “6” as the numerical value of the address 3 of the light ID of the bright line pattern region Y. The numerical value “6” of the address 3 is different from the numerical value “7” of the address 3 of the known light ID “5, 2, 7, 7, 1, 5, 3, 2, 7, 4”. However, since the numerical value “6” is a numerical value close to the numerical value “7”, the receiver 200 has a known light ID “5, 2, 7, 7, 1, 5, 3, 2, 7, 4” with a bright line. It may be determined that the pattern area Y is obtained. In the receiver, the numerical value “6” is a numerical value close to the numerical value “7” depending on whether the numerical value “6” is within the range of the numerical value “7” ± n (n is a number of 1 or more, for example). It may be determined whether or not.

FIG. 258 is a diagram illustrating another example of the receiver 200 in the present embodiment.

The receiver 200 is configured as a smartphone in the above-described example, but may be configured as a head-mounted display (also referred to as glass) including an image sensor, similar to the examples illustrated in FIGS.

Such a receiver 200 detects a predetermined signal because power consumption increases when a processing circuit for displaying an AR image as described above (hereinafter referred to as an AR processing circuit) is always activated. When this occurs, the AR processing circuit may be activated.

For example, the receiver 200 includes a touch sensor 202. The touch sensor 202 outputs a touch signal when it touches a user's finger or the like. The receiver 200 activates the AR processing circuit when detecting the touch signal.

Alternatively, the receiver 200 may activate the AR processing circuit when detecting a radio signal such as Bluetooth (registered trademark) or Wi-Fi (registered trademark).

Alternatively, the receiver 200 may include an acceleration sensor, and may activate the AR processing circuit when the acceleration sensor measures an acceleration equal to or greater than a threshold value in a direction opposite to the direction of gravity. That is, the receiver 200 activates the AR processing circuit when detecting the signal indicating the acceleration. For example, when the user pushes up the nose pad portion of the receiver 200 configured as a glass upward with a fingertip from below, the receiver 200 detects a signal indicating the acceleration and activates the AR processing circuit.

Alternatively, the receiver 200 may activate the AR processing circuit when it is detected by the GPS and the 9-axis sensor that the image sensor is directed to the transmitter 100. That is, the receiver 200 activates the AR processing circuit when detecting a signal indicating that the receiver 200 is directed in a predetermined direction. In this case, if the transmitter 100 is the above-mentioned Japanese station name mark, the receiver 200 displays an AR image indicating the English station name superimposed on the station name mark.

FIG. 259 is a flowchart illustrating another example of the processing operation of the receiver 200 in the present embodiment.

When the receiver 200 acquires the optical ID from the transmitter 100 (step S141), the receiver 200 switches the mode of noise cancellation by receiving mode designation information corresponding to the optical ID (step S142). Then, the receiver 200 determines whether or not the mode switching process should be terminated (step S143), and when it is determined that the mode switching process should not be terminated (N in step S143), the process from step S141 is repeatedly executed. The switching of the noise canceling mode is, for example, a mode (ON) for canceling noise such as an engine in an airplane and a mode (OFF) for not canceling the noise. Specifically, a user carrying the receiver 200 is listening to a sound such as music output from the receiver 200 by putting an earphone connected to the receiver 200 on the ear. When such a user gets on the airplane, the receiver 200 acquires an optical ID. As a result, the receiver 200 switches the noise cancellation mode from OFF to ON. As a result, the user can hear a voice that does not include noise such as engine noise even in the cabin. The receiver 200 also acquires the light ID when the user leaves the airplane. The receiver 200 that has acquired this optical ID switches the noise cancellation mode from ON to OFF. Note that the noise to be subject to noise cancellation is not limited to engine noise but may be any sound such as a human voice.

FIG. 260 is a diagram illustrating an example of a transmission system including a plurality of transmitters in this embodiment.

This transmission system includes a plurality of transmitters 120 arranged in a predetermined order. Similar to the transmitter 100, these transmitters 120 are transmitters according to any one of the above-described embodiments 1 to 22, and include one or a plurality of light emitting elements (for example, LEDs). The leading transmitter 120 transmits the optical ID by changing the luminance of one or a plurality of light emitting elements according to a predetermined frequency (carrier frequency). Further, the first transmitter 120 outputs a signal indicating the change in luminance to the subsequent transmitter 120 as a synchronization signal. When the subsequent transmitter 120 receives the synchronization signal, it transmits the optical ID by changing the luminance of one or more light emitting elements in accordance with the synchronization signal. Further, the subsequent transmitter 120 outputs a signal indicating the change in luminance to the subsequent subsequent transmitter 120 as a synchronization signal. Thereby, all the transmitters 120 included in the transmission system transmit the optical ID in synchronization.

Here, the synchronization signal is transferred from the first transmitter 120 to the subsequent transmitter 120, and is further transferred from the subsequent transmitter 120 to the next subsequent transmitter 120 to the last transmitter 120. reach. For example, it takes about 1 μsec to transfer the synchronization signal. Therefore, if N (N is an integer of 2 or more) transmitters 120 are provided in the transmission system, it takes 1 × Nμ seconds for the synchronization signal to reach the last transmitter 120 from the first transmitter 120. become. As a result, the transmission timing of the optical ID is shifted by a maximum of N μ seconds. For example, even if N transmitters 120 transmit optical IDs according to a frequency of 9.6 kHz, and the receiver 200 attempts to receive an optical ID at a frequency of 9.6 kHz, the receiver 200 receives light that is shifted by N μ seconds. Since the ID is received, the optical ID may not be received correctly.

Therefore, in the present embodiment, the head transmitter 120 transmits the optical ID at a higher speed according to the number of transmitters 120 included in the transmission system. For example, the first transmitter 120 transmits an optical ID according to a frequency of 9.605 kHz. On the other hand, the receiver 200 receives the optical ID at a frequency of 9.6 kHz. At this time, even if the receiver 200 receives the optical ID shifted by N μs, the frequency of the leading transmitter 120 is higher than the frequency of the receiver 200 by 0.005 kHz. Occurrence can be suppressed.

Also, the first transmitter 120 may control the frequency adjustment amount by having the synchronization signal fed back from the last transmitter 120. For example, the first transmitter 120 measures the time from when it outputs the synchronization signal itself until it receives the synchronization signal fed back from the last transmitter 120. And the head transmitter 120 transmits optical ID according to a frequency higher than a reference frequency (for example, 9.6 kHz), so that the time is long.

FIG. 261 is a diagram illustrating an example of a transmission system including a plurality of transmitters and receivers in the present embodiment.

This transmission system includes, for example, two transmitters 120 and a receiver 200. One of the two transmitters 120 transmits an optical ID according to a frequency of 9.599 kHz. The other transmitter 120 transmits an optical ID according to a frequency of 9.601 kHz. In such a case, each of the two transmitters 120 notifies the receiver 200 of the frequency of its own optical ID with a radio wave signal.

When the receiver 200 receives the notification of those frequencies, the receiver 200 tries to perform decoding according to each of the notified frequencies. That is, the receiver 200 attempts to decode the decoding image according to the frequency of 9.599 kHz. If the optical ID cannot be received by this, the receiver 200 attempts to decode the decoding image according to the frequency of 9.601 kHz. As described above, the receiver 200 attempts to decode the decoding image according to each of all the notified frequencies. In other words, the receiver 200 performs brute force for each notified frequency. Alternatively, the receiver 200 may attempt decoding according to the average frequency of all the notified frequencies. That is, the receiver 200 attempts decoding according to 9.6 kHz, which is an average frequency of 9.599 kHz and 9.601 kHz.

Thereby, it is possible to reduce the occurrence rate of reception errors due to the difference in frequency between the receiver 200 and the transmitter 120.

FIG. 262A is a flowchart illustrating an example of processing operations of the receiver 200 in the present embodiment.

First, the receiver 200 starts imaging (step S151) and initializes the parameter N to 1 (step S152). Next, the receiver 200 decodes the decoding image obtained by the imaging according to the frequency corresponding to the parameter N, and calculates an evaluation value for the decoding result (step S153). For example, parameters N = 1, 2, 3, 4, 5 are associated with frequencies such as 9.6 kHz, 9.601 kHz, 9.599 kHz, and 9.602 kHz in advance. The evaluation value indicates a higher numerical value as the decoding result is more similar to the correct optical ID.

Next, the receiver 200 determines whether or not the numerical value of the parameter N is equal to Nmax that is a predetermined integer of 1 or more (step S154). Here, if the receiver 200 determines that it is not equal to Nmax (N in step S154), it increments the parameter N (step S155) and repeats the processing from step S153. On the other hand, if the receiver 200 determines that it is equal to Nmax (Y in step S154), the frequency for which the maximum evaluation value is calculated is registered as the optimum frequency in association with the location information indicating the location of the receiver 200. To do. The optimum frequency and location information registered in this way are used for receiving the optical ID by the receiver 200 that has moved to the location indicated by the location information after registration. The location information may be information indicating a position measured by GPS, for example, or may be identification information (for example, SSID: Service Set Identifier) of an access point in a wireless LAN (Local Area Network).

In addition, the receiver 200 that has registered with the server displays, for example, the AR image as described above according to the optical ID obtained by decoding at the optimum frequency.

FIG. 262B is a flowchart illustrating an example of processing operations of the receiver 200 in the present embodiment.

After registration with the server shown in FIG. 262A is performed, the receiver 200 transmits location information indicating a location where the receiver 200 exists to the server (step S161). Next, the receiver 200 acquires the optimum frequency registered in association with the location information from the server (step S162).

Next, the receiver 200 starts imaging (step S163), and decodes the decoding image obtained by the imaging according to the optimum frequency acquired in step S162 (step S164). The receiver 200 displays an AR image as described above, for example, according to the optical ID obtained by this decoding.

Thus, after registration to the server is performed, the receiver 200 can acquire the optimum frequency and receive the optical ID without executing the processing shown in FIG. 262A. Note that the receiver 200 may acquire the optimum frequency by executing the process illustrated in FIG. 262A when the optimum frequency cannot be obtained in step S162.

[Summary of Embodiment 23]
FIG. 263A is a flowchart illustrating a display method in this embodiment.

The display method in the present embodiment is a display method in which the display device that is the above-described receiver 200 displays an image, and includes steps SL11 to SL16.

In step SL11, the image sensor captures an image of the subject to acquire a captured display image and a decoding image. In step SL12, the optical ID is acquired by decoding the decoding image. In step SL13, the optical ID is transmitted to the server. In step SL14, the AR image corresponding to the optical ID and the recognition information are acquired from the server. In step SL15, an area corresponding to the recognition information in the captured display image is recognized as a target area. In step SL16, the captured display image in which the AR image is superimposed on the target area is displayed.

Thereby, since the AR image is displayed superimposed on the captured display image, an image useful for the user can be displayed. Furthermore, it is possible to superimpose an AR image on an appropriate target area while suppressing the processing load.

That is, in general augmented reality (that is, AR), a huge number of recognition target images stored in advance are compared with the captured display image, and any of the recognition target images is included in the captured display image. It is determined whether or not If it is determined that the recognition target image is included, the AR image corresponding to the recognition target image is superimposed on the captured display image. At this time, the AR image is aligned based on the recognition target image. In this way, in general augmented reality, a huge number of recognition target images are compared with captured display images, and further, position detection of the recognition target images in the captured display image is necessary even in alignment. There is a problem that the calculation amount is large and the processing load is high.

However, in the display method according to the present embodiment, as shown in FIGS. 235 to 262B, the light ID is acquired by decoding the decoding image obtained by imaging the subject. That is, the optical ID transmitted from the transmitter that is the subject is received. Furthermore, the AR image corresponding to this optical ID and the recognition information are acquired from the server. Therefore, the server does not need to compare a huge number of recognition target images and captured display images, and can select and transmit an AR image previously associated with the optical ID to the display device. Thereby, the amount of calculation can be reduced and the processing load can be significantly suppressed.

Further, in the display method in the present embodiment, the recognition information corresponding to this light ID is acquired from the server. The recognition information is information for recognizing a target area that is an area in which an AR image is superimposed in a captured display image. This recognition information may be information indicating that a white square is the target area, for example. In this case, the target area can be easily recognized, and the processing load can be further suppressed. That is, the processing load can be further suppressed according to the content of the recognition information. Further, since the server can arbitrarily set the content of the recognition information according to the optical ID, the balance between the processing load and the recognition accuracy can be appropriately maintained.

Here, the recognition information is reference information for specifying a reference area in the captured display image. In recognition of the target area, the reference area is specified from the captured display image based on the reference information, and the captured display image is displayed. Of these, the target area may be recognized based on the position of the reference area.

Alternatively, the recognition information may include reference information for specifying a reference area in the captured display image and target information indicating a relative position of the target area with respect to the reference area. In this case, in the recognition of the target area, the reference area is identified from the captured display image based on the reference information, and the area at the relative position indicated by the target information in the captured display image with the position of the reference area as a reference, Recognize as a target area.

Thereby, as shown in FIGS. 244 and 245, the degree of freedom of the position of the target area recognized in the captured display image can be expanded.

Further, the reference information is such that the position of the reference area in the captured display image is the same as the position of the bright line pattern area composed of a plurality of bright line patterns appearing by exposure of the multiple exposure lines of the image sensor in the decoding image. May be indicated.

Thereby, as shown in FIGS. 244 and 245, the target area can be recognized with reference to the area corresponding to the bright line pattern area in the captured display image.

Further, the reference information may indicate that the reference area in the captured display image is an area in which the display of the captured display image is displayed.

Thus, as shown in FIG. 235, for example, if a station name sign is used as a display, the target area can be recognized with reference to the area where the display is displayed.

Further, in the display of the captured display image, the first AR image is displayed for a predetermined display period while suppressing the display of the second AR image different from the first AR image that is the above-described AR image. May be.

As a result, as shown in FIG. 250, when the user is viewing the first AR image once displayed, the first AR image is immediately replaced with a different second AR image. Can be suppressed.

In the display of the captured display image, decoding of a newly acquired decoding image may be prohibited during the display period.

As a result, as shown in FIG. 250, decoding of the newly acquired decoding image is a wasteful process when the display of the second AR image is suppressed. Power consumption can be reduced.

In the display of the captured display image, the acceleration of the display device may be measured by an acceleration sensor during the display period, and it may be determined whether or not the measured acceleration is equal to or greater than a threshold value. And when it determines with more than a threshold value, you may display a 2nd AR image instead of a 1st AR image by canceling suppression of a display of the 2nd AR image.

Thereby, as shown in FIG. 250, when the acceleration of the display device equal to or higher than the threshold is measured, the suppression of the display of the second AR image is released. Accordingly, for example, when the user moves the display device greatly to direct the image sensor toward another subject, the second AR image can be displayed immediately.

Further, in the display of the captured display image, it may be further determined whether or not the user's face is approaching the display device by imaging with a face camera provided in the display device. And if it determines with the face approaching, you may display a 1st AR image, suppressing the display of the 2nd AR image different from a 1st AR image. Alternatively, in the display of the captured display image, whether or not the user's face is approaching the display device may be further determined based on the acceleration of the display device measured by the acceleration sensor. And if it determines with the face approaching, you may display a 1st AR image, suppressing the display of the 2nd AR image different from a 1st AR image.

As a result, as shown in FIG. 250, when the user is looking at the first AR image and the face is brought close to the display device, the first AR image is replaced with a different second AR image. That can be suppressed.

Further, as shown in FIG. 254, in the acquisition of the captured display image and the decoding image, the captured display image and the decoding image are acquired by capturing a plurality of displays each displaying an image as a subject. Also good. At this time, in the recognition of the target area, an area in which a transmission display that is a display that transmits the light ID among the plurality of displays appears in the captured display image is recognized as the target area. Further, in the display of the captured display image, the first subtitle corresponding to the image displayed on the transmission display is superimposed on the target area as an AR image, and further, in an area larger than the target area of the captured display image, The second subtitle, which is an expanded subtitle of the first subtitle, is superimposed.

Thereby, since the first subtitle is superimposed on the image on the transmission display, the user can easily grasp which display subtitle is the subtitle for the display image of the plurality of displays. it can. In addition, since the second subtitle, which is an enlarged subtitle of the first subtitle, is also displayed, even if the first subtitle is small and difficult to read, the subtitle can be easily read by displaying the second subtitle. can do.

Further, in the display of the captured display image, it is further determined whether or not audio information is included in the information acquired from the server. When it is determined that the information is included, the first and second subtitles are determined. The voice indicated by the voice information may be output with priority.

This allows the user to reduce the burden of reading subtitles because the audio is output preferentially.

FIG. 263B is a block diagram illustrating a structure of the display device in this embodiment.

The display device 10 according to the present embodiment is a display device that displays an image, and includes an image sensor 11, a decoding unit 12, a transmission unit 13, an acquisition unit 14, a recognition unit 15, and a display unit 16. Prepare. The display device 10 corresponds to the receiver 200 described above.

The image sensor 11 acquires a captured display image and a decoding image by imaging a subject. The decoding unit 12 acquires the optical ID by decoding the decoding image. The transmission unit 13 transmits the optical ID to the server. The acquisition unit 14 acquires the AR image corresponding to the optical ID and the recognition information from the server. The recognition unit 15 recognizes a region corresponding to the recognition information in the captured display image as a target region. The display unit 16 displays a captured display image in which an AR image is superimposed on the target area.

Thereby, since the AR image is displayed superimposed on the captured display image, an image useful for the user can be displayed. Furthermore, it is possible to superimpose an AR image on an appropriate target area while suppressing the processing load.

In the present embodiment, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. Here, software for realizing the receiver 200 or the display device 10 according to the present embodiment is included in each of the flowcharts shown in FIGS. 239, 246, 250, 256, 259, and 262A to 263A. A program for causing a computer to execute steps.

[Modification 1 of Embodiment 23]
Hereinafter, a first modification of the twenty-third embodiment, that is, a first modification of the display method for realizing the AR using the optical ID will be described.

FIG. 264 is a diagram illustrating an example in which the receiver in the first modification of the twenty-third embodiment displays an AR image.

The receiver 200 acquires the above-described captured display image Pk, which is the normal captured image, and the above-described decoding image, which is the visible light communication image or the bright line image, by capturing the subject with the image sensor.

Specifically, the image sensor of the receiver 200 images the transmitter 100c configured as a robot and the person 21 adjacent to the transmitter 100c. The transmitter 100c is the transmitter according to any one of Embodiments 1 to 22, and includes one or a plurality of light emitting elements (for example, LEDs) 131. The transmitter 100c changes its luminance by blinking the one or more light emitting elements 131, and transmits an optical ID (optical identification information) by the luminance change. This light ID is the above-mentioned visible light signal.

The receiver 200 acquires the captured display image Pk on which the transmitter 100c and the person 21 are imaged by the normal exposure time. Furthermore, the receiver 200 acquires the decoding image by capturing the transmitter 100c and the person 21 with the communication exposure time shorter than the normal exposure time.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100c. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P10 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pk as a target area. For example, the receiver 200 recognizes an area on the right side of the area where the robot that is the transmitter 100c is projected as the target area. Specifically, the receiver 200 specifies the distance between the two markers 132a and 132b of the transmitter 100c displayed in the captured display image Pk. Then, the receiver 200 recognizes an area having a width and a height corresponding to the distance as a target area. That is, the recognition information indicates the shape of the markers 132a and 132b and the position and size of the target region with reference to the markers 132a and 132b.

Then, the receiver 200 superimposes the AR image P10 on the target area, and displays the captured display image Pk on which the AR image P10 is superimposed on the display 201. For example, the receiver 200 acquires an AR image P10 indicating another robot different from the transmitter 100c. In this case, since the AR image P10 is superimposed on the target area of the captured display image Pk, the captured display image Pk can be displayed so that another robot actually exists next to the transmitter 100c. As a result, the person 21 can be photographed together with the other robot together with the transmitter 100c even if no other robot exists.

FIG. 265 is a diagram illustrating another example in which the receiver 200 in the first modification of the twenty-third embodiment displays an AR image.

For example, as shown in FIG. 265, the transmitter 100 is configured as an image display device having a display panel, and transmits a light ID by changing the luminance while displaying a still image PS on the display panel. The display panel is, for example, a liquid crystal display or an organic EL (electroluminescence) display.

The receiver 200 acquires the captured display image Pm and the decoding image by imaging the transmitter 100 in the same manner as described above. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P11 and the recognition information corresponding to the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pm as a target area. For example, the receiver 200 recognizes the area where the display panel of the transmitter 100 is displayed as the target area. Then, the receiver 200 superimposes the AR image P11 on the target area, and displays the captured display image Pm on which the AR image P11 is superimposed on the display 201. For example, the AR image P11 is a moving image having the same or substantially the same picture as the still image PS displayed on the display panel of the transmitter 100 as the first picture in the display order. That is, the AR image P11 is a moving image that starts to move from the still image PS.

In this case, since the AR image P11 is superimposed on the target region of the captured display image Pm, the receiver 200 displays the captured display image Pm so that an image display device that displays a moving image actually exists. Can do.

FIG. 266 is a diagram illustrating another example in which the receiver 200 in the first modification of the twenty-third embodiment displays an AR image.

The transmitter 100 is configured as a station name symbol as shown in FIG. 266, for example, and transmits the light ID by changing the luminance.

The receiver 200 images the transmitter 100 from a position away from the transmitter 100 as shown in FIG. Thereby, the receiver 200 acquires the captured display image Pn and the decoding image in the same manner as described above. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR images P12 to P14 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes two regions corresponding to the recognition information in the captured display image Pn as first and second target regions. For example, the receiver 200 recognizes the area around the transmitter 100 as the first target area. Then, the receiver 200 superimposes the AR image P12 on the first target area, and displays the captured display image Pn on which the AR image P12 is superimposed on the display 201. For example, the AR image P12 is an arrow that prompts the user of the receiver 200 to approach the transmitter 100.

In this case, since the AR image P12 is displayed superimposed on the first target area of the captured display image Pn, the user approaches the transmitter 100 with the receiver 200 facing the transmitter 100. As the receiver 200 approaches the transmitter 100, the area of the transmitter 100 (corresponding to the reference area described above) displayed in the captured display image Pn becomes larger. When the size of the area becomes equal to or larger than the first threshold, the receiver 200 further moves to a second target area that is an area where the transmitter 100 is projected, as shown in FIG. 266 (b), for example. The AR image P13 is superimposed. That is, the receiver 200 displays the captured display image Pn on which the AR images P12 and P13 are superimposed on the display 201. For example, the AR image P13 is a message that informs the user of the outline of the vicinity of the station indicated by the station name sign. The AR image P13 is equal to the size of the area of the transmitter 100 displayed in the captured display image Pn.

Also in this case, since the AR image P12 that is an arrow is superimposed and displayed on the first target area of the captured display image Pn, the user transmits the transmitter 200 with the receiver 200 facing the transmitter 100. Approaching 100. As the receiver 200 approaches the transmitter 100, the area of the transmitter 100 (corresponding to the reference area described above) displayed in the captured display image Pn becomes larger. When the size of the area becomes equal to or larger than the second threshold, the receiver 200 changes the AR image P13 superimposed on the second target area to the AR image P14, for example, as illustrated in (c) of FIG. 266. To do. Furthermore, the receiver 200 deletes the AR image P12 superimposed on the first target area.

That is, the receiver 200 displays the captured display image Pn on which the AR image P14 is superimposed on the display 201. For example, the AR image P14 is a message that informs the user of the details around the station indicated by the station name sign. The AR image P14 is equal to the size of the area of the transmitter 100 displayed in the captured display image Pn. The area of the transmitter 100 is larger as the receiver 200 is closer to the transmitter 100. Therefore, the AR image P14 is larger than the AR image P13.

Thus, the receiver 200 enlarges the AR image and displays more information as it approaches the transmitter 100. In addition, since an arrow that prompts the user to approach, such as the AR image P12, is displayed, the user can easily grasp that a large amount of information is displayed when approaching the transmitter 100.

FIG. 267 is a diagram illustrating another example in which the receiver 200 in the first modification of the twenty-third embodiment displays an AR image.

In the example shown in FIG. 266, the receiver 200 displays a lot of information when approaching the transmitter 100, but displays a lot of information in the form of, for example, a balloon regardless of the distance to the transmitter 100. Also good.

Specifically, as shown in FIG. 267, the receiver 200 captures the captured display image Po and the decoding image by capturing the transmitter 100 in the same manner as described above. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P15 and the recognition information corresponding to the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Po as a target area. For example, the receiver 200 recognizes an area around the transmitter 100 as a target area. Then, the receiver 200 superimposes the AR image P15 on the target area, and displays the captured display image Po on which the AR image P15 is superimposed on the display 201. For example, the AR image P15 is a message that informs the user of the details of the vicinity of the station indicated by the station name in a balloon form.

In this case, since the AR image P15 is superimposed on the target area of the captured display image Po, the user of the receiver 200 can display a lot of information on the receiver 200 without approaching the transmitter 100.

FIG. 268 is a diagram illustrating another example of the receiver 200 in the first modification of the twenty-third embodiment.

The receiver 200 is configured as a smartphone in the above-described example, but is configured as a head-mounted display (also referred to as glass) including an image sensor, similar to the examples illustrated in FIGS. 19 to 21 and 258. Also good.

Such a receiver 200 acquires the optical ID by performing decoding only on a part of the decoding target area of the decoding image. For example, the receiver 200 includes a line-of-sight detection camera 203 as shown in FIG. The line-of-sight detection camera 203 images the eyes of the user wearing the head-mounted display that is the receiver 200. The receiver 200 detects the user's line of sight based on the eye image obtained by the imaging by the line-of-sight detection camera 203.

As shown in FIG. 268 (b), for example, the receiver 200 displays the line-of-sight frame 204 so that the line-of-sight frame 204 appears in the area of the user's field of view where the detected line of sight is directed. . Accordingly, the line-of-sight frame 204 moves according to the movement of the user's line of sight. The receiver 200 treats an area corresponding to the line-of-sight frame 204 in the decoding image as a decoding target area. In other words, the receiver 200 does not decode the bright line pattern area in the decoding image even if there is a bright line pattern area outside the decoding target area, and performs decoding only on the bright line pattern area in the decoding target area. . As a result, even when there are a plurality of bright line pattern areas in the decoding image, decoding is not performed for all of the bright line pattern areas, so that the processing load can be reduced and the display of an extra AR image can be suppressed. Can do.

In addition, when a plurality of bright line pattern areas for outputting sound are included in the decoding image, the receiver 200 decodes only the bright line pattern area in the decoding target area, and the bright line pattern area Only the sound corresponding to may be output. Alternatively, the receiver 200 decodes each of the plurality of bright line pattern regions included in the decoding image, outputs a large amount of sound corresponding to the bright line pattern region in the decoding target region, and outputs to the bright line pattern region outside the decoding target region. The corresponding voice may be output small. In addition, when there are a plurality of bright line pattern areas outside the decoding target area, the receiver 200 may output a larger amount of sound corresponding to the bright line pattern area as the bright line pattern area is closer to the decoding target area.

FIG. 269 is a diagram illustrating another example in which the receiver 200 in the first modification of the twenty-third embodiment displays an AR image.

For example, as shown in FIG. 269, the transmitter 100 is configured as an image display device having a display panel, and transmits an optical ID by changing luminance while displaying an image on the display panel.

The receiver 200 acquires the captured display image Pp and the decoding image in the same manner as described above by imaging the transmitter 100.

At this time, the receiver 200 specifies, from the captured display image Pp, an area having the same position as the bright line pattern area in the decoding image and the same size as the bright line pattern area. The receiver 200 may display the scanning line P100 that repeatedly moves from one end of the region to the other end.

While the scanning line P100 is displayed, the receiver 200 acquires the optical ID by decoding the decoding image and transmits the optical ID to the server. Then, the receiver 200 acquires the AR image corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pp as a target area.

When recognizing such a target area, the receiver 200 ends the display of the scanning line P100, superimposes the AR image on the target area, and displays the captured display image Pp on which the AR image is superimposed on the display 201. .

Accordingly, since the moving scanning line P100 is displayed from when the transmitter 100 is imaged until the AR image is displayed, processing such as reading of the optical ID and the AR image is performed. Can be notified to the user.

FIG. 270 is a diagram illustrating another example in which the receiver 200 in the first modification of the twenty-third embodiment displays an AR image.

Each of the two transmitters 100 is configured as an image display device having a display panel, for example, as shown in FIG. 270, and transmits an optical ID by changing the luminance while displaying the same still image PS on the display panel. is doing. Here, the two transmitters 100 transmit different optical IDs (for example, optical IDs “01” and “02”) by changing the luminance in different manners.

The receiver 200 acquires the captured display image Pq and the decoding image by capturing images of the two transmitters 100 as in the example illustrated in FIG. The receiver 200 acquires the optical IDs “01” and “02” by decoding the decoding image. That is, the receiver 200 receives the optical ID “01” from one of the two transmitters 100 and receives the optical ID “02” from the other. The receiver 200 transmits those optical IDs to the server. Then, the receiver 200 acquires the AR image P16 corresponding to the optical ID “01” and the recognition information from the server. Furthermore, the receiver 200 acquires the AR image P17 corresponding to the optical ID “02” and the recognition information from the server.

The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pq as a target area. For example, the receiver 200 recognizes an area where the display panels of the two transmitters 100 are displayed as the target area. Then, the receiver 200 superimposes the AR image P16 on the target area corresponding to the light ID “01”, and superimposes the AR image P17 on the target area corresponding to the light ID “02”. Then, the receiver 200 displays the captured display image Pq on which the AR images P16 and P17 are superimposed on the display 201. For example, the AR image P16 is a moving image having the same or substantially the same picture as the still picture PS displayed on the display panel of the transmitter 100 corresponding to the optical ID “01” as the first picture in the display order. is there. The AR image P17 is a moving image having the same or substantially the same picture as the still picture PS displayed on the display panel of the transmitter 100 corresponding to the optical ID “02” as the first picture in the display order. is there. That is, the leading pictures of the AR image P16 and the AR image P17, which are moving images, are the same. However, the AR image P16 and the AR image P17 are different moving images, and the pictures other than the head of each are different.

Therefore, since the AR image P16 and the AR image P17 are superimposed on the captured display image Pq, the receiver 200 actually has an image display device that displays different moving images reproduced from the same picture. The captured display image Pq can be displayed.

FIG. 271 is a flowchart illustrating an example of processing operations of the receiver 200 in Modification 1 of Embodiment 23. The processing operation shown by the flowchart of FIG. 271 is an example of the processing operation of the receiver 200 that individually images each of the transmitters 100 when there are two transmitters 100 shown in FIG. is there.

First, the receiver 200 acquires the first light ID by imaging the first transmitter 100 as the first subject (step S201). Next, the receiver 200 recognizes the first subject from the captured display image (step S202). That is, the receiver 200 acquires the first AR image and the first recognition information corresponding to the first light ID from the server, and recognizes the first subject based on the first recognition information. Then, the receiver 200 starts reproduction of the first moving image that is the first AR image from the beginning (step S203). That is, the receiver 200 starts reproduction from the first picture of the first moving image.

Here, the receiver 200 determines whether or not the first subject is out of the captured display image (step S204). That is, the receiver 200 determines whether or not the first subject cannot be recognized from the captured display image. Here, if it is determined that the first subject has deviated from the captured display image (Y in step S204), the receiver 200 interrupts the reproduction of the first moving image that is the first AR image (step S205).

Next, the receiver 200 captures a second transmitter 100 different from the first transmitter 100 as a second subject, thereby different from the first light ID acquired in step S201. It is determined whether or not the optical ID of the first one has been acquired (step S206). Here, when the receiver 200 determines that the second optical ID has been acquired (Y in step S206), the receiver 200 performs the same processing as steps S202 to S203 after the first optical ID is acquired. That is, the receiver 200 recognizes the second subject from the captured display image (step S207). Then, the receiver 200 starts reproduction of the second moving image that is the second AR image corresponding to the second optical ID from the beginning (step S208). That is, the receiver 200 starts playback from the first picture of the second moving image.

On the other hand, if the receiver 200 determines in step S206 that the second light ID has not been acquired (N in step S206), it determines whether or not the first subject has entered the captured display image again (step S206). S209). That is, the receiver 200 determines whether or not the first subject is recognized again from the captured display image. Here, when the receiver 200 determines that the first subject has entered the captured display image (Y in step S209), the receiver 200 further determines whether or not a predetermined time (that is, a predetermined time) has passed (step S209). Step S210). That is, the receiver 200 determines whether or not a predetermined time has elapsed from when the first subject is removed from the captured display image until it enters again. If it is determined that the predetermined time has not elapsed (Y in step S210), the receiver 200 starts reproduction from the middle of the interrupted first moving image (step S211). It should be noted that the first picture to be resumed for reproduction, which is the first picture of the first moving picture that is displayed at the beginning of the reproduction from the middle, is the next picture that was last displayed when the reproduction of the first moving picture was interrupted. The pictures may be in the display order. Alternatively, the reproduction restart top picture may be a picture preceding the last displayed picture by n (n is an integer of 1 or more) in display order.

On the other hand, if it is determined that the predetermined time has elapsed (N in step S210), the receiver 200 starts playback of the interrupted first moving image from the beginning (step S212).

In the above example, the receiver 200 superimposes the AR image on the target area of the captured display image. At this time, the brightness of the AR image may be adjusted. That is, the receiver 200 determines whether or not the brightness of the AR image acquired from the server matches the brightness of the target area of the captured display image. If the receiver 200 determines that they do not match, the receiver 200 adjusts the brightness of the AR image to match the brightness of the AR image with the brightness of the target region. Then, the receiver 200 superimposes the AR image whose brightness has been adjusted on the target area of the captured display image. Thereby, the superimposed AR image can be brought closer to the image of the actual object, and the user's uncomfortable feeling with respect to the AR image can be suppressed. Note that the brightness of the AR image is the spatial average brightness of the AR image, and the brightness of the target area is also the spatial average brightness of the target area.

Further, as shown in FIG. 247, when the receiver 200 taps an AR image, the receiver 200 may enlarge the AR image and display it on the entire display 201. In the example illustrated in FIG. 247, the receiver 200 switches the AR image on which the AR image is tapped to another AR image, but may automatically switch the AR image regardless of the tap. For example, when the AR image is displayed for a predetermined time, the receiver 200 switches the AR image to another AR image for display. Further, when the current time reaches a predetermined time, the receiver 200 switches the AR image that has been displayed so far to another AR image and displays it. Thereby, the user can easily see a new AR image without performing an operation.

[Modification 2 of Embodiment 23]
Hereinafter, a second modification of the twenty-third embodiment, that is, a second modification of the display method for realizing the AR using the optical ID will be described.

FIG. 272 is a diagram illustrating an example of a problem when displaying an AR image assumed in the receiver 200 in the twenty-third embodiment or the first modification thereof.

For example, the receiver 200 in the twenty-third embodiment or its modification 1 images the subject at time t1. Note that the above-described subject is a transmitter such as a television that transmits a light ID according to a change in luminance, or a poster, a guide board, or a signboard illuminated by light from the transmitter. As a result, the receiver 200 displays the entire image obtained by the effective pixel area of the image sensor (hereinafter, referred to as all captured images) on the display 201 as a captured display image. At this time, the receiver 200 recognizes, in the captured display image, an area corresponding to the recognition information acquired based on the light ID as a target area on which the AR image is superimposed. The target area is an area indicating an image of a transmitter such as a television or an image of a poster, for example. Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201. The AR image may be a still image or a moving image, or a character string including one or more characters or symbols.

Here, when the user of the receiver 200 approaches the subject in order to display the AR image in a large size, a region corresponding to the target region in the image sensor (hereinafter referred to as a recognition region) protrudes from the effective pixel region at time t2. The recognition area is an area in which an image of the target area in the captured display image is projected in the effective pixel area of the image sensor. That is, the effective pixel area and the recognition area in the image sensor correspond to the captured display image and the target area on the display 201, respectively.

When the recognition area protrudes from the effective pixel area, the receiver 200 cannot recognize the target area from the captured display image and cannot display the AR image.

Therefore, the receiver 200 in the present modification acquires an image having a wider angle of view than the captured display image displayed on the entire display 201 as the entire captured image.

FIG. 273 is a diagram illustrating an example in which the receiver 200 in the second modification of the twenty-third embodiment displays an AR image.

The field angle of all captured images of the receiver 200 according to this modification, that is, the field angle of the effective pixel area of the image sensor is wider than the field angle of the captured display image displayed on the entire display 201. In the image sensor, an area corresponding to an image range displayed on the display 201 is hereinafter referred to as a display area.

For example, the receiver 200 images the subject at time t1. As a result, the receiver 200 displays, on the display 201, only the image obtained by the display area narrower than the effective pixel area among all the captured images obtained by the effective pixel area of the image sensor as the captured display image. At this time, similarly to the above, the receiver 200 recognizes an area corresponding to the recognition information acquired based on the light ID among all the captured images as a target area on which the AR image is superimposed. Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201.

Here, when the user of the receiver 200 approaches the subject in order to display the AR image in a large size, the recognition area in the image sensor is expanded. At time t2, the recognition area protrudes from the display area in the image sensor. That is, an image of the target region (for example, a poster image) protrudes from the captured display image displayed on the display 201. However, the recognition area in the image sensor does not protrude from the effective pixel area. That is, the receiver 200 acquires all captured images including the target area even at time t2. As a result, the receiver 200 can recognize the target area from all the captured images, and superimposes a part of the AR image corresponding to the target area only on a part of the target area in the captured display image. Are displayed on the display 201.

Thereby, even when the user approaches the subject to display the AR image in a large size and the target region protrudes from the captured display image, the display of the AR image can be continued.

FIG. 274 is a flowchart illustrating an example of processing operations of the receiver 200 in the second modification of the twenty-third embodiment.

The receiver 200 acquires the entire captured image and the decoding image by the image sensor capturing the subject (step S301). Next, the receiver 200 acquires an optical ID by decoding the decoding image (step S302). Next, the receiver 200 transmits the optical ID to the server (step S303). Next, the receiver 200 acquires an AR image and recognition information corresponding to the optical ID from the server (step S304). Next, the receiver 200 recognizes an area corresponding to the recognition information among all captured images as a target area (step S305).

Here, the receiver 200 determines whether or not the recognition area that is the area corresponding to the image of the target area in the effective pixel area of the image sensor protrudes from the display area (step S306). Here, if it is determined that it is protruding (Yes in step S306), the receiver 200 extracts a part of the AR image corresponding to the area only in a part of the target area within the captured display image. It is displayed (step S307). On the other hand, when the receiver 200 determines that it does not protrude (No in step S306), the receiver 200 superimposes the AR image on the target area of the captured display image and displays the captured display image on which the AR image is superimposed. (Step S308).

Then, the receiver 200 determines whether or not the AR image display process should be terminated (step S309), and when it is determined that the AR image display process should not be terminated (No in step S309), the process from step S305 is repeatedly executed.

FIG. 275 is a diagram illustrating another example in which the receiver 200 in the second modification of the twenty-third embodiment displays an AR image.

The receiver 200 may switch the screen display of the AR image according to the ratio of the size of the recognition area to the display area.

When the horizontal width of the display area of the image sensor is w1, the vertical width is h1, the horizontal width of the recognition area is w2, and the vertical width is h2, the receiver uses the ratio (h2 / h1). ) And (w2 / w1), the larger ratio is compared with the threshold value.

For example, as shown in (screen display 1) in FIG. 275, the receiver 200 displays a captured display image in which the AR image is superimposed on the target region, and sets the larger ratio to the first threshold ( For example, compare with 0.9). When the larger ratio becomes 0.9 or more, the receiver 200 enlarges and displays the AR image on the entire display 201 as shown in (screen display 2) in FIG. Note that when the recognition area becomes larger than the display area and further when the recognition area becomes larger than the effective pixel area, the receiver 200 continues to enlarge and display the AR image on the entire display 201.

Further, for example, when the AR image is enlarged and displayed on the entire display 201 as shown in FIG. 275 (screen display 2), the receiver 200 sets the larger ratio to the second threshold ( For example, compare with 0.7). The second threshold is smaller than the first threshold. When the larger ratio becomes 0.7 or less, the receiver 200 displays a captured display image in which the AR image is superimposed on the target area as shown in (screen display 1) in FIG.

FIG. 276 is a flowchart illustrating another example of the processing operation of the receiver 200 in the second modification of the twenty-third embodiment.

First, the receiver 200 performs optical ID processing (step S301a). This optical ID process is a process including steps S301 to S304 shown in FIG. Next, the receiver 200 recognizes an area corresponding to the recognition information in the captured display image as a target area (step S311). Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed (step S312).

Next, the receiver 200 has a recognition area ratio, that is, a larger ratio of the ratios (h2 / h1) and (w2 / w1) is equal to or greater than a first threshold value K (for example, K = 0.9). It is determined whether or not (step S313). If it is determined that the value is not equal to or greater than the first threshold value K (No in step S313), the receiver 200 repeatedly executes the processing from step S311. On the other hand, if it determines with it being more than the 1st threshold value K (Yes of step S313), the receiver 200 will expand and display the AR image on the entire display 201 (step S314). At this time, the receiver 200 periodically switches the power supply of the image sensor between on and off. Power saving of the receiver 200 can be achieved by periodically turning off the power supply of the image sensor.

Next, the receiver 200 determines whether or not the recognition area ratio is equal to or smaller than a second threshold L (for example, L = 0.7) when the image sensor is periodically turned on. To do. Here, if it is determined that it is not equal to or less than the second threshold L (No in step S315), the receiver 200 repeatedly executes the processing from step S314. On the other hand, if it is determined that it is equal to or smaller than the second threshold value L (Yes in step S315), the receiver 200 superimposes the AR image on the target area of the captured display image and displays the captured display image on which the AR image is superimposed. (Step S316).

Then, the receiver 200 determines whether or not to end the AR image display process (step S317). If the receiver 200 determines that the AR image display process should not be ended (No in step S317), the receiver 200 repeatedly executes the processes from step S313.

Thus, by setting the second threshold value L to be smaller than the first threshold value K, the screen display of the receiver 200 can be frequently switched between (screen display 1) and (screen display 2). Can be prevented and the state of the screen display can be stabilized.

In the example shown in FIGS. 275 and 276, the display area and the effective pixel area may be the same or different. In this example, the ratio of the size of the recognition area to the display area is used. However, if the display area and the effective pixel area are different, the size of the recognition area relative to the effective pixel area is used instead of the display area. A ratio may be used.

FIG. 277 is a diagram illustrating another example in which the receiver 200 in the second modification of the twenty-third embodiment displays an AR image.

In the example shown in FIG. 277, like the example shown in FIG. 273, the image sensor of the receiver 200 has an effective pixel area wider than the display area.

For example, the receiver 200 images the subject at time t1. As a result, the receiver 200 displays, on the display 201, only the image obtained by the display area narrower than the effective pixel area among all the captured images obtained by the effective pixel area of the image sensor as the captured display image. At this time, similarly to the above, the receiver 200 recognizes an area corresponding to the recognition information acquired based on the light ID among all the captured images as a target area on which the AR image is superimposed. Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201.

Here, when the user changes the direction of the receiver 200 (specifically, the image sensor), the recognition area in the image sensor moves, for example, in the upper left direction in FIG. 277, and protrudes from the display area at time t2. That is, an image of the target region (for example, a poster image) protrudes from the captured display image displayed on the display 201. However, the recognition area in the image sensor does not protrude from the effective pixel area. That is, the receiver 200 acquires all captured images including the target area even at time t2. As a result, the receiver 200 can recognize the target area from all the captured images, and superimposes a part of the AR image corresponding to the area only on a part of the target area in the captured display image. And displayed on the display 201. Furthermore, the receiver 200 changes the size and position of a part of the displayed AR image in accordance with the movement of the recognition area in the image sensor, that is, the movement of the target area in all captured images.

When the recognition area protrudes from the display area as described above, the receiver 200 counts the number of pixels corresponding to the distance between the edge of the effective pixel area and the edge of the display area (hereinafter referred to as inter-area distance). Is compared to a threshold.

For example, the shorter distance (hereinafter referred to as the first distance) among the distance between the upper side of the effective pixel area and the upper side of the display area, and the distance between the lower side of the effective pixel area and the lower side of the display area. Let dh be the number of pixels corresponding to (distance). Further, the shorter distance (hereinafter referred to as the second distance) among the distance between the left side of the effective pixel area and the left side of the display area, and the distance between the right side of the effective pixel area and the right side of the display area. Let dw be the number of pixels corresponding to (distance). At this time, the above-mentioned inter-region distance is the shorter one of the first and second distances.

That is, the receiver 200 compares the smaller pixel number of the pixel numbers dw and dh with the threshold value N. Then, for example, when the smaller number of pixels becomes equal to or smaller than the threshold value N at time t2, the receiver 200 changes the size and position of a part of the AR image according to the position of the recognition area in the image sensor. Fix without. That is, the receiver 200 switches the screen display of the AR image. For example, the receiver 200 sets the size and position of a part of the displayed AR image to the size of the part of the AR image displayed on the display 201 when the smaller number of pixels reaches the threshold value N. Fix in position and position.

Therefore, even when the recognition area further moves and protrudes from the effective pixel area at time t3, the receiver 200 continues to display a part of the AR image similarly to time t2. That is, as long as the smaller number of pixels dw and dh is equal to or smaller than the threshold value N, the receiver 200 captures a part of the AR image whose size and position are fixed as at time t2. Continue to superimpose on the display image.

In the example illustrated in FIG. 277, the receiver 200 changes the size and position of a part of the displayed AR image in accordance with the movement of the recognition area in the image sensor, but the display magnification and position of the entire AR image are changed. It may be changed.

FIG. 278 is a diagram illustrating another example in which the receiver 200 in the second modification of the twenty-third embodiment displays an AR image. Specifically, FIG. 278 shows an example in which the display magnification of the AR image is changed.

For example, as in the example shown in FIG. 277, when the user changes the orientation of the receiver 200 (specifically, the image sensor) from the state at the time t1, the recognition area in the image sensor becomes, for example, in the upper left direction in FIG. It moves and protrudes from the display area at time t2. That is, an image of the target region (for example, a poster image) protrudes from the captured display image displayed on the display 201. However, the recognition area in the image sensor does not protrude from the effective pixel area. That is, the receiver 200 acquires all captured images including the target area even at time t2. As a result, the receiver 200 can recognize the target area from all captured images.

Therefore, in the example illustrated in FIG. 278, the receiver 200 displays the display magnification of the AR image so that the size of the entire AR image matches the size of a part of the target region in the captured display image. To change. That is, the receiver 200 reduces the AR image. Then, the receiver 200 superimposes the AR image whose display magnification has been changed (that is, reduced) on the area and displays the AR image on the display 201. Furthermore, the receiver 200 changes the display magnification and position of the displayed AR image in accordance with the movement of the recognition area in the image sensor, that is, the movement of the target area in all captured images.

In addition, when the recognition area protrudes from the display area as described above, the receiver 200 compares the smaller pixel number of the pixel numbers dw and dh with the threshold value N. Then, for example, if the smaller number of pixels becomes equal to or less than the threshold value N at time t2, the receiver 200 is fixed without changing the display magnification and position of the AR image according to the position of the recognition area in the image sensor. To do. That is, the receiver 200 switches the screen display of the AR image. For example, the receiver 200 fixes the display magnification and position of the displayed AR image to the display magnification and position of the AR image displayed on the display 201 when the smaller number of pixels reaches the threshold value N. .

Therefore, the receiver 200 continues to display the AR image similarly to the time t2 even when the recognition area further moves at the time t3 and protrudes from the effective pixel area. That is, as long as the smaller number of pixels dw and dh is equal to or smaller than the threshold value N, the receiver 200 converts an AR image with a fixed display magnification and position into a captured display image as at time t2. Continue to overlay and display.

In the above example, the smaller of the number of pixels dw and dh is compared with the threshold value, but the ratio of the smaller number of pixels may be compared with the threshold value. The ratio of the number of pixels dw is, for example, the ratio of the number of pixels dw to the number of pixels w0 in the horizontal direction of the effective pixel region (dw / w0). Similarly, the ratio of the number of pixels dh is, for example, the ratio of the number of pixels dh to the number of pixels h0 in the vertical direction of the effective pixel region (dh / h0). Alternatively, the ratio between the pixel numbers dw and dh may be expressed by using the number of pixels in the horizontal or vertical direction of the display area instead of the number of pixels in the horizontal or vertical direction of the effective pixel area. The threshold value compared with the ratio of the number of pixels dw and dh is, for example, 0.05.

Also, the smaller angle of view of the number of pixels dw and dh may be compared with a threshold value. When the number of diagonal pixels in the effective pixel area is m and the angle of view corresponding to the diagonal is θ (for example, 55 °), the angle of view corresponding to the number of pixels dw is θ × dw / m, The angle of view corresponding to the number of pixels dh is θ × dh / m.

In the example illustrated in FIGS. 277 and 278, the receiver 200 switches the screen display of the AR image based on the inter-region distance between the effective pixel region and the recognition region. The screen display of the AR image may be switched based on the relationship.

FIG. 279 is a diagram illustrating another example in which the receiver 200 in the second modification of the twenty-third embodiment displays an AR image. Specifically, FIG. 279 illustrates an example in which the screen display of the AR image is switched based on the relationship between the display area and the recognition area. In the example shown in FIG. 279, as in the example shown in FIG. 273, the image sensor of the receiver 200 has an effective pixel area wider than the display area.

For example, the receiver 200 images the subject at time t1. As a result, the receiver 200 displays, on the display 201, only the image obtained by the display area narrower than the effective pixel area among all the captured images obtained by the effective pixel area of the image sensor as the captured display image. At this time, similarly to the above, the receiver 200 recognizes an area corresponding to the recognition information acquired based on the light ID among all the captured images as a target area on which the AR image is superimposed. Then, the receiver 200 superimposes the AR image on the target area of the captured display image, and displays the captured display image on which the AR image is superimposed on the display 201.

Here, when the user changes the direction of the receiver 200, the receiver 200 changes the position of the displayed AR image according to the movement of the recognition area in the image sensor. Then, for example, the recognition area in the image sensor moves in the upper left direction in FIG. 279, for example, and at time t2, a part of the edge of the recognition area coincides with a part of the edge of the display area. That is, an image of the target area (for example, an image such as a poster) is arranged at the corner of the captured display image displayed on the display 201. As a result, the receiver 200 superimposes the AR image on the target area at the corner of the captured display image and displays the AR image on the display 201.

When the recognition area further moves and protrudes from the display area, the receiver 200 fixes the AR image displayed at time t2 without changing the size and position. That is, the receiver 200 switches the screen display of the AR image.

Therefore, the receiver 200 continues to display the AR image similarly to the time t2 even when the recognition area further moves at the time t3 and protrudes from the effective pixel area. In other words, as long as the recognition area extends beyond the display area, the receiver 200 superimposes the AR image having the same size as that at time t2 on the same position as at time t2 in the captured display image. Continue to display.

Thus, in the example shown in FIG. 279, the receiver 200 switches the screen display of the AR image depending on whether or not the recognition area protrudes from the display area. Further, the receiver 200 may include a determination area that includes the display area and is larger than the display area and smaller than the effective pixel area instead of the display area. In this case, the receiver 200 switches the screen display of the AR image depending on whether or not the recognition area protrudes from the determination area.

As described above, the screen display of the AR image has been described with reference to FIGS. 273 to 279. However, when the receiver 200 cannot recognize the target region from all the captured images, the target that has been recognized until immediately before the target region is recognized. The AR image having the size of the area may be displayed superimposed on the captured display image.

FIG. 280 is a diagram illustrating another example in which the receiver 200 in the second modification of the twenty-third embodiment displays an AR image.

In the example illustrated in FIG. 243, the receiver 200 captures the captured display image Pe and the decoding image in the same manner as described above by capturing an image of the guide plate 107 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the guide plate 107. However, if the entire surface of the guide plate 107 is a color that absorbs light (for example, dark color), the receiver 200 receives the light ID correctly because the surface is dark even when illuminated by the transmitter 100. It may not be possible. Alternatively, even if the entire surface of the guide plate 107 has a striped pattern such as a decoding image (that is, a bright line image), the receiver 200 may not be able to correctly receive the light ID.

Therefore, as shown in FIG. 280, a reflecting plate 109 may be disposed near the guide plate 107. Thereby, the receiver 200 can receive light reflected from the transmitter 100 by the reflector 109, that is, visible light (specifically, light ID) transmitted from the transmitter 100. As a result, the receiver 200 can appropriately receive the optical ID and display the AR image P5.

[Summary of Modifications 1 and 2 of Embodiment 23]
FIG. 281A is a flowchart illustrating a display method according to one embodiment of the present invention.

The display method according to one aspect of the present invention includes steps S41 to S43.

In step S41, a picked-up image is acquired by picking up an image of an object illuminated by a transmitter that transmits a signal according to a change in luminance of light as a subject with an image sensor. In step S42, a signal is decoded from the captured image. In step S43, a moving image corresponding to the decoded signal is read from the memory, and the moving image is superimposed on a target area corresponding to the subject in the captured image and displayed on the display. Here, in step S43, a predetermined image that is one of a plurality of images included in the moving image and includes display images of the image including the target object and the image including the target object. The moving image is displayed from any one of the plurality of images. For example, the predetermined number is 10 frames. Alternatively, the object is a still image, and in step S43, the moving image is displayed from the same image as the still image. The image from which the display of the moving image is started is not limited to the same image as the still image, but is the same number as the still image, that is, the predetermined number of frames in the display order from the image including the target object. It may be an image. Further, the object is not limited to a still image, and may be a doll or the like.

Note that the imaging sensor and the captured image are, for example, the image sensor and the entire captured image in the twenty-third embodiment. The still image to be lit up may be a still image displayed on the display panel of the image display device, or may be a poster, a guide board, a signboard, or the like illuminated by light from the transmitter.

Further, such a display method may further include a transmission step of transmitting a signal to the server and a reception step of receiving a moving image corresponding to the signal from the server.

Thereby, as shown in FIG. 265, for example, a moving image can be displayed virtually so that a still image starts to move, and an image useful to the user can be displayed.

Further, the still image has an outer frame of a predetermined color, and the display method according to one aspect of the present invention may further include a recognition step of recognizing a target area from the captured image by the predetermined color. In this case, in step S43, the moving image is resized so as to be the same as the size of the recognized target region, and the resized moving image is superimposed on the target region in the captured image and displayed on the display. Good. For example, the outer frame of the predetermined color is a white or black rectangular frame surrounding a still image, and is indicated by the recognition information in the twenty-third embodiment. Then, the AR image in the twenty-third embodiment is resized and superimposed as a moving image.

Thereby, the moving image can be displayed more realistically so that the moving image actually exists as a subject.

Further, only the image projected on the display area which is smaller than the imaging area among the imaging areas of the imaging sensor is displayed on the display. In this case, in step S43, if the projection area onto which the subject is projected in the imaging area is larger than the display area, an image obtained by a portion of the projection area that exceeds the display area is displayed on the display. You don't have to. Here, for example, as shown in FIG. 273, the imaging region and the projection region are an effective pixel region and a recognition region of the image sensor.

Thus, for example, as shown in FIG. 273, the image sensor approaches the still image that is the subject, so that the subject can be displayed even if a part of the image obtained by the projection area (recognition area in FIG. 273) is not displayed on the display. In some cases, the entire still image is projected onto the imaging region. Therefore, in this case, a still image that is a subject can be appropriately recognized, and a moving image can be appropriately superimposed on a target region corresponding to the subject in the captured image.

For example, the horizontal and vertical widths of the display area are w1 and h1, and the horizontal and vertical widths of the projection area are w2 and h2. In this case, in step S43, when h2 / h1 or w2 / w1 is greater than a predetermined value, a moving image is displayed on the entire screen of the display, and either h2 / h1 or w2 / w1 is displayed. If the larger value is smaller than the predetermined value, the moving image may be superimposed on the target area in the captured image and displayed on the display.

Thus, for example, as shown in FIG. 275, when the imaging sensor approaches the still image that is the subject, the moving image is displayed on the entire screen, so that the user enlarges the moving image by moving the imaging sensor closer to the still image. There is no need to display. For this reason, it is possible to prevent the signal from being unable to be decoded when the imaging sensor is too close to the still image and the projection area (the recognition area in FIG. 275) protrudes from the imaging area (effective pixel area). .

In addition, the display method according to one aspect of the present invention may further include a control step of turning off the operation of the imaging sensor when a moving image is displayed on the entire screen of the display.

Thereby, for example, as shown in step S314 in FIG. 276, the power consumption of the image sensor can be suppressed by turning off the operation of the image sensor.

In step S43, if the target area cannot be recognized from the captured image due to the movement of the imaging sensor, the moving image is displayed in the same size as the size of the target area recognized immediately before it cannot be recognized. May be. Note that the target area cannot be recognized from the captured image, for example, is a situation in which at least a part of the target area corresponding to the still image that is the subject is not included in the captured image. Thus, when the target area cannot be recognized, a moving image having the same size as the size of the target area recognized immediately before is displayed, for example, at time t3 in FIG. Accordingly, it is possible to suppress the at least part of the moving image from being displayed because the imaging sensor has been moved.

Further, in step S43, when only a part of the target area is included in the area displayed on the display of the captured image due to the movement of the imaging sensor, it corresponds to a part of the target area. A part of the spatial area of the moving image to be displayed may be superimposed on a part of the target area and displayed on the display. A part of the spatial area of the moving image is a part of each picture constituting the moving image.

Thus, only a part of the space area of the moving image (AR image in FIG. 277) is displayed on the display, for example, at time t2 in FIG. 277. As a result, the user can be informed that the image sensor is not properly directed to the still image that is the subject.

In step S43, if the target area cannot be recognized from the captured image due to the movement of the imaging sensor, the space of the moving image corresponding to a part of the target area displayed immediately before the target area cannot be recognized. A part of the area may be continuously displayed.

As a result, even when the user points the imaging sensor in a direction different from the still image that is the subject, for example, at time t3 in FIG. 277, one space area of the moving image (AR image in FIG. 277) is displayed. The part is displayed continuously. As a result, it is possible to make it easier for the user to know how the image sensor is directed to display the entire moving image.

In step S43, the horizontal and vertical widths in the imaging area of the imaging sensor are w0 and h0, respectively, and the horizontal area between the projection area where the subject is projected in the imaging area and the imaging area. When the distances in the direction and the vertical direction are dh and dw, it is determined that the target region cannot be recognized when the smaller one of dw / w0 and dh / h0 is equal to or smaller than a predetermined value. Good. The projection area is a recognition area shown in FIG. 277, for example. Alternatively, in step S43, the angle of view corresponding to the shorter one of the horizontal and vertical distances between the projection area where the subject is projected in the imaging area of the imaging sensor and the imaging area is predetermined. If the value is less than or equal to the value, it may be determined that the target area cannot be recognized.

This makes it possible to appropriately determine whether or not the target area can be recognized.

FIG. 281B is a block diagram illustrating a structure of a display device according to one embodiment of the present invention.

The display device A10 according to an aspect of the present invention includes an imaging sensor A11, a decoding unit A12, and a display control unit A13.

The imaging sensor A11 acquires a captured image by capturing a still image illuminated by a transmitter that transmits a signal according to a change in the luminance of light as a subject.

The decoding unit A12 is a decoding unit that decodes a signal from the captured image.

The display control unit A13 reads out the moving image corresponding to the decoded signal from the memory, and displays the moving image on the display by superimposing the moving image on the target area corresponding to the subject in the captured image. Here, the display control unit A13 displays the plurality of images in order from the first image that is the same image as the still image among the plurality of images included in the moving image.

Thereby, the same effect as the above-described display method can be obtained.

Further, the imaging sensor A11 may include a plurality of micromirrors and a photosensor, and the display device A10 may further include an imaging control unit that controls the imaging sensor. In this case, the imaging control unit identifies a region including a signal as a signal region in the captured image, and controls the angle of the micromirror corresponding to the identified signal region among the plurality of micromirrors. And an imaging control part makes the above-mentioned photosensor receive only the reflected light by the micro mirror by which the angle was controlled among a plurality of micro mirrors.

Thus, for example, as shown in FIG. 232A, even if a visible light signal, which is a signal represented by a change in light brightness, contains a high-frequency component, the high-frequency component can be correctly decoded.

In each of the above-described embodiments and modifications, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. For example, the program causes the computer to execute the display method shown by the flowcharts of FIGS. 271, 274, 276, and 281A.

As described above, the display method according to one or a plurality of aspects has been described based on the above-described embodiments and modifications. However, the present invention is not limited to the embodiments. Unless it deviates from the gist of the present invention, various modifications conceived by those skilled in the art have been made in the present embodiment, and forms constructed by combining components in different embodiments and modifications are also within the scope of the present invention. May be included.

[Modification 3 of Embodiment 23]
Hereinafter, a third modification of the twenty-third embodiment, that is, a third modification of the display method for realizing the AR using the optical ID will be described.

FIG. 282 is a diagram illustrating an example of expansion and movement of the AR image.

As shown in FIG. 282 (a), the receiver 200 superimposes the AR image P21 on the target area of the captured display image Ppre, as in the case of the twenty-third embodiment or the first or second modification thereof. Then, the receiver 200 displays the captured display image Ppre on which the AR image P21 is superimposed on the display 201. For example, the AR image P21 is a moving image.

Here, when receiving an instruction to change the size, the receiver 200 changes the size of the AR image P21 in accordance with the instruction, as shown in FIG. 282 (b). For example, when receiving an enlargement instruction, the receiver 200 enlarges the AR image P21 according to the instruction. The size change instruction is given by, for example, a pinch operation, a double tap, or a long press on the AR image P21 by the user. Specifically, when receiving an enlargement instruction performed by pinching out, the receiver 200 enlarges the AR image P21 according to the instruction. Conversely, when receiving an instruction for reduction performed by pinch-in, the receiver 200 reduces the AR image P21 in accordance with the instruction.

Further, as shown in FIG. 282 (c), the receiver 200 receives the position change instruction and changes the position of the AR image P21 according to the instruction. The instruction to change the position is given by, for example, swiping the AR image by the user. Specifically, when receiving an instruction to change the position performed by swiping, the receiver 200 changes the position of the AR image P21 according to the instruction. That is, the AR image P21 moves.

This makes it possible to make the AR image easier to see by enlarging the AR image that is a moving image, and to reduce the area of the captured display image Pre that is hidden in the AR image by reducing or moving the AR image that is the moving image. Can be displayed to the user.

FIG. 283 is a diagram illustrating an example of enlargement of the AR image.

As shown in FIG. 283 (a), the receiver 200 superimposes the AR image P22 on the target area of the captured display image Ppre, as in the twenty-third embodiment or the first or second modification thereof. Then, the receiver 200 displays the captured display image Ppre on which the AR image P22 is superimposed on the display 201. For example, the AR image P22 is a still image in which a character string is described.

Here, when receiving an instruction to change the size, the receiver 200 changes the size of the AR image P22 in accordance with the instruction, as shown in FIG. 283 (b). For example, when receiving an enlargement instruction, the receiver 200 enlarges the AR image P22 according to the instruction. The size change instruction is performed by, for example, a pinch operation, a double tap, or a long press on the AR image P22 by the user, as described above. Specifically, when receiving an enlargement instruction performed by pinching out, the receiver 200 enlarges the AR image P22 according to the instruction. By enlarging the AR image P22, the character string described in the AR image P22 can be easily read by the user.

Further, as illustrated in (c) of FIG. 283, when the receiver 200 further receives a size change instruction, the receiver 200 changes the size of the AR image P22 according to the instruction. For example, when receiving a further enlargement instruction, the receiver 200 further enlarges the AR image P22 according to the instruction. By enlarging the AR image P22, it is possible to make it easier for the user to read the character string described in the AR image P22.

Note that, when receiving an enlargement instruction, the receiver 200 may acquire a high-resolution AR image if the enlargement ratio of the AR image corresponding to the instruction is equal to or greater than a threshold value. In this case, the receiver 200 may enlarge and display the high-resolution AR image up to the above-described enlargement factor instead of the original AR image that has already been displayed. For example, the receiver 200 displays an AR image of 1920 × 1080 pixels instead of the AR image of 640 × 480 pixels. As a result, the AR image can be enlarged and a high-resolution image that cannot be obtained by the optical zoom can be displayed so that the AR image is actually captured as a subject.

FIG. 284 is a flowchart illustrating an example of processing operations related to enlargement and movement of the AR image by the receiver 200.

First, the receiver 200 starts imaging based on the normal exposure time and the communication exposure time, similarly to step S101 shown in the flowchart of FIG. 239 (step S401). When this imaging is started, a captured display image Ppre based on the normal exposure time and a decoding image (that is, a bright line image) Pdec based on the communication exposure time are periodically obtained. Then, the receiver 200 acquires the optical ID by decoding the decoding image Pdec.

Next, the receiver 200 performs an AR image superimposition process including the processes of steps S102 to S106 shown in the flowchart of FIG. 239 (step S402). When this AR image superimposition process is performed, the AR image is displayed superimposed on the captured display image Ppre. At this time, the receiver 200 decreases the optical ID acquisition rate (step S403). The light ID acquisition rate is a ratio of the number of decoding images (that is, bright line images) Pdec out of the number of captured images per unit time obtained by imaging started in step S401. For example, as the optical ID acquisition rate decreases, the number of decoding images Pdec obtained per unit time becomes smaller than the number of captured display images Ppre obtained per unit time.

Next, the receiver 200 determines whether or not a size change instruction has been received (step S404). If it is determined that the size change instruction has been received (Yes in step S404), the receiver 200 further determines whether the size change instruction is an enlargement instruction (step S405). If it is determined that the size change instruction is an enlargement instruction (Yes in step S405), the receiver 200 further determines whether it is necessary to reacquire the AR image (step S406). For example, when the receiver 200 determines that the AR image enlargement rate according to the enlargement instruction is equal to or greater than a threshold, the receiver 200 determines that the AR image needs to be reacquired. When the receiver 200 determines that reacquisition is necessary (Yes in step S406), the receiver 200 acquires a high-resolution AR image from, for example, a server, and converts the AR image displayed in a superimposed manner into the high-resolution AR image. Replace with the AR image (step S407).

Then, the receiver 200 changes the size of the AR image in accordance with the received size change instruction (step S408). That is, when a high-resolution AR image is acquired in step S407, the receiver 200 enlarges the high-resolution AR image. Further, when it is determined in step S406 that the re-acquisition of the AR image is unnecessary (No in step S406), the receiver 200 enlarges the superimposed AR image. If it is determined in step S405 that the size change instruction is a reduction instruction (No in step S405), the receiver 200 performs superimposition according to the received size change instruction, that is, the reduction instruction. Reduce the displayed AR image.

On the other hand, when the receiver 200 determines in step S404 that the size change instruction has not been received (No in step S404), the receiver 200 determines whether a position change instruction has been received (step S409). If it is determined that a position change instruction has been received (Yes in step S409), the receiver 200 changes the position of the superimposed AR image in accordance with the position change instruction (step S410). ). That is, the receiver 200 moves the AR image. If it is determined that the position change instruction has not been received (No in step S409), the receiver 200 repeatedly executes the processing from step S404.

When the size of the AR image is changed in step S408 or the position of the AR image is changed in step S410, the receiver 200 cannot acquire the light ID periodically acquired from step S401. It is determined whether or not (step S411). If it is determined that the light ID is no longer acquired (Yes in step S411), the receiver 200 ends the processing operation related to the expansion and movement of the AR image. On the other hand, if it is determined that the light ID is still acquired (No in step S411), the receiver 200 repeatedly executes the processing from step S404.

FIG. 285 is a diagram illustrating an example of superimposition of the AR image by the receiver 200.

As described above, the receiver 200 superimposes the AR image P23 on the target area in the captured display image Ppre. Here, as shown in FIG. 285, the AR image P23 is configured such that the transmittance of each part of the AR image P23 is higher as the part is closer to the end of the AR image P23. The transmittance is the degree to which the superimposed image is displayed through. For example, when the overall transmittance of the AR image is 100%, even if the AR image is superimposed on the target area of the captured display image, only the target area is displayed on the display 201 without displaying the AR image. Means that. Conversely, the transmittance of the entire AR image being 0% means that the target area of the captured display image is not displayed on the display 201 and only the AR image superimposed on the target area is displayed. .

For example, when the AR image P23 is rectangular, the transmittance of each part in the AR image P23 is higher as the part is closer to the upper end, lower end, left end, or right end of the rectangle. More specifically, the transmittance at those ends is 100%. In addition, in the central portion of the AR image P23, there is a rectangular area having a transmittance of 0% which is smaller than that of the AR image P23. In the rectangular area, for example, “Kyoto Station” is written in English. That is, at the peripheral edge of the AR image P23, the transmittance changes stepwise from 0% to 100% like a gradation.

The receiver 200 superimposes such an AR image P23 on the target area in the captured display image Pre as shown in FIG. At this time, the receiver 200 matches the size of the AR image P23 with the size of the target area, and superimposes the resized AR image P23 on the target area. For example, an image of a station name sign having the same background color as that of the rectangular area in the center of the AR image P23 appears in the target area. The station name has “Kyoto” written in Japanese.

Here, as described above, the transmittance of each part of the AR image P23 is higher as the part is closer to the end of the AR image P23. Therefore, when the AR image P23 is superimposed on the target area, even if the rectangular area at the center of the AR image P23 is displayed, the end of the AR image P23 is not displayed, but the end of the target area, that is, the station name mark The edge of the image is displayed.

Thereby, the deviation between the AR image P23 and the target area can be made inconspicuous. That is, even when the AR image P23 is superimposed on the target area, a shift may occur between the AR image P23 and the target area due to the movement of the receiver 200 or the like. In this case, if the overall transmittance of the AR image P23 is 0%, the end of the AR image P23 and the end of the target area are displayed, and the shift becomes conspicuous. However, in the AR image P23 in the present modification, the closer to the end, the higher the transmittance of the part, so that the end of the AR image P23 can be made difficult to be displayed. As a result, the AR image P23 and the target region It is possible to make the gap inconspicuous. Furthermore, since the transmittance changes like gradation in the peripheral portion of the AR image P23, it is difficult to notice that the AR image P23 is superimposed on the target region.

FIG. 286 is a diagram illustrating an example of superimposition of the AR image by the receiver 200.

As described above, the receiver 200 superimposes the AR image P24 on the target area in the captured display image Ppre. Here, as shown in FIG. 286, the subject to be imaged is, for example, a restaurant menu. This menu is surrounded by a white frame, and the white frame is surrounded by a black frame. That is, the subject includes a menu, a white frame surrounding the menu, and a black frame surrounding the white frame.

The receiver 200 recognizes, as a target area, an area larger than the white frame image and smaller than the black frame image in the captured display image Pre. Then, the receiver 200 matches the size of the AR image P24 with the size of the target area, and superimposes the resized AR image P24 on the target area.

Thereby, even when the superimposed AR image P24 is displaced from the target area due to the movement of the receiver 200, the AR image P24 can be continuously displayed in a state surrounded by a black frame. Therefore, the shift between the AR image P24 and the target region can be made inconspicuous.

In the example shown in FIG. 286, the color of the frame is black or white, but is not limited to these colors, and may be any color.

FIG. 287 is a diagram illustrating an example of superimposition of the AR image by the receiver 200.

For example, the receiver 200 images a poster on which a castle illuminated in the night sky is drawn as a subject. For example, the poster is illuminated by the above-described transmitter 100 configured as a backlight, and a visible light signal (ie, a light ID) is transmitted by the backlight. The receiver 200 acquires the captured display image Ppre including the image of the subject that is the poster and the AR image P25 corresponding to the light ID by the imaging. Here, the AR image P25 has the same shape as the poster image from which the region where the castle is drawn is extracted. That is, the area corresponding to the castle of the poster image in the AR image P25 is masked. Further, the AR image P25 is configured so that the transmittance of each part of the AR image P25 is higher as the part of the AR image P25 is closer to the end of the AR image P25, similarly to the AR image P23 described above. In the central portion where the transmittance is 0% in the AR image P25, fireworks launched in the night sky are displayed as moving images.

The receiver 200 matches the size of the AR image P25 with the size of the target area that is the image of the subject, and superimposes the resized AR image P25 on the target area. As a result, the castle drawn on the poster is displayed as an image of the subject, not as an AR image, and a moving image of fireworks is displayed as an AR image.

Thereby, the captured display image Ppre can be displayed as if fireworks are actually being launched in the poster. Further, the transmittance of each part of the AR image P25 is higher as the part is closer to the end of the AR image P25. Therefore, when the AR image P25 is superimposed on the target area, even if the center portion of the AR image P25 is displayed, the end of the AR image P25 is not displayed, but the end of the target area is displayed. As a result, the deviation between the AR image P25 and the target region can be made inconspicuous. Furthermore, since the transmittance changes like a gradation at the peripheral portion of the AR image P25, it is difficult to notice that the AR image P25 is superimposed on the target region.

FIG. 288 is a diagram illustrating an example of superimposition of the AR image by the receiver 200.

For example, the receiver 200 images the transmitter 100 configured as a television as a subject. Specifically, the transmitter 100 displays a castle illuminated in the night sky on a display and transmits a visible light signal (that is, a light ID). The receiver 200 acquires the captured display image Ppre displayed by the transmitter 100 and the AR image P26 corresponding to the optical ID by the imaging. Here, the receiver 200 first displays the captured display image Ppre on the display 201. At this time, the receiver 200 also displays on the display 201 a message m that prompts the user to turn it off. Specifically, the message m is, for example, “Turn off the room lighting and darken the room”.

When the message m is displayed and the user is turned off and the room in which the transmitter 100 is installed becomes dark, the receiver 200 displays the AR image P26 superimposed on the captured display image Ppre. Here, the AR image P26 has the same size as the captured display image Ppre, and the area corresponding to the castle of the captured display image Ppre in the AR image P26 is cut out. That is, the area corresponding to the castle of the captured display image Ppre in the AR image P26 is masked. Therefore, the castle of the captured display image Ppre can be shown to the user from the area. Further, in the peripheral portion of the area in the AR image P26, similarly to the above, the transmittance may change stepwise from 0% to 100% like a gradation. In this case, the shift between the captured display image Ppre and the AR image P26 can be made inconspicuous.

In the above-described example, the AR image having a high peripheral edge transmittance is superimposed on the target area of the captured display image Ppre, so that the shift between the AR image and the target area is less noticeable. However, instead of such an AR image, an AR image that is the same size as the captured display image Ppre and is entirely translucent (that is, having a transmittance of 50%) may be superimposed on the captured display image Ppre. Even in this case, the shift between the AR image and the target region can be made inconspicuous. Further, when the captured display image Ppre is generally bright, the AR image having a uniform low transparency is superimposed on the captured display image Ppre. Conversely, when the captured display image Ppre is generally dark, the transparency is uniformly uniform. A high AR image may be superimposed on the captured display image Ppre.

It should be noted that objects such as the fireworks of the AR image P25 and the AR image P26 may be expressed by CG (computer graphics). In this case, masking can be eliminated. In the example illustrated in FIG. 288, the receiver 200 displays the message m that prompts the user to turn off the light, but the light may be automatically turned off without performing such display. For example, the receiver 200 outputs a turn-off signal to the lighting device in which the transmitter 100 that is a television is set by Bluetooth (registered trademark), ZigBee, a specific low-power radio station, or the like. Thereby, the lighting device is automatically turned off.

FIG. 289A is a diagram illustrating an example of a captured display image Ppre obtained by imaging by the receiver 200.

For example, the transmitter 100 is configured as a large display installed in a stadium. Then, the transmitter 100 displays a message indicating that, for example, fast food and drinks can be ordered with the light ID, and transmits a visible light signal (that is, a light ID). When such a message is displayed, the user images the receiver 200 toward the transmitter 100. That is, the receiver 200 images the transmitter 100 configured as a large display installed in the stadium as a subject.

The receiver 200 acquires the captured display image Ppre and the decoding image Pdec by the imaging. Then, the receiver 200 acquires the optical ID by decoding the decoding image Pdec, and transmits the optical ID and the captured display image Ppre to the server.

For each light ID, the server specifies the installation information of the captured large display associated with the light ID transmitted from the receiver 200 from the installation information associated with the light ID. For example, the installation information indicates the position and orientation where the large display is installed, the size of the large display, and the like. Further, the server identifies the seat number where the captured display image Ppre was captured in the stadium based on the size and orientation of the large display displayed in the captured display image Ppre and the installation information. To do. Then, the server causes the receiver 200 to display a menu screen including the seat number.

FIG. 289B is a diagram illustrating an example of a menu screen displayed on the display 201 of the receiver 200.

The menu screen m1 includes, for example, for each product, an input field ma1 in which the number of orders for the product is input, a seat field mb1 in which the seat number of the stadium specified by the server is described, and an order button mc1. . The user operates the receiver 200 to input the order quantity of the product in the input field ma1 corresponding to the desired product, and selects the order button mc1. As a result, the order is confirmed, and the receiver 200 transmits the order contents corresponding to the input result to the server.

When the server receives the order details, it instructs the stadium staff to deliver the number of products according to the order details to the seat of the number specified as described above.

FIG. 290 is a flowchart illustrating an example of processing operation between the receiver 200 and the server.

The receiver 200 first images the transmitter 100 configured as a large stadium display (step S421). The receiver 200 acquires the optical ID transmitted from the transmitter 100 by decoding the decoding image Pdec obtained by the imaging (step S422). The receiver 200 transmits the optical ID acquired in step S422 and the captured display image Ppre obtained by imaging in step S421 to the server (step S423).

When the server receives the light ID and the captured display image Pre (step S424), the server identifies installation information of a large display installed in the stadium based on the light ID (step S425). For example, for each light ID, the server holds a table indicating installation information of a large display associated with the light ID, and the installation information associated with the light ID transmitted from the receiver 200 is stored from the table. The installation information is specified by searching.

Next, the server acquires (that is, captures) the captured display image Ppre in the stadium based on the specified installation information and the size and orientation of the large display displayed in the captured display image Ppre. The assigned seat number is specified (step S426). Then, the server transmits the URL (Uniform Resource Locator) of the menu screen m1 including the identified seat number to the receiver 200 (step S427).

When the receiver 200 receives the URL of the menu screen m1 transmitted from the server (step S428), the receiver 200 accesses the URL and displays the menu screen m1 (step S429). Here, the user operates the receiver 200 to input the order contents into the menu screen m1, and selects the order button mc1, thereby confirming the order. Thereby, the receiver 200 transmits the order details to the server (step S430).

Upon receiving the order details transmitted from the receiver 200, the server performs an order receiving process according to the order details (step S431). At this time, for example, the server instructs the staff of the stadium to deliver the number of products corresponding to the order contents to the seat of the number specified in step S426.

Thus, since the seat number is specified based on the captured display image Ppre obtained by imaging by the receiver 200, the user of the receiver 200 bothers to input the seat number when ordering a product. There is no need to do. Therefore, the user can easily place an order for a product without inputting the seat number.

In the above example, the server specifies the seat number, but the receiver 200 may specify the seat number. In this case, the receiver 200 acquires the installation information from the server, and specifies the seat number based on the installation information and the size and orientation of the large display displayed in the captured display image Pre.

FIG. 291 is a diagram for explaining the volume of sound reproduced by the receiver 1800a.

The receiver 1800a receives the light ID (visible light signal) transmitted from the transmitter 1800b configured as, for example, street digital signage, similarly to the example shown in FIG. Then, the receiver 1800a reproduces sound at the same timing as the image reproduction by the transmitter 1800b. That is, the receiver 1800a reproduces sound so as to be synchronized with the image reproduced by the transmitter 1800b. Note that the receiver 1800a may reproduce the same image as the image reproduced by the transmitter 1800b (reproduced image) or an AR image (AR moving image) related to the reproduced image together with the sound.

Here, when reproducing the sound as described above, the receiver 1800a adjusts the volume of the sound according to the distance to the transmitter 1800b. Specifically, the receiver 1800a adjusts the volume smaller as the distance to the transmitter 1800b is longer, and conversely adjusts the volume larger as the distance to the transmitter 1800b is shorter.

The receiver 1800a may specify the distance to the transmitter 1800b using a GPS (Global Positioning System) or the like. Specifically, the receiver 1800a acquires position information of the transmitter 1800b associated with the optical ID from a server or the like, and further specifies the position of the receiver 1800a by GPS. Then, the receiver 1800a specifies the distance between the position of the transmitter 1800b indicated by the position information acquired from the server and the position of the specified receiver 1800a as the distance to the above-described transmitter 1800b. . Note that the receiver 1800a may specify the distance to the transmitter 1800b by using Bluetooth (registered trademark) instead of GPS.

Further, the receiver 1800a may specify the distance to the transmitter 1800b based on the size of the bright line pattern region of the above-described decoding image Pdec obtained by imaging. Similar to the examples shown in FIGS. 245 and 246, the bright line pattern region is a region composed of a plurality of bright line patterns that appear by exposure at a communication exposure time of a plurality of exposure lines included in the image sensor of the receiver 1800 a. This bright line pattern area corresponds to the display area of the transmitter 1800b displayed in the captured display image Ppre. Specifically, the receiver 1800a specifies the shorter distance as the distance to the transmitter 1800b as the bright line pattern region is larger, and conversely specifies the longer distance as the distance to the transmitter 1800b as the bright line pattern region is smaller. . The receiver 1800a uses distance data indicating the relationship between the size of the bright line pattern region and the distance, and in the distance data, the distance associated with the size of the bright line pattern region in the captured display image Pre is The distance to the transmitter 1800b may be specified. Note that the receiver 1800a may transmit the optical ID received as described above to the server, and obtain distance data associated with the optical ID from the server.

As described above, since the volume is adjusted according to the distance to the transmitter 1800b, the user of the receiver 1800a makes the sound reproduced by the receiver 1800a like the sound actually reproduced by the transmitter 1800b. Can be heard.

FIG. 292 is a diagram showing the relationship between the distance from the receiver 1800a to the transmitter 1800b and the sound volume.

For example, when the distance to the transmitter 1800b is between L1 and L2 [m], the volume increases or decreases in proportion to the distance in the range from Vmin to Vmax [dB]. Specifically, the receiver 1800a linearly decreases the volume from Vmax [dB] to Vmin [dB] when the distance to the transmitter 1800b increases from L1 [m] to L2 [m]. Even if the distance to the transmitter 1800b is shorter than L1 [m], the receiver 1800a maintains the volume at Vmax [dB], and the distance to the transmitter 1800b is longer than L2 [m]. However, the volume is maintained at Vmin [dB].

As described above, the receiver 1800a stores the maximum volume Vmax, the longest distance L1 at which the sound of the maximum volume Vmax is output, the minimum volume Vmin, and the shortest distance L2 at which the sound of the minimum volume Vmin is output. is doing. The receiver 1800a may change the maximum volume Vmax, the minimum volume Vmin, the longest distance L1, and the shortest distance L2 according to the attributes set for the receiver 1800a. For example, when the attribute is the age of the user and the age indicates a high age, the receiver 1800a sets the maximum volume Vmax higher than the reference maximum volume and sets the minimum volume Vmin higher than the reference minimum volume. Also good. Further, the attribute may be information indicating whether audio output is performed from a speaker or an earphone.

As described above, since the minimum volume Vmin is set in the receiver 1800a, it is possible to prevent the receiver 1800a from being inaudible because the receiver 1800a is too far from the transmitter 1800b. Furthermore, since the maximum volume Vmax is set in the receiver 1800a, the receiver 1800a is too close to the transmitter 1800b, so that it is possible to suppress an excessively loud sound from being output.

FIG. 293 is a diagram illustrating an example of AR image superimposition performed by the receiver 200.

The receiver 200 images the illuminated signboard. Here, the signboard is lit up by the illumination device which is the above-described transmitter 100 that transmits the optical ID. Therefore, the receiver 200 acquires the captured display image Ppre and the decoding image Pdec by the imaging. Then, the receiver 200 acquires the optical ID by decoding the decoding image Pdec, and acquires a plurality of AR images P27a to P27c associated with the optical ID and the recognition information from the server. Based on the recognition information, the receiver 200 recognizes the periphery of the area m2 in which the signboard is displayed in the captured display image Ppre as a target area.

Specifically, as shown in FIG. 293 (a), the receiver 200 recognizes an area in contact with the left side of the area m2 as the first target area, and superimposes the AR image P27a on the first target area. To do.

Next, as illustrated in FIG. 293 (b), the receiver 200 recognizes an area including the lower side of the area m2 as the second target area, and superimposes the AR image P27b on the second target area. .

Next, as shown in FIG. 293 (c), the receiver 200 recognizes an area in contact with the upper side of the area m2 as the third target area, and superimposes the AR image P27c on the third target area.

Here, each of the AR images P27a to P27c is, for example, an image of a snowman character and may be a moving image.

Further, the receiver 200 switches the recognized target area to any one of the first to third target areas in a predetermined order and timing while continuously acquiring the optical ID. May be. That is, the receiver 200 may switch the recognized target area in the order of the first target area, the second target area, and the third target area. Alternatively, the receiver 200 may switch the recognized target area to any one of the first to third target areas in a predetermined order each time the above-described optical ID is acquired. That is, the receiver 200 first acquires the light ID, and while continuously acquiring the light ID, the receiver 200 recognizes the first target area as illustrated in FIG. Then, the AR image P27a is superimposed on the first target area. Then, when the receiver 200 cannot acquire the optical ID, the receiver 200 hides the AR image P27a. Next, when the receiver 200 acquires the light ID again, the receiver 200 recognizes the second target area as shown in FIG. 293 (b) while continuously acquiring the light ID. Then, the AR image P27b is superimposed on the second target area. Then, when the receiver 200 cannot acquire the optical ID again, the receiver 200 hides the AR image P27b. Next, when the receiver 200 acquires the light ID again, the receiver 200 recognizes the third target area as shown in FIG. 293 (c) while continuously acquiring the light ID. Then, the AR image P27c is superimposed on the third target area.

In this way, when the target area to be recognized is switched every time the light ID is acquired, the receiver 200 displays the AR image displayed once every N (N is an integer of 2 or more) times. The color may be changed. N times is the number of times the AR image is displayed, and may be 200 times, for example. That is, the AR images P27a to P27c are images of the same white character, but an AR image of a pink character, for example, is displayed at a frequency of once every 200 times. When the AR image of the pink character is displayed, the receiver 200 may give points to the user when receiving an operation on the AR image by the user.

In this way, by switching the target area on which the AR image is superimposed or changing the color of the AR image at a predetermined frequency, the user's interest can be directed to the imaging of the signboard lit up by the transmitter 100. The user can repeatedly obtain the optical ID.

FIG. 294 is a diagram illustrating an example of AR image superimposition performed by the receiver 200.

The receiver 200 functions as a so-called way finder (Way Finder) that presents a route to be followed by the user, for example, by imaging the mark M4 drawn on the floor surface at a position where a plurality of passages intersect in the building. Have The building is a hotel, for example, and the presented route is a route where the user who has checked in heads to his / her room.

The mark M4 is lit up by the illumination device that is the above-described transmitter 100 that transmits the light ID by a change in luminance. Therefore, the receiver 200 acquires the captured display image Ppre and the decoding image Pdec by capturing the mark M4. Then, the receiver 200 acquires the optical ID by decoding the decoding image Pdec, and transmits the optical ID and the terminal information of the receiver 200 to the server. The receiver 200 acquires a plurality of AR images P28 and recognition information associated with the optical ID and terminal information from the server. The optical ID and terminal information are stored in the server in association with a plurality of AR images P28 and recognition information at the time of user check-in.

Based on the recognition information, the receiver 200 recognizes a plurality of target areas around the area m4 in which the mark M4 is displayed in the captured display image Ppre. Then, as shown in FIG. 294, the receiver 200 displays an AR image P28 such as an animal footprint superimposed on each of the plurality of target regions.

Specifically, the recognition information indicates the course of turning to the right at the position of the mark M4. Based on such recognition information, the receiver 200 identifies a route in the captured display image Ppre and recognizes a plurality of target regions arranged along the route. This route is a route that goes from the lower side of the display 201 to the region m4 and turns right in the region m4. The receiver 200 arranges the AR image P28 in each of the recognized plurality of target regions as if the animal walked along the route.

Here, when specifying the route in the captured display image Pre, the receiver 200 may use the geomagnetism detected by the 9-axis sensor provided in the receiver 200. In this case, the recognition information indicates the direction to proceed at the position of the mark M4 with reference to the direction of geomagnetism. For example, the recognition information indicates west as the direction to proceed at the position of the mark M4. Based on such recognition information, the receiver 200 specifies a route from the lower side of the display 201 toward the area m4 and toward the west in the area m4 in the captured display image Ppre. Then, the receiver 200 recognizes a plurality of target areas arranged along the route. Note that the receiver 200 identifies the lower side of the display 201 by detecting gravitational acceleration using a nine-axis sensor.

In this way, since the route of the user is presented by the receiver 200, the user can easily reach the destination by following the route. Moreover, since the course is displayed as an AR image in the captured display image Ppre, the course can be presented to the user in an easy-to-understand manner.

In addition, the illuminating device which is the transmitter 100 can appropriately transmit the light ID while suppressing the brightness by illuminating the mark M4 with a short pulse of light. In addition, the receiver 200 images the mark M4. However, the receiver 200 may image the illumination device using a camera (a so-called self-taking camera) disposed on the display 201 side. The receiver 200 may capture both the mark M4 and the illumination device.

FIG. 295 is a diagram for explaining an example of how to determine the line scan time by the receiver 200.

The receiver 200 performs decoding using the line scan time when decoding the decoding image Pdec. This line scan time is the time from the start of exposure of one exposure line included in the image sensor to the start of exposure of the next exposure line. If the line scan time is known, the receiver 200 decodes the decoding image Pdec using the known line scan time. However, when the line scan time is not known, the receiver 200 obtains the line scan time from the decoding image Pdec.

For example, as shown in FIG. 295, the receiver 200 finds a line having the minimum width from among a plurality of bright lines and a plurality of dark lines constituting the bright line pattern in the decoding image Pdec. The bright line is a line on the decoding image Pdec generated when each of one or a plurality of continuous exposure lines is exposed when the luminance of the transmitter 100 is high. Further, the dark line is a line on the decoding image Pdec generated by exposure of each of one or a plurality of continuous exposure lines when the luminance of the transmitter 100 is low.

When the receiver 200 finds the line with the minimum width, the receiver 200 specifies the number of exposure lines corresponding to the line with the minimum width, that is, the number of pixels. When the carrier frequency at which the luminance changes so that the transmitter 100 transmits the optical ID is 9.6 kHz, the time when the luminance of the transmitter 100 is high or low is 104 μs at the shortest. Therefore, the receiver 200 calculates the line scan time by dividing 104 μs by the number of pixels having the specified minimum width.

FIG. 296 is a diagram for explaining an example of how to obtain the line scan time by the receiver 200.

The receiver 200 may perform a Fourier transform on the bright line pattern of the decoding image Pdec and obtain the line scan time based on the spatial frequency obtained by the Fourier transform.

For example, as shown in FIG. 296, the receiver 200 derives a spectrum indicating the relationship between the spatial frequency and the intensity of the component of the spatial frequency in the decoding image Pdec by the Fourier transform described above. Next, the receiver 200 sequentially selects each of the plurality of peaks indicated in the spectrum. Each time the receiver 200 selects a peak, the receiver 200 calculates a line scan time such that the spatial frequency of the selected peak (for example, the spatial frequency f2 in FIG. 296) is obtained by a time frequency of 9.6 kHz. Calculated as a candidate for the can time. 9.6 kHz is the carrier frequency of the luminance change of the transmitter 100 as described above. Thereby, a plurality of line scan time candidates are calculated. The receiver 200 selects the most likely candidate among the plurality of line scan time candidates as the line scan time.

In order to select the most likely candidate, the receiver 200 calculates the allowable range of line scan time based on the frame rate in imaging and the number of exposure lines included in the image sensor. That is, the receiver 200 calculates the maximum value of the line scan time by 1 × 10 ⁶ [μs] / {(frame rate) × (number of exposure lines)}. Then, the receiver 200 determines the maximum value × constant K (K <1) to the maximum value as the allowable range of the line scan time. The constant K is, for example, 0.9 or 0.8.

The receiver 200 selects a candidate within this allowable range from among a plurality of line scan time candidates as a maximum likelihood candidate, that is, a line scan time.

Note that the receiver 200 may evaluate the reliability of the calculated line scan time depending on whether or not the line scan time calculated according to the example illustrated in FIG. 295 is within the above-described allowable range.

FIG. 297 is a flowchart showing an example of how to obtain the line scan time by the receiver 200.

The receiver 200 may obtain the line scan time by trying to decode the decoding image Pdec. Specifically, first, the receiver 200 starts imaging (step S441). Next, the receiver 200 determines whether or not the line scan time is known (step S442). For example, the receiver 200 may determine whether or not the line scan time is known by notifying the server of the type and model of the receiver 200 and inquiring the line scan time according to the type and model. . If it is determined that it is known (Yes in step S442), the receiver 200 sets the reference acquisition count of the optical ID to n (n is an integer equal to or larger than 2, for example, 4) (step S443). ). Next, the receiver 200 acquires the optical ID by decoding the decoding image Pdec using the known line scan time (step S444). At this time, the receiver 200 acquires a plurality of optical IDs by performing decoding on each of the plurality of decoding images Pdec obtained sequentially by the imaging started in step S441. Here, the receiver 200 determines whether or not the same optical ID has been acquired the reference acquisition times (that is, n times) (step S445). If it is determined that it has been acquired n times (Yes in step S445), the receiver 200 trusts the optical ID and starts processing using the optical ID (for example, superimposition of an AR image) (step S446). On the other hand, if it is determined that it has not been acquired n times (No in step S445), the receiver 200 does not trust the optical ID and ends the process.

If it is determined in step S442 that the line scan time is not known (No in step S442), the receiver 200 sets the optical ID reference acquisition count to n + k (k is an integer equal to or greater than 1) (step S447). . That is, when the line scan time is not known, the receiver 200 sets a larger reference acquisition count than when the line scan time is known. Next, the receiver 200 determines a temporary line scan time (step S448). Then, the receiver 200 acquires the optical ID by decoding the decoding image Pdec using the provisional line scan time (step S449). At this time, the receiver 200 acquires a plurality of optical IDs by performing decoding on each of the plurality of decoding images Pdec obtained sequentially by the imaging started in step S441, as described above. Here, the receiver 200 determines whether or not the same optical ID has been acquired the reference acquisition times (that is, (n + k) times) (step S450).

If it is determined that (n + k) times have been acquired (Yes in step S450), the receiver 200 determines that the provisional line scan time is the correct line scan time. Then, the receiver 200 notifies the server of the type and model of the receiver 200 and the line scan time (step S451). As a result, the server stores the type and model of the receiver in association with the line scan time suitable for the receiver. Therefore, when another receiver of the same type and type starts imaging, the other receiver can specify its own line scan time by making an inquiry to the server. That is, the other receivers can determine that the line scan time is known in the determination in step S442.

Then, the receiver 200 trusts the optical ID acquired (n + k) times, and starts processing using the optical ID (for example, superimposition of an AR image) (step S446).

If it is determined in step S450 that the same light ID has not been acquired (n + k) times (No in step S450), the receiver 200 further determines whether or not an end condition is satisfied (step S452). . The end condition is, for example, that a predetermined time has elapsed since the start of imaging, or that the optical ID has been acquired more than the maximum number of acquisitions. If it is determined that such an end condition is satisfied (Yes in step S452), the receiver 200 ends the process. On the other hand, when determining that the termination condition is not satisfied (No in step S452), the receiver 200 changes the provisional line scan time (step S453). Then, the receiver 200 repeatedly executes the processing from step S449 using the changed provisional line scan time.

Thus, even if the line scan time is not known, the receiver 200 can obtain the line scan time as in the examples shown in FIGS. 295 to 297. Accordingly, regardless of the type and model of the receiver 200, the receiver 200 can appropriately decode the decoding image Pdec and obtain an optical ID.

FIG. 298 is a diagram illustrating an example of AR image superimposition performed by the receiver 200.

The receiver 200 images the transmitter 100 configured as a television. The transmitter 100 periodically transmits an optical ID and a time code by changing luminance while displaying a television program, for example. The time code is information indicating the time at the time of transmission each time it is transmitted, and may be a time packet shown in FIG. 126, for example.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec through the above-described imaging. The receiver 200 acquires the above-described optical ID and time code by decoding the decoding image Pdec while displaying the captured display image Ppre acquired periodically on the display 201. Next, the receiver 200 transmits the optical ID to the server 300. When the server 300 receives the optical ID, the server 300 transmits the audio data associated with the optical ID, the AR start time information, the AR image P29, and the recognition information to the receiver 200.

When the receiver 200 acquires the audio data, the receiver 200 reproduces the audio data in synchronization with the video of the TV program displayed on the transmitter 100. That is, the sound data is composed of a plurality of sound unit data, and the plurality of sound unit data includes a time code. The receiver 200 starts reproduction of a plurality of audio unit data from the audio unit data including the time code indicating the same time as the time code acquired from the transmitter 100 together with the optical ID in the audio data. Thereby, the reproduction of the audio data is synchronized with the video of the television program. It should be noted that such synchronization between audio and video may be performed by a method similar to the audio synchronized playback shown in each of the drawings after FIG.

When the receiver 200 acquires the AR image P29 and the recognition information, the receiver 200 recognizes an area corresponding to the recognition information in the captured display image Ppre as a target area, and superimposes the AR image P29 on the target area. For example, the AR image P29 is an image showing a crack in the display 201 of the receiver 200, and the target area is an area that crosses the image of the transmitter 100 in the captured display image Ppre.

Here, the receiver 200 displays the captured display image Ppre on which the AR image P29 as described above is superimposed at a timing according to the AR start time information. The AR start time information is information indicating the time at which the AR image P29 is displayed. That is, the receiver 200 captures a display image in which the above-described AR image P29 is superimposed at the timing of receiving the time code indicating the same time as the AR start time information among the time codes transmitted from the transmitter 100 as needed. Display Pre. For example, the time indicated by the AR start time information is the time when a scene in which a magician girl applies ice magic appears in a television program. At this time, the receiver 200 may output from the speaker of the receiver 200 a sound in which the AR image P29 is cracked due to the reproduction of the audio data.

This allows the user to view the TV program scene with a sense of reality.

Further, the receiver 200 may vibrate a vibrator provided in the receiver 200 at a time indicated by the AR start time information, or may cause the light source to emit light like a flash, and the display 201 may be instantaneously displayed. It may be brightened or flashed. Further, the AR image P29 may include not only an image showing a crack but also an image showing a state in which condensation of the display 201 is frozen.

FIG. 299 is a diagram illustrating an example of AR image superimposition performed by the receiver 200.

The receiver 200 images the transmitter 100 configured as a toy cane, for example. The transmitter 100 includes a light source, and transmits an optical ID by changing the luminance of the light source.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec through the above-described imaging. The receiver 200 acquires the above-described optical ID by decoding the decoding image Pdec while displaying the captured display image Ppre acquired periodically on the display 201. Next, the receiver 200 transmits the optical ID to the server 300. When the server 300 receives the optical ID, the server 300 transmits the AR image P30 associated with the optical ID and the recognition information to the receiver 200.

Here, the recognition information further includes gesture information indicating a gesture (that is, an action) by a person holding the transmitter 100. The gesture information indicates, for example, a gesture in which a person moves the transmitter 100 from right to left. The receiver 200 compares the gesture by the person holding the transmitter 100 displayed on each captured display image Ppre with the gesture indicated by the gesture information. Then, when the gestures coincide with each other, the receiver 200 arranges the AR images P30 such that, for example, many star-shaped AR images P30 are arranged along the trajectory of the transmitter 100 moved by the gestures. Is superimposed on the captured display image Ppre.

FIG. 300 is a diagram illustrating an example of superimposition of AR images by the receiver 200.

The receiver 200 images the transmitter 100 configured as, for example, a toy cane, as described above.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec by the imaging. The receiver 200 acquires the above-described optical ID by decoding the decoding image Pdec while displaying the captured display image Ppre acquired periodically on the display 201. Next, the receiver 200 transmits the optical ID to the server 300. When the server 300 receives the optical ID, the server 300 transmits the AR image P31 associated with the optical ID and the recognition information to the receiver 200.

Here, the recognition information includes gesture information indicating a gesture by a person holding the transmitter 100 as described above. The gesture information indicates, for example, a gesture in which a person moves the transmitter 100 from right to left. The receiver 200 compares the gesture by the person holding the transmitter 100 displayed on each captured display image Ppre with the gesture indicated by the gesture information. Then, when the gestures match, the receiver 200, for example, in the captured display image Ppre, an AR image P30 indicating a dress costume in a target area that is an area in which a person holding the transmitter 100 is projected. Is superimposed.

As described above, in the display method in the present modification, gesture information corresponding to the light ID is acquired from the server. Next, it is determined whether or not the movement of the subject indicated by the periodically acquired captured display image matches the movement indicated by the gesture information acquired from the server. And when it determines with matching, the picked-up display image Ppre on which AR image was superimposed is displayed.

Thereby, for example, an AR image can be displayed according to the movement of a subject such as a person. That is, the AR image can be displayed at an appropriate timing.

FIG. 301 is a diagram illustrating an example of the decoding image Pdec acquired according to the attitude of the receiver 200.

For example, as shown in (a) of FIG. 301, the receiver 200 images the transmitter 100 that transmits the optical ID according to the luminance change in the horizontal orientation. In the horizontal orientation, the longitudinal direction of the display 201 of the receiver 200 is an orientation along the horizontal direction. Further, each exposure line of the image sensor provided in the receiver 200 is orthogonal to the longitudinal direction of the display 201. By the imaging as described above, a decoding image Pdec including the bright line pattern region X with a small number of bright lines is acquired. In the bright line pattern region X of the decoding image Pdec, the number of bright lines is small. That is, there are few parts where the luminance changes to High or Low. Therefore, the receiver 200 may not be able to acquire the optical ID appropriately by decoding the decoding image Pdec.

Therefore, for example, as shown in (b) of FIG. 301, the user changes the attitude of the receiver 200 from landscape to portrait. The vertical orientation is a posture in which the longitudinal direction of the display 201 of the receiver 200 is along the vertical direction. The receiver 200 having such an attitude can acquire the decoding image Pdec including the bright line pattern region Y having a large number of bright lines when the transmitter 100 that transmits the light ID is imaged.

As described above, the optical ID may not be appropriately acquired according to the attitude of the receiver 200. Therefore, when the receiver 200 acquires the optical ID, the attitude of the receiver 200 that is imaging is appropriately set. It is good to change. When the posture is changed, the receiver 200 can appropriately acquire the light ID at a timing when the posture is such that the light ID can be easily acquired.

FIG. 302 is a diagram illustrating another example of the decoding image Pdec acquired according to the attitude of the receiver 200.

For example, the transmitter 100 is configured as a digital signage of a coffee shop, displays a video relating to a coffee shop advertisement during the video display period, and transmits a light ID by a change in luminance during the light ID transmission period. That is, the transmitter 100 alternately and repeatedly performs video display during the video display period and transmission of the optical ID during the optical ID transmission period.

The receiver 200 periodically acquires the captured display image Ppre and the decoding image Pdec by imaging of the transmitter 100. At this time, the decoding cycle including the bright line pattern region is synchronized with the repetition cycle of the video display period and the optical ID transmission period of the transmitter 100 and the repetition cycle of acquisition of the captured display image Ppre and the decoding image Pdec by the receiver 200. The image Pdec may not be acquired. Furthermore, the decoding image Pdec including the bright line pattern region may not be acquired depending on the attitude of the receiver 200.

For example, the receiver 200 images the transmitter 100 in a posture as shown in FIG. That is, the receiver 200 approaches the transmitter 100 and images the transmitter 100 so that the image of the transmitter 100 is projected on the entire image sensor of the receiver 200.

Here, if the timing at which the receiver 200 acquires the captured display image Ppre is within the video display period of the transmitter 100, the receiver 200 appropriately acquires the captured display image Ppre displayed by the transmitter 100. .

Even when the timing at which the receiver 200 acquires the decoding image Pdec extends between the video display period and the optical ID transmission period of the transmitter 100, the receiver 200 includes the bright line pattern region Z1. An image Pdec can be acquired.

That is, the exposure of each exposure line included in the image sensor is started sequentially from the exposure line at the upper end in the vertical direction downward. Therefore, even if the receiver 200 starts exposure of the image sensor to acquire the decoding image Pdec during the video display period, it is not possible to obtain the bright line pattern region. However, when the video display period is switched to the light ID transmission period, a bright line pattern region corresponding to each exposure line that is exposed in the light ID transmission period can be obtained.

Here, the receiver 200 images the transmitter 100 in a posture as shown in FIG. 302 (b). That is, the receiver 200 is separated from the transmitter 100 and images the transmitter 100 so that the image of the transmitter 100 is projected only on the area above the image sensor of the receiver 200. At this time, as described above, if the timing at which the receiver 200 acquires the captured display image Ppre is within the video display period of the transmitter 100, the receiver 200 captures the captured display image Prep on which the transmitter 100 is projected. Get properly. However, when the timing at which the receiver 200 acquires the decoding image Pdec extends between the video display period and the optical ID transmission period of the transmitter 100, the receiver 200 selects the decoding image Pdec including the bright line pattern region. You may not be able to get it. That is, even if the video display period of the transmitter 100 is switched to the optical ID transmission period, the image of the transmitter 100 whose luminance changes is displayed on each exposure line below the image sensor that is exposed during the optical ID transmission period. May not be projected. Therefore, the decoding image Pdec having the bright line pattern region cannot be acquired.

On the other hand, as shown in FIG. 302 (c), the receiver 200 projects the image of the transmitter 100 only on the lower region of the image sensor of the receiver 200 in a state of being separated from the transmitter 100. Then, the transmitter 100 is imaged. At this time, as described above, if the timing at which the receiver 200 acquires the captured display image Ppre is within the video display period of the transmitter 100, the receiver 200 captures the captured display image Prep on which the transmitter 100 is projected. Get properly. Furthermore, even when the timing at which the receiver 200 acquires the decoding image Pdec extends between the video display period and the optical ID transmission period of the transmitter 100, the receiver 200 acquires the decoding image Pdec including the bright line pattern region. There are things that can be done. That is, when the video display period of the transmitter 100 is switched to the optical ID transmission period, the image of the transmitter 100 whose luminance changes is displayed on each exposure line below the image sensor that is exposed during the optical ID transmission period. Projected. Therefore, the decoding image Pdec having the bright line pattern region Z2 can be acquired.

As described above, since the optical ID may not be appropriately acquired according to the attitude of the receiver 200, the receiver 200 may change the attitude of the receiver 200 when acquiring the optical ID. You may be encouraged. That is, when imaging starts, the receiver 200 displays, for example, a message “Please move” or “Shake” or output sound so that the attitude of the receiver 200 changes. Thereby, since the receiver 200 performs imaging while changing the posture, it can appropriately acquire the light ID.

FIG. 303 is a flowchart illustrating an example of the processing operation of the receiver 200.

For example, the receiver 200 determines whether or not the receiver 200 is shaken during imaging (step S461). Specifically, the receiver 200 determines whether or not it is shaken based on the output of the 9-axis sensor provided in the receiver 200. Here, if the receiver 200 determines that it is being shaken during imaging (Yes in step S461), the receiver 200 increases the above-described optical ID acquisition rate (step S462). Specifically, the receiver 200 acquires all captured images per unit time obtained during imaging as decoding images (that is, bright line images) Pdec, and decodes all the acquired decoding images. . Alternatively, the receiver 200 starts acquisition and decoding when all the captured images are acquired as the captured display image Ppre, that is, when acquisition and decoding of the decoding image Pdec are stopped.

On the other hand, if the receiver 200 determines that it is not shaken during imaging (No in step S461), the receiver 200 acquires the decoding image Pdec at a low optical ID acquisition rate (step S463). Specifically, if the optical ID acquisition rate is increased in step S462 and is still a high optical ID acquisition rate, the receiver 200 sets the optical ID acquisition rate because the current optical ID acquisition rate is high. Lower. Thereby, since the frequency with which the decoding process of the decoding image Pdec by the receiver 200 is reduced, power consumption can be suppressed.

Then, the receiver 200 determines whether or not an end condition for ending the adjustment process of the optical ID acquisition rate is satisfied (step S464). When the receiver 200 determines that the end condition is not satisfied (No in step S464), The processing from S461 is repeatedly executed. On the other hand, when the receiver 200 determines that the end condition is satisfied (Yes in step S464), the receiver 200 ends the optical ID acquisition rate adjustment process.

FIG. 304 is a diagram illustrating an example of a camera lens switching process by the receiver 200.

The receiver 200 may include a wide-angle lens 211 and a telephoto lens 212 as camera lenses. A captured image obtained by imaging using the wide-angle lens 211 is an image with a wide angle of view, and a subject is projected to be small in the image. On the other hand, a captured image obtained by imaging using the telephoto lens 212 is an image with a narrow angle of view, and a subject is projected greatly in the image.

The receiver 200 as described above may switch the camera lens used for imaging by any one of the methods A to E shown in FIG. 304 when imaging.

In Method A, the receiver 200 always uses the telephoto lens 212 when imaging, whether in normal imaging or when receiving an optical ID. Here, the case of normal imaging is a case where all captured images are acquired as captured display images Ppre by imaging. The case where the optical ID is received is a case where the captured display image Ppre and the decoding image Pdec are periodically acquired by imaging.

In Method B, the receiver 200 uses the wide-angle lens 211 in the case of normal imaging. On the other hand, when receiving the optical ID, the receiver 200 first uses the wide-angle lens 211. The receiver 200 switches the camera lens from the wide-angle lens 211 to the telephoto lens 212 if the bright line pattern region is included in the decoding image Pdec acquired when the wide-angle lens 211 is used. After this switching, the receiver 200 can acquire a decoding image Pdec with a narrow angle of view, that is, a bright line pattern region appearing large.

In Method C, the receiver 200 uses the wide-angle lens 211 in the case of normal imaging. On the other hand, when receiving the optical ID, the receiver 200 switches the camera lens between the wide-angle lens 211 and the telephoto lens 212. That is, the receiver 200 acquires the captured display image Ppre using the wide-angle lens 211 and acquires the decoding image Pdec using the telephoto lens 212.

In Method D, the receiver 200 switches the camera lens between the wide-angle lens 211 and the telephoto lens 212 in accordance with the operation by the user regardless of whether it is normal imaging or receives an optical ID.

In the method E, when receiving the optical ID, the receiver 200 decodes the decoding image Pdec acquired using the wide-angle lens 211. If the decoding image Pdec cannot be correctly decoded, the camera lens is changed from the wide-angle lens 211 to the telephoto lens 212. Switch. Alternatively, the receiver 200 decodes the decoding image Pdec acquired using the telephoto lens 212, and switches the camera lens from the telephoto lens 212 to the wide-angle lens 211 if it cannot be decoded correctly. Note that when determining whether or not the decoding image Pdec has been correctly decoded, the receiver 200 first transmits an optical ID obtained by decoding the decoding image Pdec to the server. If the optical ID matches the optical ID registered in the server, the server notifies the receiver 200 of the matching information indicating that it matches, and if it does not match, the server does not match. Is sent to the receiver 200. The receiver 200 determines that the decoding image Pdec has been correctly decoded if the information notified from the server is coincidence information. If the information notified from the server is mismatch information, the receiver 200 correctly decodes the decoding image Pdec. Judge that it was not possible. Alternatively, the receiver 200 determines that the decoding image Pdec has been correctly decoded when the optical ID obtained by decoding the decoding image Pdec satisfies a predetermined condition. On the other hand, if the condition is not satisfied, the receiver 200 determines that the decoding image Pdec has not been correctly decoded.

By switching the camera lens in this way, an appropriate decoding image Pdec can be acquired.

FIG. 305 is a diagram illustrating an example of camera switching processing by the receiver 200.

For example, the receiver 200 includes an in camera 213 and an out camera (not shown in FIG. 305) as cameras. The in-camera 213 is also referred to as a face camera or a self-portrait camera, and is a camera arranged on the same surface as the display 201 in the receiver 200. The out camera is a camera arranged on the surface of the receiver 200 opposite to the surface of the display 201.

Such a receiver 200 images the transmitter 100 configured as a lighting device with the in-camera 213 with the in-camera 213 facing upward. By this imaging, the receiver 200 acquires the decoding image Pdec, and acquires the optical ID transmitted from the transmitter 100 by decoding the decoding image Pdec.

Next, the receiver 200 acquires the AR image and the recognition information associated with the optical ID from the server by transmitting the acquired optical ID to the server. The receiver 200 starts a process of recognizing a target area corresponding to the recognition information from the captured display images Ppre obtained by the out camera and the in camera 213, respectively. Here, the receiver 200 prompts the user to move the receiver 200 when the target area cannot be recognized from any of the captured display images Ppre obtained by the out camera and the in camera 213 respectively. The user who is prompted by the receiver 200 moves the receiver 200. Specifically, the user moves the receiver 200 so that the in-camera 213 and the out-camera face the front-rear direction of the user. As a result, the receiver 200 recognizes the target area from the captured display image Ppre acquired by the out camera. That is, the receiver 200 recognizes a region in which a person is projected as a target region, superimposes an AR image on the target region in the captured display image Ppre, and displays the captured display image Ppre on which the AR image is superimposed. To do.

FIG. 306 is a flowchart illustrating an example of processing operation between the receiver 200 and the server.

The receiver 200 acquires an optical ID transmitted from the transmitter 100 by capturing an image of the transmitter 100, which is a lighting device, with the in-camera 213, and transmits the optical ID to the server (step S471). The server receives the optical ID from the receiver 200 (step S472), and estimates the position of the receiver 200 based on the optical ID (step S473). For example, the server stores, for each light ID, a table indicating a room, a building, a space, or the like in which the transmitter 100 that transmits the light ID is arranged. Then, the server estimates the room associated with the optical ID transmitted from the receiver 200 as the position of the receiver 200 in the table. Further, the server transmits the AR image and recognition information associated with the estimated position to the receiver 200 (step S474).

The receiver 200 acquires the AR image and the recognition information transmitted from the server (Step S475). Here, the receiver 200 starts a process of recognizing a target area corresponding to the recognition information from each captured display image Ppre obtained by each of the out camera and the in camera 213. Then, the receiver 200 recognizes the target area from, for example, the captured display image Ppre acquired by the out-camera (step S476). The receiver 200 superimposes the AR image on the target area in the captured display image Ppre, and displays the captured display image Ppre on which the AR image is superimposed (step S477).

In the above-described example, when the receiver 200 acquires the AR image and the recognition information transmitted from the server, in step S476, the receiver 200 selects from the captured display images Ppre obtained by the out camera and the in camera 213, respectively. The process of recognizing the target area has started. However, the receiver 200 may start the process of recognizing the target area from the captured display image Ppre obtained only by the out-camera in step S476. That is, the camera for acquiring the light ID (in-camera 213 in the above example) and the camera for acquiring the captured display image Pre on which the AR image is superimposed (out-camera in the above example) are always different. It may be allowed.

In the above-described example, the receiver 200 images the transmitter 100 that is an illumination device with the in-camera 213, but the floor illuminated by the transmitter 100 may be captured with the out-camera. Even with such an out-camera imaging, the receiver 200 can acquire the optical ID transmitted from the transmitter 100.

FIG. 307 is a diagram illustrating an example of superimposition of the AR image by the receiver 200.

The receiver 200 images the transmitter 100 configured as a microwave oven installed in a store such as a convenience store. The transmitter 100 includes a camera for imaging the inside of a microwave oven and an illumination device that illuminates the inside of the oven. And the transmitter 100 recognizes the food / beverage (namely, warming object) accommodated in the store | warehouse | chamber by imaging with a camera. In addition, when the food or drink is warmed, the transmitter 100 transmits the light ID indicating the recognized food or drink by causing the lighting device to emit light and changing the luminance of the lighting device. In addition, although this illuminating device illuminates the inside of the store | warehouse | chamber of a microwave oven, the light of the illuminating device is emitted outside from the window part which has the transparency of a microwave oven. Accordingly, the light ID is transmitted from the lighting device to the outside of the microwave oven through the window portion of the microwave oven.

Here, the user purchases food and drink at a convenience store, and puts the food and drink into the transmitter 100, which is a microwave oven, in order to warm the food and drink. At this time, the transmitter 100 recognizes the food and drink with the camera, and starts warming the food and drink while transmitting the optical ID indicating the recognized food and drink.

The receiver 200 acquires the optical ID transmitted from the transmitter 100 by capturing an image of the transmitter 100 that has started the warming, and transmits the optical ID to the server. Next, the receiver 200 acquires an AR image, audio data, and recognition information associated with the optical ID from the server.

The above-mentioned AR image includes an AR image P32a that is a moving image showing a virtual state inside the transmitter 100, an AR image P32b that shows food and drink stored in the cabinet in detail, and steam from the transmitter 100. AR image P32c which shows a state of being heated by a moving image, and AR image P32d which shows the remaining time until the completion of warming of food and drink is displayed.

For example, in the AR image P32a, if the food and drink stored in the microwave oven is a pizza, the turntable on which the pizza is placed is rotating, and a video in which a plurality of dwarfs are dancing around the pizza It is. The AR image P32b is, for example, an image showing the product name “pizza” and the material of the pizza if the food and drink stored in the warehouse is a pizza.

Based on the recognition information, the receiver 200 recognizes the area in which the window of the transmitter 100 is projected in the captured display image Ppre as the target area of the AR image P32a, and the AR image P32a is displayed in the target area. Superimpose. Further, based on the recognition information, the receiver 200 recognizes the area above the area where the transmitter 100 is projected in the captured display image Ppre as the target area of the AR image P32b, and the target The AR image P32b is superimposed on the area. Furthermore, based on the recognition information, the receiver 200 recognizes an area between the target area of the AR image P32a and the target area of the AR image P32b in the captured display image Ppre as the target area of the AR image P32c. Then, the AR image P32c is superimposed on the target area. Further, based on the recognition information, the receiver 200 recognizes an area below the area where the transmitter 100 is projected in the captured display image Ppre as the target area of the AR image P32d, and sets the target area as the target area. The AR image P32d is superimposed.

Furthermore, the receiver 200 outputs sound generated when food or drink is heated by reproducing audio data.

The AR images P32a to P32d as described above are displayed by the receiver 200, and further, by outputting sound, it is possible to attract the user's interest to the receiver 200 until the warming of food and drink is completed. . As a result, the burden on the user who is waiting for completion of warming can be reduced. Further, the AR image P32c indicating steam or the like is displayed, and a sound generated when the food or drink is heated is output, so that a sizzle can be given to the user. In addition, the display of the AR image P32d allows the user to easily know the remaining time until the completion of the heating of the food and drink. Therefore, the user can read, for example, a book displayed in the store away from the transmitter 100 that is a microwave oven until the warming is completed. The receiver 200 may notify the user that the warming is completed when the remaining time becomes zero.

In the above-described example, the AR image P32a is a moving image in which a turntable on which a pizza is placed is rotating and a plurality of dwarfs are dancing around the pizza. May be an image that virtually represents. Moreover, although AR image P32b was an image which shows the brand name and material of the food / beverage accommodated in the store | warehouse | chamber, the image which shows a nutrient component or a calorie may be sufficient. Alternatively, the AR image P32b may be an image showing a discount ticket.

As described above, in the display method according to this modification, the subject is a microwave oven provided with an illumination device, and the illumination device illuminates the interior of the microwave oven and changes the luminance to change the light ID of the microwave oven. Send to the outside. In acquiring the captured display image Ppre and the decoding image Pdec, the captured display image Ppre and the decoding image Pdec are acquired by imaging the microwave oven that transmits the optical ID. In the recognition of the target area, the window portion of the microwave oven displayed in the captured display image Ppre is recognized as the target area. In the display of the captured display image Ppre, the captured display image Ppre on which the AR image indicating the state change in the warehouse is superimposed is displayed.

Thereby, since the change in the state of the microwave oven is displayed as an AR image, the state of the oven can be easily communicated to the user of the microwave oven.

FIG. 308 is a sequence diagram showing processing operations of the system including the receiver 200, the microwave oven, the relay server, and the electronic settlement server. Note that the microwave oven includes a camera and a lighting device as described above, and transmits the light ID by changing the luminance of the lighting device. That is, the microwave oven has a function as the transmitter 100.

First, the microwave oven recognizes food and drink stored in the cabinet with the camera (step S481). Next, the microwave oven transmits a light ID indicating the recognized food and drink to the receiver 200 by a luminance change of the lighting device.

The receiver 200 receives the optical ID transmitted from the microwave oven by imaging the microwave oven (step S483), and transmits the optical ID and the card information to the relay server. The card information is information such as a credit card stored in advance in the receiver 200 and is information necessary for electronic payment.

The relay server holds a table indicating an AR image, recognition information, and product information corresponding to each optical ID. This merchandise information indicates the price of food and drink indicated by the light ID. When such a relay server receives the optical ID and card information transmitted from the receiver 200 (step S486), it finds product information associated with the optical ID from the above table. Then, the relay server transmits the merchandise information and card information to the electronic settlement server (step S486). When the electronic settlement server receives the merchandise information and the card information transmitted from the relay server (step S487), the electronic settlement server performs an electronic settlement process based on the merchandise information and the card information (step S488). Then, when the electronic payment processing is completed, the electronic settlement server notifies the relay server of the completion (step S489).

When the relay server confirms the payment completion notification from the electronic payment server (step S490), the relay server instructs the microwave to start heating the food (step S491). Further, the relay server transmits the AR image and the recognition information associated with the optical ID received in step S485 in the above table to the receiver 200 (step S493).

When the microwave oven receives a warming start instruction from the relay server, it starts warming the food and drink stored in the cabinet (step S492). Further, when the receiver 200 receives the AR image and the recognition information transmitted from the relay server, the target according to the recognition information from the captured display image Ppre periodically acquired by the imaging started from step S483. Recognize the area. Then, the receiver 200 superimposes the AR image on the target area (step S494).

Thereby, the user of the receiver 200 can easily settle the settlement and start warming the food and drink by placing the food and drink in the microwave oven and taking an image. Moreover, when payment cannot be performed, warming of food and drink by a user can be prohibited. Furthermore, when warming is started, an AR image P32a shown in FIG. 307 and the like can be displayed, and the user can be informed of the state in the warehouse.

FIG. 309 is a sequence diagram illustrating processing operations of a system including a POS terminal, a server, a receiver 200, and a microwave oven. Note that the microwave oven includes a camera and a lighting device as described above, and transmits the light ID by changing the luminance of the lighting device. That is, the microwave oven has a function as the transmitter 100. A POS (point-of-sale) terminal is a terminal installed in a store such as the same convenience store as the telegraph range.

First, the user of the receiver 200 selects a food or drink as a product at a store and heads to a place where a POS terminal is installed in order to purchase the food or drink. The store clerk operates the POS terminal and receives the price of food and drink from the user. By the operation of the POS terminal by the store clerk, the POS terminal acquires operation input data and sales information (step S501). The sales information indicates, for example, the name, number, and price of the product, the sales location, and the sales date. The operation input data indicates, for example, the sex and age of the user input by the store clerk. The POS terminal transmits the operation input data and sales information to the server (step S502). The server receives operation input data and sales information transmitted from the POS terminal (step S503).

On the other hand, when the user of the receiver 200 pays the clerk for the food and drink, he / she puts the food and drink in the potential range to warm the food and drink. The microwave oven recognizes the food and drink stored in the cabinet with the camera (step S504). Next, the microwave oven transmits the light ID indicating the recognized food and drink to the receiver 200 by the luminance change of the lighting device (step S505). And a microwave oven starts warming of food and drink (step S507).

The receiver 200 receives the optical ID transmitted from the microwave oven by imaging the microwave oven (step S508), and transmits the optical ID and the terminal information to the server (step S509). The terminal information is information stored in advance in the receiver 200 and indicates, for example, the language type (for example, English or Japanese) displayed on the display 201 of the receiver 200.

When the server is accessed from the receiver 200 and receives the optical ID and the terminal information transmitted from the receiver 200, the server determines whether the access from the receiver 200 is the first access (step S510). The first access is the first access performed within a predetermined time from the time when the process of step S503 is performed. If the server determines that the access from the receiver 200 is the first access (Yes in step S510), the server associates and stores the operation input data and the terminal information (step S511).

The server determines whether or not the access from the receiver 200 is the first access, but may determine whether or not the product indicated by the sales information matches the food or drink indicated by the light ID. . In step S511, the server may store not only the operation input data and the terminal information but also the sales information in association with them.

(Indoor use)
FIG. 310 is a diagram showing a state of indoor use such as an underground shopping center.

The receiver 200 receives the light ID transmitted from the transmitter 100 configured as a lighting device, and estimates its current position. Further, the receiver 200 displays the current position on a map to provide route guidance, or displays information on nearby stores.

In an emergency, transmission of disaster information and evacuation information from the transmitter 100 can be used when communication is congested, when a communication base station fails, or when radio waves from the communication base station do not reach. Even this information can be obtained. This is effective for a hearing impaired person who has missed an emergency broadcast or cannot hear an emergency broadcast.

That is, the receiver 200 acquires the optical ID transmitted from the transmitter 100 by taking an image, and further acquires the AR image P33 and the recognition information associated with the optical ID from the server. Then, the receiver 200 recognizes the target area corresponding to the recognition information from the captured display image Ppre obtained by the above-described imaging, and superimposes the AR image P33 in the shape of an arrow on the target area. Thereby, receiver 200 can be used as the above-mentioned way finder (refer to Drawing 294).

(Display of augmented reality object)
FIG. 311 is a diagram illustrating a state in which an augmented reality object is displayed.

The stage 2718e that displays the augmented reality is configured as the transmitter 100 described above, and the light emitting patterns and position patterns of the light emitting units 2718a, 2718b, 2718c, and 2718d are the reference positions for displaying the augmented reality object information and the augmented reality object. Send.

The receiver 200 displays the augmented reality object 2718f, which is an AR image, superimposed on the captured image based on the received information.

Note that these comprehensive or specific modes may be realized by a recording medium such as an apparatus, a system, a method, an integrated circuit, a computer program, or a computer-readable CD-ROM. You may implement | achieve in arbitrary combinations of a circuit, a computer program, or a recording medium. Moreover, the computer program which performs the method concerning one Embodiment is preserve | saved at the recording medium of the server, and may be implement | achieved in the aspect delivered to a terminal from a server according to the request | requirement of a terminal.

[Modification 4 of Embodiment 23]
FIG. 312 is a diagram showing a configuration of a display system in Modification 4 of Embodiment 23. In FIG.

This display system 500 performs object recognition and augmented reality (Augmented Reality / Mixed Reality) display using visible light signals.

The receiver 200 performs imaging, receives a visible light signal, and extracts feature quantities for object recognition or space recognition. The feature amount extraction is extraction of an image feature amount from a captured image obtained by imaging. The visible light signal may be a visible light adjacent carrier signal such as infrared light or ultraviolet light. In this modification, the receiver 200 is configured as a recognition device that recognizes an object on which an augmented reality image (that is, an AR image) is displayed. In the example illustrated in FIG. 312, the target object is, for example, the AR target object 501.

The transmitter 100 transmits information such as an ID for identifying itself or the AR object 501 as a visible light signal or a radio wave signal. The ID is identification information such as the above-described optical ID, for example, and the AR object 501 is the above-described target area. The visible light signal is a signal transmitted by a change in luminance of a light source included in the transmitter 100.

The receiver 200 or the server 300 holds the identification information transmitted from the transmitter 100 in association with the AR recognition information and the AR display information. The association may be one-to-one or one-to-many. The AR recognition information is the above-described recognition information, and is information for recognizing the AR object 501 for performing AR display. Specifically, the AR recognition information is the image feature amount (SIFT feature amount, SURF feature amount, ORB feature amount, etc.), color, shape, size, reflectance, transmittance, or three-dimensional of the AR object 501. Model etc. The AR recognition information may include identification information or a recognition algorithm indicating which recognition method is used for recognition. The AR display information is information for performing AR display, and is an image (that is, the above-described AR image), video, audio, three-dimensional model, motion data, display coordinates, display size, transmittance, or the like. Further, the AR display information may be the absolute value or change ratio of each of hue, saturation, and brightness.

The transmitter 100 may also function as the server 300. That is, the transmitter 100 may hold the AR recognition information and the AR display information and transmit the information by wired or wireless communication.

The receiver 200 captures an image with a camera (specifically, an image sensor). The receiver 200 receives a visible light signal or a radio wave signal such as WiFi or Bluetooth (registered trademark). Further, the receiver 200 acquires position information obtained by GPS or the like, information obtained by a gyro sensor or acceleration sensor, and information such as sound from a microphone, and integrates all or some of these information. AR objects existing in the vicinity may be recognized. Further, the receiver 200 may recognize the AR object using only one of the information without integrating the information.

FIG. 313 is a flowchart showing the processing operation of the display system according to the fourth modification of the twenty-third embodiment.

First, the receiver 200 determines whether or not a visible light signal has already been received (step S521). That is, for example, the receiver 200 determines whether or not the visible light signal indicating the identification information is acquired by photographing the transmitter 100 that transmits the visible light signal according to the luminance change of the light source. At this time, a captured image of the transmitter 100 is acquired by the shooting.

Here, when the receiver 200 determines that the visible light signal has already been received (Y in step S521), the AR object (object, reference point, spatial coordinate, or space in the space is determined from the received information. The position and orientation of the receiver 200 are specified. Furthermore, the receiver 200 recognizes the relative position of the AR object. This relative position is represented by the distance and direction from the receiver 200 to the AR object. For example, the receiver 200 identifies the AR object (that is, the target area that is the bright line pattern area) based on the size and position of the bright line pattern area shown in FIG. 244 and recognizes the relative position of the AR target object. To do.

Then, the receiver 200 transmits the information such as an ID included in the visible light signal and the relative position to the server 300, and uses the information and the relative position as a key, thereby registering the AR recognition information in the server 300. And AR display information are acquired (step S522). At this time, the receiver 200 acquires not only the information on the recognized AR object but also information on other AR objects existing in the vicinity of the AR object (that is, the AR recognition information and the AR display information). Also good. Thereby, when another AR object existing in the vicinity is imaged by the receiver 200, the receiver 200 recognizes the other AR object existing in the vicinity quickly and without error. Can do. For example, other AR objects present in the vicinity are different from the first recognized AR object.

Note that the receiver 200 may acquire these pieces of information from the database in the receiver 200 instead of accessing the server 300. The receiver 200 receives these pieces of information after a certain period of time has elapsed from the time of acquisition or a specific process (for example, turning off the screen, pressing a button, ending or stopping an application, displaying an AR image, or another AR object. May be discarded after the recognition). Alternatively, the receiver 200 may decrease the reliability of the information every time a certain period of time has elapsed since the acquisition of the information, and use highly reliable information among the plurality of information. Good.

Here, the receiver 200 may preferentially acquire the AR recognition information of the AR object that is effective in relation to the relative position based on the relative position to each AR object. For example, the receiver 200 acquires a plurality of visible light signals (that is, identification information) by photographing the plurality of transmitters 100 in step S521, and corresponds to the plurality of visible light signals in step S522. A plurality of AR recognition information (that is, image feature amount) is acquired. At this time, in step S522, the receiver 200 selects an image feature amount of the AR object closest to the receiver 200 that performs imaging of the transmitter 100 among the plurality of AR objects. That is, the selected image feature amount is used to specify one AR object (that is, the first object) specified using the visible light signal. Thereby, even if a plurality of image feature amounts are acquired, an appropriate image feature amount can be used for specifying the first object.

On the other hand, when it is determined that the visible light signal is not received (N in step S521), the receiver 200 further determines whether or not the AR recognition information has already been acquired (step S523). If it is determined that the AR recognition information has not been acquired (N in step S523), the receiver 200 does not use identification information such as an ID indicated by the visible light signal, or by image processing, or position information or radio wave information. The AR object candidate is recognized using other information such as (step S524). This process may be performed only by the receiver 200. Alternatively, the receiver 200 may transmit information such as a captured image or an image feature amount of the captured image to the server 300, and the server 300 may recognize the AR object candidate. As a result, the receiver 200 acquires AR recognition information and AR display information corresponding to the recognized candidate from the server 300 or its own database.

After step S522, the receiver 200 determines whether or not the AR object is detected by another method that does not use identification information such as an ID indicated by the visible light signal, such as image recognition (step S525). ). That is, the receiver 200 determines whether or not the AR object is recognized by a plurality of methods. Specifically, the receiver 200 specifies the AR object (that is, the first object) from the captured image using the image feature amount acquired based on the identification information indicated by the visible light signal. Then, the receiver 200 determines whether or not the AR object (that is, the second object) is specified from the captured image by image processing without using such identification information.

Here, when the receiver 200 determines that the AR object has been recognized by a plurality of methods (Y in step S525), the recognition result by the visible light signal is prioritized. That is, the receiver 200 confirms whether or not the AR objects recognized by the respective methods match. If they do not match, the receiver 200 selects one AR object on which the AR image is superimposed in the captured image as the AR object recognized by the visible light signal. Determination is made (step S526). That is, when the first object is different from the second object, the receiver 200 gives priority to the first object and recognizes it as the object on which the AR image is displayed. The object on which the AR image is displayed is an object on which the AR image is superimposed.

Alternatively, the receiver 200 may prioritize a method having a higher priority order based on a priority order assigned to each of the plurality of methods. That is, the receiver 200 recognizes one AR object on which the AR image is superimposed in the captured image from among the AR objects recognized by each method, for example, by a method having the highest priority. AR target is determined. Alternatively, the receiver 200 may determine one AR object on which the AR image is superimposed in the captured image by a majority decision or a majority decision with priority. If the recognition result up to that point is overturned by this process, the receiver 200 performs an error handling process.

Next, the receiver 200 determines the state of the AR object in the captured image (specifically, the absolute position, the relative position from the receiver 200, the size, the angle, and the illumination status) based on the acquired AR recognition information. Or occlusion or the like) (step S527). Then, the receiver 200 displays the AR display information (that is, the AR image) superimposed on the captured image in accordance with the recognition result (step S528). That is, the receiver 200 superimposes the AR display information on the recognized AR object in the captured image. Alternatively, the receiver 200 displays only the AR display information.

These enable recognition or detection that is difficult with image processing alone. The difficult recognition or detection includes, for example, identification of AR objects that are image-similar (such as only different text content), detection of AR objects with fewer patterns, and AR objects with high reflectivity or transmittance. Detection of an object, detection of an AR object (for example, an animal) whose shape or pattern changes, or detection of an AR object from a wide angle (in various directions). That is, in this modification, recognition of these AR objects and AR display can be performed. In addition, in image processing that does not use a visible light signal, as the number of AR objects to be recognized increases, it takes time to search for neighborhoods of image feature values, which takes time for recognition processing, and the recognition rate also deteriorates. . However, in this modification, the influence of the increase in recognition time and the deterioration of the recognition rate due to the increase in recognition objects is not at all or extremely small, and effective AR object recognition is possible. Further, efficient recognition is possible by using the relative position of the AR object. For example, by using an approximate distance to the AR object, processing for making the AR object size independent of the calculation of the image feature amount can be omitted, or a size-dependent feature can be used. Can do. In addition, when the angle of the AR object is used and it is usually necessary to evaluate the image feature amount for many angles, only the retention and calculation of the image feature amount corresponding to the angle of the AR object is performed. The calculation speed or the memory efficiency can be improved.

[Summary of Modification 4 of Embodiment 23]
FIG. 314 is a flowchart illustrating a recognition method according to an aspect of the present invention.

The display method according to one aspect of the present invention is a method for recognizing an object on which an augmented reality image (AR image) is displayed, and includes steps S531 to S535.

In step S531, the receiver 200 acquires the identification information by photographing the transmitter 100 that transmits a visible light signal by a change in luminance of the light source. The identification information is, for example, an optical ID. In step S <b> 532, the receiver 200 transmits the identification information to the server 300 and acquires an image feature amount corresponding to the identification information from the server 300. The image feature amount is indicated as AR recognition information or recognition information.

In step S533, the receiver 200 specifies the first object from the captured image of the transmitter 100 using the image feature amount. In step S534, the receiver 200 specifies the second object from the captured image of the transmitter 100 by image processing without using the identification information (that is, the optical ID).

In step S535, when the first object specified in step S533 is different from the second object specified in step S534, the receiver 200 gives priority to the first object and augments reality. Recognize as an object to be displayed.

For example, the augmented reality image, the captured image, and the target object correspond to the AR image, the captured display image, and the target region in the twenty-third embodiment and each modification thereof, respectively.

Thereby, as shown in FIG. 313, the first object specified using the identification information indicated by the visible light signal, and the second object specified by the image processing without using the identification information, Are different, the first object is preferentially recognized as the object on which the augmented reality image is displayed. Therefore, the object on which the augmented reality image is displayed can be appropriately recognized from the captured image.

In addition to the image feature amount of the first object, the image feature amount includes an image feature amount of a third object that is located in the vicinity of the first object and is different from the first object. May be.

Thereby, as shown in step S522 of FIG. 313, not only the image feature amount of the first object but also the image feature amount of the third object is acquired. When appearing in the captured image, the third object can be quickly identified or recognized.

In addition, the receiver 200 may acquire a plurality of identification information by photographing a plurality of transmitters in step S531, and may acquire a plurality of image feature amounts corresponding to the plurality of identification information in step S532. is there. In such a case, in step S533, the receiver 200 determines the image feature amount of the object closest to the receiver 200 that performs imaging by the plurality of transmitters among the plurality of objects as the first object. It may be used to identify

Thus, as shown in step S522 of FIG. 313, even when a plurality of image feature amounts are acquired, an appropriate image feature amount can be used for specifying the first object.

Note that the recognition device in the present modification is, for example, a device provided in the receiver 200 described above, and includes a processor and a recording medium. On this recording medium, a program for causing the processor to execute the recognition method shown in FIG. 314 is recorded. Moreover, the program in this modification is a program which makes a computer perform the recognition method shown in FIG.

(Embodiment 24)
FIG. 315 is a diagram illustrating an example of an operation mode of a visible light signal according to this embodiment. The present embodiment corresponds to a modification of the twentieth embodiment.

As shown in FIG. 315, there are two modes in the operation mode of the physical (PHY) layer of the visible light signal. The first operation mode is a mode in which packet PWM (Pulse Width Modulation) is performed, and the second operation mode is a mode in which packet PPM (Pulse-Position Modulation) is performed. The transmitter according to each of the above-described embodiments or modifications thereof generates and transmits a visible light signal by modulating a signal to be transmitted according to any one of these operation modes.

In the operation mode of the packet PWM, RLL (Run-Length Limited) encoding is not performed, the optical clock rate is 100 kHz, forward error correction (FEC) is repeatedly encoded, and a typical data rate is 5.5 kbps. It is.

In this packet PWM, the pulse width is modulated, and the pulse is represented by two brightness states. The two brightness states are a bright state (Bright or High) and a dark state (Dark or Low), but typically the light is on and off. A chunk of a physical layer signal called a packet (also referred to as a PHY packet) corresponds to a MAC (medium access control) frame. The transmitter can repeatedly transmit PHY packets and transmit a set of a plurality of PHY packets regardless of a special order.

Note that this packet PWM is, for example, the modulation shown in FIGS. 188 and 189A (b) and 197 described above. The packet PWM is used to generate a visible light signal transmitted from a normal transmitter.

In the operation mode of packet PPM, RLL encoding is not performed, the optical clock rate is 100 kHz, forward error correction (FEC) is repeatedly encoded, and a typical data rate is 8 kbps.

In this packet PPM, the position of a short time length pulse is modulated. That is, this pulse is a bright pulse of a bright pulse (High) and a dark pulse (Low), and the position of this pulse is modulated. The position of this pulse is indicated by the interval between the pulse and the next pulse.

Packet PPM realizes deep dimming. The format, waveform, and characteristics of the packet PPM not described in each embodiment and its modification are the same as those of the packet PWM. The packet PPM is modulated as shown in FIG. 189B, FIG. 199, FIG. The packet PPM is used to generate a visible light signal transmitted from a transmitter having a light source that emits very bright light.

In each of the packet PWM and the packet PPM, dimming in the physical layer of the visible light signal is controlled by the average luminance of the optional field.

<Packet PWM PPDU format>
Here, a format of a PPDU (physical-layer data unit) will be described.

FIG. 316 is a diagram illustrating an example of a PPDU format in mode 1 of packet PWM. FIG. 317 is a diagram illustrating an example of a PPDU format in mode 2 of the packet PWM. FIG. 318 is a diagram illustrating an example of a PPDU format in mode 3 of the packet PWM.

In the mode 1 and the mode 2, the packet modulated by the packet PWM includes a PHY payload A, an SHR (synchronization header), a PHY payload B, and an optional field as shown in FIGS. 316 and 317. SHR is a header for PHY payload A and PHY payload B. The PHY payload A and the PHY payload B are collectively referred to as a PHY payload.

In mode 3, as shown in FIG. 318, the packet modulated by the packet PWM includes SHR, PHY payload, SFT (synchronization footer), and optional field. SHT is a header for the PHY payload, and SFT is a footer for the PHY payload.

In each of the modes 1 to 3, in the PHY payload A, SHR, PHY payload B, and SFT, the first and second luminance values that are different from each other appear alternately on the time axis. The first luminance value is Bright or High, and the second luminance value is Dark or Low.

Here, the SHR of the packet PWM includes two or four pulses. These pulses are Bright or Dark brightness pulses.

FIG. 319 is a diagram showing an example of a pulse width pattern in each SHR of the packet PWM modes 1 to 3.

As shown in FIG. 319, in the mode 1 of the packet PWM, the SHR includes two pulses. Of the two pulses, the pulse width H1 of the first pulse in the order of transmission is 100 μsec, and the pulse width H2 of the second pulse is 90 μsec. In mode 2 of packet PWM, the SHR includes 4 pulses. Of the four pulses, the pulse width H1 of the first pulse in the transmission order is 100 μsec, the pulse width H2 of the second pulse is 90 μsec, and the pulse width H3 of the third pulse is 90 μs, and the pulse width H4 of the fourth pulse is 100 μs. In mode 3 of packet PWM, the SHR includes 4 pulses. Of the four pulses, the pulse width H1 of the first pulse in the transmission order is 50 μsec, the pulse width H2 of the second pulse is 40 μsec, and the pulse width H3 of the third pulse is It is 40 μs, and the pulse width H4 of the fourth pulse is 50 μs.

The PHY payload includes 6-bit data (ie, x ₀ -x ₅ ) as a transmission target signal in mode 1, and 12-bit data (ie, x ₀ -x ₁₁ ) as a transmission target signal in mode 2. Including. In the mode 3, the PHY payload includes data with a variable number of bits (ie, x ₀ -x _n ) as a transmission target signal. n is an integer of 1 or more, but more specifically is an integer obtained by subtracting 1 from a multiple of 3.

The parameter yk _is defined as _{y k = y k = x 3k} + x 3k + 1 × 2 + x 3k + 2 × 4. In mode 1, k is 0 or 1, and in mode 2, k is 0, 1, 2, or 3. In mode 3, k is an integer from 0 to {(n + 1) / 3-1}.

In each of mode 1 and mode 2, the signal to be transmitted included in the PHY payload A has two pulse widths P _A1 and P _A2 , with a pulse width P _Ak = 120 + 30 × (7−y _k ) [μs], Or, it is modulated to four pulse widths P _A1 to P _A4 . The signal to be transmitted included in the PHY payload B is modulated into two pulse widths P _B1 and P _B2 or four pulse widths P _B1 to P _B4 by a pulse width P _Bk = 120 + 30 × y _k [μsec]. The

In mode 3, the signal to be transmitted included in the PHY payload is modulated to (n + 1) / 3 pulse widths P1, P2,... By the pulse width P _k = 100 + 20 × y _k [μs]. Is done.

In mode 1 and mode 2, half of all payloads including PHY payload A and PHY payload B are optional. That is, the transmitter may transmit PHY payload A and PHY payload B, or only one of them. Further, the transmitter may transmit only a part of the PHY payload A and only a part of the PHY payload B. Specifically, in mode 2, the transmitter transmits a pulse having a pulse width P _{A3 and} a pulse having a pulse width P _{A4 in} the PHY payload A, and a pulse having a pulse width P _{B1 and} a pulse having a pulse width P _{B2 in} the PHY payload B. May be sent.

Mode 3 SFT includes four pulses with pulse widths F1 to F4 of 40 μsec, 50 μsec, 60 μsec, and 40 μsec, respectively. The SFT is optional. Therefore, the transmitter may transmit the next SHR instead of the SFT.

The transmitter may transmit any type of signal as a signal included in the optional field. However, the signal must not contain the SHR pattern. Such an optional field is used for direct current compensation or dimming control.

<PPDU format of packet PPM>
FIG. 320 is a diagram showing an example of a PPDU format in mode 1 of the packet PPM. FIG. 321 is a diagram illustrating an example of a PPDU format in mode 2 of the packet PPM. FIG. 322 is a diagram illustrating an example of a PPDU format in mode 3 of the packet PPM.

In the mode 1 and the mode 2, the packet modulated by the packet PPM includes the SHR, the PHY payload, and the optional field as shown in FIG. 320 and FIG. SHR is a header for the PHY payload.

In mode 3, the packet modulated by the packet PPM includes SHR, PHY payload, SFT, and optional field as shown in FIG. The SFT is a footer for the PHY payload.

In each of the modes 1 to 3, in the SHR, the PHY payload, and the SFT, the first and second luminance values that are different from each other appear alternately along the time axis. The first luminance value is Bright or High, and the second luminance value is Dark or Low.

The time length of short and bright pulses in packet PPM (L in FIGS. 320 to 322) is shorter than 10 μsec. Thereby, it can darken, suppressing the average brightness | luminance of a visible light signal.

The SHR time length of the packet PPM includes three intervals H1 to H3. Each of the three intervals H1 to H3 is an interval of four consecutive pulses (specifically, the above-described bright pulse).

FIG. 323 is a diagram illustrating an example of an interval pattern in each SHR of modes 1 to 3 of the packet PPM.

As shown in FIG. 323, in the mode 1 of the packet PPM, the three intervals H1 to H3 are each 160 μsec. In mode 2 of the packet PWM, the first interval H1 among the three intervals H1 to H3 is 160 μsec, the second interval H2 is 180 μsec, and the third interval H3 is 160 μsec. In mode 3 of the packet PPM, the first interval H1 of the three intervals H1 to H3 is 80 μsec, the second interval H2 is 90 μsec, and the third interval H3 is 80 μsec.

The PHY payload includes 6-bit data (ie, x ₀ -x ₅ ) as a transmission target signal in mode 1, and 12-bit data (ie, x ₀ -x ₁₁ ) as a transmission target signal in mode 2. Including. In the mode 3, the PHY payload includes data with a variable number of bits (ie, x ₀ -x _n ) as a transmission target signal. n is an integer of 5 or more, but more specifically is an integer obtained by subtracting 1 from a multiple of 3.

The parameter yk _is defined as _{y k = y k = x 3k} + x 3k + 1 × 2 + x 3k + 2 × 4. In mode 1, k is 0 or 1, and in mode 2, k is 0, 1, 2, or 3. In mode 3, k is an integer from 0 to {(n + 1) / 3-1}.

In each of mode 1 and mode 2, the signal to be transmitted included in the PHY payload is modulated into two intervals P1 and P2 or four intervals P1 to P4 by an interval P _k = 180 + 30 × y _k [μsec]. Is done.

In mode 3, the signal to be transmitted included in the PHY payload is modulated into (n + 1) / 3 intervals P1, P2,... By the interval P _k = 100 + 20 × y _k [μsec]. . In mode 3, a PHY payload that continues until SFT or the next SHR is transmitted.

In addition, the mode 3 SFT includes three intervals F1 to F3, and each of the intervals F1 to F3 is 90 μsec, 80 μsec, and 90 μsec. The SFT is optional. Therefore, the transmitter may transmit the next SHR instead of the SFT.

The transmitter may transmit any type of signal as a signal included in the optional field. However, the signal must not contain the SHR pattern. Such an optional field is used for direct current compensation or dimming control.

<PHY frame format>
Hereinafter, the PHY frame in mode 1 of each of the packet PWM and the packet PPM will be described.

The PHY payload includes 6 bits of data (ie, x ₀ -x ₅ ) as described above. The packet address A (a ₀ , a ₁ ) of the packet containing the data is indicated by (x ₁ , x ₄ ). Packet data D (d ₀ , d ₁ , d ₂ , d ₃ ) is indicated by (x ₀ , x ₂ , x ₃ , x ₅ ). The PHY frame which is the above-mentioned MAC frame is composed of 16 bits including packet data D ₀₀ , D ₀₁ , D ₁₀ and D ₁₁ of four packets. Here, the packet data Dk is packet data D of a packet having an address A indicating k.

Here, as described above, 2 bits (x ₁ , x ₄ ) out of 6 bits (x ₀ -x ₅ ) are used for the packet address A (a ₀ , a ₁ ). As a result, the time length of the 6-bit PHY payload can be shortened, and as a result, a visible light signal can be transmitted to a long distance. That is, since each of 2 bits (x ₂ , x ₅ ) out of 6 bits (x ₀ -x ₅ ) is not used for the packet address A, it can be set to 0. The 2 bits (x ₂ , x ₅ ) are multiplied by a large coefficient 4 by the above-described y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4, and based on the multiplication result The pulse width or interval is determined. Therefore, when each of the 2 bits (x ₂ , x ₅ ) is 0, the time length of the PHY payload can be shortened, and as a result, the transmission distance of the visible light signal can be extended.

Also, since 2 bits (x ₀ , x ₃ ) of 6 bits (x ₀ -x ₅ ) are not used for the packet address A, reception errors can be suppressed. That is, the influence of 2 bits (x ₀ , x ₃ ) out of 6 bits (x ₀ -x ₅ ) on the above-described parameter y _k (x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4) is small. Therefore, if these 2 bits (x ₀ , x ₃ ) are used for packet address A, the same numerical value of parameter y _k , that is, the same pulse width or interval, is determined for different packet addresses A. There is a possibility. As a result, the receiver may mistake the packet address A. The PHY frame reception error rate is larger when the packet address A is wrong than when a part of the packet data is wrong. Therefore, reception errors can be suppressed by using each of (x ₁ , x ₄ ) instead of 2 bits (x ₀ , x ₃ ) of 6 bits (x ₀ -x ₅ ) for packet address A. .

By the way, MPDU (medium-access-control protocol-data unit) has a very large overhead for PHY frames, and most of its fields are repeated MSDU (medium-access-control service-data unit). Is unnecessary. Therefore, the PHY frame does not have MHR (medium-access-control footer), and MFR (medium-access-control footer) is optional.

Next, the PHY frame in mode 2 of each of the packet PWM and the packet PPM will be described.

FIG. 324 is a diagram illustrating an example of 12-bit data included in the PHY payload.

As described above, the PHY payload includes 12 bits of data (ie, x ₀ -x ₁₁ ). This data includes packet address A (all or part of a ₀ -a ₃ ), packet data Da (all or part of d _a0 -d _a6 ), and packet data Db (all or part of d _b0 -d _b3 ) And a stop bit S (s).

That is, as shown in FIG. 324, 3 bits (x ₀ , x ₁ , x ₂ ) indicate (d _a0 , s, d _b0 ), and 3 bits (x ₃ , x ₄ , x ₅ ) indicate (d _a1 , A ₀ or d _a6 , d _b1 ). Further, 3 bits (x ₆ , x ₇ , x ₈ ) represent (d _a2 , a ₁ or d _a5 , d _b2 ), and 3 bits (x ₉ , x ₁₀ , x ₁₁ ) represent (d _a3 , a ₂ Or d _a4 , a ₃ or d _b3 ).

Note that the 12-bit data shown in FIG. 324 is the same as the data shown in FIG. That is, the codes w1, w2, w3, and w4 shown in FIG. 215 are 3 bits (x ₀ , x ₁ , x ₂ ), (x ₃ , x ₄ , x ₅ ), (x ₆ , x ₇ , x ₈ ), respectively. ) And (x ₉ , x ₁₀ , x ₁₁ ).

Bits x ₄ , x ₇ , x ₁₀ and x ₁₁ are used for any one of the packet address and the packet data according to the packet division rule.

FIG. 325 to FIG. 332 are diagrams showing processing for dividing the PHY frame into packets. The processing shown in FIGS. 325 to 332 is the same as the packet generation processing shown in FIGS. 216 to 226, but the point that the parity is not included in the packet generated by the division is shown in FIGS. 216 to 226. Different from the process shown in. Also, the numerical value on the second line from the top in each box shown in FIGS. 325 to 332 indicates the bit size, and the numerical value on the third line from the top indicates the bit value (0 or 1).

FIG. 325 is a diagram illustrating a process of storing the PHY frame in one packet. In other words, FIG. 325 shows a process for storing 7-bit data included in the PHY frame in one packet without dividing the PHY frame.

Specifically, out of the 7 bits of the PHY frame, 4-bit packet data Da (0) and 3-bit packet data Db (0) consist of a 1-bit stop bit and a 4-bit packet address. Together with packet 0. The stop bit indicates “1” and the packet address indicates “0000”.

FIG. 326 is a diagram illustrating processing for dividing the PHY frame into two packets.

Of the 18 bits of the PHY frame, packet data Da (0) consisting of 7 bits and packet data Db (0) consisting of 4 bits are stored in packet 0 together with 1 stop bit. The stop bit indicates “0”. Of the 18 bits of the PHY frame, packet data Da (1) consisting of 4 bits and packet data Db (1) consisting of 3 bits are packeted together with a 1-bit stop bit and a 4-bit packet address. 1 The stop bit indicates “1”, and the packet address indicates “1000”.

FIG. 327 is a diagram showing processing for dividing the PHY frame into three packets.

Of the 27 bits of the PHY frame, 6-bit packet data Da (0) and 4-bit packet data Db (0) are transmitted in packet 0 together with a 1-bit stop bit and a 1-bit packet address. Can be stored. The stop bit indicates “0” and the packet address indicates “0”. Of the 27 bits of the PHY frame, packet data Da (1) consisting of 6 bits and packet data Db (1) consisting of 4 bits are packeted together with 1 stop bit and 1 bit packet address. 1 The stop bit indicates “0” and the packet address indicates “1”. Further, of the 27 bits of the PHY frame, packet data Da (2) consisting of 4 bits and packet data Db (2) consisting of 3 bits are packeted together with a 1-bit stop bit and a 4-bit packet address. 2 The stop bit indicates “1” and the packet address indicates “0100”.

FIG. 328 is a diagram showing processing for dividing the PHY frame into four packets.

Of the 34 bits of the PHY frame, packet data Da (0) consisting of 5 bits and packet data Db (0) consisting of 4 bits are transferred to packet 0 together with a 1-bit stop bit and a 2-bit packet address. Can be stored. The stop bit indicates “0” and the packet address indicates “00”. Of the 34 bits of the PHY frame, packet data Da (1) consisting of 5 bits and packet data Db (1) consisting of 4 bits are packeted together with a 1-bit stop bit and a 2-bit packet address. 1 The stop bit indicates “0” and the packet address indicates “10”. Of the 34 bits of the PHY frame, packet data Da (2) consisting of 5 bits and packet data Db (2) consisting of 4 bits are packeted together with a 1-bit stop bit and a 2-bit packet address. 2 The stop bit indicates “0” and the packet address indicates “01”. Further, of the 34 bits of the PHY frame, the packet data Da (3) consisting of 4 bits and the packet data Db (3) consisting of 3 bits are packeted together with a 1-bit stop bit and a 4-bit packet address. 3 The stop bit indicates “1”, and the packet address indicates “1100”.

FIG. 329 is a diagram showing processing for dividing the PHY frame into 5 packets.

Of the 43 bits of the PHY frame, packet data Da (0) consisting of 5 bits and packet data Db (0) consisting of 4 bits are transferred to packet 0 together with a 1-bit stop bit and a 2-bit packet address. Can be stored. The stop bit indicates “0” and the packet address indicates “00”. Similarly, in packet 1 to packet 3, packet data Da consisting of 5 bits and packet data Db consisting of 4 bits are stored together with a 1-bit stop bit and a 2-bit packet address. The stop bits of these packets indicate “0”. Further, of the 34 bits of the PHY frame, packet data Da (4) consisting of 4 bits and packet data Db (4) consisting of 3 bits are packeted together with a 1-bit stop bit and a 4-bit packet address. 4 The stop bit indicates “1” and the packet address indicates “0010”.

FIG. 330 is a diagram showing processing for dividing a PHY frame into N (N = 6, 7 or 8) packets.

Of the (8N-1) bits of the PHY frame, 4-bit packet data Da (0) and 4-bit packet data Db (0) are a 1-bit stop bit and a 3-bit packet address. Together with packet 0. The stop bit indicates “0” and the packet address indicates “000”. Similarly, packet 1 to packet (N-2) also contain 4-bit packet data Da and 4-bit packet data Db together with a 1-bit stop bit and a 3-bit packet address. The stop bits of these packets indicate “0”. Further, of the (8N−1) bits of the PHY frame, the packet data Da (N−1) consisting of 4 bits and the packet data Db (N−1) consisting of 3 bits are expressed as 1 stop bit. A packet (N-1) is stored together with a 4-bit packet address. The stop bit indicates “1”.

FIG. 331 is a diagram showing a process of dividing the PHY frame into 9 packets.

Of the 71 bits of the PHY frame, 4-bit packet data Da (0) and 4-bit packet data Db (0) are transmitted in packet 0 together with a 1-bit stop bit and a 3-bit packet address. Can be stored. The stop bit indicates “0” and the packet address indicates “000”. Similarly, packet 1 to packet 7 also contain 4-bit packet data Da and 4-bit packet data Db together with a 1-bit stop bit and a 3-bit packet address. The stop bits of these packets indicate “0”. Further, out of 71 bits of the PHY frame, packet data Da (8) consisting of 4 bits and packet data Db (8) consisting of 3 bits are packeted together with 1 stop bit and 4 bit packet address. 8 The stop bit indicates “1” and the packet address indicates “0001”.

FIG. 332 is a diagram showing processing for dividing the PHY frame into N (N = 10 to 16) packets.

Of the 7N bits of the PHY frame, packet data Da (0) consisting of 4 bits and packet data Db (0) consisting of 3 bits are transferred to packet 0 together with a 1-bit stop bit and a 4-bit packet address. Can be stored. The stop bit indicates “0” and the packet address indicates “0000”. Similarly, packets 1 to (N-2) also contain 4-bit packet data Da and 3-bit packet data Db together with a 1-bit stop bit and a 4-bit packet address. The stop bits of these packets indicate “0”. Further, of 7N bits of the PHY frame, packet data Da (N-1) consisting of 4 bits and packet data Db (N-1) consisting of 3 bits are composed of 1 stop bit and 4 bits. It is stored in the packet (N-1) together with the packet address. The stop bit indicates “1”.

Also, when transmitting a large amount of data such as data (PHY frame) or stream data exceeding 112 bits, the transmitter sets the stop bit of the packet 15 to “0” without setting it to “1”. Then, the transmitter stores the data that could not be included in the packet 0 to the packet 15 out of the large amount of data described above and newly transmits it in each packet arranged from the packet 0. In other words, the transmitter stores data that could not be included in the packets 0 to 15 in each packet having a packet address starting from “0000” and transmits the data.

Like the PHY frame in mode 1, the PHY frame in mode 2 does not have MHR, and MFR is optional.

(Summary of Embodiment 24)
The visible light signal generation method according to Embodiment 24 is shown by the flowchart in FIG. 230A.

That is, this visible light signal generation method is a method for generating a visible light signal transmitted by a change in luminance of a light source provided in a transmitter, and includes steps SD1 to SD3.

In step SD1, a preamble which is data in which each of the first and second luminance values, which are different luminance values, appears alternately along the time axis is generated.

In step SD2, in the data in which the first and second luminance values appear alternately along the time axis, the time length in which each of the first and second luminance values continues is determined according to the signal to be transmitted. The first payload is generated by determining according to the first method.

Finally, in step SD3, a visible light signal is generated by combining the preamble and the first payload.

For example, as shown in FIGS. 316 to 318, the first and second luminance values are Bright (High) and Dark (Low), and the first data is PHY payload (PHY payload A or PHY payload B). ). By transmitting the visible light signal generated in this way, as shown in FIGS. 191 to 193, the number of received packets can be increased and the reliability can be increased. As a result, communication between various devices can be enabled.

In addition, the visible light signal generation method further sets a time length in which each of the first and second luminance values continues in the data in which the first and second luminance values appear alternately along the time axis. The second payload having a complementary relationship with the brightness expressed by the first payload may be generated by determining according to the second scheme according to the signal to be transmitted. In this case, in the generation of the visible light signal, the visible light signal is generated by combining the preamble and the first and second payloads in the order of the first payload, the preamble, and the second payload.

For example, as shown in FIGS. 316 and 317, the first and second luminance values are Bright (High) and Dark (Low), and the first and second payloads are PHY payload A and PHY payload B. It is.

Thereby, since the brightness of the first payload and the brightness of the second payload have a complementary relationship, the brightness can be kept constant regardless of the signal to be transmitted. Furthermore, since the first payload and the second payload are data obtained by modulating the same transmission target signal according to different systems, if the receiver receives only one of the payloads, the payload is transmitted to the transmission target. The signal can be demodulated. In addition, a header (SHR) that is a preamble is arranged between the first payload and the second payload. Therefore, if the receiver receives only a part on the rear side of the first payload, a header, and a part on the front side of the second payload, it can demodulate them into a signal to be transmitted. it can. Therefore, the visible light signal reception efficiency can be increased.

For example, the preamble is a header for the first and second payloads, in which the first luminance value of the first time length and the second luminance value of the second time length in that order, respectively. Luminance value appears. Here, the first time length is 100 μsec, and the second time length is 90 μsec. That is, as shown in FIG. 319, a pattern of time length (pulse width) of each pulse included in the header (SHR) in mode 1 of the packet PWM is defined.

The preamble is a header for the first and second payloads, and in the header, the first luminance value having the first time length, the second luminance value having the second time length, and the third time length. The respective luminance values appear in the order of the first luminance value and the second luminance value of the fourth time length. Here, the first time length is 100 μsec, the second time length is 90 μsec, the third time length is 90 μsec, and the fourth time length is 100 μsec. is there. That is, as shown in FIG. 319, a pattern of time length (pulse width) of each pulse included in the header (SHR) in mode 2 of the packet PWM is defined.

Thus, since the pattern of each header of the mode 1 and mode 2 of the packet PWM is defined, the receiver can appropriately receive the first and second payloads in the visible light signal.

The signal to be transmitted consists of 6 bits from the first bit x ₀ to the bit x ₅ sixth, in each of the first and second payload, a first luminance value of the third time length The respective luminance values appear in the order of the second luminance value of the fourth time length. Here, when the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4 (k is 0 or 1), in the generation of the first payload, in the first payload, Each of the third and fourth time lengths is determined according to the first method, which is the time length P _k = 120 + 30 × (7−y _k ) [μ seconds]. In the generation of the second payload, each of the third and fourth time lengths in the second payload is determined according to the second method, the time length P _k = 120 + 30 × y _k [μsec]. . In other words, as shown in FIG. 316, in the mode 1 of the packet PWM, the signal to be transmitted is the time of each pulse included in each of the first payload (PHY payload A) and the second payload (PHY payload B). Modulated as long (pulse width).

The signal to be transmitted consists of 12 bits from the first bit x ₀ to bit x ₁₁ of the 12, in each of the first and second payload, a first luminance value of the fifth duration The respective luminance values appear in the order of the second luminance value having the sixth time length, the first luminance value having the seventh time length, and the second luminance value having the eighth time length. Here, when the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4 (k is 0, 1, 2, or 3), in the generation of the first payload, Each of the fifth to eighth time lengths in one payload is determined in accordance with the first method of time length P _k = 120 + 30 × (7−y _k ) [μ seconds]. Further, in the generation of the second payload, each of the fifth to eighth time lengths in the second payload is determined according to the second method, the time length P _k = 120 + 30 × y _k [μsec]. . That is, as shown in FIG. 317, in the mode 2 of the packet PWM, the time of each pulse included in each of the first payload (PHY payload A) and the second payload (PHY payload B) is the signal to be transmitted. Modulated as long (pulse width).

As described above, in the mode 1 and the mode 2 of the packet PWM, since the transmission target signal is modulated as the pulse width of each pulse, the receiver appropriately transmits the visible light signal based on the pulse width. The signal can be demodulated.

The preamble is a header for the first payload, and in the header, the first luminance value having the first time length, the second luminance value having the second time length, and the first luminance having the third time length. The luminance values appear in the order of the luminance value of the second and the second luminance value of the fourth time length. Here, the first time length is 50 μsec, the second time length is 40 μsec, the third time length is 40 μsec, and the fourth time length is 50 μsec. is there. That is, as shown in FIG. 319, a pattern of time length (pulse width) of each pulse included in the header (SHR) in mode 3 of the packet PWM is defined.

Thus, since the pattern of the header of the mode 3 of the packet PWM is defined, the receiver can appropriately receive the first payload in the visible light signal.

The signal to be transmitted consists of 3n bits from the first bit x ₀ through bit x _3n-1 of the 3n (n is an integer of 2 or more), the time length of the first payload, a respective one Alternatively, it consists of first to nth time lengths in which the second luminance value continues. Here, when the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4 (k is an integer from 0 to (n−1)), the first payload is generated. Then, each of the first to n-th time lengths in the first payload is determined according to the first method, the time length P _k = 100 + 20 × y _k [μsec]. In other words, as shown in FIG. 318, in the mode 3 of the packet PWM, the signal to be transmitted is modulated as the time length (pulse width) of each pulse included in the first payload (PHY payload).

As described above, in the mode 3 of the packet PWM, the signal to be transmitted is modulated as the pulse width of each pulse. Therefore, the receiver appropriately demodulates the visible light signal to the signal to be transmitted based on the pulse width. can do.

FIG. 333A is a flowchart showing another visible light signal generation method according to Embodiment 24. This visible light signal generation method is a method for generating a visible light signal transmitted by a change in luminance of a light source provided in a transmitter, and includes steps SE1 to SE3.

In step SE1, a preamble which is data in which each of the first and second luminance values, which are different luminance values, appears alternately along the time axis is generated.

In step SE2, in the data in which the first and second luminance values appear alternately along the time axis, an interval from when the first luminance value appears until the next first luminance value appears is set as a transmission target. 1st payload is produced | generated by determining according to the system according to this signal.

In step SE3, a visible light signal is generated by combining the preamble and the first payload.

FIG. 333B is a block diagram showing a configuration of another signal generating apparatus according to Embodiment 24. The signal generation device E10 is a signal generation device that generates a visible light signal transmitted by the luminance change of the light source provided in the transmitter, and includes a preamble generation unit E11, a payload generation unit E12, and a combining unit E13. . Further, the signal generation device E10 executes the processing of the flowchart shown in FIG. 333A.

That is, the preamble generation unit E11 generates a preamble that is data in which the first and second luminance values, which are different luminance values, appear alternately on the time axis.

In the data in which the first and second luminance values appear alternately along the time axis, the payload generation unit E12 sets an interval from when the first luminance value appears until the next first luminance value appears. A first payload is generated by determining according to a method according to a signal to be transmitted.

The combining unit E13 generates a visible light signal by combining the preamble and the first payload.

For example, as shown in FIGS. 320 to 322, the first and second luminance values are Bright (High) and Dark (Low), and the first payload is a PHY payload. By transmitting the visible light signal generated in this way, as shown in FIGS. 191 to 193, the number of received packets can be increased and the reliability can be increased. As a result, communication between various devices can be enabled.

For example, the time length of the first luminance value in each of the preamble and the first payload is 10 μsec or less.

This makes it possible to suppress the average luminance of the light source while performing visible light communication.

Also, the preamble is a header for the first payload, and the time length of the header includes three intervals from when the first luminance value appears until the next first luminance value appears. Here, each of the three intervals is 160 μs. That is, as shown in FIG. 323, an interval pattern between pulses included in the header (SHR) in mode 1 of the packet PPM is defined. Each of the pulses is a pulse having a first luminance value, for example.

Also, the preamble is a header for the first payload, and the time length of the header includes three intervals from when the first luminance value appears until the next first luminance value appears. Here, the first interval among the three intervals is 160 μsec, the second interval is 180 μsec, and the third interval is 160 μsec. That is, as shown in FIG. 323, an interval pattern between pulses included in the header (SHR) in mode 2 of the packet PPM is defined.

Also, the preamble is a header for the first payload, and the time length of the header includes three intervals from when the first luminance value appears until the next first luminance value appears. Here, the first one of the three intervals is 80 μs, the second interval is 90 μs, and the third interval is 80 μs. That is, as shown in FIG. 323, an interval pattern between pulses included in the header (SHR) in mode 3 of the packet PPM is defined.

Thus, since the header patterns of the mode 1, mode 2 and mode 3 of the packet PPM are defined in this way, the receiver can appropriately receive the first payload in the visible light signal.

The signal to be transmitted consists of 6 bits from the first bit x ₀ to the bit x ₅ in the sixth, the time length of the first payload, first from the first luminance value appears in the following 1 Two intervals until the brightness value of 2 appears. Here, when the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4 (k is 0 or 1), in the generation of the first payload, in the first payload, Each of the two intervals is determined according to the interval P _k = 180 + 30 × y _k [μsec], which is the above-described method. That is, as shown in FIG. 320, in the mode 1 of the packet PPM, the transmission target signal is modulated as an interval between each pulse included in the first payload (PHY payload).

The signal to be transmitted consists of 12 bits from the first bit x ₀ to bit x ₁₁ of the 12, the time length of the first payload, first from the first luminance value appears in the following 1 This includes four intervals until the luminance value appears. Here, when the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4 (k is 0, 1, 2, or 3), in the generation of the first payload, Each of the four intervals in one payload is determined according to the interval P _k = 180 + 30 × y _k [μsec], which is the above-described method. That is, as shown in FIG. 321, in mode 2 of the packet PPM, the signal to be transmitted is modulated as an interval between each pulse included in the first payload (PHY payload).

The signal to be transmitted consists of 3n bits from the first bit x ₀ through bit x _3n-1 of the 3n (n is an integer of 2 or more), the time length of the first payload, a first It includes n intervals from when the luminance value appears until the next first luminance value appears. Here, when the parameter y _k is expressed as y _k = x _3k + x _{3k + 1} × 2 + x _{3k + 2} × 4 (k is an integer from 0 to (n−1)), the first payload is generated. Then, each of the n intervals in the first payload is determined according to the interval P _k = 100 + 20 × y _k [μsec], which is the above-described method. That is, as shown in FIG. 322, in the mode 3 of the packet PPM, the signal to be transmitted is modulated as an interval between each pulse included in the first payload (PHY payload).

As described above, in the mode 1, mode 2 and mode 3 of the packet PPM, the signal to be transmitted is modulated as an interval between each pulse, so that the receiver appropriately transmits a visible light signal based on the interval. It can be demodulated into the signal of interest.

In the visible light signal generation method, a footer for the first payload may be further generated. In the generation of the visible light signal, the footer may be combined next to the first payload. That is, as shown in FIGS. 318 and 322, in mode 3 of the packet PWM and the packet PPM, a footer (SFT) is transmitted following the first payload (PHY payload). Thereby, since the end of the first payload can be clearly specified by the footer, visible light communication can be performed efficiently.

In addition, when generating a visible light signal, if a footer is not transmitted, a header for a signal next to a transmission target signal may be combined instead of the footer. That is, in the mode 3 of the packet PWM and the packet PPM, instead of the footer (SFT) shown in FIGS. 318 and 322, the header (SHR) for the next first payload is followed by the first payload (PHY payload). ) Is sent. Thus, the end of the first payload can be clearly specified by the header for the next first payload, and since the footer is not transmitted, visible light communication can be performed more efficiently.

The configuration of the signal generation device according to Embodiment 24 is shown by the block diagram in FIG. 230B.

That is, the signal generation device D10 according to the twenty-fourth embodiment is a signal generation device that generates a visible light signal transmitted by a change in luminance of a light source included in a transmitter, and includes a preamble generation unit D11, a data generation unit D12, and the like. And a connecting part D13.

The preamble generation unit D11 generates a preamble that is data in which the first and second luminance values, which are different from each other, appear alternately along the time axis.

In the data in which the first and second luminance values appear alternately along the time axis, the data generation unit D12 determines the time length that each of the first and second luminance values continues according to the transmission target signal. By determining according to the first method, the first payload is generated.

The combining unit D13 generates a visible light signal by combining the preamble and the first payload.

By transmitting the visible light signal generated by the signal generation device D10, the number of received packets can be increased and the reliability can be increased as shown in FIGS. 191 to 193. As a result, communication between various devices can be enabled.

In each of the above-described embodiments and modifications, each component may be configured by dedicated hardware or may be realized by executing a software program suitable for each component. Each component may be realized by a program execution unit such as a CPU or a processor reading and executing a software program recorded on a recording medium such as a hard disk or a semiconductor memory. For example, the program causes a computer to execute the visible light signal generation method shown by the flowcharts of FIGS. 230A and 333A.

As described above, the visible light signal generation method according to one or a plurality of aspects has been described based on the above-described embodiments and modifications. However, the present invention is not limited to the embodiments. Unless it deviates from the gist of the present invention, various modifications conceived by those skilled in the art have been made in the present embodiment, and forms constructed by combining components in different embodiments and modifications are also within the scope of the present invention. May be included.

(Embodiment 25)
In this embodiment, a visible light signal decoding method, an encoding method, and the like will be described.

FIG. 334 is a diagram showing a format of a MAC frame in MPM.

The format of the MAC (medium access control) header in MPM (Mirror Pulse Modulation) consists of MHR (medium access control header) and MSDU (medium access control service-data unit). The MHR field includes a sequence number subfield. The MSDU includes a frame payload and has a variable length. The bit length of MPDU (medium access control control protocol-data unit) including MHR and MSDU is set as macMpmMpduLength.

Note that MPM is a modulation scheme in the twentieth and twenty-fourth embodiments. For example, as shown in FIGS. 188 to 189B, 197 to 230B, and 315 to 332, information to be transmitted is provided. Alternatively, the signal is modulated.

FIG. 335 is a flowchart showing the processing operation of the encoding device for generating a MAC frame in MPM. Specifically, FIG. 335 is a diagram illustrating how to determine the bit length of the sequence number subfield. Note that the encoding device is provided in, for example, the above-described transmitter or transmission device that transmits a visible light signal.

The sequence number subfield includes a frame sequence number (also called a sequence number). The bit length of the sequence number subfield is set as macMpmSnLength. When the bit length of the sequence number subfield is set to a variable length, the first bit in the sequence number subfield is used as the last frame flag. That is, in this case, the sequence number subfield includes a final frame flag and a bit string indicating the sequence number. The final frame flag is set to 1 for the final frame, and is set to 0 for the other frames. That is, the final frame flag indicates whether or not the processing target frame is the final frame. This final frame flag corresponds to the above-described stop bit. The sequence number corresponds to the above address.

First, the encoding device determines whether or not SN is set to a variable length (step S101a). SN is the bit length of the sequence number subfield. That is, the encoding apparatus determines whether macMpmSnLength indicates 0xf. When macMpmSnLength indicates 0xf, SN is a variable length, and when macMpmSnLength indicates other than 0xf, SN is a fixed length. When determining that the SN is not set to a variable length, that is, the SN is set to a fixed length (N in step S101a), the encoding apparatus determines the SN to a value indicated by macMpmSnLength (step S102a). . At this time, the encoding apparatus does not use the final frame flag (that is, LFF).

On the other hand, when determining that the SN is set to a variable length (Y in step S101a), the encoding apparatus determines whether the processing target frame is the last frame (step S103a). Here, when the encoding apparatus determines that the processing target frame is the final frame (Y in step S103a), the encoding apparatus determines SN to be 5 bits (step S104a). At this time, the encoding apparatus determines a final frame flag indicating 1 as the first bit in the sequence number subfield.

Further, when the encoding apparatus determines that the processing target frame is not the final frame (N in step S103a), the encoding apparatus determines whether the value of the sequence number of the final frame is 1-15 (step S105a). The sequence number is an integer assigned to each frame in ascending order from 0. In the case of N in step S103a, the number of frames is 2 or more. Therefore, in this case, the value of the sequence number of the last frame can take any of 1-15 except 0.

If the encoding apparatus determines in step S105a that the value of the sequence number of the final frame is 1, it determines SN as 1 bit (step S106a). At this time, the encoding apparatus determines that the value of the last frame flag which is the first bit in the sequence number subfield is 0.

For example, when the sequence number value of the last frame is 1, the sequence number subfield of the last frame is represented as (1, 1) including the last frame flag (1) and the sequence number value (1). . At this time, the encoding apparatus determines that the bit length of the sequence number subfield of the processing target frame is 1 bit. That is, the encoding apparatus determines a sequence number subfield including only the last frame flag (0).

If the encoding device determines in step S105a that the value of the sequence number of the last frame is 2, it determines SN to 2 bits (step S107a). Also at this time, the encoding apparatus determines the value of the final frame flag as 0.

For example, when the sequence number value of the last frame is 2, the sequence number subfield of the last frame is represented as (1, 0, 1) including the last frame flag (1) and the sequence number value (2). Is done. The sequence number is indicated by a bit string. In the bit string, the leftmost bit is LSB (leastlesignificant bit) and the rightmost bit is MSB (most significant bit). Therefore, the value (2) of the sequence number is expressed as a bit string (0, 1). Thus, when the value of the sequence number of the last frame is 2, the encoding apparatus determines the bit length of the sequence number subfield of the processing target frame to be 2 bits. That is, the encoding apparatus determines the sequence number subfield including the last frame flag (0) and the bit (0) or (1) indicating the sequence number.

If the encoding device determines in step S105a that the value of the sequence number of the final frame is 3 or 4, it determines SN to 3 bits (step S108a). Also at this time, the encoding apparatus determines the value of the final frame flag as 0.

If the encoding apparatus determines in step S105a that the value of the sequence number of the last frame is any integer of 5-8, it determines SN as 4 bits (step S109a). Also at this time, the encoding apparatus determines the value of the final frame flag as 0.

When the encoding apparatus determines in step S105a that the sequence number value of the final frame is any integer of 9-15, it determines SN as 5 bits (step S110a). Also at this time, the encoding apparatus determines the value of the final frame flag as 0.

FIG. 336 is a flowchart showing the processing operation of the decoding device for decoding the MAC frame in MPM. Specifically, FIG. 336 is a diagram illustrating how to determine the bit length of the sequence number subfield. Note that the decoding device is provided, for example, in the above-described receiver or receiving device that receives a visible light signal.

Here, the decoding apparatus determines whether or not SN is set to a variable length (step S201a). That is, the decoding apparatus determines whether macMpmSnLength indicates 0xf. If the decoding apparatus determines that SN is not set to a variable length, that is, that SN is set to a fixed length (N in step S201a), SN is determined to be a value indicated by macMpmSnLength (step S202a). At this time, the decoding device does not use the final frame flag (ie, LFF).

On the other hand, when the decoding apparatus determines that the SN is set to a variable length (Y in step S201a), the decoding apparatus determines whether the value of the final frame flag of the decoding target frame is 1 or 0 (step S203a). . That is, the decoding apparatus determines whether or not the decoding target frame is the last frame. Here, when the decoding apparatus determines that the value of the final frame flag is 1 (1 in step S203a), it determines SN to be 5 bits (step S204a).

When the decoding apparatus determines that the value of the last frame flag is 0 (0 in step S203a), the value indicated by the bit string from the second bit to the fifth bit in the sequence number subfield of the last frame is 1. It is determined which of −15 (step S205a). The final frame has a final frame flag indicating 1 and is a frame generated from the same source as the decoding target frame. Each source is specified by a position in the captured image. The source is divided into a plurality of frames (that is, packets) as shown in FIGS. 325 to 332, for example. That is, the last frame is the last frame among a plurality of frames generated by dividing one source. The value indicated by the bit string from the second bit to the fifth bit in the sequence number subfield is the value of the sequence number.

If the decoding apparatus determines in step S205a that the value indicated by the bit string is 1, it determines the SN to be 1 bit (step S206a). For example, when the sequence number subfield of the last frame is 2 bits of (1, 1), the last frame flag is 1, and the sequence number of the last frame, that is, the value indicated by the bit string is 1. At this time, the decoding apparatus determines that the bit length of the sequence number subfield of the decoding target frame is 1 bit. That is, the decoding apparatus determines (0) as the sequence number subfield of the decoding target frame.

If the decoding apparatus determines in step S205a that the value indicated by the bit string is 2, the decoding apparatus determines SN to be 2 bits (step S207a). For example, if the sequence number subfield of the last frame is 3 bits (1, 0, 1), the last frame flag is 1, and the sequence number of the last frame, that is, the value indicated by the bit string (0, 1). Is 2. In the above bit string, the leftmost bit is LSB (least significant bit) and the rightmost bit is MSB (most） significant bit).

At this time, the decoding apparatus determines that the bit length of the sequence number subfield of the decoding target frame is 2 bits. That is, the decoding apparatus determines the sequence number subfield of the decoding target frame to be (0, 0) or (0, 1).

If it is determined in step S205a that the value indicated by the bit string is 3 or 4, the decoding apparatus determines SN to be 3 bits (step S208a).

If the decoding apparatus determines in step S205a that the value indicated by the bit string is any integer of 5-8, it determines SN as 4 bits (step S209a).

When the decoding apparatus determines in step S205a that the value indicated by the bit string is any integer of 9-15, it determines SN as 5 bits (step S210a).

FIG. 337 is a diagram showing the attributes of the MAC PIB.

The MAC PIB (physical-layer personal-area-network information base) attributes include macMpmSnLength and macMpmMpduLength. macMpmSnLength is any integer value in the range of 0x0-0xf, and indicates the bit length of the sequence number subfield. Specifically, when macMpmSnLength is any integer value in the range of 0x0-0xe, the integer value is indicated as a fixed bit length of the sequence number subfield. When macMpmSnLength is 0xf, it indicates that the bit length of the sequence number subfield is variable.

MacMpmMpduLength is any integer value in the range of 0x00-0xff and indicates the bit length of the MPDU.

FIG. 338 is a diagram for explaining an MPM dimming method.

MPM has a dimming function. Examples of the MPM dimming method include (a) analog dimming method, (b) PWM dimming method, (c) VPPM dimming method, and (d) field insertion dimming method shown in FIG. .

In the analog dimming method, for example, as shown in (a2), the visible light signal is transmitted by changing the luminance. Here, when darkening the visible light signal, for example, as shown in (a1), the overall luminance of the visible light signal is lowered. Conversely, when the visible light signal is brightened, for example, as shown in (a3), the overall luminance of the visible light signal is increased.

In the PWM dimming method, for example, as shown in (b2), the visible light signal is transmitted by changing the luminance. Here, when darkening the visible light signal, for example, as shown in (b1), the luminance is lowered for a short period in the period in which the high-luminance light shown in (b2) is output. On the contrary, when the visible light signal is brightened, for example, as shown in (b3), the luminance is raised only for a short period in the period when the low-luminance light shown in (b2) is output. It should be noted that the short period described above must be less than 1/3 of the original pulse width and less than 50 μsec.

In the VPPM dimming method, for example, as shown in (c2), the visible light signal is transmitted by changing the luminance. Here, when darkening the visible light signal, for example, as shown in (c1), the timing of the fall of the brightness is advanced. On the other hand, when the visible light signal is brightened, for example, as shown in (c3), the timing of the fall of the luminance is delayed. The VPPM modulation method can be used only for the PHY PPM mode in MPM.

In the field insertion dimming method, for example, as shown in (d2), a visible light signal including a plurality of PPDUs (physical-layer data units) is transmitted. Here, when darkening the visible light signal, for example, as shown in (d1), a dimming field having a luminance lower than the luminance of the PPDU is inserted between the PPDUs. On the other hand, when the visible light signal is brightened, for example, as shown in (d3), a dimming field having a luminance higher than that of the PPDU is inserted between the PPDUs.

FIG. 339 is a diagram showing attributes of the PHY PIB.

PHY (physical layer) PIB attributes include phyMpmMode, phyMpmPlcpHeaderMode, phyMpmPlcpCenterMode, phyMpmSymbolSize, phyMpmOddSymbolBit, phyMpmEvenSymbolBit, phyMpmSymbolOffset, and phyMpmSymbolUnit.

PhyMpmMode is 0 or 1, and indicates the PHY mode of MPM. Specifically, when phyMpmMode is 0, it indicates that the PHY mode is the PWM mode, and when it is 1, it indicates that the PHY mode is the PWM mode.

PhyMpmPlcpHeaderMode is any integer value in the range of 0x0-0xf, and indicates a PLCP (Physical Layer Conversion Protocol) header subfield mode and a PLCP footer subfield mode.

PhyMpmPlcpCenterMode is any integer value in the range of 0x0-0xf and indicates the PLCP center subfield mode.

PhyMpmSymbolSize is any integer value in the range of 0x0-0xf, and indicates the number of symbols in the payload subfield. Specifically, when phyMpmSymbolSize is 0x0, it indicates that the number of symbols is variable, and is referred to as N.

phyMpmOddSymbolBit is an any integer value in the range of up to OxO-Oxf, shows the bit length included in each odd symbol in the payload field and are referred to as M _odd.

phyMpmEvenSymbolBit is any integer value in the range of 0x0-0xf, indicates the bit length included in each even symbol of the payload subfield, and is referred to as M _even .

phyMpmSymbolOffset is an any integer value in the range of up 0x00-0xFF, it indicates the offset value of the symbol of the payload field and are referred to as W _1.

phyMpmSymbolUnit is an any integer value in the range of up 0x00-0xFF, indicates the unit value of the symbol of the payload field and are referred to as W _2.

FIG. 340 is a diagram for explaining the MPM.

MPM consists only of PSDU (PHY service data unit) fields. The PSDU field includes an MPDU converted by the MPM PLCP.

MPM PLCP converts MPDU into 5 subfields as shown in FIG. The five subfields are a PLCP header subfield, a front payload subfield, a PLCP center subfield, a back payload subfield, and a PLCP footer subfield. The PHY mode of MPM is set as phyMpmMode.

As shown in FIG. 340, the MPM PLCP includes a bit rearrangement unit 301a, a duplication unit 302a, a front conversion unit 303a, and a back conversion unit 304a.

Here, (x ₀ , x ₁ , x ₂ ,...) Is each bit included in the MPDU, L _SN is the bit length of the sequence number subfield, and N is each payload subfield. The number of symbols. The bit rearrangement unit 301a rearranges (x ₀ , x ₁ , x ₂ ,...) To (y ₀ , y ₁ , y ₂ ,...) According to the following (Equation 1).

This relocation, each bit included in the sequence number subfield at the head of the MPDU, it moves rearward only L _SN. The duplication unit 302a duplicates the MPDU after the bit rearrangement.

Each of the front payload subfield and the back payload subfield consists of N symbols. Here, M _odd is the bit length included in the odd-numbered symbol, M _even is the bit length included in the even-numbered symbol, and W ₁ is the symbol value offset (the above-described offset value). , _{W 2} is the symbol value unit (unit value described above). N, M _odd , M _even , W ₁ , and W ₂ are set by the PHY PIB shown in FIG. 339.

The front conversion unit 303a and the back conversion unit 304a convert payload bits (y0, y1, y2,...) Of the rearranged MPDU into z _i according to the following (Expression 2) to (Expression 5).

The front conversion unit 303a calculates the i-th symbol (that is, the symbol value) of the front payload subfield using z _i by the following (Equation 6).

The back conversion unit 304a calculates the i-th symbol (that is, the symbol value) of the back payload subfield using z _{i according} to (Equation 7) below.

Note that the symbol values calculated by (Expression 6) and (Expression 7) correspond to the time lengths D _R1 to D _R4 and D _L1 to D _L4 shown in FIG. 188, for example.

FIG. 341 is a diagram showing a PLCP header subfield.

As shown in FIG. 341, the PLCP header subfield is composed of four symbols in the PWM mode, and is composed of three symbols in the PPM mode.

FIG. 342 is a diagram showing a PLCP center subfield.

As shown in FIG. 342, the PLCP center subfield is composed of four symbols in the PWM mode, and is composed of three symbols in the PPM mode.

FIG. 343 is a diagram showing a PLCP footer subfield.

As shown in FIG. 343, the PLCP footer subfield is composed of four symbols in the PWM mode, and is composed of three symbols in the PPM mode.

FIG. 344 is a diagram illustrating a PHY PWM mode waveform in the MPM.

In PWM mode, a symbol must be transmitted as one of two states of light intensity: bright or dark. In the PHY PWM mode in MPM, the symbol value corresponds to a continuous time in microseconds. For example, as shown in FIG. 344, the first symbol value corresponds to the continuous time of the first bright state, and the second symbol value corresponds to the continuous time of the next dark state. In the example shown in FIG. 344, the initial state of each subfield is a bright state, but may be a dark state.

FIG. 345 is a diagram showing a PHY PPM mode waveform in MPM.

In the PPM mode, as shown in FIG. 345, the symbol value represents the time from the start of the bright state to the start of the next bright state in units of microseconds. The bright state time must be shorter than 90% of the symbol value.

For both modes, the transmitter can transmit only some of the symbols. However, the transmitter must transmit all the symbols of the PLCP center subfield and at least N symbols. Each of the at least N symbols is a symbol included in either the front payload subfield or the back payload subfield.

(Summary of Embodiment 25)
FIG. 346 is a flowchart illustrating an example of the decoding method according to the twenty-fifth embodiment. Note that the flowchart shown in FIG. 346 corresponds to the flowchart shown in FIG.

This decoding method is a method for decoding a visible light signal composed of a plurality of frames, and includes step S310b, step S320b, and step S330b as shown in FIG. Each of the plurality of frames includes a sequence number and a frame payload.

In step S310b, based on macSnLength which is information for determining the bit length of the subfield in which the sequence number is stored in the decoding target frame, it is determined whether or not the bit length of the subfield is variable Process.

In step S320b, the bit length of the subfield is determined based on the result of the variable length determination process. In step S330b, the decoding target frame is decoded based on the determined bit length of the subfield.

Here, the determination of the bit length of the subfield in step S320b includes steps S321b to S324b.

That is, in the variable length determination process of step S310b, when it is determined that the bit length of the subfield is not variable, the bit length of the subfield is determined to the value indicated by the above macSnLength (step S321b). ).

On the other hand, if it is determined in the variable length determination process in step S310b that the bit length of the subfield is variable length, it is determined whether or not the decoding target frame is the last frame of the plurality of frames. A final determination process is performed (step S322b). If it is determined that the frame is the final frame (Y in step S322b), the bit length of the subfield is determined to be a predetermined value (step S323b). On the other hand, if it is determined that the frame is not the last frame (N in step S322b), the bit length of the subfield is determined based on the sequence number value of the last frame (step S324b).

As a result, as shown in FIG. 346, regardless of whether the bit length of the subfield in which the sequence number is stored (specifically, the sequence number subfield) is fixed or variable, the subfield The bit length can be determined appropriately.

Here, in the final determination process in step S322b, it may be determined whether the decoding target frame is the final frame based on the final frame flag indicating whether the decoding target frame is the final frame. Specifically, in the final determination process in step S322b, when the final frame flag indicates 1, when it is determined that the decoding target frame is the final frame, and when the final frame flag indicates 0, the decoding target frame May not be the last frame. For example, the last frame flag may be included in the first bit of the subfield.

Thereby, as shown in step S203a of FIG. 336, it is possible to appropriately determine whether or not the decoding target frame is the final frame.

More specifically, in the determination of the bit length of the subfield in step S320b, if it is determined in the final determination process in step S322b that the decoding target frame is the final frame, the bit length of the subfield is set as described above. The predetermined value of 5 bits may be determined. That is, as shown in step S204a of FIG. 336, the bit length SN of the subfield is determined to be 5 bits.

Further, in the determination of the bit length of the subfield in step S320b, when it is determined in the final determination process in step S322b that the decoding target frame is not the final frame, the sequence number value of the final frame is 1. The bit length of the subfield may be determined as 1 bit. Further, when the value of the sequence number of the last frame is 2, the bit length of the subfield may be determined to be 2 bits. Further, when the value of the sequence number of the last frame is 3 or 4, the bit length of the subfield may be determined to be 3 bits. Further, when the value of the sequence number of the last frame is any integer from 5 to 8, the bit length of the subfield may be determined to be 4 bits. Further, when the value of the sequence number of the last frame is any integer from 9 to 15, the bit length of the subfield may be determined to be 5 bits. That is, as shown in steps S206a to S210a in FIG. 336, the bit length SN of the subfield is determined to be any one of 1 to 5 bits.

FIG. 347 is a flowchart illustrating an example of the encoding method according to the twenty-fifth embodiment. Note that the flowchart shown in FIG. 347 corresponds to the flowchart shown in FIG.

This encoding method is a method of encoding information to be encoded into a visible light signal composed of a plurality of frames. As shown in FIG. 347, step S410a, step S420a, step S430a, including. Each of the plurality of frames includes a sequence number and a frame payload.

In step S410a, based on macSnLength which is information for determining the bit length of the subfield in which the sequence number is stored in the processing target frame, it is determined whether or not the bit length of the subfield is variable. Process.

In step S420a, the bit length of the subfield is determined based on the result of the variable length determination process. In step S430a, based on the determined bit length of the subfield, a part of the encoding target information is encoded into the processing target frame.

Here, the determination of the bit length of the subfield in step S420a includes steps S421a to S424a.

That is, in the variable length determination process in step S410a, when it is determined that the bit length of the subfield is not variable length, the bit length of the subfield is determined to the value indicated by the above-described macSnLength (step S421a). ).

On the other hand, if it is determined in the variable length determination process in step S410a that the bit length of the subfield is variable length, it is determined whether or not the processing target frame is the last frame of the plurality of frames. A final determination process is performed (step S422a). If it is determined that the frame is the last frame (Y in step S422a), the bit length of the subfield is determined to be a predetermined value (step S423a). On the other hand, when it is determined that the frame is not the last frame (N in step S422a), the bit length of the subfield is determined based on the sequence number value of the last frame (step S424a).

As a result, as shown in FIG. 347, regardless of whether the bit length of the subfield in which the sequence number is stored (specifically, the sequence number subfield) is fixed or variable, the subfield The bit length can be determined appropriately.

Note that the decoding device according to the present embodiment includes a processor and a memory, and a program that causes the processor to execute the decoding method shown in FIG. 346 is recorded in the memory. The encoding apparatus in the present embodiment includes a processor and a memory, and a program that causes the processor to execute the encoding method shown in FIG. 347 is recorded in the memory. The program in this embodiment is a program that causes a computer to execute the decoding method shown in FIG. 346 or the encoding method shown in FIG. 347.

(Embodiment 26)
In this embodiment, a transmission method for transmitting an optical ID by a visible light signal will be described. Note that the transmitter and the receiver in this embodiment may have the same functions and configurations as the transmitter (or the transmission device) and the receiver (or the reception device) in each of the above embodiments.

FIG. 348 is a diagram illustrating an example in which the receiver according to the present embodiment displays an AR image.

The receiver 200 in the present embodiment is a receiver including an image sensor and a display 201, and is configured as a smartphone, for example. Such a receiver 200 acquires the above-described captured display image Pa, which is the normal captured image, and the above-described decoding image, which is the visible light communication image or the bright line image, by capturing the subject with the image sensor.

Specifically, the image sensor of the receiver 200 images the transmitter 100. The transmitter 100 has a shape such as a light bulb, for example, and includes a glass bulb 141 and a light emitting unit 142 that sways while shining like a flame inside the glass bulb 141. The light emitting unit 142 emits light when one or more light emitting elements (for example, LEDs) provided in the transmitter 100 are turned on. The transmitter 100 changes in luminance by blinking the light emitting unit 142, and transmits an optical ID (optical identification information) by the luminance change. This light ID is the above-mentioned visible light signal.

The receiver 200 captures the transmitter 100 with the normal exposure time, thereby acquiring the captured display image Pa projected by the transmitter 100, and the transmitter 100 with a communication exposure time shorter than the normal exposure time. A decoding image is acquired by imaging. The normal exposure time is the exposure time in the above-described normal photographing mode, and the communication exposure time is the exposure time in the above-described visible light communication mode.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P42 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pa as a target area. Then, the receiver 200 superimposes the AR image P42 on the target area, and displays the captured display image Pa on which the AR image P42 is superimposed on the display 201.

For example, similarly to the example shown in FIG. 245, the receiver 200 recognizes the area at the upper left of the area where the transmitter 100 is projected as the target area according to the recognition information. As a result, for example, the AR image P <b> 42 showing a fairy is displayed so as to fly around the transmitter 100.

FIG. 349 is a diagram illustrating another example of the captured display image Pa on which the AR image P42 is superimposed.

As shown in FIG. 349, the receiver 200 displays the captured display image Pa on which the AR image P42 is superimposed on the display 201.

Here, the above-described recognition information indicates that a range having a luminance equal to or higher than a threshold in the captured display image Pa is a reference region. Further, the recognition information indicates that the target area is in a predetermined direction with respect to the reference area, and that the target area is separated from the center (or center of gravity) of the reference area by a predetermined distance. Show.

Therefore, when the light emitting unit 142 of the transmitter 100 imaged by the receiver 200 fluctuates, the AR image p42 superimposed on the target region of the captured display image Pa also moves as shown in FIG. Move to sync. That is, when the light emitting unit 142 fluctuates, the image 142a of the light emitting unit 142 displayed in the captured display image Pa also fluctuates. This image 142a is a range having a luminance equal to or higher than the above-described threshold and is a reference area. That is, since the reference area moves, the receiver 200 moves the target area so that the distance between the reference area and the target area is maintained at a predetermined distance. The AR image P42 is superimposed. As a result, when the light emitting unit 142 shakes, the AR image P42 superimposed on the target area of the captured display image Pa also moves in synchronization with the movement of the light emitting unit 142. Note that the center position of the reference region may move even when the light emitting unit 142 is deformed. Therefore, even when the light emitting unit 142 is deformed, the AR image 42 may move so that the distance from the center position of the moving reference area is maintained at a predetermined distance.

In the above example, the receiver 200 recognizes the target area based on the recognition information and superimposes the AR image P42 on the target area. However, the AR image P42 may be vibrated around the target area. . That is, the receiver 200 vibrates the AR image P42 in the vertical direction, for example, according to a function indicating the change in amplitude with respect to time. The function is a trigonometric function such as a sine wave.

In addition, the receiver 200 may change the size of the AR image P42 according to the size of a range having a luminance equal to or higher than the above-described threshold. That is, the receiver 200 increases the size of the AR image P42 as the area of the bright region in the captured display image Pa increases, and conversely decreases the size of the AR image P42 as the area of the bright region decreases.

Alternatively, the receiver 200 increases the size of the AR image P42 as the average luminance in the range having the luminance equal to or higher than the above-described threshold is increased, and conversely decreases the size of the AR image P42 as the average luminance is lower. May be. Instead of the size of the AR image P42, the transparency of the AR image P42 may be changed according to the average luminance.

In the example shown in FIG. 349, in the image 142a of the light emitting unit 142, any pixel has a luminance equal to or higher than a threshold value, but any pixel may be less than the threshold value. That is, the range corresponding to the image 142a and having a luminance equal to or higher than the threshold value may be annular. Also in this case, a range having a luminance equal to or higher than the threshold is specified as the reference region, and the AR image P42 is superimposed on a target region that is separated from the center (or centroid) of the reference region by a predetermined distance.

FIG. 350 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

For example, as shown in FIG. 350, the transmitter 100 is configured as an illuminating device, and transmits a light ID by changing the luminance while illuminating a figure 143 composed of, for example, three circles drawn on a wall. Since the figure 143 is illuminated by the light from the transmitter 100, the luminance changes in the same manner as the transmitter 100, and the light ID is transmitted.

The receiver 200 acquires the captured display image Pa and the decoding image in the same manner as described above by capturing the figure 143 illuminated by the transmitter 100. The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the graphic 143. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P43 and the recognition information corresponding to the optical ID from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pa as a target area. For example, the receiver 200 recognizes an area where the graphic 143 is projected as a target area. Then, the receiver 200 superimposes the AR image P43 on the target area, and displays the captured display image Pa on which the AR image P43 is superimposed on the display 201. For example, the AR image P43 is a character face image.

Here, the figure 143 is composed of three circles as described above, but the figure 143 has few geometric features. Therefore, it is difficult to appropriately select and acquire an AR image corresponding to the graphic 143 from many images stored in the server from only the captured image obtained by imaging the graphic 143. However, in the present embodiment, the receiver 200 acquires an optical ID and acquires an AR image P43 corresponding to the optical ID from the server. Therefore, even if many images are stored in the server, the AR image P43 corresponding to the light ID can be appropriately selected and acquired as an AR image corresponding to the graphic 143 from the many images. it can.

FIG. 351 is a flowchart showing the operation of the receiver 200 in the present embodiment.

The receiver 200 in the present embodiment first acquires a plurality of AR image candidates (step S541). For example, the receiver 200 acquires a plurality of AR image candidates from the server through wireless communication (such as BTLE or Wi-Fi) different from visible light communication. Next, the receiver 200 captures an image of the subject (step S542). The receiver 200 acquires the captured display image Pa and the decoding image as described above by this imaging. However, when the subject is a photograph of the transmitter 100, since the optical ID is not transmitted from the subject, the receiver 200 does not acquire the optical ID even when decoding the decoding image. Can not.

Therefore, the receiver 200 determines whether or not the light ID has been acquired, that is, whether or not the light ID has been received from the subject (step S543).

Here, if it is determined that the optical ID is not received (No in step S543), the receiver 200 determines whether or not the AR display flag set for itself is 1 (step S544). The AR display flag is a flag indicating whether or not an AR image may be displayed only from the captured display image Pa even if the light ID is not acquired. When the AR display flag is 1, the AR display flag indicates that the AR image may be displayed only from the captured display image Pa. When the AR display flag is 0, the AR display flag is This indicates that the AR image should not be displayed only from the captured display image Pa.

If it is determined that the AR display flag is 1 (Yes in step S544), the receiver 200 selects a candidate corresponding to the captured display image Pa from the plurality of AR image candidates acquired in step S541 as an AR image. (Step S545). That is, the receiver 200 extracts a feature amount included in the captured display image Pa, and selects a candidate associated with the extracted feature amount as an AR image.

Then, the receiver 200 displays the selected candidate AR image superimposed on the captured display image Pa (step S546).

On the other hand, if it is determined that the AR display flag is 0 (No in step S544), the receiver 200 does not display the AR image.

If it is determined in step S543 that an optical ID has been received (Yes in step S543), the receiver 200 selects a candidate associated with the optical ID from among a plurality of AR image candidates acquired in step S541. It selects as AR image (step S547). Then, the receiver 200 displays the AR image that is the selected candidate so as to be superimposed on the captured display image Pa (step S546).

In the above example, the AR display flag is set in the receiver 200, but may be set in the server. In this case, the receiver 200 inquires of the server whether the AR display flag is 1 or 0 in step S544.

Thus, whether or not to display an AR image on the receiver 200 when the optical ID is not received even when the receiver 200 performs imaging can be controlled by the AR display flag.

FIG. 352 is a diagram for explaining the operation of the transmitter 100 in the present embodiment.

For example, the transmitter 100 is configured as a projector. Here, the intensity of the light emitted from the projector and reflected on the screen varies depending on factors such as the aging of the light source of the projector or the distance from the light source to the screen. When the light intensity is low, the optical ID transmitted from the transmitter 100 is difficult to be received by the receiver 200.

Therefore, the transmitter 100 according to the present embodiment adjusts a parameter for causing the light source to emit light in order to suppress a change in light intensity according to each factor. This parameter is at least one of the value of the current input to the light source for causing the light source to emit light and the light emission time (more specifically, the light emission time per unit time). For example, the light intensity of the light source increases as the current value increases or the light emission time increases.

That is, the transmitter 100 adjusts the parameter so that the light from the light source becomes stronger as the light source deteriorates with age. Specifically, the transmitter 100 includes a timer, and adjusts the parameter to increase the light of the light source as the usage time of the light source measured by the timer is longer. That is, the transmitter 100 increases the current value of the light source or lengthens the light emission time as the usage time is longer. Alternatively, the transmitter 100 detects the intensity of light emitted from the light source, and adjusts the parameters so that the detected light intensity does not decrease. That is, the transmitter 100 adjusts the parameter so as to increase the light intensity as the detected light intensity decreases.

In addition, the transmitter 100 adjusts the parameter so that the light from the light source becomes stronger as the irradiation distance from the light source to the screen becomes longer. Specifically, the transmitter 100 detects the intensity of the light that is irradiated and reflected by the screen, and adjusts the parameter so that the light from the light source increases as the detected light intensity decreases. That is, the transmitter 100 increases the current value of the light source or lengthens the light emission time as the detected light intensity decreases. Thereby, the parameters are adjusted so that the intensity of the reflected light is constant regardless of the irradiation distance. Alternatively, the transmitter 100 detects the irradiation distance from the light source to the screen by a distance measuring sensor, and adjusts the parameter so that the light from the light source is increased as the detected irradiation distance is longer.

Also, the transmitter 100 adjusts the parameter so that the lighter the light of the light source is, the darker the screen is. Specifically, the transmitter 100 detects the color of the screen by imaging the screen, and adjusts the parameter so that the light from the light source is increased as the detected color is blacker. That is, the transmitter 100 increases the current value of the light source or lengthens the light emission time as the detected color is black. Thus, the parameters are adjusted so that the intensity of the reflected light is constant regardless of the color of the screen.

In addition, the transmitter 100 adjusts the parameter so that the light from the light source increases as the external light increases. Specifically, the transmitter 100 detects the difference between the screen brightness when the light source is turned on and irradiated with light and the screen brightness when the light source is turned off and no light is irradiated. Then, the transmitter 100 adjusts the parameter so as to increase the light of the light source as the brightness difference is smaller. That is, the transmitter 100 increases the current value of the light source or lengthens the light emission time as the difference in brightness is smaller. Thus, the parameters are adjusted so that the S / N ratio of the light ID is constant regardless of the external light. Alternatively, when the transmitter 100 is configured as an LED display, for example, the intensity of sunlight may be detected, and the parameter may be adjusted to increase the light from the light source as the intensity of sunlight increases. .

It should be noted that the parameter adjustment as described above may be performed when an operation by the user is performed. For example, the transmitter 100 includes a calibration button, and performs the above-described parameter adjustment when the calibration button is pressed by the user. Alternatively, the transmitter 100 may periodically adjust the parameters described above.

FIG. 353 is a diagram for describing another operation of the transmitter 100 in the present embodiment.

For example, the transmitter 100 is configured as a projector, and irradiates the screen with light from the light source through the front member. When the projector is a liquid crystal projector, the front member is a liquid crystal panel, and when the projector is a DLP (registered trademark) projector, the front member is a DMD (Digital Mirror Device). That is, the front member is a member that adjusts the luminance of the image for each pixel. The light source irradiates light toward the front member, and switches the intensity of the light between High and Low. In addition, the light source adjusts the time-average brightness by adjusting the High time per unit time.

Here, when the transmittance of the front member is 100%, for example, the light source is dark so that the image projected from the projector onto the screen is not too bright. That is, the light source shortens the High time per unit time.

At this time, the light source widens the pulse width of the light ID when transmitting the light ID due to a change in luminance.

On the other hand, when the transmittance of the front member is 20%, for example, the light source becomes bright so that the image projected from the projector onto the screen is not too dark. That is, the light source lengthens High time per unit time.

At this time, the light source narrows the pulse width of the light ID when transmitting the light ID due to a change in luminance.

As described above, when the light source is dark, the pulse width of the light ID is widened. Conversely, when the light source is bright, the pulse width of the light ID is narrowed. It is possible to prevent the intensity of the light from being too weak or too bright.

In the above example, the transmitter 100 is a projector, but may be configured as a large LED display. The large LED display includes a pixel switch and a common switch as shown in FIGS. 173, 175, and 180B. An image is expressed by turning on and off the pixel switch, and an optical ID is transmitted by turning on and off the common switch. In this case, functionally, the pixel switch corresponds to the front member, and the common switch corresponds to the light source. When the average luminance by the pixel switch is high, the pulse width of the light ID by the common switch may be shortened.

FIG. 354 is a diagram for describing another operation of the transmitter 100 in the present embodiment. Specifically, FIG. 354 shows the dimming degree of the transmitter 100 configured as a spotlight with dimming function and the current (specifically, the peak current value) input to the light source of the transmitter 100. Show the relationship.

The transmitter 100 receives the dimming degree specified for the light source provided in the transmitter 100, and causes the light source to emit light at the specified dimming degree. The dimming degree is a ratio of the average luminance of the light source to the maximum average luminance. The average luminance is not instantaneous luminance but luminance in time average. In addition, the dimming degree is adjusted by adjusting the value of the current input to the light source or adjusting the time during which the luminance of the light source is low. The time when the luminance of the light source becomes Low may be the time when the light source is turned off.

Here, when transmitting the transmission target signal as an optical ID, the transmitter 100 generates an encoded signal by encoding the transmission target signal in a predetermined mode. Then, the transmitter 100 transmits the encoded signal as an optical ID (that is, a visible light signal) by changing the luminance of the light source provided in the transmitter 100 according to the encoded signal.

For example, when the designated dimming degree is 0% or more and x3 (%) or less, the transmitter 100 generates an encoded signal by encoding the transmission target signal in the PWM mode with a duty ratio of 35%. . x3 (%) is, for example, 50%. In the present embodiment, the PWM mode with a duty ratio of 35% is also referred to as a first mode, and the above x3 is also referred to as a first value.

That is, when the designated dimming degree is 0% or more and x3 (%) or less, the transmitter 100 sets the dimming degree of the light source to the value of the peak current while maintaining the duty ratio of the visible light signal at 35%. Adjust by.

When the designated dimming degree is greater than x3 (%) and equal to or less than 100%, the transmitter 100 encodes the encoded signal by encoding the transmission target signal in the PWM mode with a duty ratio of 65%. Generate. In the present embodiment, the PWM mode with a duty ratio of 65% is also referred to as a second mode.

That is, when the designated dimming degree is greater than x3 (%) and equal to or less than 100%, the transmitter 100 sets the dimming degree of the light source to the peak current while maintaining the duty ratio of the visible light signal at 65%. Adjust according to the value of.

Thus, the transmitter 100 according to the present embodiment accepts the dimming degree designated for the light source as the designated dimming degree. Then, when the designated dimming degree is equal to or less than the first value, the transmitter 100 transmits the signal encoded in the first mode by the luminance change while causing the light source to emit light at the designated dimming degree. In addition, when the value of the designated dimming degree is larger than the first value, the transmitter 100 transmits the signal encoded in the second mode by changing the luminance while causing the light source to emit light at the designated dimming degree. To do. Specifically, the duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode.

Here, since the duty ratio of the second mode is larger than the duty ratio of the first mode, the change rate of the peak current with respect to the dimming degree in the second mode is expressed as the change of the peak current with respect to the dimming degree in the first mode. It can be made smaller than the rate.

Also, when the designated dimming degree exceeds x3 (%), the mode is switched from the first mode to the second mode. Accordingly, at this time, the peak current can be instantaneously reduced. That is, when the designated dimming degree is x3 (%), the peak current is y3 (mA). However, when the designated dimming degree slightly exceeds x3 (%), the peak current exceeds y2 (mA). Can be suppressed. In addition, y3 (mA) is 143 mA, for example, and y2 (mA) is 100 mA, for example. As a result, in order to increase the dimming degree, it is possible to suppress the peak current from becoming larger than y3 (mA), and it is possible to suppress the deterioration of the light source due to the large current flowing.

Also, when the specified dimming degree exceeds x4 (%), the peak current becomes larger than y3 (mA) even if the mode is the second mode. However, when the specified dimming rate is less than x4 (%), deterioration of the light source can be suppressed. In the present embodiment, x4 described above is also referred to as a second value. In the example shown in FIG. 354, x4 (%) is less than 100%, but may be 100%.

That is, in the transmitter 100 according to the present embodiment, when the designated dimming degree is larger than the first value and equal to or smaller than the second value, the signal encoded in the second mode is transmitted by the luminance change. When the specified dimming level is the first value, the peak current value of the light source is smaller than the peak current value of the light source for transmitting the signal encoded in the first mode by a change in luminance.

As a result, the peak current value of the light source when the designated dimming degree is larger than the first value and equal to or smaller than the second value by switching the mode for encoding the signal is the designated dimming degree being the first value. It becomes smaller than the peak current value of the light source in some cases. Therefore, as the designated dimming degree is increased, a large peak current can be suppressed from flowing to the light source. As a result, deterioration of the light source can be suppressed.

Furthermore, when the designated dimming degree is less than or equal to x1 (%) and less than x2 (%), the transmitter 100 in the present embodiment causes the light source to emit light at the designated dimming degree, while in the first mode. In addition to transmitting the signal encoded by the luminance change, the peak current value is maintained at a constant value with respect to the designated dimming change. x2 (%) is smaller than x3 (%). In the present embodiment, x2 described above is also referred to as a third value.

That is, when the designated dimming degree is smaller than x2 (%), the transmitter 100 increases the time for turning off the light source as the designated dimming degree becomes smaller, and the specified dimming degree becomes smaller. The light source is caused to emit light, and the peak current value is maintained at a constant value. Specifically, the transmitter 100 lengthens the cycle for transmitting each of the plurality of encoded signals while maintaining the duty ratio of the encoded signal at 35%. Thereby, the time to turn off the light source, that is, the extinguishing period becomes longer. As a result, the dimming degree can be reduced while keeping the peak current value constant. Even when the specified dimming level is small, the peak current value is kept constant, so that the visible light signal (that is, the light ID), which is a signal transmitted by a change in luminance, is easily received by the receiver 200. be able to.

Here, the transmitter 100 determines the time to turn off the light source so that one cycle obtained by adding the time to transmit the encoded signal due to the luminance change and the time to turn off the light source does not exceed 10 milliseconds. . For example, if the time to turn off the light source is too long and one period exceeds 10 milliseconds, the luminance change of the light source for transmitting the encoded signal may be recognized by the human eye as flickering. . Therefore, in this embodiment, since the time for turning off the light source is determined so that one period does not exceed 10 milliseconds, flicker can be prevented from being recognized by a person.

Further, even when the designated dimming degree is smaller than x1 (%), the transmitter 100 transmits the signal encoded in the first mode by the luminance change while causing the light source to emit light at the designated dimming degree. At this time, the transmitter 100 causes the light source to emit light at the specified dimming degree that becomes smaller by reducing the value of the peak current as the designated dimming degree becomes smaller. x1 (%) is smaller than x2 (%). In the present embodiment, the above x1 is also referred to as a fourth value.

Thereby, even if the designated dimming degree is smaller, the light source can be appropriately emitted with the designated dimming degree.

In the example shown in FIG. 354, the maximum peak current value in the first mode (ie, y3 (mA)) is greater than the maximum peak current value in the second mode (ie, y4 (mA)). Small but the same. That is, the transmitter 100 encodes the transmission target signal in the first mode until the designated dimming degree is x3a (%) larger than x3 (%). Then, when the designated dimming degree is x3a (%), the transmitter 100 turns on the light source with the same peak current value as the maximum peak current value in the second mode (ie, y4 (mA)). Make it emit light. In this case, x3a is the first value. Note that the maximum peak current value in the second mode is the peak current value when the specified dimming degree is the maximum value, that is, 100%.

In other words, in the present embodiment, the value of the peak current of the light source when the designated dimming degree is the first value is the same as the value of the peak current of the light source when the designated dimming degree is the maximum value. There may be. In this case, since the range of the dimming degree that causes the light source to emit light with a peak current of y3 (mA) or more is widened, it is possible to make the receiver 200 easily receive the light ID in a wide range of dimming degree. In other words, even in the first mode, since a large peak current can be passed through the light source, it is possible to make it easier for the receiver to receive a signal transmitted by a change in luminance of the light source. In this case, since the period during which a large peak current flows becomes long, the light source easily deteriorates.

FIG. 355 is a diagram showing a comparative example for explaining the ease of receiving the optical ID in the present embodiment.

In this embodiment, as shown in FIG. 354, the first mode is used when the dimming degree is small, and the second mode is used when the dimming degree is large. The first mode is a mode in which the increase in peak current is increased even if the increase in dimming degree is small, and the second mode is a mode in which the increase in peak current is suppressed even if the increase in dimming degree is large. Therefore, since the second mode suppresses a large peak current from flowing to the light source, deterioration of the light source can be suppressed. Furthermore, the first mode allows the receiver 200 to easily receive the optical ID because a large peak current flows through the light source even if the dimming degree is small.

On the other hand, when the second mode is used even when the dimming degree is small, as shown in FIG. 355, when the dimming degree is small, the value of the peak current is small. It becomes difficult to make it receive.

Therefore, in transmitter 100 according to the present embodiment, it is possible to achieve both suppression of deterioration of the light source and ease of reception of the optical ID.

Further, when the value of the peak current of the light source exceeds the fifth value, the transmitter 100 may stop transmission of a signal due to a change in luminance of the light source. The fifth value may be, for example, y3 (mA).

Thereby, the deterioration of the light source can be further suppressed.

Also, the transmitter 100 may measure the usage time of the light source, as in the example shown in FIG. When the usage time is equal to or longer than the predetermined time, the transmitter 100 may transmit a signal by a change in luminance using a parameter value for causing the light source to emit light with a dimming degree greater than the specified dimming degree. . In this case, the parameter value may be a peak current value or a time for turning off the light source. Thereby, it is possible to prevent the optical ID from becoming difficult to be received by the receiver 200 due to deterioration of the light source over time.

Alternatively, the transmitter 100 may measure the usage time of the light source, and if the usage time is equal to or longer than the predetermined time, the transmitter 100 may increase the pulse width of the current of the light source as compared to the case where the usage time is less than the predetermined time. . Thereby, it can suppress that it becomes difficult to receive optical ID by deterioration of a light source like the above-mentioned.

In the above-described embodiment, the transmitter 100 is switched between the first mode and the second mode according to the designated dimming degree, but even if the mode is switched according to the operation by the user. Good. That is, when the switch is operated by the user, the transmitter 100 switches the first mode to the second mode, or conversely switches the second mode to the first mode. Further, the transmitter 100 may notify the user when the mode is switched. For example, the transmitter 100 may notify the user of the mode switching by sounding a sound, blinking a light source at a period that can be visually recognized by a person, or turning on a notification LED. Further, the transmitter 100 may notify the user that the relationship changes not only at the mode switching but also at the time when the relationship between the peak current and the dimming degree changes. The time point is, for example, a time point when the dimming degree shown in FIG. 354 changes from x1 (%) or a time point when the dimming degree changes from x2 (%).

FIG. 356A is a flowchart showing the operation of the transmitter 100 in the present embodiment.

First, the transmitter 100 receives the dimming degree designated for the light source as the designated dimming degree (step S551). Next, the transmitter 100 transmits a signal according to a change in luminance of the light source (step S552). Specifically, when the designated dimming degree is equal to or less than the first value, the transmitter 100 causes the light source to emit light at the designated dimming degree, and changes the signal encoded in the first mode according to the luminance change. Send. In addition, when the designated dimming degree is larger than the first value, the transmitter 100 transmits the signal encoded in the second mode by the luminance change while causing the light source to emit light at the designated dimming degree. Here, when the designated dimming degree is larger than the first value and equal to or smaller than the second value, the value of the peak current of the light source for transmitting the signal encoded in the second mode by the luminance change is When the designated dimming level is the first value, it is smaller than the value of the peak current of the light source for transmitting the signal encoded in the first mode by the luminance change.

FIG. 356B is a block diagram showing a configuration of transmitter 100 in the present embodiment.

The transmitter 100 includes a reception unit 551 and a transmission unit 552. The accepting unit 551 accepts the dimming degree designated for the light source as the designated dimming degree (step S551). The transmission unit 552 transmits a signal according to a change in luminance of the light source. Specifically, when the designated dimming level is equal to or smaller than the first value, the transmission unit 552 causes the light source to emit light at the designated dimming level, and the signal encoded in the first mode is changed by the luminance change. Send. In addition, when the designated dimming degree is larger than the first value, the transmission unit 552 transmits the signal encoded in the second mode by the luminance change while causing the light source to emit light at the designated dimming degree. Here, when the designated dimming degree is larger than the first value and equal to or smaller than the second value, the value of the peak current of the light source for transmitting the signal encoded in the second mode by the luminance change is When the designated dimming level is the first value, it is smaller than the value of the peak current of the light source for transmitting the signal encoded in the first mode by the luminance change.

As a result, as shown in FIG. 354, the peak current value of the light source when the designated dimming degree is greater than the first value and equal to or less than the second value by switching the signal encoding mode is It becomes smaller than the value of the peak current of the light source when the luminous intensity is the first value. Therefore, as the designated dimming degree is increased, a large peak current can be suppressed from flowing to the light source. As a result, deterioration of the light source can be suppressed.

FIG. 357 is a diagram illustrating another example in which the receiver 200 according to the present embodiment displays an AR image.

The receiver 200 acquires the above-described captured display image Pk, which is the normal captured image, and the above-described decoding image, which is the visible light communication image or the bright line image, by capturing the subject with the image sensor.

Specifically, the image sensor of the receiver 200 images the transmitter 100 configured as signage and the person 21 adjacent to the transmitter 100. The transmitter 100 is the transmitter in each of the above embodiments, and includes one or a plurality of light emitting elements (for example, LEDs) and a light transmitting plate 144 having a light transmitting property like ground glass. The one or more light emitting elements emit light inside the transmitter 100, and light from the one or more light emitting elements is transmitted through the translucent plate 144 and irradiated outside. As a result, the translucent plate 144 of the transmitter 100 shines brightly. Such a transmitter 100 changes in luminance by blinking one or more light emitting elements, and transmits an optical ID (light identification information) by the change in luminance. This light ID is the above-mentioned visible light signal.

Here, a message “Please hold your smartphone here” is written on the translucent plate 144. Therefore, the user of the receiver 200 places the person 21 next to the transmitter 100 and instructs the person 21 to put his arm on the transmitter 100. Then, the user images the camera (that is, the image sensor) of the receiver 200 toward the person 21 and the transmitter 100. The receiver 200 captures the transmitter 100 and the person 21 with the normal exposure time, thereby obtaining a captured display image Pk on which they are projected. Furthermore, the receiver 200 acquires a decoding image by capturing the transmitter 100 and the person 21 with a communication exposure time shorter than the normal exposure time.

The receiver 200 acquires the optical ID by decoding the decoding image. That is, the receiver 200 receives the optical ID from the transmitter 100. The receiver 200 transmits the optical ID to the server. Then, the receiver 200 acquires the AR image P44 corresponding to the optical ID and the recognition information from the server. The receiver 200 recognizes an area corresponding to the recognition information in the captured display image Pk as a target area. For example, the receiver 200 recognizes an area in which signage as the transmitter 100 is displayed as a target area.

Then, the receiver 200 superimposes the AR image P44 on the captured display image Pk so that the target area is covered with the AR image P44, and displays the captured display image Pk on the display 201. For example, the receiver 200 acquires an AR image P44 indicating a soccer player. In this case, since the AR image P44 is superimposed so as to cover the target area of the captured display image Pk, the captured display image Pk can be displayed so that a soccer player actually exists next to the person 21. . As a result, even if the soccer player is not next to the person 21, the person 21 can be photographed together with the soccer player. More specifically, in the state where the arm of the person 21 is put on the shoulder of the soccer player, the photograph can be taken together with the soccer player.

(Embodiment 27)
In this embodiment, a transmission method for transmitting an optical ID by a visible light signal will be described. Note that the transmitter and the receiver in this embodiment may have the same functions and configurations as the transmitter (or the transmission device) and the receiver (or the reception device) in each of the above embodiments.

FIG. 358 is a diagram for explaining the operation of the transmitter 100 in the present embodiment. Specifically, FIG. 358 shows the dimming degree of the transmitter 100 configured as a spotlight with dimming function and the current (specifically, the peak current value) input to the light source of the transmitter 100. Show the relationship.

When the designated dimming degree is 0% or more and x14 (%) or less, the transmitter 100 according to the present embodiment encodes a transmission target signal in a PWM mode with a duty ratio of 35%. Is generated. In other words, when the designated dimming level changes from 0% to x14 (%), the transmitter 100 increases the peak current value while maintaining the duty ratio of the visible light signal at 35%. The light source is caused to emit light at the specified dimming degree. Note that the PWM mode with a duty ratio of 35% is also referred to as a first mode, as in the twenty-sixth embodiment, and the above-described x14 is also referred to as a first value. For example, x14 (%) is a value within the range of 50 to 60%.

Further, when the designated dimming degree is not less than x13 (%) and not more than 100%, the transmitter 100 generates an encoded signal by encoding the transmission target signal in the PWM mode with a duty ratio of 65%. . That is, when the specified dimming level changes from 100% to x13 (%), the transmitter 100 suppresses the peak current value while maintaining the duty ratio of the visible light signal at 65%. The light source is caused to emit light at the specified dimming degree. Note that the PWM mode with a duty ratio of 65% is also referred to as a second mode, as in the twenty-sixth embodiment, and the above x13 is also referred to as a second value. Here, x13 (%) is a value smaller than x14 (%), for example, a value within a range of 40 to 50%.

As described above, in this embodiment, when the specified dimming degree increases, the PWM mode is changed from the PWM mode with the duty ratio of 35% to the PWM mode with the duty ratio of 65% at the dimming degree x14 (%). Can be switched. On the other hand, when the specified dimming degree decreases, the PWM mode is a dimming degree x13 (%) smaller than the dimming degree x14 (%), and the PWM mode from the PWM mode having a duty ratio of 65% to the PWM mode having a duty ratio of 35%. Can be switched to. That is, in the present embodiment, the dimming degree at which the PWM mode is switched is different when the designated dimming degree is increased and when the designated dimming degree is reduced. Hereinafter, the dimming degree at which the PWM mode is switched is referred to as a switching point.

Therefore, in this embodiment, frequent switching of the PWM mode can be suppressed. In the example shown in FIG. 354 of the twenty-sixth embodiment, the switching point of the PWM mode is 50%, and is the same when the designated dimming degree increases and when the designated dimming degree decreases. . As a result, in the example shown in FIG. 354, when the increase / decrease in the specified dimming degree is repeated around 50%, the PWM mode is frequently switched between the PWM mode with a duty ratio of 35% and the PWM mode with a duty ratio of 65%. Can be switched. However, in this embodiment, since the switching point of the PWM mode is different between the case where the designated dimming degree increases and the designated dimming degree decreases, such frequent switching of the PWM mode is performed. Can be suppressed.

In the present embodiment, similarly to the example shown in FIG. 354 of the twenty-sixth embodiment, when the designated dimming degree is small, the PWM mode with a small duty ratio is used, and conversely, the designated dimming degree is used. When is large, a PWM mode with a large duty ratio is used.

Therefore, when the specified dimming degree is large, the PWM mode with a large duty ratio is used, so the rate of change of the peak current with respect to the dimming degree can be reduced, and the light source emits light with a large dimming degree with the small peak current Can be made. For example, in the PWM mode with a duty ratio as small as 35%, unless the peak current is 250 mA, the light source cannot emit light with a dimming degree of 100%. However, in the present embodiment, for a large dimming degree, a PWM mode with a large duty ratio such as a duty ratio of 65% is used. Therefore, for example, the light source is set to 100 by simply setting the peak current to a smaller 154 mA. % Light can be emitted. That is, it is possible to suppress the overcurrent from flowing through the light source and shorten the life of the light source.

Also, when the specified dimming degree is small, the PWM mode with a small duty ratio is used, so that the rate of change of the peak current with respect to the dimming degree can be increased. As a result, a visible light signal can be transmitted with a large peak current while causing the light source to emit light with a small dimming degree. The light source emits light brighter as the input current increases. Therefore, when a visible light signal is transmitted with a large peak current, the visible light signal can be easily received by the receiver 200. In other words, the range of the dimming level in which the visible light signal receivable to the receiver 200 can be transmitted can be expanded to a smaller dimming level. For example, as shown in FIG. 385, if the peak current is equal to or greater than Ia (mA), the receiver 200 can receive a visible light signal transmitted by the peak current. In this case, in the PWM mode with a large duty ratio such as a duty ratio of 65%, the range of dimming degree capable of transmitting a receivable visible light signal is x12 (%) or more. However, in the PWM mode with a small duty ratio such as a duty ratio of 35%, the range of the dimming degree capable of transmitting a receivable visible light signal is set to x11 (%) or more smaller than x12 (%). Can do.

Thus, by switching the PWM mode, the life of the light source can be extended and a visible light signal can be transmitted in a wide dimming range.

FIG. 359A is a flowchart showing a transmission method according to the present embodiment.

The transmission method in the present embodiment is a transmission method for transmitting a signal according to a change in luminance of a light source, and includes a reception step S561 and a transmission step S562. In the reception step S561, the transmitter 100 receives the dimming degree designated for the light source as the designated dimming degree. In the transmission step S562, the transmitter 100 transmits the signal encoded in the first mode or the second mode by the luminance change while causing the light source to emit light at the designated dimming degree. Here, the duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode. Further, in the transmission step S562, when the designated dimming level is changed from a small value to a large value, the transmitter 100 changes the mode used for signal encoding when the designated dimming level is the first value. Switch from the first mode to the second mode. Furthermore, when the designated dimming level is changed from a large value to a small value, the transmitter 100 changes the mode used for signal encoding from the second mode when the designated dimming level is the second value. Switch to the first mode. Here, the second value is smaller than the first value.

For example, the first mode and the second mode are the PWM mode with a duty ratio of 35% and the PWM mode with a duty ratio of 65% shown in FIG. 358, respectively. Further, the first value and the second value are x14 (%) and x15 (%) shown in FIG. 358, respectively.

Thereby, the designated dimming degree (that is, the switching point) at which the switching between the first mode and the second mode is performed differs depending on whether the designated light intensity increases or decreases. Therefore, frequent switching between these modes can be suppressed. That is, the occurrence of so-called chattering can be suppressed. As a result, the operation of the transmitter 100 that transmits a signal can be stabilized. Further, the duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode. Therefore, similarly to the transmission method shown in FIG. 354, the larger the designated dimming degree, the more the large peak current can be suppressed from flowing to the light source. As a result, deterioration of the light source can be suppressed. Further, since deterioration of the light source can be suppressed, communication between various devices can be performed for a long time. Further, when the designated dimming degree is small, the first mode with a small duty ratio is used. Therefore, the above-described peak current can be increased, and a signal that is easily received by the receiver 200 can be transmitted as a visible light signal.

In addition, in the transmission step S562, when switching from the first mode to the second mode is performed, the transmitter 100 determines the peak current of the light source for transmitting the encoded signal by the luminance change, The current value of 1 is changed to a second current value smaller than the first current value. Furthermore, when switching from the second mode to the first mode is performed, the transmitter 100 changes the peak current from the third current value to a fourth current value that is larger than the third current value. change. Here, the first current value is larger than the fourth current value, and the second current value is larger than the third current value.

For example, the first current value, the second current value, the third current value, and the fourth current value are respectively the current value Ie, the current value Ic, the current value Ib, and the current value Id shown in FIG. is there.

This makes it possible to appropriately switch between the first mode and the second mode.

FIG. 359B is a block diagram showing a configuration of transmitter 100 in the present embodiment.

The transmitter 100 in the present embodiment is a transmitter that transmits a signal according to a change in luminance of a light source, and includes a reception unit 561 and a transmission unit 562. The accepting unit 561 accepts the dimming degree designated for the light source as the designated dimming degree. The transmission unit 562 transmits the signal encoded in the first mode or the second mode by the luminance change while causing the light source to emit light at the designated dimming degree. Here, the duty ratio of the signal encoded in the second mode is larger than the duty ratio of the signal encoded in the first mode. Further, when the designated dimming level is changed from a small value to a large value, the transmission unit 562 changes the mode used for signal encoding from the first mode when the designated dimming level is the first value. Switch to the second mode. Furthermore, when the designated dimming level is changed from a large value to a small value, the transmission unit 562 changes the mode used for signal encoding from the second mode when the designated dimming level is the second value. Switch to the first mode. Here, the second value is smaller than the first value.

The transmission method of the flowchart shown in FIG. 359A is realized by such a transmitter 100.

FIG. 360 is a diagram illustrating an example of a detailed configuration of a visible light signal according to the present embodiment.

Such a visible light signal is a PWM mode signal as in FIGS. 188 and 189A (b), FIGS. 197, 212, 316, and 317.

The visible light signal packet is composed of an L data part, a preamble, and an R data part. Note that each of the L data portion and the R data portion corresponds to a payload.

The preamble corresponds to the preambles in FIGS. 188 and 189A (b), 197 and 212, and corresponds to the SHR in FIGS. 316 and 317. Specifically, the preamble alternately indicates High and Low luminance values along the time axis. In other words, the preamble indicates a high luminance value only for the time length C ₀ , indicates a low luminance value for the next time length C ₁ , indicates a high luminance value for the next time length C ₂ , and displays the next time length C _2. Only ₃ indicates a low luminance value. The time lengths C ₀ and C ₃ are, for example, 100 μs. Further, the time lengths C ₁ and C ₂ are 90 μs shorter than the time lengths C ₁ and C ₃ by 10 μs, for example.

The L data portion corresponds to the data L in FIGS. 188 and 189A (b), 197 and 212, and corresponds to the PHY payload A in FIGS. 316 and 317. Specifically, the L data portion alternately indicates High and Low luminance values along the time axis, and is arranged immediately before the preamble. That is, the L data portion indicates a high luminance value for the time length D ′ ₀ , indicates a low luminance value for the next time length D ′ ₁ , indicates a high luminance value for the next time length D ′ ₂ , The next time length D ′ ₃ indicates a low luminance value. Note that the time lengths D ′ ₀ to D ′ ₃ are determined according to a mathematical formula corresponding to the signal to be transmitted. This equation is expressed as D ′ ₀ = W ₀ + W ₁ × (3-y ₀ ), D ′ ₁ = W ₀ + W ₁ × (7−y ₁ ), D ′ ₂ = W ₀ + W ₁ × (3-y ₂₎ ), And D ′ ₃ = W ₀ + W ₁ × (7−y ₃ ). Here, the constant W ₀ is, for example, 110 μs, and the constant W ₁ is, for example, 30 μs. The variables y ₀ and y ₂ are any integer from 0 to 3 represented by 2 bits, and the variables y ₁ and y ₃ are any integer from 0 to 7 represented by 3 bits. Variables y ₀ to y ₃ are signals to be transmitted. In FIGS. 360 to 363, “*” is used as a symbol meaning multiplication.

The R data portion corresponds to the data R in FIGS. 188 and 189A (b), FIGS. 197 and 212, and corresponds to the PHY payload B in FIGS. 316 and 317. Specifically, the R data portion alternately indicates High and Low luminance values along the time axis, and is arranged immediately after the preamble. That is, the R data portion indicates a high luminance value for the time length D ₀ , indicates a low luminance value for the next time length D ₁ , indicates a high luminance value for the next time length D ₂ , and displays the next time length. the length _{D 3} shows the luminance value of Low. Note that the time lengths D ₀ to D ₃ are determined according to a mathematical expression corresponding to the signal to be transmitted. This formula is D ₀ = W ₀ + W ₁ × y ₀ , D ₁ = W ₀ + W ₁ × y ₁ , D ₂ = W ₀ + W ₁ × y ₂ , and D ₃ = W ₀ + W ₁ × y ₃ .

Here, the L data portion and the R data portion have a complementary relationship with respect to brightness. That is, if the brightness of the L data portion is bright, the brightness of the R data portion is dark. Conversely, if the brightness of the L data portion is dark, the brightness of the R data portion is bright. That is, the sum of the time length of the L data portion and the time length of the R data portion is constant regardless of the signal to be transmitted. In other words, the brightness of the time average of the visible light signal transmitted from the light source can be made constant regardless of the signal to be transmitted.

Further, D ′ ₀ = W ₀ + W ₁ × (3-y ₀ ), D ′ ₁ = W ₀ + W ₁ × (7−y ₁ ), D ′ ₂ = W ₀ + W ₁ × (3-y ₂ ), And by changing the ratio of 3 and 7 in D ′ ₃ = W ₀ + W ₁ × (7−y ₃ ), the duty ratio of the PWM mode can be changed. The ratio between 3 and 7 corresponds to the ratio between the maximum values of the variables y ₀ and y _{2 and} the maximum values of the variables y ₁ and y ₃ . For example, when the ratio is 3: 7, a PWM mode with a small duty ratio is selected. Conversely, when the ratio is 7: 3, a PWM mode with a large duty ratio is selected. Therefore, by adjusting the ratio, the PWM mode can be switched between the PWM mode with a duty ratio of 35% shown in FIGS. 354 and 358 and the PWM mode with a duty ratio of 65%. In addition, a preamble may be used to notify the receiver 200 which PWM mode is being switched to. For example, the transmitter 100 notifies the receiver 200 of the switched PWM mode by including in the packet the preamble of the pattern associated with the switched PWM mode. The preamble pattern is changed according to the time lengths C ₀ , C ₁ , C ₂ and C ₃ .

However, in the visible light signal shown in FIG. 360, since the packet includes two data parts, it takes time to transmit the packet. For example, when the transmitter 100 is a DLP projector, the transmitter 100 projects each video image of red, green, and blue in a time division manner. Here, the transmitter 100 desirably transmits a visible light signal when projecting a red video. This is because the visible light signal transmitted at this time has a red wavelength and thus is easily received by the receiver 200. The period during which the red video is continuously projected is, for example, 1.5 ms. This period is hereinafter referred to as a red projection period. In such a short red projection period, it is difficult to transmit a packet including the L data portion, the preamble, and the R data portion described above.

Therefore, a packet having only the R data portion of the two data portions is recalled.

FIG. 361 is a diagram showing another example of the detailed configuration of the visible light signal in the present embodiment.

Unlike the example shown in FIG. 360, the visible light signal packet shown in FIG. 361 does not include the L data portion. Instead, the visible light signal packet shown in FIG. 361 includes invalid data and an average luminance adjustment unit.

The invalid data alternately indicates High and Low luminance values along the time axis. In other words, the invalid data indicates a high luminance value for the time length A ₀ and indicates a low luminance value for the next time length A ₁ . The time length A ₀ is, for example, 100 μs, and the time length A ₁ is represented by, for example, A ₁ = W ₀ −W ₁ . Such invalid data indicates that the L data portion is not included in the packet.

The average luminance adjustment unit alternately indicates High and Low luminance values along the time axis. In other words, the invalid data indicates a high luminance value for the time length B ₀ and indicates a low luminance value for the next time length B ₁ . The time length B ₀ is represented by, for example, B ₀ = 100 + W ₁ × ((3-y ₀ ) + (3-y ₂ )), and the time length B ₁ is, for example, B ₁ = W ₁ × ((7−y ₁ ) + (7−y ₃ )).

By such an average luminance adjusting unit, the average luminance in the packet can be made constant regardless of the signals y ₀ to y ₃ to be transmitted. That is, the sum of the lengths of time when the luminance value is High in the packet (that is, the total ON time) can be set to A ₀ + C ₀ + C ₂ + D ₀ + D ₂ + B ₀ = 790. Furthermore, the sum of the time lengths when the luminance value is low in the packet (that is, the total OFF time) can be set to A ₁ + C ₁ + C ₃ + D ₁ + D ₃ + B ₁ = 910.

However, even in such a configuration of the visible light signal, it is impossible to shorten the effective length of time E ₁ which is a part of the time length of the total duration E ₀ in the packet. The effective duration E _1, first from appeared luminance values of High, a time until the brightness of the last occurrence High is completed, the receiver 200 demodulates or decodes the packets of the visible light signal in packet Is the time required to do. Specifically, the effective time length E ₁ is E ₁ = A ₀ + A ₁ + C ₀ + C ₁ + C ₂ + C ₃ + D ₀ + D ₁ + D ₂ + D ₃ + B ₀ . Note that the total time length E ₀ is E ₀ = E ₁ + B ₁ .

That is, the effective time length E ₁ is 1700 μs at the maximum even for the visible light signal having the configuration shown in FIG. 361. Therefore, the transmitter 100 continues for the effective time length E ₁ during the red projection period. It is difficult to transmit one packet.

Therefore, in order to shorten the effective time length E ₁ and make the average brightness of the packet constant regardless of the signal to be transmitted, not only the time lengths of the brightness values of High and Low but also the brightness value of High It is also recalled to adjust.

FIG. 362 is a diagram illustrating another example of the detailed configuration of the visible light signal according to the present embodiment.

In the visible light signal packet shown in FIG. 362, unlike the example shown in FIG. 361, in order to shorten the effective time length E ₁ , the time length B ₀ of the high luminance value of the average luminance adjustment unit is the signal to be transmitted. Regardless, it is fixed at the shortest 100 μs. Instead, in the visible light signal packet shown in FIG. 362, the High luminance value is in accordance with the variables y ₀ and y ₂ included in the transmission target signal, that is, in accordance with the time lengths D ₀ and D _2. Adjusted. For example, when the time lengths D ₀ and D ₂ are short, the transmitter 100 adjusts the High luminance value to a large value as illustrated in FIG. When the time lengths D ₀ and D ₂ are long, the transmitter 100 adjusts the High luminance value to a small value as shown in FIG. Specifically, when the time lengths D ₀ and D ₂ are the shortest W ₀ (for example, 110 μs), the High luminance value is 100% brightness. When the time lengths D ₀ and D ₂ are the maximum “W ₀ + 3W ₁ ” (for example, 200 μs), the high luminance value is 77.2% brightness.

In such a visible light signal packet, the sum of the time lengths of the brightness value High (that is, the total ON time) can be set to, for example, A ₀ + C ₀ + C ₂ + D ₀ + D ₂ + B ₀ = 610 to 790. . Furthermore, the sum of the time lengths when the luminance value is Low (that is, the total OFF time) can be set to A ₁ + C ₁ + C ₃ + D ₁ + D ₃ + B ₁ = 910.

However, in the visible light signal shown in FIG. 362, the shortest time lengths of the total time length E ₀ and the effective time length E _{1 in} the packet can be made shorter than the example shown in FIG. The length cannot be shortened.

Therefore, to shorten the effective length of time E _1, and, in order to make constant regardless of the average brightness of the packet signal to be transmitted in response to a signal to be transmitted, and L data portion as the data portion included in the packet It is recalled that it can be understood using the R data part.

FIG. 363 is a diagram illustrating another example of the detailed configuration of the visible light signal according to the present embodiment.

In the visible light signal shown in FIG. 363, unlike the examples shown in FIG. 360 to FIG. 362, in order to shorten the effective time length, the L data portion is selected according to the sum of the variables y ₀ to y ₃ that are signals to be transmitted. And a packet including the R data portion are selectively used.

That is, when the sum of the variables y ₀ to y ₃ is 7 or more, the transmitter 100 generates a packet including only the L data portion of the two data portions as shown in FIG. 363 (a). . Hereinafter, this packet is referred to as an L packet. In addition, when the sum of the variables y ₀ to y ₃ is 6 or less, the transmitter 100 generates a packet including only the R data portion of the two data portions as shown in FIG. 363 (b). . Hereinafter, this packet is referred to as an R packet.

The L packet includes an average luminance adjustment unit, an L data unit, a preamble, and invalid data, as shown in FIG.

The average luminance adjusting unit of the L packet indicates the low luminance value for the time length B ′ ₀ without indicating the high luminance value. The time length B ′ ₀ is represented by, for example, B ′ ₀ = 100 + W ₁ × (y ₀ + y ₁ + y ₂ + y ₃ −7).

The invalid data of the L packet alternately indicates high and low luminance values along the time axis. That is, the invalid data indicates a high luminance value for the time length A ′ ₀ and indicates a low luminance value for the next time length A ′ ₁ . The time length A ′ ₀ is indicated by A ′ ₀ = W ₀ −W ₁ and is, for example, 80 μs, and the time length A ′ ₁ is, for example, 150 μs. Such invalid data indicates that the R data portion is not included in the packet having the invalid data.

In such an L packet, the total time length E ′ ₀ is E ′ ₀ = 5W ₀ + 12W ₁ + 4b + 230 = 1540 μs regardless of the signal to be transmitted. The effective time length E ′ ₁ is a time length corresponding to a signal to be transmitted and is in the range of 900 to 1290 μs. In addition, while the total time length E ′ ₀ is a constant 1540 μs, the sum of the time lengths where the luminance value is High (that is, the total ON time) varies in the range of 490 to 670 μs depending on the signal to be transmitted. To do. Therefore, the transmitter 100 also sets the high luminance value to 100% to 73 in this L packet according to the total ON time, that is, according to the time lengths D ₀ and D ₂ , as in the example shown in FIG. Change in the range of 1%.

As in the example shown in FIG. 361, the R packet includes invalid data, a preamble, an R data portion, and an average luminance adjustment portion as shown in FIG. 363 (b).

Here, the R packet shown in (b) in FIG. 363, in order to shorten the effective length of time E _1, the time length B ₀ of the luminance values of High in the average luminance adjusting unit, shortest regardless signal to be transmitted Of 100 μs. Further, the time length B ₁ of the Low luminance value in the average luminance adjustment unit is set to, for example, B ₁ = W ₁ × (6− (y ₀ + y ₁ + y ₂ + y ₃ ) in order to make the total time length E ₁ constant. Further, also in the R packet shown in FIG. 363 (b), High is made according to the variables y ₀ and y ₂ included in the transmission target signal, that is, according to the time lengths D ₀ and D _2. The brightness value of is adjusted.

In such an R packet, the total time length E ₀ is E ₀ = 4W ₀ + 6W ₁ + 4b + 260 = 1280 μs regardless of the signal to be transmitted. The effective time length E ₁ is a time length corresponding to the signal to be transmitted, and is in the range of 1100 to 1280 μs. In addition, while the total time length E ₀ is a constant 1280 μs, the total time length (ie, the total ON time) when the luminance value is High varies within the range of 610 to 790 μs depending on the signal to be transmitted. . Accordingly, the transmitter 100, also in this L packets, similarly to the example shown in FIG. 362, in accordance with the total ON time, i.e. according to the time length _{D 0} and _{D 2,} the luminance value of the High 80.3% Change in the range of ~ 62.1%.

As described above, in the visible light signal illustrated in FIG. 363, the maximum value of the effective time length in the packet can be shortened. Therefore, the transmitter 100 can continuously transmit one packet during the above-described red projection period by the effective time length E ₁ or E ′ ₁ .

Here, in the example illustrated in FIG. 363, the transmitter 100 generates an L packet when the sum of the variables y ₀ to y ₃ is 7 or more, and when the sum of the variables y ₀ to y ₃ is 6 or less. , R packets are generated. In other words, since the sum of the variables y ₀ to y ₃ is an integer, the transmitter 100 generates an L packet when the sum of the variables y ₀ to y ₃ is greater than 6, and the variables y ₀ to y ₃ R packet is generated when the sum of is less than or equal to 6. That is, in this example, the threshold value for switching the packet type is 6. However, the threshold for switching the packet type is not limited to 6, and may be any value from 3 to 10.

FIG. 364 is a diagram showing the relationship between the sum of the variables y ₀ to y ₃ and the total time length and effective time length. The total time length shown in FIG. 364 is the larger one of the total time length E ₀ of the R packet and the total time length E ′ ₀ of the L packet. The effective length of time shown in FIG. 364 is a time length of larger of the maximum value of the effective duration E ₁ of R packets, the maximum value of the effective time length E _'1 of L packets. In the example shown in FIG. 364, the constants W ₀ , W ₁ , and b are W ₀ = 110 μs, W ₁ = 15 μs, and b = 100 μs, respectively.

As shown in FIG. 364, the total time length changes according to the sum of the variables y ₀ to y ₃ , but the sum is about 10 and becomes the minimum. Further, as shown in FIG. 364, the effective time length changes according to the sum of the variables y ₀ to y ₃ , but the sum is about 3 and becomes the minimum.

Therefore, the packet type switching threshold may be set in the range of 3 to 10 depending on which of the total time length and the effective time length is to be shortened.

FIG. 365A is a flowchart showing a transmission method according to the present embodiment.

The transmission method in the present embodiment is a transmission method for transmitting a visible light signal by a change in luminance of a light emitter, and includes a determination step S571 and a transmission step S572. In the determination step S571, the transmitter 100 modulates the signal to determine a luminance change pattern. In the transmission step S572, the transmitter 100 transmits a visible light signal by changing the luminance of red expressed by the light source included in the light emitter according to the determined pattern. Here, the visible light signal includes data, a preamble, and a payload. In the data, a first luminance value and a second luminance value smaller than the first luminance value appear along the time axis, and at least one of the first luminance value and the second luminance value is displayed. The length of time during which one continues is less than or equal to the first predetermined value. In the preamble, each of the first and second luminance values appears alternately along the time axis. In the payload, the first and second luminance values appear alternately along the time axis, the length of time that each of the first and second luminance values continues is greater than the first predetermined value, and It is determined according to the above signal and a predetermined method.

For example, data, preamble, and payload are invalid data, preamble, and L data part or R data part shown in (a) and (b) of FIG. 363, respectively. For example, the first predetermined value is 100 μs.

Thus, as shown in FIGS. 363 (a) and (b), the visible light signal has one waveform payload (that is, L data portion or R data portion) determined according to the signal to be modulated. Contains no two payloads. Therefore, the visible light signal, that is, the packet of the visible light signal can be shortened. As a result, for example, even if the light emission period of red light expressed by the light source included in the light emitter is short, a packet of visible light signals can be transmitted during the light emission period.

In the payload, the first luminance value of the first time length, the second luminance value of the second time length, the first luminance value of the third time length, the second luminance value of the fourth time length Each luminance value may appear in the order of the luminance value. In this case, in the transmission step S572, the transmitter 100 determines that the first time length and the third time length when the sum of the first time length and the third time length is smaller than the second predetermined value. Is larger than the second predetermined value, the value of the current flowing in the light source is increased. Here, the second predetermined value is larger than the first predetermined value. Note that the second predetermined value is, for example, a value larger than 220 μs.

Accordingly, as shown in FIGS. 362 and 363, when the sum of the first time length and the third time length is small, the current value of the light source is increased, and the first time length and the third time length are increased. When the sum of the lengths is large, the current value of the light source is reduced. Therefore, the average luminance of the packet including data, preamble, and payload can be kept constant regardless of the signal.

In the payload, the first luminance value of the first time length D ₀ , the second luminance value of the second time length D ₁ , the first luminance value of the third time length D ₂ , the fourth in order of the second luminance value of the time length D _3, it may appear the respective luminance values. In this case, when the sum of the four parameters y _k (k = 0, 1, 2, 3) obtained from the signal is equal to or smaller than the third predetermined value, the first to fourth time lengths D ₀ to D ₃ Each is determined according to D _k = W ₀ + W ₁ × y _k (W ₀ , W ₁ is an integer of 0 or more). For example, as shown in FIG. 363 (b), the third predetermined value is 3.

As a result, as shown in FIG. 363 (b), it is possible to generate a payload having a short waveform according to the signal while setting each of the first to fourth time lengths D ₀ to D ₃ to be W ₀ or more.

When the sum of the four parameters y _k (k = 0, 1, 2, 3) is equal to or smaller than the third predetermined value, in the transmission step S572, the data, the preamble, and the payload are replaced with the data, the preamble, and the payload. You may transmit in order. In the case of the example shown in FIG. 363 (b), the payload is an R data portion.

Thereby, as shown in FIG. 363 (b), the fact that the packet of the visible light signal including the data (that is, invalid data) does not include the L data portion indicates that the receiver 200 receives the packet with the data. Can let you know.

When the sum of the four parameters y _k (k = 0, 1, 2, 3) is larger than the third predetermined value, each of the first to fourth time lengths D ₀ to D ₃ is D _0. = W ₀ + W ₁ × (A−y ₀ ), D ₁ = W ₀ + W ₁ × (By ₁ ), D ₂ = W ₀ + W ₁ × (A−y ₂ ), and D ₃ = W ₀ + W _It may be determined according to ₁ × (By ₃ ) (A and B are each integers of 0 or more).

Thereby, as shown in FIG. 363 (a), each of the first to fourth time lengths D ₀ to D ₃ (that is, the first to fourth time lengths D ′ ₀ to D ′ ₃ ) is set to W ₀ or more. However, even if the total sum is large, a short waveform payload can be generated according to the signal.

If the sum of the four parameters y _k (k = 0, 1, 2, 3) is larger than the third predetermined value, in step S572, the data, preamble, and payload are replaced with the payload, preamble, and data. You may transmit in order. In the case of the example shown in FIG. 363 (a), the payload is the L data portion.

As a result, as shown in FIG. 363 (a), the fact that the packet of the visible light signal including data (that is, invalid data) does not include the R data portion is indicated to the receiving device that receives the packet by the data. I can inform you.

The light emitter has a plurality of light sources including a red light source, a blue light source, and a green light source. In the transmission step S572, a visible light signal is generated using only the red light source among the plurality of light sources. You may send it.

Thereby, the light emitter can display an image using a red light source, a blue light source, and a green light source, and can transmit a visible light signal having a wavelength that is easy to receive to the receiver 200.

The light emitter may be a DLP projector, for example. As described above, the DLP projector may include a plurality of light sources including a red light source, a blue light source, and a green light source, but may include only one light source. That is, the DLP projector may include one light source, a DMD (Digital Micromirror Device), and a color wheel disposed between the light source and the DMD. In this case, the DLP projector is visible during a period in which red light is output among red light, blue light, and green light that are output in time division from the light source to the DMD via the color wheel. Transmit an optical signal packet.

FIG. 365B is a block diagram showing a configuration of transmitter 100 in the present embodiment.

The transmitter 100 according to the present embodiment includes a transmitter that transmits a visible light signal according to a change in luminance of a light emitter, a determination unit 571, and a transmission unit 572. The determination unit 571 determines a luminance change pattern by modulating a signal. The transmission unit 572 transmits the visible light signal by changing the luminance of red expressed by the light source included in the light emitter according to the determined pattern. Here, the visible light signal includes data, a preamble, and a payload. In the data, a first luminance value and a second luminance value smaller than the first luminance value appear along the time axis, and at least one of the first luminance value and the second luminance value is displayed. The length of time during which one continues is less than or equal to the first predetermined value. In the preamble, each of the first and second luminance values appears alternately along the time axis. In the payload, the first and second luminance values appear alternately along the time axis, the length of time that each of the first and second luminance values continues is greater than the first predetermined value, and It is determined according to the above signal and a predetermined method.

Such a transmitter 100 implements the transmission method of the flowchart shown in FIG. 365A.

The transmission method of the present invention can be used for, for example, a transmission device that transmits a visible light signal from a display or illumination, and can be used particularly for a transmission device that transmits a visible light signal from, for example, a spotlight.

100 Transmission Device 551 Reception Unit 552 Transmission Unit

Claims (15)

1. A transmission method for transmitting a signal by luminance change of a light source,
   An accepting step of accepting the dimming degree designated for the light source as the designated dimming degree;
   When the designated dimming degree is equal to or less than the first value,
   While emitting the light source at the specified dimming degree, the signal encoded in the first mode is transmitted by luminance change,
   When the designated dimming degree is larger than the first value,
   A transmission step of transmitting the signal encoded in the second mode by a luminance change while causing the light source to emit light at the specified dimming degree,
   The peak current value of the light source for transmitting the signal encoded in the second mode by a change in luminance when the designated dimming degree is greater than the first value and less than or equal to a second value. Is
   When the specified dimming level is the first value, the signal encoded in the first mode is smaller than the peak current value of the light source for transmitting the luminance change.
   Transmission method.
2. When the designated dimming degree is smaller than the third value,
   While causing the light source to emit light at the specified dimming level, the signal encoded in the first mode is transmitted by a luminance change,
   Maintaining the value of the peak current at a constant value with respect to the change in the specified dimming degree,
   The third value is less than the first value;
   The transmission method according to claim 1.
3. When the designated dimming degree is smaller than the third value,
   As the specified dimming level decreases, the light source is caused to emit light at the specified dimming level which is reduced by increasing the time for turning off the light source, and the peak current value is maintained at a constant value. ,
   The transmission method according to claim 2.
4. When the designated dimming degree is smaller than the fourth value,
   While causing the light source to emit light at the specified dimming level, the signal encoded in the first mode is transmitted by a luminance change,
   By reducing the value of the peak current as the specified dimming level decreases, the light source is caused to emit light at the specified dimming level that decreases.
   The fourth value is smaller than the second value;
   The transmission method according to claim 1.
5. Determining the time to turn off the light source so that one cycle of the time for transmitting the signal due to luminance change and the time to turn off the light source does not exceed 10 milliseconds;
   The transmission method according to claim 3.
6. A value of a peak current of the light source when the designated dimming is the first value;
   When the designated dimming degree is the maximum value, the value of the peak current of the light source is the same.
   The transmission method according to claim 1.
7. The transmission method according to claim 1, wherein a duty ratio of the signal encoded in the second mode is larger than a duty ratio of the signal encoded in the first mode.
8. When the peak current value of the light source exceeds a fifth value,
   Stopping transmission of the signal due to a change in luminance of the light source;
   The transmission method according to claim 1.
9. Measure the usage time of the light source,
   When the usage time is a predetermined time or more,
   Using the value of a parameter for causing the light source to emit light with a dimming degree greater than the specified dimming degree, and transmitting the signal by a change in brightness;
   The transmission method according to claim 1.
10. Measure the usage time of the light source,
    When the usage time is a predetermined time or more,
    Than when the usage time is less than a predetermined time,
    Increasing the pulse width of the current of the light source,
    The transmission method according to claim 1.
11. A transmission method for transmitting a signal by luminance change of a light source,
    An accepting step of accepting the dimming degree designated for the light source as the designated dimming degree;
    A transmission step of transmitting the signal encoded in the first mode or the second mode by a luminance change while causing the light source to emit light at the specified dimming degree,
    The duty ratio of the signal encoded in the second mode is greater than the duty ratio of the signal encoded in the first mode;
    In the transmission step,
    When the designated dimming level is changed from a small value to a large value, when the designated dimming level is the first value, the mode used for encoding the signal is changed from the first mode to the second mode. Switch to the mode
    When the designated dimming level is changed from a large value to a small value, when the designated dimming level is the second value, the mode used for encoding the signal is changed from the second mode to the first mode. Switch to the mode
    The second value is smaller than the first value;
    Transmission method.
12. In the transmission step,
    When the switching from the first mode to the second mode is performed, the peak current of the light source for transmitting the encoded signal by a luminance change is changed from the first current value to the first current value. Change to a second current value smaller than the current value of 1,
    When the switching from the second mode to the first mode is performed, the peak current is changed from a third current value to a fourth current value larger than the third current value;
    The transmission method according to claim 11, wherein the first current value is larger than the fourth current value, and the second current value is larger than the third current value.
13. A program for causing a computer to execute the transmission method according to claim 1 or 11.
14. A transmission device that transmits a signal by luminance change of a light source,
    A reception unit that receives the dimming degree specified for the light source as the specified dimming degree;
    When the designated dimming degree is equal to or less than the first value,
    While emitting the light source at the specified dimming degree, the signal encoded in the first mode is transmitted by luminance change,
    If the dimming degree is greater than the first value,
    A transmitter that transmits the signal encoded in the second mode by a luminance change while causing the light source to emit light at the specified dimming degree,
    The peak current value of the light source for transmitting the signal encoded in the second mode by a change in luminance when the designated dimming degree is greater than the first value and less than or equal to a second value. Is
    When the specified dimming level is the first value, the signal encoded in the first mode is smaller than the peak current value of the light source for transmitting the luminance change.
    Transmitter device.
15. A transmission device that transmits a signal by luminance change of a light source,
    A reception unit that receives the dimming degree specified for the light source as the specified dimming degree;
    A transmitter that transmits the signal encoded in the first mode or the second mode by a luminance change while causing the light source to emit light at the specified dimming degree,
    The duty ratio of the signal encoded in the second mode is greater than the duty ratio of the signal encoded in the first mode;
    The transmitter is
    When the designated dimming level is changed from a small value to a large value, when the designated dimming level is the first value, the mode used for encoding the signal is changed from the first mode to the second mode. Switch to the mode
    When the designated dimming level is changed from a large value to a small value, when the designated dimming level is the second value, the mode used for encoding the signal is changed from the second mode to the first mode. Switch to the mode
    The second value is smaller than the first value;
    Transmitter device.

Images