JP5713061B2 - Information transmitting apparatus, information transmitting method, and program - Google Patents

Description

The present invention relates to an information transmission device, an information transmission method, and a program applied to an information transmission system using a spatial transmission technology.

  In recent years, so-called digital signage has attracted attention as an information transmission system using spatial light transmission technology. Digital signage is a system that transmits information using a display device connected to a network in places other than the home, such as outdoors, transportation, stores, and public facilities.

  FIG. 1 is a usage diagram of digital signage. In this figure, an outdoor scene 100 (here, a state in town) is shown in the center. The scene 100 includes a pedestrian 101 and a vehicle 102, and includes distant buildings 103 to 105. In particular, a large display device 106 (hereinafter referred to as a display terminal 106) is provided on the wall surface of the central building 104. ) Is attached.

  The display terminal 106 is a digital signage display device and visualizes and displays information 107 sent from a server (not shown). The information 107 includes various information relating to a special sale product, for example. In the illustrated example, a picture of a wristwatch is displayed as the product. Now, there are various kinds of additional information about such special sale products, such as discount rate, special sale period, and sales location. Displaying all of the information on the display terminal 106 is a congestion of display information. This is not preferable because it makes it difficult to see the screen of the display terminal 106.

  For this reason, in digital signage, additional information or the like is transmitted using spatial light transmission technology, and product information including the additional information is reproduced and displayed on the screen of the mobile electronic device with camera 108 such as a mobile phone. ing.

  According to this, when a product advertisement (display on the display terminal 106) that is anxious in the town is found, the user can know detailed information about the product simply by shooting the advertisement using the portable electronic device 108 at hand. be able to. For example, in the illustrated example, the screen 109 of the mobile electronic device 108 has a picture 110 of the product, detailed information about the product (discount information such as 50% OFF, a discount period such as 11:30 to 15:00, etc. ) Is displayed.

  Here, the transmission of information in digital signage is based on a spatial “light” transmission technique, and in the illustrated example, light modulation regions 106 a to 106 a on a part of the screen of the display terminal 106 (four corners in the figure). 106d is provided, and required information transmission is performed by time-series changes of light in these modulation regions 106a to 106d.

  As such a spatial “light” transmission technique, for example, the following is known.

  Patent Document 1 below describes a technology of a spatial light transmission system including a light emitting unit and a light receiving unit. The gist of this technique is that “the light emitting unit logically determines the bit string that constitutes the information to be transmitted, and, according to the determination result, alternatively from the two bit pattern sequences having low correlation with each other. A bit pattern series is selected, and the light is modulated and transmitted according to the selection result, and the light receiving unit receives the light and generates a binarized signal corresponding to the intensity of the light, and binarizes the light. When the bit pattern sequence included in the signal corresponds to one of the two bit pattern sequences, the logic signal 1 or the logic signal 0 is generated to reproduce the information included in the light. Is.

  Patent Document 2 below describes a technique that realizes pointing using blinking signals of the same hue value with color, and Patent Document 3 uses a specified change in hue difference. In addition to the color change, Patent Document 4 blinks one of the colors at high speed and adds a photodiode separate from the image sensor to detect the intensity of light. Thus, a technique for realizing high-speed communication in which another signal is superimposed is described, and Patent Document 5 describes a technique for performing transmission using a combination of three color light emitters.

JP 2003-179556 A JP 2005-267169 A JP 2006-072778 A JP 2010-287820 A JP 2009-186203 A

However, in the technique of Patent Document 1, optical transmission is performed using the logic signal 1 and the logic signal 0 (logic signal 1 = light on, logic signal 0 = light off), in short, binary. Since this is only a transmission technique based on typical light flickering, it takes a considerably long time to reproduce information by receiving a signal due to binary light flickering with a general-purpose camera having a general frame rate (about 30 fps). (Approximately 2 seconds, details will be described later).
In addition, in the techniques of Patent Documents 2 to 5, any technique is affected by variations due to individual differences in hue values such as a display device or a light source on the transmission side, and stable information transmission cannot be performed. There is a problem.

Therefore, an object of the present invention is to provide an information transmitting apparatus, an information transmitting method, and a program that can perform stable information transmission.

The present invention provides a color change prepared in advance in a plurality of stages according to a signal included in information to be transmitted , a degree of color change when changing color temporally, or a distance between colors in a predetermined color space. A table in which the information is correlated, modulation means for modulating the information to be transmitted by referring to this table into a signal composed of a change in color, and temporally based on the signal modulated by the modulation means And a light emission control means for controlling light emission means for emitting light in a plurality of colors so as to emit light while changing the color.

  According to the present invention, stable information transmission can be performed.

It is a utilization state figure of digital signage. 1 is a configuration diagram of an information transmission system according to an embodiment. It is a figure which shows the modulation area | region (light emission part 5) provided in a part of display terminal of digital signage. It is a figure which shows the physical format of the optical communication of embodiment which employ | adopted color modulation. It is a figure which shows the characteristic of the color filter of a light-emitting device and a camera. It is the figure which converted the color space of FIG. 5 into HSV space. It is a figure which shows the encoding table which converts a data value into a light emission signal sequence. FIG. 3 is a diagram showing internal processing of the entire light receiving device 3. FIG. 9 is a detailed view of the process in step S3 of FIG. FIG. 9 is a detailed view of the process in step S4 of FIG. It is a figure which shows the method of shape evaluation. It is a figure which shows the buffering state of the buffer memory formed in RAM12c. It is a figure which shows the example of a candidate area | region table. FIG. 9 is a detailed view of the process in step S7 of FIG. FIG. 15 is a detailed view of the process of step S72 of FIG. It is a figure which shows the connection image of an area | region table. It is a figure which shows the image which regrouped the area | region determined to be connected continuously. It is a figure which shows the example of hue data extraction. It is explanatory drawing of a threshold setting. It is a figure which shows the example of the change which exists on a hue value.

Embodiments of the present invention will be described below with reference to the drawings.
FIG. 2 is a configuration diagram of the information transmission system according to the embodiment. In this figure, the information transmission system 1 includes a light emitting device 2, a light receiving device 3, and a service server 4.

  The light-emitting device 2 includes a light-emitting unit 5, a storage unit 6 that stores information such as a tag ID, and a modulation unit 7 that modulates and drives the light-emitting unit 5 with the information. Signage, that is, a system that transmits information using a display device connected to a network outside the home such as outdoors, transportation, stores, public facilities, etc. An example of digital signage provided with such a light emitting device 2 is shown in FIG. In FIG. 1, a predetermined area (modulation areas 106 a to 106 d) functioning as the light emitting unit 5 is provided in a part (four corners in the figure) of the display terminal 106 of digital signage. Necessary information (information such as tag ID) can be transmitted to the space by optical communication depending on the color and intensity. Thus, by providing the modulation regions 106a to 106d (light emitting unit 5) in a part of the display terminal 106, it is possible to provide information over a long distance by optical communication, and display content without reducing the display area of the signage The design impact on can be minimized.

  The light receiving device 3 includes an optical system 8 including a photographing lens, a camera 9, an input sensor 10, a display unit 11 such as a liquid crystal display, a control / communication unit 12, and the like. As this light-receiving device 3, portable electronic devices (for example, refer to portable electronic device 108 of Drawing 1), such as a cellular phone with a camera (or smart phone), can also be included, for example. The service server 4 is a server that operates, for example, an information providing site on the Internet and a product sales site (so-called net shop) linked to information transmitted by digital signage. In the case of a merchandise sales site, a user of a portable electronic device can directly purchase merchandise obtained through digital signage from a net shop on the Internet.

  The camera 9 of the light receiving device 3 is composed of a two-dimensional imaging device such as a CCD or a CMOS on which a color filter is mounted. An image within a predetermined angle of view is captured at a period of several tens of frames per second and output to the control / communication means 12. To do. Here, the imaging cycle of the camera 9 is set to 30 frames per second (30 fps) following the example of a general (also called general-purpose) two-dimensional imaging device, and the modulation frequency in the modulation means 7 of the light emitting device 2 is half that, that is, , 15 Hz.

The input sensor 10 of the light receiving device 3 is, for example, a sensor for detecting various types of information input by a user operation, and is specifically a QWERTY keyboard or a touch panel including a numeric keypad.
The display means 11 of the light receiving device 3 is a high-definition display device such as a liquid crystal display, and visualizes and displays any information appropriately output from the control / communication means 12.

  The control / communication means 12 of the light receiving device 3 includes a communication interface with the service server 4, a computer or microcomputer (hereinafter referred to as CPU) 12 a, a read-only semiconductor memory (hereinafter referred to as ROM) 12 b, and writing / reading. Is a control element of a program control system including a semiconductor memory (hereinafter referred to as RAM) 12c and a peripheral circuit (not shown). A control program stored in the ROM 12b in advance is loaded into the RAM 12c and executed by the CPU 12a. Various processes are executed sequentially to control the overall operation of the light receiving device 3. The ROM 12b may be a write-type nonvolatile semiconductor memory (PROM, EPROM, etc.).

[Modulation method and physical format]
FIG. 3 is a diagram showing a modulation region (light emitting unit 5) provided in a part of a display terminal of digital signage. As shown to (a), the light emission part 5 is expressed as a collection of several pixels in the predetermined part (for example, corner part of a terminal) of the display terminal of digital signage. A set of x2 pixels (or pixels; hereinafter, unified with pixels).

  The periphery of the light emitting unit 5 composed of 2 × 2 pixels is surrounded by a pixel frame (a frame composed of 12 pixels labeled Bk here) for separation from other display images of digital signage, These pixel frames are always black pixels that are not lit, but the light-emitting unit 5 (2 × 2 pixels) for optical communication surrounded by the pixel frame emits light at a predetermined cycle while changing the color tone and intensity. Can be done. For example, when all the light emitting parts 5 for optical communication are turned on in red (R), the state becomes as shown in (a), or when all the light emitting parts 5 for optical communication are turned off, as shown in (b). All of the 2 × 2 pixels are black (Bk).

  Note that the shape of the twelve pixels for the separation frame is not limited to this example. In the figure, 12 pixels form a quadrangle. However, for example, a round shape or other shapes such as an ellipse, a rhombus, and a rectangle may be used. In the figure, a 4 × 4 pixel configuration including 12 pixels for the separation frame is used, but the present invention is not limited to this. In an actual pixel configuration, it may be configured by a larger number of pixels. The number of pixels is determined solely by the balance with other display image areas of digital signage.

FIG. 4 is a diagram illustrating a physical format of optical communication according to the present embodiment. In this figure, the physical format 13 is one of the three colors of red (R), blue (B), and green (G), followed by a header portion 13a consisting of one non-emission (black) pulse and then 9 pulses. And a data portion 13b that lights up. The number of colors (three colors) of the data portion 13b is three values of R, G, and B with respect to the binary values of black (non-lighting) and white (lighting) described at the beginning. Will be going.
The reason why one pulse of “black” is used for the header is that the luminance value of “black” is clearly greatly different from that of other chromatic colors, and is easily separated without being affected by the color mixture.
In this figure, three colors of red, blue, and green are used. That is, ternary modulation is performed. However, the present invention is not limited to this. For example, three colors of cyan, magenta, and yellow may be used. Alternatively, seven colors obtained by adding white to them may be used, and in short, a multi-value exceeding three values may be used. Which color configuration (multi-value number) is adopted is exclusively a design matter. For example, an appropriate color configuration may be adopted in consideration of the color separation characteristics of the camera unit 9 and the AWB (auto white balance) characteristics.

FIG. 5 is a diagram illustrating characteristics of the color filter of the light emitting device and the camera. As shown in this figure, even when only red is lit, green wavelength components and blue wavelength components are slightly included, and in reality, color separation is often poor.
For this reason, in the present embodiment, three primary colors, red, blue, and green, which are relatively easy to separate, are used. Note that, as long as color separation is possible, the present invention is not limited to this, and complementary colors such as cyan, magenta, and yellow may be used. Further, a combination of multiple colors such as red, blue, green, and yellow may be used as long as an algorithm process capable of color separation can be executed.

  FIG. 6 is a diagram in which the color space of FIG. 5 is converted to HSV. As you can see, the chromatic colors R, G, B, etc. have a certain saturation or more, and even if the brightness decreases due to the AWB functioning due to the nature of the saturation, the saturation itself does not change. It can be seen that the distinction is easy.

  Therefore, by adopting “color modulation” (see FIG. 4) as in the present embodiment, if it is desired to output 12 bits of the data portion, for example, if a three-color modulation method (three-value modulation) is used, “ 3 ^ 9 ≧ 14 bits ”, and 14 pulses or more can be expressed by 9 pulses.

  FIG. 7 is a diagram showing an encoding table for converting a data value into a light emission signal sequence. This encoding table is stored in advance in the ROM 12 b of the control / communication means 12 of the light receiving device 3. In this figure, only the encoding for three pulses is shown, but this is for convenience of explanation. That is, if it is shown for 9 pulses, it becomes a huge table, and in order to simplify the table, the case where it is divided every 3 pulses is described.

  The encoded signals “1”, “2”, and “3” in this table represent “red”, “blue”, and “green”, respectively. “123” indicates that light is emitted in the order of “red → blue → green”. For example, when the encode signal is “132”, that is, when light is emitted in the order of “red → green → blue”, four types of encode results are obtained. For example, in the illustrated encoding table, “8” is obtained as the first encoding result, “7” is obtained as the second encoding result, “7” is obtained as the third encoding result, and “2” is obtained as the fourth encoding result. . The first encoding result is one of 27 values from 1 to 27, the second encoding result is one of 24 values from 1 to 24, and the third encoding result is 1 to 27. 8 is one of eight values up to 8, and the fourth encoding result is one of six values from 1 to 6. Hereinafter, the first encoding result is “no redundancy”, the second encoding result is “redundancy / weak”, the third encoding result is “redundancy / medium”, and the fourth encoding is performed. The result is called “redundancy / strong”.

  In the case of “no redundancy”, 27 values can be indicated by 3 pulses. For this reason, a lot of information can be transmitted at a much higher speed than what is conventionally performed with binary values (specifically, four values with four pulses). However, in the case of “no redundancy”, it is not practical because a pixel region in which the same color continues for a certain period of time is likely to be mistaken as a part of a background image that does not change.

  On the other hand, if only “111”, “222”, and “333” are eliminated and “redundancy / weak” is used to provide a slight redundancy, a background including a pixel area that does not change color for a certain period of time. The information can be easily deleted from the candidate pixels for information demodulation.

  In addition, what is indicated by “redundancy / medium” is encoded with a difference (distance) in hue from the first color as a reference. This is because an information signal can be obtained even if the hue rotates due to a change in color due to the AWB on the camera side.

  Furthermore, with “redundancy / strong”, the obtained value becomes six values, but since different hues enter each of the three pulses, there is an advantage that the influence on noise can be sufficiently eliminated.

  These encoding results (no redundancy / redundancy / weak / redundancy / medium / redundancy / strong) may be appropriately selected according to the environment and characteristics of the device in which the light emitting device 5 is installed.

Next, the operation of the light receiving device 3 will be described.
FIG. 8 is a diagram showing internal processing of the entire light receiving device 3. In this process, first, the frame counter on the RAM 12c is reset (step S1), the frame output from the camera 9 is captured in the RAM 12c (step S2), a binary image of the frame is generated (step S3), and the RAM 12c. A candidate area table is created (step S4) and registered in the list (step S5). Then, it is determined whether or not the frame counter is a predetermined value n (step S6). If it is not the predetermined value n, the process returns to step S2, and if it is the predetermined value n, a decoding process is performed (step S7). Is displayed (step S8), and step S1 and subsequent steps are repeated again.

  In this way, in the processing from step S1 to step S6, the image captured in the RAM 12c is analyzed, a candidate area table is created in the RAM 12c, and the captured frames are stored in a time-series list (hereinafter, referred to as “list”). (Table list) is created.

More specifically, in step S3, the following processing is performed.
[Detection of candidate area for each frame and registration in candidate area table]
(A) Color twist (color enhancement correction process):
When the modulated light by digital signage is photographed by the camera 9, if expressed using the values in the RGB color space as they are, depending on the performance of the camera 9 and the background color of the place where the modulated light exists, the color separation performance in the modulated light However, in the HSV color space processing, there may be a disadvantage that the performance is deteriorated when the saturation S and the hue value H are separated. Therefore, as a process prior to the HSV color space process, color conversion from RGB to R′G′B ′ is performed using a conversion matrix as shown in the following equation (1). By this processing, a thing with high saturation can be made more conspicuous and can be easily separated.

  Each component a, b, and c of the matrix of Formula (1) is a value like the following Formula (2), for example. The vector space is a process of drawing values inside the color space closer to the respective RGB axes.

a = (a11, a22, a33) = (− 1.7, −0.65, −0.1)
b = (b11, b22, b33) = (− 0.9, 1.9, −0.1)
c = (c11, c22, c33) = (− 0.1, −0.1, 1.1)
(2)

(B) Convert to HSV color space:
The saturation of the image obtained by photographing the light color-modulated by digital signage or the like with the camera 9 is almost unchanged without being influenced by the surrounding environment. On the other hand, each RGB color-separated color vector value is susceptible to the influence of the surrounding environment. In order to cope with such a property in the conventional luminance-based search, the search is performed by converting to the HSV color system instead of expressing in RGB.

  In the case of the color modulation method and the color modulation as shown in the physical format, the imaging result of the pulse in the data portion is always the saturation (S) regardless of the emission color even if the hue slightly changes. ) Is a high value. Further, if it is a transmission source that emits light, it may be handled as having a constant brightness. Only in the case of a black pulse in the header, the lightness and saturation are lowered, and it shows a constant property in the color variation, which is convenient for searching for a region candidate within a frame unit.

  FIG. 9 is a detailed view of the process in step S3 of FIG. As shown in this figure, in the process of step S3, the captured image in the RGB color space is converted into an image in the HSV color space (step S31), and S (in the image expressed in the converted HSV color space is displayed. For each pixel of an image expressed by a saturation (saturation) parameter (hereinafter referred to as S image) and an image expressed by a V (brightness: I (Intensity)) parameter (hereinafter referred to as V image), Appropriate threshold values are set and binarized (step S32). Then, a product obtained by ANDing the obtained S image and V image is set as a black and white binary image representing a candidate communication region (step S33).

[Labeling process]
By this processing, an image of a region that includes the light-modulated region excluding the header portion 13a and that coincides with the color characteristics by chance is obtained as a monochrome binary image. Thereby, for example, an image in which a region with a high likelihood of color features is white on a black background is obtained.

  FIG. 10 is a detailed view of the process in step S4 of FIG. As shown in this figure, first, so-called labeling processing is performed to identify continuous regions and basic shape parameters (step S41). More specifically, a process for identifying each white region connected to the candidate image and obtaining its shape information is performed. In the present embodiment, it is assumed that the center of gravity, the area (pixel area) of the region, and the four corner coordinates of the circumscribed rectangular region are obtained.

  In the subsequent process, one continuous area obtained by the process is extracted (step S42), and narrowing down is performed according to the shape condition. First, an area that is too small in area (for example, 2 × 2 pixel angle) may be excluded as noise (step S43).

  Next, the shape of the area detected by the result of the labeling process in step S4a is evaluated. In the present embodiment, in order to simplify the process, the aspect ratio is used as the shape likelihood (step S44).

FIG. 11 is a diagram showing a shape evaluation method. 11A and 11B are diagrams showing the aspect ratio of the target shape (outlined portion), where H is the long side length of the circumscribed rectangle of the target shape, and W is the short side length of the circumscribed rectangle of the target shape. is there. FIGS. 11C and 11D are diagrams showing the area filling rate of the target shape with respect to a predetermined area P (10 × 10 pixels in the figure), and this area filling rate is the area of the target shape (outlined portion). A is obtained by dividing A by a predetermined area P (10 × 10 pixels).
As shown in FIG. 3 above, the modulation area is a regular square, and considering the positional relationship with the camera 9, the aspect ratio is preferably 1.0, but the aspect ratio is as shown in FIGS. As shown in b), a threshold value is set after taking into account that it is a circumscribed rectangle and is distorted due to the positional relationship with the camera 9, etc., and below that (obviously different in shape) Be excluded.

  Next, since the area actually captured may not be seen as a square depending on its shape, the area of the circumscribed rectangle and the area in it are used to easily determine whether or not it is such an area. A threshold value close to an ideal value with an area ratio of 1.0: 1.0 is provided, and those that do not have an area filling rate equal to or greater than the threshold value (those that are too small) are excluded (step S45).

  Since it is highly possible that a region that has not been excluded by performing such processing is a modulation region (see 106a to 106d in FIG. 3), there is a possibility of a modulation region after registration in the candidate region table (step S46). Until it is determined that the processing has been completed for all candidate regions (YES in step S47), the above steps S42 to S45 are repeatedly executed.

  As a result, as many list entries as necessary frames from step S1 to step S6 are obtained and registered as a table list. However, since the phase relationship is indefinite, sampling by frame is twice the pulse period, that is, 30 fps for 15 Hz pulse-based modulation. Therefore, since the number of constituent pulses of the block is 10 pulses according to the above description, n = 2 × 10 = 20 is this list entry, which is a buffering state as shown in FIG.

  In actual implementation, the frame analysis is performed, and the number of detected candidate areas continues to be zero or more in the process of creating the candidate area table and adding to the table list. If there is a shooting situation (considering surrounding environment, camera shake, etc.) or a situation that is clearly not suitable for communication, the processing may be reset in the middle of the process.

[Narrowing process of decoding candidate]
FIG. 12 is a diagram showing a buffering state of the buffer memory formed in the RAM 12c. The upper F0 to Fn are frames, and the lower table represents the buffering state of the candidate area table for each frame F0 to Fn.
The candidate area table is prepared for n (n is a natural number) frames, and when the predetermined number of frames is reached, the contents are rewritten.
Furthermore, the reason for using the candidate area table for each frame is that, when processing time-series images, information is compressed as a candidate area table instead of processing multiple images in the time direction at the pixel data level. This is to drastically reduce the calculation amount by using.

Here, an example of a candidate area table generated for each captured (imaged) frame will be described.
FIG. 13 is a diagram illustrating an example of a candidate area table with a frame number Fn = 0. In this figure, the region No. For each of (A), centroid coordinates (cx, xy), area (Size), and center coordinate optical signal value (here, hue value Hue) are illustrated. 0, centroid coordinates (10, 50), area (70), and center coordinate optical signal value (80) are illustrated. 1 exemplifies the barycentric coordinates (111, 321), the area (23), and the central coordinate optical signal value (200). Thus, in the present embodiment, the barycentric coordinates (cx, xy), the area of the area (Size), and the optical signal value (Hue) of the area barycentric coordinates are sequentially stored in the candidate area table. Go.

  In the following description, identification of individual candidate areas is expressed as Fn: Am, and identification of the internal parameters is expressed as Fn: Am: (cx, cy). The evaluation value indicates the “closeness” of the feature amount obtained by the single image analysis, that is, the likelihood that the feature value is derived from the same signal.

  Typically, using the centroid coordinates of the area, the distance between two area centroids (x1, y1) and (x2, y2) is converted into a normal two-dimensional distance “√ ((x2−x1) ^ 2 + (Y2-y1) ^ 2) "is most generally considered to be evaluated, but in the present embodiment, area information is used in order to obtain a more reliable likelihood.

  However, since the area is already a square dimension with respect to the coordinates, consideration should be given so that it can be added as an evaluation scale of the same dimension. By the way, if this square root calculation is not performed, a slight difference in area will be effective as a square, and the degree of deviation between the center of gravity distance and area cannot be considered heavy.

√ ((x2-x1) ^ 2 + (y2-y1) ^ 2
+ (√Size2-√Size1) ^ 2) (3)

  In short, the expression (3) means that after calculating the square root of the area, the normal distance calculation of the vector composed of the third order is performed. Actually, since this expression (3) is operated within a relatively small threshold value (for example, 0 to 10), it is possible to obtain an evaluation value of the same level while reducing the amount of calculation such as the square. Alternatively, the following equation (4) may be used.

(X2-x1) + (y2-y1)
+ √ (Size2-Size1) (4)

  In both formulas (3) and (4), it can be considered that “the evaluation value is small” = “the same region”.

FIG. 14 is a flowchart showing details of the process in step S7 of FIG.
In this flowchart, a candidate point of a processing target frame (precisely, this data is area structure data that holds area information, hue value, etc. in addition to coordinate data, but will be referred to as “candidate point” in the following description) and 1 Candidate points in frames acquired in the past for three frames are input to the above equation (4) to calculate an evaluation value. Then, using this evaluation value, a linked list is created between the candidate points of the frame to be processed and the candidate points of the frames acquired 1-3 frames in the past from this frame (step S71).
Next, while interpolating black (non-light-emitting) candidate points that do not exist as black data within the allowed range (for this time, 2 frames), the linked list between the two frames is sequentially integrated and linked, A series of candidate point groups over 18 frames is determined (step S72, see FIG. 15 for details).
Next, the elements of discontinuous connection in which 3 frames have passed without any candidate point being deleted are deleted, and the objects in which the candidate point group for 18 frames has been collected are rearranged so that the start of the connection is black (step S73). .
(Note: In this physical format, black means header, there are 1-2 pulses, and no more.)
Then, a completely connected region consisting of related candidate point groups for 18 frames is extracted (step S74), the Hue value (optical signal value sequence) corresponding to each candidate point is decoded (step S75), and an effective decoding result is obtained. It is determined whether (decoded value) has been obtained (step S76).
If it is obtained, the display system processing coordinates and data request queuing are performed so that the balloon 111 (see FIG. 1) and the like are displayed (step S77), and a candidate point group for all 18 frames. It is determined whether or not the process has ended (step S78). Even if it is not obtained, the end of processing for the candidate point group for all 18 frames is determined as it is. In any case, if it is not determined that all the processes for the candidate point group for all 18 frames have been completed, step S74 and subsequent steps are repeated.

FIG. 15 is a flowchart showing details of the process in step S72 of FIG.
In this flowchart, first, a processing target frame Fx is defined, and this is advanced by one in the forward direction in time (step S721), one region candidate Ax is extracted from the processing target frame (step S722), and the processing target frame Fx is processed. Is defined as Fd = Fn + 1 (step S723).
Next, evaluation values of the region candidate Ax and each element of the processing target frame Fd are calculated using the above equation (4) (step S724), and the minimum connection combination is determined based on the evaluation value (step S725). ).
Next, it is determined whether or not the determined evaluation value of the minimum connection combination is equal to or less than a preset threshold (step S726). If it is determined that it is not below the threshold, it is determined whether or not the current processing target frame Fd is the Fn + 3th frame (step S728). If it is determined that the processing target frame Fd is the Fn + 3th frame, it is determined that a region to be connected has not been found (step S729), and step S722 and subsequent steps are repeated.
On the other hand, if it is determined in step S728 that the processing target frame Fd is not the Fn + 3th frame, dummy data (“skip”) is entered and the processing target frame Fd is moved forward by one in time. (Step S730), Step S724 and subsequent steps are repeated again.
On the other hand, when it is determined in the previous step S726 that the evaluation value of the minimum connection combination is equal to or less than the preset threshold, the current list is registered as an adjacent list in the connection list (step S727), and the processing target frame It is determined whether or not all region processing has been completed (step S731). If it is determined that it has not been completed, the processing from step S722 is repeated again. If it is determined that it has been completed, it is determined whether or not the connection evaluation of all frames has been completed (step S732), and it is determined that it has not been completed. If so, the process returns to step S721, and if it is determined that the process has been completed, the process ends.

In the present embodiment, in the above formula (4), for example, an evaluation value of 30 or less is regarded as “there are some movements and shape variations but are regarded as the same”.
In the physical format of this embodiment, although the connection is made when the saturation and the lightness are high, the header (black) is always included, so the connection of the area table for each frame when the processing of FIG. 15 is performed is shown in FIG. As shown in FIG.

FIG. 16 is a diagram illustrating a connection image of region tables.
In this image, a portion connected by a solid line indicates a connection state of each candidate region performing visible light communication.
On the other hand, the dotted line indicates a situation where it is determined that the visible light communication is clearly performed but is connected at the evaluation value level.

  In step S73, the obtained connection is evaluated. At this time, in the area where the modulation signal exists, there are always paths (connection of decoding results) connected in all 20 frames. Therefore, other areas are deleted.

  FIG. 17 is a diagram illustrating an image obtained by rearranging areas determined to be continuously connected. Then, an image in which the connection is skipped is set to black (extinguish), complemented with saturation 0, and the others are extracted with hue data of the area Am.

  FIG. 18 is a diagram illustrating an example of hue data extraction when the hue value (Hue) has a range of 0 to 360. As illustrated in FIG. 17, the results are finally rearranged as connection candidates. Correspondingly, only the hue values held by each candidate area element data (individual, F0: A0, etc.) are arranged.

  In this way, a candidate for temporal connection is determined from the color / shape candidates of one frame. In the present embodiment, the optical signal value of the connection region in the previous figure (when the hue value Hue is changed to the connection in this embodiment) is assumed to have a value range of 0 to 360 according to the general definition. 18 and so on. When it is determined that the light is off and skip is determined, the value is clearly excluded from the hue value range (for example, −1).

  Returning to the flowchart of FIG. 14, the CPU 12a first eliminates a connected area that cannot exist as a modulation area. It is a candidate area for the condition “there is no header section 13a (−1 state)”. When 19 frames are sampled in the above-described physical format, there is always a turn-off time of 1 to 2 (depending on the phase relationship between the signal pulse and the frame capture timing) in the signal region. "No continuation" is excluded (taken from -1 to another value, and -1 is not a signal). A value series in the region that satisfies this condition is selected. For example, taking FIG. 18 as an example, area number 1 is excluded because it has no turn-off timing.

Next, the hue value series is cyclically shifted so as to start from one or two consecutive -1. Note that there are cases where a plurality of visible light communication sources are asynchronous with each other (whether pulse level or data block level), so this circulation is also performed for each.
The process up to this point is the process of step S73.

  In step S74 and step S75, the validity of each connection result as modulation is confirmed, and the phase is selected and demodulated. In this embodiment, it is assumed that the threshold type and color decoding process is simple (generally general purpose / not optimal). However, since there is a frame drop in the implementation, such simple phase selection is performed. Not. In the present embodiment, for convenience of explanation, this simple threshold process will be described as a simple example.

[Threshold setting]
FIG. 19 is an explanatory diagram of threshold setting. As shown in this figure, on the hue axis, a threshold in a range that can be taken by the color issue pulse of the modulation signal is set in consideration of the color characteristics (including the influence of dynamic AWB control) on the transmission side light source and the camera side. . For example, the R threshold is around 0 (or 360), the G threshold is around 120, and the B threshold is around 240.
In this embodiment, a fixed threshold is used, but this threshold is set according to the environment that matches the characteristics of the camera 9 or is dynamically optimized by setting a threshold in the valley of the peak of the hue distribution. You may do it.

  Furthermore, it is desirable to optimize these threshold values for each light emitting point because a plurality of light emitting signals having different color characteristics can be received more stably.

  As described above, since the color changing at 15 Hz (pulse period 66 ms) is sampled by the camera 9 at 30 fps, it can be considered that the 18 sample series is composed of two phases A and B. As described above, an example of a change on the hue value starting from the peak that can be regarded as extinguished is as follows.

  FIG. 20 is a diagram illustrating an example of a change on the hue value. In this figure, the vertical axis represents the hue value, the horizontal axis represents the frame number, and the A phase and the B phase are arranged along the horizontal axis. As shown in this figure, there are various change patterns on the hue value, but here, a good example shows a case where the A phase is relatively the optimum phase and the B phase is always a color change. In the phase relationship with the light source, both the A phase and the B phase may take values within the threshold. For example, when R-> B near 0, the intermediate value is not G, but around 300 between B and Upper R. On the other hand, when going from R closer to 340 to G, the intermediate value is not around the B phase but around Y 60.

  In any case, when the region candidate is a modulated signal, one of the A phase and B phase sequences is always in the range threshold.

  The communication area can be determined as described above, and the change in the optical signal at the observed value level can be used as the color information string applied to the decode table. Thereafter, the transmission bit data can be obtained by comparing the color information sequence with the decoder table of FIG.

Of course, noise regions that coincided by chance, such as saturation, spatial shape, temporal connectivity, etc. are also removed by the rule of FIG. 7 with redundancy, so it is here that changes in the natural world are regarded as accidental data. It is thought that there will be almost no at all.
In the processing of this embodiment, fluctuations in the natural world may coincide by chance, but it is desirable to prevent reception errors by error detection and correction of higher layers.

Since it is as above, according to this embodiment, there can exist the following effects.
A. Since the optical transmission system is color-modulated to at least ternary values, for example, if ternary modulation is used, “3 ^ 9 ≧ 14 bits”, and 14 pulses or more can be expressed by 9 pulses. As a result, the transmission time can be shortened.
B. In the decoding process of image sensor communication, the time direction processing of frames is performed on a table basis, so that the amount of processing can be dramatically reduced.
C. D. Since the temporal change of the region is performed by the change of the center of gravity and the size, the connection determination with high validity becomes possible. Since the transmission side is fixed-length repetitive transmission, the response time for communication acquisition is obtained by storing the data block length without searching for the header and starting sampling for communication, and then searching for the header in that data block. Can be minimized.

The above embodiment may be modified as follows. For example, shape analysis data such as a more detailed value of shape feature data, a second moment around the center of gravity (= inclination direction, etc.) is added in order to further improve the temporal connectivity in the region. May be.
Further, in the temporal connection of regions, it is preferable to eliminate the influence of blurring and moving components of the entire image, such as camera shake, on the entire elements in advance because a more stable detection can be performed on a fixed object.
In addition, if region connection is performed and the connection determination is made in more detail based on the validity of the movement vector variation between the connection candidates, even if the size is the same as in this embodiment, it is canceled due to a large amount of movement. It is possible to cope with such a case where the movement is intense.

As mentioned above, although several embodiment of this invention was described, this invention is not limited to these, The invention described in the claim, and its equal range are included.
The invention described in the claims of the present application will be appended below.

(Appendix 1)
The invention described in claim 1
A light emitting means for transmitting information by shining a predetermined area;
A light receiving unit that images the predetermined region in time series, decodes the information from the captured image, and outputs the decoded information;
The predetermined area emits color-modulated light that has been multi-valued into at least three values according to the information, and the light receiving means is based on the multi-valued color modulation information of the predetermined area. An information transmission system for decoding the information.

(Appendix 2)
The invention according to claim 2
Imaging means for periodically imaging a subject at a predetermined frame rate;
A predetermined area included in a subject imaged by the imaging unit is extracted from a periodically captured image of the imaging unit, and light transmission is performed using the predetermined area as a light source from a time-series light change mode of the predetermined area. Decoding means for decoding the received information;
Output means for outputting the information decoded by the decoding means,
The decoding means is a light receiving device that decodes the information based on color modulation information multi-valued into at least three values of the predetermined region.

(Appendix 3)
The invention described in claim 3
An information transmission method in a communication system including: a light emitting device that transmits information by illuminating a predetermined region; and a light receiving device that images the predetermined region in time series and decodes and outputs the information from the captured image Because
The predetermined area emits color-modulated light that has been multi-valued into at least three values according to the information, and the light receiving device is based on the multi-valued color modulation information of the predetermined area. The information transmission method is characterized by decoding the information.

(Appendix 4)
The invention according to claim 4
A computer of an electronic device having an imaging means;
Information obtained by extracting a predetermined area included in the subject in the captured image periodically picked up by the imaging unit and optically transmitting the predetermined area as a light source from a time-series light change mode of the predetermined area Decoding means for decoding
Functioning as output means for outputting information decoded by the decoding means, and the decoding means decodes the information based on color modulation information multi-valued into at least three values of the predetermined area. Is a program characterized by

DESCRIPTION OF SYMBOLS 1 Information transmission system 2 Light-emitting device 3 Light-receiving device 5 Light emission part 6 Memory | storage means 7 Modulation means 9 Camera 11 Display means 12 Control / communication means 12a CPU (computer)

Claims (7)

A signal included in information to be transmitted , color change information prepared in advance in a plurality of stages according to the degree of color change when changing color temporally or the distance between each color in a predetermined color space , A table associated with
Modulation means for modulating information to be transmitted into a signal consisting of a change in color by referring to this table;
A light emission control means for controlling the light emission means for emitting light in a plurality of colors so as to emit light while changing the color temporally based on the signal modulated by the modulation means;
An information transmission device comprising: The information transmission apparatus according to claim 1, wherein the color change configuration is determined according to a color separation characteristic or an auto white balance characteristic of a light receiving sensor included in a device that receives the color change . The information transmitting apparatus according to claim 2, wherein the light receiving sensor includes an image sensor .   4. The information transmitting apparatus according to claim 1, wherein the light emitting means is a partial pixel region of the video display means that emits light in a plurality of colors in units of pixels.   5. The information transmission apparatus according to claim 4, wherein a frame display area for discriminating from a video displayed outside the area is set around the partial pixel area. A signal included in information to be transmitted and color change information prepared in advance in a plurality of stages according to the degree of color change when changing color temporally or the distance between each color in a predetermined color space. A modulation step for modulating information to be transmitted into a signal composed of a change in color by referring to the associated table;
A light emission control step for controlling light emitting means for emitting light in a plurality of colors so as to emit light while changing the color temporally based on the signal modulated in this modulation step;
An information transmission method comprising: Computer
A signal included in information to be transmitted and color change information prepared in advance in a plurality of stages according to the degree of color change when changing color temporally or the distance between each color in a predetermined color space. Modulation means for modulating information to be transmitted into a signal composed of a change in color by referring to the associated table;
A light emission control means for controlling the light emission means for emitting light in a plurality of colors so as to emit light while changing the color temporally based on the signal modulated by the modulation means;
A program characterized by functioning as

Images