JP6433014B2 - Information acquisition apparatus and information transmission system - Google Patents

Description

  The present invention relates to an information acquisition device that acquires information embedded in a display image displayed by a display device such as a liquid crystal monitor or a projector, and an information transmission system including the information acquisition device.

  In recent years, image sensors are mounted on various electronic devices such as mobile terminals and robots, and various systems using an imaging function of such electronic devices have been proposed. Among such systems, a system in which a display device and an electronic device operate in cooperation (specifically, the electronic device captures some information when the electronic device captures a display image displayed by the display device). The system that is used for operation) is attracting particular attention.

  In a system in which a display device and an electronic device operate in cooperation, for example, augmented reality (Augmented Reality) in which an image of an object that does not exist in a mobile terminal that captures a display image and displays a preview image is combined and displayed in the preview image. : AR) The display device (or the mobile device) that identifies the display device (or the computer that generates the display image data and outputs the display image data) by displaying or displaying the display image. , Remote operation of the computer), autonomous operation of a robot that operates based on information acquired by capturing a display image, and the like are possible.

  However, in a system in which the display device and the electronic device operate in cooperation as described above, the “structure” for acquiring information is displayed in the display image so that the electronic device that captured the display image can acquire the information. It is necessary to provide in.

  As the simplest method, a method of displaying a marker representing the information in a display image can be considered. However, this method causes a problem that the display image becomes unsightly. Therefore, the “structure” can be recognized by an electronic device that captures a display image, but is required to be unrecognizable (or extremely difficult to recognize) by the human vision of the display image. .

  Systems that satisfy this requirement have been proposed in Patent Documents 1 and 2 and Non-Patent Documents 1 and 2. Note that the systems proposed in Patent Document 2 and Non-Patent Document 2 are the same.

  In Patent Literature 1, a display device of which display device is picked up by a light pen that picks up an image on the display surface of the display device by changing the method of modulating the luminance of the backlight in each of the plurality of display devices. A system has been proposed that makes it possible to identify whether or not.

  Non-Patent Document 1 includes a projector that projects a two-dimensional code in an extremely short time that cannot be recognized by human vision, and the projection is performed by synchronizing the timing of projection by the projector with the timing of imaging by the camera. A system for generating image data in which a two-dimensional code can be identified has been proposed.

  In Patent Document 2 and Non-Patent Document 2, two colors (blue and yellow), which are complementary colors, are displayed at high speed switching, and a continuous display of a single color (white or gray) formed by additively mixing the two colors is human. A system that embeds information in a display image by utilizing the fact that cannot be identified has been proposed.

  Specifically, in this system, a portable terminal equipped with a rolling shutter type image sensor employed in a CMOS (Complementary Metal Oxide Semiconductor) image sensor or the like is used. Note that the rolling shutter method is to sequentially generate pixel data having different exposure timings for each unit by sequentially driving each imaging pixel circuit constituting the image sensor for each predetermined unit (for example, row unit). Further, it is a method of generating one image data by connecting these pixel data. In addition, the display device uses the above-described two types of display methods to display “cells” provided in a predetermined area in a display image (a blank area in which characters and drawings are not displayed, for example, an outer frame area). By selectively displaying, information is embedded in the display image.

  When this “cell” is imaged by a mobile terminal equipped with a rolling shutter image sensor, the “cell” that performs high-speed switching display of two colors is in the blue display period in the image data generated by the mobile terminal. A state in which the blue pixel data obtained by imaging and the yellow pixel data obtained by imaging during the yellow display period are mixed (specifically, a blue and yellow striped pattern state) It becomes. On the other hand, all the “cells” that are continuously displayed in a single color are filled with the single color in the image data generated by the mobile terminal. Therefore, in the mobile terminal, the region corresponding to the “cell” in the image data generated by capturing the display image is in a state where two colors (blue and yellow) are mixed, or is filled with a single color (white or gray). Information is obtained by identifying whether it is in a state of being recorded.

JP 2014-41540 A US Patent Application Publication No. 2013/0089133

Anselm Grundhofer, et al., “Dynamic Adaptation of Projected Imperceptible Codes”, In Proc. ISMAR 2007, pp.181-190. Grace Woo, et al., “VRCodes: Unobtrusive and Active Visual Codes for Interaction by Exploiting Rolling Shutter”, In Proc. ISMAR 2012, pp.59-64.

  However, in the system proposed in Patent Document 1, a modulation device for modulating the luminance of the backlight is added to the backlight, and a sensor and a demodulation device for identifying a method for modulating the luminance of the backlight are written in the backlight. Since it needs to be added to the pen, the system configuration becomes large and complicated. Furthermore, in this system, information is transmitted using the brightness of the backlight that does not change in the spatial direction but only in the time direction, so the amount of information that can be acquired by the light pen per unit time is extremely small, and display is possible. Only very simple information such as an identification code of the device can be transmitted.

  On the other hand, in the system proposed in Non-Patent Document 1, since information is transmitted using a two-dimensional code, even complex information can be transmitted. However, in this system, a synchronization unit is required to connect to each of the projector that projects the two-dimensional code and the camera that images the projected two-dimensional code, and synchronizes these operations. The configuration of will be large. Furthermore, in this system, since the camera other than the camera connected to the synchronization unit cannot generate image data that can identify the two-dimensional code, the versatility of the system is extremely low.

  In contrast to these systems, in the systems proposed in Patent Document 2 and Non-Patent Document 2, a portable terminal having a rolling shutter image sensor is displayed in a “cell” in a display image. It is possible to acquire dimension information. That is, in this system, it is possible to acquire two-dimensional information in any portable terminal.

  However, in the systems proposed in Patent Document 2 and Non-Patent Document 2, since the mobile terminal identifies the color in the image data obtained by imaging and acquires information, the display device becomes a “cell”. There is a limitation that it is impossible to display a display image including characters and drawings of colors to be displayed (blue, yellow, white, gray). Furthermore, in this system, since the “cell” is recognized as a region painted with a single color (white or gray) in human vision, the “cell” is placed in the region where characters and drawings in the display image are displayed. Since it cannot be provided, there is a restriction that “cells” must be provided in a very narrow area such as a frame area where characters and drawings are not displayed.

  As described above, the systems proposed in Patent Document 2 and Non-Patent Document 2 impose various restrictions such as restrictions on the contents of the display image and restrictions on the position where the “cell” is provided in the display image. , It becomes a problem.

  Therefore, the present invention provides an information acquisition device and information that can acquire information embedded in a display image with a simple configuration without limiting the contents of the display image and the position where the information in the display image is embedded. The purpose is to provide a transmission system.

In order to achieve the above object, the present invention sequentially generates captured image data by sequentially capturing a display image in which a pattern in which a display method is sequentially switched is embedded in an original image in each imaging period of a series of imaging frames. An imaging unit; and an image processing unit that generates output pattern data indicating the pattern based on a difference between different frames of the series of imaging frames of the captured image data sequentially generated by the imaging unit, and The display image is obtained by changing the original pixel at the embedding position in the original image to form the pattern. When the display method of the pattern is sequentially switched, in human vision, the pattern and the original image are displayed. wherein in the embedding position in the image are set so that the original pixels are indistinguishable, the imaging unit, the display image corresponding to one of said pattern When a deviation occurs between a display timing that is a display period to be displayed and an imaging timing that is the imaging period in which the imaging unit sequentially captures the display image, within one imaging period in which the deviation occurs In addition, it is configured to sequentially capture two display images in which two consecutive patterns having different display methods are embedded separately at a time ratio corresponding to the shift, and the imaging in the series of imaging frames information acquisition timing, with respect to the display timing appearing at a predetermined period, so that the time ratio corresponding to the deviation of one of the imaging period varies with respect to the time direction, characterized in that it is set Providing equipment.

  According to this information acquisition device, output pattern data is generated based on the difference between different frames of captured image data, so even with a simple configuration that does not require synchronization of the display operation and the imaging operation, The information embedded in the display image can be acquired without limiting the contents and the position where the information in the display image is embedded.

  Furthermore, according to this information acquisition apparatus, since the display timing and the imaging timing shift fluctuate with respect to the time direction, it is possible to avoid the generation of only the captured image data in which the pattern is completely canceled, and to identify the pattern. It is possible to generate output pattern data that is possible.

Furthermore, Oite the information acquisition device having the above characteristics, the imaging unit, the said display image the pattern is embedded to the method displayed in a predetermined pattern display switching rate to the original image is sequentially switched, predetermined imaging frame rate In order to generate the captured image data sequentially by sequentially capturing images, the deviation between the display timing and the imaging timing varies periodically with respect to the time direction, and a beat occurs. It is preferable that the pattern display switching rate and the imaging frame rate are set.

  According to this information acquisition apparatus, it is possible to vary the display timing and the imaging timing with respect to the time direction by a simple method of setting the pattern display switching rate and the imaging frame rate to a size that causes a beat. Therefore, it is possible to simplify the operation control of the imaging unit.

  Furthermore, in the information acquisition apparatus having the above characteristics, the image processing unit is obtained from a predetermined number of the captured image data that is equal to or more than the number of repetitive unit frames that is a number obtained by dividing the captured frame rate by the beat frequency. It is preferable to generate the output pattern data based on the difference.

  According to this information acquisition apparatus, differences between frames are calculated using all types of captured image data having different display timings and imaging timings. Therefore, it is possible to increase the possibility that the calculated difference between the frames of the captured image data includes a case where the absolute value of the difference in the position where the pattern is embedded is sufficiently larger than 0. Become.

  Furthermore, in the information acquisition apparatus having the above characteristics, it is preferable that the image processing unit generates the output pattern data based on the difference obtained from all combinations of the predetermined number of the captured image data.

  According to this information acquisition device, there is a possibility that the difference between calculated frames of captured image data includes a case where the absolute value of the difference between positions where the pattern is embedded is sufficiently larger than 0. It becomes possible to make it higher.

  Furthermore, in the information acquisition device having the above characteristics, it is preferable that the pattern display switching rate and the imaging frame rate are set so that the number of repeating unit frames is an odd number.

  According to this information acquisition apparatus, the mixture ratio of the display images in the captured image data generated by displaying a plurality of display images during the exposure period is uniform over the entire captured image data generated sequentially. Thus, it is possible to avoid that the absolute value of the difference in the position where the pattern is embedded becomes small as a whole.

  Furthermore, in the information acquisition device having the above characteristics, it is preferable that the pattern display switching rate and the imaging frame rate are set so that the imaging frame rate is smaller than the pattern display switching rate.

  According to this information acquisition device, even when a display image is displayed using a three-plate projector or when the imaging unit includes a rolling shutter image sensor, highly reliable output pattern data is generated. It becomes possible.

  Furthermore, in the information acquisition device having the above characteristics, the pattern display switching rate and the imaging frame rate are set such that when one value is fixed, the other has the maximum beat frequency. It is preferable.

  According to this information acquisition device, the number of captured image data used for calculating the difference can be minimized, and the influence of fluctuations in the captured image data can be reduced. Reliability can be improved.

  Furthermore, in the information acquisition apparatus having the above characteristics, when the display method of the pattern is two types, the pattern display switching rate and the imaging frame rate are 3/4 times the imaging frame rate of the pattern display switching rate. It is preferable that it is set to be.

  According to this information acquisition apparatus, the beat frequency is maximum and the number of repeating unit frames can be made odd.

  Furthermore, in the information acquisition device having the above characteristics, it is preferable that the length of the exposure period of the imaging unit is set to be equal to the pattern switching cycle that is the reciprocal of the pattern display switching rate.

  According to this information acquisition device, it is possible to reduce the possibility of generating captured image data in which display images are excessively mixed and the pattern is significantly canceled, and various types of display devices and images are used. The sensor can be used.

  Furthermore, in the information acquisition apparatus having the above characteristics, it is preferable that the pattern in the display image is obtained by changing the chromaticity of the original pixel at the embedding position in the original image.

  According to this information acquisition device, it is possible to capture a pattern by capturing an image of a display device in which a pattern that is difficult to recognize with human vision is embedded.

  In addition, an information transmission system of the present invention includes the above-described information acquisition device and an information providing device that displays the display image, and the information providing device inputs input to original image data indicating the original image. A pattern embedding unit that sequentially generates display image data indicating the display image by changing a pixel value of the original pixel at the embedding position of the pattern indicated by pattern data, and the pattern embedding unit generates A display unit that sequentially displays the display images indicated by the display image data.

  According to the information acquisition device and the information transmission system having the above characteristics, even in a simple configuration that does not require synchronization of the display operation and the imaging operation, the contents of the display image and the position where the pattern in the display image is embedded are limited. It is possible to acquire information embedded in the display image. Furthermore, it is possible to avoid generating only captured image data in which the pattern is completely canceled, and it is possible to generate output pattern data in which the pattern can be identified.

The block diagram shown about the structural example of the information transmission system which concerns on embodiment of this invention. The figure shown about the specific operation example of the pattern embedding part and display part in an information provision apparatus. The figure shown about the outline of the production | generation method of the output pattern data based on the difference between the frames of captured image data. The figure shown about the outline of the production | generation method of the output pattern data based on the difference between the frames of captured image data. The graph which shows the relationship between an imaging frame rate and the beat frequency. The timing diagram in case an imaging frame rate is 1 / ²ⁿ times the pattern display switching rate. The timing diagram in case an imaging frame rate is 1 / ²ⁿ times the pattern display switching rate. The timing diagram in case an imaging frame rate is 3/4 times the pattern display switching rate. The figure shown about the specific example of the production | generation method of the output pattern data by an image process part. The figure shown about the specific example of the production | generation method of the output pattern data by an image process part. The timing diagram when an imaging frame rate is 2/3 times the pattern display switching rate. FIG. 6 is a timing chart when there are three types of pattern display methods. The timing diagram shown about the operation example of the information transmission system which concerns on another embodiment of this invention.

<< Information transmission system >>
An information transmission system according to an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a block diagram showing a configuration example of an information transmission system according to an embodiment of the present invention.

  As shown in FIG. 1, the information transmission system 1 according to the embodiment of the present invention includes an information providing device 10 and an information acquisition device 20. The information providing apparatus 10 generates and displays a display image in which a pattern that is information to be transmitted to the information acquisition apparatus 20 is embedded. The information acquisition device 20 acquires a pattern embedded in the display image by capturing a display image displayed by the information providing device 10.

  A typical example of the information providing apparatus 10 is a stationary computer system having a medium or large display device such as a display or a projector and a screen, a digital signage, or the like. A typical example of the information acquisition device 20 is a wearable terminal such as a smartphone having a moving image capturing camera, a mobile terminal such as a tablet terminal, a notebook computer, or a head-mounted display represented by smart glasses.

<Information provision device>
The information providing apparatus 10 includes a pattern embedding unit 11 and a display unit 12. For example, the pattern embedding unit 11 includes an arithmetic device such as a CPU (Central Processing Unit) and a frame memory that temporarily holds data. For example, the display unit 12 includes a display device such as a liquid crystal monitor or a projector.

  The pattern embedding unit 11 generates display image data indicating the display image by changing the pixel value of the original pixel at the pattern embedding position indicated by the input pattern data with respect to the original image data indicating the original image. . The display unit 12 sequentially displays the display images indicated by the display image data generated by the pattern embedding unit 11. Hereinafter, pixels in the original image and the original image data are referred to as “original pixels”, and pixels in the display image and the display image data are referred to as “display pixels”.

  Here, specific operation examples of the pattern embedding unit 11 and the display unit 12 in the information providing apparatus 10 will be described with reference to the drawings. FIG. 2 is a diagram illustrating a specific operation example of the pattern embedding unit and the display unit in the information providing apparatus. Note that the original image shown in FIG. 2 is a still image, and the original image is a still image in the following description.

  In the operation example shown in FIG. 2, the pattern embedding unit 11 changes the pixel value of the original pixel at the embedding position P in the original image data alternately by applying two different types of variation methods. Display image data indicating a display image in which different patterns are alternately embedded is sequentially generated. Hereinafter, the display image data generated by applying one pixel value variation method to the original image data and the display image indicated by the display image data are both referred to as “A”, and the original image data On the other hand, the display image data generated by applying the other pixel value variation method and the display image indicated by the display image data are both referred to as “B”.

  In the operation example shown in FIG. 2, the display unit 12 alternately switches the display images “A” and “B” indicated by the display image data sequentially generated by the pattern embedding unit 11 at a predetermined display frame rate. To display. The display frame rate is a frame rate (for example, 60 fps) at which the display unit 12 draws a display image.

  As shown in FIG. 2, the display frame rate and the rate at which the display of the display images “A” and “B” are switched (hereinafter referred to as “pattern display switching rate”) generally match, but may not match. obtain. For example, when the display unit 12 draws a display image “A”, a display image “A”, a display image “B”, and a display image “B” at a display frame rate of 60 fps, the pattern display switching rate is 30 fps, It does not match the display frame rate (60 fps). Further, for example, the display unit 12 has a display frame rate of 120 fps, a display image “A”, a dark image (a black image inserted to suppress an afterimage on a liquid crystal monitor or the like), a display image “B”, a dark image, In the case of drawing, each of the display images “A” and “B” is displayed for 1/120 seconds, but the pattern display switching rate is 60 fps, which does not match the display frame rate (120 fps). However, in the following, for simplification of description, a case where the display frame rate and the pattern display switching rate are equal as illustrated in FIG. 2 will be described.

  In the operation example illustrated in FIG. 2, the display unit 12 alternately displays the display images “A” and “B”. As described above, the display unit 12 is embedded in the display images “A” and “B”. Each pattern is required to be unrecognizable (or very difficult to recognize) by human vision. Therefore, when the display images “A” and “B” are alternately switched at the pattern display switching rate, the pattern embedding unit 11 causes the pattern and the embedding position P in the original image at the embedding position P due to the afterimage effect. A method of changing the pixel value of the original pixel of the original image data (that is, a pattern display method) that cannot be distinguished from the original pixel is selected.

Specifically, the pattern embedding unit 11 averages the display pixels (that is, pattern pixels) at the embedding position P in the display images “A” and “B” in human vision. A variation method is selected so as to be the same as the original pixel at the embedded position P. For example, when the pixel value of the original pixel at the embedding position P in the original image data is C _O , the pattern embedding unit 11 determines the pixel value C _A of the display pixel at the embedding position P of the display image data “A”. _A variation method is selected such that the pixel value C _B of the display pixel at the embedding position P of the display image data “B” is (C _A + C _B ) / 2 = C _O. In this case, the pixel value changing method is an arithmetic expression for changing the pixel value C _O to the pixel value C _A , and the other is changing the pixel value C _O to the pixel value C _B. This is an arithmetic expression for

  There are an infinite number of such pixel value variation methods, but it is preferable to select a variation method that makes it difficult for human vision to recognize the pattern. In particular, the pattern embedding unit 11 uses the property that human beings are sensitive to changes in luminance but insensitive to changes in chromaticity, so that the pattern embedding unit 11 performs luminance values of original pixels at the embedding position P of original image data. It is preferable to select a changing method that changes only the chromaticity value without changing the value because it is possible to embed a pattern that is not easily recognized by human vision in the display images “A” and “B”.

In selecting such a variation method in which only the chromaticity value is changed without changing the luminance value of the original pixel at the embedding position P of the original image data, if necessary, the original pixel of the original image data is changed. The pixel value is converted into a pixel value in the L ^* a ^* b ^* color space. Note that the pixel value in the L ^* a ^* b ^* color space means that the pixel value is a value of L ^* representing luminance (a value that can take 0 or more, and the larger the pixel becomes, the brighter the pixel), The value of a ^* that represents chromaticity (a value that can be positive or negative, the redness of the pixel increases as the value increases positively, and the greenness of the pixel increases as the value increases negatively), and b ^* that indicates chromaticity in the yellow-blue direction (A value that can be positive or negative, the pixel value increases as the value increases positively, and the pixel value increases as the value increases negatively).

For example, when converting a pixel value in the RGB color space into a pixel value in the L ^* a ^* b ^* color space, first, the pixel value in the RGB color space is converted to a pixel value in the XYZ color space as shown in the following equation (1). Convert to In addition, each value of the matrix M in the following formula (1) is determined according to the selection of the white reference light, and each value of the matrix M illustrated in the following formula (1) is D65 as the white reference light. If selected.

Then, the pixel value in the XYZ color space obtained by the above equation (1) is converted into a pixel value in the L ^* a ^* b ^* color space by the following equation (2). The function f (t), constants X _n , Y _n and Z _n in the following formula (2) are as shown in the following formula (3). The values of constants X _n , Y _n and Z _{n in} the following formula (3) are those when D65 is selected as the white reference light.

In this case, the pattern embedding unit 11 uses at least the pixel values of the original pixel at the embedding position P in the original image data to calculate the pixels in the L ^* a ^* b ^* color space by the calculations of the above formulas (1) to (3). Display image data “A” and “B” are generated by converting the values into values and changing the pixel values in the a ^* b ^* plane. Then, the pattern embedding unit 11 performs inverse conversion of the pixel values in the L ^* a ^* b ^* color space to the pixel values in the RGB color space by performing the inverse operation to the above formulas (1) to (3). Then, display image data “A” and “B” composed of pixel values in the RGB color space are generated.

  As the pattern display switching rate decreases, the afterimage effect is weakened and the pattern is more easily recognized by human vision. Therefore, it is preferable that the pattern embedding unit 11 is configured to select a variation method in which the variation amount of the pixel value is smaller as the pattern display switching rate is smaller. Further, it is preferable that a pixel value variation method that is difficult to be recognized by human vision is examined in advance by experiments or the like, and the pattern embedding unit 11 selects the variation method.

  The pattern embedding unit 11 may select a variation method according to the pixel value of the original pixel at the embedding position P in the original image data. For example, the pattern embedding unit 11 may prevent the pixel value of the display pixel at the embedding position P in the display image data (that is, the pixel value after change) from exceeding the upper limit value or the lower limit value. The variation amount and the variation direction may be determined according to the pixel value of the original pixel at the embedding position P (that is, the pixel value before the variation).

<Information acquisition device>
The information acquisition device 20 includes an imaging unit 21 and an image processing unit 22. For example, the imaging unit 21 includes an image sensor such as a CCD (Charge Coupled Device) image sensor or a CMOS image sensor. Further, for example, the image processing unit 22 includes an arithmetic device such as a CPU and a frame memory that temporarily holds data. Note that the image sensor included in the imaging unit 21 may be a rolling shutter system or a global shutter system (a system in which the exposure timings of all imaging pixel circuits constituting the image sensor are the same).

  The imaging unit 21 sequentially generates captured image data including data indicating the display image by sequentially imaging the display unit 12 that displays the display image at a predetermined imaging frame rate. Further, the image processing unit 22 generates output pattern data indicating a pattern embedded in the display image based on a difference between different frames of the captured image data sequentially generated by the imaging unit 21.

  Here, an outline of a method for generating output pattern data based on a difference between frames of captured image data by the image processing unit 22 and problems in the generation method will be described with reference to the drawings. 3 and 4 are outlines of a method for generating output pattern data based on differences between captured image data frames. 3 and 4 show a case where the pattern display switching rate and the imaging frame rate match. However, in FIGS. 3 and 4, the state of deviation between the display timing of the display images “A” and “B” by the display unit 12 and the imaging timing of the imaging unit 21 is different.

  By the way, actual captured image data may include a subject other than the display image. However, for simplification of description below, it is assumed that the captured image data does not include a subject other than the display image as in the captured image data illustrated in FIGS. 3 and 4. In addition, the actual captured image data may have different imaging areas depending on the shake of the user who holds the information acquisition device 20 including a mobile terminal or the movement of the user. However, the difference between the imaging regions is extremely small and can be ignored between the captured image data whose imaging timings are close in time. Therefore, it is assumed that the captured image data described below does not have a difference in the imaging region, similarly to the captured image data illustrated in FIGS. 3 and 4.

  In FIG. 3, the display timing of the display images “A” and “B” by the display unit 12 and the imaging timing of the imaging unit 21 are the same. In this case, the imaging unit 21 performs imaging in an exposure period (a period in which the imaging timing is “A”) that coincides with a period in which the display unit 12 displays the display image “A” (a period in which the display timing is “A”). Thus, captured image data indicating the display image “A” (hereinafter, referred to as captured image data “A”) is generated. Similarly, the imaging unit 21 performs imaging in an exposure period (period in which the imaging timing is “B”) that coincides with a period in which the display unit 12 displays the display image “B” (period in which the display timing is “B”). Thus, captured image data indicating the display image “B” (hereinafter referred to as captured image data “B”) is generated.

  In FIG. 3, each of the captured image data “A” and the captured image data “B” includes patterns displayed by different display methods. That is, the captured image data “A” and the captured image data “B” are data that differ only in the pixel value of the pattern. Therefore, for these captured image data “A” and “B”, when the difference in pixel value is calculated for each corresponding position, the absolute value of the difference is greater than 0 because the pixel value is different at the position where the pattern is embedded. The difference becomes approximately 0 at other positions. Accordingly, when the display timing and the imaging timing are shifted from each other as shown in FIG. 3, two-dimensional output pattern data that can identify the pattern is generated.

  On the other hand, in FIG. 4, the display timing of the display images “A” and “B” by the display unit 12 and the imaging timing of the imaging unit 21 are shifted by a half cycle. In this case, the imaging unit 21 has an exposure period in which the display unit 12 displays the display image “A” in the first half and a period in which the display unit 12 displays the display image “B” in the second half (the imaging timing is “A_B The captured image data (hereinafter referred to as captured image data “A_B”) indicating the display image obtained by mixing (ie, averaging) the display images “A” and “B” by half is generated. Similarly, the imaging unit 21 has an exposure period in which the display unit 12 displays the display image “B” in the first half and a period in which the display unit 12 displays the display image “A” in the second half (the imaging timing is “B_A”). The captured image data (hereinafter, referred to as captured image data “B_A”) indicating the display image obtained by mixing (ie, averaging) the display images “B” and “A” by half is generated.

  As described above, the patterns included in the display images “A” and “B” are set so as to be the same as the original pixel at the embedding position P of the original image when averaged on human vision. Yes. Therefore, the captured image data “A_B” and “B_A” in FIG. 4 both indicate the same image as the original image, and are captured image data in which the pattern is completely canceled. Therefore, when the difference between pixel values is calculated for each corresponding position for the captured image data “A_B” and “B_A”, the difference between all positions is approximately regardless of whether the pattern is embedded or not. 0. Therefore, when the display timing and the imaging timing are shifted as shown in FIG. 4, output pattern data that cannot identify the pattern is generated.

  3 and 4, since the pattern display switching rate and the imaging frame rate are the same, the display timing and the imaging timing are uniform, and this deviation is maintained no matter how much time elapses. The Therefore, in the case of FIG. 3, output pattern data whose pattern can be identified is continuously generated, but in the case of FIG. 4, output pattern data that cannot identify the pattern is continuously generated.

  As one method for avoiding the situation shown in FIG. 4, the operations of the display unit 12 and the imaging unit 21 are performed so that the display timing and the imaging timing are always shifted as shown in FIG. Can be considered to be synchronized. However, as described above, when the configuration in which the operations of the display unit 12 and the imaging unit 21 are synchronized is adopted, the configuration of the information transmission system 1 is increased in scale, and the patterns embedded in the plurality of information providing apparatuses 10 are set to one. Since another problem such as inability to identify with the information acquisition device 20 occurs, it is preferable that the operations of the display unit 12 and the imaging unit 21 are asynchronous.

  Therefore, in the information transmission system 1 according to the embodiment of the present invention, the difference between the display timing and the imaging timing varies with respect to the time direction by intentionally providing a predetermined difference between the pattern display switching rate and the imaging frame rate. (Details will be described later).

  Accordingly, it is possible to avoid the situation in which output pattern data that cannot identify a pattern as shown in FIG. 4 is continuously generated, and to generate output pattern data that can at least identify a pattern. Therefore, even if the operations of the display unit 12 and the imaging unit 21 are asynchronous, output pattern data that can identify the pattern can be generated.

  In addition, if a difference is provided between the pattern display switching rate and the imaging frame rate so that the display timing and the imaging timing are shifted with respect to the time direction, the display timing and the imaging timing are periodically shifted. fluctuate. This periodic fluctuation is “beat”, and details will be described later. In the information transmission system 1 according to the embodiment of the present invention, a pattern display switching rate and an imaging frame rate at which this “beat” occurs are selected. To do.

[Specific examples of pattern display switching rate and imaging frame rate]
Specific examples of a pattern display switching rate and an imaging frame rate for obtaining output pattern data that can identify a pattern will be described with reference to the drawings. FIG. 5 is a graph showing the relationship between the imaging frame rate and the beat frequency. FIG. 5 is a graph in the case where the pattern display switching rate is 60 fps and the display unit 12 alternately switches between two types of display images “A” and “B”. In the following description, it is assumed that the exposure period of the imaging unit 21 is equal to the pattern switching period that is the reciprocal of the pattern display switching rate.

As shown in FIG. 5, when the display unit 12 alternately switches two kinds of display image "A" and "B", the (n when the imaging frame rate S _R is 60/2 ⁿ 0 or an integer , And so on), the beat frequency Q is 0 times / s. When this relationship is generalized with the pattern display switching rate as D _R and the pattern display method as k types (k is an integer of 2 or more, the same applies hereinafter), “the imaging frame rate S _R is D _R / k ^{When n} , the beat frequency Q becomes 0 times / s ”.

Further, as shown in FIG. 5, the beat frequency Q, when the imaging frame rate and _{S R,} the smaller the value of S R -60/2 ^{n + 1} and 60/2 ⁿ -S _R (provided that 60/2 ^{n + 1} ≦ S _R ≦ 60/2 ⁿ ). Incidentally, this relationship, the pattern display switching rates _{D R,} Generalizing type of display method of pattern as k type, "beat frequency Q ^{_{is, S R -D R / k n}} + 1 and _D R / ^{k n} - smaller values of S _R (provided ^{_{that, D R / k n + 1}} ≦ S R ≦ D R / k n) becomes "and may be expressed.

Here, a case where the pattern display switching rate and the imaging frame rate are set to a size that does not cause a beat will be described with reference to the drawings. 6 and 7 are timing diagrams when the imaging frame rate is ½ ⁿ times the pattern display switching rate. 6 and 7 are diagrams exemplifying cases where the pattern display switching rate is 60 fps and the imaging frame rates are 60 fps, 30 fps, 15 fps, and 7.5 fps, respectively. Further, the case where the imaging frame rate shown in FIGS. 6 and 7 is 60 fps is a case where the pattern display switching rate and the imaging frame rate match as shown in FIGS. 3 and 4.

  As shown in FIGS. 6 and 7, when the beat frequency is 0 times / s, the display timing and imaging timing shift is uniform as in FIGS. 3 and 4. Specifically, in FIG. 6, as in FIG. 3, the display timing and the imaging timing always coincide. On the other hand, in FIG. 7, as in FIG. 4, the display timing and the imaging timing are always shifted by a half cycle.

  As described above, when the beat frequency is 0 times / s, as shown in FIGS. 4 and 7, there may occur a situation in which output pattern data that cannot identify the pattern is continuously generated.

  By the way, as shown in FIGS. 6 and 7, when the pattern display switching rate and the imaging frame rate are large enough not to cause a beat, the display can be performed even if the exposure period of the imaging unit 21 is lengthened or shortened. The difference between the timing and the imaging timing cannot be changed with respect to the time direction. Therefore, in order to change the shift between the display timing and the imaging timing with respect to the time direction, a method for selecting the pattern display switching rate and the imaging frame rate is important.

  Next, a case where the pattern display switching rate and the imaging frame rate are set to a size that causes a beat will be described with reference to the drawings. FIG. 8 is a diagram illustrating captured image data obtained when the imaging frame rate is 3/4 times the pattern display switching rate. FIG. 8 is a diagram illustrating a case where the pattern display switching rate is 60 fps, the imaging frame rate is 45 fps, and the beat frequency is 15 times / s. In FIG. 8, three examples in which the display timing and the imaging timing are different from each other are shown side by side.

  As shown in FIG. 8, when the pattern display switching rate and the imaging frame rate are set to be large, the shift in display timing and imaging timing varies with respect to the time direction, which is not uniform. Specifically, as shown in FIG. 8, when the beat frequency is 15 times / s, the display timing and imaging timing shift fluctuate within a 1/15 second period that is the reciprocal of the beat frequency.

  For example, as shown in Example 1 of FIG. 8, at the imaging timing “F11”, the display image “A” is displayed over the entire exposure period. At the next imaging timing “F12” after the imaging timing “F11”, the display image “B” is displayed about 70% before the exposure period, and the display image “A” is displayed about 30% after the exposure period. At the next imaging timing “F13” after the imaging timing “F12”, the display image “A” is displayed about 30% before the exposure period, and the display image “B” is displayed about 70% after the exposure period. In Example 2 and Example 3 in FIG. 8, as in Example 1, the display timing and imaging timing shift fluctuates within a period of 1/15 seconds (however, the shift state is different from Example 1). In the following description, the captured image data obtained by imaging at the imaging timings “F11” to “F13”, “F21” to “F23”, and “F31” to “F33” are also “F11” to “F13”. ”,“ F21 ”to“ F23 ”, and“ F31 ”to“ F33 ”.

As shown in Example 1 of FIG. 8, at the next imaging timing after the imaging timing “F13”, the display image “A” is displayed over the entire exposure period, and the display returns to the imaging timing “F11”. Yes. That is, in Example 1 of FIG. 8, the imaging timings “F11” to “F13” for three frames are repeated at the beat frequency of 15 times / s. In Example 2 and Example 3 in FIG. 8, as in Example 1, the imaging timings “F21” to “F23” and “F31” to “F33” for three frames are 15 times / s, which is the beat frequency. Will be repeated. Further, when this relationship is generalized with the imaging frame rate as S _R and the beat frequency as Q, it can be expressed as “the imaging timing for S _R / Q frames is repeated at the beat frequency Q”.

  As described above, when the pattern display switching rate and the imaging frame rate are large enough to generate a beat, it is possible to vary the shift between the display timing and the imaging timing with respect to the time direction. Therefore, it is possible to avoid continuously generating only captured image data in which the pattern is completely canceled, such as the captured image data “A_B” and “B_A” illustrated in FIGS. 4 and 7. Therefore, the image processing unit 22 can generate output pattern data that can identify the pattern based on the difference between frames of the captured image data whose pattern is not completely canceled.

  In FIG. 8, regarding the shift between the display timing and the imaging timing, the imaging timings “F11” to “F13” in Example 1, the imaging timings “F21” to “F23” in Example 2, and the imaging timings “F31” in Example 3 to Although each of “F33” is illustrated as a repeating unit, this is merely for the convenience of explaining the periodicity of the shift of the display timing and the imaging timing. As a matter of course, in the example shown in FIG. 8, if it is an imaging timing for three consecutive frames, it becomes a repeating unit of deviation no matter how it is selected. Specifically, for example, the imaging timings “F12”, “F13”, and “F11” in Example 1 of FIG. 8 are also a unit for repeating the deviation, and the imaging timings “F13”, “F11”, and “F12” are also the repetition of the deviation. Unit.

[Specific example of generation method of output pattern data]
Next, a specific example of a method for generating output pattern data by the image processing unit 22 using captured image data obtained in the case illustrated in FIG. 8 (pattern display switching rate: 60 fps, imaging frame rate: 45 fps) will be described. Will be described with reference to FIG. 9 and 10 are diagrams illustrating a specific example of a method for generating output pattern data by the image processing unit. The method for generating output pattern data illustrated in FIGS. 9 and 10 uses captured image data obtained by capturing a display image as a pattern by changing the chromaticity of the original pixel at the embedding position in the original image. Thus, output pattern data is generated.

In the method for generating output pattern data shown in FIGS. 9 and 10, since the number of frames (S _R / Q: hereinafter referred to as the number of repeating unit frames) that is a repetition unit of the display timing and the imaging timing is 3, it is continuous. The captured image data F1 to F3 for three frames are used. Specifically, for example, in the case of Example 1 in FIG. 8, the captured image data “F11”, “F12”, and “F13” may be used as the captured image data F1 to F3, or the captured image data “F12” and “F13” may be used. F13 "and" F11 "may be used, and captured image data" F13 "," F11 ", and" F12 "may be used.

In the output pattern data generation method of this example, the image processing unit 22 uses the a ^* and b ^* values representing chromaticity among the pixel values in the L ^* a ^* b ^* color space for the captured image data F1 to F3. Performs the operation for. This is because, when the pattern is obtained by changing the chromaticity in the original image, the pattern can be extracted with high accuracy by performing an operation on the values of a ^* and b ^* representing the chromaticity. Because.

First, the image processing unit 22 performs the calculations of the above formulas (1) to (3) as necessary, thereby F1 (a ^* ) which is the value of a ^* for each position of the captured image data F1 to F3. To F3 (a ^* ) and F1 (b ^* ) to F3 (b ^* ), which are values of b ^* for each position of the captured image data F1 to F3, are calculated.

Next, the image processing unit 22 calculates a difference for each position between different frames for the captured image data F1 to F3. Specifically, for each of a ^* and b ^* , the image processing unit 22 performs inter-frame processing for all combinations of the three captured image data F1 to F3 (three kinds of F1 and F2, F2 and F3, and F3 and F1). The difference for each position is calculated. Thus, the image processing unit 22, as the difference ^{a *,} each position of the difference of the imaged image data F1 and ^{F2 | F1 (a *) -F2} (a *) | and, for each position of the captured image data F2 and F3 Difference | F2 (a ^* ) − F3 (a ^* ) | and a difference | F3 (a ^* ) − F1 (a ^* ) | for each position of the captured image data F3 and F1. Similarly, the image processing unit 22, as a difference ^{b *,} each position of the difference of the imaged image data F1 and ^{F2 | F1 (b *) -F2} (b *) | and the position of the captured image data F2 and F3 The difference | F2 (b ^* ) − F3 (b ^* ) | for each position and the difference | F3 (b ^* ) − F1 (b ^* ) | for each position of the captured image data F3 and F1 are calculated.

  As described with reference to FIG. 3 and FIG. 4, the difference in the position where the pattern is not embedded is approximately 0, but the difference in the position where the pattern is embedded is in a state of deviation between the display timing and the imaging timing. Various values can be taken depending on the case. However, the captured image data F1 to F3 includes only captured image data in which the pattern as shown in FIG. 4 is completely canceled because the display timing and the shift in the imaging timing vary with respect to the time direction. There is no (see FIG. 8). For this reason, in the output pattern data generation method of this example, the absolute value of the difference in the position where the pattern is embedded can be made larger than 0 in at least one of the three types of difference calculation results.

As described above, in the method for generating output pattern data of this example, using the captured image data having the number of repetitive unit frames (S _R / Q, 3 in this example) of the display timing and the imaging timing shift, The difference is calculated. That is, the difference between frames is calculated using all types of captured image data with different display timing and imaging timing. Furthermore, in the output pattern data generation method of this example, the difference between frames is calculated for all combinations of the captured image data F1 to F3. Therefore, the possibility that the calculated difference between the frames of the captured image data F1 to F3 includes a case where the absolute value of the difference between the positions where the patterns are embedded is sufficiently larger than 0 is increased. Is possible.

Next, the image processing unit 22 calculates a total value for each position of the calculated difference for each chromaticity value a ^* and b ^* . Thus, the image processing unit 22, the chromaticity value ^{a *,} the sum ^SUM at each position of the difference ^{(a *) = | F1 (} a *) -F2 (a *) | + | F2 (a *) - F3 (a ^* ) | + | F3 (a ^* ) − F1 (a ^* ) | is calculated. Similarly, the image processing unit 22, the chromaticity value ^{b *,} the total value ^SUM (* ^b) at each position of difference ^{= | F1 (b *) -F2} (b *) | + | F2 (b *) - F3 (b ^* ) | + | F3 (b ^* ) − F1 (b ^* ) | is calculated.

  As described above, when the calculation results of the differences between the frames of the captured image data F1 to F3 are summed, even if the absolute value of the difference between the positions where the patterns are embedded is small overall in the three types of differences, It is possible to enlarge the difference between the difference at the position where the pattern is not embedded and the difference at the position where the pattern is embedded, to the extent that can be identified.

Next, the image processing unit 22 calculates an evaluation value POW corresponding to the magnitude of the vector for each position having the total values SUM (a ^* ) and SUM (b ^* ) for each difference position as components. To do. For example, the image processing unit 22 calculates the sum of squares of the total values SUM (a ^* ) and SUM (b ^* ) for each position as the evaluation value POW for each position.

  Then, for example, the image processing unit 22 generates output pattern data by binarizing the evaluation value POW for each position to 1 if the evaluation value POW is equal to or greater than a predetermined threshold, and 0 if it is less than the threshold. . The output pattern data generated in this way is data in which the pattern is represented by 1 and other than the pattern is represented by 0.

  As described above, the information acquisition system 1 according to the embodiment of the present invention generates output pattern data based on differences between different frames of captured image data, and thus synchronizes the operations of the display unit 12 and the imaging unit 21. Even with a simple configuration that is unnecessary, the information embedded in the display image can be acquired without limiting the contents of the display image and the position where the information in the display image is embedded.

  Furthermore, the information acquisition system 1 according to the embodiment of the present invention changes the display timing and the imaging timing with respect to the time direction by changing the pattern display switching rate and the imaging frame rate so as to generate a beat. Only the captured image data in which the pattern is completely canceled can be avoided, and the output pattern data in which the pattern can be identified can be generated.

  Note that when captured image data is sequentially input to the image processing unit 22 and the image processing unit 22 sequentially generates output pattern data, the image processing unit 22 captures captured image data for the three frames most recently input. It may be used to generate output pattern data.

  For example, when the captured image data captured at the imaging timing shown in Example 1 of FIG. 8 is sequentially input to the image processing unit 22, the image processing unit 22 captures the captured image data “F11”, “F12”, “ F13 ”is used to generate output pattern data, and then the output pattern data is used using the captured image data“ F12 ”and“ F13 ”used in the previous calculation and the newly input captured image data“ F11 ”. Data is generated, and then, output pattern data is generated using the captured image data “F13” and “F11” used in the previous calculation and the newly input captured image data “F12”. Good.

  When the output pattern data is generated in this way, it is possible to generate new output pattern data each time new captured image data is input. Therefore, the generation density of the output pattern data is increased, and the output pattern data It is possible to improve the real-time property of generation.

  In addition, as described above, when captured image data is sequentially input to the image processing unit 22 and the image processing unit 22 sequentially generates output pattern data, the image processing unit 22 can detect the interval between successive frames of the captured image data. Only the difference may be calculated.

  For example, in the case where captured image data captured at the imaging timing illustrated in Example 1 of FIG. 8 is sequentially input to the image processing unit 22, the first “F11”, the first “F12”, and the first “ Assume that captured image data is sequentially input in the order of “F13”, second “F11”, second “F12”, and second “F13”. At this time, the image processing unit 22 uses the respective differences of the first “F11” and the first “F12”, the first “F12” and the first “F13”, the first “F13” and the second “F11”. Output pattern data, and then each of the first “F12” and the first “F13”, the first “F13” and the second “F11”, the second “F11” and the second “F12”. The output pattern data is generated using the difference, and then the first “F13” and the second “F11”, the second “F11” and the second “F12”, the second “F12” and the second “F13”. The output pattern data may be generated using each difference.

  In this case, the two differences used in the immediately preceding calculation can be used for the next calculation. Therefore, it is possible to simplify the frame memory included in the image processing unit 22 and reduce the calculation amount of the image processing unit 22.

  However, in this case, the difference is calculated using captured image data for four consecutive frames. For this reason, the influence of fluctuations in the captured image data (for example, fluctuations in the imaging region due to shaking of the user holding the information acquisition device 20 or movement of the user) tends to increase. Therefore, when the difference is calculated using captured image data for four consecutive frames, the output pattern data is compared with the case where the difference is calculated using captured image data for three consecutive frames as described above. The reliability of the can be reduced.

<< Deformation, etc. >>
[1] In FIGS. 8 to 10, the method of generating output pattern data when the imaging frame rate (45 fps) that maximizes the beat frequency is selected when the pattern display switching rate is 60 fps has been described. The pattern display switching rate and the imaging frame rate do not necessarily need to maximize the beat frequency.

  Here, display timing and imaging timing when the combination of the pattern display switching rate and the imaging frame rate is different from those shown in FIGS. 8 to 10 will be described with reference to the drawings. FIG. 11 is a timing chart when the imaging frame rate is 2/3 times the pattern display switching rate. FIG. 11 is a diagram illustrating a case where the pattern display switching rate is 60 fps, the imaging frame rate is 40 fps, and the beat frequency is 10 times / s. In FIG. 11, two examples in which the display timing and the imaging timing are different from each other are shown side by side.

As shown in FIG. 11, when the beat frequency is set to 10 times / s, the display timing and the imaging timing are different from each other within a period of 1/10 seconds that is the reciprocal of the beat frequency. In this case, the number of repetitive unit frames (S _R / Q) of the display timing and imaging timing difference is 4, but the difference between the display timing and imaging timing is the same as the example shown in FIG.

  Therefore, similarly to the case of FIGS. 8 to 10, the output pattern data can be generated also in the case shown in FIG. In this case, as shown in FIGS. 9 and 10, output pattern data may be generated using captured image data for three consecutive frames, or using captured image data for four consecutive frames. Output pattern data may be generated.

  As shown in FIG. 11, when the number of repetitive unit frames of display timing and imaging timing is set to an even number (4 in this example), the symmetry of the shift becomes high. Therefore, as shown in Example 2 of FIG. 11, the mixture ratio of the display images “A” and “B” in the captured image data generated by displaying each of the display images “A” and “B” during the exposure period. However, it may be uniform throughout the captured image data sequentially generated, and the absolute value of the difference between the positions where the patterns are embedded may be reduced as a whole. From the viewpoint of avoiding this, as shown in FIGS. 8 to 10, it is preferable that the number of repetitive unit frames of display timing and imaging timing shift is an odd number (3 in the example of FIG. 8).

Further, when the difference between frames is calculated using captured image data that is equal to or greater than the number of repetitive unit frames (S _R / Q) between the display timing and the imaging timing, the difference is calculated as the beat frequency Q is increased. Therefore, it is possible to reduce captured image data used for this purpose. In particular, by maximizing the beat frequency Q, it is possible to minimize the number of captured image data used for calculating the difference. Then, by reducing the captured image data used to calculate the difference, the influence of fluctuations in the captured image data (for example, fluctuations in the imaging region due to camera shake or movement of the user holding the information acquisition device 20) is reduced. Since it can be reduced, the reliability of the output pattern data can be improved.

[2] In the above-described embodiment, the case where there are mainly two types of pattern display methods (the method of changing the pixel value of the original pixel in the original image data) is illustrated, but there may be three or more types. . Here, the case where there are three types of pattern display methods will be described with reference to the drawings. FIG. 12 is a timing chart when there are three types of pattern display methods. FIG. 12 is a diagram illustrating a case where the pattern display switching rate is 60 fps, the imaging frame rate is 40 fps, and the beat frequency is 20 times / s. In FIG. 12, two examples in which the display timing and the imaging timing are different from each other are shown side by side.

  As shown in FIG. 12, even when there are three (or more) pattern display methods, the display timing and the imaging timing are shifted in the time direction as in the case where there are two pattern display methods. It is possible to vary.

[3] In the above-described embodiment, the case where the original image is a still image is mainly illustrated, but the original image may be a moving image. Even if the original image is a moving image, there is not much difference in several frames that are close in time, and therefore output pattern data can be generated based on the difference between frames of the captured image data as described above.

  However, when the original image is a moving image, the difference between positions where patterns are not embedded tends to increase. Therefore, it becomes difficult to identify the pattern in the output pattern data. Therefore, in order to make it easy to identify the pattern in the output pattern data, the number of captured image data used for generating the output pattern data and the number of combinations of captured image data for calculating the difference between frames may be increased. Alternatively, sequentially generated output pattern data may be accumulated and synthesized (for example, a logical product is obtained).

[4] The pattern display switching rate and the imaging frame due to the influence of the specifications of the display device included in the display unit 12 and the image sensor included in the imaging unit 21 (for example, the number and arrangement of display pixel circuits and imaging pixel circuits, driving method). At least one of the rates may be slightly different from the target magnitude. However, as described above, output pattern data capable of identifying a pattern can be generated as long as the shift between the display timing and the imaging timing can be changed with respect to the time direction. For this reason, the pattern display switching rate and the imaging frame rate do not necessarily have to exactly match the target size.

[5] In the above-described embodiment, the case where the image pickup frame rate is made lower than the pattern display switching frame rate to generate beat is exemplified, but the image pickup frame rate is set higher than the pattern display switching frame rate. It may cause a beat.

  However, in this case, the exposure period of the imaging unit 21 is shorter than the period during which the display unit 12 displays one display image. Therefore, when the display unit 12 includes a three-plate projector (a projector that sequentially projects red, green, and blue images when projecting one display image), or when the imaging unit 21 is a rolling shutter image sensor. In this case, captured image data obtained by capturing a complete display image cannot be obtained, and reliable output pattern data may not be obtained. Therefore, when employing at least these configurations, it is preferable to make the imaging frame rate smaller than the pattern display switching frame rate.

[6] Similar to the above-described embodiment, when the image pickup frame rate is made smaller than the pattern display switching frame rate to generate beat, the exposure period of the image pickup unit 21 may be increased to the limit at which the image pickup timing does not overlap. Is possible.

  However, if the exposure period of the imaging unit 21 is longer than the pattern switching period that is the reciprocal of the pattern display switching rate, a plurality of display images are displayed during the exposure period. Therefore, there is a high possibility that captured image data in which the display image is excessively mixed and the pattern is significantly canceled is generated.

  On the contrary, if the exposure period of the image pickup unit 21 is shorter than the pattern switching period, it becomes difficult to use a three-plate projector or a rolling shutter type image sensor as described in [5].

  Accordingly, when the exposure period of the imaging unit 21 is set to be equal to the pattern switching period that is the reciprocal of the pattern display switching rate, the captured image data in which the display image is excessively mixed and the pattern is significantly canceled is set. Can be reduced, and various types of display devices and image sensors can be used.

[7] When a difference is provided between the pattern display switching rate and the imaging frame rate, the display device constituting the display unit 12 may be controlled to change the pattern display switching rate, or the image sensor constituting the imaging unit 21 May be changed to change the imaging frame rate, or both the pattern display switching rate and the imaging frame rate may be changed. However, it is preferable to adopt a change method that does not obstruct the operations of the information providing apparatus 10 and the information acquisition apparatus 20 as much as possible.

  For example, when the display frame rate (pattern display switching rate) of the display device constituting the display unit 12 is fixed to a predetermined size (for example, 60 fps) in the industry standard, the information acquisition device 20 acquires the pattern. When the operation mode is shifted to the operation mode, the imaging frame rate of the image sensor constituting the imaging unit 21 may be controlled to be a predetermined size (for example, 45 fps).

  In addition, when the imaging frame rate is changed by controlling the image sensor that configures the imaging unit 21, the information acquisition device 20 may dynamically change the imaging frame rate.

  For example, when the information acquisition device 20 shifts to an operation mode for acquiring a pattern, the imaging frame rate is first controlled to be a predetermined size (for example, 45 fps). If it is detected that the pattern cannot be accurately identified (for example, the pattern cannot be identified, the position of the pattern is not determined, etc.), the imaging frame rate is different from the predetermined size (for example, 40 fps). You may control so that it may become.

[8] The embodiment has been described so far in which the display timing and the imaging timing shift are changed with respect to the time direction by appropriately setting the pattern display switching rate and the imaging frame rate. Even in the form, it is possible to vary the shift between the display timing and the imaging timing with respect to the time direction. Here, as an example, a mode in which the shift of the display timing and the imaging timing is changed with respect to the time direction by making the imaging intervals different without equal intervals will be described with reference to the drawings. FIG. 13 is a timing chart showing an operation example of the information transmission system according to another embodiment of the present invention. FIG. 13 is a diagram illustrating a case where the pattern display switching rate is 60 fps and the imaging frame rate is 30 fps (as a whole). In FIG. 13, two examples in which the display timing and the imaging timing are different from each other are shown side by side.

  FIG. 13 shows that the imaging frame rate of the imaging unit 21 as a whole is 30 fps (an imaging frame rate at which no beat occurs if the imaging interval is equal, see FIG. 5). However, in the example shown in FIG. 13, the imaging intervals are not equal, but are different. Specifically, in the example shown in FIG. 13, after two continuous imagings are performed at a short imaging interval, a standby with a longer imaging interval is provided, and these operations are sequentially repeated. The total is 30 fps.

  As described above, the display timing and the imaging timing can be shifted with respect to the time direction by changing the imaging intervals without making them equal. However, as in the embodiments described so far, it is easier to control the operation of the imaging unit 21 by adjusting the relationship between the pattern display switching rate and the imaging frame rate while keeping the imaging interval equal. This is preferable because an existing imaging device can be used.

  The present invention can be applied to an information acquisition device that acquires information embedded in a display image displayed by a display device, and an information transmission system that includes the information acquisition device.

1: communication system 10: the information providing apparatus 11: pattern embedding part 12: display unit 20: Information acquisition apparatus 21: imaging unit 22: image processing unit P: embedding position Q: beat frequency S _R: imaging frame rate D _R: pattern display switching frame rate

Claims (11)

An imaging unit that sequentially generates captured image data by sequentially capturing a display image in which a pattern in which a display method is sequentially switched is embedded in an original image in each imaging period of a series of imaging frames ;
An image processing unit that generates output pattern data indicating the pattern based on a difference between different frames of the series of imaging frames of the captured image data that the imaging unit sequentially generates,
The display image is obtained by changing an original pixel at an embedding position in the original image to form the pattern. When the display method of the pattern is sequentially switched, in human vision, the pattern and the It is set so that it cannot be distinguished from the original pixel at the embedding position in the original image,
The imaging unit has a difference between a display timing that is a display period in which the display image corresponding to one pattern is displayed and an imaging timing that is the imaging period in which the imaging unit sequentially captures the display image. If there are two display images in which two consecutive patterns with different display methods are embedded in the imaging period in which the deviation occurs, at a time ratio corresponding to the deviation. Configured to capture images sequentially,
The imaging timing in the series of imaging frames, with respect to the display timing appearing at a predetermined period, so that the time ratio corresponding to the deviation of one of the imaging period varies with respect to the time direction, is set An information acquisition device characterized in that:
The imaging unit sequentially captures the captured image data by sequentially capturing the display image in which the display method in which a display method is sequentially switched at a predetermined pattern display switching rate is embedded in the original image at a predetermined imaging frame rate. Is to generate
The pattern display switching rate and the imaging frame rate are set so that the deviation between the display timing and the imaging timing periodically fluctuates in the time direction. The information acquisition apparatus according to claim 1 .   The image processing unit obtains the output pattern data based on the difference obtained from a predetermined number of the captured image data that is equal to or greater than the number of repetitive unit frames, which is a number obtained by dividing the imaging frame rate by the beat frequency. The information acquisition apparatus according to claim 2, wherein the information acquisition apparatus generates the information acquisition apparatus.   The information acquisition apparatus according to claim 3, wherein the image processing unit generates the output pattern data based on the differences obtained from all combinations of the predetermined number of the captured image data.   5. The information acquisition apparatus according to claim 3, wherein the pattern display switching rate and the imaging frame rate are set such that the number of repeating unit frames is an odd number.   The said pattern display switching rate and the said imaging frame rate are set so that the said imaging frame rate may become smaller than the said pattern display switching rate, The any one of Claims 2-5 characterized by the above-mentioned. Information acquisition device.   7. The pattern display switching rate and the imaging frame rate are set such that when one value is fixed, the other is set to maximize the beat frequency. Information acquisition device. When the display method of the pattern is two types,
The information acquisition apparatus according to claim 7, wherein the pattern display switching rate and the imaging frame rate are set such that the imaging frame rate is 3/4 times the pattern display switching rate.   The length of the exposure period of the imaging unit is set to be equal to the pattern switching cycle that is the reciprocal of the pattern display switching rate. The information acquisition device described in 1.   The said pattern in the said display image changes chromaticity of the said original pixel in the said embedding position in the said original image, The any one of Claims 1-9 characterized by the above-mentioned. Information acquisition device. The information acquisition device according to any one of claims 1 to 10,
An information providing device for displaying the display image,
The information providing apparatus includes:
A pattern for sequentially generating display image data indicating the display image by changing a pixel value of the original pixel at the embedding position of the pattern indicated by input pattern data with respect to the original image data indicating the original image. An embedding part;
And a display unit that sequentially displays the display images indicated by the display image data generated by the pattern embedding unit.

Images