JP4640483B2 - Imaging apparatus and program - Google Patents

Description

  INDUSTRIAL APPLICABILITY The present invention can be used for various purposes such as landmark display of buildings, advertisement display, amusement facility guidance and congestion status display such as amusement parks, product descriptions at stores, and exhibit descriptions at museums and exhibitions. The present invention relates to an imaging device and a program.

  Conventionally, landmarks such as buildings, advertisements, amusement parks such as amusement parks, crowds, etc., product descriptions at stores, exhibitions at museums and exhibitions, such as paper, banners, billboards, plates, etc. This is done using “visual information” such as letters and figures written on the “information presentation”.

  However, when it is desired to search for a specific information display object in an environment where there are a large number of information display objects, there arises a problem that the desired information display object becomes inconspicuous by surrounding information display objects. For example, if the product A and the product B are displayed adjacent to each other at the store, the product description of the product A may be mistaken for the product description of the product B.

  Therefore, as prior art 1, by utilizing information transmission by optical space transmission, the correspondence between the information presentation target product and the presentation information is clarified so as not to misidentify the information, and further, the information integrity Is also considered (for example, see Patent Document 1).

  Specifically, in this prior art 1, an optical tag device attached near a product arranged on a display shelf is photographed and stored in time series with an electronic still camera, and light existing in the stored image is stored. Recognizing that the information is luminance-modulated from the time-series luminance change of the tag device, extracting information related to the product from the luminance-modulated information, and displaying the extracted information superimposed on the captured image. Is. However, in this prior art 1, for example, when there is a light source (for example, disturbance light such as flicker of a fluorescent lamp) nearby that changes in luminance in the same pattern as the luminance change pattern of the flashing optical tag device, There has been a technical problem that information extraction from the optical tag device is not performed correctly.

  In view of such a problem, as the conventional technique 2, an information receiving apparatus, an information transmission system, and an information transmission method that can always extract information correctly while avoiding adverse effects due to disturbance light are considered (for example, Patent Document 2).

  Specifically, this prior art 2 uses a light receiving unit having an image sensor such as a CCD, and from a light emitting unit (corresponding to the optical tag device in the prior art 1) existing within the photographing field angle of the image sensor. Detects a change in brightness, binarizes the degree of change in brightness as modulation information into a bit pattern sequence, and matches the binarized bit pattern sequence to any of the pre-correlated bit pattern sequences The transmission information from the light emitting unit is reproduced by generating a binary logic signal (1/0) based on the determination result, and the light emission pattern of this light emitting unit By making it unique, the influence of ambient light is reduced.

JP 2001-245253 A (2nd page, 5th page, FIG. 1, FIG. 10) JP 2003-179556 A (Pages 7 to 8, FIGS. 7 to 10)

However, although the prior art 2 can avoid the adverse effects of ambient light, there is a problem that the image processing is wasteful.
That is, in the prior art 2, “partial readout” is performed from an image within the shooting angle of view of the image sensor, and the readout area is shifted every one period, while the above binarization processing or binarization processing is performed. Even if the logical processing of the bit pattern series is executed, the processing remains the same for all the pixels of the image frame output from the image sensor, in other words, within the shooting angle of view of the image sensor. The point light source is detected and captured for the entire area of the image. Therefore, as described below, useless processing cannot be denied.

  Assuming that all pixels (full dots) within the shooting angle of view of the image sensor are 1280 × 960 dots, and that this image sensor is capable of partial readout of 320 × 240 dots, In other words, the above binarization processing or the logical processing of the bit pattern series after binarization processing is performed for each area obtained by dividing a 1280 × 960 dot image by 4 × 4. In this case, since one process is performed on pixels of 320 × 240 dots for one area, if attention is paid to only one process, the processing load is surely small. However, the point light source of the light emitting unit, especially the point light source located in the distance, is considerably small, and in many cases it is enough to fit in one divided area. Processing for other divided areas that are not present is useless.

  An object of the present invention is to provide an imaging apparatus and a program capable of recording a captured image and received luminance modulation information in association with each other as, for example, one image file.

According to the first aspect of the present invention, there is provided an imaging unit and a light source that emits light with a luminance change in a time series included in an angle of view captured by the imaging unit, or an object irradiated with light emitted from the light source. , Continuously receiving the light whose luminance changes in time series, and restoring means for restoring information from the received light; an image captured by the imaging means; and information restored by the restoring means Recording means for recording in association, instruction determination means for determining whether or not re-light reception of the light has been instructed, and determining that re-light reception has been instructed by the instruction determination means, the image captured by the imaging means In a held state, a re-light receiving start unit that starts re-reception of the light, an image captured by the imaging unit, and information restored from the light received by the restoring unit are associated Characterized in that it comprises a recording control means for controlling said recording means so as to store.

According to a second aspect of the present invention, in the first aspect of the present invention, the recording control unit further includes a completion determining unit that determines whether or not light reception corresponding to information restored by the restoring unit is completed. When the completion determining unit determines that the light reception is completed , the image captured by the imaging unit and the information restored by the restoring unit are stored in association with each other.

According to a third aspect of the present invention, in the second aspect of the present invention, when the completion determining unit determines that the light has been received, a notification unit that notifies the fact is further provided.

According to a fourth aspect of the present invention, there is provided an imaging unit and a light source that emits light with a luminance change in a time series included in an angle of view captured by the imaging unit, or an object irradiated with light emitted from the light source. , Continuously receiving the light whose luminance changes in time series, and restoring means for restoring information from the received light; an image captured by the imaging means; and information restored by the restoring means In the image pickup apparatus including the recording means for recording in association, the luminance modulation is repetition of turning on and off the specific wavelength light, and controls the image pickup means to pick up an image at the turn-off timing of the specific wavelength light. further comprising an imaging control means, said recording means, especially that that associates and stores the recovered information by the restoring means and the image captured by the control by the imaging control means To.

The invention according to claim 5 is a computer that the imaging apparatus has, from a light source that emits light that changes in luminance in a time series included in the captured angle of view, or an object that is irradiated with light emitted from the light source, Reconstructing means that continuously receives light that changes in luminance in time series and restores information from the received light, and associates the captured image with information restored by the restoring means in a recording unit Recording means for recording, instruction determination means for determining whether or not re-light reception of the light has been instructed, and when it is determined that re-light reception has been instructed by the instruction determination means, the image captured by the imaging means is retained A re-light receiving start means for starting re-light reception of the light, and an image captured by the imaging means and information restored from the light received by the restoring means in association with each other Wherein the function as a recording control means for controlling the recording means.

The invention according to claim 6 is a computer having an imaging device, from a light source that emits light with a luminance change in a time series included in a captured angle of view, or an object irradiated with light emitted from the light source, Reconstructing means for continuously receiving the light whose luminance changes in time series and restoring information from the received light, and a recording unit associating the captured image with the information restored by the restoring means Further, the luminance modulation is a repetition of turning on and off the specific wavelength light, and functions as an imaging control means for controlling to take an image at the turn-off timing of the specific wavelength light. , in the recording means, to characterized in that to function to store in association with recovered information by the restoring means and the image captured by the control by the imaging control means .

  According to the present invention, a photographed image and received light intensity modulation information can be recorded in association with each other as, for example, one image file, and can be applied to image number-of-prints management and copyright management.

  Embodiments of the present invention will be described below with reference to the drawings. It should be noted that the specific details or examples in the following description and the illustrations of numerical values, character strings, and other symbols are only for reference in order to clarify the idea of the present invention, and the present invention may be used in whole or in part. Obviously, the idea of the invention is not limited. In addition, a well-known technique, a well-known procedure, a well-known architecture, a well-known circuit configuration, and the like (hereinafter, “well-known matter”) are not described in detail, but this is also to simplify the description. Not all or part of the matter is intentionally excluded. Such well-known matters are known to those skilled in the art at the time of filing of the present invention, and are naturally included in the following description.

<First embodiment>
FIG. 1 is a utilization state diagram of the information transmission system in the first embodiment. First, (a) will be described. An advertisement display board 15 is installed on the roof of a building 14 such as a building. As shown in the front view of (b) and the side view of (c), the advertisement display board 15 has an arbitrary explanatory character string (in the figure, “X department store special sale is in progress!”, “Light of this signboard”). The tag information can be used to obtain an electronic coupon that can be used when visiting the store. ”) It has a signboard 16 with a large illustration and a back fixed foot 17 that supports the signboard 16 from the back. It can be confirmed visually.

  An optical tag device 18 is attached to an arbitrary position of the signboard 16 (substantially the central portion in the illustrated example). The optical tag device 18 is a point light source that blinks in two pattern series, and when the optical tag device 18 is only visually observed, it can be seen only as blinking of light. By receiving the flashing light of the optical tag device 18 through an information receiving device (here, a camera-equipped mobile phone) corresponding to the information transmission system in the first embodiment, desired digital information (for example, used in an X department store) Information on electronic coupons that can be played back) and displayed on the information receiving device.

  In the illustrated example, there are a plurality of persons 19 to 21 at a position where the signboard 16 can be seen, and only some of them (for convenience, two persons 19 and 20) are the information in the first embodiment. Since the camera-equipped mobile phone 22 (information receiving device) corresponding to the transmission system is possessed, only these two persons 19 and 20 can obtain an electronic coupon that can be used in the X department store, and the remaining persons Since 21 does not have the camera-equipped mobile phone 22, the electronic coupon cannot be acquired.

  Note that the optical tag device 18 does not necessarily have to be attached to a large object such as the signboard 16, and can of course be attached to a very small object such as a product tag. The optical tag device 18 can be installed, for example, “I want to incentivize users who have come to a (relatively large) area to attract customers to that area” or “I want to see certain advertisements on the spot. “I want to give incentives to users”, “I want to automatically provide more detailed information (or links to it) to the user terminal than information such as signboards,” etc. The person who came to the area of (1) is provided with various information while being related to the physical entity (sign 16 in the illustrated example), or a specific service (electronic coupon in the illustrated example). As long as it is provided, it is not limited to the illustrated example, and any mode may be used.

  Here, an application example in which an optical tag device 18 is arranged on a signboard 16 and an electronic coupon is distributed as an incentive to a user who sees the advertisement of the signboard 16 is taken as an example. In this case, the “coupon” is an electronic image file, and a link destination is separately specified and downloaded from there.

  FIG. 2 is a front view and a rear view of the camera-equipped mobile phone 22 corresponding to the information transmission system in the first embodiment. The camera-equipped mobile phone 22 has a body 23 composed of a foldable lid 231 and a main body 232. The main body 232 includes an antenna 24, and the lid 231 includes a status indicator 25, a speaker 26, and a liquid crystal. A display unit 27 such as a display panel, a camera key 28, an on-hook button 29, an off-hook button 30, a cursor key (decision instruction when the center portion is pressed) 31, a numeric key button group 32, a microphone 33, and the like are provided. A photographing lens 34, a condensing lens 35, and the like are provided on the rear surface of H.231. Note that the photographing lens 34 and the condenser lens 35 are arranged as close as possible to the distance L in order to eliminate parallax.

  FIG. 3 is a sequence diagram showing a relationship between a user operation on the camera-equipped mobile phone 22 and an internal operation of the camera-equipped mobile phone 22. It is assumed that the code pattern of the optical tag device 18 is inputted in advance as desired by the provider. Also, for example, as a campaign announcement in advance, media such as TV commercials or magazines, or in the form of a mail magazine on the camera-equipped mobile phone 22, “Look at the XXX outdoor signage! Various services with optical tags It is preferable that information such as “XXXXXXX” is acquired and widely notified. “X” is an arbitrary character or an arbitrary number or an arbitrary symbol. However, instead of inputting individual access codes for providing information, an access code that anyone can use may be input.

  First, when the user of the camera-equipped mobile phone 22 finds the signboard 16 having the optical tag device 18, the user enters an access code (step S11) and presses the camera key 28 of the camera-equipped mobile phone 22 (step S12). In response to this button operation, the camera-equipped mobile phone 22 switches from the normal telephone use state (standby state or call state) to the optical tag detection and reception state (steps S13 and S14), and the camera takes a picture. The image is displayed on the display unit 27, and a small index (see the aiming frame 40 in FIG. 4A) is superimposed and displayed in the image (step S15).

  FIG. 4A is a diagram illustrating a display example of the display unit 27. In this figure, a mobile phone reception status indicator 36, a current time indicator 37, a remaining battery indicator 38, and the like are displayed at the upper end portion of the display portion 27, and an operation guidance is provided at the lower end portion of the display portion 27. A message 39 is displayed. A through image (moving image of several tens of frames per second) of the camera of the camera-equipped cellular phone 22 is displayed in a wide range other than the upper end and the lower end of the display unit 27. For example, in the illustrated example, FIG. A through image including the sign 16 of (a) is displayed. The star symbol in the through image schematically shows the optical tag device 18 in FIG. 1A, and the blinking light of the optical tag device 18 is information to be acquired. However, it is not possible to know the contents of the information just by looking at the flashing light. By receiving the light using a predetermined terminal (camera mobile phone 22), the contents of the information are reproduced inside the camera mobile phone 22 and displayed on the display unit 27 of the camera mobile phone 22. It is like that.

  The camera-equipped mobile phone 22 in the first embodiment does not process the entire through image displayed on the display unit 27 and reproduce the above information, but is surrounded by a small aiming frame 40 in the through image. Only the specific part is processed in a limited manner to reproduce the above information. This avoids useless processing and improves the responsiveness of information reproduction. That is, unlike the prior art 2, the information of the optical tag device 18 is reproduced by processing only within a specific range in the through image.

  In the example of FIG. 4A, the aiming frame 40 is not on the optical tag device 18 indicated by the star symbol. The white arrow 41 and the broken line frame 42 represent the moving direction of the camera-equipped mobile phone 22 and the camera photographing range after the movement necessary for overlaying the aiming frame 40 on the optical tag device 18 for convenience. Yes.

  FIG. 4B is a diagram illustrating a display example of the display unit 27 when the aiming frame 40 is overlaid on the optical tag device 18. Thus, the user operates the cursor key 31 to superimpose the aiming frame 40 on the optical tag device 18 (step S16), and the camera-equipped mobile phone 22 lights or blinks the status indicator lamp 25, or It detects that the center part of the cursor key 31 was pressed, and starts the information detection process of the optical tag apparatus 18 (step S17).

  FIG. 5A is a diagram illustrating a display example of the display unit 27 when information acquisition from the optical tag device 18 fails. For example, when information cannot be acquired even after a predetermined time has elapsed, or when correct information cannot be acquired due to the influence of ambient light or camera shake. In such a case, as shown in the figure, together with the information acquisition failure message 43, balloon messages 44 and 45 for each cause of failure are generated and displayed on the display unit 27.

  FIG. 5B is a diagram illustrating a display example of the display unit 27 when information acquisition from the optical tag device 18 is successful. For example, the character information acquired from the optical tag device 18 together with the information acquisition success message 46 Is generated and displayed on the display unit 27.

  During this time, the camera-equipped mobile phone 22 displays the detection result (step S18), and determines whether there is an error such as garbled characters in the detection result (step S22). If the detection result is an error, that is, if the answer is No in Step S22, the camera-equipped mobile phone 22 returns to the standby state before performing this process, but if Yes, the process of Step S20 described later is performed. Also, the user confirms the start of the action according to the result in step S18 and executes an execution instruction (step S19). If the information acquisition is successful, for example, execution of a designated command, login to a web service, A command execution result such as information download is displayed (step S20), and the user uses the acquired information (for example, coupon use) (step S21).

  Fig.6 (a) is a figure which shows the example of a display of the display part 27 when a coupon image file is downloaded, and Fig.6 (a) is a figure which shows the image of a coupon. The user uses the coupon by showing the screen shown in FIG.

  FIG. 7 is an internal block diagram of the camera-equipped mobile phone 22 in the first embodiment. In this figure, the camera-equipped mobile phone 22 includes a data communication unit 48 including an antenna 24, an operation button unit 51 (28 to 32) including buttons such as a data buffer 49, a decoding bit buffer 50, a camera key 28, and a speaker 26. And a voice input / output unit 52 including a microphone 33, a display unit 27, a CPU 53, an acceleration sensor 54, a camera shake correction actuator 55, a photo detector unit 56, a photographing lens 34, a photographing image sensor 57 such as a CCD or CMOS, and an image buffer 58. A display buffer 59 and a storage unit 60. In addition to this, a power supply unit such as a battery is provided, but is omitted in order to avoid congestion in the drawing.

  The photo detector unit 56 includes a condenser lens 35, a photo detector 61, an amplification and A / D converter 62, a timing generator 63, a matched filter 64, and a filter buffer 65.

  There is nothing special to note about the configuration of the camera portion including the photographing lens 34 and the photographing image sensor 57. There may be a finder-like function capable of aiming at the optical tag intended by the user or a function for displaying the position of the information light source in the actual image. Therefore, even if the frame rate of the camera portion is, for example, a transmission signal speed of several tens of bits / second or several hundreds of bits / second, there is no problem in aiming confirmation, for example, an image of about 10 frames / second. Acquisition ability is sufficient. It should be noted that information such as the gain control of the camera part may be used for setting the gain control of the photodetector unit 56 without being closed inside the camera part.

  The photodetector unit 56 is a block that digitizes the luminance variation signal of the optical tag and performs the evaluation up to the correlation degree. Details will be described later.

  8 and 9 are conceptual diagrams of camera shake correction in the camera-equipped mobile phone 22 according to the first embodiment. As shown in FIG. 8A, the photodetector 61 is located on the optical axis of the condenser lens 35. In the first embodiment, the position of the photodetector 61 is changed by the actuator 55 provided along with the photodetector 61 so that the photodetector 61 is arranged on the optical axis connecting the light A and the detection point B of the optical tag device 18. Yes.

  For example, in the case where the actuator 55 is not provided, as shown in FIG. 8B, the optical axis shift due to camera shake occurs, and the photodetector 61 is not positioned at the detection point B, but in FIG. As shown, if the photo detector 61 is provided with an actuator 55 so that the position of the photo detector 61 can be changed, the photo detector 61 can be moved in the C direction and the optical axis deviation can be corrected. Therefore, the photodetector 61 can reliably capture the light A from the optical tag device 18 even if some camera shake is detected by the user.

  As shown in FIG. 9, the acceleration sensor 54 detects a camera shake caused by a user operation in two axial directions. The CPU 53 generates a camera shake correction control signal having a magnitude corresponding to the detection signal of the acceleration sensor 54, and drives the actuator 55 with the camera shake correction control signal. As described above, it is possible to correct the camera shake during the aiming of the optical tag by detecting the biaxial acceleration.

  In the first embodiment, for the sake of convenience, the detection field angle of the photo detector 61 is described as a narrow field angle of 2 degrees × 2 degrees, and the correction range is described as 1 degree. It is said that a beginner who is not very familiar with the operation of the camera causes camera shake at an angular velocity of about 5 degrees / second and in a range of about 1 degree. Normally, this level of camera shake correction function is equipped in general digital cameras, binoculars, and the like, and is considered sufficient for aiming at a narrow angle of view (2 degrees × 2 degrees) of the photodetector 61. As will be described later, this camera shake correction may be omitted depending on the sensitivity, the detection angle of view, and the distance and size standard with respect to the optical tag.

  Here, a design example in which the operation of the present invention is specifically realized with respect to the arrangement configuration of the photographing lens 34 (photographing image sensor 57) and the photodetector unit 56 in the camera-equipped mobile phone 22 will be described.

  FIG. 10 is a diagram showing an imaging field angle (wide angle of view of an elevation angle—an depression angle of about 32 degrees) of the photographing lens 34 (imaging image sensor 57) in the camera-equipped cellular phone 22, and FIG. 22 is a diagram showing a detected field angle (a narrow field angle of about 2 degrees in elevation angle−an depression angle) of the photo detector section 56 in FIG. In the first embodiment, it is assumed that a typical use distance is about several m to 100 m, and that the displacement of the angle of view due to the arrangement of the optical system is not affected, and the two optical axes are parallel. However, in consideration of utilization at a short distance, it is desirable to mount the two optical systems as close as possible (see the distance L in FIG. 2B). In the case of mainly using at a short distance, the angles of the optical axes of the two optical systems may be adjusted as appropriate by performing a bias for camera shake correction.

  As shown in FIG. 10, the photographic lens 34 and the like have a normal camera angle of view so that the user can easily aim at the image range captured by the user and is conscious of the operation in the normal camera mode. . Here, specifically, it is assumed that an image of 480 dots × 480 dots is acquired in the range of elevation angle−decline angle 32 degrees × left and right angles 32 degrees. On the other hand, as shown in FIG. 11, it is desirable that the photo detector 61 (the condensing lens 35, etc.) of the photo detector section 56 perform sufficient condensing with a large lens so that a distant signal source can be sufficiently condensed. . In the first embodiment, the angle of view is set to an elevation angle—an depression angle of 2 degrees × a horizontal angle of 2 degrees. This angle of view corresponds to the size of the aiming frame 40.

  In the first embodiment, for the convenience of the user, the aiming frame 40 is displayed at a position slightly above the center at the angle of view of the camera image, and the optical system is adapted to that. For this reason, as shown in FIG. 11, the photodetector 61 is disposed at a position slightly shifted from the center of the optical axis.

  10 and FIG. 11 both have a “matching lens” configuration, but this is, for example, for the convenience of explanation to clearly show that the lens is telephoto, and other configurations of optical systems (for example, , No single lens). In the final implementation, rather, the optical lens requires a complex lens group to ensure image quality, but details thereof are well known for various purposes and will be omitted.

  On the other hand, the optical system of the photodetector 61 does not accurately form an aiming region, and it is only necessary that the corresponding field angle region is focused on the photodetector 61 (with a lens having a necessary brightness). Since the depth of field or the required accuracy can be made low, an extremely simple and low-cost configuration including materials is possible. The focus may be pan focus in consideration of the use distance.

  FIGS. 12A and 12B are internal block configuration diagrams of the optical tag device 18. The optical tag device 18 includes a timing generator 66, a 1 / N frequency divider 67, a transmission data memory 68, a pattern data generation unit 69, a first control / drive unit 70, and a second control / drive unit 71. The first control / drive unit 70 includes two AND gates 72 and 73, two inverter gates 74 and 75, and one OR gate 76. The second control / drive unit 71 includes a front light 77 and a reflection. Type liquid crystal shutter 78.

  FIG. 13A is a diagram showing the front light 77 and the reflective liquid crystal shutter 78. The front light 77 only needs to illuminate the reflective liquid crystal shutter 78. That is, as shown in the figure, it is only necessary to shine from a place that does not interfere with the aiming. The light source of the optical tag device 18 is not limited to this example. For example, as shown in FIG. 13B, the light source 80 that directly emits light and the lowest possible luminance when the ambient light wraps around or is turned off. Although it is good also as a structure which included the food | hood 81, the structure of Fig.13 (a) may be able to be comprised more low-powered by the point which can ensure sufficient brightness | luminance fluctuation | variation outdoors on a clear day. So desirable.

  Next, the luminance, distance, area, light detector sensitivity, dynamic range, S / N, etc. of the light source will be described. A rough example of actual values of the light source and the detection element is as follows. Needless to say, the following approximate example is based on a model that considerably simplifies calculations related to luminous intensity.

  First, in the first embodiment, the optical tag device 18 is configured by a front light 77 and a reflection panel 78 as shown in FIG. 13, and can be used as an environment from the outdoors on a sunny day to the outdoors at night.

  The size of the reflection panel 78 is 1 square meter, and the reflection coefficient when the front light 77 is in a light emitting state is 30%, and the reflection coefficient when the front light 77 is not emitting light is 2%. Further, the front light 77 can uniformly irradiate the entire surface (1 square meter) of the reflective panel 78 with light of 1000 lumens (about one household bulb).

The luminance calculation method is generally obtained by “luminance = reflection coefficient × illuminance / π”.
(A) Brightness is XA (ambient environment: sunny weather outdoors)
(B) Brightness is XB (Ambient environment: under night street lights)
FIG. 14 illustrates a case where the luminance of the optical tag is calculated under these extreme conditions.

FIG. 14 is a graph showing the luminance calculation result of the optical tag.
(A) When the luminance is XA (100,000 lux) in the environmental illuminance (logarithmic X axis) a-1: the luminance of only the reflective panel 78 (point A)
(100,000 × 0.3) / π = 9543cd (per square meter)
a-2: Brightness (point B) when the amount of irradiation of the front light 77 is added
Assume that 1000 lumens of light that the front light 77 irradiates 1 square meter is 1000 lux. In this case, the brightness increase by the front light 77 is 1000 × 0.3 / π = 10 cd (per square meter)
And when added to the luminance of a-1,
9559 cd (per square meter).
a-3: Luminance (point C) of only the reflective panel 78 in a state where the front light 77 is not emitting light
(100,000 × 0.02) / π = 636 cd (per square meter)
From this result, when the luminance is XA (ambient environment: sunny weather outdoors), the ratio of the luminance when the front light 77 emits light and when it does not emit light is 15: 1.

(B) When the luminance is XB (100 lux) in environmental illuminance (logarithmic X axis) b-1: luminance of only the reflective panel 78 (point D)
(100 × 0.3) /π=9.54 cd (per square meter)
b-2: Luminance (point E) when the amount of irradiation of the front light 77 is added
The brightness increase by the front light 77 is 1000 × 0.3 / π = 10 cd (per square meter)
And adding to the luminance of b-1,
19.54 cd (per square meter)
b-3: Luminance (point F) of only the reflective panel 78 in a state where the front light 77 is not emitting light.
(100 × 0.02) /π=0.6 cd (per square meter)
From this result, when the luminance is XB (ambient environment: under night street lights), the ratio of the luminance when the front light 77 emits light and when it does not emit light is 32: 1.

  As described above, in a bright environment such as outdoors in sunny daytime, a sufficient fluctuation range can be ensured by a luminance difference due to ambient light reflection, while in a dark environment such as outdoors at night, a sufficient fluctuation range by a luminance difference due to the front light 77 can be secured. Can be secured.

  If attention is paid to the luminance, a dynamic range of 9559: 0.6 = 42 db can be obtained even if this value is observed as it is at a short distance. In fact, since there is a luminance meter with a dynamic range of about 45 db even at present, if a modulation method as described later with a strong noise resistance such as spread spectrum is used, it is possible to measure as it is, but a high-precision photo detector In addition, since the number of A / D bits is required to be large (8 bits or more), in the first embodiment, as a method for configuring the terminal at low cost even with the current device, the gain of the photodetector (in some cases, the lens aperture is also included). ) And the following gain adjustment is performed as indicated by a trapezoidal figure frame 81 in FIG.

(A) In a sunny outdoor environment with a large range of fluctuation, lower the gain (in addition to using a lens diaphragm or the like) to prevent saturation.
(B) Increase the gain in the dark outdoors with little fluctuation.

  In this way, it is possible to perform control using an A / D converter of about 8 bits with a dynamic range of about 20 db in accordance with environmental illuminance conditions. Note that the information for controlling the gain of the photodetector is finely controlled so that the camera system can shoot normally with a small dynamic range, and this may be used to control the system of the photodetector.

  Actually, the luminance of the optical tag device 18 is observed by the photodetector 61 of a remote terminal (camera-equipped mobile phone 22). Since the photodetector 61 of the present invention is a single unit, the size of the light receiving sensor should be sufficiently large several times to several tens of times as compared with a photodiode used in a normal image sensor (photographing image sensor 57). Even if it operates at high speed, sufficient sensitivity and dynamic range can be secured. Since the optical tag device 18 is actually observed within a viewing angle of 2 ° × 2 °, if the panel has a size of 1 square meter, the photodetector 61 has a narrow angle of view of 2 ° × 2 °. For example, since the luminance fluctuation can be detected with full angle of view within 30 meters, the detection is very good.

  FIG. 15 is a conceptual diagram of detection by the photodetector 61. As shown in the figure, even when the field angle of the photo detector 61 is set to 2 degrees × 2 degrees, it is possible to detect the luminance variation at the full field angle at a short distance (approximately within 30 meters). When the distance is longer, for example, a distance of 60 meters, a panel with a size of 1 square meter has a luminance variation of 1/4 due to the average luminance and power of the surroundings, and a further distance, for example, At 90 meters, it will be 1/9. Practically, depending on the gain control and A / D accuracy, the fluctuation of the original optical tag device 18 is a sufficiently high ratio such as “15: 1” or “32: 1” under certain conditions. Therefore, even a panel with a size of 1 square meter can be set to about 100 meters. This is clear from the fact that, even at the level of a digital camera, if the subject has an area larger than a pixel and is properly exposed, it can distinguish white paper from black paper even at very long distances. It is. In addition, in order to perform stable detection even at a longer distance, a method such as using a larger panel or a front light 77 having a higher luminous intensity, such as a more accurate camera shake correction and a narrower detection angle of view of the photodetector. It is possible. On the other hand, if it is within a relatively short distance, a method of providing a detection angle of view with a margin by omitting the camera shake correction mechanism is also conceivable.

Next, data format and optical tag modulation and demodulation will be described.
FIG. 16 is a format structure diagram of a data format. The source data block 82 has a fixed-length head block 83 and variable-length parameter blocks that follow the head block (in the figure, there are three first to third parameter blocks, but this number of blocks is an example) 84 to 86. The head block 83 includes a command code part 87, a first parameter block byte length storage part 88, a second parameter block length storage part 89, and a third parameter block length storage part 90. The first to third parameter block length storage parts The actual lengths (byte lengths) of the first to third parameter blocks 84 to 86 are stored in 88 to 90, respectively.

  FIG. 17 is a diagram showing the storage code of the command code portion 87 and the meaning of the code. As shown in this figure, for example, codes from “0” to “6” can be stored in the command code portion 87, and these codes are commands for instructing the operation of the terminal (mobile phone). For example, an application such as mobile phone web access, mail, or scheduler can be controlled based on the acquired data. Incidentally, the case of the code “0” and only the parameter “1” corresponds to the prior art 2.

  FIG. 18 is a diagram illustrating an example of data. In this example, the code “2” is set in the command code portion 87, and “107”, “20”, and “0” are set in the first to third parameter block length storage portions 88 to 90, respectively. Yes. Then, the first parameter block 84 and the second parameter block 85 are connected to the display information (“information acquisition complete! Necessary access information (ID and password) is stored. In the illustrated example, the third parameter block 86 is unused.

  FIG. 19 is a process diagram of the optical tag device 18. In this figure, when receiving the source data block 82, the optical tag device 18 performs data compression, error correction, and addition of a flag sequence to generate a data block 820. Then, on / off control of light emission is performed using a data block 821 obtained by converting them into two types of bit pattern sequences having low correlation to each other.

  FIG. 20 is an explanatory diagram of a pattern spreading method of a bit pattern series. In the first embodiment, the length of the bit pattern series (SA, SB) is increased to 15 bits each of SA = “111101011001000” and SB = “0000010100110111”. Now, assuming that the minimum ON / OFF slot time of the optical tag device 18 (minimum period of the timing generator 66 in FIG. 10) is 1 millisecond, the time required to send 1 bit is 15 milliseconds. The bit rate is about 67 bits / second. However, since the data to be transmitted is mainly text data, the actual bit rate is considered to be higher than the error correction and the data compression exceeds this. The time required for detection is 50 × 15 = 0.75 seconds assuming that a certain frame finally becomes 50 bytes. Therefore, one frame can be acquired in 0.75 seconds. However, if the data acquisition from a certain point is in the middle of the frame, the information can be acquired in 1.5 × 2 = 3 seconds at worst.

  Next, details of the modulation processing performed by the terminal (camera-equipped mobile phone 22) will be described. First, regarding synchronization, in the case of code spread modulation using such a pattern, it is necessary to synchronize both the optical tag device 18 and the terminal (camera-equipped mobile phone 22). In the first embodiment, the terminal is a camera-equipped mobile phone 22 and has a communication function on the terminal side, so it is easy to ensure independent synchronization with the optical tag device 18 with a required accuracy or more, or Alternatively, the detection timing may be shifted on the terminal side by a constant phase step until the detection is successful.

  In the present invention, since a continuous amount is observed by the photodetector 61, as a synchronization and detection method suitable for this, the start timing of one cycle of the pattern is obtained by using the peak of a so-called matched filter. And In this example, for example, synchronous supplementation (using a SAW device in the GHz band) by a SAW (surface acoustic wave) matched filter in CDMA (Code Division Multiple Access) communication is applied to the KHz band of the first embodiment.

  FIG. 21 is a conceptual structural diagram of the photo detector section 56 of the camera-equipped mobile phone 22 (corresponding to the internal details of the broken line section in FIG. 7). In the first embodiment, the luminance value within the target angle of view observed under gain control is A / D-sampled at a period four times the time slot of the pattern code, that is, 4 KHz, and placed in the FIFO buffer. The phase of the sample clock may be asynchronous with the light source (optical tag device 18), but the period error needs to be kept constant. As for the length of the buffer, 15 bits as the code pattern are sampled at a period of 4 times, so that 60 samples are sequentially stored in the FIFO. The specific addresses of the FIFO are taken out every four, and the data group corresponding to the sign bit 1 and the data group corresponding to the sign bit 0 constitute a matched filter that performs summation calculation while being weighted. The sampled FIFO data may be sequentially read by the CPU and calculated in software.

The range of this filter indicates the correlation value of the input signal,
Maximum: 255 × 8/8 + (0 × 7) / 7 = 255
Minimum: 0 × 8 / 8− (255 × 7) / 7 = −255
In the case of DC input, 0 × 8 / 8− (255 × 7) / 7 = 0. However, it is from the start of input of the target pattern until detection synchronization can be achieved. When the target pattern does not exist, it takes a small positive or negative value.

  FIG. 22 is a state diagram at the time when the target pattern signal is input but the phase is not yet matched (shifted by 4 signed bits = 16 samples). The graph in the figure shows a time-series change in the data sample value stored in the FIFO. As shown in the figure, the actual observation is a signal fluctuation that is smaller than the actual detection scale (MAX255). Actually, noise is added to the observation data as shown in the figure, but for the sake of simplicity, if it is assumed that “140” at one level and “120” at two levels, the output of this filter is about − 1.

  FIG. 23 is a state diagram when the detection phases are matched. In this state, the output outputs the variation width “20” of the added pattern. From this point, this peak lasts for a period of 4 samples (until the broken line on the graph), and the output drops rapidly. In fact, if the observed signal can be detected as “140” in the bright state and “120” in the dark state, for example, when the output at each sample time is calculated, only the four sample times in phase are peaked. “20” is indicated, and after that, “−1.4” is obtained. In this way, since the peak can be detected very easily, it is easy to match the detection timing thereafter. The filter output is stored in a FIFO for four samples as shown in FIG.

  FIG. 24 is a flowchart showing processing from the FIFO until the CPU determines a 0/1 bit and stores it in the decoded bit buffer 50. In this processing, since the bit string is decoded and stored in the decoded bit buffer 50 while containing some errors, the frame detection (in this embodiment, the data frame) The first and last is a sequence of 1), and an error correction block is extracted and error correction is performed, and the transmitted source data block is restored and processing is performed accordingly.

  Specifically, first, the data in the filter buffer 65 is read (step S31), and it is determined whether or not the threshold has been exceeded for four consecutive sample times in order to perform stable extraction of “1” (step S32). .

  When the threshold is exceeded for four consecutive sample times, t = -1 data (midst stable timing from the oversampled data: -2 may be used) is extracted (step S33), and the filter output value It is determined whether or not exceeds the threshold (step S34). If the filter output value exceeds the threshold, “1” is stored in the decoded bit buffer 50 (step S35). If the filter output value does not exceed the threshold, the filter output value is the threshold × (−1). Is determined (step S36).

  If the filter output value exceeds the threshold × (−1), “0” is stored in the decoded bit buffer 50 (step S37). If the filter output value does not exceed the threshold × (−1), "0" is randomly stored in the decoded bit buffer 50 (step S38). Next, it is determined whether or not there is a detection stop from the upper sequence (step S39). If there is a detection stop, the process is terminated. If there is no detection stop, after waiting for 4 sample times (step S40), t = -1 Data is extracted (step S41), and the processing from step S34 onward is repeated.

Since it is as above, according to 1st embodiment, the following effects are acquired.
(A) Since the data communication is received by the dedicated sensor (photodetector 61), the data transfer speed, while maintaining the convenience of the prior art 2 of taking the actual camera angle-of-view image and its position and information. That is, the delay time until the acquisition of information and the transmission speed can be increased.
(B) Since the light modulation processing is buffering or arithmetic processing for one data, a very simple processing configuration can be achieved.
(C) Since camera shake correction is added to the optical system of the photo detector 61, it is possible to detect stably even a terminal that performs hand-held shooting with a narrow detection angle of view.
(D) Since it was applied to a usage style in which outdoor advertising and mobile phones are used together, this enables advertisers to conduct limited service promotions and quantitative measurement of advertising effectiveness for outdoor advertising. . On the other hand, since the user is also a signboard with special information provision, there is an effect that the recognition degree to the advertisement is naturally improved.
(E) In addition, for example, it is possible to target an extremely large area compared to information provision using a two-dimensional code by an IC tag or a camera, and furthermore, information provision combined with a physical entity such as a signboard is possible. Therefore, a unique advertising effect can be brought about.
(F) Since various communication cooperation commands can be incorporated into the transmission data, it is possible to provide very effective information.
(G) Since code modulation is used for pulse modulation, even a weak luminance signal can be detected stably.
(H) Since a matched filter is configured for oversampled continuous data, phase synchronization between the optical tag device side and the terminal side can be easily performed.
(G) Since the optical tag device 18 is configured by combining the front light 77 and the reflective liquid crystal shutter 78, stable brightness fluctuations are possible not only indoors but also in various situations from outdoor in the daytime to outdoor in the nighttime. Can bring.
(K) Since the signage 16 and the optical tag device 18 are arranged, it is easy for the user to recognize and can bring about a unique advertising effect.
(C) Since the gain control of the photo detector 61 is performed from the exposure control data of the camera, it is possible to secure an optimal dynamic range and sensitivity in accordance with the use situation.

  In the prior art 2, regarding the modulation of the final light source, only a modulation method that diffuses bits into patterns can be used as a method that can transmit from discrete and complete image data with a high S / N ratio. In one embodiment, the method using the photo detector 61 allows high-speed and continuous observation, so if the characteristics of the detected data can be ensured, a simpler modulation scheme, for example, pulse modulation at a specific carrier frequency (10 KHz, etc.) You may do it. The same applies to pulse frequency switching modulation.

  Further, as described in the related art 2, the aiming on the camera and the screen is omitted, and only the photo detector 61 is aimed at only a simple tube or a telephoto optical finder to display and operate. It is also possible.

<Second embodiment>
By the way, in the above first embodiment, the photodetector 61 is used as the light receiving means for receiving the bright spot of the optical tag device 18, but the present invention is not limited to this. What is necessary is just to have an angle of view corresponding to the size of the information providing object (the optical tag device 18). For example, an image sensor such as a CCD or CMOS may be used, or a plurality of photodetectors may be arranged on a plane.

Hereinafter, an embodiment using an image sensor such as a CCD or a CMOS (hereinafter referred to as a second embodiment) will be described.
FIG. 25 is an internal block diagram of the camera-equipped cellular phone 220 according to the second embodiment. In this figure, the camera-equipped cellular phone 220 uses an image sensor (hereinafter, referred to as “communication image sensor 610” in order to distinguish it from the imaging image sensor 57) instead of the photodetector 61 of the above-described embodiment. There is a feature.

  That is, the camera-equipped mobile phone 220 in the second embodiment includes an operation button unit 51 (28 to 32) including buttons such as a data communication unit 48 including the antenna 24, a data buffer 49, a decoding bit buffer 50, and a camera key 28. , A voice input / output unit 52 including a speaker 26 and a microphone 33, a display unit 27, a CPU 53, an acceleration sensor 54, a camera shake correction actuator 55, a communication image sensor unit 560, a photographing lens 34, and a photographing image such as a CCD or CMOS. A sensor 57, an image buffer 58, a display buffer 59, a storage unit 60, and the like are provided. In addition to this, a power supply unit such as a battery is provided, but is omitted in order to avoid congestion in the drawing.

  The communication image sensor unit 560 includes a condenser lens 35, a communication image sensor 610, an amplification and A / D converter 62, a timing generator 63, a matched filter 64, and a filter buffer 65.

  Similar to the above-described embodiment, there is nothing to be noted about the configuration of the camera portion including the photographing lens 34 and the photographing image sensor 57. There may be a finder-like function capable of aiming at the optical tag intended by the user or a function for displaying the position of the information light source in the actual image. Therefore, even if the frame rate of the camera portion is, for example, a transmission signal speed of several tens of bits / second or several hundreds of bits / second, there is no problem in aiming confirmation, for example, an image of about 10 frames / second. Acquisition ability is sufficient. Information such as gain control of the camera part may be used for setting gain control of the communication image sensor unit 560 without being closed inside the camera part.

  The communication image sensor unit 560 is a block that digitizes the luminance fluctuation signal of the optical tag device 18 and performs correlation evaluation, and basically has the same function as the photodetector unit 56 (see FIG. 7) of the above embodiment. However, the photo detector unit 56 of the above embodiment includes one photoelectric conversion element (photo detector 61), whereas the communication image sensor unit 560 of the second embodiment is configured by a CCD, a CMOS, or the like. The difference is that a two-dimensional image sensor (communication image sensor 610) including a plurality of photoelectric conversion elements (pixels) is provided.

  The photodetector 61 and the communication image sensor 610 are both devices that “convert light into an electrical signal”. The photodetector 61 is composed of a single photoelectric conversion element, whereas the communication image sensor 610 is different in that it is composed of a plurality of photoelectric conversion elements (a plurality of pixels). However, the number of pixels of the communication image sensor 610 may be smaller than that of the imaging image sensor 57. In other words, the communication image sensor 610 that only needs to detect a bright spot within a narrow angle of view has a lower resolution (for example, 30 dots × 30 dots) than the shooting image sensor 57 that requires shooting image quality. It doesn't matter.

  The light source (optical tag device 18) that emits information is assumed to sample a 1 kHz baseband signal at four times the speed as in the above embodiment. Accordingly, the frame rate of the communication image sensor 61 ′ is 4000 frames / second, and 1/400 [second] = 0.25 milliseconds, so that the necessary dynamic range is satisfied with a constant shutter speed of 0.25 milliseconds. The device should be designed as follows.

  As described above, assuming that the resolution of the communication image sensor 610 is 30 × 30 dots, the data rate for reading all pixels of the frame in this case is 30 × 30 × 4000 = 3,600 with 8-bit data for one dot. 1,000 frames / second.

  In the case of a general VGA (640 × 480 dot) image sensor, 30 fps (frame / second) is a normal bandwidth, and the data rate of 30 bps is 640 × 480 × 30 = 9, 216,000 dots. Therefore, if the data rate of the communication image sensor 610 of the second embodiment is about 3,600,000 frames / second, it can be realized with a margin.

  Next, the relationship between specific design numerical values, detection angles, distances, and object sizes will be described. Assume that the assumed usage state is the same as in FIG. In addition, the distance between the optical tag device 18 and the terminal (camera mobile phone 220) is 50 m, and the size of the light source of the optical tag device 18 is a 10 cm × 10 cm rectangle (10 cm square).

  FIG. 26 is a diagram showing an imaging angle of view (wide angle of view of about 26 degrees × 39 degrees) of the imaging lens 34 (imaging image sensor 57) in the camera-equipped mobile phone 220 according to the second embodiment. It is a figure which shows the detection field angle (narrow field angle of about 6 degree x 6 degree | times) of the image sensor 610 for communication in the mobile phone 220 with the camera.

  Here, as an example of a target value for realizing camera shake and rough aiming, a 5 m range 50 m ahead is set as a sensor field angle (view angle of the communication image sensor 610: 6 degrees × 6 degrees). 6 degrees is given as a tangent (5 m / 50 m). The coverage angle per dot of the communication image sensor 610 having an angle of view of 6 degrees and 30 × 30 dots is given by about 6/30 = 0.2 degrees square.

FIGS. 28A to 28D are views showing the apparent dot size G at each distance of the area (10 cm × 10 cm) of the illumination portion of the optical tag device 18.
For example, as shown in FIG. 28A, when the optical tag device 18 is 5 m away from the camera-equipped mobile phone 220, that is, 10 cm≈1.1 degrees, the dot size G of the optical tag device 18 is 1.1 / 0. .2 = 5.5 dot width and 100% luminance change can be obtained. As shown in FIG. 28B, when the optical tag device 18 is 10 m ahead of the camera-equipped mobile phone 220, that is, 10 cm≈0.6 degrees, the dot size G of the optical tag device 18 is 0.6 / 0. .2 = approx. 4 dots wide and 100% change in luminance. As shown in FIGS. 28C and 28D, when the optical tag device 18 is 50 m away from the camera-equipped mobile phone 220, that is, 10 cm≈0.11 degrees, the dot size G of the optical tag device 18 is 0. .11 / 0.2 = approx. 0.5 dot width, resulting in a luminance change of an area of 1 dot or less, and the power becomes about 1/4 to 1/16.

  As described above, the signal strength is ¼ or less at a location 50 m away from the optical tag device 18 (the location of the camera-equipped mobile phone 220), but such a decrease in signal does not cause any inconvenience. In the case of a general image sensor, the dynamic range per dot is about 30 to 50 dB. Therefore, if the detection level of the ON light source with one dot covered is 50% to the saturation level of the dynamic range. Even if the detection voltage falls by 1/4 (that is, −6 dB), no inconvenience occurs.

  Although the shutter speed is 0.25 milliseconds, generally, the absolute luminance of the light source is almost constant and is very high with respect to the surroundings due to the requirement for visibility. Even if the 610 is operated at the above-described 4000 frames / second (shutter time of about 0.25 milliseconds), it is possible to easily cope with the lens aperture, gain, and the like, so that the blinking information of the light source can be sufficiently obtained. Note that there is a slight increase in luminance due to reflection of ambient illumination, but this is not a big problem as compared to the ON / OFF luminance change of the signal itself.

  Of course, for example, when the surroundings become dark, the surrounding image becomes a so-called “black shadow”, or when it is fine, the so-called “white stripes”, so at a fixed shutter speed, it looks like a photograph. However, since the communication image sensor 61 is exclusively “communication” and is not used for monitoring or photographing, surrounding images other than the information light source are “white stripes” and “ Even if “blackout” occurs, it is not affected at all.

  Next, in the second embodiment, the camera shake mechanism (actuator 55) can be eliminated. This is because the angle of view of the communication image sensor 610 of the second embodiment is as large as “6 degrees × 6 degrees” with respect to the angle of view “2 degrees × 2 degrees” of the photodetector 61 of the first embodiment. . That is, the allowable amount of camera shake is 3 × 3 = 9 times, and resistance to large camera shake (hand shake, walking, use from a vehicle, etc.) is enhanced. Here, the angle of view of the communication image sensor 610 is set to “6 degrees × 6 degrees”, but this is only an example. From the standpoint of camera shake resistance, a larger angle of view can be used.

  FIG. 29 is a conceptual diagram of the second embodiment that does not require a camera shake mechanism. As shown in this figure, the communication image sensor 610 is disposed behind the condenser lens 35, and the image of the light spot A is formed on the light receiving surface of the communication image sensor 610 through the condenser lens 35. Now, as shown in (a), when the light spot A is located on the optical axis of the condensing lens 35, the image of the light spot A is connected to a position B substantially at the center of the light receiving surface of the communication image sensor 610. However, when the light spot A is shifted from the optical axis of the condenser lens 35 as shown in (b), the image of the light spot A is also shifted from the optical axis of the condenser lens 35. ′ Is imaged. At this time, if the size of the light receiving surface of the communication image sensor 610 is sufficient, that is, if it is large enough to include the position B ′, the image of the position B as well as the image of the position A is taken. Thus, it is possible to perform alignment (superimpose on the image at position A), and hence it is possible to perform camera shake correction by image processing. Therefore, it is not necessary to provide a camera shake correction mechanism (actuator 55).

As an image processing method for correcting camera shake, the following can be used.
FIG. 30 is a corresponding limit schematic diagram of a motion capable of capturing a pattern. When each dot of the communication image sensor 610 is processed in parallel, as shown in this figure, if overlapping dot coordinates H do not occur through a plurality of frames (for example, 15 frames) in two pattern series, No dot indicating correlation is generated, and therefore pattern detection cannot be performed.

The following expression (1) is a general expression of the corresponding limit of the motion that can capture the pattern.
d> mpf × n × x (1)
d: Target spot diameter (diameter)
mpf: Number of moving dots per frame time (MovePerFrame)
n: Spreading code length (or data block length)
x: Double speed of capture for target signal

In the above formula (1), n = 15 (number of bits of SA and SB patterns = 15 bits), d = 4 (10 em ahead 10 m≈0.6 degrees 0.6 / 0.2 = about 4 dot width), When x = 4 (4 because 1 kHz baseband signal is captured by 4 kHz capture) is applied and mpf is solved under this condition, the following equation (2) is obtained.
mpf <d / (n × x) = 4 / (15 × 4)
≒ 0.67 dots / 1 frame time (2)

  Since 0.67 dots is approximately 0.134 degrees, it can be seen that an angular velocity per second of 0.134 × 4000 = 536 degrees / second can be accommodated. Of course, the maximum angular velocity is 536 degrees / second, but in the case of the second embodiment, the fluctuation range must be within 6 degrees square (the angle of view of the communication image sensor 61 ′ is 6 degrees × 6 degrees). I must.

  These exemplified values are appropriately set when the present invention is actually used. For example, in order to cope with a smaller signal light source and a farther signal light source, the resolution of the communication image sensor 610 may be increased. Further, when the allowable perimeter such as camera shake may be small, it is possible to cope with a smaller dot area by narrowing the angle of view of the communication image sensor 610. Furthermore, the frame rate may be further increased in order to deal with extremely severe blurring. Incidentally, it goes without saying that increasing the frame rate increases the communication speed.

  In any case, in the second embodiment, as long as it is within the angle of view (6 degrees square) of the communication image sensor 610, there is no practical problem no matter how the signal is captured. The camera shake mechanism (actuator 55) 610 can be eliminated.

Since it is as above, according to 2nd embodiment, the following effects are acquired.
First, the characteristics of two-dimensional communication can be brought out. That is, the communication image sensor 610 can simultaneously receive a plurality of signals, and each can be discriminated in position (discrimination based on the pixel position of the communication image sensor 610).

  In addition, both the minimum detection angle and the detection range can be achieved. That is, in the second embodiment, the detection aiming range is set to 6 × 6 degrees, so that the minimum light source area is reduced to 0 while expanding the area range to 9 times the same range (2 degrees × 2 degrees) of the above embodiment. It can be set to 9 times as well as 2 degrees or less. For this reason, if the light source is the same, detection is possible even at a distance three times that of the above embodiment, or if the light source is the same distance, even a light source having a size 1/9 smaller than that of the above embodiment is detected. Further, since the detection angle (view angle) can be widened, the range of the aiming area performed by the user can be increased while maintaining the detection limit angle and the like. Therefore, rough aiming is possible, and operability at the time of information acquisition can be improved.

  Further, the mechanism can be simplified. That is, when aiming at a narrow limit detection angle, the camera shake correction mechanism (actuator 55) as shown in the previous embodiment can be eliminated.

  In addition, cost merit can be achieved. That is, the number of pixels of the communication image sensor 610 may be smaller than that of the imaging image sensor 57, and a low-resolution one (inexpensive one) can be used, and the accuracy of the optical system is not required. Cost can be reduced.

  Further, since a two-dimensional sensor (communication image sensor 610) specialized for communication is used, adjustment and design of sensor sensitivity become extremely simple. In other words, a general landscape has a very large dynamic range of illumination from nighttime to sunny daytime, so a camera that captures a landscape combines real-time illumination of several tens to 100 dB with a combination of shutter speed, lens aperture, gain, and the like. The luminance range must be covered.

  However, if a dedicated image sensor (communication image sensor 610) is used as in the second embodiment, it becomes a sensor only for optical communication. On the other hand, since the information light source side has a constant absolute luminance, gain adjustment only needs to be performed on the target signal, so sensitivity adjustment and design become extremely simple.

  FIGS. 31 to 33 are flowcharts of a control program (shooting mode program) that is preferably applied to the mobile phone with camera 220 of the second embodiment. When this flowchart is started, first, a camera unit such as the image sensor 57 for shooting is activated (step S51), and a frame image captured by the camera unit is displayed on the display unit 27 as a through image for composition confirmation (step S52). ).

  Next, it is determined whether or not a user has instructed to display a predetermined submenu (step S53). The “predetermined submenu” is a menu for receiving information from the optical tag device 18, and the menu selection button is provided in the operation unit 51 in advance.

  When the sub menu display instruction is not performed, the normal processing for the photographing mode is executed (step S54). On the other hand, when the sub menu display instruction is performed, the following process is executed.

  That is, the predetermined submenu is displayed on the display unit 27 (step S55), and information from the optical tag device 18 is received in the submenu and whether or not the information is stored together with the photographed image. To the user (step S56).

  If the user selects negative (does not store information), the process returns to the normal processing (step S55). If the user selects affirmative (stores information), the display unit A predetermined “grid” is displayed in the through image being displayed on the screen 27 (step S57).

  Here, the “grid” is a rectangular frame representing the angle of view (6 degrees × 6 degrees) of the communication image sensor 610, and the aiming frame 40 of the above-described embodiment (see FIG. 4A). It is equivalent to. The user who wants to receive information from the optical tag device 18 is directed to the camera-equipped mobile phone 220 so that the entire optical tag device 18 or a signboard including the optical tag device 18 is contained in the grid. Adjust (shooting direction).

  Next, it is determined whether or not a bright spot is detected in the grid (step S58).

  When no bright spot is detected in the grid (when the determination result in step S58 is “NO”), it is determined whether or not the shutter button (not shown) of the operation unit 51 is pressed (step S59). If not, the process returns to step S57 again. When the shutter operation is performed, a photographed image of the photographing image sensor 57 at that time is captured (step S60), and a predetermined message for warning that information from the optical tag device 18 has not been received is displayed. The image is displayed on the unit 27 (step S61), and an image saving instruction by the user is detected (step S62). If there is no save instruction, the captured image is discarded (step S63), and the flowchart ends (returns to the main program). On the other hand, when an image save instruction by the user is detected, the captured image is converted into a file (step S64). ) After saving the file in the storage unit 60 (step S65), the flowchart is ended (return to the main program).

  As described above, when a bright spot is not detected in the grid, after displaying a predetermined message for warning that the information from the optical tag device 18 is not received, the same as in normal camera shooting. The captured image at that time can be filed and stored. For this reason, the user can know from the warning message that information from the optical tag device 18 could not be received. That is, it is possible to intuitively know that the stored image is the same as a normal camera image (an image without information).

  On the other hand, when a bright spot is detected in the grid (when the determination result in step S58 is “YES”), the bright spot portion in the grid is differentiated and displayed (step S66). The user confirms the bright spot portion displayed in a differentiated manner, determines whether the bright spot portion is within the grid, and if necessary, finely adjusts the direction (photographing direction) of the camera-equipped mobile phone 220. To do.

  Next, a depression operation of a shutter button (not shown) of the operation unit 51 is determined (step S67). If the shutter operation is not performed, the process returns to step S66 again. If the shutter operation is performed, the captured image of the imaging image sensor 57 at that time is captured (step S68).

  Here, the imaging image sensor 57 continuously generates and outputs images of several tens of frames per second, but the bright spots in the grid (the bright spots of the optical tag device 18) are also several times or several tens of times per second. Since the light blinks in a cycle, the image sensor 57 for shooting generates and outputs two types of images, an image including a bright spot and an image not including the bright spot. It is desirable to use an image that does not include a bright spot as a captured image for storage and storage. This is because when an image including a bright spot is stored, the bright spot becomes an obstacle and the image looks worse. For this reason, in the second embodiment, when capturing an image for storage in step S68, an image that does not include a bright spot is selected.

  When the image capture is completed, it is next determined whether or not the reception of information from the optical tag device 18 is completed (step S69). If not completed, an icon such as an hourglass indicating that information is being received, a symbol or a message is displayed on the display unit 27 (step S70), and step S69 is repeated.

  When the reception of information from the optical tag device 18 is completed, the image captured in step S68 (image not including the bright spot) and the received information are combined to generate one image file (step S71). : Association means) and the received information is displayed on the display unit 27 (step S72: display means).

  Next, it is determined whether or not there is an image saving instruction by the user (step S73). If there is no image saving instruction, it is determined whether or not there is a request for re-reception of information from the user (step S74: operation acceptance). Means, re-reception execution means). If there is a request for re-reception of information, it is considered that there is an error in the information displayed on the display unit 27, and step S69 and subsequent steps are re-executed. Here, when the information is re-received, the image is not captured (step S68). This is because there is no inconvenience in the image captured in step S68, and only a trouble (communication failure or the like) has occurred when receiving information from the optical tag device 18. In this way, it is possible to reduce the processing burden and the processing time when re-receiving information.

  If the information is not received again, the image file generated in step S71 is discarded (step S74), and the flowchart is ended (returns to the main program). If the reception of information is completed normally and an image save instruction is detected (“YES” in step S73), the image file generated in step S71, that is, the image file is imported in step S68. An image file including the placed image (image not including a bright spot) and reception information is stored in the storage unit 60, and the flowchart is ended (return to the main program).

  Here, as an example of the image file including the image (image not including the bright spot) captured at step S68 and the reception information, Exif (Exchangeable image file format: main image) of a digital camera. This is a general-purpose image file format defined so that various attached information and other optional information at the time of shooting can be saved together with the data of.) Files can be used. In the Exif file, a storage area for text information that can be arbitrarily used is defined. By writing the reception information from the optical tag device 18 in this storage area, the image data and the reception information are combined in one image file. Can be paid.

  Thus, according to the flowchart shown in the figure, the received information from the optical tag device 18 can be stored and stored together with the captured image in a single image file. Therefore, by reading out the information at the time of reproduction of the image file, it is possible to realize various services such as the use of coupons as in the application example in the above embodiment.

  Further, since the image data included in the image file is an image that does not include the bright spot of the optical tag device 18, the appearance of the image is not deteriorated.

  In addition, when the information is re-received, the image capture (step S68) is passed, so that a wasteful image capture process is not performed, and the processing load and the processing time can be reduced accordingly.

  Note that the image captured in step S68 is not limited to a still image. A simple video may be used. In other words, the image generated and output from the image sensor 57 for shooting is an image of several tens of frames per second, and each of these images is a still image. Then, since it looks like a moving image with movement, frame images taken during an arbitrary time may be continuously captured as simple moving image data. However, when capturing the moving image data, it is desirable to exclude an image including a bright spot from the capturing target in order to avoid deterioration in the appearance of the reproduced moving image.

  Also, generally, the simple moving image data captured in this way has an enormous file size when stored as a moving image file as it is, and presses the capacity of the storage unit 27. For example, I picture, B picture, and P picture In this file compression, an I picture may be composed only of images that do not include a bright spot, and a B picture or a P picture may be generated from the I picture. .

It is a utilization state figure of the information transmission system in a first embodiment. It is the front view and back view of the mobile telephone 22 with a camera corresponding to the information transmission system in 1st embodiment. FIG. 6 is a sequence diagram showing a relationship between a user operation on the camera-equipped mobile phone 22 and an internal operation of the camera-equipped mobile phone 22. FIG. 6 is a diagram showing a display example of the display unit 27 and a diagram showing a display example of the display unit 27 when the aiming frame 40 is superimposed on the optical tag device 18. FIG. 6 is a diagram illustrating a display example of the display unit 27 when information acquisition from the optical tag device 18 fails, and a diagram illustrating a display example of the display unit 27 when information acquisition from the optical tag device 18 is successful. It is the figure which shows the example of a display of the display part 27 when downloading the image file of a coupon, and the figure which shows the image of a coupon. It is an internal block block diagram of the mobile telephone 22 with a camera in 1st embodiment. It is a key map of camera shake amendment in cellular phone 22 with a camera of a first embodiment. It is a key map of camera shake amendment in cellular phone 22 with a camera of a first embodiment. It is a figure which shows the angle of view (for example, 32 degrees x 32 degrees) of the optical system of a camera system. It is a figure which shows the angle of view (for example, 2 degree x 2 degree | times) of the optical system of a photodetector system. FIG. 3 is an internal block configuration diagram of the optical tag device 18. It is a figure which shows the front light 77 and the reflective liquid-crystal shutter. It is a graph which shows the brightness | luminance calculation result of an optical tag. 3 is a detection conceptual diagram of a photodetector 61. FIG. It is a format structure diagram of a data format. It is a figure which shows the storage code of the command code part 87, and the meaning of the code. It is a figure which shows an example of data. FIG. 5 is a process diagram of the optical tag device 18. It is explanatory drawing of a pattern spreading | diffusion system. It is a conceptual structure figure of photo detector part 56 of cellular phone 22 with a camera (equivalent to the internal details of the broken line part of Drawing 7). It is a state diagram at the time when the target pattern signal is input but the phase is not yet matched (shifted by 4 signed bits = 16 samples). It is a state diagram when a detection phase is suitable. FIG. 6 is a flowchart showing processing from the FIFO until the CPU determines a 0/1 bit and stores it in a decoded bit buffer 50. It is an internal block block diagram of mobile telephone 22 'with a camera in 2nd embodiment. It is a figure which shows the imaging field angle (wide field angle of about 26 degree x 39 degree | times) of the imaging lens 34 (image sensor 57 for imaging | photography) in the mobile phone 220 with a camera of 2nd embodiment. It is a figure which shows the detection field angle (narrow field angle of about 6 degree x 6 degree | times) of the image sensor 610 for communication in the mobile phone 220 with a camera of 2nd embodiment. It is a figure which shows the apparent dot size in each distance of the area (10 cm x 10 cm) of the illumination part of the optical tag apparatus 18. FIG. It is a conceptual diagram which does not require the camera shake mechanism in 2nd embodiment. It is a corresponding limit schematic diagram of the movement which can capture a pattern. It is a figure (1/3) which shows the flowchart of a control program (photographing mode program) preferable when applied to the mobile phone with camera 220 of the second embodiment. It is a figure (2/3) which shows the flowchart of a control program (photographing mode program) preferable when applied to the mobile phone with camera 220 of the second embodiment. It is a figure (3/3) which shows the flowchart of a preferable control program (photographing mode program) applied to the mobile phone 220 with a camera of 2nd embodiment.

Explanation of symbols

18 Optical tag device (information output device)
22 Mobile phone with camera (information receiver)
25 Status indicator (notification means)
27 Display section (display means)
31 Cursor keys (specifying means, moving means, light receiving instruction means, specifying means)
34 Shooting lens (imaging means)
35 Condensing lens (light receiving means, optical system)
40 Aiming frame (index)
47 Message balloon (reproduction means)
50 Decoding bit buffer (restoration means)
53 CPU (designation means, light reception control means, conversion means, logic signal output means, second judgment means, imaging control means, conversion means, logic signal output means, association means, first judgment means, designation means, restoration means , Control means)
54 Acceleration sensor (control means)
55 Shake correction actuator (control means)
57 Image sensor for imaging (imaging means)
60 storage unit (storage means)
61 Photodetector (light receiving element, light receiving means)
69 Pattern data generator (selection means)
70 First control / drive unit (modulation means)
77 Front light (output means)
78 Reflective type liquid crystal shutter (output means)
220 Mobile phone with camera (information receiving device)
610 (light receiving means, two-dimensional image sensor)

Claims (6)

Imaging means;
A light source that emits light that changes in luminance in a time series included in the angle of view captured by the imaging means, or an object that is irradiated with light emitted from the light source, continuously emits light that changes in luminance in a time series. Means for receiving light automatically and restoring information from the received light;
A recording unit that records the image captured by the imaging unit in association with the information restored by the restoration unit ;
Instruction determination means for determining whether re-light reception of the light is instructed;
When it is determined that re-light reception is instructed by the instruction determination unit, a re-light reception start unit that starts re-reception of the light in a state where the image captured by the imaging unit is held;
An image pickup apparatus comprising: a recording control unit that controls the recording unit to store an image captured by the imaging unit and information restored from the light received by the restoration unit in association with each other . A completion judging means for judging whether or not the light reception corresponding to the information restored by the restoring means is completed ;
The recording control unit stores the image captured by the imaging unit and the information restored by the restoration unit in association with each other when the completion judgment unit judges that the light reception has been completed. The imaging device according to claim 1. The imaging apparatus according to claim 2, further comprising notification means for notifying that when the completion determination means determines that the light reception has been completed. Imaging means;
A light source that emits light that changes in luminance in a time series included in the angle of view captured by the imaging means, or an object that is irradiated with light emitted from the light source, continuously emits light that changes in luminance in a time series. An image pickup apparatus comprising: a restoration unit that receives light and restores information from the received light; an image captured by the imaging unit; and a recording unit that records the information restored by the restoration unit in association with each other ,
The luminance modulation is repetition of turning on and off a specific wavelength light,
An image pickup control means for controlling the image pickup means to pick up an image of the specific wavelength light extinguishing timing;
The image recording apparatus, wherein the recording unit stores an image captured by control by the image capturing control unit and information restored by the restoration unit in association with each other . A computer included in the imaging apparatus;
A light source that changes in luminance in a time series and emits light included in the captured angle of view or an object irradiated with light emitted from the light source continuously receives the light that changes in luminance in a time series. , Restoration means for restoring information from the received light,
A recording unit that records the captured image and the information restored by the restoration unit in a recording unit in association with each other;
Instruction determination means for determining whether re-light reception of the light has been instructed;
When it is determined that re-light reception is instructed by the instruction determination unit, a re-light reception start unit that starts re-reception of the light in a state where the image captured by the imaging unit is held;
Recording control means for controlling the recording means so as to store the image captured by the imaging means and the information restored from the light received by the restoring means in association with each other.
A program characterized by functioning as A computer included in the imaging apparatus;
A light source that emits light with a time-series change in luminance included in the captured angle of view or an object irradiated with light emitted from the light source is continuously received with the light that changes in luminance in a time series. , Functioning as a restoring means for restoring information from the received light, and a recording means for associating and recording the captured image and information restored by the restoring means to the recording unit,
The luminance modulation is repetition of turning on and off a specific wavelength light,
Function as an imaging control means for controlling to capture an image of the specific wavelength light extinguishing timing,
A program for causing the recording unit to function to store an image captured by control by the imaging control unit and information restored by the restoration unit in association with each other .

Images