JP2006277552A - Information display system and information display method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an information display system that enables remote control of a personal computer or the like, for example, in the process of a presentation. <P>SOLUTION: Input control information is converted into a predetermined pattern series and modulated to specific wavelength light with time series changes depending on the converted predetermined pattern series, the modulated specific wavelength light is radiated to an object, and the object irradiated with the specific wavelength light is shot at predetermined frame rates. The area irradiated with the specific wavelength light is extracted from a plurality of shot frames, the predetermined pattern series is generated from the modulation state of the extracted specific wavelength light irradiation area, the control information is generated from the generated predetermined pattern series, and a display means for displaying the shot frames is controlled according to the control information. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to an information display system and an information display method. Specifically, an information display system capable of sending a required command (operation command) in a cordless manner to the projector itself or a personal computer connected to the projector while viewing the projection image of the projector at a site such as a presentation, and the like The present invention relates to an information display method.

  In general, during a presentation while a personal computer screen is enlarged and projected onto a screen by a projector, the personal computer must be operated each time the page is moved or changed to another document, for example. In that case, it is not very smart to interrupt and operate the presentation one by one. If there is a person in charge of operating the personal computer, the presentation does not have to be interrupted, but human resources are necessary, and if the communication is not sufficiently communicated in the first place, wrong materials will be projected etc. Mistakes can also occur.

  With such a background, there is a need for a technology that allows a personal computer to be remotely operated without taking your eyes off the screen (while continuing the presentation).

  As a first prior art, for example, a laser pointer with a built-in wireless remote control is known (for example, see Non-Patent Document 1). According to this, it is possible to send various operation commands to a personal computer at a remote location simply by pressing the switch of the laser pointer during the presentation. However, since this conventional technology uses "radio waves", there are inconveniences unique to radio waves, such as the effects of electromagnetic waves on the human body, restrictions on laws and regulations (radio law), and effects on precision equipment in hospitals, etc. In addition, when there is an illegal radio station with a high output nearby, there is a possibility that a communication failure may occur due to the influence.

  Then, the 2nd prior art using "light" which does not have such an inconvenience is known (for example, refer patent document 1). This technology is a combination of a light pen, a video camera, and a data processing device (personal computer). In short, irradiate the spot light from the light pen to any place on the display screen of the data processing device, take this light spot image with a video camera, and determine the position coordinates of the light spot image (coordinates on the display screen). For example, the movement of the mouse cursor on the display screen is remotely controlled by indexing and sending the coordinates to the data processing device.

  This second prior art uses “light” that is harmless to the human body, is not subject to legal restrictions, and has no effect on precision instruments. There is no.

  In addition, this second conventional technology, when replacing the “display screen” with the “projector screen screen”, “irradiates the spot light from the light pen to any place on the“ projector screen screen ”.・ ・ ・ "Moving the mouse cursor on the projector screen" can be read as "remotely controlled", so "part" (below) Explanation) can also be achieved.

"Remote control with laser pointer RC-LUWPP", [online], ELECOM Co., Ltd., [Search on December 24, 2004], Internet <URL: http://www2.elecom.co.jp/products/RC-LUWPP .html> Japanese Patent Laid-Open No. 2002-140058

  By the way, in the second conventional technique, a light spot image on a display screen (or a projector screen screen) may be taken with a video camera, and the “coordinate” of the light spot is calculated and sent to a data processing device. It ’s just that. For this reason, for example, only the movement of the mouse cursor can be controlled only by such coordinate information, and other mouse operations (click operation, drag operation, etc.) still need to be operated by direct contact with the mouse. is there. Therefore, since the second prior art only achieves “part” of the problem at the beginning, there is still a problem to be solved in this respect.

  Accordingly, an object of the present invention is to selectively generate two or more commands using light and input these commands to a personal computer (not limited to a projector or the like), for example, An object of the present invention is to provide an information display system and an information display method capable of remotely operating a personal computer or the like while continuing a presentation.

According to the first aspect of the present invention, there is provided control information input means for inputting control information, conversion means for converting the control information input by the control information input means into a predetermined pattern series, and the predetermined information converted by the conversion means. In accordance with the pattern sequence, the modulation means for modulating the specific wavelength light with time series change, the irradiation means for irradiating the target with the specific wavelength light modulated by the modulation means, and the specific wavelength light emitted by the irradiation means The specific wavelength light is emitted from a photographing means for photographing the target object at a predetermined frame rate, a display means for displaying a frame photographed by the photographing means, and a plurality of frames photographed by the photographing means. Extraction means for extracting the region, and generating the predetermined pattern series from the modulation state of the irradiation region of the specific wavelength light extracted by the extraction means Based on the first generating means, the second generating means for generating control information from the predetermined pattern sequence generated by the generating means, and the control information generated by the second generating means, An information display system comprising display control means for controlling display of a frame displayed on the display means.
The invention according to claim 2 is characterized in that the display means includes a projection means for projecting a frame photographed by the photographing means, and the target is a projection object projected by the projection means. An information display system according to claim 1.
The invention according to claim 3 further includes control information storage means for storing a plurality of sets of the control information in association with the pattern series, and the conversion means is based on the control information input by the control information input means. 3. The information display system according to claim 1, further comprising selection means for selecting a corresponding pattern series from the control information storage means.
According to a fourth aspect of the present invention, the photographing unit photographs a frame including the specific wavelength light region in at least a half modulation section, and the first generating unit is time-sequentially by the photographing unit. Frame storage means for storing a plurality of shot frames in the order of shooting, reading means for sequentially reading frames every other frame from the frame storage means, and the control information storage for the frames read by the reading means A multiplication image generation means for generating a multiplication image obtained by multiplying a coefficient corresponding to the pattern sequence stored in the means; an addition image generation means for generating an addition image from the multiplication image generated by the multiplication means; and the addition image generation Determining means for determining whether or not there is a pixel area exceeding the threshold in the added image generated by the means; and 4. The information according to claim 3, further comprising: a pattern sequence specifying unit that specifies a pattern sequence based on an applied multiplication coefficient of a multiplication image that is a source of the added image when it is determined that a pixel region is present. It is a display system.
The invention according to claim 5 is a control information input step for inputting control information, a conversion step for converting the control information input in the control information input step into a predetermined pattern sequence, and conversion in the conversion step. In accordance with a predetermined pattern sequence, a modulation step for modulating light with a specific wavelength with a time series change, an irradiation step for irradiating the target with the specific wavelength light modulated in the modulation step, and specifying with the irradiation step From the imaging step of imaging the target irradiated with wavelength light at a predetermined frame rate, the display step of displaying the frame imaged in the imaging step, and the plurality of frames imaged in the imaging step, the specific An extraction step for extracting a region irradiated with wavelength light, and a modulation for the specific wavelength light irradiation region extracted by the extraction step A first generation step for generating the predetermined pattern sequence from the state, a second generation step for generating control information from the predetermined pattern sequence generated by the generation step, and generation by the second generation step And a display control step for controlling the display of the frame displayed in the display step based on the control information.
The invention according to claim 6 is characterized in that the display step includes a projection step of projecting the frame photographed in the photographing step, and the target is a projection object projected by the projection step. The information display method according to claim 5.
The invention according to claim 7 includes a memory that stores a plurality of sets of the control information in association with the pattern series, and the conversion step corresponds to the memory based on the control information input in the control information input step. The information display method according to claim 5, further comprising a selection step of selecting a pattern series to be performed.
According to an eighth aspect of the present invention, the imaging step captures a frame including the specific wavelength light region in at least a half modulation section, and the first generation step is time-series in the imaging step. A frame storing step for storing a plurality of frames photographed in the order of photographing, a reading step for sequentially reading frames from the plurality of frames stored by the frame storing step, and a reading step for reading the frames. A multiplication image generation step of generating a multiplication image obtained by multiplying a frame corresponding to a pattern sequence stored in the memory by a coefficient, and an addition image generation step of generating an addition image from the multiplication image generated by the multiplication step And the added image generated by the added image generating step includes a pixel area exceeding the threshold value. A determination step for determining whether or not a pattern sequence is determined based on an applied multiplication coefficient of a multiplication image that is a source of the added image when it is determined that there is a pixel region that exceeds a threshold value The information display method according to claim 7, further comprising a sequence identification step.

According to the present invention, the target irradiated with the specific wavelength light is imaged at a predetermined frame rate, the region irradiated with the specific wavelength light is extracted from the plurality of captured frames, and the extracted specific wavelength is extracted. Display means for generating the predetermined pattern series from the modulation state for the light irradiation region, generating control information from the generated predetermined pattern series, and displaying a photographed frame based on the control information Control. Therefore, if the present invention is used, for example, in the place of presentation, a personal computer or the like can be remotely operated while continuing the presentation without resting on the way. Further, if the light having the specific wavelength is light having a wavelength outside the visible region, that is, light that is not visible to the human eye, the spot light is not irradiated to the target such as a screen. There is also no need to consciously align the cursor and the spot light when performing a moving operation of.
Alternatively, the projection means can be, for example, a projector, and in this case, the projector can be remotely operated.
Alternatively, a plurality of control information can be selectively generated, and various control signals necessary for, for example, a personal computer or a projector can be generated.

  Embodiments of the present invention will be described below with reference to the drawings. It should be noted that the specific details or examples in the following description and the illustrations of numerical values, character strings, and other symbols are only for reference in order to clarify the idea of the present invention, and the present invention may be used in whole or in part. Obviously, the idea of the invention is not limited. In addition, a well-known technique, a well-known procedure, a well-known architecture, a well-known circuit configuration, and the like (hereinafter, “well-known matter”) are not described in detail, but this is also to simplify the description. Not all or part of the matter is intentionally excluded. Such well-known matters are known to those skilled in the art at the time of filing of the present invention, and are naturally included in the following description.

<Usage example>
FIG. 1 is a usage example diagram of this embodiment. Although an example of use in a presentation is shown here, this is only one example that can be used.
In this figure, the light emitting unit 2 is held in the hand of the user 1, and the light spot 4 formed by the spot light 3 irradiated from the light emitting unit 2 is projected at an arbitrary position on the screen 5, Further, a rectangular area (hereinafter referred to as a projection area 8) by the enlarged projection light 7 irradiated from the projector 6 is also displayed on the screen 5. If necessary, screen information 12 including a cursor 11 from the personal computer 13 is input to the projector 6 via the cable 9. The projector 6 is displayed on the screen 14 of the personal computer 13. The screen information 12 including the same cursor 11 as the existing cursor 11 ′ is projected on the screen 5 on the enlarged projection light 7. However, as will be described later, the spot light 3 is light having a wavelength outside the visible region, and the light spot 4 is invisible to the human eye. This light spot 4 can be understood to be “projected” for the first time when it is imaged by an imaging device 15 described later.

  A photographing device 15 such as a video camera or a digital camera is connected to the personal computer 13 via a cable 12. The photographing range 16 of the photographing device 15 is a size that covers at least the entire screen 5, and preferably a size that covers the entire projection region 8 of the projector 6, but strictness is not required. It does not have to deviate significantly from the screen 5 or the projection area 8 such as greatly deviating from the field of view of the user 1, and may be an approximate position and size.

  Note that the shooting conditions (focal length, zoom magnification, aperture, etc.) of the shooting device 15 are optimal in consideration of the distance from the subject (screen 5), the brightness of the place, etc., as in normal camera shooting. Of course, it should be set.

  Here, before starting detailed description, the outline | summary is demonstrated about operation peculiar to this embodiment in this usage example. First, the user 1 performs unfolding of the screen 5, installation of the projector 6 and the photographing device 15 and necessary adjustments, etc. with his / her own hand (or with the help of a third party), and then the personal computer 13 and those The imaging device 15 and the projector 6 are connected by cables 9 and 12. Then, the personal computer 13 is activated to display the presentation material, and then prepared so that the light emitting unit 1 can be held immediately, and stands by until the presentation start time.

  Needless to say, in a venue where these facilities (screen 5, projector 6, personal computer 13, photographing device 15, etc.) are permanently installed, the user 1 simply removes the removable media in which the presentation material is stored in the personal computer 13. After making a slight adjustment of the device, the light emitting unit 1 is prepared so that it can be held immediately, and it is sufficient to stand by until the presentation start time.

  When the presentation is started, the user 1 proceeds with the presentation while operating the personal computer 13 according to a scenario prepared in advance. During this period, in the present embodiment, the personal computer 13 (and a mouse attached thereto) There is no need to touch any existing input device such as a keyboard.

  As will be described below, the light emitting unit 2 is provided with a number of buttons necessary for existing mouse operations. By appropriately pressing these buttons, the mouse cursor can be moved, left and right clicked, This is because the drag operation can be performed remotely.

  These operation “commands” (replace existing mouse commands) are finally made by the photographing device 15. The imaging device 15 continuously shoots images in the shooting range 16 at a predetermined frame rate (shooting interval: typically 30 frames per second), processes the frame images, and transmits light into the shooting range 16. When the point 4 (the light spot 4 formed by the spot light 3 irradiated from the light emitting unit 2) is detected, the blinking pattern of the light spot 4 is specified, a command corresponding to the pattern is generated, and the personal computer 13 is generated. Send it out.

  Therefore, as can be understood from this outline description, in this embodiment, a plurality of mouse commands (in this embodiment, cursor movement, 4 commands (for example, left click, right click, and drag) can be generated, and the personal computer 13 can be remotely operated while giving a presentation. Therefore, not the “partial” problem as in the second prior art can be achieved, but the first problem can be completely achieved. Moreover, since light is used, there is no inconvenience peculiar to radio waves as in the first prior art.

Next, detailed description of the present embodiment will be started.
<Configuration and operation of light emitting unit 2>
FIG. 2 is an external view of the light emitting unit 2 and an enlarged view of the switch. In this figure, the light emitting unit 2 includes a case 20 having a shape suitable for hand-holding (here, a cylindrical shape, but this is an example), a cap 21 capable of closing the bottom of the case 20, and the cap 21. A battery 22 to be removed and inserted into the case 20, an electronic unit 23 incorporated in the case 20, an optical unit 24 attached to the tip of the case 20 so as to allow a slight amount of degeneration adjustment, A light emitting unit 25 built in the optical unit 24 and a power switch 26 and an operation switch 27 provided at a position where the case 20 is easy to operate are provided.

  Here, the light emitting unit 25 includes a light emitting element 25a that generates light having a wavelength outside the visible region, that is, light invisible to human eyes. As the light emitting element 25a, a light emitting element having a sufficient amount of light (high light power) is used. When the light power of one light emitting element 25a is insufficient, a plurality of light emitting elements 25a may be arranged as illustrated.

  An important point equal to or more than the optical power is the response speed of the light emitting element 25a. Although it is desirable to select the one that is as fast as possible, if it is difficult, it is necessary to select one that can perform the pattern blinking described later with at least sufficient resolution. This is because if the response speed is slow, only slow blinking can be performed in accordance with the response speed, and the amount of pattern information is reduced accordingly. Alternatively, it takes a long time to send the required amount of information. In this respect, an LED (Light Emitting Diode) has a response speed much faster than that of a filament light emitting element, and an element having sufficient power outside the visible region is available at a low cost. Suitable for the light emitting element 25a. However, the filament light-emitting element is not excluded from the idea of the invention. This is because the light emitting element 25a of the present embodiment can be used without any inconvenience as long as it generates light having a sufficient power outside the visible region and satisfies a necessary and sufficient response speed.

  The above “light having a wavelength outside the visible region” must be light in a wavelength region that can be imaged by the imaging device 15. In this respect, many of the existing video cameras and digital cameras have an imaging device that has imaging sensitivity up to the “infrared region” in addition to the visible region, so It is realistic to regard “light having a wavelength of” as infrared light. Therefore, in the following description, the “light having a wavelength outside the visible region” is referred to as “infrared light”. However, the invention is not limited to this (infrared light). Any light in a wavelength region that can be photographed by the photographing device 15 and is invisible to human eyes may be used.

  The optical unit 24 attached to the tip of the case 20 enables a slight amount of degeneracy adjustment as described above. This is because the spread (α) of the spot light 3 can be adjusted with a predetermined width by adjusting the distance between the internal lens 24a and the light emitting element 25a. Thereby, the diameter of the light spot 4 projected on the screen 5 can be increased or decreased and can be freely adjusted.

  However, since the spot light 3 from the light emitting element 25a that is the origin of the light spot 4 is infrared light that is invisible to human eyes, the size of the light spot 4 cannot be changed while visually observing. As a countermeasure for this, for example, a distance scale 20a previously determined from the relationship between the distance and the size of the light spot 4 is engraved or printed on the surface of the case 20, and the visual distance to the screen 5 and the distance scale are used when used. The amount of degeneration of the optical unit 24 may be adjusted to match the scale of 20a.

  Note that the degeneracy adjustment mechanism of the optical unit 24 may be a simple slide type, but is preferably a screw type in order to prevent inadvertent sliding. That is, it is preferable that a spiral thread groove is cut on both contact surfaces of the optical unit 24 and the case 20 so that the amount of degeneration can be adjusted by rotating the optical unit 24. In addition to preventing inadvertent movement of the optical unit 24, it is possible to finely adjust the amount of degeneration.

  The operation switch 27 has a seesaw-type push button 27a. When the front of the push button 27a is referred to as "A" and the rear thereof is referred to as "B", neither A nor B is pressed ( b) Three operating states can be defined: a state in which A is pressed (c), and a state in which B is pressed (d). In addition to the above, even in the state (c) in which A is pressed, two states of “short press” and “long press” can be defined depending on the length of time.

  Hereinafter, a state in which neither A nor B is pressed is assigned to the “cursor movement operation”, a state in which A is pressed shortly is assigned to the “left click operation”, and a state in which A is pressed long is referred to as “ Assume that a state where B is pressed is assigned to a “right click operation”. These assignments are only examples for explanation, and are not the result of a thorough examination of operability. The mouse operation includes not only the above-described operation but also a double click operation, for example. If a double-click operation is also added, or if it is replaced with one of the above-described exemplary operations, the switch structure may be changed, or the operation method may be changed.

  FIG. 3 is an electrical configuration diagram of the light emitting unit 2. In this figure, the operation switch 27 includes a push button 27a that can be swung by a predetermined amount about a fulcrum 27b, and a first micro switch 27c provided immediately below the push button 27a ("A"). And a second micro switch 27d provided immediately below the push button 27a ("B"). The voltage Vcc from the battery 22 is applied to one terminal of the first micro switch 27c via the power switch 26. Similarly, the one terminal of the second micro switch 27d is also connected to the one terminal of the second micro switch 27d from the battery 22 via the power switch 26. Voltage Vcc is applied.

  Signals S1 and S2 are taken out from the other terminals of the first and second microswitches 27c and 27d, respectively. These signals S1 and S2 are normally dropped (pulled down) through resistors R1 and R2 to a ground potential (0 V, hereinafter referred to as a low level). When the push button 27a is pressed down ("A" or "B"), the potential (Vcc) of one terminal is taken out via the ON contact. In this case, the signals S1 and S2 change to the potential of Vcc (hereinafter referred to as high level).

  Although the electronic unit 23 is not particularly limited thereto, it is configured by, for example, a one-chip MPU (Micro Processing Unit) including a CPU 23a, a ROM 23b, a RAM 23c, an input / output unit 23d, and the like. The ROM 23b stores a control program necessary for the operation of the light emitting unit 2. The CPU 23a loads the control program into the RAM 23c and executes it, thereby reading the signals S1 and S2 from the operation switch 27. Then, the state determination processing of the signals S1 and S2, the pattern sequence selection processing according to the determination result, the control waveform C1 generation processing corresponding to the selected pattern sequence, and the like are performed.

  The drive unit 28 receives the control waveform C1 from the electronic unit 23, creates a drive voltage P1 obtained by modulating the voltage Vcc of the battery 22 with the control waveform C1, and drives the light emitting element 25a of the light emitting unit 25 at high speed with the drive voltage P1. To do.

  FIG. 4 is a flowchart of a control program executed by the CPU 23a of the electronic unit 23. When the power switch 26 of the light emitting unit 2 is turned on, the power supply voltage (Vcc) from the battery 22 is supplied to the electronic unit 23. In response to this power supply, the CPU 23a performs a predetermined error check, initial setting processing, and the like, then loads a control program from the ROM 23b to the RAM 23c, and starts executing the program.

  In this control program, first, the states of the signals S1 and S2 from the operation switch 27 are determined (step S10). Here, it is determined whether one of the front “A” or the rear “B” of the push button 27a of the operation switch 27 is pressed. As described above, the signals S1 and S2 from the operation switch 27 are normally at the low level, but when the front “A” of the push button 27a is pressed, the signal S1 changes from the low level to the high level. When the rear “B” of the push button 27a is pressed, the signal S2 changes from the low level to the high level. In this step S10, it is determined whether one of the signals S1 and S2 is the high level. Will be.

  If the determination result in step S10 is negative (“NO”), that is, if neither the front “A” or the rear “B” of the push button 27a of the operation switch 27 is pressed, the above definition is used. In this case, the cursor movement process (step S11) is executed.

  On the other hand, if the determination result in step S10 is affirmative (“YES”), that is, if either the front “A” or the rear “B” of the push button 27a of the operation switch 27 is pressed, from the above definition. First, the signal S2 is at a high level in order to determine that the operation is one of operations other than the “cursor movement operation” (“left click operation”, “drag operation”, or “right click operation”). It is determined whether or not (step S12).

  When the determination result is affirmative, that is, when the rear “B” of the push button 27a of the operation switch 27 is pressed, it is determined as a “right click operation” from the above definition. Right click processing (step S13) is executed.

  Alternatively, when the determination result in step S12 is negative, that is, when the front “A” of the push button 27a of the operation switch 27 is pressed, either “left click operation” or “drag operation” is determined from the above definition. In order to separate these, first, the high level period is measured and the measurement result is set in the variable ST (step S14), and then whether ST <T1 is satisfied. Determination is made (step S15). Here, T1 is a short reference time of about several tens of milliseconds. It is desirable that the reference time T1 can be adjusted according to the habit of the user 1 (early click operation).

  In the case of ST <T1, that is, when the front “A” of the push button 27a of the operation switch 27 is pressed for a short time, it is determined from the above definition that it is “left click operation”. (Step S16) is executed. Alternatively, if ST <T1, that is, if the front “A” of the push button 27a of the operation switch 27 is pressed for a long time, it is determined as a “drag operation” from the above definition, and in this case, a drag process is performed. (Step S17) is executed.

Next, cursor movement processing, left click processing, right click processing, and drag processing will be described.
<Cursor movement processing>
FIG. 5 is a diagram showing a schematic flowchart of cursor movement processing. In this flowchart, an n-bit pattern sequence (hereinafter referred to as a first pattern sequence, here referred to as a “1010” pattern sequence where n = 4) assigned to the cursor movement operation in advance is read from the ROM 23b (step S11a), then, one unit of control waveform C1 based on the first pattern series is generated (step S11b), and finally, the control waveform C1 is output to the drive unit 28 (step S11c), and then, as shown in FIG. A series of processing of returning to the control program is executed.

  Here, the logic 1 section (the high level section of the control waveform C1) of the first pattern series is the “lighting” section of the light emitting unit 25, and the logic 0 section (control waveform C1 of the first pattern series). (Low level section) is a “light-off” section of the light emitting unit 25.

  Note that “lit” and “off” generally refer to a lighted state and a lighted state as the words indicate, but the present specification is not limited to this interpretation. It may be interpreted as “high luminance” and “low luminance”. In other words, “lighting” may be referred to as brightly shining, and “lighting off” may be referred to as darkly shining.

  The drive unit 28 generates a drive voltage P1 that repeats a high potential (Vcc) and a low potential (0 V) in a cycle synchronized with the control waveform C1, and drives the light emitting unit 25 with the drive voltage P1. As a result, the light emitting element 25a of the light emitting unit 25 repeats the operation of turning on in the logic 1 section of the first pattern series and turning off in the logic 0 section.

  As described above, by executing the cursor movement process, the light emitting element 25a of the light emitting unit 25 is changed in the order of “ON” → “OFF” → “ON” → “OFF” according to the first pattern series (1010). Can be flashed. Note that this blinking does not end halfway. Each time the cursor movement process is executed, one unit of flashing ("ON" → "OFF" → "ON" → "OFF") is always performed and completed. However, since this cursor movement process is repeatedly executed while no push button 27a is being pressed, in practice, “lighting” → “lighting out” → “lighting” → “lighting out” is set as one unit. The unit is repeated endlessly.

The length of each section of the control waveform C1 is represented by T _SLOT . Since T _SLOT is the same length in each section and the number of sections is equal to the number n of bits of the first pattern series, the total length of one unit control waveform C1 is “n × T _SLOT ”. T _SLOT is set in advance so as to be equal to twice the length (T _FRAME ) of one frame of the photographing apparatus 15 (T _SLOT = 2 × T _FRAME ). This (T _SLOT = 2 × T _FRAME ) is the same in other processes (right-click process, left-click process, and drag process).

<Right click processing>
FIG. 6 is a schematic flowchart of the right click process. In this flowchart, an n-bit pattern sequence (hereinafter referred to as a second pattern sequence, here referred to as “1110” pattern sequence where n = 4) assigned to the right click operation in advance is read from the ROM 23b (step S13a), then, one unit of control waveform C1 based on the second pattern series is generated (step S13b). Finally, the control waveform C1 is output to the drive unit 28 (step S13c), and then the control of FIG. A series of processing of returning to the program is executed.

  In this right click process, a mechanism for repeating the output of the control waveform C1 a predetermined number of times is put in place assuming that an instantaneous right click operation is performed (step S13d). If an instantaneous right-click operation is performed, the flashing of the light emitting unit 25 may end in one unit of “lighting” → “lighting” → “lighting out” → “lighting off”. This is because there is no way to deal with misrecognition of the pattern (misinterpretation of lighting as unlit or misidentification of unlit as lighting). If the output of the control waveform C1 is repeated a predetermined number of times (a predetermined number of units), even if an instantaneous right-click operation is performed, “lighting” → “lighting” → “ This is because the unit of “off” → “off” is repeated, so that even if a certain unit is mistakenly recognized as being turned off, or even if it is mistaken as being turned off, it can be correctly recognized in the next unit.

  Similar to the cursor movement process, the logic 1 section of the second pattern series (the high level section of the control waveform C1) is the “lighting” section of the light emitting unit 25, and the logic 0 of the second pattern series. This section (the low level section of the control waveform C1) is the “light-off” section of the light emitting unit 25. The drive unit 28 generates a drive voltage P1 that repeats a high potential (Vcc) and a low potential (0 V) in a cycle synchronized with the control waveform C1, and drives the light emitting unit 25 with the drive voltage P1. As a result, the light emitting element 25a of the light emitting unit 25 repeats the operation of turning on in the logic 1 section of the second pattern series and turning off in the logic 0 section.

  As described above, by executing the right click process, the light emitting element 25a of the light emitting unit 25 is changed in the order of “ON” → “ON” → “ON” → “OFF” according to the second pattern series (1110). Blink (note that the blink pattern is different from the cursor movement process). Note that, similarly to the cursor movement process described above, this blinking does not end midway. Every time the right-click process is executed, one unit of flashing ("ON" → "ON" → "ON" → "OFF") is always performed and completed. However, this right-click process is not completed with only one unit. It is always completed by blinking several units (refer to the predetermined times in step S13d). For example, if “predetermined times” is 3, the unit of “lighting” → “lighting” → “lighting” → “lighting off” is repeated three units, and then the process is completed.

<Left click process>
FIG. 7 is a diagram showing a schematic flowchart of the left click process. In this flowchart, an n-bit pattern sequence (hereinafter referred to as a third pattern sequence, here referred to as “1100” pattern sequence where n = 4) assigned to the left click operation in advance is read from the ROM 23b (step S16a) Next, one unit of control waveform C1 based on the third pattern series is generated (step S16b), and the control waveform C1 is output to the drive unit 28 (step S16c), and then the control program of FIG. A series of processing of returning is executed.

  Also in this left click process, a mechanism for repeating the output of the control waveform C1 a predetermined number of times is put in the same manner as in the previous right click process (step S16d).

  Similar to the cursor movement process and the right click process, the logic 1 section of the third pattern series (the high level section of the control waveform C1) is the “lighting” section of the light emitting unit 25, and the third pattern The section of the logical 0 of the series (low level section of the control waveform C1) is the “light-off” section of the light emitting unit 25. The drive unit 28 generates a drive voltage P1 that repeats a high potential (Vcc) and a low potential (0 V) in a cycle synchronized with the control waveform C1, and drives the light emitting unit 25 with the drive voltage P1. As a result, the light emitting element 25a of the light emitting unit 25 repeats the operation of turning on in the logic 1 section of the third pattern series and turning off in the logic 0 section.

  In this way, by performing the left click process, the light emitting element 25a of the light emitting unit 25 is changed in the order of “ON” → “ON” → “OFF” → “OFF” according to the third pattern series (1100). Blink (note that the blink pattern is different from the cursor movement process and the right-click process). Note that, similarly to the cursor movement process and the right click process described above, this blinking does not end midway. Every time the left click process is executed, one unit of flashing ("lit" → "lit" → "extinguished" → "extinguished") is always performed and completed. However, this left click process is not completed with only one unit, as in the previous right click process. It is always completed by blinking several units (refer to the predetermined times in step S16d). For example, if “predetermined times” is 3, the unit of “lighting” → “lighting” → “lighting off” → “lighting off” is repeated three units, and then the process is completed.

<Drag processing>
FIG. 8 is a diagram showing a schematic flowchart of the drag process. In this flowchart, an n-bit pattern sequence (hereinafter referred to as a fourth pattern sequence, here referred to as a “1000” pattern sequence where n = 4) assigned to the drag operation is read from the ROM 23b (step S17a). Then, one unit of control waveform C1 based on the fourth pattern series is generated (step S17b). Finally, the control waveform C1 is output to the drive unit 28 (step S17c), and then the control program of FIG. A series of processes of returning to the process is executed.

  In this drag process, the mechanism in the previous right click process or left click process (a mechanism for repeatedly outputting the control signal C1 several times: see step S13c in FIG. 6 or step S16c in FIG. 7) is not necessary. This is because the drag operation is performed by “long pressing” of the push button 27a.

  Similar to the cursor movement process, right click process, and left click process, the logic 1 section of the fourth pattern series (the high level section of the control waveform C1) is the “lighting” section of the light emitting unit 25, and The logic 0 section (low level section of the control waveform C1) of the fourth pattern series is the “light-off” section of the light emitting unit 25. The drive unit 28 generates a drive voltage P1 that repeats a high potential (Vcc) and a low potential (0 V) in a cycle synchronized with the control waveform C1, and drives the light emitting unit 25 with the drive voltage P1. As a result, the light emitting element 25a of the light emitting unit 25 repeats the operation of turning on in the logic 1 section of the fourth pattern series and turning off in the logic 0 section.

  As described above, by executing the drag process, the light emitting element 25a of the light emitting unit 25 blinks in the order of “ON” → “OFF” → “OFF” → “OFF” in accordance with the fourth pattern series (1000). (Note that the blinking pattern is different from the cursor movement process, right click process, and left click process). Note that, similarly to the cursor movement process, the right click process, and the left click process, the blinking does not end in the middle. Each time the drag process is executed once, one unit of flashing ("ON" → "OFF" → "OFF" → "OFF") is always performed and completed. However, since the drag process is repeatedly executed while the push button 27a is “long pressed”, in practice, during the long press, “ON” → “OFF” → “OFF” → “OFF” "Is a unit, and that unit is repeated endlessly.

<Summary of light emitting unit 2>
As described above, according to the light emitting unit 2, the first pattern series corresponding to the “cursor moving operation” when the power switch 26 is kept ON (if the operation button 27 is not pressed). According to (1010), it is possible to emit the spot light 3 that repeats “ON” → “OFF” → “ON” → “OFF”. In addition, by pressing “B” behind the operation button 27, a spot that repeats “lighting” → “lighting” → “lighting” → “lighting off” in accordance with the second pattern sequence (1110) corresponding to “right click operation”. Light 3 can be fired. In addition, by briefly pressing “A” in front of the operation button 27, “ON” → “ON” → “OFF” → “OFF” is repeated according to the third pattern sequence (1100) corresponding to “left click operation”. Spot light 3 can be emitted. In addition, by long pressing “A” in front of the operation button 27, a spot that repeats “ON” → “OFF” → “OFF” → “OFF” according to the third pattern sequence (1000) corresponding to “drag operation”. Light 3 can be fired.

Here, an operation example of the light emitting unit 2 will be described.
FIG. 9 is a diagram illustrating an operation example of the light emitting unit 2. As shown in this figure, the user 1 first performs “cursor movement operation” continuously for 5 minutes, then performs “left click operation” once and “right click operation” once, then 10 Consider an operation example in which a “drag operation” is performed continuously for a second and then the operation returns to the “cursor movement operation” again.

  The cursor movement process shown in FIG. 5 is repeatedly executed for 5 minutes during the first “cursor movement operation”. Therefore, the spot light 3 repeats the blinking pattern of the first pattern series (1010) for 5 minutes. That is, the spot light 3 repeats “ON” → “OFF” → “ON” → “OFF”.

  When 5 minutes have elapsed, the “right click operation” is performed once, and the right click process of FIG. 6 is executed. As a result, the spot light 3 blinks in the second pattern series (1110). The number of executions of this right click process changes according to the pressing time of the rear “B” of the push button 27a. The number of executions increases if the pressing time is long, and decreases if the pressing time is short. Even when the pressing time is extremely short (instantaneous), since the step S13d of FIG. 6 is included, the blinking pattern is repeated at least a predetermined number of times. For example, if “predetermined times” is 3, even if an instantaneous right-click operation is performed, the blinking pattern of the second pattern series (1110) is repeated at least 3 times (3 units). That is, in this case, the spot light 3 is “lighted” → “lighted” → “lighted” → “lighted off” → “lighted” → “lighted” → “lighted” → “lighted off” → “lighted” → “lighted” → Flashes in the order of “ON” → “OFF”.

  Next, the “left click operation” is performed once, and the left click process of FIG. 7 is executed. As a result, the spot light 3 blinks in the third pattern series (1100). This left click process is executed once. This is because the front “A” of the push button 27a is pushed only for a short time (T1 or less). In this way, even if the left click process is executed only once, since the step S16d of FIG. 7 is included, the blinking pattern can be repeated at least a predetermined number of times. For example, if “predetermined times” is 3, the blinking pattern of the third pattern series (1100) is repeated 3 times (3 units). In other words, the spot light 3 is “ON” → “ON” → “OFF” → “OFF” → “ON” → “ON” → “OFF” → “OFF” → “ON” → “ON” → “OFF” → Flashes in the order of “off”.

  Finally, the drag process of FIG. 8 is repeatedly executed for 10 seconds during which the “drag operation” is performed. Therefore, the spot light 3 repeats the blinking pattern of the fourth pattern series (1000) for 10 seconds. That is, the spot light 3 repeats “ON” → “OFF” → “OFF” → “OFF”.

  FIG. 10 is a diagram in which the blinking patterns of the spot light 3 in the case of the operation example are arranged in time series. The spot light 3 is a pattern sequence (first pattern sequence: 1010, second pattern sequence: 1110, third pattern sequence: 1100, first pattern sequence of cursor movement operation, right click operation, left click operation and drag operation. The fourth pattern series: 1000) is repeated as one unit. For example, in the case of the illustrated operation example, first, blinking of 1010 is repeated by the number of units corresponding to the time (5 minutes), then 1110 is repeated at least 3 units (when predetermined times = 3), and then 1100 is repeated at least 3 units (when the predetermined number of times is set to 3), and finally, a blinking pattern in which 1000 blinking is repeated by the number of units corresponding to time (10 seconds).

  The imaging device 15 described below continuously captures a subject image including an image of the light spot 4 projected on the screen 5 by the spot light 3 that repeats such a specific blinking pattern at a predetermined frame rate. The unit of the pattern is detected from a plurality of frame images, the pattern series is specified, a mouse command corresponding to the specified pattern series is generated, and output to the personal computer 13.

<Configuration and Operation of Imaging Device 15>
Next, the configuration and operation of the photographing apparatus 15 will be specifically described. Although a digital camera will be described as an example here, the present invention is not limited to this. For example, a video camera may be used.
FIG. 11 is an external view of the photographing device 15. In this figure, the photographing apparatus 15 has a retractable lens barrel 31, a strobe light emission window 32, a finder front window 33 and the like disposed on the front surface of a box-shaped body 30, and a power switch 34 and a shutter on the upper surface of the body 30. A button 35 and the like are arranged, and further, a finder rear window 36, a mode switch 37, a zoom switch 38, a MENU button 39, an up / down / left / right movement button 40, a SET button 41, a DISP button 42, a liquid crystal monitor 43, etc. In addition, a hole 44 a for attaching the tripod 44 to the bottom surface of the body 30 is provided.

  FIG. 12 is an internal block diagram of the photographing apparatus 15. In this figure, the photographing device 15 can be classified into an imaging system 45, a control system 46, an image file storage system 47, a display system 48, an operation system 49, an external interface system 50, and the like according to function.

  Explaining for each of these systems, the image pickup system 45 picks up a photographic lens group 51 with a zoom function and an autofocus function housed in a lens barrel 31, and a subject image that has passed through the photographic lens group 51. An electronic image pickup unit 52 composed of a CCD (Charge Coupled Devices), a CMOS (Complementary Metal Oxide Semiconductor), or the like, which is converted into an image at a frame rate (hereinafter referred to as a frame image) and output, and a frame image from the electronic image pickup unit 52 A video processing unit 53 for performing required image processing, and an image buffer memory 54 including at least 1 + 8 frame memories (F0 to F8) for temporarily storing frame images after image processing. A focus drive unit 55 for driving a focus mechanism (not shown) of the lens barrel 31 and a zoom drive for driving the zoom mechanism. 56, a strobe light emission unit 57 provided in the strobe light emission window 32 on the front surface of the body, a strobe drive unit 58 for driving the strobe light emission unit 57, and each of these units (electronic imaging unit 52, video processing unit 53, focus) A driving control unit 59 for controlling the driving unit 55, the zoom driving unit 56, and the strobe driving unit 58).

  Here, the frame memory F0 is a frame memory for normal photographing, but the other eight frame memories F1 to F8 are FIFO type memories for use in pattern reading processing to be described later. A FIFO (First-In First-Out) type memory is a memory configuration in which data is extracted in the order in which it was stored. The most recently stored data is retrieved last. Each of the eight frame memories F1 to F8 has a capacity for one frame image, and transfers frame images sequentially input in time series in the order of F1, F2, F3,... F8.

  The control system 46 controls each of the above systems to centrally control the operation of the photographing apparatus 15, a program memory 61 that stores various programs and data necessary for the operation of the CPU 60, and an image. And a built-in flash memory 62 for data storage.

The image file storage system 47 includes an interface unit 63 and an external flash memory 64 that is detachably connected to the interface unit 63.
The display system 48 includes a display control unit 66 including a video memory (VRAM 65) that temporarily holds display data output appropriately from the CPU 60, and a liquid crystal monitor 43 that displays an output signal of the display control unit 66.

  The operation system 49 includes various operation buttons provided on each part of the body 30, that is, a shutter button 35, a mode switch 37, a zoom switch 38, a MENU button 39, an up / down / left / right movement button 40, a SET button 41, and a DISP button. 42 and an input circuit 68 for inputting an operation signal from the operation input unit 67 to the CPU 60.

  The external interface system 50 includes a USB (Universal Serial Bus) terminal 69 for connecting to an external device such as the personal computer 13 (see FIG. 1), and an external device such as the personal computer 13 and the CPU 60 via the USB terminal 69. And a USB interface unit 70 for controlling the exchange of data with each other. Needless to say, “USB” is merely an example, and other general-purpose interface standards may be used. For example, general-purpose interface standards such as IEEE 1394, wired LAN, wireless LAN, Bluetooth, and optical fiber (Fibre Channel) may be used.

  FIG. 13 is a diagram illustrating an operation flowchart of the photographing apparatus 15. In this figure, when the power switch 34 is turned on, first, the current switch position of the mode switch 37 is checked to determine whether the mode is "shooting mode" or "reproduction mode" (step S20). If the determination result is “reproduction mode”, a required reproduction mode process is executed (step S21).

  The reproduction mode processing is a mode for displaying and confirming a photographed image on the liquid crystal monitor 43 when the photographing apparatus 15 is used as a normal digital camera. That is, in this playback mode, a list of captured images (images stored in the internal flash memory or the external flash memory 64) (reduced image list or file name list display) is displayed on the liquid crystal monitor 43, and the user's An arbitrary image is read from the built-in flash memory or the external flash memory 64 in accordance with a selection operation (operation to select with the up / down / left / right movement button 40 and confirm with the SET button 41), and decompress and display on the liquid crystal monitor 43.

  On the other hand, if “shooting mode” is determined in step S20, it is next determined whether or not the MENU button 39 is pressed (step S22). If the MENU button 39 is not pressed, a required shooting mode process is executed (step S22). Step S23).

  The photographing mode process is a mode when the photographing device 15 is used as a normal digital camera for photographing. This mode is also described briefly because it is not directly related to the present invention. In this shooting mode, a frame image sequentially output from the electronic imaging unit 52 at a predetermined frame rate is stored in one image memory buffer 54. The frame memory F0 is written, the image of the frame memory F0 is transferred to the VRAM 65 of the display control unit 66, and a through image (image for composition confirmation) is displayed on the liquid crystal monitor 43 while waiting for the operation of the shutter button 35. In response to the operation of the shutter button 35, the frame image at that time is compressed and stored in the built-in flash memory 62 or the external flash memory 64.

  If it is determined in step S22 that the MENU button 39 has been pressed, a shooting mode MENU process is executed (step S24).

  FIG. 14 is a flowchart of the shooting mode MENU process. In this flowchart, the shooting mode menu screen (see FIG. 15) is displayed first (step S24a), and then it is determined whether any menu item on the shooting menu screen is selected (step S24b). . If no menu item is selected, the shooting mode menu screen is closed (step S24c), and the process returns to the flowchart of FIG. 13. However, when any menu item is selected, the selected menu item is selected. After branching determination is performed (step S24d) and processing corresponding to the selected menu item (for example, mouse processing described later: step S24e) is executed, the process returns to the flowchart of FIG.

  FIG. 15 is a diagram showing a shooting mode menu screen. In this figure, the shooting mode menu screen 71 is a menu screen of an arbitrary design having, for example, a shooting tab 72 and a setting tab 73, but at least in the lower menu screen 75 of the shooting mode menu 74, In addition to various shooting mode menus (eg, still image shooting mode menu 76 and moving image shooting mode menu 77) used for shooting, a menu 78 specific to the present embodiment (referred to as “mouse menu 78” in this specification) is provided. It is characterized in that it is displayed in a selectable manner.

  When the mouse menu 78 is selected, the photographing device 15 can be used in a special way different from normal photographing and reproduction. An example of special usage is the usage example of FIG. That is, the flashing pattern of the light spot 4 formed on the screen 5 by the spot light 3 emitted from the light emitting unit 2 is detected by the photographing device 15, and the pattern series is specified to “cursor movement”, “ A mouse command such as “right click”, “left click” or “drag” is generated and output to the personal computer 13.

  In the case of such special usage, the photographing device 15 behaves like a mouse when viewed from the personal computer 13.

  FIG. 16 is a diagram showing a flowchart of mouse processing (step S24e in FIG. 14). When this flowchart is started, first, the connection with the personal computer 13 is confirmed (steps S25 and S26). If not yet connected, a required message is displayed on the LCD monitor 43 to prompt connection (step S27), and if it is not connected even after a predetermined time (step S28), an incorrect menu selection has occurred. It is determined that the process has ended, and the process exits the flowchart and returns to the flowchart of FIG.

  On the other hand, if it is determined in step S26 that the personal computer 13 is connected, next, the required operation guidance is continuously displayed on the liquid crystal monitor 43 until the user's consent operation (for example, pressing the SET button 41 is performed) ( Step S29, Step S30). The required operation guidance is, for example, a storyboard or an explanatory text showing an example of use as shown in FIG. In addition, the operation guidance may include instructions on how to use the light emitting unit 2 and installation of devices such as the projector 6 and the screen 5.

  When the user performs an understanding operation, next, a process “pattern reading process” (step S31) unique to the present embodiment is executed. Before the details of the “pattern reading process” are described, the process is used in the usage example. The configuration and operation of the remaining devices (personal computer 13 and projector 6) among the various devices will be described.

FIG. 17 is a schematic block diagram of the electrical configuration of the personal computer 13 and the projector 6.
The personal computer 13 includes a CPU 79, a RAM 80, a storage device 81, a keyboard 82, a mouse 83, a video adapter 84, a VRAM 85, and a display device 86.

  The video adapter 84 generates a display video signal (RGB signal or NTSC signal) and outputs the generated video signal to the display device 86 including a liquid crystal panel and its drive circuit. The VRAM 85 generates a display signal generated by the video adapter 84. Store image data as needed. The video adapter 84 is provided with a video output terminal 87, and one end of the cable 9 is connected to the video terminal 87. The video adapter 84 displays an image for display according to a command from the CPU 79. Data can be output via the cable 9.

  A USB interface circuit (not shown) is connected to the CPU 79, and the cable 12 from the photographing device 15 is connected to the USB terminal 88 of the USB interface circuit. Then, any mouse command (“cursor movement command”, “right click command”, “left click command”, “drag command”) is sent from the photographing device 15 through the cable 12 at an arbitrary timing. Then, the CPU 79 interprets these commands and generates events (cursor movement event, right click event, left click event, drag event) similar to the commands generated by its own mouse 83.

  The projector 6 includes a CPU 89, a ROM 90, a RAM 91, an image input unit 92, a display unit 93 including an optical system, and an operation unit 94. A video input terminal 95 is connected to the image input unit 92, and the image input unit 92 receives a display signal for projection from an image input from the personal computer 13 via the video input terminal 95 and the cable 9. The display signal is generated and sent to the display unit 93, and the display unit 93 modulates high-intensity light such as a krypton lamp with the display signal and enlarges and projects it onto the screen 5 through an optical system such as a projection lens.

<Frame memory structure>
Next, the frame memories F1 to F8 will be described.
FIG. 18 is a conceptual structural diagram of the eight frame memories F1 to F8 referred to in the pattern reading process. In this figure, the eight frame memories F1 to F8 provided in the image buffer memory 54 have a FIFO structure as described above. In these frame memories F1 to F8, frame images created by the electronic imaging unit 52 and subjected to required signal processing by the video signal processing unit 53 are sequentially input at a predetermined frame period (T _FRAME ).

  Of the eight frame images (PC1 to PC8) captured in the frame memories F1 to F8, odd-numbered images (PC1, PC3, PC5, PC7) are read by the CPU 60, and a program (pattern reading) executed by the CPU 60 is read. Process). Four multipliers 96 to 99 and one adder 100 in the figure represent multiplication processing functions and addition processing functions virtually realized by this program. Note that ki_1 to ki_4 input to the four multipliers 96 to 99 are multiplication coefficients. A number from 1 to 14 is entered in “i” of ki (see FIG. 21). A number after the underscore (_) represents the number of digits of the multiplication coefficient. For example, as will be described later, since the multiplication coefficient k1 is (1, -1, 1, -1), k1_1 indicates the leftmost "1", k1_2 indicates the second "-1" from the left, k1_3 indicates the third “1” from the left, and k1_4 indicates the rightmost “−1”.

  The four images (PC1, PC3, PC5, and PC7) captured by the CPU 60 are multiplied by coefficients by adders 96 to 99, respectively, and then added by the adder 100. GΣ represents an image after addition processing (added image). These coefficient multiplication and addition processes are performed in order to eliminate unnecessary information such as background images and leave only information necessary for pattern detection. The principle and mechanism will be described in detail later. .

<Number of frame memories>
FIG. 19 is an operation conceptual diagram of the eight frame memories F1 to F8. The validity of the number of frame memories F1 to F8 (eight) will be described with reference to FIG. However, in this figure, the in-frame operation (exposure, readout, and transfer) of the image capturing device 15 is ignored, and the image captured in _TFRAME is simply stored in the frame memory as it is. In addition, a large number of cells (thick cells and fine cells) in the figure represent eight frame memories F1 to F8, respectively. The reason why a large number of cells are arranged will become clear from the following explanation.

The light emitting unit 2 described above emits the spot light 3 that blinks in any of the first pattern series to the fourth pattern series. These pattern sequences include n-bit information. That is, the first pattern series includes “1010”, the second pattern series includes “1110”, the third pattern series includes “1100”, and the fourth pattern series includes “1000”. Therefore, the value of n in this embodiment is “4”. There are theoretically 2 ⁿ types of pattern series that can be used in the light emitting unit 2. Since n = 4 in the present embodiment, 2 ⁴ = 16 pattern sequences can be used at the maximum. However, only four of them (first to fourth pattern series) are actually used. If four are acceptable (that is, if future extensibility is not taken into consideration), n = 2 is sufficient. However, in this embodiment, four types of mouse commands (cursor movement, right click, Future expandability based on the fact that it is not limited to (left click and drag) (for example, there is also a double click, etc.) and can be applied to other than mouse operation (for example, remote control of the projector 6). N = 4, leaving room for.

  Hereinafter, description will be made assuming that n = 4. The number of frame memories is 8 from F1 to F8. This corresponds to twice the value of n. In FIG. 19, a pattern series “1010” is now considered. “1” turns on the spot light 3 and “0” turns it off. For convenience, information (“1”) of the first section of the pattern series is referred to as B1. Similarly, the second section information (“0”) is referred to as B2, the third section information (“1”) is referred to as B3, and the last section information (“0”) is referred to as B4. To do. Note that Bx is information (“1”) of the first section of the pattern unit of the next unit.

As described above, the section length T _SLOT of the pattern series is adjusted to be exactly twice the frame period T _FRAME of the image capturing device 15. Since the light emitting unit 2 has only the function of transmitting information to the outside (emitting the spot light 3), it can be said to be a unidirectional optical communication device. Therefore, T _SLOT cannot be synchronized with T _FRAME at the phase level. (A) to (g) in the figure schematically illustrate the phase relationship of the eight frame memories F1 to F8 for each section of the pattern series. (A) is an ideal state in which the phases match, and the other (A) to (K) are examples of states in which the phases are shifted. Because of asynchrony, such phase shifts can occur with high probability.

  Now, in the case of (a), B1 to B4 correctly enter each of the eight frame memories F1 to F8. That is, B1 enters F1 and F2, B2 enters F3 and F4, B3 enters F5 and F6, and B4 enters F7 and F8. Refer to every other frame memory (odd or even) By doing so, four frame images including the pattern series “1010” (B1 to B4) can be taken out without any trouble.

  On the other hand, in the cases where the phases are shifted (i) to (g), the information on the adjacent section of the pattern sequence “1010” is stored in the even-numbered frame memory (F2, F4, F6, F8). Or, since the information (B1 + B2, B2 + B3, B3 + B4) of the transition point is included, these four frames including the pattern sequence “1010” (B1 to B4) from the even-numbered frame memories (F2, F4, F6, F8). The frame image cannot be extracted.

  However, as is apparent from the figure, the odd-numbered frame memories (F1, F3, F5, F7) indicated by bold lines always include the pattern series “1010” (B1 to B4) regardless of the phase shift. Contains four frame images. For this reason, not only the phase matching state of (A) but also the phase shift states of (A) to (K) are referred to the odd-numbered frame memories (F1, F3, F5, F7). Thus, four frame images including the pattern series “1010” (B1 to B4) can be taken out without any trouble.

  This is because the number of frame memories is 2n, and each frame image captured in time series at a predetermined frame rate is sequentially stored in the 2n frame memories F1 to F8. is there.

<Pattern reading process>
Next, details of the pattern reading process will be described.
FIG. 20 is a diagram illustrating a flowchart of the pattern reading process. This flowchart is a subroutine program of step S31 of the “mouse processing” program (see FIG. 16) described above.
When this flowchart chart is started, first, four images PC1, PC5, PC5, and PC7 (see FIG. 18) are extracted from the odd-numbered frame memories (F1, F3, F5, and F7) of the image buffer memory 54, and each of them is a variable. Set to G1 to G4 (step S40). Here, G1 = F1, G2 = F3, G3 = F5, and G4 = F7.

  Next, pattern detection processing is executed (step S41). Details of this pattern detection processing will be described later. Next, it is determined whether or not a predetermined value (in this case, “0000”) indicating pattern non-detection is set by looking at the result (detection pattern) of the pattern detection process (step S42). If the detection pattern is not 0000, it is determined that a valid pattern (any one of the first to fourth pattern series) has been detected. After passing through the determination of “Different from previous detection pattern?” (Step S43), a mouse command output process is executed (step S44). The details of the mouse command output process and the reason for performing the determination in step S43 ("different from the previous detection pattern?") Will be described in detail later.

  When the mouse command output process is completed, the determination of the end of the use of the “mouse”, that is, “use of the photographing device 15 in a special way different from normal photographing and reproduction” described above ends. If it is determined whether or not the user 1 has performed any end operation (for example, a pressing operation of the MENU button 39 of the photographing apparatus 15), the process exits this flowchart, and the MENU process of FIG. 14 is performed. Return.

<Pattern detection process>
Next, details of the pattern detection processing in step S41 will be described.
First, a pattern sequence to be detected, a detection range thereof, and a multiplication coefficient applied to each detection range will be described.
FIG. 21 is a diagram illustrating a pattern series to be detected, a detection range thereof, and a multiplication coefficient applied to each detection range. In this figure, four types of pattern series of the light emitting unit 2 are shown at the left end. In order from the top, “1010” is the first pattern series (corresponding to the cursor movement), “1110” is the second pattern series (corresponding to the right click), and “1100” is the second pattern series (corresponding to the left click). , “1000” is the fourth pattern series (corresponding to drag). At the center and the right end, a pattern detection range, a multiplication coefficient applied to the range, and a coefficient identification number (ki) are shown, respectively.

  A number sequence in the pattern detection range column indicates repetition of the pattern. For example, “10101010” at the top indicates that one unit (1010) of the first pattern series is repeated. Note that only two units are shown for convenience of illustration. A solid horizontal rectangle shown in the same column is a detection range when the phase is not shifted (hereinafter referred to as a basic detection range 101), and a broken horizontal rectangle is a detection range when the phase is shifted ( Hereinafter, it is referred to as a phase shift countermeasure detection range 102).

  In the case of the first pattern series (1010), “1” and “0” appear alternately, so the phase shift countermeasure detection range 102 may be one type, but other pattern series (1110, 1100, 1000) In this case, since the appearance intervals of “1” and “0” are unequal, three types of phase shift countermeasure detection ranges 102 in which the positions are shifted by 1 bit each are necessary.

  Accordingly, there are 14 types of pattern detection ranges in the entire four pattern series (first to fourth pattern series). Since the multiplication coefficient is necessary for each detection range, the multiplication coefficient is also 14 types (k1 to k14).

  The multiplication coefficient is a 4-digit numerical value having polarity, and the format is (± ki_1, ± ki_2, ± ki_3, ± ki_4). “±” represents polarity, “ki” represents a count identification number, and “_1, _2, _3, and _4” represent digit numbers. Each digit of the coefficient corresponds to each bit within the pattern detection range. For example, since the multiplication coefficient k1 is (1, -1,1, -1) and the pattern detection range to be applied is “1010”, ± ki_1 = 1 is set to the image of the first bit (1). ± ki_2 = −1 is applied to the image of the second bit (0), ± ki — 3 = 1 is applied to the image of the third bit (1), and ± ki — 4 = −1 is applied to the fourth bit (0). It means to apply to the image of bit (0).

Based on the above points, the actual application method of the multiplication coefficient will be specifically described.
FIG. 22A is a three-dimensional luminance distribution diagram of an image including a bright spot (eg, a light spot 4 on the screen 5) that stands out in a background having a constant luminance. The vertical axis (Z-axis) represents luminance, and the luminance increases (brighter) as it goes upward. The vertical and horizontal sides (Y axis and X axis) represent the vertical size and horizontal size of the image. In this figure, the central portion 103 is slightly raised, which indicates a portion having higher brightness than the surroundings. Needless to say, this three-dimensional luminance distribution is a simplified version of the actual one, but it can be said that the characteristics of the actual image are well represented in many respects.

  FIG. 22B is a model diagram in which the three-dimensional luminance distribution of FIG. 22A is extremely simplified. In this figure, a flat base portion 104 having a slight thickness is a background portion in an image, and a cylindrical portion 105 protruding in the Z-axis direction at a substantially central portion of the base portion 104 is a high luminance portion. is there. Hereinafter, the actual pattern detection process will be described with reference to the simplified model diagram.

  FIG. 23 and FIG. 24 are diagrams for explaining the operation when the multiplication coefficient k1 is applied to the uppermost pattern detection range (1010) of FIG. First, referring to FIG. On the left side of the figure, the first model 106 including the base portion 104 and the cylindrical portion 105 in order from the top, the second model 107 including only the base portion 104, the third model 108 including the base portion 104 and the cylindrical portion 105, And the 4th model 109 only of the base part 104 is drawn. A model including the base portion 104 and the cylindrical portion 105 (first model 106 and third model 108) corresponds to an image of bit 1 within the pattern detection range, and a model including only the base portion 104 (second model 107 and first model 108). The four models 109) correspond to the bit 0 image within the pattern detection range. That is, these first to fourth models 106 to 109 represent four images in the uppermost pattern detection range (1010) in FIG.

  Right-pointing white arrows 110 to 113 drawn in order from the top in the center of this figure indicate that each model (first to fourth models 106 to 109) is multiplied by a specific coefficient. (Refer to the multipliers 96 to 99 in FIG. 18). According to FIG. 21, the first model 106 is multiplied by “+1” of k1_1, the second model 107 is multiplied by “−1” of k1_2, and the third model 108 is multiplied by “+1” of k1_3. ”And the fourth model 109 is multiplied by“ −1 ”of k1_4.

  In simple terms, the multiplication of a coefficient is to invert the direction of the luminance of the image according to the polarity of the coefficient (when the coefficient is negative) or to double the luminance by the value of the coefficient. It is. That is, in the example shown in the figure, the first model 106 and the third model 108 are both multiplied by a positive value and a coefficient of value 1, so that the multiplication result remains unchanged, and the first model 106 and the third model 108 are not changed. A first model 106 'and a third model 108' identical to the model 108 are retrieved. On the other hand, the values of the second model 107 and the fourth model 109 are both 1, but they are multiplied by a negative coefficient, so that the multiplication result inverts the second model 107 and the fourth model 109. The second model 108 ′ and the fourth model 109 ′ in the state of being made are taken out.

  These multiplication results (first model 106 'to fourth model 109') are added in FIG. 24 (see adder 100 in FIG. 18). The first to fourth models 106 'to 109' have the same height (Z-axis height, that is, luminance), but the second model 106 'and the third model 108' The polarities of the model 107 'and the fourth model 109' are opposite. Accordingly, the base portions 104 of the first model 106 'and the third model 108' are canceled out by the base portions 104 of the second model 107 'and the fourth model 109', and are removed. And the cylindrical portion 105 of the third model 108 'are added together to obtain an added image GΣ in which only the high luminance portion 114 remains.

  In this way, an appropriate multiplication coefficient is applied to each image in the pattern detection range, and the multiplication results are added to remove the disturbing background image, and the high luminance portion (that is, the light spot 4). An added image GΣ in which only the image portion) is emphasized can be obtained.

  FIG. 25 to FIG. 30 are flowcharts of the pattern detection process. This flowchart is a subroutine program of Step 41 of the “pattern detection process” program (see FIG. 20) described above. FIGS. 31 to 37 are conceptual diagrams for explaining the operation of the pattern detection process.

  As shown in FIG. 25, this pattern detection process includes detection steps (step S50, step S60, step S70, step S80) of each of the four pattern series (1010, 1110, 1100, 1000) in an appropriate order. For example, the detection processing is executed in the order of pattern “1010” → pattern “1110” → pattern “1100” → pattern “1000”. As will be described in detail later, in this pattern detection process, when a pattern is detected in the preceding sequence of steps, the subsequent steps are not executed and the process returns to the flowchart of FIG. For example, when the pattern “1010” is detected in step S50, subsequent steps S60, S70, and S80 are not executed.

<Detection of first pattern sequence (1010)>
FIG. 26 shows the detailed processing contents of the detection processing (step S50) of the pattern “1010” in FIG. In this detailed processing content, first, in order to perform pattern detection of the basic detection range 101, four evaluation images (G1 to G4) are multiplied by a coefficient k1 as shown in FIG. G1 ′ to G4 ′) are obtained (step S50a, step S50b). The four images (G1 to G4) correspond to the first to fourth models 106 to 109 in FIG. 23, and the four evaluation images (G1 ′ to G4 ′) correspond to the first to fourth models 106 ′ in FIG. Corresponds to ~ 109 '. Next, an addition image (GΣ) of four evaluation images (G1 ′ to G4 ′) is obtained (step S50c), and a high-luminance portion exceeding a predetermined threshold (SL) is included in the addition image (GΣ). It is determined whether or not there is (step S50d). This high luminance portion corresponds to the high luminance portion 114 in FIG. And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the object image (G1-G4) is an image of a 1st pattern series (1010), The pattern series (1010) is set (step S50e), and the process returns to the “pattern detection process” program of FIG.

  On the other hand, when a pattern is not detected in the basic detection range 101, four patterns (G1 to G4) are displayed as shown in FIG. Multiply the coefficient k2 to obtain four evaluation images (G1 ′ to G4 ′) (step S50f, step S50g). Similarly to the above, the four images (G1 to G4) correspond to the first to fourth models 106 to 109 in FIG. 23, and the four evaluation images (G1 ′ to G4 ′) correspond to the first to fourth models in FIG. It corresponds to four models 106'-109 '. Next, an added image (GΣ) of four evaluation images (G1 ′ to G4 ′) is obtained (step S50h), and a high-luminance portion exceeding a predetermined threshold (SL) is included in the added image (GΣ). It is determined whether or not there is (step S50i). This high luminance portion corresponds to the high luminance portion 114 in FIG. And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the object image (G1-G4) is an image of a 1st pattern series (1010), The pattern series (1010) is set (step S50e), and the process returns to the “pattern detection process” program of FIG. 20, but if there is no high-luminance part exceeding the threshold (SL), The pattern detection process (pattern “1110” detection process: step S60) is performed.

  FIG. 27 shows a conceptual diagram of the above-described pattern “1010” detection process (step S50). This conceptual diagram summarizes the same processing contents as in FIG. 26 in the form of a table for easy viewing.

  That is, in the first row (the number of rows is the numerical value described on the left outside the table), four images (G1 to G4) are multiplied by the coefficient k1 in order to perform pattern detection in the basic detection range 101, and four evaluation images are obtained. (G1 ′ to G4 ′) are obtained. The second row indicates that an added image (GΣ) of four evaluation images (G1 ′ to G4 ′) is obtained, and the third row shows a predetermined threshold (SL) in the added image (GΣ). This indicates that it is determined whether or not there is a high-luminance portion exceeding. And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the object image (G1-G4) is an image of a 1st pattern series (1010), The pattern series (1010) is set (step S50e), and the process returns to the “pattern detection process” program of FIG. Accordingly, the first line in FIG. 27 corresponds to steps S50a and S50b in FIG. 26, the second line corresponds to step S50c in FIG. 26, and the third line corresponds to step S50d in FIG. To do.

  Further, in the fourth row, when no pattern is detected in the basic detection range 101, four images (G1 to G4) are multiplied by a coefficient k2 to perform pattern detection in the phase shift countermeasure detection range 102. The 5th line indicates that an evaluation image (G1 ′ to G4 ′) is obtained, and the 5th line indicates that an addition image (GΣ) of 4 evaluation images (G1 ′ to G4 ′) is obtained. This indicates that it is determined whether or not there is a high-luminance portion that exceeds a predetermined threshold value (SL) in the added image (GΣ). And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the object image (G1-G4) is an image of a 1st pattern series (1010), The pattern series (1010) is set (step S50e), and the process returns to the “pattern detection process” program of FIG. Accordingly, the fourth line in FIG. 27 corresponds to step S50f and step S50g in FIG. 26, the fifth line corresponds to step S50h in FIG. 26, and the sixth line corresponds to step S50i in FIG. To do. Note that the 0th line (black inversion line) in FIG. 27 is a simple comment line indicating that the detection target pattern of this process is “1010”.

  Hereinafter, detection of other pattern series will be described with reference to the same conceptual diagram (processing contents are summarized in a table form for easy viewing: FIGS. 28, 29, and 30).

<Detection of Second Pattern Series (1110)>
When no result is obtained by the detection processing of the first pattern series (1010) (when pattern detection cannot be performed with one basic detection range 101 and one phase shift countermeasure detection range 102), In addition, the second pattern series (1110) shown in the conceptual diagram of FIG. 28 is detected (see step S60 of FIG. 25). In step S60, first, in order to detect the pattern of the basic detection range 101, As shown in FIG. 32A, four evaluation images (G1 ′ to G4 ′) are obtained by multiplying the four images (G1 to G4) by a coefficient k3. Similarly to the above, the four images (G1 to G4) correspond to the first to fourth models 106 to 109 in FIG. 23, and the four evaluation images (G1 ′ to G4 ′) correspond to the first to fourth models in FIG. It corresponds to four models 106'-109 '. Next, an addition image (GΣ) of four evaluation images (G1 ′ to G4 ′) is obtained, and whether or not there is a high-luminance portion in the addition image (GΣ) that exceeds a predetermined threshold (SL). Determine. This high luminance portion corresponds to the high luminance portion 114 in FIG. And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the target image (G1-G4) is an image of a 2nd pattern series (1110), The pattern series (1110) is set (step S61), and the process returns to the “pattern detection process” program of FIG.

  On the other hand, if no pattern is detected in the basic detection range 101, four images (G1 to G1) are detected as shown in FIG. 32 (b) in order to perform pattern detection in the first phase shift countermeasure detection range 102. G4) is multiplied by a coefficient k4 to obtain four evaluation images (G1 ′ to G4 ′). Similarly to the above, the four images (G1 to G4) correspond to the first to fourth models 106 to 109 in FIG. 23, and the four evaluation images (G1 ′ to G4 ′) correspond to the first to fourth models in FIG. It corresponds to four models 106'-109 '. Next, an addition image (GΣ) of four evaluation images (G1 ′ to G4 ′) is obtained, and whether or not there is a high-luminance portion in the addition image (GΣ) that exceeds a predetermined threshold (SL). Determine. This high luminance portion corresponds to the high luminance portion 114 in FIG. And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the target image (G1-G4) is an image of a 2nd pattern series (1110), The pattern series (1110) is set (step S61), and the process returns to the “pattern detection process” program of FIG.

  When no pattern is detected in the first phase shift countermeasure detection range 102, four images are displayed as shown in FIG. 33A in order to detect the pattern in the second phase shift countermeasure detection range 102. (G1 to G4) is multiplied by a coefficient k5 to obtain four evaluation images (G1 ′ to G4 ′). Similarly to the above, the four images (G1 to G4) correspond to the first to fourth models 106 to 109 in FIG. 23, and the four evaluation images (G1 ′ to G4 ′) correspond to the first to fourth models in FIG. It corresponds to four models 106'-109 '. Next, an addition image (GΣ) of four evaluation images (G1 ′ to G4 ′) is obtained, and whether or not there is a high-luminance portion in the addition image (GΣ) that exceeds a predetermined threshold (SL). Determine. This high luminance portion corresponds to the high luminance portion 114 in FIG. And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the target image (G1-G4) is an image of a 2nd pattern series (1110), The pattern series (1110) is set (step S61), and the process returns to the “pattern detection process” program of FIG.

  If no pattern is detected even in the second phase shift countermeasure detection range 102, four images are displayed to detect the pattern in the third phase shift countermeasure detection range 102 as shown in FIG. (G1 to G4) is multiplied by a coefficient k6 to obtain four evaluation images (G1 ′ to G4 ′). Similarly to the above, the four images (G1 to G4) correspond to the first to fourth models 106 to 109 in FIG. 23, and the four evaluation images (G1 ′ to G4 ′) correspond to the first to fourth models in FIG. It corresponds to four models 106'-109 '. Next, an addition image (GΣ) of four evaluation images (G1 ′ to G4 ′) is obtained, and whether or not there is a high-luminance portion in the addition image (GΣ) that exceeds a predetermined threshold (SL). Determine. This high luminance portion corresponds to the high luminance portion 114 in FIG. And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the target image (G1-G4) is an image of a 2nd pattern series (1110), The pattern series (1110) is set (step S61), and the process returns to the “pattern detection process” program of FIG.

<Detection of Third Pattern Sequence (1100)>
If no result is obtained even in the detection processing of the second pattern series (1110) (when pattern detection cannot be performed with one basic detection range 101 and three detection ranges 102 for countering phase shift), The third pattern series (1100) shown in the conceptual diagram of FIG. 29 is detected (see step S70 of FIG. 25). In step S70, first, in order to detect the pattern of the basic detection range 101, FIG. As shown in (a), four evaluation images (G1 ′ to G4 ′) are obtained by multiplying the four images (G1 to G4) by a coefficient k7. Similarly to the above, the four images (G1 to G4) correspond to the first to fourth models 106 to 109 in FIG. 23, and the four evaluation images (G1 ′ to G4 ′) correspond to the first to fourth models in FIG. It corresponds to four models 106'-109 '. Next, an addition image (GΣ) of four evaluation images (G1 ′ to G4 ′) is obtained, and whether or not there is a high-luminance portion in the addition image (GΣ) that exceeds a predetermined threshold (SL). Determine. This high luminance portion corresponds to the high luminance portion 114 in FIG. And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the object image (G1-G4) is an image of a 3rd pattern series (1100), The pattern series (1100) is set (step S71), and the process returns to the “pattern detection process” program of FIG.

  On the other hand, if no pattern is detected in the basic detection range 101, four images (G1 to G1) are detected as shown in FIG. 34 (b) in order to perform pattern detection in the first phase shift countermeasure detection range 102. G4) is multiplied by a coefficient k8 to obtain four evaluation images (G1 ′ to G4 ′). Similarly to the above, the four images (G1 to G4) correspond to the first to fourth models 106 to 109 in FIG. 23, and the four evaluation images (G1 ′ to G4 ′) correspond to the first to fourth models in FIG. It corresponds to four models 106'-109 '. Next, an addition image (GΣ) of four evaluation images (G1 ′ to G4 ′) is obtained, and whether or not there is a high-luminance portion in the addition image (GΣ) that exceeds a predetermined threshold (SL). Determine. This high luminance portion corresponds to the high luminance portion 114 in FIG. And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the object image (G1-G4) is an image of a 3rd pattern series (1100), The pattern series (1100) is set (step S71), and the process returns to the “pattern detection process” program of FIG.

  When a pattern is not detected in the first phase shift countermeasure detection range 102, four images are detected as shown in FIG. 35A in order to perform pattern detection in the second phase shift countermeasure detection range 102. (G1 to G4) is multiplied by a coefficient k9 to obtain four evaluation images (G1 ′ to G4 ′). Similarly to the above, the four images (G1 to G4) correspond to the first to fourth models 106 to 109 in FIG. 23, and the four evaluation images (G1 ′ to G4 ′) correspond to the first to fourth models in FIG. It corresponds to four models 106'-109 '. Next, an addition image (GΣ) of four evaluation images (G1 ′ to G4 ′) is obtained, and whether or not there is a high-luminance portion in the addition image (GΣ) that exceeds a predetermined threshold (SL). Determine. This high luminance portion corresponds to the high luminance portion 114 in FIG. And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the object image (G1-G4) is an image of a 3rd pattern series (1100), The pattern series (1100) is set (step S71), and the process returns to the “pattern detection process” program of FIG.

  If no pattern is detected even in the second phase shift countermeasure detection range 102, four images are displayed to detect the pattern in the third phase shift countermeasure detection range 102 as shown in FIG. (G1 to G4) is multiplied by a coefficient k10 to obtain four evaluation images (G1 ′ to G4 ′). Similarly to the above, the four images (G1 to G4) correspond to the first to fourth models 106 to 109 in FIG. 23, and the four evaluation images (G1 ′ to G4 ′) correspond to the first to fourth models in FIG. It corresponds to four models 106'-109 '. Next, an addition image (GΣ) of four evaluation images (G1 ′ to G4 ′) is obtained, and whether or not there is a high-luminance portion in the addition image (GΣ) that exceeds a predetermined threshold (SL). Determine. This high luminance portion corresponds to the high luminance portion 114 in FIG. And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the object image (G1-G4) is an image of a 3rd pattern series (1100), The pattern series (1100) is set (step S71), and the process returns to the “pattern detection process” program of FIG.

<Detection of Fourth Pattern Series (1000)>
If no result is obtained even in the detection processing of the third pattern series (1100) (when pattern detection cannot be performed with one basic detection range 101 and three detection ranges 102 for countering phase shift), The fourth pattern series (1000) shown in the conceptual diagram of FIG. 30 is detected (see step S80 of FIG. 25). In step S80, first, in order to detect the pattern of the basic detection range 101, FIG. As shown in (a), four evaluation images (G1 ′ to G4 ′) are obtained by multiplying the four images (G1 to G4) by a coefficient k11. Similarly to the above, the four images (G1 to G4) correspond to the first to fourth models 106 to 109 in FIG. 23, and the four evaluation images (G1 ′ to G4 ′) correspond to the first to fourth models in FIG. It corresponds to four models 106'-109 '. Next, an addition image (GΣ) of four evaluation images (G1 ′ to G4 ′) is obtained, and whether or not there is a high-luminance portion in the addition image (GΣ) that exceeds a predetermined threshold (SL). Determine. This high luminance portion corresponds to the high luminance portion 114 in FIG. And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the object image (G1-G4) is an image of a 4th pattern series (1000), The pattern series (1000) is set (step S81), and the process returns to the “pattern detection process” program of FIG.

  On the other hand, if no pattern is detected in the basic detection range 101, four images (G1 to G1) are detected as shown in FIG. 36 (b) in order to perform pattern detection in the first phase shift countermeasure detection range 102. G4) is multiplied by a coefficient k12 to obtain four evaluation images (G1 ′ to G4 ′). Similarly to the above, the four images (G1 to G4) correspond to the first to fourth models 106 to 109 in FIG. 23, and the four evaluation images (G1 ′ to G4 ′) correspond to the first to fourth models in FIG. It corresponds to four models 106'-109 '. Next, an addition image (GΣ) of four evaluation images (G1 ′ to G4 ′) is obtained, and whether or not there is a high-luminance portion in the addition image (GΣ) that exceeds a predetermined threshold (SL). Determine. This high luminance portion corresponds to the high luminance portion 114 in FIG. And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the object image (G1-G4) is an image of a 4th pattern series (1000), The pattern series (1000) is set (step S81), and the process returns to the “pattern detection process” program of FIG.

  When a pattern is not detected in the first phase shift countermeasure detection range 102, four images are displayed as shown in FIG. 37A in order to perform pattern detection in the second phase shift countermeasure detection range 102. (G1 to G4) is multiplied by a coefficient k13 to obtain four evaluation images (G1 ′ to G4 ′). Similarly to the above, the four images (G1 to G4) correspond to the first to fourth models 106 to 109 in FIG. 23, and the four evaluation images (G1 ′ to G4 ′) correspond to the first to fourth models in FIG. It corresponds to four models 106'-109 '. Next, an addition image (GΣ) of four evaluation images (G1 ′ to G4 ′) is obtained, and whether or not there is a high-luminance portion in the addition image (GΣ) that exceeds a predetermined threshold (SL). Determine. This high luminance portion corresponds to the high luminance portion 114 in FIG. And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the object image (G1-G4) is an image of a 4th pattern series (1000), The pattern series (1000) is set (step S81), and the process returns to the “pattern detection process” program of FIG.

  If no pattern is detected even in the second phase shift countermeasure detection range 102, four images are displayed to detect the pattern in the third phase shift countermeasure detection range 102 as shown in FIG. (G1 to G4) is multiplied by a coefficient k14 to obtain four evaluation images (G1 ′ to G4 ′). Similarly to the above, the four images (G1 to G4) correspond to the first to fourth models 106 to 109 in FIG. 23, and the four evaluation images (G1 ′ to G4 ′) correspond to the first to fourth models in FIG. It corresponds to four models 106'-109 '. Next, an addition image (GΣ) of four evaluation images (G1 ′ to G4 ′) is obtained, and whether or not there is a high-luminance portion in the addition image (GΣ) that exceeds a predetermined threshold (SL). Determine. This high luminance portion corresponds to the high luminance portion 114 in FIG. And if the high-intensity part exceeding a threshold value (SL) exists, it will judge that the object image (G1-G4) is an image of a 4th pattern series (1000), The pattern series (1000) is set (step S81), and the process returns to the “pattern detection process” program of FIG.

  On the other hand, if the result is not obtained even in the detection process of the fourth pattern series (1000), it is determined that the information is not sent from the light emitting unit 2 or is not received correctly even if the information is sent, A predetermined value (“0000”) indicating pattern non-detection is set in the detection pattern (step S82), and the process returns to the “pattern detection process” program of FIG.

  In this way, each image (G1) of the pattern detection ranges of the first pattern series (1010), the second pattern series (1110), the third pattern series (1100), and the fourth pattern series (1000). ˜G4), an appropriate multiplication coefficient ki is applied, and the multiplication results (G1 ′ to G4 ′) are added to remove the disturbing background image, and the high luminance part (that is, the image of the light spot 4). An added image GΣ in which only the (part) is emphasized can be obtained. Then, the first pattern series (1010), the second pattern series (1110), and the third pattern series are determined by determining the presence or absence of a high-luminance part (part exceeding the threshold SL) of the added image GΣ. (1100) and the fourth pattern series (1000) can be detected.

<Mouse command output processing>
FIG. 38 is a flowchart of mouse command output processing. This flowchart is a subroutine program of step 44 of the “pattern detection process” program (see FIG. 20) described above.
When this flowchart is started, first, light spot coordinates are detected (step S90). Here, the light spot coordinates refer to the coordinates of the light spot 4 projected on the screen 5. The size (diameter) of the light spot 4 is much larger than that of a laser beam or the like. This is because the spot light 3 that is the origin of the light spot 4 has a slight spread (see α in FIG. 2). Therefore, since the area of the high-luminance portion (image portion of the light spot 4) of the added image GΣ also increases in the same manner, for example, when the position of the mouse cursor is moved using the position of the high-luminance portion as it is, In addition to slow movement of the cursor, there is an inconvenience that detailed movement cannot be expected. For example, the following measures may be taken.

  FIG. 39 is a preferred conceptual view of detection of light spot coordinates. In this figure, the high-luminance portion (image portion of the light spot 4) of the added image GΣ is a region that exceeds the threshold value SL. This region includes a large number of pixels, and each pixel has a luminance value. Assuming that the luminance values are not saturated, a luminance value peak should be observed at the approximate center of the region. This is because the amount of light in the vicinity of the center of the optical axis of the spot light 3 is maximum, and the brightness of the central portion of the light spot 4 formed by the spot light 3 is also maximum. From such a point, pixel coordinates (x, y) that exceed the threshold value SL in the added image GΣ and have the maximum luminance value may be used as the light point coordinates. Coordinate detection can be performed with an accuracy of one pixel or several pixels. Therefore, the movement of the cursor when the position of the mouse cursor is moved can be finely controlled.

  When the light spot coordinates are detected in this way, the pattern is branched in order to output a mouse command corresponding to the pattern specified in the pattern detection process (step S91). If the specified pattern is the first pattern series (1010), a cursor movement command is output together with the light spot coordinates (step S92), or the specified pattern is the second pattern series (1110). If there is, the right click command is output together with the light spot coordinates (step S93), or if the specified pattern is the third pattern series (11000), the left click command is output together with the light spot coordinates (step S94). ) Or if the specified pattern is the fourth pattern series (1000), after outputting the drag command together with the light spot coordinates (step S95), the process returns to the “pattern detection processing” program of FIG.

<Summary of Embodiment>
As described above, according to the present embodiment, by operating the light emitting unit 2, the mouse movement operation, right click operation, left click operation, or drag operation of the personal computer 10 can be performed while the presentation is being performed. A special effect is possible. In addition, since these operations are performed using “light”, there are no inconveniences such as radio waves. Furthermore, since the spot light 3 from the light emitting unit 2 is light (infrared rays) having a wavelength outside the visible region, the light spot 4 is also invisible to human eyes. Therefore, not only the operator (user 1) of the light emitting unit 3 but also other people who are watching the screen 5 are not aware of the existence of the light spot 4, and the screen information 12 displayed on the screen 5 is displayed. And the cursor 11 will not be distracted. In addition, since the light spot 4 is not visible, the user 1 performs an operation such as cursor movement or clicking while gazing at an actual operation target (such as a cursor on the screen 5). Since the operation target also follows with (accurately, the direction of the lens 20a located at the tip of the case 20), some familiarity is necessary, but there is no operational trouble at all.

<Measures to prevent false detection at pattern transition points>
Next, the determination “is different from the previous detection pattern?” In step S43 in FIG. 20 will be described.

  First, as is clear from the above description, the light emitting unit 2 transmits one piece of information of any of the four patterns (first to fourth pattern series) one unit at a time (for example, see FIG. 10) for photographing. The device 15 detects the pattern while dividing the information transmitted from the light emitting unit 2 into a range of n bits (basic detection range 101 and phase shift countermeasure detection range 102).

  Assume that the same pattern (first pattern series) as 1010... 1010. In such a case, it is 1010 or 0101 regardless of where n bits are divided, and a correct pattern can be detected in either the basic detection range 101 or the phase shift countermeasure detection range 102. However, when switching to another pattern in the middle, for example, when switching from the first pattern series (1010) to the third pattern series (1100), the switching portion (hereinafter referred to as a pattern transition point or simply a transition). May cause inconvenience.

  FIG. 40 is an inconvenience explanatory diagram at a pattern transition point. In this figure, the 4-bit sequence spanned by the transition point surrounded by the dashed-dotted line range 115 is “1011”. This bit arrangement does not match the first pattern series (1010) before the transition and does not match the third pattern series (1100) after the transition. When such a bit arrangement (1011) is set as a pattern search target range, the pattern detection process of the present embodiment causes a disadvantage that it is mistaken for the second pattern series (1110). This is because, when the column of the second pattern arrangement (1110) in FIG. 21 is seen, the same bit arrangement (1011) exists third from the top of the column.

  Therefore, in this case, it should be correctly determined as either the first pattern series (1010) or the third pattern series (1100). In the detection process of “pattern (1110)” in FIG. This results in an inconvenient result of passing the pattern detection using the detection area 102 for the countermeasure against deviation and eventually proceeding to the mouse command output process and outputting an incorrect command (right click command).

  The determination “is different from the previous detection pattern?” In step S43 in FIG. 20 is entered to avoid such inconvenience. In other words, the difference between the previous detection pattern and the current detection pattern may be a correct pattern transition, but in some cases, the wrong pattern as described above (for example, 1011 in FIG. 40) is detected. Therefore, when such a pattern transition is detected, the mouse command output process does not proceed for the sake of safety, and it waits for a predetermined time (Td) to deviate from the transition point ( Step S46). The predetermined time (Td) may be a time corresponding to at least n bits, that is, one unit of time. If the pattern detection is restarted after the elapse of a predetermined time (Td) at the time when the transition point is detected, for example, even in the example of FIG. 40, it is possible to erroneously determine the second pattern series (1110). The correct pattern (third pattern series) can be determined after the transition point.

<Modification>
The embodiment of the present invention is not limited to the above-described examples, and various modifications and developments are naturally included. For example, the following may be adopted.
FIG. 41 is an external view showing another example of the light emitting unit. In addition to having all the functions of the light emitting unit 2 in the above-described embodiment, the light emitting unit 2 ′ is further characterized in that it can also be used as a normal laser pointer (a point that is combined). That is, the light emitting unit 2 ′ includes all the configurations of the light emitting unit 2 in the above embodiment, and further includes a laser pointer unit 116 inside a case 20 ′ suitable for holding by hand. The laser beam 117 from the laser pointer section 116 can be projected from the end face of the case 20 'to the outside. For this reason, the light emitting unit 2 ′ includes some buttons 118 and 119 for a mouse operation (for example, right click operation and right click operation), and the light emitting unit 2 ′ is used as a mouse operation. Alternatively, a select switch 120 for switching whether to use as a normal laser pointer is provided. For example, when the front side of the select switch 120 is pressed, the laser beam 117 is emitted from the laser emitting lens 121, and when the rear side is pressed, the spot light 3 is emitted from the mouse operating lens 24a. It has become.

  Therefore, according to the light emitting unit 2 ', since it is a dual-purpose type that can be used as a laser pointer, it can be used effectively in presentations and the like that do not require a mouse operation, thereby improving convenience.

It is a utilization example figure of the input device of this embodiment. It is the external view of the light emission unit 2, and the enlarged view of a switch. 3 is an electrical configuration diagram of a light emitting unit 2. FIG. It is a figure which shows the flowchart of the control program run by CPU23a of the electronic unit 23. FIG. It is a figure which shows the schematic flowchart of a cursor movement process. It is a figure which shows the schematic flowchart of a right click process. It is a figure which shows the schematic flowchart of a left click process. It is a figure which shows the schematic flowchart of a drag process. FIG. 4 is a diagram illustrating an operation example of the light emitting unit 2. It is the figure which arranged the blink pattern of the spot light 3 in the case of an operation example in time series. 1 is an external view of a photographing device 15. FIG. 3 is an internal block diagram of the image capturing device 15. FIG. FIG. 10 is a diagram illustrating an operation flowchart of the photographing apparatus 15. It is a figure which shows the flowchart of MENU processing for imaging | photography modes. It is a figure which shows the menu screen for imaging | photography modes. It is a figure which shows the flowchart of a mouse | mouth process (step S24e of FIG. 14). 2 is a schematic block diagram of an electrical configuration of a personal computer 13 and a projector 6. FIG. It is a conceptual structure figure of eight frame memories F1-F8 referred by pattern reading processing. It is an operation | movement conceptual diagram of eight frame memory F1-F8. It is a figure which shows the flowchart of a pattern reading process. It is a figure which shows the pattern series of a detection target, its detection range, and the multiplication coefficient applied to each detection range. FIG. 4 is a three-dimensional luminance distribution diagram of an image including a bright spot (for example, a light spot 4 on the screen 5) that stands out in a background having a constant luminance, and a model diagram that is an extremely simplified model thereof. FIG. 22 is an operation explanatory diagram (1/2) when the multiplication coefficient k1 is applied to the uppermost pattern detection range (1010) of FIG. FIG. 22 is an operation explanatory diagram (2/2) when the multiplication coefficient k1 is applied to the uppermost pattern detection range (1010) of FIG. It is a figure which shows the flowchart of a pattern detection process. It is a figure which shows the flowchart of a pattern detection process (detection of 1010). It is a conceptual diagram of a pattern detection process (detection of 1010). It is a conceptual diagram of a pattern detection process (detection of 1110). It is a conceptual diagram of a pattern detection process (detection of 1100). It is a conceptual diagram of a pattern detection process (detection of 1000). It is a conceptual diagram (1/7) for demonstrating the effect | action of a pattern detection process. It is a conceptual diagram (2/7) for demonstrating the effect | action of a pattern detection process. It is a conceptual diagram (3/7) for demonstrating the effect | action of a pattern detection process. It is a conceptual diagram (4/7) for demonstrating the effect | action of a pattern detection process. It is a conceptual diagram (5/7) for demonstrating the effect | action of a pattern detection process. It is a conceptual diagram (6/7) for demonstrating the effect | action of a pattern detection process. It is a conceptual diagram (7/7) for demonstrating the effect | action of a pattern detection process. It is a figure which shows the flowchart of a mouse command output process. It is a preferable detection conceptual diagram of a light spot coordinate. It is inconvenient explanatory drawing in a pattern transition point. It is an external view which shows the other example of a light emission unit.

Explanation of symbols

2 Light emitting unit (irradiation means)
3 Spot light (specific wavelength light)
4 Light spots (irradiation area)
5 screen (target)
6 Projector (projection means)
14 screens (display means)
15 Shooting device (shooting means)
23a CPU (selection means)
23b ROM (control information storage means)
27 Operation switch (control information input means)
28 Drive unit (modulation means)
45 Imaging system (photographing means)
54 Image buffer (frame storage means)
60 CPU (extraction means, first generation means, second generation means, readout means, multiplication image generation means, addition image generation means, determination means, pattern identification means)
79 CPU (display control means)
96 multiplier (multiplication image generation means)
97 Multiplier (Multiplication image generation means)
98 multiplier (multiplication image generation means)
99 multiplier (multiplication image generation means)
100 adder (added image generating means)

Claims (8)

Control information input means for inputting control information;
Conversion means for converting the control information input by the control information input means into a predetermined pattern series;
A modulation means for modulating the light into a specific wavelength with a time series change according to the predetermined pattern series converted by the conversion means;
Irradiating means for irradiating the target with the specific wavelength light modulated by the modulating means;
Imaging means for imaging the target irradiated with the specific wavelength light by the irradiation means at a predetermined frame rate;
Display means for displaying a frame photographed by the photographing means;
Extracting means for extracting the region irradiated with the specific wavelength light from a plurality of frames photographed by the photographing means;
First generation means for generating the predetermined pattern series from the modulation state of the specific wavelength light irradiation region extracted by the extraction means;
Second generation means for generating control information from the predetermined pattern series generated by the generation means;
An information display system comprising: display control means for controlling display of a frame displayed on the display means based on the control information generated by the second generation means. The display means includes projection means for projecting a frame photographed by the photographing means,
The information display system according to claim 1, wherein the target is a projection projected by the projection unit.   The system further includes control information storage means for storing a plurality of sets of the control information in association with the pattern series, and the conversion means corresponds to the control information storage means based on the control information input by the control information input means. 3. The information display system according to claim 1, further comprising selection means for selecting a pattern series. The photographing means photographs a frame including the specific wavelength light region in at least a half modulation section,
The first generation means includes:
Frame storage means for storing a plurality of frames photographed in time series by the photographing means in order of photographing;
Reading means for sequentially reading frames from the frame storing means every other frame;
Multiplication image generation means for generating a multiplication image obtained by multiplying the frame read by the reading means by a coefficient corresponding to the pattern series stored in the control information storage means;
Addition image generation means for generating an addition image from the multiplication image generated by the multiplication means;
Determining means for determining whether or not there is a pixel region exceeding a threshold in the added image generated by the added image generating means;
A pattern sequence specifying unit that specifies a pattern sequence based on an applied multiplication coefficient of a multiplication image that is a source of the added image when the determination unit determines that there is a pixel region that exceeds a threshold value. The information display system according to claim 3. A control information input step for inputting control information;
A conversion step of converting the control information input in the control information input step into a predetermined pattern series;
In accordance with the predetermined pattern sequence converted in the conversion step, a modulation step that modulates to a specific wavelength light accompanied by a time series change,
An irradiation step of irradiating the target with the specific wavelength light modulated in the modulation step;
An imaging step of imaging the target irradiated with the specific wavelength light in the irradiation step at a predetermined frame rate;
A display step for displaying a frame shot in the shooting step;
An extraction step for extracting a region irradiated with the specific wavelength light from a plurality of frames photographed in the photographing step;
A first generation step of generating the predetermined pattern series from the modulation state for the specific wavelength light irradiation region extracted by the extraction step;
A second generation step of generating control information from the predetermined pattern series generated by the generation step;
An information display method comprising: a display control step for controlling display of a frame displayed in the display step based on the control information generated in the second generation step. The display step includes a projecting step of projecting the frame photographed in the photographing step,
The information display method according to claim 5, wherein the target is a projection projected by the projecting step.     A selection step of selecting a corresponding pattern sequence from the memory based on the control information input in the control information input step, the memory storing a plurality of sets of the control information in association with the pattern sequence The information display method according to claim 5 or 6, characterized by comprising: The imaging step captures a frame including the specific wavelength light region in a modulation interval of at least 1/2.
The first generation step includes:
A frame storing step for storing a plurality of frames shot in time series in the shooting step in the shooting order;
A reading step of sequentially reading out frames every other frame from the plurality of frames stored in the frame storing step;
A multiplication image generation step of generating a multiplication image obtained by multiplying the frame read in the reading step by a coefficient corresponding to the pattern series stored in the memory;
An addition image generation step of generating an addition image from the multiplication image generated by the multiplication step;
A determination step of determining whether or not there is a pixel region exceeding a threshold in the addition image generated by the addition image generation step;
A pattern sequence specifying step for specifying a pattern sequence based on an applied multiplication coefficient of a multiplication image that is a source of the added image when the determination step determines that there is a pixel region that exceeds a threshold value. The information display method according to claim 7.

Images