JP2009070245A - Image projection display device, image projection display method, image projection display program and storage medium - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To allow an operation without using a mouse or the like in a computer having a certain level of screen size. <P>SOLUTION: An image is projected to a projecting surface 13 of a screen 12 by a projector 15 set in a casing 14. When a user views the image projected to the screen 12, and touches a wiring surface 11 on the surface side of the screen 12 by the finger, a CCD camera 16 takes an image of the finger's shadow, and supplies image data thereof to an information processing device 100. The information processing device 100 recognizes the motion of the user's finger based on the image data from the CCD camera 16, and performs activation processing or moving processing of a window displayed on the screen, start-up processing of a program or the like. <P>COPYRIGHT: (C)2009,JPO&INPIT

Description

  The present invention relates to an image projection display device, an image projection display method, and an image projection display program, and more particularly to an image projection display device, an image projection display method, and an image projection display program suitable for use in a tabletop computer.

When using a tabletop computer, there is a problem that it is cumbersome to use with a mouse that is generally used in the past when writing on the tabletop or moving an object on the tabletop. is there. Further, when a table top type computer is used, there is a problem due to its shape. That is, when the user sits in front of the table and uses it, the reachable range of the user's hand is fixed, so when writing or installing an object in a portion far from the user on the screen, the posture becomes unreasonable ( Non-patent document 1).
However, if the screen is made in a size that can be reached, the screen becomes smaller and the resolution becomes lower. Therefore, there is a method of specifying the pointing and the position of the user using a shadow (see, for example, Patent Documents 2 and 3).

As a method for effectively using a limited screen size while reducing the burden on the user's posture when using a large screen, several methods have been proposed for wall-type display interaction (for example, non-patented). Reference 2 and Patent Reference 1).
In Non-Patent Document 2, a camera capable of acquiring distance information is installed on the upper part of the user, and user operation information is acquired based on the distance information to interact with the display. Further, in the invention of Patent Document 1, a user's operation information is acquired using a Pohimass sensor, and interaction with a display is performed.
JP 2003-280817 A JP-A-2005-78291 JP 2005-148555 A JP 2004-109402 A Aaron Toney, Bruce H. Thomas, `` Considering Reach in Tangible and Table Top Design, TABLETOP '''06.2. Tetsuo Fukasawa, Kentaro Fukuchi, Hideki Koike "Study on Non-Contact Dialogue with Wall Display, 119th HI Workshop"

However, when using a camera that can acquire distance information, it is necessary to install the camera on the ceiling surface. When using a Pohimass sensor, it is necessary to place the device within the movable range of the user and the sensor. There was a problem.
The present invention has been made in view of the above, and in a table top type computer having a certain screen size, it is not necessary to install special equipment in a place other than the apparatus such as a ceiling surface, and the operation of the user A function that can be operated without using a mouse is proposed.

The image projection display device according to claim 1, a projection surface, a projection device that projects a predetermined image on the projection surface, and an image in which pixels are two-dimensionally arranged on the image projected on the projection surface An image pickup means for picking up an image by a unit, a time measuring means for measuring time, and a detection means for detecting a shadow of a user's finger included in the image picked up by the image pickup means and detecting a pointing position corresponding to the shadow And processing means for executing processing corresponding to the stationary time measured by the time measuring means for the shadow detected by the detecting means.
The image projection display device may further include a threshold initial setting unit that stores a predetermined threshold used when the detection unit detects a shadow, and the threshold initial setting unit includes the projection plane. A state in which a shadow is generated within a predetermined range of the image projected on the image is picked up by the imaging means, and a luminance value of a shadow within the predetermined range of the picked-up image is determined as the predetermined threshold value It is characterized by.
According to a third aspect of the present invention, the processing means compares the luminance value of the shadow imaged by the imaging means with the threshold value stored in the threshold value initial setting means, and determines whether the shadow is for pointing purposes. It is characterized by determining.

According to a fourth aspect of the present invention, the apparatus further comprises an activating unit that activates the window that has not been activated when the shadow detected for the purpose of pointing is stationary for a predetermined time.
According to a fifth aspect of the present invention, the apparatus further comprises window moving means for moving the window by the movement of the pointing position detected by the detecting means after activating the window.
According to a sixth aspect of the present invention, when there is an object that can be clicked at the pointing position detected by the detection unit, the display unit further includes a click determination unit that determines that the click is performed when the pointing operation is stopped.
According to a seventh aspect of the present invention, when the shadow stillness is detected in a state that does not have a sufficient luminance value such that the pointing position is not detected, the detected user's position and the window at the destination connecting the pointing position are drawn. It is characterized by that.
According to an eighth aspect of the present invention, in the state where the shadow position is not detected in a state where the pointing position is not detected, and the stationary position of the shadow is detected, the scrolling of the window at the destination connecting the detected user position and the pointing position is performed. It is characterized by attracting bars.
According to a ninth aspect of the present invention, when a pointing operation is detected during the pulling of the window or the scroll bar, the pulling of the window or the scroll bar is immediately stopped.

According to a tenth aspect of the present invention, a projection step of projecting a predetermined image onto a projection plane by a projection device, and an image projected onto the projection plane by an imaging unit is captured by an imaging unit in which pixels are two-dimensionally arranged. An imaging step, a time measuring step for measuring time by a time measuring means, and a detecting means for detecting a shadow of a user's finger included in the image captured by the imaging means and determining a pointing position corresponding to the shadow. A detection step of detecting, and a processing step of executing processing corresponding to a time when the shadow detected in the detection step timed in the time measuring step is stationary by the processing means.
According to an eleventh aspect of the present invention, a projecting step of projecting a predetermined image onto a projection surface on a computer, an imaging step of capturing an image projected on the projection surface with an imaging unit in which pixels are two-dimensionally arranged, and time A timing step for measuring the time, a detecting step for detecting a shadow of a user's finger included in the image captured by the imaging means, and a pointing position corresponding to the shadow, and a timing for the timing And a processing step of executing processing corresponding to the time when the shadow detected in the detection step is stationary.
A twelfth aspect is characterized in that the computer-readable recording medium according to the eleventh aspect records the program according to the tenth aspect.

  According to the image projection display device, the image projection display method, and the image projection display program of the present invention, in a table top type computer having a certain screen size, it is necessary to install special equipment in a place other than the device such as a ceiling surface. It is possible to operate without using a mouse or the like according to the user's operation.

(First embodiment)
FIG. 1 is a diagram illustrating a configuration example of a first embodiment of the present invention. As shown in the figure, an image display device 1 according to the present embodiment includes a writing surface 11, a screen 12, a projection surface 13, a housing 14, a projector 15 installed in the housing 14, and It comprises a CCD (Charge Coupled Device) camera 16 and an information processing device 100 to which a projector 15 and a CCD camera 16 are connected.
The screen 12 is a transmissive screen that has been subjected to diffusion treatment. The projector 15 is installed toward the screen 12, and an image from the projector 15 is projected on the screen 12. The CCD camera 16 continues to shoot the screen 12. A projector 15 and a CCD camera 16 are connected to the information processing apparatus 100, and an input image from the CCD camera 16 and an output image to the projector 15 are processed. The writing surface 11 is made of a transparent member having a certain level of strength for use as a desk.

Hereinafter, this embodiment will be described in detail. FIG. 2 shows a usage pattern of the present embodiment. As shown in FIG. 2, the image display apparatus 1 according to the present embodiment includes a rectangular plane part (writing surface 11), and includes a desk-like display part (screen 12) and an information processing device 100. Yes. The display unit (screen 12) is configured such that an image projected from the inside of the housing 14 is displayed at the central portion of a rectangular flat surface (writing surface 11).
As shown in FIG. 1, the image display device 1 includes a screen 12, a casing 14 that supports the screen 12, a projector 15 that is provided in the casing 14 and projects an image on the screen 12, and an interior of the casing 14. And a CCD camera 16 for photographing the screen 12 from the back side. The CCD camera 16 disposed in the housing 14 and the information processing device 100 are connected by a cord, and the projector 15 disposed in the housing 14 and the information processing device 100 are optically coupled.

The screen 12 has a double structure, a projection surface 13 on which a projection image is projected is provided on the back side of the screen 12, and a drawing that can be drawn using an aqueous pen, a whiteboard marker, or the like on the front side of the screen 12. The insertion surface 11 is provided, and the projection surface 13 and the writing surface 11 are in close contact with each other.
The writing surface 11 and the projection surface 13 are both transparent, but there are fine irregularities on the surface of the projection surface 13 that is in close contact with the writing surface 11, and an image is projected onto the projection surface 13. Since the light is slightly diffused and transmitted, the projected image can be viewed from various angles from the surface side of the screen 12 provided with the writing surface 11. Here, the surface of the writing surface 11 may be covered with a transparent protective sheet so as not to be scratched, or may be coated with a transparent paint or the like.
The information processing apparatus 100 captures and recognizes a shadow image captured by the CCD camera 16 and operates a projection image projected from the projector 15 on the back side of the screen 12 according to the acquired image. be able to. The information processing apparatus 100 may be created exclusively for the image display device 1 of the present embodiment, or may be configured by installing predetermined software on a personal computer. Further, the information processing apparatus 100 may be disposed in the housing 14.

The projector 15 is coupled to a display (not shown) of the information processing device 100 by an optical system such as a reflection mirror and a beam splitter (not shown), and can project an image generated by the information processing device 100 onto the projection surface 13 of the screen 12. It is like that.
The CCD camera 16 is connected to the information processing apparatus 100 by a code via, for example, a USB (Universal Serial Bus) interface, and an object placed on the surface side of the screen 12, that is, the writing surface 11, or the writing surface 11 The image drawn on the writing surface 11 and the shadow reflected on the writing surface 11 are photographed at a predetermined time interval from the back surface side of the screen 12, that is, the projection surface 13 side, and image data can be obtained. Image data corresponding to the photographed image is supplied to the information processing apparatus 100.

FIG. 2 shows a usage pattern of the image display apparatus 1 as the present embodiment. The user M uses the image display device 1 by sitting in front of the image display device 1. In the image display device 1, a screen 12 is embedded in a portion corresponding to a table-shaped framework top plate. The user M can write on the surface side of the screen 12 with a pen.
Under the table, a projector 15 and a CCD camera 16 are installed. An image from the projector 15 is projected on the screen 12, and characters and figures written by the user M with the pen on the screen 12 and the shadow of the user M reflected on the screen 12 are photographed and photographed by the camera 16. The image data of the obtained image is supplied to the information processing apparatus 100 and processed.

  FIG. 3 is a block diagram illustrating a configuration example of the information processing device 100. The information processing device 100 controls the entire information processing device 100, and operates a CPU (Central Processing Unit) 101 that operates according to a program stored in the RAM 102, and a RAM (Random) that temporarily stores programs and data. Access Memory) 102, HDD (Hard Disk Drive) 103, which is a large-capacity external storage device that stores programs and data, and a display controller 104 that controls display of image data output as an operation result of the CPU 101 on a display. A USB (Universal Serial Bus) controller 105 for controlling a CCD (Charge Coupled Device) camera 16, which will be described later, and the like. A projector 15 is connected to the display controller 104, and a CCD camera 16 is connected to the USB controller 105.

FIG. 4 shows a configuration example of software developed in the RAM 102 and executed by the CPU 101. This software includes basic software 200 and application software 205. Under the control of the basic software 200, there are a USB device driver 201, a window control 202, and a display control driver 203.
The USB device driver 201 controls the CCD camera 16, and the window control 202 is a window position and size of various applications 207 executed on the basic software 200, the front-to-back relationship between windows, and active / inactive of windows. Manage and control.
The display control driver 203 generates a video and controls its output. The image processing program 206 is read from the HDD 103 as necessary, developed in the RAM 102, and executed. Then, a predetermined calculation process is performed on the image from the CCD camera 16 obtained via the USB device driver 201, and the calculation result is passed to the window control 202.
The various applications 207 are so-called business applications such as word processing software, spreadsheet software, and Internet browsing software, and operate individually on the basic software 200.

FIG. 5 shows the processing procedure of each means. In this embodiment, a CCD camera 16 as an imaging unit, a USB device controller 201 as an image capturing unit, an image processing program 206 as an image processing unit, a window control 202 as a window processing unit, and a display control as an image generation unit It comprises a driver 203, a projector 15 as projection means, and the like.
As described above, the imaging unit represents the CCD camera 16, the image capturing unit represents the USB device controller 201, the image processing unit represents the image processing program 206 configured as software in the information processing apparatus 100, and the window processing unit. Represents the window control 202 of the basic software 200, the image generation means represents the display control driver 203 of the basic software 200, and the projection means represents the projector 15.
Data captured by the image capturing unit is captured by the information processing device 100 by the image capturing unit. The captured data is subjected to image processing and analysis by the image processing means. Depending on the analysis contents, various windows are managed and controlled by the window processing means, and as a result, the output image of the image generating means is changed. The changed data is output to the projection means and projected on the screen.

Next, a method for activating and attracting a window in the embodiment of the present invention will be described. The window is activated by the user M pointing at the screen 12 for a certain period.
FIG. 6 shows the current state of the desktop W0. On the desktop W0, an active window W1 and a window W2 in the background are overlapped. These windows W1 and W2 are controlled by the window control 202.
FIG. 7 shows a conceptual diagram of window activation. Touch the screen 12 at a certain position in the window W1 to be activated. As a result, a shadow image of the fingertip size is captured by the imaging means (CCD camera 16), and image data of the fingertip size shadow image is captured by the image capturing means (USB device controller 201) for image processing. Supplied to the means (image processing program 206). As a result, the window processing means (window control 202) activates the active window W1 at the position of the shadow of the fingertip.

FIG. 8 is a flowchart showing an activation processing procedure. First, in step S11, an image difference and a shadow size are acquired. That is, a difference image is acquired by the image processing program 206 from the image acquired from the CCD camera 16 via the image capturing means (USB device controller 201) and the reference image stored in the RAM 102.
FIG. 9 is a flowchart illustrating a procedure for acquiring a difference image. First, in step S <b> 31, the image processing program 206 reads a reference image from the RAM 102. FIG. 10 shows an example of a reference image. Next, in step S32, a current image captured by the CCD camera 16 is acquired. FIG. 11 shows an example of an image captured when the user touches the screen 12 while touching it.
Next, in step S33, the image processing program 206 calculates a difference in luminance at the same position between the reference image and the current image. FIG. 12 shows a difference image obtained by calculating the difference between the reference image shown in FIG. 10 and the current image shown in FIG. The position of the user M is represented by a point P1, and the position of the fingertip is represented by a point P2. Next, in step S34, the difference image obtained by the calculation is supplied to the RAM 102 and stored.

FIG. 13 is a flowchart showing a shadow size acquisition procedure. First, in step S41, the image processing program 206 reads from the RAM 102 the difference image stored in the RAM 102 in step S34 of FIG. Next, in step S42, it is determined whether or not the luminance of each pixel of the difference image read from the RAM 102 is equal to or higher than a threshold value.
As a result, when it is determined that the luminance of each pixel of the difference image is equal to or higher than the threshold value, the process proceeds to step S43, the size counter is added, and the luminance of the corresponding part is stored in the RAM 102 as a shadow image in step S44. For example, the threshold is about 40 for 256 gradations. On the other hand, if it is determined in step S42 that the luminance of each pixel of the difference image read from the RAM 102 is equal to or lower than the threshold value, no processing is performed and the process ends.
Thereafter, returning to step S12 in FIG. 8, the image processing program 206 determines whether or not the area of the shadow image pixel is larger than the reference value A1 and smaller than the reference value A2, and the shadow image pixel area is determined. Is larger than the reference value A1 and smaller than the reference value A2, it is recognized that the user M is operating, and the process proceeds to step S13. In step S13, a shadow area is extracted using a threshold value.

On the other hand, if it is determined that the pixel area of the difference image (pixel area of the shadow image) is smaller than the reference value A1 or larger than the reference value A2, the process does not proceed to step S13, and an error or user It is determined that M is performing another operation, and the process proceeds to step S20 to reset the counter and wait.
The reference values A1 and A2 are set by calibration or are set as arbitrary values. Although not shown, as a calibration method, shadow information when a hand is held without touching the screen 12 and when a hand is placed on the screen 12 is read, and the touch information is not touched on the screen 12. The threshold value A1 and the threshold value A2 are set by the calibration program so that the area of the shadow that is recognized is almost eliminated when the hand is held over and the area of the shadow that is recognized when the hand is placed on the screen 12 is present. Stored in the RAM 102, the HDD 103, and the like.

FIG. 14 is a flowchart showing a procedure for separating the size of the shadow image according to the threshold value. First, in step S <b> 51, the image processing program 206 reads a shadow image from the RAM 102. Next, in step S52, the image processing program 206 compares the luminance value of each pixel constituting the shadow image with a threshold value, and determines whether the luminance value of each pixel is larger than the threshold value.
As a result, when it is determined that the luminance value of the pixel is larger than the threshold value (the value is above), 1 is added to the value of the threshold value upper shadow counter. Add 1 to the counter value. Note that the luminance value of the pixel is, for example, a value of 256 gradations. The above determination is made for all the pixels constituting the shadow image, and the value of the upper threshold shadow counter and the lower threshold counter are set.
Next, returning to step S14 in FIG. 8, it is determined whether the number of pixels below the threshold is less than the number of pixels above the threshold, and the number of pixels below the threshold is If it is determined that the number is lower than the number of pixels above, it is determined by the image processing program 206 that there is a large area where the hand is in contact with the screen 12, and the process proceeds to step S15.

On the other hand, when the number of pixels below the threshold is not less than the number of pixels above the threshold (when the number of pixels above the threshold is greater than or equal to the number of pixels below the threshold). It is determined that the area where the hand is in contact with the screen 12 is small, and the process does not proceed to the next step S15, but proceeds to step S20, resets the counter and waits. The threshold value is set as calibration or an arbitrary value.
Next, in step S <b> 15, the image processing program 206 recognizes the position of the pixel that is grounded to the edge of the screen 12 among the pixels constituting the difference image. In addition, an intermediate point between successive pixels at the edge is recognized as the position of the user M. Next, in step S <b> 16, the position of the tip portion of the shadow image is recognized as the finger position and stored in the RAM 102.

FIG. 15 is a flowchart showing a procedure for acquiring the size and position of the shadow of the fingertip in step S16 of FIG. First, in step S <b> 61, the image processing program 206 reads a shadow image from the RAM 102. Next, in step S 62, the position of the tip of the shadow farthest from the position of the user M is recognized as the finger position and stored in the RAM 102.
Next, in step S63, the luminance value of each pixel at the tip of the shadow is compared with a threshold value to determine whether the luminance value is greater than the threshold value. As a result, when it is determined that the luminance value is larger than the threshold value, the process proceeds to step S64, and 1 is added to the value of the size counter. On the other hand, when it is determined that the luminance value is not greater than the threshold value, addition to the size counter is not performed. By executing the processing of step S63 and step S64 for all the pixels, the area of the tip is obtained as the value of the size counter.
Next, returning to step S17 in FIG. 8, the image processing program 206 determines whether or not the area of the tip portion of the shadow image is larger than the reference value C1 and smaller than the reference value C2. As a result, when it is determined that the area of the tip portion of the shadow image is larger than the reference value C1 and smaller than the reference value C2, the image processing program 206 recognizes that a certain point is pointed, and the next step S18. Proceed to the process.

On the other hand, when it is determined that the area of the tip portion is smaller than the reference value C1 or larger than the reference value C2, the process does not proceed to the next step S18 but proceeds to step S20, and the image processing program 206 Reset the counter and wait.
In step S18, the image processing program 206 compares the position of the tip recognized this time with the position of the tip recognized last time, and determines whether or not the position of the tip recognized this time is equal to the position of the tip recognized last time. judge. As a result, if the image processing program 206 determines that the position of the tip recognized last time and the position of the tip recognized this time are substantially equal, the process proceeds to the next step S19. On the other hand, if it is determined that the position of the tip recognized last time is not equal to the position of the tip recognized this time, it is determined that there is no intention of the user M, and the process proceeds to step S20 without proceeding to the next step S19. Reset the counter and wait.
In step S19, the image processing program 206 adds 1 to the counter value. Next, in step S21, the counter value is compared with the reference value B, and it is determined whether or not the counter value is greater than the reference value B. As a result, if it is determined that the counter value is greater than the reference value B, the process proceeds to step S22. On the other hand, if it is determined that the value of the counter is equal to or smaller than the reference value B, since the intention cannot be determined at this point, the process waits. B is set as an arbitrary value, and this process ends. This is the operation range of the image processing program 206.
In step S22, a window activation process is performed. The window activation process is performed by the window control 202.

FIG. 16 is a flowchart showing a procedure for activating a window. With the processing in FIG. 8, the image processing program 206 estimates the position of the finger of the user M from the shadow image and passes the position to the window control 202.
In step S <b> 71, the window control 202 acquires the position of the finger of the user M passed from the image processing program 206. Next, in step S72, the window control 202 determines whether or not a window exists at the position of the finger of the user M. As a result, if it is determined that a window exists at the position of the finger, the process proceeds to step S73, where the window is set as the target window and activated. On the other hand, if it is determined in step S72 that there is no window at the position of the finger, the window activation process is not performed and the process ends.

(Second Embodiment)
Next, a method for moving an activated window with a finger will be described as a second embodiment. FIG. 17 is a schematic diagram showing a method of moving an activated window with a finger.
The position P1 of the user M and the position P2 of the fingertip of the hand are acquired by the procedure shown in the first embodiment. By moving the finger while touching the screen 12, the activated window W3 can be moved.
FIG. 18 is a flowchart showing a processing procedure when moving a window. First, in step S81, the image processing program 206 compares the image acquired from the CCD camera 16 through the image capturing unit and the reference image stored in the RAM 102, acquires a difference image, and calculates its size.
FIG. 9 is a flowchart showing the difference image acquisition procedure. First, in step S <b> 31, the image processing program 206 reads a reference image from the RAM 102. Next, in step S32, the current image is acquired from the CCD camera 16. Next, in step S33, a difference in luminance at the same position between the reference image and the current image is calculated. Next, in step S <b> 34, the calculated difference image is stored in the RAM 102.

FIG. 13 is a flowchart showing a shadow size acquisition procedure. First, in step S <b> 41, the image processing program 206 reads the difference image from the RAM 102. Next, in step S42, the luminance value of each pixel is compared with a threshold value to determine whether the luminance value of each pixel is greater than the threshold value. As a result, when it is determined that the luminance value of each pixel is larger than the threshold value, the process proceeds to step S43, and only 1 is added to the value of the size counter. In step S44, the luminance value of the corresponding part is stored in the RAM 102 as a shadow image (difference image). On the other hand, if it is determined in step S42 that the luminance value of each pixel of the difference image read from the RAM 102 is equal to or less than the threshold value, the process ends without performing any processing.
Next, returning to step S82 in FIG. 18, it is determined whether or not the pixel area of the shadow image (difference image) is larger than the reference value A1 and smaller than the reference value A2. As a result, when it is determined that the pixel area of the difference image is larger than the reference value A1 and smaller than the reference value A2, the image processing program 206 recognizes that the user M is operating, and The process proceeds to step S83. In step S83, a shadow area is extracted using a threshold value.
On the other hand, when it is determined that the pixel area of the difference image is smaller than the reference value A1 or larger than the reference value A2, it is determined that an error or the user M is performing another operation, and the process proceeds to step S90. , Reset the counter and activate state and wait. The reference value A1 and the reference value A2 are set by calibration or are set as arbitrary values. Since the calibration means is as described in the first embodiment, its description is omitted here.

FIG. 14 is a flowchart showing a procedure for separating the size of the shadow image according to the threshold value. First, in step S <b> 51, the image processing program 206 reads a shadow image from the RAM 102. Next, in step S52, the image processing program 206 compares the luminance value of each pixel constituting the shadow image with a threshold value, and determines whether the luminance value of each pixel is larger than the threshold value.
As a result, when it is determined that the luminance value of the pixel is larger than the threshold value (the value is above), 1 is added to the value of the threshold value upper shadow counter. Add 1 to the counter value. Note that the luminance value of the pixel is, for example, a value of 256 gradations. The above determination is made for all the pixels constituting the shadow image, and the value of the upper threshold shadow counter and the lower threshold counter are set.
Next, returning to step S84 in FIG. 18, the number of pixels below the threshold (pixels with a luminance value smaller than the threshold) is smaller than the number of pixels above the threshold (pixels with a luminance value greater than the threshold). In this case, the image processing program 206 determines that the area where the hand is in contact with the screen 12 is large, and proceeds to the processing of the next step S85.

On the other hand, if it is determined that the number of pixels below the threshold (pixels whose luminance value is smaller than the threshold) is larger than the number of pixels above the threshold (pixels whose luminance value is larger than the threshold), the hand is on the screen. 12, it is determined that the area that is in contact with the ground is small, and the process does not proceed to the next step S <b> 85 but proceeds to step S <b> 90 to reset the counter and the activated state and wait. The threshold value is set as calibration or an arbitrary value.
Next, in step S85, the image processing program 206 recognizes a position of the pixels of the difference image that are in contact with the edge of the screen 12. Next, the image processing program 206 recognizes an intermediate point between successive pixels at the edge as the position of the user M. Next, in step S <b> 86, the position of the tip portion of the shadow image is recognized as the finger position and stored in the RAM 102.

FIG. 15 is a flowchart showing a procedure for acquiring the size and position of the shadow of the fingertip in step S86 of FIG. First, in step S <b> 61, the image processing program 206 reads a shadow image from the RAM 102. Next, in step S 62, the position of the tip of the shadow farthest from the position of the user M is recognized as the finger position and stored in the RAM 102.
Next, in step S63, the luminance value of each pixel at the tip of the shadow is compared with a threshold value to determine whether the luminance value is greater than the threshold value. As a result, when it is determined that the luminance value is larger than the threshold value, the process proceeds to step S64, and 1 is added to the value of the size counter. On the other hand, when it is determined that the luminance value is not greater than the threshold value, addition to the size counter is not performed. By executing the processing of step S63 and step S64 for all the pixels, the area of the tip is obtained as the value of the size counter.
Next, returning to step S87 in FIG. 18, the image processing program 206 determines whether the area of the tip portion of the shadow image is larger than the reference value C1 and smaller than the reference value C2. As a result, when it is determined that the area of the tip portion of the shadow image is larger than the reference value C1 and smaller than the reference value C2, the image processing program 206 recognizes that a certain point is pointed, and the next step S88. Proceed to the process.

On the other hand, when it is determined that the area of the tip portion is smaller than the reference value C1 or larger than the reference value C2, the process does not proceed to the next step S88 but proceeds to step S90, and the image processing program 206 Reset the counter and activate status and wait.
In step S88, the image processing program 206 compares the tip position recognized this time with the tip position recognized last time, and determines whether or not the tip position recognized this time and the tip position recognized last time are equal. judge. As a result, if it is determined by the image processing program 206 that the position of the tip recognized last time is substantially equal to the position of the tip recognized this time, the process proceeds to the next step S91.
On the other hand, if it is determined that the position of the tip recognized last time is not equal to the position of the tip recognized this time, it is determined that there is no intention of the user M, and the process proceeds to step S89 without proceeding to the next step S91. . In step S89, it is determined whether the window has been activated. As a result, when it is determined that it has not been activated, the process proceeds to step S90, and the image processing program 206 resets the counter and the activated state and waits.
On the other hand, if it is determined that it has been activated, the process proceeds to step S92, and the tip position and the identification information of the window to be moved are sent to the window control 202 in order to move the window to the tip position recognized previously. The window control 202 that has received the information from the image processing program 206 moves the corresponding window to the tip position.

In step S91, the image processing program 206 adds 1 to the counter value. Next, in step S93, the counter value is compared with the reference value B to determine whether or not the counter value is greater than the reference value B. As a result, if it is determined that the counter value is greater than the reference value B, the process proceeds to step S94. On the other hand, if it is determined that the value of the counter is equal to or smaller than the reference value B, since the intention cannot be determined at this point, the process waits. The reference value B is set as an arbitrary value. Thereafter, this process is terminated. This is the operation range of the image processing program 206.
In step S94, a window activation process is performed. The window activation process is performed by the window control 202, and the activation state is saved.

(Third embodiment)
Next, a procedure for starting a program will be described as a third embodiment. FIG. 19 is a conceptual diagram showing a method for starting a program. The position P1 of the user M and the position P2 of the fingertip of the hand are acquired by the method described in the first embodiment.
If the window control 202 determines that the position touched with the fingertip for a certain period of time is clickable, the category is sequentially displayed as in the case of the mouse click operation, and the target program is started by selecting the category. can do. Of course, the method of recognizing the fingertip, pointing when it stops for a certain period of time, and executing it when the fingertip is released can be used not only for starting the program but also for other purposes such as menu operations associated with windows. Yes.

FIG. 20 is a flowchart showing a program start processing procedure. First, in step S <b> 101, the image processing program 206 acquires a difference image between an image acquired from the camera 16 through an image capturing unit and a reference image stored in the RAM 102.
FIG. 9 is a flowchart illustrating a procedure for acquiring a difference image. First, in step S <b> 31, the image processing program 206 reads a reference image from the RAM 102. Next, in step S32, a current image captured by the CCD camera 16 is acquired. Next, in step S33, the image processing program 206 calculates a difference in luminance at the same position between the reference image and the current image. Next, in step S34, the difference image obtained by the calculation is supplied to the RAM 102 and stored.

FIG. 13 is a flowchart showing a shadow size acquisition procedure. First, in step S41, the image processing program 206 reads from the RAM 102 the difference image stored in the RAM 102 in step S34 of FIG. Next, in step S42, it is determined whether or not the luminance value of each pixel of the difference image read from the RAM 102 is greater than a threshold value.
As a result, if the luminance of each pixel of the difference image is larger than the threshold value, the process proceeds to step S43, the size counter is added, and the luminance of the corresponding part is stored in the RAM 102 as a shadow image in step S44. For example, the threshold is about 40 for 256 gradations. On the other hand, if the luminance value of each pixel of the difference image read from the RAM 102 is equal to or less than the threshold value in step S42, no processing is performed and the process ends.

Next, returning to step S102 in FIG. 20, it is determined whether or not the pixel area of the shadow image (difference image) is larger than the reference value A1 and smaller than the reference value A2. As a result, when it is determined that the pixel area of the difference image is larger than the reference value A1 and smaller than the reference value A2, the image processing program 206 recognizes that the user M is operating, and The process proceeds to step S103. In step S103, a shadow area is extracted using a threshold value.
On the other hand, when it is determined that the pixel area of the difference image is smaller than the reference value A1 or larger than the reference value A2, it is determined that an error or the user M is performing another operation, and the process proceeds to step S110. , Reset the counter and activate state and wait. The reference value A1 and the reference value A2 are set by calibration or are set as arbitrary values. Since the calibration means is as described in the first embodiment, its description is omitted here.

FIG. 14 is a flowchart showing a procedure for separating the size of the shadow image according to the threshold value. First, in step S <b> 51, the image processing program 206 reads a shadow image from the RAM 102. Next, in step S52, the image processing program 206 compares the luminance value of each pixel constituting the shadow image with a threshold value, and determines whether the luminance value of each pixel is larger than the threshold value.
As a result, when it is determined that the luminance value of the pixel is larger than the threshold value (the value is above), 1 is added to the value of the threshold value upper shadow counter. Add 1 to the counter value. Note that the luminance value of the pixel is, for example, a value of 256 gradations. The above determination is made for all the pixels constituting the shadow image, and the value of the upper threshold shadow counter and the lower threshold counter are set.
Next, returning to step S104 in FIG. 20, the number of pixels below the threshold (pixels with a luminance value smaller than the threshold) is smaller than the number of pixels above the threshold (pixels with a luminance value greater than the threshold). In this case, the image processing program 206 determines that the area where the hand is in contact with the screen 12 is large, and proceeds to the processing of the next step S105.

On the other hand, if it is determined that the number of pixels below the threshold (pixels whose luminance value is smaller than the threshold) is larger than the number of pixels above the threshold (pixels whose luminance value is larger than the threshold), the hand is on the screen. 12, it is determined that the area that is in contact with the ground is small, and the process does not proceed to the next step S <b> 105, but proceeds to step S <b> 110 to reset the counter and the activated state and wait. The threshold value is set as calibration or an arbitrary value.
Next, in step S <b> 105, the image processing program 206 recognizes the position of the pixels of the difference image that are in contact with the edge of the screen 12. Next, the image processing program 206 recognizes an intermediate point between successive pixels at the edge as the position of the user M. Next, in step S <b> 106, the position of the tip portion of the shadow image is recognized as the finger position and stored in the RAM 102.

FIG. 15 is a flowchart showing a procedure for acquiring the size and position of the shadow of the fingertip in step S106 of FIG. First, in step S <b> 61, the image processing program 206 reads a shadow image from the RAM 102. Next, in step S 62, the position of the tip of the shadow farthest from the position of the user M is recognized as the finger position and stored in the RAM 102.
Next, in step S63, the luminance value of each pixel at the tip of the shadow is compared with a threshold value to determine whether the luminance value is greater than the threshold value. As a result, when it is determined that the luminance value is larger than the threshold value, the process proceeds to step S64, and 1 is added to the value of the size counter. On the other hand, when it is determined that the luminance value is not greater than the threshold value, addition to the size counter is not performed. By executing the processing of step S63 and step S64 for all the pixels, the area of the tip is obtained as the value of the size counter.
Next, returning to step S107 in FIG. 20, the image processing program 206 determines whether the area of the tip portion of the shadow image is larger than the reference value C1 and smaller than the reference value C2. As a result, when it is determined that the area of the tip portion of the shadow image is larger than the reference value C1 and smaller than the reference value C2, the image processing program 206 recognizes that a certain point is pointed, and the next step S108. Proceed to the process.

On the other hand, when it is determined that the area of the tip portion is smaller than the reference value C1 or larger than the reference value C2, the process does not proceed to the next step S88 but proceeds to step S110, and the image processing program 206 Reset the counter and activate status and wait.
In step S108, the image processing program 206 compares the position of the tip recognized this time with the position of the tip recognized last time, and determines whether or not the position of the tip recognized this time is equal to the position of the tip recognized last time. judge. As a result, if the image processing program 206 determines that the position of the tip recognized last time and the position of the tip recognized this time are approximately equal, the process proceeds to the next step S109.
On the other hand, if it is determined that the position of the tip recognized last time is not equal to the position of the tip recognized this time, it is determined that the user M has no intention, and the process proceeds to step S110 without proceeding to the next step S109. .

In step S109, the image processing program 206 adds 1 to the counter value. Next, in step S111, the value of the counter is compared with the reference value B, and it is determined whether or not the value of the counter is larger than the reference value B. As a result, when it is determined that the counter value is greater than the reference value B, the process proceeds to step S112. On the other hand, if it is determined that the value of the counter is equal to or smaller than the reference value B, since the intention cannot be determined at this point, the process waits. The reference value B is set as an arbitrary value. Thereafter, this process is terminated.
In step S112, the window control 202 determines whether or not the target location is clickable. If it is determined by the window control 202 that it is clickable, the clicked result is displayed and the program is activated. On the other hand, if it is determined that it cannot be clicked, it waits. Thereafter, this process is terminated.

(Fourth embodiment)
Next, a method for drawing the target window will be described. FIG. 21 and FIG. 22 are conceptual diagrams showing the target window drawing method. In the method shown in FIG. 21, the target window W4 is drawn together with the window W4. In the method shown in FIG. 22, the scroll bars B1 and B2 of the target window W4 are drawn by scrolling. These pulling methods can be selected depending on the state of the window W4, the position of the hand of the user M, and the like.
FIG. 23 is a flowchart showing the procedure of the target window drawing process. The drawing of the target window W4 is a process performed when the position of the hand of the user M is away from the screen 12 and stays at the same place for a certain time.
First, in step S121, an image difference and a shadow size are acquired. That is, a difference image (image difference) is acquired by the image processing program 206 from the image acquired from the CCD camera 16 via the image capturing means (USB device controller 201) and the reference image stored in the RAM 102.

FIG. 9 is a flowchart illustrating a procedure for acquiring a difference image. First, in step S <b> 31, the image processing program 206 reads a reference image from the RAM 102. FIG. 10 shows an example of a reference image. Next, in step S32, a current image captured by the CCD camera 16 is acquired. FIG. 11 shows an image captured when the user touches the screen 12 while touching it.
Next, in step S33, the image processing program 206 calculates a difference in luminance at the same position between the reference image and the current image. FIG. 12 shows a difference image obtained by calculating the difference between the reference image shown in FIG. 10 and the current image shown in FIG. Next, in step S34, the difference image obtained by the calculation is supplied to the RAM 102 and stored.

FIG. 13 is a flowchart showing a shadow size acquisition procedure. First, in step S41, the image processing program 206 reads from the RAM 102 the difference image stored in the RAM 102 in step S34 of FIG. Next, in step S42, it is determined whether or not the luminance of each pixel of the difference image read from the RAM 102 is equal to or higher than a threshold value.
As a result, when it is determined that the luminance of each pixel of the difference image is equal to or higher than the threshold value, the process proceeds to step S43, the size counter is added, and the luminance of the corresponding part is stored in the RAM 102 as a shadow image in step S44. For example, the threshold is about 40 for 256 gradations. On the other hand, if it is determined in step S42 that the luminance of each pixel of the difference image read from the RAM 102 is equal to or lower than the threshold value, no processing is performed and the process ends.
Thereafter, the process returns to step S122 in FIG. 23, and the image processing program 206 determines whether or not the area of the shadow image pixel is larger than the reference value A1 and smaller than the reference value A2, and the shadow image pixel area is determined. Is larger than the reference value A1 and smaller than the reference value A2, it is recognized that the user M is operating, and the process proceeds to step S123. In step S123, a shadow region is extracted using a threshold value.

On the other hand, if it is determined that the pixel area of the difference image (pixel area of the shadow image) is smaller than the reference value A1 or larger than the reference value A2, the process does not proceed to step S123, and an error or user It is determined that M is performing another operation, and the process proceeds to step S129 to reset the counter and wait.
The reference values A1 and A2 are set by calibration or are set as arbitrary values. Although not shown, as a calibration method, shadow information when a hand is held without touching the screen 12 and when a hand is placed on the screen 12 is read, and the touch information is not touched on the screen 12. The threshold value A1 and the threshold value A2 are set by the calibration program so that the area of the shadow that is recognized is almost eliminated when the hand is held over and the area of the shadow that is recognized when the hand is placed on the screen 12 is present. Stored in the RAM 102, the HDD 103, and the like.

FIG. 14 is a flowchart showing a procedure for separating the size of the shadow image according to the threshold value. First, in step S <b> 51, the image processing program 206 reads a shadow image from the RAM 102. Next, in step S52, the image processing program 206 compares the luminance value of each pixel constituting the shadow image with a threshold value, and determines whether the luminance value of each pixel is larger than the threshold value.
As a result, when it is determined that the luminance value of the pixel is larger than the threshold value (the value is above), 1 is added to the value of the threshold value upper shadow counter. Add 1 to the counter value. Note that the luminance value of the pixel is, for example, a value of 256 gradations. The above determination is made for all the pixels constituting the shadow image, and the value of the upper threshold shadow counter and the lower threshold counter are set.
Next, returning to step S124 in FIG. 23, it is determined whether or not the number of pixels below the threshold is smaller than the number of pixels above the threshold, and the number of pixels below the threshold is If it is determined that the number is lower than the number of pixels above, it is determined by the image processing program 206 that there is a large area where the hand is in contact with the screen 12, and the process proceeds to step S125.

On the other hand, when the number of pixels below the threshold is not less than the number of pixels above the threshold (when the number of pixels above the threshold is greater than or equal to the number of pixels below the threshold). It is determined that the area where the hand is in contact with the screen 12 is small, and the process does not proceed to the next step S125, but proceeds to step S129, resets the counter and waits. The threshold value is set as calibration or an arbitrary value.
Next, in step S125, the image processing program 206 recognizes the position of the pixel that is grounded to the edge of the screen 12 among the pixels constituting the difference image. In addition, an intermediate point between successive pixels at the edge is recognized as the position of the user M. Next, in step S126, the position of the tip portion of the shadow image is recognized as the finger position and stored in the RAM 102.

FIG. 15 is a flowchart showing a procedure for acquiring the size and position of the shadow of the fingertip in step S126 of FIG. First, in step S <b> 61, the image processing program 206 reads a shadow image from the RAM 102. Next, in step S 62, the position of the tip of the shadow farthest from the position of the user M is recognized as the finger position and stored in the RAM 102.
Next, in step S63, the luminance value of each pixel at the tip of the shadow is compared with a threshold value to determine whether the luminance value is greater than the threshold value. As a result, when it is determined that the luminance value is larger than the threshold value, the process proceeds to step S64, and 1 is added to the value of the size counter. On the other hand, when it is determined that the luminance value is not greater than the threshold value, addition to the size counter is not performed. By executing the processing of step S63 and step S64 for all the pixels, the area of the tip is obtained as the value of the size counter.
Next, returning to step S127 in FIG. 23, the image processing program 206 compares the position of the tip recognized this time with the position of the tip recognized last time, and the position of the tip recognized this time and the position of the tip recognized last time are compared. Whether or not are equal. As a result, if the image processing program 206 determines that the position of the tip recognized last time and the position of the tip recognized this time are substantially equal, the process proceeds to the next step S128.

On the other hand, if it is determined that the position of the tip recognized last time is not equal to the position of the tip recognized this time, it is determined that there is no intention of the user M, and the process does not proceed to the next step S128 but proceeds to step S129. Reset the counter and wait.
In step S128, the image processing program 206 adds 1 to the counter value. Next, in step S130, the counter value is compared with the reference value B, and it is determined whether or not the counter value is greater than the reference value B. As a result, when it is determined that the counter value is greater than the reference value B, the process proceeds to step S131. On the other hand, if it is determined that the value of the counter is equal to or smaller than the reference value B, since the intention cannot be determined at this point, the process waits. B is set as an arbitrary value, and this process ends. This is the operation range of the image processing program 206.
Next, in step S131, window drawing processing is performed. FIG. 24 is a flowchart showing window drawing selection processing. Through the processing of the flowchart shown in FIG. 23, the image processing program 206 estimates the position of the hand of the user M from the shadow image, and passes the position of the hand of the user M as a value to the window control 202.

FIG. 25 is a conceptual diagram showing a method for designating a target window. As shown in FIG. 25, the window control 202 sets a target window to a window W4 on an extension line of the line extending from the position P1 of the user M to the position P2 of the hand. If there is no target window, the drawing selection process is terminated.
When the window control 202 determines that the target window has a scroll bar, the position P1 estimated to be a hand is compared with the position of the window W4. When the place P1 recognized as the position of the hand and the window W4 overlap, the window control 202 confirms whether or not there is room for the scroll bars B1 and B2 to scroll toward the user M. If there is room, the window W4 is scrolled (FIG. 22). When there is no room, the window control 202 performs an operation of pulling the window (FIG. 21).
If the hand and the window W4 do not overlap, the window control 202 draws the window (FIG. 21). When the window W4 does not have the scroll bars B1 and B2 and the position of the hand does not overlap the window W4, the window control 202 pulls the window W4 (FIG. 21). When the hand position and the window overlap, the window control 202 scrolls the window W4 (FIG. 22).

FIG. 25 is a conceptual diagram illustrating a method of detecting a certain direction of the target window. The window control 202 assumes that the circle mark in front is recognized as the position P1 of the user M, and the circle mark right in the center of the screen is recognized as the hand position P2 by the image processing program 206. The window W4 on the straight line connecting the position P2 is recognized as the target window.
FIG. 26 shows a reference image, and FIG. 27 shows an actual image when the hand is away from the screen 12. FIG. 28 shows a difference image obtained by taking the difference between the actual image when the hand is separated from the screen 12 shown in FIG. 27 and the reference image shown in FIG. In FIG. 28, the point P1 represents the position of the user M, and the point P2 represents the position of the fingertip.

FIG. 29 shows an actual image when the pen is written on the screen 12 and the hand is in contact with the screen 12. FIG. 30 shows the case where the hand shown in FIG. 29 is in contact with the screen 12. The difference image which took the difference of an actual image and the reference image shown in FIG. 26 is shown. In FIG. 30, a point P1 represents the position of the user M, and a point P2 represents the position of the fingertip.
As described above, the image processing program 206 can acquire the shadow of the hand based on the difference from the reference image, and can know the state of the hand of the user M based on the luminance. Thereby, interaction can be performed through the window control 202.
As described above, in each embodiment, the intention of the user M can be determined based on the shadow of the user M falling on the screen 12. The point of the present embodiment is the determination of the operation of the user M based on the shape analysis of the shadow falling on the diffusion screen 12 and the time transition using the counter.
By using the diffusing screen 12, the brightness of the shadow changes depending on the distance between the screen 12 and the hand of the user M. Further, by acquiring the user M's intention that the operation of the user M is temporarily stopped as a result, the malfunction can be prevented. For example, when activating overlapping windows, by pointing the fingertip on the target window, the shape can be read by the system and activated after a certain period of time.

Further, when moving the window, the window is moved by moving the fingertip while touching the activated window to the target location. The operation of sliding on the screen 12 with a finger while touching can also be used for starting a program.
The program can be started by tracing the position on the screen 12 by recognition including a fingertip pause operation and finally releasing the finger from the screen 12. Of course, it is possible not only to start the program but also to operate the window menu by the same operation.
Alternatively, when the user M reaches out and pauses in the air, the user M determines that he wants to use the area at the tip of the hand and draws the area. As a method for drawing an area, a method of “drawing a window” or “scrolling a window” can be considered. Which drawing method is selected is determined by determining what form is displayed in the target region.
The target area may be determined by identifying the position of the user M using a shadow and determining that the area is the area where the position of the user M and the shadow of the hand are connected by a line. After the area that the user M wants to use is drawn, the user M can approach the screen 12 to determine the difference in the shadow of the hand and stop the drawing.
It should be noted that the configuration and operation of the above-described embodiment are examples, and it goes without saying that they can be changed as appropriate without departing from the spirit of the present invention.

  The present invention can be applied not only to a projection-type image display device but also to other various display devices.

It is a figure which shows the structural example of 1st Embodiment of the image display apparatus with which this invention is applied. It is a perspective view which shows the usage condition of an image display apparatus. It is a block diagram which shows the structural example of information processing equipment. It is a figure which shows the structural example of software. It is a figure which shows the process sequence of each means. It is a figure which shows the present condition of a desktop. It is a figure which shows the conceptual diagram of activation of a window. It is a flowchart which shows the process sequence of activation. It is a flowchart which shows the procedure which acquires a difference image. It is a figure which shows the example of a reference image. It is a figure which shows the example of the image imaged when it earth | grounds and points at the screen. It is a figure which shows the example of a difference image. It is a flowchart which shows the size acquisition procedure of a shadow. It is a flowchart which shows the procedure which classifies the size of a shadow image according to a threshold value. It is a flowchart which shows the procedure which acquires the size and position of the shadow of a fingertip. It is a flowchart which shows the procedure which activates a window. It is a schematic diagram which shows the method of moving the activated window with a finger | toe. It is a flowchart which shows the process sequence when moving a window. It is a conceptual diagram which shows the method of starting a program. It is a flowchart which shows the process sequence of a program starting. It is a conceptual diagram which shows the drawing method of an object window. It is a conceptual diagram which shows the drawing method of an object window. It is a flowchart which shows the procedure of the drawing process of an object window. It is a flowchart which shows the drawing selection process of a window. It is a conceptual diagram which shows the method of designating a target window. It is a figure which shows a reference image. It is a figure which shows an actual image when a hand is separated from the screen. It is a figure which shows the example of a difference image. It is a figure which shows the actual image when writing with a pen on the screen and the hand is touching the screen. It is a figure which shows the example of a difference image.

Explanation of symbols

  DESCRIPTION OF SYMBOLS 1 Image display apparatus, 11 Writing surface, 12 Screen, 13 Projection surface, 14 Housing | casing, 15 Projector, 16 CCD camera, 100 Information processing apparatus, 101 CPU, 102 RAM, 103 ROM, 104 Display controller, 105 USB controller, 200 Basic software, 201 USB device driver, 202 Window control, 203 Display control driver, 204 Setting value, 205 Application software, 206 Image processing program, 207 Various applications

Claims (12)

A projection plane;
A projection device for projecting a predetermined image onto the projection plane;
Imaging means for imaging an image projected on the projection plane with an imaging unit in which pixels are two-dimensionally arranged;
A time measuring means for measuring time;
Detecting means for detecting a shadow of a user's finger included in the image picked up by the image pickup means and detecting a pointing position corresponding to the shadow;
An image projection display device comprising: processing means for executing processing corresponding to the stationary time measured by the time measuring means for the shadow detected by the detection means. The image projection display device further includes a threshold initial setting unit that stores a predetermined threshold used when the detection unit detects a shadow,
The threshold initial setting means captures a state in which a shadow is generated in a predetermined range of the image projected on the projection plane by the imaging means, and the brightness of the shadow within the predetermined range of the captured image The image projection display device according to claim 1, wherein a value is determined as the predetermined threshold value.   The processing means compares the brightness value of the shadow imaged by the imaging means with the threshold value stored in the threshold value initial setting means to determine whether or not the shadow is for pointing purposes. The image projection display device according to claim 2.   The image projection display device according to claim 2, further comprising an activating unit that activates a window that has not been activated when the shadow detected to be a pointing object is stationary for a predetermined time. .   The image projection display apparatus according to claim 4, further comprising: a window moving unit that moves the window by moving the pointing position detected by the detecting unit after activating the window.   The image according to claim 4, further comprising: a click determination unit that determines that a click has been made when a pointing operation is stopped when there is an object that can be clicked at the pointing position detected by the detection unit. Projection display device.   When the shadow stillness is detected in a state that does not have a sufficient luminance value such that the pointing position is not detected, the detected user's position and the window at the destination connecting the pointing position are drawn. The image projection display device according to claim 1.   In the state where the shadow position is not detected in a state where the pointing position is not detected, and when the stillness of the shadow is detected, the detected user position and the scroll bar of the window at the destination of the pointing position are drawn. The image projection display device according to claim 1.   9. The image projection display device according to claim 7, wherein when the pointing operation is detected during the pulling of the window or the scroll bar, the pulling of the window or the scroll bar is immediately stopped. A projection step of projecting a predetermined image onto the projection plane by the projection device;
An imaging step of imaging an image projected on the projection plane by an imaging unit with an imaging unit in which pixels are two-dimensionally arranged;
A timing step for measuring time by means of timing;
A detecting step of detecting a shadow of a user's finger included in the image picked up by the image pickup means by a detecting means and detecting a pointing position corresponding to the shadow;
An image projection display method comprising: a processing step of executing a process corresponding to a time when the shadow detected in the detection step measured in the timing step is stationary by a processing unit. On the computer,
A projecting step of projecting a predetermined image onto a projection surface;
An imaging step of imaging an image projected on the projection plane with an imaging unit in which pixels are two-dimensionally arranged;
A timing step to measure time;
A detection step of detecting a shadow of a user's finger included in the image captured by the imaging means and detecting a pointing position corresponding to the shadow;
And a processing step of executing a process corresponding to a time when the shadow detected in the detection step measured in the timing step is stationary.   The computer-readable recording medium which recorded the program of Claim 11.

Images