JP2012181721A - Position input device, projector, control method for projector, and display system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To more accurately specify an indicated position when an operation for indicating, with an indicator, a position on a projection surface on which an image is being projected by a projector is performed.SOLUTION: A projector 11 acquires an original image and a captured image obtained by capturing an image projected on a screen SC by a projector 11 based on the original image, detects an image of an indicator 12 for indicating a position on the screen SC and an image of a shadow of the indicator 12 from the captured image based on the acquired original image, and specifies the indication position indicated by the indicator 12 and an indication state based on a relative position of the image of the indicator 12 and the image of the shadow of the indicator 12 that are detected.

Description

  The present invention relates to a position input device that detects an instructed position, a projector, a projector control method, and a display system.

  2. Description of the Related Art Conventionally, there has been known an apparatus that detects an indicated position when a position on a projection surface on which an image is projected by a projector is designated (see, for example, Patent Document 1). The apparatus described in Patent Document 1 detects the position of the tip of the indicator from a photographed image obtained by photographing the projection surface from the front when the position is instructed by an indicator such as a stick on the projection surface during projection, The indicated position is specified.

Japanese Patent Laid-Open No. 2000-200169

By the way, the whole image of the indicator such as a stick is often shown in the photographed image of the projection surface, and the position of the indicator pointed by the person operating the indicator is specified. It is not easy to do. In order to solve this problem, the apparatus described in Patent Literature 1 specifies the indicated position by detecting the tip from the captured image. However, when an object having no sharp tip such as a finger is used as the indicator, it is difficult to detect the tip from the captured image. Further, when such an object is used as an indicator, the indicated position is not necessarily the tip position.
The present invention has been made in view of the above-described circumstances, and when an operation for indicating a position on a projection surface on which an image is projected by a projector is performed with an indicator, the indicated position is more accurately detected. The purpose is to specify.

In order to solve the above problems, the present invention acquires an original image and an image acquisition unit that acquires a captured image obtained by capturing an image projected by a projector on a projection surface based on the original image, and the image acquisition unit acquires the original image. Based on the original image, an indicator image detecting means for detecting an image of a pointer indicating a position on the projection surface and a shadow image of the indicator from a photographed image, and the indicator image detecting means And an instruction state determination unit that specifies an instruction position and an instruction state indicated by the indicator based on a relative position between the detected image of the indicator and a shadow image of the indicator.
According to the present invention, by detecting the image of the indicator that indicates the position on the projection surface and the shadow image of the indicator, for example, whether the indicator is in contact with the projection surface along with the indicated position. Etc. can be specified. Thereby, the instructed position can be accurately specified regardless of the shape of the indicator. In addition, it is possible to more accurately reflect the intention of the person who operates the indicator and to specify the indicated position.

Further, according to the present invention, in the position input device, the indication state determination unit is configured to determine the indication based on a distance between the indicator image detected by the indicator image detection unit and a shadow image of the indicator. It is determined whether or not a body is in contact with the projection surface.
According to the present invention, by determining whether or not the indicator is in contact with the projection surface, it is possible to accurately specify the indicated position regardless of the shape of the indicator.

In the position input device according to the present invention, the image acquisition unit acquires a captured image captured from a direction other than the front of the projection surface.
Here, the front refers to the direction facing the projection surface from the position facing the projection surface on the front surface side of the projection surface, and the directions other than the front include the above, below the projection surface, excluding the front surface, It includes the left direction, the right direction, and the diagonal direction between them. In other words, the direction other than the front refers to a direction in which the axis of the lens is not perpendicular to the projection plane when the photographing means for photographing a photographed image has a lens.
According to the present invention, the position pointed to by the indicator is specified using a photographed image photographed from a direction other than the front of the projection surface, so that it is not necessary to install a photographing device or the like on the front of the projection surface, and the image is projected. There is an advantage that it is easy to install the projector. Furthermore, since the indicator is not hidden behind the shadow of the operator standing in front of the projection surface as in the case of shooting from the front, the position of the indicator can be reliably detected.

According to the present invention, in the position input device, the indicator is provided with a predetermined pattern that can be identified by the indicator image detection means, and the indicator image detection means is acquired by the image acquisition means. The predetermined pattern is detected from a captured image.
According to the present invention, the indicator can be easily detected in the photographed image, and since it becomes easy to distinguish the indicator from the shadow of the indicator, it is also easy to detect the shadow of the indicator. Can be speeded up.

In order to solve the above problems, the present invention is a projector, which is attached to an original image and projects an image onto a projection surface, and images the image projected by the projection unit onto the projection surface. An image acquisition unit that acquires a photographing unit, an original image, and a captured image obtained by photographing the image projected by the projector based on the original image, and the original image acquired by the image acquisition unit. An indicator image detecting means for detecting an image of a pointer indicating a position on the projection surface and a shadow image of the indicator from a photographed image, and the indication detected by the indicator image detecting means An indication state determination unit that specifies an indication position and an indication state indicated by the indicator based on a relative position between a body image and a shadow image of the indicator is provided.
According to the present invention, the projector detects the image of the indicator that indicates the position on the projection surface and the shadow image of the indicator, so that, for example, the indicator contacts the projection surface together with the indicated position. An instruction state such as whether or not there is can be specified. Accordingly, the instructed position can be accurately specified using the projector that projects an image regardless of the shape of the indicator. In addition, it is possible to more accurately reflect the intention of the person who operates the indicator and to specify the indicated position.

In order to solve the above problem, the present invention projects an image on a projection surface based on an original image, captures an image projected on the projection surface based on the original image, acquires a captured image, Based on the original image, an image of a pointer indicating the position on the projection surface and a shadow image of the pointer are detected from the captured image, and the detected image of the pointer and the shadow of the pointer are detected. The pointing position and the pointing state pointed to by the pointer are specified based on the relative position to the image.
According to the present invention, the projector detects the image of the indicator that indicates the position on the projection surface and the shadow image of the indicator, so that, for example, the indicator contacts the projection surface together with the indicated position. An instruction state such as whether or not there is can be specified. Accordingly, the instructed position can be accurately specified using the projector that projects an image regardless of the shape of the indicator. In addition, it is possible to more accurately reflect the intention of the person who operates the indicator and to specify the indicated position.

In order to solve the above problems, the present invention provides a display system including a projector that projects an image on a projection surface and a photographing device that photographs the projection surface, the original image and the original image An image acquisition unit that acquires an image acquired by the imaging device capturing an image projected by the projector on the projection surface based on an image, and an image acquisition unit acquired by the image acquisition unit Based on the original image, an indicator image detecting means for detecting an image of a pointer indicating a position on the projection surface and a shadow image of the indicator from the photographed image, and the indicator image detecting means And an instruction state determination means for specifying an instruction position and an instruction state indicated by the indicator based on a relative position between the image of the indicator detected by the step and a shadow image of the indicator. That.
According to the present invention, by detecting the image of the indicator that indicates the position on the projection surface and the shadow image of the indicator, for example, whether the indicator is in contact with the projection surface along with the indicated position. Etc. can be specified. Thereby, the instructed position can be accurately specified regardless of the shape of the indicator. In addition, it is possible to more accurately reflect the intention of the person who operates the indicator and to specify the indicated position.

  According to the present invention, when an operation for indicating the position on the projection surface is performed by the indicator, the intention of the person who operates the indicator is more accurately reflected regardless of the shape of the indicator. Can be identified.

It is a figure which shows the structure of the display system which concerns on the 1st Embodiment of this invention. It is the perspective view which looked at the projector from the top | upper surface side. It is a block diagram which shows the functional structure of a display system. It is a figure which shows an example of the picked-up image image | photographed with the imaging device of the projector. It is a figure which shows the example of the picked-up image at the time of detecting the instruction | indication position and instruction | indication state by a pointer, (A) shows a picked-up image when a pointer is separated from a screen, (B) shows a pointer on a screen. A photographed image in the case of contact is shown, and (C) shows a photographed image in a case where the indicator is moved while being in contact with the screen. It is a flowchart which shows operation | movement of a projector. It is a figure which shows the example of a calibration, (A) shows operation of the indicator in calibration, (B) shows the pattern projected during execution of calibration. It is a figure which shows the structure of the display system which concerns on the 2nd Embodiment of this invention.

[First Embodiment]
Embodiments to which the present invention is applied will be described below with reference to the drawings.
FIG. 1 is a diagram illustrating a configuration of a display system 10 using a projector 11 according to the first embodiment. A part of the figure is enlarged and shown in a circle.
The projector 11 according to the present embodiment is wired or wirelessly connected to a PC (Personal Computer) 13 as an image output device, and projects an image corresponding to image information received from the PC 13 onto a screen SC as a projection surface. The projector 11 can project whether the image corresponding to the image information received from the PC 13 is a still image or a moving image. As shown in FIG. 1, the projector 11 is mainly in a state in which a side surface portion (a top surface portion 2 </ b> A to be described later) provided with a first opening 25 </ b> B through which projection light is emitted from the projector 11 is directed downward. In this embodiment, the arm 14 attached to the wall surface on which the screen SC is formed is used by fixing a side surface portion (a bottom surface portion 2B described later) opposite to the side surface portion. The The screen SC is not limited to a flat plate fixed to the wall surface, and the wall surface itself can be used as the screen SC.

  The projector 11 projects an image corresponding to the image information received from the PC 13. In addition, when a position input operation for designating a position with the indicator 12 is performed on the screen SC, the projector 11 identifies the position designated by the indicator 12 and relates to the position designated by the indicator 12. Information is transmitted to the PC 13. Based on the position information received from the projector 11, the PC 13 specifies a position instructed by the position input operation, and performs processing such as creating a new projection image by drawing the projected image on the position. The content of the position input operation is reflected on the projected image. In this way, the display system 10 functions as an interactive whiteboard system that projects an image on the screen SC and receives a position input operation by the indicator 12 on the screen SC and reflects the position input operation on the projected image. . Specifically, the PC 13 generates an image in which a line indicating the locus of the indication position of the indicator 12 is superimposed on an image of a presentation material or the like, and transmits image information of the generated image to the projector 11. As a result, the projector 11 projects an image in which a line indicating the locus of the indicator 12 is reflected on the screen SC.

The projector 11 includes a projection optical system 33 (FIG. 2) that projects an image and an imaging device 5 (FIG. 3) that captures the screen SC. The top surface 2A of the projector 11 is provided with a first opening 25B for emitting projection light from the projection optical system 33 in the projector 11 to the screen SC, and projects from the top surface 2A toward the screen SC obliquely below. An image is projected. An imaging optical system 51 (FIG. 2) of the imaging device 5 is disposed on the top surface portion 2A. The imaging device 5 displays the screen SC from almost the same direction as the projection optical system 33 projects a projection image. Take a picture.
The projector 11 specifies the position instructed by the position input operation by the indicator 12 based on the captured image captured by the imaging device 5.

In the present embodiment, the indicator 12 is a finger of an operator (presenter) as shown enlarged in the circle of FIG. 1, and a cap-shaped marker 12A is attached to the finger. The marker 12 </ b> A is provided with a predetermined pattern (pattern) so that an image can be easily detected from the captured image data. The pattern of the marker 12A only needs to be detectable from an image captured by the imaging device 5 using an image processing technique, and is preferably distinguishable from a projected image on the screen SC, a background such as a wall, or a floor. Specifically, there is a pattern in which a dark-colored frame having a predetermined shape is arranged and characters, symbols, figures, or the like are arranged in the frame. In the example of FIG. 1, a typical example is a pattern in which characters are arranged in a black square frame similar to a so-called AR marker. In addition, as the pattern of the marker 12A, a checkered pattern, a checkerboard pattern, a geometric pattern, or the like, which is a combination of two shades of light and shade, may be used.
The projector 11 detects the marker 12A from the photographed image of the imaging device 5, and recognizes the position indicated by the fingertip on which the marker 12A is attached as the instruction input position. Compared to the case where the operator's finger itself is detected from the photographed image, there is an advantage that the load of image processing required for detection is light and the detection is successful with high probability. These patterns may be drawn on the entire side surface of the marker 12A, or may be drawn only on the surface facing the marker 12A.

FIG. 2 is a perspective view of the projector 11 according to the present embodiment as viewed from the top surface 2A side.
As described with reference to FIG. 1, the projector 11 projects the image from the surface facing downward toward the front side (side facing the screen SC) obliquely downward. As shown in FIG. 2, the projector 11 is configured by housing each part in an outer casing 2 made of synthetic resin having an overall substantially rectangular parallelepiped shape. Each surface of the exterior casing 2 is a pair of side surfaces facing each other, a top surface portion 2A, a bottom surface portion 2B, side surface portions 2C and 2D constituting left and right side surfaces, and a side surface portion on the side close to the screen SC. The front part 2F and the back part 2E which is a side part far from the screen SC are used. The exterior casing 2 has a configuration in which a cover member 22 is attached to a casing main body 21 that houses the apparatus main body of the projector 11, and a part of the top surface portion 2A, the bottom surface portion 2B, and the side surface portion 2D is a cover member. 22. A slit-shaped opening 21 </ b> C is formed in the side surface 2 </ b> C of the housing body 21. An air filter and an intake fan (not shown) are provided inside the opening 21C, and the intake fan supplies cooling air for cooling the apparatus main body into the exterior housing 2 via the opening 21C and the air filter. Introduce. An exhaust port (not shown) to which an exhaust fan (not shown) for discharging the air introduced from the opening 21C is attached is formed on the side surface portion 2D opposite to the side surface portion 2C.

As shown in FIG. 1, the projector 11 is installed with the top surface portion 2A facing downward. As shown in FIG. 2, an operation panel 23 is disposed at the end of the top surface 2A. In addition, the top surface portion 2A has a V-shaped cross section composed of the first inclined surface 21A and the second inclined surface 21B on the back side with respect to the substantially center position in the front-rear direction (direction from the front surface portion 2F to the back surface portion 2E). A recess is formed.
One first inclined surface 21 </ b> A is a surface facing the screen SC in the installed state of the projector 11, and is formed with a recess 25 </ b> A that is recessed obliquely downward on the back side. A first opening 25B is formed at the bottom of the recess 25A. The first opening 25B is covered with a translucent cover, and a projection optical system 33 for projecting an image is disposed behind the cover. The first inclined surface 21A is formed with a second opening 25 in which the imaging optical system 51 is disposed, and a third opening 25D in which the remote control light receiving unit 4 is disposed, both of which are screens. It faces the SC side. The second opening 25C is covered with a cover that transmits an area including visible light, and the imaging optical system 51 of the imaging device 5 is disposed behind the cover. In addition, the lens which comprises the imaging optical system 51 may serve as the cover of 2nd opening part 25C. The third opening 25D is covered with a cover that transmits infrared light, and at the back of the cover is a light reception unit that receives an infrared signal from a remote controller (not shown) operated by an operator standing in front of the screen SC. A remote control light receiving unit 4 having an element (not shown) is arranged.

The second inclined surface 21B is connected to the first inclined surface 21A and is inclined upward (in a direction away from the bottom surface portion 2B) in FIG. 2 from the back surface portion 2E toward the front surface portion 2F.
The operation panel 23 is provided in the range from the end on the back side to the first inclined surface 21A and on the side opposite to the concave portion 25A side (side surface portion 2D side). The operation panel 23 is provided with a plurality of keys for operating the projector 11.

  Further, as shown in FIG. 1, the projector 11 is installed so that the bottom surface portion 2B faces upward, and the bottom surface portion 2B is provided with a plurality of attachment portions to which metal parts connected to the arm 14 can be attached. ing. The bottom surface portion 2B is provided with a cable connection portion (not shown) to which a communication cable connected to an external device such as the PC 13 or a power cable of the projector 11 is connected. These cables are inserted through the arm 14 and wired to the ceiling or wall surface of the room where the display system 10 is installed.

FIG. 3 is a block diagram illustrating a functional configuration of the projector 11.
The projector 11 includes an I / F (interface) unit 101 connected to an external device such as a PC 13, a video playback device, or a DVD playback device. For example, the I / F unit 101 includes a USB interface, a wired or wireless LAN interface, a VGA terminal to which an analog video signal is input, a DVI (Digital Visual Interface) to which a digital video signal is input, NTSC, PAL, SECAM, and other composites. An S video terminal to which a video signal is input, an RCA terminal to which a composite video signal is input, a D terminal to which a component video signal is input, an HDMI connector conforming to the HDMI (registered trademark) standard, and the like are provided.
The projector 11 is connected to the PC 13 via the LAN interface or USB interface of the I / F unit 101.

The projector 11 is roughly divided into an optical system that forms an optical image and an image processing system that electrically processes a video signal. The optical system includes an illumination optical system 31, a liquid crystal panel 32 that is a light modulation device, and a projection optical system 33. The illumination optical system 31 includes a light source including a xenon lamp, an ultra high pressure mercury lamp, an LED (Light Emitting Diode), and the like. The illumination optical system 31 may include a reflector and an auxiliary reflector that guide light emitted from the light source to the liquid crystal panel 32, and a lens group (not shown), a polarizing plate, A light control element or the like that reduces the amount of light emitted from the light source on the path to the liquid crystal panel 32 may be provided.
The liquid crystal panel 32 receives a signal from an image processing system, which will be described later, and forms an image on the panel surface. The liquid crystal panel 32 includes three liquid crystal panels corresponding to the three primary colors of RGB in order to perform color projection. Therefore, the light from the illumination optical system 31 is separated into three color lights of RGB, and each color light enters each corresponding liquid crystal panel. The color light modulated by passing through each liquid crystal panel is combined by a combining optical system such as a cross dichroic prism and emitted to the projection optical system 33.

  The projection optical system 33 is a zoom lens that enlarges / reduces the projected image and adjusts the focus, a zoom adjustment motor that adjusts the degree of zoom, a focus adjustment motor that adjusts the focus, and directs the projection light toward the screen SC. A concave mirror is provided. The projection optical system 33 zooms and adjusts the incident light modulated by the liquid crystal panel 32 through a lens group, guides the light passing through the lens group toward the screen SC with a concave mirror, and forms an image on the screen SC. . The optical system includes a projection optical system driving unit 121 that drives each motor included in the projection optical system 33 according to the control of the control unit 103, and a light source drive that drives a light source included in the illumination optical system 31 according to the control of the control unit 103. The unit 117 is connected. The specific configuration of the projection optical system 33 is not limited to the above example. For example, the light modulated by the liquid crystal panel 32 is projected onto the screen SC by a lens to form an image by using a configuration that does not use a mirror including a concave mirror. It is also possible.

  On the other hand, the image processing system is configured around a control unit 103 that controls the entire projector 11 in an integrated manner, and includes a storage unit 105 that stores data processed by the control unit 103 and a program executed by the control unit 103, and an operation panel 23. And an input processing unit 123 that detects an operation through the remote control light receiving unit 4, a display control unit 107 that processes an input video input through the I / F unit 101, an image processing unit 113, and a display control unit 107. A light modulator driving unit 119 that performs drawing by driving the liquid crystal panel 32 based on the video signal.

  The control unit 103 controls each unit of the projector 11 by reading and executing the control program stored in the storage unit 105. The control unit 103 detects the content of the operation performed by the operator based on the operation signal input from the input processing unit 123, and in accordance with this operation, the display control unit 107, the projection optical system driving unit 121, and the light source driving The unit 117 is controlled to project an image on the screen SC.

As described above, the outer casing 2 of the projector 11 is provided with the operation panel 23 (FIG. 2) provided with various switches and indicator lamps. The operation panel 23 is connected to the input processing unit 123, and the input processing unit 123 lights or blinks the indicator lamp of the operation panel 23 as appropriate according to the operation state and setting state of the projector 11 according to the control of the control unit 103. . When the switch of the operation panel 23 is operated, an operation signal corresponding to the operated switch is output from the input processing unit 123 to the control unit 103.
Further, the projector 11 receives an infrared signal transmitted in response to a button operation by a remote controller (not shown) used by the operator by the remote controller light receiving unit 4. The remote control light receiving unit 4 decodes the infrared signal received from the remote control, generates an operation signal indicating the operation content of the remote control, and outputs the operation signal to the control unit 103.

  A display control unit 107 is connected to the I / F unit 101. An image processing unit 113 having a frame memory 115 is connected to the display control unit 107. The image processing unit 113 develops the input video input via the I / F unit 101 in the frame memory 115, appropriately executes various conversion processes such as analog / digital conversion, interlace / progressive conversion, resolution conversion, and the like. A video signal in the set format is generated and output to the display control unit 107. The display control unit 107 outputs the video signal processed by the image processing unit 113 to the light modulation device driving unit 119 and displays it on the liquid crystal panel 32. An image acquired by the display control unit 107 via the I / F unit 101 and an image developed and drawn in the frame memory 115 by the display control unit 107 correspond to an original image.

  Further, the projector 11 includes an imaging device 5 that captures the screen SC from the first inclined surface 21A (FIG. 2). The imaging device 5 includes an imaging optical system 51 disposed on the first inclined surface 21A, an imaging unit 52 that includes the imaging device and an output circuit that performs imaging using the imaging optical system 51 and outputs a captured image. With. The imaging device 5 captures the screen SC in accordance with the control of the control unit 103, and particularly performs capturing while the projector 11 projects a projection image on the screen SC, and outputs captured image data to the control unit 103. .

The control unit 103 detects the image of the indicator 12 from the captured image data input from the imaging device 5 and specifies the position indicated by the indicator 12. In this processing, the control unit 103 detects the difference between the projection image data that is developed in the frame memory 115 and then output to the light modulation device driving unit 119, and the captured image data captured by the imaging device 5, and the captured image is detected. The component of the projected image is removed from If the image received and projected from the PC 13 is a moving image, the frame data projected at the time of shooting is acquired from the frame memory 115. In this case, the control unit 103 controls the shooting timing and the switching timing of the frame being projected.
Furthermore, the projector 11 stores the pattern of the marker 12A in advance in the storage unit 105 as the indicator pattern 105A, and the control unit 103 detects a pattern similar or coincident with the indicator pattern 105A from the photographed image. Thereby, the image of the indicator 12 can be detected more quickly.

FIG. 4 is a diagram illustrating an example of a captured image captured by the imaging device 5.
The photographed image 200 includes an image 210 of an operator's hand as an indicator and an image 211 of a hand shadow on the screen SC on the projection image projected by the projector 11 on the screen SC. The control unit 103 compares the data of the photographed image 200 with the data of the projection image output to the light modulation device driving unit 119 after being developed in the frame memory 115, and obtains the difference. A hand shadow image 211 is extracted.

By the way, when the operator is in front of the screen SC, the operator's fingers are reflected in the captured image regardless of the intention of the operator. For this reason, the position of the hand image is not necessarily the position intended by the operator. Also, depending on the relationship between the angle of the hand and the shooting direction, the hand image may have an unpredictable shape. For example, the fingertip cannot be specified and it is difficult to determine which position of the hand image should be detected. There is a case.
Therefore, the projector 11 extracts a hand image and a hand shadow image from the captured image, and obtains a distance between them. For example, if the distance between the hand image and the hand shadow image in the captured image is zero or a value close to zero, the hand is in contact with the screen SC at that position. It can be said that the hand is in contact with the screen SC is the position that the operator intends to instruct. In this way, the control unit 103 extracts the hand image and the hand shadow image from the photographed image, and determines whether or not the screen SC is touched as the instruction state by obtaining the distance between them. it can. Therefore, it is possible to accurately specify the indicated position while accurately reflecting the operator's intention. Also, depending on the shooting direction and the hand angle, the shape of the hand image may be unsharp, but in such a case as well, the position closest to the shadow image in the hand image can be specified as the designated position. . In other words, the pointing position can be specified using the shadow image of the hand.

Further, in the projector 11, the imaging device 5 captures images from substantially the same position as the projection unit 3 projects the projection image toward the screen SC. Therefore, the image of the hand and the shadow of the hand are quickly captured from the captured image. And can be extracted.
Since the imaging device 5 captures the screen SC that is a plane perpendicular to the horizontal plane from above, the screen SC of the captured image 200 is distorted so that the upper side is wide and the lower side is narrow, as shown in FIG. It is reflected.
Similarly, the image projected by the projector 11 on the screen SC is distorted so that the upper side is narrow and the lower side is wide on the screen SC. However, since the projector 11 is assumed to project obliquely with respect to the screen SC, the projection optical system 33 is designed so as not to cause the above-described distortion. For this reason, when a rectangular image is drawn on the liquid crystal panel 32, the image is appropriately deformed through the projection optical system 33, and a rectangular projection image is formed on the screen SC. For this reason, the display control unit 107 does not need image processing for correcting so-called trapezoidal distortion on the screen SC. Note that the function of the display control unit 107 may perform a process of widening the upper part of the original image to be projected and narrowing the lower part so as to correct the trapezoidal distortion of the projected image on the screen SC.

  That is, while the projection image of the projector 11 is rectangular, the projection image that appears in the captured image of the imaging device 5 is deformed due to trapezoidal distortion. The projector 11 deforms the captured image to eliminate the influence of the trapezoidal distortion, makes an image including a rectangular projection image, and compares the image with an image drawn on the liquid crystal panel 32 to obtain a difference. Thereby, the difference between the captured image and the projected image can be detected by a relatively simple process, and the portion of the projected image can be removed from the captured image 200. Thereby, the image of a hand and the image of a hand shadow can be extracted quickly.

When the shooting direction is a direction other than the front of the screen SC (for example, when shooting the screen SC from the lower side of the screen SC or from the left and right sides), the shot image is shot from above. Almost the same distortion occurs. Here, if the projection direction and the shooting direction do not match, after changing the projection image according to the distortion of the shot image, or after changing the shot image according to the trapezoidal distortion correction state of the projection image If the difference is obtained, the hand image and the shadow image of the hand can be quickly extracted by a relatively simple process as in the above case.
Furthermore, by shooting from the side including the lower side and the left and right sides instead of the front side of the screen SC, the indicator is hidden in the shadow of the operator standing in front of the projection surface as in the case of shooting from the front side. And the position of the indicator can be reliably detected.

FIG. 5 shows an operation of drawing a locus of movement according to an operation of moving the indicator 12 as an example of operation when the indication position by the indicator 12 and the indication state, that is, presence or absence of contact with the screen SC are detected. . (A) is a photographed image when the indicator 12 is separated from the screen SC while the indicator 12 is moving, and (B) is a photographed image when the indicator 12 is brought into contact with the screen SC again. (C) is a photographed image when the indicator 12 is moved while being in contact with the screen SC.
The projector 11 shoots the screen SC by the imaging device 5 at a preset cycle under the control of the control unit 103, and indicates the indicator based on the data of the photographic image and the data of the projection image projected at the time of photographing. 12 is detected, and it is determined whether or not the indicator 12 is in contact with the screen SC. Each time the above processing is performed, information indicating the position of the indicator 12 and an indication state indicating whether or not the indicator 12 is in contact with the screen SC is transmitted to the PC 13.

The PC 13 receives information periodically transmitted from the projector 11 and performs operations such as drawing in response to the position input operation of the indicator 12. The photographed image 201 shown in FIG. 5A shows a state where the locus of movement of the point where the hand image 210 and the shadow image 211 of the hand touch is drawn as a curve 251. When the indicator 12 moves away from the screen SC at the position B, the hand image 210 moves away from the shadow image 211 of the hand, so that the projector 11 detects a change in the pointing state. When the PC 13 detects that the indicator 12 has left the screen SC based on the information received from the projector 11, the PC 13 stops drawing the movement locus.
A photographed image 202 shown in FIG. 5B shows a state where the indicator 12 is in contact with the screen SC. The curve 251 is interrupted when the indicator 12 is separated from the screen SC. However, when the indicator 12 comes into contact with the screen SC at a position C away from the curve 251, the indicator 12 moves along a trajectory from which the indicator 12 moves. A curve 252 is drawn.
A photographed image 203 shown in FIG. 5C is a photographed image when the indicator 12 moves in contact with the screen SC from position C to position D in FIG. A curve 252 is drawn along the locus of movement of the indicator 12 during this time. In the present embodiment, the curves 251 and 252 are combined with image information transmitted from the PC 13 to the projector 11, and the projector 11 does not perform the process of drawing the curves 251 and 252.
The function of the display system 10 is that, for example, an operation icon or the like is projected as a projected image in addition to the locus drawing shown in FIGS. 5A and 5B, and the indicator 12 touches the display position of the icon. An example of executing an operation associated with an icon when an operation is performed is given. In this case, the PC 13 performs icon display (projection), determination of the correspondence between the icon display position and the position of the indicator 12, and execution of the operation corresponding to the icon.

FIG. 6 is a flowchart showing the operation of the projector 11 and shows the operation of detecting the position of the indicator 12. When the operation shown in FIG. 6 is executed, the control unit 103 functions as an image acquisition unit, a pointer image detection unit, and an instruction state determination unit, and the projector 11 functions as a position input device.
The control unit 103 of the projector 11 starts a position input mode for detecting position input by the indicator 12 in accordance with an operation on a remote controller (not shown) or the operation panel 23 (step S11). When starting the position input mode, the control unit 103 first executes calibration (step S12). Here, the state of calibration will be described.

FIG. 7 is a diagram illustrating an example of calibration executed by the projector 11, (A) shows the operation of the indicator in the calibration, and (B) shows a pattern projected during the execution of calibration. Note that the arrow in FIG. 7B indicates the movement of the pattern (◯) during calibration, and this arrow is not included in the pattern.
This calibration is an operation for calibrating the position of the indicator 12 detected by the projector 11 to match the indicated position of the actual indicator 12 on the screen SC. During the execution of calibration, the calibration pattern 205 is projected onto the screen SC. The calibration pattern 205 in the example of FIG. 7A is a symbol ◯, and an operator who uses his / her hand as the indicator 12 puts the tip of the finger (indicator 12) on the symbol ◯ on the screen SC. Touch the SC. The projector 11 shoots with the imaging device 5, extracts the image of the indicator 12 and the shadow image of the indicator 12 from the captured image, and detects the position of the symbol ◯ indicated by the indicator 12. Then, calibration is executed by aligning the position of the symbol ◯ detected from the captured image with the position of the symbol ◯ in the projection image. The projector 11 may interrupt the calibration when the image of the indicator 12 and the shadow image of the indicator 12 are separated from each other in the captured image.

  The calibration is preferably performed at many positions included in the operation area of the position input operation. For this reason, as shown in FIG. 7B, the projector 11 projects the calibration patterns 205 (symbols ◯) while sequentially moving them on the screen SC, and the position of each calibration pattern 205 is indicated by the indicator 12. Instruct, detect the indicated position, and calibrate. In the example of FIG. 7B, the calibration pattern 205 is displayed from the upper left corner of the screen SC, but the calibration is completed when the above operation is executed to the lower right corner of the screen SC as a whole. By performing this calibration, the accuracy of position detection based on the captured image can be increased.

After executing the calibration, the control unit 103 controls the imaging device 5 to capture the screen SC and obtain a captured image (step S13). As described above, the captured image acquired by the control unit 103 is an image obtained by capturing the screen SC from above, and the screen SC and the projected image are distorted into a trapezoid. For this reason, the control unit 103 corrects the captured image so that the shape of the screen SC or the projected image shown in the captured image is the same shape (rectangle) as the projection image drawn by the display control unit 107. (Step S14).
Next, the control unit 103 acquires the projection image data projected on the screen SC at the time of shooting from the display control unit 107 and extracts the difference between the acquired projection image data and the corrected shot image data ( Step S15). The control unit 103 extracts a shadow image of the indicator 12 based on the luminance data of each pixel constituting the extracted difference image (step S16). That is, pixels whose luminance data is lower than the threshold value are extracted as pixels constituting the shadow image of the indicator 12. This threshold value may be stored in advance in the storage unit 105, or may be a threshold value calculated from the captured image data acquired in step S13.
Further, the control unit 103 extracts the image of the indicator 12 from the difference image between the captured image and the projected image (step S17). When the marker 12A is attached to the finger as the indicator 12, the control unit 103 reads the indicator pattern 105A stored in advance in the storage unit 105, and makes a difference between patterns that match or are similar to the indicator pattern 105A. Detect with images.

The control unit 103 detects the position (the closest point) at which the shadow image of the indicator 12 extracted in step S16 and the image of the indicator 12 extracted in step S17 are closest (step S18). It is determined whether or not the distance between the image of the indicator 12 and the shadow image of the indicator 12 at the closest point is equal to or less than a preset distance (step S19). This distance may be obtained from the coordinate position of the end point of the indicator 12 image and the end point of the indicator 12 shadow image, or may be obtained from the number of pixels between the end points.
When the distance between the image of the indicator 12 and the shadow image of the indicator 12 is equal to or less than the set value (step S19; Yes), the control unit 103 determines that the indicator 12 is a contact operation in contact with the screen SC ( In step S20), the position of the closest point in the image of the indicator 12 and information indicating the instruction state, that is, the contact operation are transmitted to the PC 13 (step S21), and it is determined whether or not to end the position input mode. (Step S22). When continuing the position input mode, the process returns to step S13.

  On the other hand, when the distance between the image of the indicator 12 and the shadow image of the indicator 12 is larger than the set value (step S19; No), the control unit 103 does not contact the screen SC with the non-contact operation. (Step S23), the position of the closest point in the image of the indicator 12 and the information indicating the indication state, that is, the non-contact operation are transmitted to the PC 13 (Step S24), and the process proceeds to Step S22.

As described above, in the display system 10 according to the embodiment to which the present invention is applied, the control unit 103 of the projector 11 acquires an image (original image) acquired by the display control unit 107 based on the image information input from the I / F unit 101. Image) and a captured image obtained by photographing the image projected on the screen SC by the projector 11, and based on the acquired original image, the image of the indicator 12 indicating the position on the screen SC and the indicator 12 Since the shadow image is detected from the photographed image, the indication position and the indication state indicated by the indicator 12 are specified based on the relative position between the detected image of the indicator 12 and the shadow image of the indicator 12. Along with the position indicated by the body 12, it is possible to specify an instruction state such as whether or not the indicator 12 is in contact with the screen SC. Thereby, regardless of the shape of the indicator 12, the indicated position can be accurately specified. In addition, it is possible to more accurately reflect the intention of the person who operates the indicator 12 and specify the indicated position.
Further, the projector 11 determines whether or not the indicator 12 is in contact with the screen SC based on the distance between the image of the indicator 12 detected from the captured image and the shadow image of the indicator 12. Regardless of the shape of the body 12, the indicated position can be accurately specified.

Further, the projector 11 obtains a photographed image photographed by the imaging device 5 from above the screen SC and discriminates the designated position and the designated state of the indicator 12, so that a photographing device or the like is installed in front of the screen SC. There is no need, and there is an advantage that it is easy to install a projector that projects an image. This effect is not limited to the upper side of the screen SC, but is the same even when shooting from other directions (including, for example, up / down / left / right and diagonal directions) except the front of the screen SC. In other words, the same effect can be obtained from any direction as long as the axis of the lens of the imaging device 5 is not in a direction (front) that is perpendicular to the screen SC. When the screen SC is opaque and the indicator 12 cannot be photographed from the back surface of the screen SC, the photographing direction is defined as a direction excluding the front surface of the screen SC on the front surface side of the screen SC.
Further, the same effect can be obtained when the imaging device 5 is separate from the projector 11 and the captured image of the imaging device 5 is input to the control unit 103 via a wired or wireless communication path. In this case, the imaging device 5 may be arranged so as to capture the screen SC from a direction other than the front.

  In the display system 10, the marker 12 </ b> A with the indicator pattern 105 </ b> A that can be identified by the control unit 103 can be used as the indicator 12. When this marker 12 </ b> A is used, the indicator pattern 105 </ b> A is extracted from the captured image. By detecting, the image of the indicator 12 can be easily detected. Further, since it becomes easy to distinguish the image of the indicator 12 from the shadow of the indicator 12, it is also easy to detect the shadow image of the indicator 12. As a result, it is possible to speed up the process of specifying the designated position.

In addition, it is also possible to perform a position input operation without attaching the marker 12A as the indicator 12.
In this case, the control unit 103 extracts a finger image as the indicator 12 and a shadow image of the finger from the difference image between the captured image and the projection image. The shadow image of the finger can be detected using the threshold value of luminance as in the above embodiment. On the other hand, the image of a finger stores, for example, a plurality of shape data that can be considered as the shape of a finger in advance, and a method of detecting an image close to the stored shape from a captured image, or finger color reference data Is stored, and a color similar to this color is detected from the photographed image. The color of the finger in this case is not limited to the color stored in advance, and the reference data of the color of the finger may be determined by statistically processing the color of each pixel constituting the captured image.

  In the above-described embodiment, the projector 11 can project from above the screen SC, and a configuration having a function of projecting an image on the screen SC and a function of photographing the screen SC will be described as an example. However, the present invention is not limited to this, and a configuration using an imaging device separate from the projector may be employed.

[Second Embodiment]
FIG. 8 is a diagram showing a configuration of a display system 10A according to the second embodiment to which the present invention is applied.
The display system 10A is a system that includes a projector 11A that projects an image from below on a screen SC provided on a wall surface, and projects a still image or a moving image that the PC 13 transmits to the projector 11A. The projector 11A is placed on the floor in front of the screen SC via a pedestal, and projects projection light upward at a predetermined tilt angle in accordance with the position of the screen SC.
The PC 13 is configured in the same manner as in the first embodiment, and the projector 11A is configured to include each part of the projector 11 (FIG. 3) except for the imaging device 5. The illustration and description of the components configured in the same manner as in the first embodiment are omitted.

The display system 10A also includes an imaging device 5A attached via an arm 14 above the screen SC. The imaging device 5A includes an imaging optical system 51 and an imaging unit 52, like the imaging device 5 included in the projector 11 of the first embodiment, and the imaging unit 52 controls the projector 11A through a wired or wireless communication path. Connected to the unit 103 or the PC 13.
The imaging device 5A captures the screen SC from above according to control information transmitted from the control unit 103 or the PC 13 of the projector 11A. Then, the imaging device 5A transmits captured image data to the projector 11A or to the projector 11A via the PC 13.

The display system 10A can execute the position input mode, and in this position input mode, the position designated on the screen SC is specified. That is, when an operator standing in front of the screen SC moves his / her finger as the indicator 12 on the screen SC and points to a specific position, the coordinates of the position pointed to by the finger are detected. It is determined whether or not the finger is in contact with the screen SC at the coordinates.
In the position input mode, the projector 11A causes the imaging device 5A to periodically perform shooting to acquire shot image data, and the projection image data projected on the screen SC at the time of shooting from the built-in frame memory 115. get. Then, based on the photographed image and the projected image, the same processing as in the first embodiment (FIG. 6) is performed to determine the indication position and the indication state by the indicator 12.

In the second embodiment, the projector 11 </ b> A and the imaging device 5 </ b> A are configured as separate bodies, but there are the same effects as in the first embodiment. Further, an image is projected from the lower side of the screen SC by the projector 11A, and the indicator 12 and the shadow of the indicator 12 by the projected light are photographed from above by the imaging device 5A. In other words, if the shooting direction and the projection direction are different, the shooting direction can be selected regardless of the projection direction. Therefore, an environment having an optimal condition for detecting whether or not the indicator 12 is in contact with the screen SC can be selected. realizable. For this reason, the indication state by the indicator 12 can be determined accurately and promptly. In addition, the display system 10A fixes the relatively light-weight imaging device 5A to the wall surface, and is installed on the front surface of the screen SC and can be diverted from a type of projector that projects from the front surface. Therefore, the display system 10A can be easily realized at low cost. There is.
In the configuration of the second embodiment, the position of the imaging device 5A is not limited to the upper side of the screen SC, and may be taken from the left and right sides of the screen SC.

Each of the above-described embodiments is merely an example of a specific mode to which the present invention is applied, and the present invention is not limited thereto. The present invention can be applied as a mode different from the above-described embodiment. For example, in the above embodiment, the projector 11, 11 </ b> A has been described by taking as an example a configuration in which the position indicated by the indicator 12 is detected, the indication state is determined, and information indicating the result is transmitted to the PC 13. The present invention is not limited to this, and the PC 13 acquires and analyzes the projection image projected by the projector 11 and the captured image data captured by the imaging devices 5 and 5A. It is good also as a structure which specifies.
In the above-described embodiment, the control unit 103 compares the captured image of the imaging device 5 with the projected image and extracts the difference, and corrects the captured image so that the shape of the captured image matches the shape of the projected image. However, the present invention is not limited to this, and the image acquired from the display control unit 107 may be corrected according to the distortion state of the captured image without correcting the captured image.

  In the above embodiment, the light modulation device that modulates the light emitted from the light source has been described by taking as an example a configuration using three transmissive or reflective liquid crystal panels 32 corresponding to each color of RGB. However, the present invention is not limited to this, for example, a system using one liquid crystal panel and a color wheel, a system using three digital mirror devices (DMD), a single digital mirror device and a color. You may comprise by the DMD system etc. which combined the wheel. Here, when only one liquid crystal panel or DMD is used as the light modulation device, a member corresponding to a combining optical system such as a cross dichroic prism is unnecessary. In addition to the liquid crystal panel and DMD, any light modulation device capable of modulating light emitted from the light source can be employed without any problem.

  In the above embodiment, the configuration in which the indicator 12 is the operator's finger has been described as an example. However, the present invention is not limited to this, and the position can be indicated on the screen SC. Any member can be used as the indicator 12. For example, an indicator bar or the like that can indicate an arbitrary position on the screen SC may be used as the indicator 12.

  Further, in the above embodiment, the description has been given of the configuration in which the input control image is selected by the display control unit 107 and processed by the image processing unit 113 from a plurality of input sources. The configuration is not limited, and the series of processes may be executed by an image supply apparatus such as the PC 13 externally connected to the projector 11. In this case, the image supply apparatus can include the functions of the control unit 103, the storage unit 105, the display control unit 107, and the image processing unit 113 included in the projector 11. The present invention can also be realized as a program executed by such an apparatus. In these configurations, the projector 11 only needs to project an image input from the image supply device onto the screen SC. In the above-described embodiment, the PC 13 is configured to output image information to the projector 11, but an image may be projected using image data built in the projector 11. Further, in the above-described embodiment, the projector 11 transmits position information related to the indicated position of the indicator 12 to the PC 13, and the PC 13 performs drawing on the projected image so as to superimpose a new original image for projection. Although the processing such as creation is performed to reflect the contents of the position input operation on the projection image, this series of processing may be performed only by the projector 11. That is, the projector 11 may generate and project a new image based on the indication position of the indicator 12.

  Further, each functional unit of the projector 11 illustrated in FIG. 3 indicates a functional configuration of the projector 11, and a specific mounting form is not particularly limited. That is, it is not always necessary to mount hardware corresponding to each function unit individually, and it is of course possible to adopt a configuration in which the functions of a plurality of function units are realized by one processor executing a program. In addition, in the above embodiment, a part of the function realized by software may be realized by hardware, or a part of the function realized by hardware may be realized by software. In addition, specific detailed configurations of other parts of the projector 11 and the display system 10 can be arbitrarily changed without departing from the gist of the present invention.

  DESCRIPTION OF SYMBOLS 3 ... Projection part, 5 ... Imaging device (imaging means), 5A ... Imaging device, 10, 10A ... Display system, 11, 11A ... Projector (position input device), 12 ... Indicator, 12A ... Marker, 13 ... PC, 32 ... Liquid crystal panel, 33 ... Projection optical system, 51 ... Imaging optical system, 52 ... Imaging unit, 103 ... Control unit (image acquisition means, indicator image detection means, and instruction state determination means), 105 ... Storage unit, 105A ... indicator pattern, 115 ... frame memory, 200, 201, 202 ... photographed image, 205 ... calibration pattern, SC ... screen (projection surface).

Claims (7)

Image acquisition means for acquiring an original image and a captured image acquired by capturing an image projected on a projection surface by a projector based on the original image;
Based on the original image acquired by the image acquisition means, an indicator image detection means for detecting an image of a pointer indicating the position on the projection surface and a shadow image of the pointer from the captured image; ,
Based on the relative position between the image of the indicator detected by the indicator image detection means and the shadow image of the indicator, an instruction state determination means for specifying an instruction position and an instruction state indicated by the indicator;
A position input device comprising:   The indication state determination unit determines whether the indicator is in contact with the projection surface based on a distance between the indicator image detected by the indicator image detection unit and a shadow image of the indicator. The position input device according to claim 1, wherein the position input device is discriminated.   The position input device according to claim 1, wherein the image acquisition unit acquires a captured image captured from a direction other than the front surface of the projection surface.   A predetermined pattern that can be identified by the pointer image detection unit is attached to the pointer, and the pointer image detection unit detects the predetermined pattern from the captured image acquired by the image acquisition unit. The position input device according to claim 1, wherein the position input device is characterized by the following. A projector,
Projection means for projecting an image on the projection surface based on the original image;
A photographing means for photographing an image projected by the projection means on the projection surface;
Image acquisition means for acquiring the original image, and a captured image obtained by capturing the image projected by the projector based on the original image;
Based on the original image acquired by the image acquisition means, an indicator image detection means for detecting an image of a pointer indicating the position on the projection surface and a shadow image of the pointer from the captured image; ,
Based on the relative position between the image of the indicator detected by the indicator image detection means and the shadow image of the indicator, an instruction state determination means for specifying an instruction position and an instruction state indicated by the indicator;
A projector comprising: Project an image on the projection surface based on the original image,
Taking an image projected on the projection surface based on the original image to obtain a captured image,
Based on the original image, an image of a pointer that indicates a position on the projection surface and a shadow image of the pointer are detected from the captured image;
Specifying an indication position and an indication state indicated by the indicator based on a relative position between the detected indicator image and a shadow image of the indicator;
A projector control method characterized by the above. A display system configured to include a projector that projects an image on a projection surface and a photographing device that photographs the projection surface,
Image acquisition means for acquiring an original image and a captured image acquired by the imaging device capturing an image projected on the projection surface by the projector based on the original image;
Based on the original image acquired by the image acquisition means, an indicator image detection means for detecting an image of a pointer indicating the position on the projection surface and a shadow image of the pointer from the captured image; ,
Based on the relative position between the image of the indicator detected by the indicator image detection means and the shadow image of the indicator, an instruction state determination means for specifying an instruction position and an instruction state indicated by the indicator;
A display system comprising:

Images