JP6328304B1 - I / O system - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an input / output system capable of effectively detecting an operation on an operated device. An input / output system includes: an input / output processing device that detects a user operation based on a photographed image captured by a photographing device; and a user that has passed through one side of the sheet and the sheet. A camera 20 that captures an image including a shadow, a translucent sheet 30, and a connection that connects the sheet 30 and the user's limbs at an interval at which a shadow of a part of the user's limbs is projected onto the sheet 30. It has the member 70 and the to-be-operated apparatus 90 which performs the operation from the user output from the input-output processing apparatus 10. FIG. [Selection] Figure 1

Description

  The present invention relates to an input / output system.

  For example, Patent Literature 1 includes a photographing device and a computer device, and is a pointing system for an operator to perform a pointing operation on a remote display. An imaging unit that images the operator, an imaging data processing unit that acquires skeleton information including a plurality of parts on the body of the operator and their positions based on the captured imaging data of the operator; A skeleton information output unit that outputs the skeleton information of the operator, and the computer device includes a skeleton information acquisition unit that acquires the skeleton information of the operator from the imaging device, and the skeleton information includes An operation point that is a part of the operator's body and is used when the operator performs the pointing operation is determined. A point determination unit, a base point determination unit for determining a base point that is a fixed point when the operator performs the pointing operation, and the operator at least about the first position and the second position on the display Based on the skeleton information of the operator when performing a calibration operation that is a pointing operation, the position when the second position is indicated from the position of the operation point when the first position is indicated A virtual touch panel calculator that calculates a rectangular virtual touch panel having a line connecting the positions of the operation points as a diagonal line and a part of a straight line extending in a vertical direction or a horizontal direction from the first position as one side; and the calibration When the operator moves the operation point after the operation, the operation point is displayed on the virtual touch panel based on the skeleton information of the operator. An instruction position specifying unit that determines a position on the display corresponding to position information on the virtual touch panel based on a positional relationship between the virtual touch panel and the base point; and the instruction position specifying unit And a pointing position display information output unit that outputs pointing position display information for indicating on the display the position on the display determined in (1).

Japanese Patent Laid-Open No. 2006-134002

  An object of this invention is to provide the input / output system which can detect the operation with respect to a to-be-operated apparatus effectively.

  An input / output system according to the present invention includes a translucent screen, a camera that captures an image including the screen at a plurality of timings, and the screen at a plurality of timings based on captured images captured by the camera. Operation detecting means for detecting a user operation based on the position of the user.

  Preferably, the operation detection unit detects a user operation based on a position of a user's shadow reflected on the screen at a plurality of timings.

  Preferably, the screen is plate-like, and a shadow generated by the user blocking the light irradiated by the light source is projected on the other surface of the screen, and the camera is projected on the other surface of the screen. The shadow projected on the screen is photographed at a plurality of times from one surface of the screen, and the operation detection unit is a shadow reflected by light transmitted to the one surface of the screen in the photographed image photographed by the camera. The user's operation is detected based on the position.

  Preferably, a shadow specifying means for specifying a shadow position, a shadow moving amount, a shadow moving direction, a shadow moving speed, a shadow size, or a shadow shade in the screen, and the camera is photographed. Screen specifying means for specifying the position of the screen in the captured image, the size of the screen, the amount of movement of the screen, the tilt of the screen, or the shape of the screen, and the operation detecting means is the shadow specifying means. Based on the combination of the position, movement amount, movement direction, movement speed, size or shade specified by the above, and the position, movement amount, inclination, or shape specified by the screen specifying means. Detect operation.

  Preferably, on the basis of a photographed image photographed by the camera, a pixel identifying unit that identifies a shadow pixel in the screen, and a shading that identifies a change in shading in the shadow pixel identified by the pixel identifying unit. A change specifying unit, and the operation detecting unit detects a user operation based on a change in shade of the shadow specified by the shade change specifying unit.

  Preferably, on the basis of a photographed image photographed by the camera, a pixel identifying unit that identifies a shadow pixel in the screen and a change in the size of the shadow pixel identified by the pixel identifying unit are identified. A size change specifying unit is further included, and the operation detecting unit detects a user operation based on a change in the size of the shadow specified by the size change specifying unit.

  Preferably, the apparatus further includes a moving distance specifying means for specifying a moving distance of the screen based on a plurality of captured images taken by the camera, and the operation detecting means is specified by the moving distance specifying means. The user's operation is detected based on the distance.

  Preferably, the apparatus further includes a connection member that connects the screen and the user's limb at an interval at which a shadow of a part of the user's limb is projected onto the screen, and the connection member is The screen and the user's limbs are connected at a position where the other surface of the screen and a part of the user's limbs face each other.

  Preferably, the image processing apparatus further includes a shape correction unit that corrects a shape of the screen in a captured image captured by the camera, and the pixel specifying unit is configured to detect from within the screen in the captured image corrected by the shape correction unit. Identify shadow pixels.

  Preferably, the image processing apparatus further includes an inclination specifying unit that specifies an inclination of the screen with respect to the captured image based on a captured image captured by the camera, and the operation detection unit is configured to specify the screen specified by the inclination specifying unit. The user operation is detected based on the inclination of.

  Preferably, the screen is formed in a polygonal or circular shape, and on one surface, all the areas are colored in a single color, and based on the color developed by the screen in the photographed image, The image processing apparatus further includes area specifying means for specifying the area of the screen, and the shade change specifying means specifies a change in shade of shadow in the screen area specified by the area specifying means.

  Preferably, the screen is formed so that a color to be developed in the central region and a color to be developed in the peripheral region are different on one surface, and the color of the central region of the screen in the photographed image and the peripheral region And an area specifying means for specifying the area of the screen based on at least one of the colors of the colors, wherein the shade change specifying means is in the screen within the area specified by the area specifying means. Identify changes in shadow shading.

  Preferably, a light source that emits light to the other surface of the screen and a connecting member that variably connects a separation distance between the screen and the light source.

  Preferably, a second screen that develops a color different from that of the screen, and an identification unit that identifies, based on the captured image, the color that the screen develops and the color that the second screen develops The operation detection unit detects a user operation for each screen based on the shadow of the user in each screen identified by the identification unit.

  Preferably, the display device further includes a buffer member that is connected to the other surface of the screen and has translucency and elasticity, and the pixel specifying unit is a shadow that passes through the screen and the buffer member. Identify the pixels.

  Preferably, the image processing apparatus further includes an area dividing unit that divides the screen area based on the captured image and distributes the operation contents to the divided areas. In the divided area, a change in shadow shading is specified, and the operation detection means detects the operation content distributed to the area in which the change in shadow density is specified as a user operation.

  Preferably, the synchronization means for matching the coordinates of the predetermined position of the screen in the photographed image with the coordinates of the pointer in the display range of the operated device, and the operation content detected by the operation detecting hand is output to the operated device. The operation detecting hand further includes a screen position and a movement amount specified by the screen specifying means, and a shadow of a fingertip of the user's hand specified by the shadow specifying means. The movement of the pointer is detected based on the position, the amount of movement of the shadow, the direction of movement of the shadow, and the speed of movement of the shadow, or the size of the shadow of the user's fingertip specified by the shadow specifying means Detecting a click operation at the coordinates of the pointer based on a change or a change in shading of the shadow, and the operation content output means is a movement operation of the pointer detected by the operation detection hand, Based on the click operation, and outputs to the pointer in the display range of synchronized the operated device.

  Preferably, the apparatus further includes a connection member that connects the screen and the user's limb at an interval at which a shadow of a part of the user's limb is projected onto the screen, and the connection member is The screen and the user's limbs are connected at a position where the other surface of the screen and a part of the user's limbs face each other.

  Further, the input / output processing device according to the present invention is based on the input image data at a plurality of timings, the screen position of the screen at a plurality of timings, the size of the screen, the amount of screen movement, the inclination of the screen, or Based on the screen specifying means for specifying the shape of the screen and the input image data at a plurality of timings, the position of the shadow reflected on the screen, the amount of movement of the shadow, the moving direction of the shadow, the moving speed of the shadow, The shadow specifying means for specifying the size or shade of the shadow, and the position, movement amount, moving direction, moving speed, size, or shade of the shadow specified by the screen specifying means, and the shadow specifying means And an operation detection unit that detects a user operation based on the identified position, size, movement amount, inclination, or shape of the screen.

  The input / output method according to the present invention includes a step in which a camera captures a screen at a plurality of timings, a step of detecting a user operation based on the position of the screen at a plurality of timings, and a detected user operation. Output to the operated device.

  In addition, the program according to the present invention determines the screen position, the clean size, the screen movement amount, the screen tilt, or the screen shape at a plurality of timings based on the input image data at a plurality of timings. Based on the screen identification function to be identified and the input image data at a plurality of timings, the position of the shadow on the screen, the amount of movement of the shadow, the direction of movement of the shadow, the speed of movement of the shadow, the size of the shadow, or The shadow specifying function for specifying the shade of the shadow and the position, moving amount, moving direction, moving speed, size, or shade of the shadow specified by the screen specifying function and the screen specified by the shadow specifying function. Allows a computer to realize an operation detection function that detects a user operation based on the position, size, movement amount, inclination, or shape.

  According to the present invention, an operation on the operated device can be detected effectively.

It is a figure explaining the composition of input / output system 1 in an embodiment. It is a figure explaining the input / output system 1 of FIG. 1 in detail. It is a figure explaining the outline | summary of the analysis method of the shadow which permeate | transmitted the back surface of the sheet | seat 30 in FIG. 2 is a diagram illustrating a hardware configuration of an input / output processing device 10. FIG. 2 is a diagram illustrating a functional configuration of an input / output processing device 10. FIG. It is a figure explaining the sheet | seat specific | specification part 500 of FIG. 4 in detail. It is a figure explaining the shadow specific part 510 of FIG. 4 in detail. 5 is a diagram for explaining an image analysis image of a user's shadow transmitted through a sheet 30. FIG. 3 is a diagram illustrating an image analysis image that identifies the shape of a sheet 30. FIG. It is a flowchart explaining the input / output process (S10) in the input / output system 1. 6 is a flowchart for explaining sheet input processing (S20) in the input / output system 1. It is a flowchart explaining the shadow input process (S30) in the input / output system 1. It is a figure which illustrates the modification 1 of the input-output system 1 in embodiment. It is a figure which illustrates the modification 2 of the input-output system 1 in embodiment. It is a figure which illustrates the modification 3 of the input-output system 1 in embodiment. It is a figure which illustrates the function structure of the input / output processing apparatus 10 in the modification 3. It is a figure which illustrates the modification 4 of the input-output system 1 in embodiment. It is a figure which illustrates the modification 5 of the input-output system 1 in embodiment. It is a figure which illustrates the modification 6 of the input-output system 1 in embodiment. It is a figure explaining the sheet | seat specific | specification part 500 in the modification 6 in detail. It is a figure which illustrates the modification 7 of the input-output system 1 in embodiment.

Embodiments in the present invention will be described with reference to the drawings.
FIG. 1 is a diagram illustrating a configuration of an input / output system 1 in the embodiment.
FIG. 2 is a diagram for explaining the input / output system 1 of FIG. 1 in more detail.
As illustrated in FIGS. 1 and 2, the input / output system 1 according to the present embodiment includes an input / output processing device 10, a camera 20, a sheet 30, a connecting member 70, and an operated device 90.
In the input / output system 1, the input / output processing device 10, the camera 20, and the operated device 90 are disposed indoors, and the seat 30 and the connecting member 70 are disposed outdoors (in the hallway or outdoors). It is assumed that there is an obstacle between the room and the outdoors.
The input / output processing device 10, the camera 20, and the operated device 90 disposed in the room are connected to each other by wire or wirelessly, and the input / output processing device 10 and the camera 20 of this embodiment are connected via a cable 40A. To each other, and information can be sent from the camera 20 to the input / output processing device 10. Further, the input / output processing device 10 and the operated device 90 can be connected to each other via the cable 40B, and information can be sent from the input / output processing device 10 to the operated device 90.
In addition, the camera 20 is arranged with the shooting direction facing the obstacle. Here, the obstacle is, for example, a transparent obstacle to the extent that a shadow transmitted through the back surface of the sheet 30 can be photographed, and the obstacle in the present embodiment is a window glass. That is, the camera 20 is arranged with the shooting direction facing the outdoor.
On the other hand, a user outside the room wears the seat 30 using the connecting member 70, and the position where the seat 30 and the shadow of the user's hand reflected in the sheet 30 are included within the shootable range of the camera 20 in the room. Perform input operation with. Note that the shadow of the user's hand reflected on the sheet 30 is a shadow generated on the surface of the sheet 30 when the user's hand blocks a part of light incident on the surface of the sheet 30 from the light source L. It is a concept that includes a shadow generated on the front surface of the sheet 30 and a shadow transmitted through the back surface of the sheet 30 (a shadow caused by transmitted light).

  The input / output processing device 10 is a computer terminal, controls the camera 20 to cause the camera 20 to capture an image including the sheet 30, accepts a user input operation from the captured image, and accepts the received input operation to the operated device 90. Output to. The input / output processing device 10 according to the present embodiment accepts a user input operation based on the relative movement distance of the sheet 30 with respect to the captured image and the shadow of the user's hand that has passed through the back surface of the sheet 30 in the captured image. The received operation content is output to the operated device 90. Here, the relative movement distance of the sheet 30 with respect to the photographed image is compared and specified based on the photographed image 22 before the movement of the sheet 30 and the photographed image 22 after the movement, as illustrated in FIG. Specifically, the movement distance moved from the reference point (SX, SY) of the sheet 30 with respect to the photographed image 22 before movement to the reference point (EX, EY) of the sheet 30 with respect to the photographed image 23 (not shown) after movement. Based on the above, the moving distance of the sheet 30 is specified. In addition, the user input operation based on the shadow includes an operation based on the shadow position (for example, a pointer operation), an operation based on the movement amount, movement direction, and movement speed of the shadow (for example, a drag operation), a shadow operation, This is a concept including an operation (for example, a click operation) based on the density and size of the image.

  The camera 20 is a photographing device that captures still images or moving images, and captures images at a plurality of timings. The camera 20 of this example is a digital camera that captures an image including one surface (back surface) of the sheet 30 through a glass plate of a window. The camera 20 shoots so that the one surface (back surface) of the sheet 30 is included in the shootable range in a state where the light source L and a part of the user's limbs are arranged on the other surface (front surface) of the sheet 30.

The sheet 30 is a plate-like screen having translucency. The sheet 30 may be a colored plate shape, a translucent plate shape, or a colored translucent plate shape as long as it has translucency. The sheet 30 can be colored as long as it has a color that is not substantially the same color as the transmitted shadow (for example, preferably white). On the other hand, the sheet 30 is not suitable for black or gray, which is a color in which the transmitted shadow cannot be recognized. The sheet 30 of this example is a plate-like screen colored with a specific color C other than black or gray. The colored sheet 30 can be colored in a single color or a plurality of colors.
Further, the sheet 30 has a rigidity that does not substantially deform due to the weight of the sheet 30 itself, and preferably has a rigidity that can maintain the shape even when an external force is applied from the outside.
The sheet 30 is formed in a size that allows projection of a shadow of a part of the user's limb. Here, the part of the user's limb is at least a part of the user's arm or leg, and the part of the user's limb is a concept indicating a hand or a foot. In addition, the size capable of projecting a shadow of a part of the user's limb is a size capable of projecting the shadow of the user's hand or foot. The size of the sheet 30 in this example is a size that can project a shadow of a hand (a range from the wrist to the fingertip) that is a part of the user's arm.
Further, the sheet 30 is formed in a polygonal shape or a circular shape (including an ellipse). In addition, the shape of the sheet | seat 30 of this example is a rectangle (rectangle).

  The connecting member 70 is a connecting fixture that connects the sheet 30 and the user's arm at an interval at which the shadow of the user's hand is projected onto the sheet 30. The connecting member 70 arranges the sheet 30 at a position facing the user's arm when the sheet 30 is viewed. The connecting member 70 of this embodiment is a multi-axis arm, and the angle and position of the seat 30 with respect to the user's hand can be appropriately adjusted in a state where the seat 30 and the user's arm are connected.

  The operated device 90 is an output device that executes a user operation output from the input / output processing device 10. The operated device 90 is, for example, a personal computer 90a, an industrial robot 90b, an image display device 90c, or the like.

  The light source L is a lighting fixture such as a light bulb, a fluorescent lamp, or an LED lamp. The light source L is disposed on the back surface of the surface of the sheet 30 taken by the camera 20 and is disposed at a position where the surface of the sheet 30 can be irradiated with light. The light source L includes the sun and the moon when the user is outdoors.

  As described above, in the input / output system 1 according to the present embodiment, with the above-described configuration, the camera 20 captures a state in which the user moves the sheet 30 connected to the hand a plurality of times, and the sheet corresponding to the captured image is captured from the captured image. Based on the relative movement distance of 30 and the shadow of the hand transmitted to the back surface of the sheet 30, it is detected as a user input operation, and the detected operation content is output to the operated device 90.

Next, an outline of a method for analyzing a shadow transmitted through the back surface of the sheet 30 will be described.
FIG. 3 is a diagram for explaining the outline of the analysis method of the shadow transmitted through the back surface of the sheet 30 in FIG.
As illustrated in FIG. 3, the light source L, which is indoor lighting such as a fluorescent lamp, emits diffusive light in order to uniformly illuminate every corner of the room, and the light emitting portion is manufactured with a surface instead of a point. Yes. The shadow that can be obtained by shielding the light emitted from the light source L of the room illumination is such that the closer the distance between the light shielding hand and the light source L is, the larger the projected shadow area is, and The thickness of the film becomes lighter (FIG. 3A). On the other hand, the shorter the distance between the object that shields light and the sheet that projects the shadow, the larger the projected shadow area is and the darkness of the shadow. Becomes darker (FIG. 3B). In other words, with the plain thin A4 paper plane facing the ceiling lighting, hold your hand between the light and the A4 paper, and put your hand on the A4 paper or release your hand from the A4 paper. The size of the shadow area transmitted through the A4 sheet or the change in the shade of the shadow occurs. This is dark because the influence of blurring around the shadow due to the penumbra by the surface light source, scattered light from other objects in the room, reflected light, and the like is reduced.

  Utilizing the above principle, the input / output system of the present embodiment makes the distance between the light source L and the sheet 30 constant. In the input / output system, as illustrated in FIG. 3A, when the separation distance between the sheet 30 and the user's hand is further increased from a certain distance, the shadow region becomes large and the shadow color becomes light. On the other hand, as illustrated in FIG. 3B, when the separation distance between the sheet 30 and the user's hand is shortened from a certain distance, the shadow area becomes smaller and the shadow color becomes darker.

  The input / output system captures the above two states with the camera 20 and compares each captured image with a reference value, so that when the separation distance between the sheet 30 and the user's hand is equal to or greater than the reference value, the input / output system Detects user operations based on. Specifically, the input / output system determines in advance a reference separation distance between the sheet 30 and the user's hand, and calculates the reference value by calculating the number of shadow pixels or the pixel density at that distance. As a memory record. Next, the number of shadow pixels or pixel density in the captured image when the distance between the sheet 30 and the user's hand is increased or decreased is compared with the number of shadow pixels or pixel density as a reference value. To do. As a result of comparison, when the reference value is equal to or greater than the reference value, a user operation based on the shadow is detected.

The input / output system detects a user operation based on at least one of the number of shadow pixels and the density or size of the pixels.
When the input / output system detects the user's operation based on the number of pixels of the shadow, it captures a subtle shade of the shadow that has passed through the sheet 30 depending on the type and performance of the camera element or camera computer driver. Since it may not be captured as an image, it is preferable to detect the user's operation by determining the separation distance using the number of pixels of the transmitted shadow. When the light source L is not a surface light source such as a fluorescent lamp but a point light source such as sunlight or a filament light bulb, it is more accurate to count the number of shadow pixels and detect the user's operation.

  In addition, when the input / output system detects a user's operation based on the shaded color of the transmitted shadow, it is not necessary to always count all the pixels of the transmitted shadow, so the processing speed is high with a light processing load. User operations can be detected (because the distance can be determined by extracting one or a few transparent shadow pixels and analyzing the shadow's color intensity).

Further, the input / output system can improve the accuracy when detecting the user's operation based on the number of shaded shadow pixels and the shaded color as compared to detecting the user operation only with the number of pixels. This is because when the shape of the user's hand in the vicinity of the sheet 30 changes (for example, transition from a state where only the index finger is raised to a state where all fingers are raised), the shape of the shadow also changes accordingly. Along with that, the number of shadow pixels changes and it is perceived as a change in separation distance rather than an input operation, and there is a risk that accuracy may be reduced only by detecting the number of pixels, This is because it is possible to prevent a decrease in accuracy by using the combination.
Note that the input / output system of the present embodiment detects a user's operation based on both the shade of shade (the shade change specifying unit 520) and the size of the shadow region (size change specifying unit 521). However, the present invention is not limited to this, and the user's operation may be detected using only one of the size of the shadow area or the shade of the shadow.

FIG. 4 is a diagram illustrating a hardware configuration of the input / output processing device 10.
As illustrated in FIG. 4, the input / output processing device 10 includes a CPU 100, a memory 102, an HDD 104, a network interface 106 (network IF 106), a display device 108, and an input device 110. Are connected to each other.
The CPU 100 is, for example, a central processing unit.
The memory 102 is, for example, a volatile memory and functions as a main storage device.
The HDD 104 is, for example, a hard disk drive device, and stores a computer program (for example, the input / output program 5 in FIG. 5) and other data files (for example, the setting information database 590 in FIG. 5) as a nonvolatile recording device.
The network IF 106 is an interface for wired or wireless communication, and realizes communication between the input / output processing device 10 and the camera 20 or the operated device 90, for example.
The display device 108 is, for example, a liquid crystal display, and the input device 110 is, for example, a keyboard and a mouse. The display device 108 and the input device 110 are exclusively used for the parameter setting values of the input / output system preliminarily stored in the setting information database 590 of FIG. Etc. are used when the user himself / herself manually adjusts, etc., but is not always necessary when this adjustment operation is unnecessary or when automatic adjustment is performed from a pre-captured image of the camera 20.

FIG. 5 is a diagram illustrating a functional configuration of the input / output processing device 10.
As illustrated in FIG. 5, the input / output processing device 10 of this example is installed with an input / output program 5 and a setting information database 590 (setting information DB 590).
The input / output program 5 includes a sheet specifying unit 500, a shadow specifying unit 510, an operation detection unit 530, and an operation signal output unit 540. Part or all of the input / output program 5 may be realized by hardware such as an ASIC, or may be realized by borrowing a part of an OS (Operating System) function.

(Sheet identification part 500)
FIG. 6 is a diagram illustrating the sheet specifying unit 500 in FIG. 5 in more detail.
In the input / output program 5, the sheet specifying unit 500 includes an area specifying unit 502, a shape specifying unit 504, a shape correcting unit 507, and a moving distance specifying unit 508, as illustrated in FIG.

  The area specifying unit 502 specifies the area of the sheet 30 based on the color of the sheet 30 in the captured image captured by the camera 20. The area specifying unit 502 in this example sequentially scans and analyzes the pixels of the captured image and specifies an area in which the color values (RGB values) of the specific color C are continuous, whereby the sheet 30 colored with the specific color C is displayed. Identify the area.

  The shape identifying unit 504 identifies the shape of the sheet 30 in the captured image captured by the camera 20. The shape specifying unit 504 in this example extracts the pixel portion of the specific color C of the sheet 30 specified by the region specifying unit 502, and specifies the overall shape of the sheet 30 including the shadow portion from the extracted pixel portion.

  The shape correcting unit 507 corrects the deformation (for example, inclination or deflection) of the sheet 30 to the shape before deformation from the overall shape of the sheet 30 specified by the shape specifying unit 504. The shape correcting unit 507 of the present embodiment uses the reference value and the shape specifying unit 504 specified by using the vertical / horizontal ratio of the rectangular sheet 30 preliminarily input to the setting information database 590 as a reference value. Compared with the shape of the sheet 30, the difference caused by the perspective and the deflection due to the original aspect ratio and inclination of the sheet 30 is calculated, and the shape is corrected so as to become the original rectangular shape based on the calculation result.

  The movement distance identification unit 508 identifies the movement amount of the sheet 30 based on a plurality of captured images captured at a plurality of timings by the camera 20. The movement distance specifying unit 508 compares the captured image before the movement of the sheet 30 with the captured image after the movement of the sheet 30 from the plurality of captured images captured by the camera 20, and the amount of movement of the sheet 30 in the captured image. Is identified. The moving distance specifying unit 508 of this example uses the start position coordinates of the left corner of the sheet 30 in the captured image as a reference point, and specifies the amount of movement of the reference point from the captured images before and after the movement of the sheet 30.

(Shadow identification unit 510)
FIG. 7 is a diagram for explaining the shadow specifying unit 510 of FIG. 5 in more detail.
As illustrated in FIG. 7, the shadow specifying unit 510 includes a reference value setting unit 511, a pixel specifying unit 512, a position specifying unit 514, a moving distance specifying unit 516, a moving direction specifying unit 517, a moving speed specifying unit 518, and a light / dark change. A specifying unit 520, a size change specifying unit 521, and a separation distance specifying unit 522 are included.

The reference value setting unit 511 uses the pixel value of the shadow region (more specifically, the RGB value indicated by the color value of the pixel) at the predetermined separation distance between the sheet 30 and the user's hand, and the number of pixels as the reference value. Set as. The reference value set in the reference value setting unit 511 is the pixel value and the number of pixels of the shadow area for each predetermined predetermined separation distance between the sheet 30 and the user's hand, and the sheet 30 and the user's hand are arbitrarily set When the distance is within the distance, it becomes an index for relative calculation of the distance. The reference value setting unit 511 separately measures in advance the pixel value and the number of pixels of the shadow area when the sheet 30 and the user's hand are arranged at a predetermined separation distance, and directly inputs them as a reference value. Alternatively, the reference value may be automatically set from a photographed image when the sheet 30 and the user's hand are arranged at the above-described predetermined separation distance or a specific distance. In addition, the pixel value set by the reference value setting unit 511 as a reference value when automatically setting from a photographed image is set such that a light / dark change specifying unit 520 described later is arranged with the sheet 30 and the user's hand at the predetermined separation distance. In this case, the density specifying operation is performed based on the color data values of the pixels in the shadow area generated on the sheet 30, that is, the RGB values. On the other hand, the number of pixels set as the reference value is the size change specifying unit 521 or The pixel specifying unit 512 performs a specifying operation by counting the number of pixels in the shadow area, and the specified pixel value and the number of pixels are stored and set by the reference value setting unit 511 as a reference value.
When the reference value is automatically set, the reference value setting unit 511 improves the accuracy by setting the dynamically corrected pixel value by capturing the influence of the surrounding illuminance environment from the photographed image at regular intervals. You may plan.
Note that the reference value setting unit 511 of this example sets the reference value by, for example, prior measurement, for example, a state in which the user's hand (for example, a hand with an index finger raised) is in contact with the sheet 30 (reference separation distance Z0). ) The pixel value and the number of pixels in (input state) are set as a reference value C0, and the pixel value and the number of pixels in a state where the sheet 30 and the user's hand are separated (reference separation distance Z5) (non-input state) A reference value C5 is set.

Based on the captured image captured by the camera 20, the pixel specifying unit 512 specifies a pixel in a shadow area in the sheet 30 that is generated when the user blocks light emitted from the light source L onto the sheet 30. Specifically, the pixel specifying unit 512 specifies pixels in the shadow area within the area of the sheet 30 specified by the area specifying unit 502. The pixel specifying unit 512 analyzes each image captured at a plurality of timings by the camera 20 and specifies the pixels in the shadow area within the area of the sheet 30. For example, the pixel specifying unit 512 assumes that the shadow color is black (that is, the RGB value = 0) and specifies a pixel having the same color as black or a color close to black. This is an area.
In addition, the pixel specifying unit 512 may specify, for example, pixels in a portion other than the pixel area that is the color value (RGB value) of the specific color C of the sheet 30, and a region where the specified pixels are continuous may be used as a shadow region. . As a result, the pixels of the shadow area in the area of the sheet 30 are indirectly specified. Note that the pixel specifying unit 512 of the present embodiment specifies a region in which pixels of the same color as black or a color similar to black are continuous, thereby setting a shadow region in the region of the sheet 30.

The position specifying unit 514 specifies the position of the shadow of the user's hand or a specific part of the user's hand in the area of the sheet 30 in the captured image taken by the camera 20. Specifically, the position specifying unit 514 is an area for all the pixels corresponding to the user's hand among the pixels in the shadow area (exactly, the shadow area) in the sheet 30 specified by the pixel specifying unit 512. Alternatively, a pixel area of a specific part of the user's hand is extracted and its position is specified.
The position specifying unit 514 according to the present exemplary embodiment specifies the position of the fingertip on the sheet 30 of the captured image among the shadows of the user's hand specified by the pixel specifying unit 512. As for the fingertip specifying method, it may be specified by regarding the coordinates of the pixel at the upper left or upper right position in the camera-captured image as the fingertip position in the above-mentioned shadow area, or by pointing the index finger. When the sheet 30 is pressed down, the shadow in the vicinity of the fingertip is the darkest. Therefore, using the principle, all the pixels of the shadow of the hand that are more than a certain shade are extracted, It may be specified by determining the pixel at the center position and considering it as the shadow position of the fingertip. Of course, the system may be configured by regarding the entire shadow area as a user's hand.
The position specifying unit 514 can calculate the relative position of the shadow of the user's hand in the sheet 30 by making a relative comparison with the sheet 30 region in the camera-captured image specified by the sheet specifying unit 500 or the region specifying unit 502. It is.

  The movement distance specifying unit 516 specifies the movement amount of the user's shadow at a plurality of timings taken by the camera 20. Specifically, the moving distance specifying unit 516 determines the amount of movement by calculating the position change of the shadow of the user's hand specified by the position specifying unit 514 at the above-described multiple timings. To do. The movement distance specifying unit 516 of this embodiment specifies the movement amount in the shadow of the user's fingertip specified by the position specifying unit 514. The movement distance specifying unit 516 compares the positions of the shadows of the user's fingertips before and after the movement from a plurality of captured images captured at a plurality of times by the camera 20, and the shadows of the user's fingertips in the captured images (in the sheet 30). Specify the amount of movement.

  The movement direction identification unit 517 identifies the movement direction of the shadow at a plurality of timings taken by the camera 20. Specifically, the moving direction specifying unit 517 specifies the moving direction with respect to the captured image (in the sheet 30) in the shadow of the user's hand specified by the position specifying unit 514. The moving direction specifying unit 517 of this embodiment specifies the moving direction in the shadow of the user's fingertip specified by the position specifying unit 514. The movement direction specifying unit 517 moves the shadow of the user's fingertip in the captured image (in the sheet 30) based on the position of the shadow of the user's fingertip before and after the movement, from the captured images captured at a plurality of timings by the camera 20. Identify the direction.

  The moving speed specifying unit 518 specifies the moving speed of the shadow at a plurality of timings taken by the camera 20. The moving speed specifying unit 518 specifies the moving speed of the shadow based on the movement amount of the user's shadow specified by the moving distance specifying unit 516 and the shooting interval time of the shot images shot at a plurality of timings. The moving speed specifying unit 518 according to the present embodiment uses the captured image (in the sheet 30) based on the movement amount of the shadow of the fingertip of the user specified by the moving distance specifying unit 516 and the shooting interval time of the taken image before and after the movement. ) To specify the moving speed of the shadow of the user's fingertip.

  The light / dark change specifying unit 520 specifies the light / dark change in the shadow pixel specified by the pixel specifying unit 512. Specifically, the light / dark change specifying unit 520 uses, as a reference value, a pixel value based on a predetermined separation distance between the sheet 30 and the user's hand as a reference value, and a shadow of an image captured at a plurality of timings. The pixel value of the region is compared, and the increase / decrease in the pixel value is defined as a change in the shade of the shadow. Also, the light / dark change specifying unit 520 specifies binary information of shadow pixels based on whether or not the pixel value is darker than the reference value.

  The size change specifying unit 521 compares the size of the shadow region generated by the user blocking the light emitted from the light source L from a plurality of image data captured by the camera 20 at a plurality of timings. Identify changes in size. The size change specifying unit 521 according to the present embodiment sets the increase or decrease in the size of the shadow area specified by the pixel specifying unit 512 as a change in the size of the shadow. Specifically, the size change specifying unit 521 specifies a change in the size of the shadow area based on the number of pixels in the shadow area. The size change specifying unit 521 uses, as a reference value, the number of pixels based on a predetermined separation distance between the sheet 30 and the user's hand as a reference value, and the number of pixels in a shadow region of an image captured at a plurality of timings. And the increase / decrease in the size of the shadow area is defined as a change in the size of the shadow.

The separation distance specifying unit 522 is a distance from the sheet 30 to the user based on the change in shading specified by the shading change specifying unit 520 or the change in the size of the shadow area specified by the size change specifying unit 521. Is identified. Here, the identification of the distance is not limited to measuring the actual distance (numerical value), but is a change state of the distance (approaching / moving away, or the closest moment) Alternatively, it may be a determination result that is closer or farther than the reference value.
The separation distance specifying unit 522 of the present example is based on the change in shading specified by the shading change specifying unit 520 and the change in the size of the shadow area specified by the size change specifying unit 521. It is determined whether or not the distance between the closest contact point (for example, the fingertip) and the sheet 30 is closer than a predetermined reference distance (a distance that can be regarded as touching the sheet 30).

(Operation detection unit 530)
As illustrated in FIG. 5, the operation detection unit 530 monitors an input operation by the sheet 30 and an input operation by a shadow transmitted through the sheet 30 and detects it as a user operation.
As an input operation by the sheet 30, the operation detection unit 530 detects the amount of movement of the sheet 30 specified by the sheet specifying unit 500 based on the captured image captured by the camera 20.
In addition, as an input operation using a shadow, the operation detection unit 530 includes a position of the shadow specified by the shadow specifying unit 510, a movement amount of the shadow, a moving direction of the shadow, a moving speed of the shadow, a shading density or a shadow. The user's operation is detected based on the combination with the size. The operation detection unit 530 of this example is based on the position of the fingertip of the shadow, and the shadow movement operation based on the movement amount, movement direction, and movement speed from the reference position and the size of the shadow, or The user's operation is based on a combination with a shadow input / output operation based on a change in shadow shade.

(Operation signal output unit 540)
When the operation detection unit 530 detects a user operation, the operation signal output unit 540 outputs a response signal corresponding to the detected user operation in a form corresponding to the operated device 90.
The operation signal output unit 540 of this example, when a user operation is detected by the operation detection unit 530, the detected movement amount of the sheet 30, the detected shadow position, movement amount, movement direction of the user's fingertip, Also, the input operation based on the moving speed etc. is used as the pointer display coordinate control signal, and the input / output operation based on the change of the shadow size of the user's fingertip or the shade density is controlled. The ON / OFF signal is output to the operated device 90 (for example, the image display device 90c).
When the operated device 90 is an industrial robot 90b or the like, an input operation based on the above-described fingertip shadow position or moving direction may be output as a signal for controlling the position or moving direction of the robot. .
As described above, the input / output program 5 includes the sheet specifying unit 500, the shadow specifying unit 510, the operation detecting unit 530, and the operation signal output unit 540.

FIG. 8 is a diagram for explaining an image analysis image of a user's shadow transmitted through the sheet 30.
FIG. 8A is a diagram illustrating an image analysis image when the separation distance between the sheet 30 and the user's hand is longer than the reference separation distance Z5, and FIG. It is a figure which illustrates an image analysis image in case the separation distance of is nearer than the reference separation distance Z5.
The input / output program 1 previously stores pixel values (more specifically, pixel color values) in a state where the sheet 30 and the user's hand are separated by the reference value setting unit 511 (reference separation distance Z5) (non-input state). RGB values) and the number of pixels are set as the reference value C5.

As illustrated in FIG. 8, the camera 20 captures a state where the sheet 30 and the user's hand are separated (reference separation distance Z7), and the input / output device 20 analyzes the captured image.
The pixel specifying unit 512 sequentially scans and analyzes the pixels in the area of the sheet 30 of the photographed image from the left corner or the right corner of the photographed image, and the position specifying unit 514 selects a pixel having the same color as black or a color similar to black. The pixel of the part specified for the first time is specified as a pixel located at the fingertip, for example. The reason for paying attention to the pixel at the fingertip position is that the fingertip is closest to the sheet 30 in the hand during the input operation.
The light / dark change specifying unit 520 compares the pixel value of the pixel at the position of the fingertip specified by the position specifying unit 514 with the reference value C5 of the pixel value set by the reference value setting unit 511, and compares the relative values of the two. Binary information indicating whether the image is brighter than the gray value or its reference value is specified.

As illustrated in FIG. 8A, the light / dark change specifying unit 520 has a case where the pixel value of the pixel at the position of the fingertip specified by the position specifying unit 514 is brighter (lighter) than the pixel value of the reference value C5. The value of binary information indicating a bright (light) state is output.
The size change specifying unit 521 calculates the relative value of the number of pixels in the shadow area of the user specified by the pixel specifying unit 512 and the number of pixels of the reference value C5, and obtains the relative shadow obtained thereby. When the area is large, a value indicating a state where the shadow area is large is output.
The separation distance specifying unit 516 is a binary information value indicating a bright (thin) state specified by the shading change specifying unit 520, and the size change specifying unit 521 is a value indicating a state in which the more specified shadow region is large. Based on the above, it is determined by the reference separation distance Z5 that the separation distance Z7 is farther from the seat 30 and the user's hand. That is, the separation distance specifying unit 516 determines that the sheet 30 and the user's hand are far away, that is, OFF.

On the other hand, as illustrated in FIG. 8B, the shading change specifying unit 520 has a darker (darker) pixel value of the pixel at the fingertip position specified by the position specifying unit 514 than the pixel value of the reference value C5. ), A binary information value indicating a dark (dark) state is output.
The size change specifying unit 521 calculates the relative value of the number of pixels in the shadow area of the user specified by the pixel specifying unit 512 and the number of pixels of the reference value C5, and obtains the relative shadow obtained thereby. When the area is small, a value indicating that the shadow area is small is output.
The separation distance specifying unit 516 is a binary information value indicating the dark (dark) state specified by the light / dark change specifying unit 520, and the size change specifying unit 521 is a value indicating a state in which the more specified shadow region is small. Based on the above, it is determined from the reference separation distance Z5 that the separation distance Z7 is closer to the user's hand. That is, the separation distance specifying unit 516 determines that the sheet 30 and the user's hand are close, that is, ON.
Note that the separation distance specifying unit 516 of the input / output system of the present embodiment is based on both the shadow density (shading change specifying unit 520) and the size of the shadow area (size change specifying unit 521). However, the present invention is not limited to this, and the distance is specified using only one of the size of the shadow area or the shade of the shadow. May be.
In the perspective determination between the sheet 30 and the user's hand, the separation distance specifying unit 516 uses both the above-described shade density (binary information) and the size of the shadow region (relative increase / decrease in the size of the shadow region). It is possible to improve perspective determination accuracy by making a determination using Further, the separation distance specifying unit 516 can improve the processing speed by performing perspective determination using only one of the size of the shadow region or the shade of the shadow. For example, if the determination is made using only the shading density (binary information), it is not necessary to extract all the pixels in the shadow area, and the pixel value at the position of the user's fingertip specified by the position specifying unit 514 is analyzed. Therefore, the processing speed can be improved.
As described above, the input / output system 1 scans and analyzes the pixels of the entire image by performing image analysis when focusing on the pixel located at the fingertip of the user's hand in the shadow region in the captured image. Therefore, since the position of the user's fingertip can be specified, the user operation can be detected at high speed with a light processing load. Thereby, for example, a system suitable for a touch pad (track pad or the like) disposed in the air can be obtained.

FIG. 9 is a diagram illustrating an image analysis image that identifies the shape of the sheet 30. FIG. 9A is a diagram illustrating an image analysis image of the bent sheet 30 in the captured image, and FIG. 9B is an image analysis image of the inclined sheet 30 in the captured image. FIG.
As illustrated in FIG. 9A, the camera 20 captures an image of a sheet 30 that is inclined at an angle R1 with respect to the upper peripheral edge of the imaging range and an angle R2 with respect to the lower peripheral edge of the imaging range. Therefore, the sheet 30 at the time of image analysis is inclined by an angle R1 with respect to the upper side of the outer periphery of the captured image and an angle R2 with respect to the lower side.
The shape specifying unit 504 extracts the pixel portion of the specific color C of the sheet 30 specified by the region specifying unit 502, and specifies the overall shape of the sheet 30 including the shadow portion from the extracted pixel portion.
The shape correcting unit 507 calculates the angle inclinations of the angle R1 and the angle R2 from the specified overall shape of the sheet 30, and corrects it to the shape of the sheet 30 before deformation.
Note that the calculated values of the angle R1 and the angle inclination of the angle R2 may be extended so that the operation signal output unit 540 can output the values together with the shadow position and movement amount of the user's fingertip. In this way, for example, when the user greatly tilts the seat 30, it is possible to add the same meaning as the operation of the user pressing a key such as the Shift key on the keyboard (that is, touch with the Shift key held down). Pad operation can be reproduced), and various operations can be performed only by operating the sheet 30.

Further, as illustrated in FIG. 9B, the camera 20 captures the sheet 30 inclined at an angle T with respect to the capturing direction. For example, the sheet 30 is inclined so that the upper side is closer to the camera 20 and the lower side is farther from the camera 20. Therefore, the sheet 30 at the time of image analysis has an inverted trapezoidal shape in which the number of pixels on the upper side A1 is larger than the number of pixels on the lower side A2.
The shape specifying unit 504 extracts the pixel portion of the specific color C of the sheet 30 specified by the region specifying unit 502, and specifies the overall shape of the sheet 30 including the shadow portion from the extracted pixel portion.
The shape correcting unit 507 can calculate the ratio of the upper side A1 and the lower side A2 from the specified overall shape of the sheet 30, and obtain the inclination angle T based on the perspective principle. Thereby, the shape correction | amendment part 507 can correct | amend to the shape of the sheet | seat 30 before a deformation | transformation. Further, the operation detection unit 530 detects the calculated value of the tilt angle T as a user operation content in addition to the shadow position and the amount of movement of the shadow, and the operation signal output unit 540 includes the operation detection unit. The operation content detected by 530 may be extended to be output to the operated device 90. Various operations are possible.
Although the case of the upper side and the lower side has been described, the same applies to the case of the right side and the left side, and the description thereof will be omitted.

FIG. 10 is a flowchart illustrating the input / output process (S10) in the input / output system 1. The input / output process (S10) will be described on the assumption that the reference value setting by the reference value setting unit 511 has been completed. The input / output process (S10) is a process performed every time the camera 20 captures an image.
First, the setting of the reference value will be described.
The user sets the reference value before performing the input / output process (S10). The user arranges the sheet 30 at a position facing the user's hand through the connecting member 70, and the state in which the hand is brought into contact with the sheet 30 (reference separation distance Z 0) (input state), and the sheet 30 and the hand are The camera 20 is caused to photograph the separated state (reference separation distance Z5) (non-input state).
The reference value setting unit 511 sets the pixel value and the number of pixels when the hand is in contact with the sheet 30 (reference separation distance Z0) as the reference value C0, and the sheet 30 is separated from the hand (reference separation distance Z5). ) And the number of pixels are set as the reference value C5. At this time, the description will be made assuming that the shape of the user's hand is, for example, a state in which the index finger is raised.

As illustrated in FIG. 10, in step 100 (S <b> 100), the camera 20 periodically captures the sheet 30 and transmits the captured image to the input / output processing device 10.
In step 102 (S102), the input / output process (S10) performs the sheet input process (S20). In the input / output process (S10), after the process of the sheet input process (S20) is completed, the process proceeds to the process of step 104 (S104).
In step 104 (S104), the input / output process (S10) performs the shadow input process (S30). In the input / output process (S10), after finishing the shadow input process (S30), the process proceeds to step 106 (S106).
In step 106 (S106), the operation signal output unit 540 displays the operation contents output by the sheet input process (S20) and the operation contents output by the shadow input process (S30) as the user operation contents. Is output to the operated device 90.
The operation content output by the process of the sheet input process (S20) is a shadow input operation (pointer movement operation) based on the movement amount of the sheet 30, and the operation output by the process of the shadow input process (S30). The contents are the shadow movement operation (pointer movement operation) based on the movement amount, movement direction, and movement speed of the fingertip of the shadow, and the input of the shadow based on the change in shadow size or shade This is a combination operation with an operation (so-called click operation).

FIG. 11 is a flowchart illustrating sheet input processing (S20) in the input / output system 1.
As illustrated in FIG. 11, in step 200 (S200), the region specifying unit 502 sequentially scans and analyzes the shot images shot by the camera 20, and the color value (RGB value) of the specific color C is continuous. By specifying the region, the region of the sheet 30 colored with the specific color C is specified. The shape specifying unit 504 extracts the pixel portion of the specific color C of the sheet 30 specified by the region specifying unit 502, and specifies the overall shape of the sheet 30 from the extracted pixel portion.

In step 202 (S202), the shape correction unit 507 determines whether or not the specified overall shape of the sheet 30 is deformed (for example, tilted or bent).
In the sheet input process (S20), when the shape of the sheet 30 specified by the shape correction unit 507 is deformed, the process proceeds to S204, and when the shape of the specified sheet 30 is not deformed, The process proceeds to S206.

  In step 204 (S204), the shape correction unit 507 identifies the deformation (for example, inclination or deflection) of the sheet 30, and corrects the shape before the deformation.

In step 206 (S206), the operation detection unit 530 detects whether or not the sheet 30 has moved. The operation detection unit 530 detects the presence or absence of the movement amount of the sheet 30 specified by the movement distance specification unit 508.
The sheet input process (S20) proceeds to the process of S208 when a user operation is detected, and ends the sheet input process (S20) when the user operation is not detected.

  In step 208 (S208), the moving distance specifying unit 508 compares the captured images before and after the movement of the sheet 30 from a plurality of captured images captured by the camera 20. The movement distance specifying unit 508 specifies the start position coordinates of the left corner of the sheet 30 as a reference point, and specifies the movement distance of the reference point as the movement amount of the sheet 30. Note that the flowchart of FIG. 11 is merely an example. For example, when it is used as one of output values of the tilt input / output system 1 as well as the position and movement amount of the sheet 30, it is necessary to add the processing separately.

In step 210 (S210), the sheet specifying unit 500 outputs a movement detection signal of the detected sheet 30 to the operation detecting unit 530 and outputs an operation signal indicating the movement amount of the sheet 30 specified by the moving distance specifying unit 508. Output to the unit 540. Instead of outputting the detection signal, a flag for monitoring the movement state of the sheet 30 is prepared in an appropriate shared memory space, and the state of no movement is set by turning on the external processing routine or object or object. The operation device 90 or the like may be referred to at an arbitrary timing.
The sheet input process (S20) proceeds to the process of S104 of the input / output process (S10) after the process of S208 ends.

FIG. 12 is a flowchart illustrating the shadow input process (S30) in the input / output system 1.
As illustrated in FIG. 12, in step 300 (S <b> 300), the pixel specifying unit 512 displays a shadow region (the same color as black indicating black or a color similar to black) within the region of the sheet 30 specified by the region specifying unit 502. A region where the pixels are continuous is specified. In addition, as a specific algorithm example used for specifying at this time, an RGB value close to black is specified and specified from pixels in the sheet 30 area, or an RGB value of a specific color C which is the color of the sheet 30 For example, a method for extracting and specifying the other region using the. Note that the pixel specifying unit 512 of this embodiment specifies a shadow area in the area of the sheet 30 by specifying a color that is the same color as black or similar to black and specifying an area in which the specified pixels are continuous.

In step 302 (S302), the shadow specifying unit 510 specifies whether or not to specify an area in which pixels in the shadow area specified by the pixel specifying unit 512 (the same color as that of black or a color similar to black) are continuous. judge.
The shadow input process (S30) proceeds to the process of S304 when the shadow area is specified by the shadow specifying unit 510, and when the shadow area is not specified (that is, the user's hand does not exist near the sheet 30), The main shadow input process (S30) is terminated.
In this determination, the number of continuous pixels in the shadow region, that is, the threshold value of the shadow region area may be increased to reduce the noise of the photographed image to increase the accuracy. Conversely, the sensitivity of detecting a shadow area may be increased by determining that there is a shadow area when even one pixel in the shadow area is found.

  In step 304 (S304), the position specifying unit 514 specifies the coordinates of the shadow pixel of the fingertip portion of the user's hand in the shadow area specified by the pixel specifying unit 512. In addition, as a specific algorithm example used for identification at this time, it is specified by regarding the coordinates of the pixel at the upper left or upper right position in the shadow area specified by the pixel specifying unit as the position of the fingertip. And a method of identifying the pixel at the center position of the shadow area and identifying it as the position of the fingertip.

  In step 306 (S306), the moving distance specifying unit 516 specifies the amount of movement in the shadow of the user's hand specified by the position specifying unit 514, and the moving direction specifying unit 517 is the user specified by the position specifying unit 514. The moving direction in the shadow of the fingertip is specified. In addition, the moving speed specifying unit 518 uses the moving amount of the shadow of the user's fingertip specified by the moving distance specifying unit 516 and the shooting interval time of the shot images before and after the movement in the shot image (in the sheet 30). The moving speed of the shadow of the user's fingertip is specified. (So-called pointer movement operation)

  In step 308 (S308), the light / dark change specifying unit 520 specifies the light / dark change in the pixel value (ie, RGB value) of the shadow pixel of the fingertip portion of the user's hand specified by the position specifying unit 514. In addition, the size change specifying unit 520 specifies the pixel range, that is, the size of the shadow of the fingertip portion of the user's hand specified by the position specifying unit 514.

In step 310 (S310), the light / dark change specifying unit 520 relatively compares the pixel value of the shadow pixel of the fingertip portion of the specified user's hand with the pixel value of the reference value C5 set by the reference value setting unit 511. . Further, the size change specifying unit 520 relatively compares the shadow region of the fingertip portion of the specified user's hand with the pixel region of the reference value C5 set by the reference value setting unit 511.
The separation distance specifying unit 522 is a distance from the sheet 30 to the user based on the change in shading specified by the shading change specifying unit 520 and the change in the size of the shadow area specified by the size change specifying unit 521. Is calculated.

In step 312 (S312), the shadow specifying unit 510 determines the pixel value of the shadow pixel of the fingertip portion of the user's hand specified by the shading change specifying unit 520 and the pixel of the reference value C5 set by the reference value setting unit 511. Based on the comparison result with the value and the comparison result between the shadow region of the fingertip portion of the user's hand specified by the size change specifying unit 520 and the pixel region of the reference value C5 set by the reference value setting unit 511 Then, it is determined whether or not the shadow of the fingertip portion of the user's hand is darker than the reference value C5.
In the shadow input process (S30), when the pixel is darker (darker) than the reference value by the shadow specifying unit 510, in other words, when the sheet 30 is close to the user's fingertip, the process proceeds to S314, and the pixel is lighter (brighter) than the reference value. In other words, when the sheet 30 is far from the user's fingertip, the process proceeds to S316.

  In step 314 (S314), the light / dark change specifying unit 520 is dark (dark) when the pixel value of the pixel at the fingertip position specified by the position specifying unit 514 is darker than the pixel value of the reference value C5. ) ON indicating the state is set in the binary information.

In step 316 (S316), the light / dark change specifying unit 520 is light (bright) when the pixel value of the pixel at the fingertip position specified by the position specifying unit 514 is thinner (brighter) than the pixel value of the reference value C5. ) OFF indicating the state is set in the binary information.
In the case of setting the binary information, the method is not limited to the method of comparing the shades based on the reference values as in S314 and S316 of this example, for example, based on the separation distance calculated in S310 described above, The binary information may be set to ON when the separation distance between the sheet 30 and the fingertip of the user's hand is 5 mm or less, and OFF when the separation distance is 5 mm or more.

In step 318 (S318), the shadow specifying unit 510 is specified by the position and moving distance specifying unit 516 of the shadow specified by the position specifying unit 514 together with some operation detection signal based on the shadow detected by the operation detecting unit 530. Numerical values such as binary information indicating the amount of movement, the separation distance of the user's hand from the sheet 30 identified by the separation distance identification unit 522, and the state of the near and far separation distance identified by the light / dark change identification unit 20 Are output to the operation signal output unit 540.
Instead of outputting a detection signal, a flag indicating whether or not some operation is detected by a shadow is prepared in an appropriate shared memory space, and it is turned on to determine whether or not an operation by a shadow is detected by an external processing routine or object, Alternatively, the operated device 90 or the like may be referred to at an arbitrary timing.
The shadow input process (S30) proceeds to the process of S106 of the input / output process (S10) after the process of S312 ends.

As described above, according to the input / output system 1 of the present embodiment, by detecting the user's operation from the captured image including the user and the shadow transmitted through the sheet 30, the camera 20 and the user can be detected. Even if a window or the like is interposed, the operated device 90 can be a pointing device that can be operated from a remote place. Further, since the input / output system 1, the input / output device 10, the camera 20, and the operated device 90 can be disposed indoors, the input / output system 1 is not exposed to outdoor wind and rain, and the risk of theft can be suppressed.
Further, the input / output system 1 binarizes the captured image by using the light source L by the light transmitted through the sheet 30 and the shadow of the user's hand between the sheets, and analyzes the binarized image. The processing load on the computer is low, and it is highly accurate and fast compared to a system that determines the position of the user's hand from the captured image as it is. In addition, the input / output system 1 accurately identifies the position of the fingertip because the user's hand and fingertip are not hidden in the user's body and hand unlike the case of using shadows reflected from the sheet. Yes, if the user's fingertips do not approach the sheet more than a certain amount, the transmitted shadow is difficult to be captured by the camera. . In addition, the degree of masking can be adjusted by adjusting the thickness of the sheet or the light transmittance.
The seat 30 is lightweight because it does not include a cord, a communication device (wireless device, infrared device), or the like. Therefore, since the burden on the user who operates can be reduced, a mouse-type pointing device can be implemented by supporting the seat 30 on either the left or right arm and moving it hollow, as will be described in detail later. Moreover, although it is limited to the environment where the camera 20 is transparent enough to photograph the sheet 30, it can be used underwater.
In addition, although the input / output system 1 of this embodiment demonstrated the case where the sheet | seat 30 was mounted | worn with a user's arm, it is not limited to this, You may fix the sheet | seat 30 to a window glass beforehand. In this case, it is preferable to fix the seat 30 at a position that does not hinder the user's field of view.

Next, a modification of the above embodiment will be described.
In the modification, elements having substantially the same functions and configurations as those in the above embodiment are denoted by the same reference numerals, and redundant description is omitted.
[Modification 1]
The input / output system 100 according to the first modification is a system in which the color of the peripheral area of the sheet 30 is different from the color of the central area, and the sheet 30 is specified based on the color of the peripheral area of the sheet 30.
FIG. 13 is a diagram illustrating a first modification of the input / output system 1 in the embodiment.
As illustrated in FIG. 13, the sheet 30 of the first modification is a translucent plate-like screen colored with a specific color C other than black or gray as in the above embodiment. However, the sheet 30 of the first modification is formed so as to be colored in a plurality of colors. The sheet 30 is formed so that, for example, the color that develops in the central region and the color that develops in the peripheral region are different. In the sheet 30 of the first modification, the color that develops in the central region is the specific color C, and the color that develops in the peripheral region is the specific color CA. As a coloring method of the peripheral region of the sheet 30, it can be realized by adhesion of a colored cloth or paper, painting, or the like.

  In addition, the area specifying unit 502 in the sheet specifying unit 500 of the input / output program 5 takes a picture with the camera 20 in order to specify the sheet 30 formed so that the color to be developed in the central area is different from the color to be developed in the peripheral area. The region of the sheet 30 is specified based on the color (specific color CA) of the peripheral region of the sheet 30 in the captured image. In other words, the area specifying unit 502 specifies an area surrounded by the specific color CA.

As described above, the input / output system 100 of Modification 1 forms the color of the peripheral area of the sheet 30 to be different from the color of the central area, and causes the area specifying unit 502 to specify the area surrounded by the color of the peripheral area. Thus, the detection accuracy can be improved as compared with the case where the monochromatic sheet 30 is specified from the captured image.
Further, the input / output system 1 causes the area specifying unit 502 to specify an area surrounded by the specific color CA at the time of image analysis, so that the specific color C in the central area of the sheet 30 is not included in the area surrounded by the specific color CA. Even if an object M having the same color as that of the sheet 30 exists, it can be processed as noise, and erroneous detection of the sheet 30 can be prevented.

[Modification 2]
The input / output system 102 according to the second modification is an input / output system in which the distance between the light source L and the sheet 30 is variably fixed.
FIG. 14 is a diagram illustrating a second modification of the input / output system 1 in the embodiment. FIG. 14A is a diagram illustrating a photographed image when the sheet 30 is moved away from the user's hand, and FIG. 14B illustrates a photographed image when the sheet 30 is brought close to the user's hand. FIG.
The input / output system 102 according to the second modification includes a connection member 80 in addition to the input / output processing device 10, the camera 20, the sheet 30, the connection member 70, and the operated device 90.
The connecting member 80 is a connecting tool that connects the light source L and the sheet 30. The connecting member 80 connects the light source L that irradiates light to the other surface of the sheet 30 and the separation distance between the sheet 30 so as to be variable. For example, the connecting member 80 is slidably connected to a support member that supports the light source L and a support member that supports the sheet 30.
As illustrated in FIG. 14A, the connecting member 80 moves the sheet 30 away from the user's hand, so that the shadow area projected on the sheet 30 in the image captured by the camera 20 is It becomes larger than the size of the hand.
On the other hand, as illustrated in FIG. 14B, the connecting member 80 brings the sheet 30 close to the user's hand, so that the shadow area projected on the sheet 30 in the image captured by the camera 20 is Is approximately equal to the size of the hand.
As described above, the connecting member 80 can adjust the size of the shadow projected on the sheet 30 by changing the separation distance between the light source L and the sheet 30.
As a result, the input / output system 102 can detect, for example, even a minute movement of the fingertip by enlarging the shadow size, and thus can prevent misdetection even in an input operation that requires a minute movement of the fingertip. . For example, in the case of a drag operation that is output by an operation of shaking the wrist to the left or right, the amount of movement to be output can be increased by detecting a small movement of the wrist while the shadow size is increased. . That is, the amount of movement to be output can be increased or decreased by enlarging or reducing the size of the shadow on the sheet 30.

[Modification 3]
The input / output system 104 according to the third modification is an input / output system that identifies inputs from a plurality of sheets 30 and detects user input operations.
FIG. 15 is a diagram illustrating a third modification of the input / output system 1 in the embodiment.
As illustrated in FIG. 15, the input / output system 104 includes an input / output processing device 10, a camera 20, a seat 30, and an operated device 90. The input / output system 104 of the third modification shoots the shadow of the hand of the user 1 projected on the sheet 30A and the shadow of the hand of the user 2 projected on the sheet 30B so that the camera 20 fits in the same image. The input / output processing device 10 detects a user operation on each sheet 30 from the captured image, and outputs the detected user operation to the image display device 90c.
There are a plurality of sheets 30 of the third modification, and each of them develops a different color. Specifically, the sheet 30 includes a sheet 30A that develops a specific color C and a sheet 30B that develops a specific color C2.

FIG. 16 is a diagram illustrating a functional configuration of the input / output processing device 10 according to the third modification.
As illustrated in FIG. 16, the input / output program 5 installed in the input / output processing device 10 according to the third modification includes a sheet specification unit 500, a shadow specification unit 510, an operation detection unit 530, and an operation signal output unit 540. In addition to this, it further includes an identification unit 550.
The identification unit 550 identifies the color that the sheet 30A develops and the color that the sheet 30B develops based on the captured image captured by the camera 20. Specifically, the identification unit 550 specifies the color of the specific color C2 that the color of the specific color C (RGB value) that the sheet 30A develops and the specific color C2 that the sheet 30B develops that are specified by the region specifying unit 502. An identification ID is assigned to each region where values (RGB values) are continuous.

  Further, in the input / output process (S10) of the input / output system 1 of the third modification, the identification unit 550 assigns an identification ID to each sheet 30 after the captured image is transmitted to the input / output processing device 10 (S100). (S101), the processing from S102 to S106 is performed for each sheet 30 identified by the identifying unit 550. In addition, in the process from S102 to S106, since it is the same process as the said embodiment, duplication description is abbreviate | omitted.

As described above, according to the input / output system 104 of the third modification, it is possible to detect and output operations from a plurality of users with one camera 20.
The input / output system 104 according to the third modification has been described with respect to a mode in which a plurality of users use sheets 30 of different colors. However, the present invention is not limited to this. Different colored sheets 30 may be used.

[Modification 4]
The input / output system 106 according to the fourth modification is an input / output system that reproduces an input operation on a flexible material.
FIG. 17 is a diagram illustrating a fourth modification of the input / output system 1 in the embodiment.
As illustrated in FIG. 17, the input / output system 106 in Modification 4 detects the user's operation from the photographed image, and outputs the detected operation to the image display device 90 c, thereby allowing the user to operate. This is a system in which the character image to be displayed changes according to the operation content.
The input / output system 106 includes a buffer member 85 in addition to the input / output processing device 10, the camera 20, the sheet 30, and the operated device 90.
The cushioning member 85 is a cushioning material that is attached to the surface of the sheet 30 and gives an appropriate resistance to the operation of the user's hand or fingers. The buffer member 85 of the fourth modification is a bag having an injection port for injecting air, and can be filled with air and enclosed. The material of the buffer member 85 is formed of a synthetic resin having translucency and elasticity. Specifically, the buffer member 85 is made of soft vinyl chloride (soft PVC) that is transparent or translucent, or rubber that is transparent or translucent. Therefore, the buffer member 85 can be formed as a bag that can be freely deformed in accordance with a user's hand operation (for example, an operation of pressing with a fingertip). Further, the buffer member 85 can freely adjust the amount of air to be filled.
Thus, by attaching the buffer member 85 to the surface of the sheet 30, a three-dimensional pointing device that can freely change flexibility and elasticity according to the operation of the user's hand can be obtained.
Therefore, the input / output system 106 synchronizes the output of the user operation based on the position of the shadow with the user's hand buried in the buffer member 85 and the output of the image change of the character image displayed on the image display device 90c. Can be displayed. As a result, the user can expect to feel as if he / she directly touched the displayed character image.
Also, the input / output system 106 is different from a pointing device using a piezoelectric element that uses a piezo effect or a change in capacitance, based on the position, density, or size of a user's hand shadow. Since the operation is performed, the separation distance between the sheet 30 and the hand according to the amount of burying of the buffer member 85 can be easily specified.

[Modification 5]
The input / output system 108 of the fifth modification is an input / output system in which a user's input operation is synchronized with a work window.
FIG. 18 is a diagram illustrating a fifth modification of the input / output system 1 in the embodiment.
As illustrated in FIG. 18, in the input / output system 106 according to the fifth modification, the input / output processing device 10 detects a moving operation of the sheet 30 and an input operation based on a shadow from a captured image as a user operation, and the detection is performed. In this system, the operation window W is output to the image display device 90c to operate the work window W in accordance with the user operation contents.

In the input / output program 5 installed in the input / output processing apparatus 10 according to the fifth modification, the sheet specifying unit 500 includes the region specifying unit 502, the shape specifying unit 504, the shape correcting unit 507, and the moving distance specifying unit 508. And further includes a synchronization unit 503.
The synchronization unit 503 includes a reference point of the sheet 30 within the photographing range of the camera 20 (for example, a start position coordinate of the left corner of the sheet 30 in the image analysis image) and a reference coordinate of the work window W within the display range of the image display device 90c. (For example, the start position coordinates of the left corner). The synchronization unit 503 can synchronize the amount of movement of the sheet 30 in the captured image with the amount of movement of the work window W within the display range by matching the coordinates serving as the reference.
Accordingly, the user can freely move the work window W displayed on the image display device 90c in conjunction with the operation of moving the sheet 30. Further, the user can perform an operation (click operation or the like) on the work window W by performing an operation of moving the work window W and projecting an input operation by projecting a hand shadow onto the sheet 30.

[Modification 6]
The input / output system 110 according to Modification 6 is an output system that detects a user input operation based on the inclination or shape of the seat 30. For example, the input / output system 106 allows the user to operate the sleep mode or the input end operation content by performing an operation such as laying the sheet 30 horizontally with respect to the shooting direction or folding the sheet 30. Output to 90.
FIG. 19 is a diagram illustrating a sixth modification of the input / output system 1 in the embodiment.
As illustrated in FIG. 19, the input / output system 110 allows the user to lay down the sheet 30 horizontally with respect to the shooting direction, or to perform an operation such as folding the sheet 30, thereby The operation content is output to the operated device 90.
FIG. 20 is a diagram for explaining the sheet specifying unit 500 in Modification 6 in more detail.
As illustrated in FIG. 20, in the input / output program 5 installed in the input / output processing device 10 according to the modified example 6, the sheet specifying unit 500 includes an area specifying unit 502, a shape specifying unit 504, a shape correcting unit 507, and In addition to the movement distance specifying unit 508, an inclination specifying unit 506 and an operation content determining unit 509 are further included.

  The tilt identifying unit 506 identifies the tilt of the sheet 30 with respect to the captured image based on the captured image captured by the camera 20. Specifically, the inclination specifying unit 506 specifies the inclination of the sheet 30 based on the shape of the region of the sheet 30 specified by the shape specifying unit 504. The inclination specifying unit 506 of Modification 6 pre-records the shape of the sheet 30 (for example, the aspect ratio) in the setting information database 590 as a reference value, and uses the recorded reference value of the shape of the sheet 30 and the shape specifying unit 504. The inclination of the sheet 30 is specified by comparing with the shape of the specified sheet 30.

The operation content determination unit 509 determines the operation content of the user based on the shape of the sheet 30 specified by the shape specifying unit 504 or the inclination of the sheet 30 specified by the inclination specifying unit 506. In this example, the setting information DB 590 has a plurality of operation contents (for example, sleep mode, input end, etc.) and a plurality of shapes (for example, the shape and inclination of the sheet 30 laid horizontally with respect to the shooting direction, and the shooting direction). And the operation content determination unit 509 specifies the shape specified by the shape specification unit 504 and the inclination specification unit 506. The inclination, the shape of the sheet 30 registered in the setting information DB 590, and the inclination are compared, and the operation content associated with the closest approximation shape is determined as the user operation content. In principle, when the sheet 30 is horizontal with respect to the shooting surface of the camera 20 as shown in FIG. 19, the sheet is hardly captured in the camera-captured image, or the sheet in the horizontal direction with respect to the camera-captured image. Since the image is only slightly captured as a line of the same color as 30, the process may be shifted to a sleep mode or an input end process when the sheet is not captured from the image captured by the camera.
As described above, according to the input / output system 110 of the present modification, the sleep mode or the input end can be performed by performing an operation such as placing the sheet 30 horizontally with respect to the photographing direction, or folding the sheet 30. The operation content can be output to the operated device 90.

[Modification 7]
The input / output system 112 of Modification 7 synchronizes the position and amount of movement of the sheet 30 in the captured image with the position of the mouse pointer displayed on the image display device 90c, so that the fingertip position where the sheet 30 is pressed is placed. Based on this, the system performs a right click or left click input operation and outputs it to the operated device 90.
FIG. 21 is a diagram illustrating a modified example 7 of the input / output system 1 in the embodiment.
As illustrated in FIG. 21, in the input / output system 112, the input / output processing device 10 detects a movement operation of the sheet 30 and an input operation based on a shadow as a user operation from a captured image, and the detected operation content is displayed as an image. By outputting to the display device 90c, the control of the position of the mouse pointer and the click operation are performed according to the content of the user's operation.
Note that the input / output system 112 of Modification 7 uses the connecting member 70 to connect the seat 30 and the user's arm, and keeps the spacing between the seat 30 and the user's arm constant.

In the input / output program 5 installed in the input / output processing apparatus 10 according to the modified example 7, the sheet specifying unit 500 includes the area specifying unit 502, the shape specifying unit 504, the shape correcting unit 507, and the moving distance specifying unit 508. In addition, an area dividing unit 501 and a synchronizing unit 503 are further included.
The area dividing unit 501 divides the area of the sheet 30 within the captured imaging range into a plurality of parts, and distributes the operation contents. Specifically, the area dividing unit 501 divides the operation content into two or more areas in the horizontal direction or the vertical direction of the sheet 30. The area dividing unit 501 of the present modification 7 divides the sheet 30 into, for example, two areas (left area and right area) in the left-right direction, and assigns the left area to, for example, a left click operation, and assigns the operation to the right area. For example, a right click operation is performed.

  The synchronization unit 503 obtains the reference point of the sheet 30 within the photographing range of the camera 20 (for example, the start position coordinate of the left corner of the sheet 30 in the image analysis image) and the coordinates of the mouse pointer within the display range of the image display device 90c. Match. The synchronization unit 503 can synchronize the movement amount of the sheet 30 in the captured image with the movement amount of the mouse pointer within the display range of the image display device 90c by matching the coordinates. At this time, the synchronization unit 503 increases the amount of movement of the mouse pointer when the user moves the sheet 30 quickly. Conversely, when the user moves the sheet 30 slowly, the amount of movement of the mouse pointer increases accordingly. The amount of movement of the sheet 30 per unit time, that is, the sheet 30 speed may be set as a coefficient so as to be multiplied by the amount of movement of the mouse pointer so as to decrease. As a result, the mouse pointer can be moved greatly without particularly moving the arm or the like to which the seat 30 is fixed. Further, the synchronization unit 503 may increase or decrease the amount of movement of the mouse pointer depending on the inclination of the sheet 30.

As described above, in the input / output system 112 according to the seventh modified example, in the two areas of the sheet 30 divided by the area dividing unit 501, the position of the fingertip specified by the position specifying unit 514 is one of the two areas. The operation detection unit 530 detects and distributes the contents of the operation to the operated device 90 by performing a shadow input operation (so-called click operation) based on a change in shadow size or shadow density. Can be output.
Unlike a mouse that is an existing pointing device, the input / output system 112 can operate the mouse pointer by the movement of the user's arm. In addition, unlike the mouse, there is no need to fix the device to the desk or the floor, and the hollow operation by the sheet 30 supported by either the left or right arm is possible, and the movement amount of the mouse pointer is increased or decreased by the inclination of the sheet 30. Can be extended. Furthermore, the touch operation or the click operation is not only divided into two areas but also divided into three areas (left area, middle area, right area), and the operation assigned to the left area is, for example, left click operation, and assigned to the middle area. For example, a scroll operation may be used, and an operation assigned to the right area may be a right click operation, for example, and if divided into smaller areas, a device capable of various click operations can be realized.

DESCRIPTION OF SYMBOLS 1 Input / output system 10 Input / output processing device 20 Camera 30 Sheet 70 Connecting member 90 Operated device 5 Input / output program

Claims (7)

A translucent screen;
A camera that captures an image including the screen at a plurality of timings;
Based on the image captured by the camera, the position of the screen at a plurality of timings, size, amount of movement, inclination, or has perforated the operation detection means for detecting the operation of the user based on the shape ,
The camera is a camera that captures a captured image including an area where RGB values of the screen are continuous,
The operation detection means detects a user operation based on the position of the user's shadow reflected on the screen at a plurality of timings ,
A connection member for connecting the screen and the user's limb at an interval at which a shadow of a part of the user's limb is projected on the screen;
Output system the connecting member is in the screen surface view of the other side of the screen, at a position where a portion is opposed limb of the user, characterized by connecting the limb of the screen and the user . The screen is plate-like, and a shadow generated by the user blocking the light irradiated by the light source is projected on the other surface of the screen,
The camera shoots a shadow projected on the other side of the screen from one side of the screen at a plurality of timings,
Said operation detecting means, according to claim 1, characterized in that said at images picked up by the camera, based on the position of the shadow captured by light transmitted on one surface of the screen, it detects an operation of the user The input / output system described in 1 . A translucent screen;
A camera that captures an image including the screen at a plurality of timings;
Based on the image captured by the camera, the position of the screen at a plurality of timings, size, amount of movement, inclination, or has perforated the operation detection means for detecting the operation of the user based on the shape ,
The camera is a camera that captures a captured image including an area where RGB values of the screen are continuous,
The operation detection means detects a user operation based on the position of the user's shadow reflected on the screen at a plurality of timings ,
Output system characterized by further comprising a light source for irradiating light on the other surface of the screen, a coupling member for coupling the distance between the screen variably possible. A translucent screen;
A camera that captures an image including the screen at a plurality of timings;
Based on the image captured by the camera, the position of the screen at a plurality of timings, size, amount of movement, inclination, or has perforated the operation detection means for detecting the operation of the user based on the shape ,
The camera is a camera that captures a captured image including an area where RGB values of the screen are continuous,
The operation detection means detects a user operation based on the position of the user's shadow reflected on the screen at a plurality of timings ,
Pixel specifying means for specifying a shadow pixel in the screen based on a captured image captured by the camera;
A shade change specifying means for specifying a change in shade in the shadow pixel specified by the pixel specifying means;
The operation detection means detects a user's operation based on a change in shadow shade specified by the shade change specifying means .
It is provided connected to the other surface of the screen, and further has a light-transmitting and elastic buffer member,
The input / output system characterized in that the pixel specifying means specifies a shadow pixel transmitted through the screen and the buffer member. The screen is plate-like, and a shadow generated by the user blocking the light irradiated by the light source is projected on the other surface of the screen,
The camera shoots a shadow projected on the other side of the screen from one side of the screen at a plurality of timings,
Said operation detecting means, according to claim 4, characterized in that said at images picked up by the camera, based on the position of the shadow captured by light transmitted on one surface of the screen, it detects an operation of the user The input / output system described in 1 . A translucent screen;
A camera that captures an image including the screen at a plurality of timings;
Based on the image captured by the camera, the position of the screen at a plurality of timings, size, amount of movement, inclination, or has perforated the operation detection means for detecting the operation of the user based on the shape ,
The camera is a camera that captures a captured image including an area where RGB values of the screen are continuous,
The operation detection means detects a user operation based on the position of the user's shadow reflected on the screen at a plurality of timings ,
A shadow specifying means for specifying the position of the shadow in the screen, the amount of movement of the shadow, the direction of movement of the shadow, the speed of movement of the shadow, the size of the shadow, or the shade of the shadow;
A screen specifying means for specifying the position of the screen, the size of the screen, the amount of movement of the screen, the tilt of the screen, or the shape of the screen in the captured image captured by the camera;
The operation detecting means includes the position, moving amount, moving direction, moving speed, or size or shade specified by the shadow specifying means, and the position, moving amount, inclination, specified by the screen specifying means, or Based on the combination with the shape, it detects the user's operation ,
Synchronization means for matching the coordinates of the predetermined position of the screen in the captured image with the coordinates of the pointer in the display range of the operated device;
Further comprising an operation content outputting means for outputting an operation content detected by the operation detection hand stage the operated device,
The operation detection hand stage, the position of the screen specified by the screen identification means, and, movement and position, the shadow of the moving amount of the shadow of the fingertip of the hand of the user identified by the shadow specifying means, shadow Based on the moving direction and the moving speed of the shadow, a pointer moving operation is detected, or a change in the size of the shadow of the user's fingertip specified by the shadow specifying means or a change in the shade of the shadow Based on the coordinates of the pointer is detected,
The operation content output means outputs the pointer to the pointer within the display range of the synchronized operated device based on the movement operation of the pointer detected by the operation detection hand or the click operation .
A connection member for connecting the screen and the user's limb at an interval at which a shadow of a part of the user's limb is projected on the screen;
Output system the connecting member is in the screen surface view of the other side of the screen, at a position where a portion is opposed limb of the user, characterized by connecting the limb of the screen and the user . The screen is plate-like, and a shadow generated by the user blocking the light irradiated by the light source is projected on the other surface of the screen,
The camera shoots a shadow projected on the other side of the screen from one side of the screen at a plurality of timings,
It said operation detecting means, the images picked up by the camera, based on the position of the shadow captured by light transmitted on one surface of the screen, according to claim 6, characterized in that to detect the operation of the user The input / output system described in 1 .

Images