JP2012238293A - Input device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an input device that discriminates between an operator and a background which becomes noise and does not malfunction even if a person other than the operator is present within a photographic range of a camera and moves a hand.SOLUTION: A right eye imaging device A and a left eye imaging device B are disposed side by side, and a distance to a subject of which images have been taken by the imaging devices A and B is measured by a binocular parallax method by distance measurement means C. Subject discrimination means D compares the measured distance with a previously set depth distance that is identified by the binocular parallax method based on an interval between the imaging devices A and B, and when the distance to the subject and the depth distance coincide with each other within a predetermined acceptable range, operation instruction generating means E detects movement of the subject and generates an operation instruction depending on the result of the detection.

Description

  The present invention is used by being connected to an information device such as an information terminal device or a personal computer, takes an operation image of an operator by an imaging device such as a camera, and controls cursor operation of the information device and selection and execution of an application program. It relates to an input device.

  In recent years, various input devices have been proposed in which an operator is photographed with a camera, image analysis is performed, and an audio device, an air conditioner, or the like is operated using the image analysis result.

  For example, a hand region is extracted from an image captured by an imaging device such as a camera, and the shape of the hand (such as tilting a hand or bending a finger) is determined. A corresponding manual operation instruction is notified to the operator by voice or a project image.

  By the way, in this type of input device that captures the operation of the operator by the camera, it is necessary to accurately identify the operator and the background that causes noise. For this reason, techniques that do not capture an excessive background in an image have been proposed.

  For example, an input device that extracts only object reflected light as an image using an image sensor that irradiates infrared light and operates in synchronization with the infrared light is known. Because the intensity of the reflected light is generally inversely proportional to the square of the distance to the object, objects in the background with a distance do not reach the reflected light, so nearby objects can be easily distinguished from the background in the distance. It is intended to reduce malfunctions due to unrelated movements by utilizing the fact that it can be performed (see, for example, Patent Document 1).

  There is also known an input device having a device for imaging the temperature distribution of the imaging range of far-infrared rays using an infrared camera. In this conventional technology, when the hand is held in front of the infrared camera, the far infrared ray corresponding to the temperature of the surface of the hand that is emitted from the hand and enters the infrared camera increases the amount of incident light to the infrared camera as the hand is closer, Increasing the distance reduces the amount of incident light and uses the fact that it is inversely proportional to the square of the distance between the hand and the infrared camera so as not to capture an extra background (see, for example, Patent Document 2).

JP 2001-216069 A Japanese Patent Laid-Open No. 2005-50177

  The above-described prior art requires a special sensor or camera, and when the operator and the background are close to each other, the background is also captured as noise, which causes a malfunction.

  Therefore, the present invention provides an input device that accurately identifies an operator and a background that causes noise and does not malfunction even when a person close to the operator moves within the shooting range of the camera. It is an object.

  In order to achieve the above object, the present invention is an input device that processes an operator's image obtained by an imaging device and generates an operation instruction according to the operation content of the operator. , A left-eye imaging device arranged in parallel with the right-eye imaging device at a predetermined distance from the right-eye imaging device, the right-eye imaging device and the left-eye imaging for a subject A distance measuring unit that measures a distance from the image of the apparatus to the subject by a binocular parallax method, and an index by the binocular parallax method based on the interval between the right eye imaging device and the left eye imaging device The depth distance set in advance and the distance to the subject measured by the distance measuring means are compared, and the subject in which the distance to the subject and the depth distance match within a predetermined allowable range is determined. Select the target to be identified as the operator And body determination means, said detecting movement of the subject object determining means has determined, is characterized by comprising an operation instruction generating means for generating an operation instruction corresponding to the detection result.

  A display unit configured to display a recognition result of the operation instruction, and the operation instruction generation unit detects the movement of the first form by the operator's hand or finger while waiting to display a cursor; The virtual cursor is displayed in the first state, and when the first movement by the hand or finger is stopped or the first movement is detected during the display in the first state, the virtual cursor is displayed. In addition to displaying the cursor in the second state, a cursor display control means for displaying the actual cursor is provided.

  Then, when the hand or finger moves in the first form when the virtual cursor is displayed in the second state, the operation instruction generating means moves the cursor to the position of the hand or finger. Cursor movement control means for generating a first signal for instructing the movement is provided.

  Then, the operation instruction generating means generates a second signal that recognizes a click operation when detecting the movement of the second form by the hand or the finger when the virtual cursor is displayed in the first state. When the movement of the third form following the movement of the second form is detected, click control means for generating a third signal that is recognized as a double click operation is provided.

  Then, the operation instruction generating means instructs to scroll the screen of the display unit when detecting the movement of the fourth form by the hand or the finger when the virtual cursor is displayed in the first state. A screen scroll control means for generating a fourth signal is provided.

  When the operation instruction generating unit detects the movement of the fifth form by the hand or the finger while the real cursor is superimposed on the object displayed on the display unit, the object is a drag target object. And when the movement of the first form by the hand or finger is detected, the fifth signal instructing the movement of the object following the movement by the hand or finger after the recognition is generated. A drag-and-drop control means for generating a sixth signal instructing to drop the object is provided.

  When the operation instruction generating unit detects that the two hands or fingers perform the movement of the first form, the operation instruction generating unit displays a two-point cursor, and the hand or finger performing the movement of the first form. The interval between the two-point cursors is enlarged or reduced according to the interval between the two points being enlarged or reduced, and the display unit is enlarged according to the interval between the two-point cursors when the movement of the first mode is stopped. A screen enlargement / reduction control means for generating a seventh signal instructing display or reduction display is provided.

  Then, the operation instruction generating means displays the two-point cursor when detecting that the two hands or fingers respectively perform the movement of the first form and displays the two-point cursor and the hand performing the movement of the first form. Alternatively, the relative position of the two-point cursor is changed up and down in accordance with the up-and-down change in the relative position of the finger, and depending on the up-and-down positional relationship of the two-point cursor when the movement of the first form is stopped. And a screen rotation control means for generating an eighth signal for instructing a right rotation or a left rotation of the display on the display unit.

  Further, the display unit displays the operator's image by the imaging device in a transparent manner. Further, the display unit is characterized in that the image is transparently displayed when the operator is moving the operation instruction.

  The display unit may display an effect of an operation instruction performed by the operator by the movement of the hand or finger.

  An input device according to the present invention is an input device that processes an operator's image obtained by a video camera and generates an operation instruction according to an operation content of the operator, and includes a right-eye color camera and the right eye For a color image output from the color camera for the left eye that is arranged alongside the color camera for the right eye at a position that is a predetermined distance away from the color camera for the eye, and the color image for the right eye that captures the subject A right eye side image processing program for performing graying processing, image division / binarization processing, inter-frame difference processing, histogram processing, activity rectangle area extraction processing, and extracting the right eye side activity rectangle area of the operator; Gray image processing, image segmentation / binarization processing, inter-frame difference processing, histogram processing, active rectangular area extraction processing for a color image output from the left-eye color camera capturing the subject A left-eye side image processing program for extracting the left-eye side activity rectangular region of the operator, a coordinate axis of the right-eye side activity rectangular region obtained by the right-eye side image processing program, and the left-eye side image processing program The distance to the subject is measured by the binocular parallax method by calculating the center coordinate distance with the coordinate axis of the left-eye side active rectangular area obtained in step, and the measured distance to the subject is the right-eye imaging device. The distance to the subject and the depth distance are within a predetermined allowable range compared with the depth distance that is determined by the binocular parallax method based on the distance between the imaging device for the left eye and the left-eye imaging device. Selected by the activity region selection processing program and the activity region selection processing program for selecting the right-eye activity rectangular region and the left-eye activity rectangular region that are coincident with each other as the activity rectangular region by the operator An image processing program for performing a virtual cursor control process / screen control process on the active rectangle area detected to detect the movement of the hand or fingertip of the subject and to generate an operation instruction according to the detection result; It is characterized by having prepared.

  In the virtual cursor control process / screen control process, when there is one active rectangular area group on the virtual cursor active area image, a Carlsol control instruction or a screen scroll instruction is generated based on the shape and presence / absence of movement. It is characterized by. On the other hand, when there are two active rectangle area groups on the virtual cursor active area image, the virtual cursor control process / screen control process is any one of the screen rotation instruction, the screen enlargement instruction, and the screen reduction instruction based on the moving direction. It is characterized by generating.

  The active rectangular area extracting process is characterized in that a virtual cursor active area image is created from the histogram using a histogram statistical processing result. The active rectangular area extraction process is characterized in that a multi-stage rectangular object extraction process is performed on the virtual cursor active area image to remove noise components.

  Further, an enlargement / reduction rectangle mask creation process is added, and a change area rectangle on the virtual cursor active area image is selected from the color images obtained by the color camera by the enlargement / reduction rectangle mask creation process. It is characterized by extracting an image and cutting other images to remove noise components.

  An input device according to the present invention is an input device that processes an image of an operator obtained by a video camera, generates an operation instruction according to the operation content of the operator, and controls the operation of a remote operation target device. An input device housing formed in a box shape, a color camera body for the right eye attached to the front left side of the input device housing, and a color camera body for the left eye attached to the front right side of the input device housing Arranged in the housing of the input device and separated from the right-eye color camera body by a graying processing circuit, an image division / binarization processing circuit, an inter-frame difference processing circuit, a histogram processing circuit, and an active rectangular area extraction processing circuit. A right eye side image processing board that processes the output color image and extracts the right eye side activity rectangular area of the operator, and is arranged in the input device casing, and is a graying processing circuit, image segmentation / binary value A color image output from the color camera body for the left eye is processed by a processing circuit, an inter-frame difference processing circuit, a histogram processing circuit, and an active rectangular area extraction processing circuit, and the left-eye active rectangular area of the operator is processed. Left eye side image processing board to be extracted and placed in the input device casing, and the right eye side using binocular parallax method by an active rectangular area selection processing circuit, virtual cursor control processing / screen control processing circuit Binocular disparity method by calculating the distance from the coordinate axis of the right eye side activity rectangular area obtained by the image processing program and the coordinate axis of the left eye side activity rectangular area obtained by the left eye side image processing program to the subject The distance to the subject is measured by the binocular parallax method based on the distance between the right-eye imaging device and the left-eye imaging device. The right eye activity rectangular area and the left eye activity rectangular area in which the distance to the subject and the depth distance match within a predetermined tolerance range are compared with the depth distance set in advance by the operator. Select as the activity rectangular area, perform virtual cursor control processing / screen control processing on the selected activity rectangular area, detect the movement of the hand or fingertip of the subject, and generate pointing data according to the detection result And a common processing board for controlling the operation of the remote operation target device.

  According to the present invention, in order to reliably identify the operator and the background by setting the depth of the space in the field of view of the camera for recognizing the operation of the operator, the movement of the hand of a person other than the operator is detected. Will not be entered by mistake. In addition, in order to perform such identification by setting the camera field of view, a highly accurate input device is provided with a normal camera without using a special sensor or camera.

1 shows a block diagram of a first embodiment of an input device according to the invention. The schematic diagram of the positional relationship of an operator, a personal computer, and an imaging device is shown. The block diagram of the specific structural example of the input device by FIG. 1 is shown. The schematic diagram explaining the virtual touch screen screen from a camera visual field and the virtual touch screen screen from the operator visual field with respect to the display part of a personal computer is shown. 7 is a flowchart illustrating a detailed operation example when initially setting a plane model of a virtual touch screen from the operator's field of view. The schematic diagram explaining the parameter used for the calculation which initializes the plane model of the virtual touch screen from an operator visual field is shown. Fig. 4 shows a flowchart of a detailed operation example of the input device shown in Fig. 3. FIG. 8 shows a flowchart of a detailed operation example of grayed / binarized image processing shown in FIG. 7. FIG. 8 is a flowchart of a detailed operation example of the inter-frame difference / histogram creation process shown in FIG. 7. 8 shows a flowchart of a detailed operation example of the activity rectangular area extraction process shown in FIG. 7. FIG. 11 shows a step 41 ′ to be processed instead of the step 41 in the flowchart shown in FIG. 10. Step 43 ′ is shown in place of step 43 in the flowchart shown in FIG. 10. The flowchart of the detailed operation example of the active rectangle area | region selection process shown in FIG. 7 is shown. FIG. 14 shows a step 57 ′ to be processed instead of the step 57 in the flowchart shown in FIG. 13. 8 is a flowchart of a detailed operation example of the virtual cursor control process and the screen control process shown in FIG. FIG. 4 is a schematic diagram of an original histogram example used in the input device shown in FIG. 3 and a histogram example after extraction of a changed region. The schematic diagram of the example of a histogram after the change area extraction used with the input device shown in FIG. 3, and a virtual cursor active area image example is shown. The schematic diagram of the example of a histogram after the change area extraction used with the input device shown in FIG. 3, and a virtual cursor active area image example is shown. In the input device shown in FIG. 3, the schematic diagram of an example of the active rectangular area group selected by the active rectangular area selection process is shown. In the input device shown in FIG. 3, the schematic diagram of an example of the active rectangular area group selected by the active rectangular area selection process is shown. The schematic diagram of an example of the multistage object extraction process used with the input device shown in FIG. 3 is shown. The schematic diagram of an example of the multistage object extraction process used with the input device shown in FIG. 3 is shown. The schematic diagram of the outline | summary of the binocular parallax method used with the input device shown in FIG. 3 is shown. In the input device shown in FIG. 3, the schematic diagram of the relationship between the right active rectangular area and the left active rectangular area before performing correction by the binocular parallax method is shown. In the input device shown in FIG. 3, the schematic diagram of the relationship between the right active rectangular area and the left active rectangular area after correcting by the binocular parallax method is shown. In the input device shown in FIG. 3, the schematic diagram of the example of the right active rectangular area after correction | amendment by a binocular parallax method and a left active rectangular area is shown. The schematic diagram of the example of a relationship between the motion of the operator's hand image | photographed with the input device shown in FIG. 3, and a virtual cursor is shown. The schematic diagram of the example of a relationship between the motion of the operator's hand image | photographed with the input device shown in FIG. 3, and a virtual cursor is shown. FIG. 4 is a schematic diagram illustrating an example of a fine adjustment operation of an actual cursor controlled by the input device illustrated in FIG. 3. The schematic diagram of the example of color control of the virtual cursor controlled by the input device shown in FIG. 3 is shown. The schematic diagram of the example of a histogram after the change area extraction used with the input device shown in FIG. 3, and a virtual cursor active area image example is shown. The schematic diagram of the example of a relationship between the motion of the operator's hand image | photographed with the input device shown in FIG. 3, and click motion is shown. The schematic diagram of an example when there exist two each active rectangle area | region groups on the virtual cursor active area image obtained by the input device shown in FIG. The schematic diagram of the positional relationship example of each active rectangular area group on the virtual cursor active area image obtained by the input device shown in FIG. 3 is shown. The schematic diagram of an example when there exist two each active rectangle area | region groups on the virtual cursor active area image obtained by the input device shown in FIG. The schematic diagram of the positional relationship example of each active rectangular area group on the virtual cursor active area image obtained by the input device shown in FIG. 3 is shown. In the input device shown in FIG. 3, the schematic diagram explaining the operation | movement which removes as a noise what generate | occur | produced by the operation | movement which an operator does not intend in the extraction activity area | region is shown. FIG. 2 is a schematic diagram for explaining the relationship between the camera viewing angle, the subject width, and the distance to the subject in one camera. The schematic diagram explaining the measurement of the exact distance from a camera to a to-be-photographed object by the binocular parallax method by the two cameras of the input device shown in FIG. 2 shows a block diagram of a second embodiment of an input device according to the invention. FIG. 6 shows a flowchart of another embodiment of an input device according to the present invention. FIG. 42 shows a schematic diagram of an operation example of the flowchart shown in FIG. 41. An explanatory view of tapping operation by a fingertip is shown. An explanatory view from feedback to display of a virtual cursor is shown. An explanatory view of a state where a virtual cursor will be in a locked state is shown. FIG. 5 is an explanatory diagram of palm movement for instructing display of a virtual cursor. The explanatory view of the hand movement of click operation is shown. An explanatory view of hand movement of a double click operation is shown. The explanatory view of the movement of the hand of flip operation is shown. An explanatory view of hand movement of a drag and drop operation is shown. FIG. 4 is an explanatory diagram of a hand movement of an operation for instructing enlargement / reduction of a screen. An explanatory view of a hand movement of an operation instructing rotation of a screen is shown.

  FIG. 1 is a block diagram showing a first embodiment of an input device 1a according to the present invention, in which A is a right-eye imaging device using a visible light camera such as a web camera or a video camera for photographing a subject, and B is this right This is a left-eye imaging device using a visible light camera, such as a web camera or a video camera, that is arranged alongside the right-eye imaging device at a position spaced from the eye imaging device A by a predetermined distance. C is a distance measuring unit that measures the distance from the image of the subject captured from the right-eye imaging device A and the left-eye imaging device B to the subject by binocular parallax, and D is the right-eye imaging device A and left-eye imaging device. It is determined whether the subject is an operator by comparing the depth distance based on the binocular parallax method set in advance based on the distance between the imaging device B and the distance to the subject measured by the distance measuring means C. Subject discrimination means. E is an operation instruction generation unit that detects the movement of the hand or fingertip of the subject when the subject determination unit D determines that the subject is an operator, and generates an operation instruction according to the detection result.

  The input device 1a of the present invention includes a right-eye image pickup device A and a right-eye image pickup device A arranged side by side in order to narrow the depth of the space in the field of view from the right-eye image pickup device A and the left-eye image pickup device B. The depth distance from the left-eye imaging device B to the subject determined by binocular parallax is set in advance in the detection area. Then, when the right-eye imaging device A and the left-eye imaging device B capture an actual subject, the distance measuring unit C calculates the distance to the subject by binocular parallax from the images of the subject captured from the imaging devices A and B. taking measurement. Then, the subject discrimination means D compares the distance to the subject measured by the distance measurement means C with a preset depth distance, and if the subject is within an allowable error range, the subject is an operator. The operation instruction generation means E determines the movement of the operator's hand or fingertip, and generates an operation instruction to a personal computer (not shown) according to the detection result.

  As described above, the input device 1a according to the present invention evaluates the distance to the subject obtained by the binocular parallax method to distinguish it from the background image, and the operator moves the hand or fingertip within a certain distance range from the imaging device. It is determined whether or not.

  The positional relationship among the subject (operator), the personal computer, and the imaging devices A and B will be further described with reference to FIG. 2, where (a) in FIG. An operator who is in a distance range, for example, “0.3 m” to “0.8 m”, is distinguished from other persons who are at other distances. Then, as shown by (a) and (c) in FIG. 2, when the distance range d is reduced, the region for detecting the movement of the operator's hand, fingertip, etc. approaches from the space to the plane. When the distance d is set to be small until it substantially matches the depth distance from the interval between the right-eye imaging device A and the left-eye imaging device B to the subject determined by the binocular parallax method, Is formed between the operator and the imaging devices A and B. The virtual touch screen F achieves a function similar to that when the operator inserts a fingertip into a plane in the field of view from the imaging device to detect the coordinates and map the coordinates on the display unit of the personal computer 2.

  Therefore, even if the operator moves his / her hand, fingertip, etc. in a space outside the depth distance and this image is picked up by the image pickup devices A and B, it is not regarded as an input operation to the virtual touch screen F, and subject discrimination is performed. Since it is ignored by the means D, it is not detected as an input operation by the operation determination means E. Since the input device 1a having such a virtual touch screen has all background operations other than those on the virtual touch screen F disabled, the operation as the input device is ensured. According to the binocular parallax method, the distance between the imaging devices A and B arranged in the horizontal direction is about 5 cm and the depth distance is 1 cm.

  FIG. 3 is a block diagram showing a specific configuration example of the input device 1a according to FIG. The input device 1a shown in this figure includes a built-in web camera 4 that is a left-eye imaging device provided in a display unit 3 of a personal computer 2, a video capture 5 provided in the personal computer 2, and a display unit 3 of the personal computer 2. An external web camera 6 that is an imaging device for the right eye provided in the computer, a USB interface 7 provided in the personal computer 2, a hard disk 8 provided in the personal computer 2, a CPU 9 provided in the personal computer 2, and a personal computer. 2 and a memory 10 provided in 2. The input device 1a analyzes the color images obtained by the web cameras 4 and 6 using the binocular parallax method, and based on the distance from the installation position of the web cameras 4 and 6, the operator and the others A virtual cursor 25 (see FIG. 27B) displayed on the display unit 3 of the personal computer 2 is detected by detecting only the movement of the operator's hand, fingertips, etc. while distinguishing it from other people at a distance. The target screen (OS screen, application screen) and the like are controlled, and the currently running application is controlled.

  The web camera 4 is a color camera having a resolution of about 320 pixels × 240 pixels. When a shooting instruction is issued from the video capture 5, the color video signal obtained by shooting the operator is sent to the video capture 5. Supply.

  When a shooting instruction is issued from the CPU 9 via the system bus 12, the video capture 5 controls the web camera 4 to shoot an operator, captures a color video signal obtained by the shooting operation, The signal is converted to a color image and supplied to the CPU 9.

  The web camera 6 is a color camera having a resolution of about 320 pixels × 240 pixels attached to the upper edge of the display unit 3 at a predetermined distance in the horizontal direction from the web camera 4. When the photographing instruction is issued, the YUV signal obtained by photographing the operator is supplied to the USB interface 7.

  The USB interface 7 controls the web camera 6 to take an image of the operator and captures a YUV signal obtained by the shooting operation when a shooting instruction is issued from the CPU 9 via the system bus 12. The data is supplied to the CPU 9 and converted into a color image in the RGB signal format.

  The hard disk 8 includes an OS storage area 13 in which an OS (Operating System) and constant data are stored, an application storage area 14 in which application programs such as an Internet Explorer program and a browser program are stored, and a right eye used in the present invention. A side image processing program, a left eye side image processing program, an image processing program storage area 15 in which the image processing program is stored, and a specific color (for example, skin color) preset in HSV (hue / saturation / lightness) method And an image storage area 16 for storing a color mask, a binarized image, a histogram, a virtual cursor activity area image 27 (see FIG. 17), a virtual button click activity area image, etc. I have. When a read instruction is output from the CPU 9, the CPU 9 captures this via the system bus 12 and stores the OS, constant data, application program, image processing program, binary image, histogram, The virtual cursor activity area image 27, the virtual button click activity area image, and the like are read out and supplied to the CPU 9 via the system bus 12. Further, when a write instruction and data are output from the CPU 9, these are taken in via the system bus 12 and stored in an area designated by the write instruction, for example, the image storage area 16.

  The CPU 9 generates display data specified by the OS, constant data, application program and the like stored in the hard disk 8 and supplies the display data to the display interface 11 connected to the system bus 12, and displays the operation target screen on the display unit 3. Display. In addition, the image processing described in the right eye side image processing program, the left eye side image processing program, the image processing program, etc. is performed, the control of the size and position of the virtual cursor displayed on the operation target screen, click control, Scroll control, screen rotation control, screen enlargement control, screen reduction control, etc.

  The memory 10 has a capacity of about several hundred megabytes to several gigabytes, and temporary data when the CPU 9 performs processing specified by an application program, a right eye side image processing program, a left eye side image processing program, an image processing program, or the like. Used as storage area.

  The input device 1a according to the present invention forms a virtual touch screen between the operator and the web cameras 4 and 6, but the operator operates the display unit 3 of the personal computer 2 to operate the operator. When viewed from the side, a virtual touch screen is formed between the operator and the display unit 3. Therefore, depending on the attachment positions of the web cameras 4 and 6, there is a problem that the positions of both virtual touch screens are greatly different.

  For example, in the case where the web cameras 4 and 6 are shown as part (a) in FIG. 4 with the ceiling facing down from the ceiling, the virtual touch screen screen S1 from the camera field of view and the operator field of view on the display unit 3 are viewed. Since the coordinate axis of the virtual touch screen screen S2 is greatly different, the input device 1a that inputs the images from the web cameras 4 and 6 may not accurately capture the operation performed by the operator toward the display unit 3. is there.

  Therefore, depending on the attachment position of the web cameras 4 and 6, as shown in FIG. 4B, the plane coordinates Pin = of the virtual touch screen screen S1 from the camera field of view obtained by the binocular web cameras 4 and 6 are obtained. It is preferable to convert (x, y, d) to the plane coordinates Pin = (X, Y, D) of the virtual touch screen screen S2 from the operator's field of view. Hereinafter, this conversion method will be described.

  The plane coordinate Pin = (x, y, d) of the virtual touch screen screen S1 from the camera view is converted into the plane coordinate Pin = (X, Y, D) of the virtual touch screen screen S2 from the operator view. The input device 1a needs to perform a calibration operation for initial setting of the planar model of the virtual touch screen screen S2 from the operator's field of view.

<Initial setting of planar model>
The initial setting of the planar model that the operator operates assuming that there is a virtual touch screen in front of the display unit 3 will be described based on the flowchart of FIG. 5 and the schematic diagram of FIG.

First, the operator taps three points P1, P2, and P3 in the order of upper left, upper right, and lower left on the screen of the display unit 3 of the personal computer 2 that can be seen in the field of view. P1 = (x1, y1, d1), P2 = (x2, y2, d2), and P3 = (x3, y3, d3) are measured on the virtual touch screen screen S1 (step S101). Here, x, y, d are as follows.
x: Horizontal axis value on the webcam image y: Vertical axis value on the webcam image d: Distance from the webcam

And CPU9 calculates | requires the normal vector n of virtual touch screen screen S1 plane (step S102). Since the normal vector n of the plane is orthogonal to the plane including the three input points, it is orthogonal to the vector connecting any two of the three points simultaneously. Therefore, the normal vector n is obtained by the following equation. Finally, the norm is normalized to a vector of 1.
n = (P ₁ −P ₂ ) × (P ₁ −P ₃ ) (× indicates outer product calculation)

Next, the CPU 9 obtains a position vector p of the virtual touch screen screen S1 plane (step S103). The plane position vector p may be a vector to any point on the plane, but here it is set to be a vector to the center of the virtual screen. For this purpose, the input point P1 is tentatively determined as a plane position vector, and this position vector and its position vector p _l = (0, 0, 0) with respect to a straight line perpendicular to the virtual touch screen screen S1 plane from the webcams 4 and 6 ) And the direction vector d ₁ = (0, 0, 1).

The intersection point c is obtained by using the straight line position vector p _l and the direction vector d _l , the plane position vector p _p and the normal vector n _p as follows. In the equation, · represents inner product calculation.

  Then, the CPU 9 obtains X and Y axis basis vectors of the virtual touch screen screen S2 (step S104). The basis vector is used when calculating coordinates X, Y, and D when a new input point is projected onto the virtual touch screen screen S2, and the basis vector of D uses a normal vector of the plane. Only the X and Y axes are created.

Basis vector _{n x} of the X-axis, X-axis from the position vector p of the plane _{(p l = (0,0,0),} d l = (1,0,0)) and a vector calculation to that plane intersects do it. Y-axis basis vector n _y is calculated from a plane position vector p to a point where the plane intersects the Y-axis (p ₁ = (0, 0, 0), d ₁ = (0, _1, 0)). do it. That is, each is subtracted by a plane position vector p to obtain a direction vector from p.

Then, add the determined basis vectors _n x, a _{n y} in the parameter plane. Finally _{n x} and _{n y} are normalized to norm 1.

As described above, when the operator inserts a fingertip into the virtual touch screen to give an instruction to the personal computer 2, the point Pin = (x, y, d) on the virtual touch screen screen S1 detected by the web cameras 4 and 6 is , the normal vector n of the plane parameters, the position vector p, basis vectors _{n x,} using a base vector _{n y} CPU 9 is by performing the following calculation, point Pin = (X on the virtual touch screen screen S2, Y , D).
X = n _x · (P _in −p)
Y = _ny · (P _in −p)
D = n · (P _in −p)
(・ Represents inner product calculation)

  By performing the correction as described above, the virtual touch screen from the camera field of view and the virtual touch screen from the operator field of view can be matched, and an input device with more flexible installation locations of the webcams 4 and 6 is provided. Is done. It should be noted that a rectangular area effective for tapping with coordinates on the virtual touch screen from the operator's field of view is held in advance, and the X axis is converted to 0 to 320, the Y axis is converted to 0 to 240, etc. based on the area values. The subsequent processing after subject determination is set to a value that is easy to handle.

  Next, an image processing operation, a cursor control operation, a screen control operation, and the like will be described for an input operation to the input device 1a.

<< Binary image generation and storage >>
First, when the personal computer 2 is turned on and an application program, a right eye side image processing program, a left eye side image processing program, and an image processing program are started, the video capture is performed by the CPU 9 as shown in the flowchart of FIG. 5 is controlled, the color video signal obtained by the photographing operation of the web camera 4 is taken in, and is converted into a color image in the RGB signal format and temporarily stored in the memory 10 or the like (step S1).

  In parallel with this operation, the USB interface 7 is controlled by the CPU 9 so that the YUV signal obtained by the photographing operation of the web camera 6 is taken in and converted into a color image in the RGB signal format and temporarily stored in the memory 10 or the like. Stored (step S2).

  In parallel with these operations, among the color images (color images obtained by the shooting operations of the web cameras 4 and 6) temporarily stored in the memory 10 or the like by the CPU 9, the operator from the personal computer 2 side. The color image obtained by the web camera corresponding to the right eye, for example, the external web camera 6 is read (step S3).

  Thereafter, the CPU 9 starts graying / binarized image processing (step S4). That is, as shown in the flowchart of FIG. 8, the color mask obtained by the external web camera 6 is masked by the color mask stored in the image storage area 16 of the hard disk 8 and set in advance from the color image. A color image (skin color image) of a specific color (for example, skin color) is extracted (step S21), and the color image obtained by the external web camera 6 is gray-processed and set in advance. The image is converted into a monochrome image of a certain gradation, and the image capacity for one frame is reduced (step S22).

  Then, the CPU 9 checks whether or not a screen division instruction is set. If there is a screen division instruction, the monochrome image is divided into a plurality of areas (each area is composed of several to several tens of pixels). If there is no screen division instruction, the division process is skipped, and then the monochrome image is binarized by the maximum likelihood threshold method to create a binarized image (step S23).

  Next, the logical sum of the binarized image and the skin color image is taken by the CPU 9 and the skin color portion in the binarized image is extracted (step S24), and this is the binarized image for one frame (on the right eye side). (Binary image) is stored in the image storage area 16 of the hard disk 8 (step S25).

  Thereafter, among the color images (color images obtained by the photographing operations of the web cameras 4 and 6) temporarily stored in the memory 10 or the like by the CPU 9 as shown in the flowchart of FIG. When the operator is viewed, the color image obtained by the web camera corresponding to the left eye, for example, the built-in web camera 4 is read (step S5).

  Next, the CPU 9 starts graying / binarized image processing (step S6). That is, as shown in the flowchart of FIG. 8, the color image obtained by the built-in web camera 4 is masked by the color mask stored in the image storage area 16 of the hard disk 8 and preset from the color image. A color image (skin color image) of a specific color (for example, skin color) is extracted (step S21), and the color image obtained by the built-in web camera 4 is subjected to gray processing, and is set in advance. The image is converted into a tone monochrome image, and the image capacity for one frame is reduced (step S22).

  Then, the CPU 9 checks whether or not a screen division instruction is set. If there is a screen division instruction, the monochrome image is divided into a plurality of areas (each area is composed of several to several tens of pixels). If there is no screen division instruction, the division process is skipped, and then the monochrome image is binarized by the maximum likelihood threshold method to create a binarized image (step S23).

  Next, the logical sum of the binarized image and the skin color image is obtained by the CPU 9 and the skin color portion in the binarized image is extracted (step S24), and this is converted into a binarized image for one frame (on the left eye side). (Binary image) is stored in the image storage area 16 of the hard disk 8 (step S25).

  Thereafter, the above-described image processing is repeated, and the right eye side binarized image and the left eye side binarized image are in the image storage area 16 of the hard disk 8 in FIFO (First / In / First / Out) format. A frame to several tens of frames are accumulated.

《Difference between frames, creation of histogram》
In parallel with this operation, as shown in the flowchart of FIG. 7, the right eye side is selected from the binarized images for several frames to several tens of frames stored in the image storage area 16 of the hard disk 13 by the CPU 9. Several consecutive frames of binarized images including the latest binarized image corresponding to are sequentially read (step S7).

  Then, the number of frames of the binarized image that can be read out by the CPU 9 is checked. If the number of frames is equal to or greater than the predetermined number (step S8), the interframe difference / histogram creation process is started (step S9). That is, as shown in the flowchart of FIG. 9, inter-frame difference processing is performed on the binarized images for two consecutive frames among the respective binarized images (steps S31 and S32), and the inter-frame difference is performed. Each difference image obtained by the process is cumulatively added for each divided area to create a right-eye histogram and stored in the image storage area 16 of the hard disk 8 (steps S33 and S34).

  Next, as shown in the flowchart of FIG. 7, the latest 2 corresponding to the left eye side from among the binary images for several frames to several tens of frames stored in the image storage area 16 of the hard disk 13 by the CPU 9. Several consecutive frames of binarized images including the binarized image are sequentially read out (step S10).

  Then, the number of frames of the binarized image that can be read out by the CPU 9 is checked. If the number of frames is equal to or greater than the predetermined number (step S11), the inter-frame difference / histogram creation process is started (step S12). That is, as shown in the flowchart of FIG. 9, inter-frame difference processing is performed on the binarized images for two consecutive frames among the respective binarized images (steps S31 and S32), and the inter-frame difference is performed. Each difference image obtained by the process is cumulatively added for each divided area to create a left eye side histogram and stored in the image storage area 16 of the hard disk 8 (steps S33 and S34).

《Statistical processing, virtual cursor activity area determination, change area extraction》
Thereafter, as shown in the flowchart of FIG. 7, the right-eye histogram stored in the image storage area 16 of the hard disk 8 is read by the CPU 9 (step S13), and the active rectangular area extraction process is started. (Step S14). That is, as shown in the flowchart of FIG. 10, statistical processing is performed on the density value of each divided area of the histogram, and an average value, density variance value, maximum value, deviation (± 1σ, ± 2σ), and the like are calculated ( Step S41).

  Next, the CPU 9 extracts from the divided areas of the histogram each divided area having a density value larger than the threshold value for extracting the change area rectangle (for example, the average value −1σ). A rectangular change area rectangle (see FIGS. 17 and 18) is determined so as to include (activity division area), and is stored in the image storage area 16 of the hard disk 8.

  In parallel with this operation, the CPU 9 extracts the virtual cursor rectangle out of each divided area 20 (see FIGS. 17 and 18) constituting the histogram as shown in the three-dimensional density distribution diagram of FIG. A divided area (activity divided area 21) having a density value larger than a threshold value (for example, maximum value −1σ) is extracted.

  As a result, when the operator turns the fingertip greatly, the rectangular activity rectangular area 26 is determined so as to include each activity division area 21 as shown in FIG. Such a virtual cursor activity area screen 27 on the right eye side is created and stored in the image storage area 16 of the hard disk 8.

  Further, when the operator moves both hands, a rectangular activity rectangular area 26 is determined so as to include each activity division area 21 as shown in FIG. 18, and a virtual cursor on the right eye side as shown in FIG. An active area image 27 is created and stored in the image storage area 16 of the hard disk 8 (step S42).

《Determine activity area for virtual button click》
Thereafter, the CPU 9 reads the right-eye histogram stored in the image storage area 16 of the hard disk 8 and extracts a threshold value (for example, the maximum value − 2)) divided areas (activity divided areas) having a density value greater than 2σ) are extracted, and a rectangular activity rectangular area is determined so as to include each of these activity divided areas, and a virtual button on the right eye side is determined. A click activity area image (not shown) is created and stored in the image storage area 16 of the hard disk 8 (step S43).

《Multi-stage rectangular object extraction processing, shadow effect removal》
Subsequently, the CPU 9 uses the virtual cursor rectangle extraction threshold value (for example, the maximum value-1σ) (for example, the virtual cursor active region image 27 on the right eye side obtained by using the virtual cursor rectangle extraction threshold value). For example, with respect to the virtual button click activity region image on the right eye side obtained using the maximum value −2σ), it is checked whether each of the activity rectangular regions 26 can be divided into left and right. As shown in FIG. 21, the horizontal center point “A” of the active rectangular area 26 is obtained, and an inactive area on the left side of the horizontal center point “A”, an active area (for example, an active division area 21), Boundary point “B”, a non-active area on the right side of the horizontal center point “A”, and a boundary point “C” between the active area (for example, the active division area 21) are detected. B ”,“ C ” Are determined as the respective activity rectangular areas 26, and the other active areas are determined as unnecessary active areas due to the shadow of the operator and are invalidated (two-point extraction process).

  Thereafter, the CPU 9 checks whether or not each of the activity rectangular areas 26 for which the two-point extraction processing has been completed can be divided vertically, and if it can be divided vertically, as shown in FIG. The vertical center point “A” of the area 26 is obtained, and the boundary point “B” between the inactive area above the vertical center point “A” and the active area (for example, the active division area 21) is detected. Then, the area including the boundary point “B” is determined as the active rectangular area 26, and the lower active area is determined as an unnecessary active area due to the shadow of the operator or the like, and is invalidated (minimization process) (step) S44).

  Subsequently, the CPU 9 performs a virtual cursor activity region image 27 on the right eye side including the activity rectangular region 26 obtained by the multi-stage rectangular object extraction process constituted by the two-point extraction process and the minimization process, and the right-eye virtual image. The button click activity area image is stored in the image storage area 16 of the hard disk 8 (step S45).

《Statistical processing, virtual cursor activity area determination, change area extraction》
After that, as shown in the flowchart of FIG. 7, the CPU 9 reads the left eye side histogram stored in the image storage area 16 of the hard disk 8 (step S15), and starts the active rectangular area extraction process. (Step S16). That is, as shown in the flowchart of FIG. 10, statistical processing is performed on the density value of each divided area of the histogram, and an average value, density variance value, maximum value, deviation (± 1σ, ± 2σ), and the like are calculated ( Step S41).

  Next, the CPU 9 extracts each divided area 20 having a density value larger than the threshold for extracting the change area rectangle (for example, average value −1σ), and includes each of these divided areas (activity divided areas). Thus, a rectangular change area rectangle (see FIGS. 17 and 18) is determined and stored in the image storage area 16 of the hard disk 8.

  In parallel with this operation, the CPU 9 uses the CPU 9 to extract a virtual cursor rectangle from the divided areas 20 (see FIGS. 17 and 18) constituting the histogram as shown in the three-dimensional density distribution diagram of FIG. A divided area (activity divided area 21) having a density value larger than a threshold value (for example, maximum value −1σ) is extracted.

  As a result, when the operator turns the fingertip greatly, the rectangular activity rectangular area 26 is determined so as to include each activity division area 21 as shown in FIG. Such a left-eye virtual cursor activity area screen 27 is created and stored in the image storage area 16 of the hard disk 8.

  Further, when the operator moves both hands, a rectangular activity rectangular area 26 is determined so as to include each activity division area 21 as shown in FIG. 18, and the left-eye virtual cursor as shown in FIG. An active area image 27 is created and stored in the image storage area 16 of the hard disk 8 (step S42).

《Determine activity area for virtual button click》
Next, the left-eye histogram stored in the image storage area 16 of the hard disk 8 is read out by the CPU 9, and a threshold value for extracting a virtual button click rectangle in each divided area 20 (for example, the maximum value −2σ) ) A divided area (activity divided area) having a larger density value is extracted, and a rectangular activity rectangular area is determined so as to include each of these activity divided areas, and a virtual button click on the left eye side is clicked An active area image (not shown) is created and stored in the image storage area 16 of the hard disk 8 (step S43).

《Multi-stage rectangular object extraction processing, shadow effect removal》
Next, the virtual cursor click area extraction threshold value 27 for the left-eye virtual cursor active region image 27 obtained by the CPU 9 using the threshold value for virtual cursor rectangle extraction (for example, the maximum value −1σ) ( For example, with respect to the virtual button click activity region image on the left eye side obtained using the maximum value −2σ), it is checked whether each of the activity rectangular regions 26 can be divided into left and right. As shown in FIG. 21, the horizontal center point “A” of the active rectangular area 26 is obtained, and an inactive area on the left side of the horizontal center point “A”, an active area (for example, an active division area 21), Boundary point “B”, a non-active area on the right side of the horizontal center point “A”, and a boundary point “C” between the active area (for example, the active division area 21) are detected. B ”,“ C ” Are determined as the respective activity rectangular areas 26, and the other active areas are determined as unnecessary active areas due to the shadow of the operator and are invalidated (two-point extraction process).

  Thereafter, the CPU 9 checks whether or not the activity rectangular area 26 can be divided vertically with respect to each of the activity rectangular areas 26 for which the two-point extraction processing has been completed. 26, the center point “A” in the vertical direction is obtained, and the boundary point “B” between the inactive region and the active region (for example, the active division area 21) above the vertical center point “A” is detected. The area including the boundary point “B” is determined to be the active rectangular area 26, and the lower active area is determined to be an unnecessary active area due to the shadow of the operator and is invalidated (minimization process) (step S44).

  Next, the left eye-side virtual cursor activity region image 27 including the activity rectangle region 26 obtained by the multi-step rectangular object extraction process constituted by the two-point extraction process and the minimization process by the CPU 9, and the left-eye virtual button The click activity area image is stored in the image storage area 16 of the hard disk 8 (step S45).

  As described above, the movement of the operator's hand can be captured by the virtual area activity area image 27 and the virtual button click activity area image of the right eye and the left eye. It would be even better if motion could be detected. The activity rectangular area 26 obtained as a result of extracting the activity area in the form of a histogram in multiple stages is an action area on the histogram-formed data even when the movement is not pointing or tapping intended by the operator, for example, when the hand is shaken from side to side. Extracted. Therefore, if it can be determined that this is not due to the pointing operation intended by the operator, the reliability of the input device 1a increases.

  In order to determine whether or not the extracted activity rectangular area is valid, the following processing is performed. That is, in the process of step S41 in FIG. 10, the process of step S41 ′ shown in FIG. 11 is performed, and the CPU 9 extracts the active rectangular area 26 from the latest difference data among the multi-stage difference image data that is made into a histogram. And hold for a certain time while tracking those points.

  Next, in the process of step S43 in FIG. 10, the process of step S43 ′ shown in FIG. 12 is performed, and the CPU 9 stores the virtual button click activity area (maximum value −2σ) in step S41 ′. When the corresponding tracking is obtained and the tracking result is active over an area of a specific size, it is determined that the activity rectangular area 26 is invalid.

  FIG. 37 shows an activity region extracted when the operator moves his / her hand from the lower right to the upper left, further to the lower left, and then to the upper right. The activity region generated from the tracking data of the activity rectangular region 26 is large. Judge what was generated in motion and ignore it as noise.

  In the above processing by the CPU 9, the extraction activity region generated by the operation not intended by the operator can be removed as noise, and the operability is improved.

《Activity rectangular area selection》
Thereafter, as shown in the flowchart of FIG. 7, the active rectangle area selection process is started by the CPU 9 (step S17). That is, as shown in the flowchart of FIG. 13, the virtual cursor activity area image 27 on the right eye side and the virtual button click activity area image on the right eye side stored in the image storage area 16 of the hard disk 8 are read out. As shown in the schematic diagram of FIG. 23, position correction by the binocular parallax method is performed for each active rectangular area 26 included in the virtual cursor activity area image 27 on the eye side and the virtual button click activity area image on the right eye side. The center coordinates of each active rectangular area 26 so as to correspond to the attachment position (horizontal distance “B”, vertical distance, etc.) of each webcam 4, 6, the focal length “f” of each webcam 4, 6, etc. After “P _R (X _R , Y _R )” is corrected, numbers are added in order of magnitude (steps S51 and S52).

Next, the CPU 9 reads the left eye side virtual cursor activity region image 27 and the left eye side virtual button click activity region image stored in the image storage area 16 of the hard disk 8, and these left eye side virtual cursor activity regions are read out. As shown in the schematic diagram of FIG. 23 based on the coordinates P (X, Y, D) of the position of the operator's hand with respect to each active rectangular area 26 of the area image 27 and the virtual button click activity area image on the left eye side. Position correction by the binocular parallax method is performed so as to correspond to the attachment position (horizontal distance “B”, vertical distance, etc.) of each webcam 4, 6, the focal length “f”, etc. of each webcam 4, 6. After the coordinates “P _L (X _L , Y _L )” of each active rectangular area 26 are corrected, numbers are added in order of size (steps S53 and S54).

  By performing position correction by the binocular parallax method in this way, the operator's hand is at a focus position corresponding to the focal length of each web camera 4, 6, for example, a position away from each web camera 4, 6, Before performing position correction by the binocular parallax method, as shown in the schematic diagram of FIG. 24, even if the positions of the activity rectangle areas 26 on the right eye side and the activity rectangle areas 26 on the left eye side are misaligned, By performing position correction by the parallax method, as shown in the schematic diagram of FIG. 25, each activity rectangle area 26 on the right eye side and each activity rectangle area 26 on the left eye side are completely matched (or almost matched). Can do.

Thereafter, as shown in the schematic diagram of FIG. 26 by the CPU 9, the center coordinates “X _R , Y _R ” of the active rectangular area 26 corresponding to the right eye side to which the number “1” is assigned correspond to the left eye side. The distance (center coordinate distance) with the center coordinates “X _L , Y _L ” of the active rectangular area 26 is calculated and stored in the memory 10 together with the number “1”.

Although it has been described that the position correction by the binocular parallax method is performed based on the coordinates P (X, Y, D) of the position of the operator's hand, P (X, Y, D) is a virtual touch from the operator. The coordinates on the screen screen S2 are as follows. As described with reference to FIG. 4, the CPU 9 at this time uses the plane vector direction vector d, position vector n, base vector n _x , by performing the following calculation using the base vector n _y, coordinates P on the virtual touch panel screen S1 from webcam 4,6 field (x, y, d) is converted from. If correction by such calculation is performed, even when the installation location of the web cameras 4 and 6 is far from the display unit 3 of the personal computer 2, the position of the display unit 3 in the field of view of the operator can be detected.
X = n _x · (P _in −p)
Y = _ny · (P _in −p)
D = n · (P _in −p)

Next, in the processing of steps S55 and S56, the central coordinates “X _R , Y _R ” of the active rectangular region 26 corresponding to the right eye side to which the next number “2” to the last number “N” are assigned by the CPU 9 are displayed. And the distance (center coordinate distance) from the center coordinates “X _L , Y _L ” of the activity rectangular area 26 corresponding to the left eye side are sequentially calculated, and these are calculated from the next number “2” to the last number “N”. Is stored in the memory 10 together.

The processing in step 55 is an operation as the distance measuring means C described in FIG. That is, if the distance between the web cameras 4 and 6 and the focal length f are set in advance, the distance to the subject is correlated with | X _L −X _R |, so the calculation of the center coordinate distance measures the distance to the subject. Will be.

When these processes are completed, the CPU 9 sequentially reads out the respective central coordinate distances (| X _L −X _R |) stored in the memory 10 and indicates a predetermined depth distance. Compare with the value. This depth distance is a depth distance based on the binocular parallax method set in advance based on the interval between the web cameras 4 and 6. As a result of the comparison between the center coordinate distance and the depth distance, if the values match within the allowable range, the activity rectangle area 26 on the right eye side and the activity rectangle area 26 on the left eye side at this center coordinate distance are effective activities. It is determined as a rectangular area.

  When the operator's hand is at a position within an allowable range from a preset depth distance from each of the web cameras 4 and 6, the right-eye activity rectangular area 26 corresponding to the operator's hand, the left-eye activity Each rectangular area 26 is determined to be a valid active rectangular area 26. This means that the virtual touch screen is formed at a preset depth distance, and the other is not an operation on the virtual touch screen. It is determined that each of the activity rectangle areas 26 is an invalid activity rectangle area 26 (step S57). At this time, by setting a narrow allowable range when comparing the center coordinate distance and the depth distance, a flatter virtual touch screen is formed, and the background movement other than the operation by the operator is captured. As a result, the operation as an input device is ensured.

  The processing in step 57 is an operation as the subject discrimination means D described in FIG. That is, the distance to the subject measured in the process of step 55 which is the distance measuring means C is compared with the preset depth distance, and if it does not match within the allowable range, the active rectangular area is invalidated. Thus, only the active rectangular area corresponding to the hand on the virtual touch screen by the operator is selected.

  Thus, using the binocular parallax method, while correcting the position of the active rectangular area, the coordinate distance in the horizontal direction (center coordinate distance) between the center points of the active rectangular areas 26 of the right eye and the left eye to the subject. Therefore, the effectiveness of the active area 26 can be determined with high accuracy. However, an error occurs in the center portion and the end portion of the subject in the field of view of the web cameras 4 and 6, and correction of this will be described.

  The distance is measured by calculating a correction value from the center coordinate distance in the active rectangular area 26 of each of the fields of view of the webcams 4 and 6 with the camera viewing angle measured in advance as a constant. Will be described.

  FIG. 38 explains the conversion of the distance from the viewing angle of one web camera 4A. From the definition of the trigonometric function, the relationship of the formula [Equation 2] is established.

ｗ：画像上での被写体の幅[pixel]
ｉｍｇ＿ｗ：カメラ画像の横画素数[pixel]
ｄ：カメラと被写体までの距離[ｍ]
δ：被写体の幅／２[ｍ]
θｗ：カメラの視野角／２[rad]（予め計測）
但し、ｗと（ｉｍｇ＿ｗ）は、それぞれ画像中の大きさのため物理量としては「角度」となる。

w: Width of the subject on the image [pixel]
img_w: Number of horizontal pixels of the camera image [pixel]
d: Distance between camera and subject [m]
δ: Subject width / 2 [m]
θw: Camera viewing angle / 2 [rad] (measured in advance)
However, w and (img_w) are “angles” as physical quantities because of their sizes in the image.

  Solving this with respect to d yields the formula [Equation 3].

  In Equation (3), θw and img_w are constants, so the distance d, which is a value to be obtained, depends on δ and w. Here, since δ varies depending on the subject 60 to be reflected, it is necessary to obtain the distance d without depending on δ.

  Therefore, in the present invention using two web cameras 4 and 6, the distance d can be obtained without depending on δ. FIG. 39 explains distance conversion from the viewing angles of the two main and sub webcams. The distance d is expressed by the equation [Equation 4].

ｗ’：サブカメラの画像上での被写体の中心の横座標[pixel]と、メインカメラの画像上での被写体の中心横座標[pixel]の差の絶対値 （｜Ｘ_Ｌ−Ｘ_Ｒ｜）
δ’：被写体の位置で、サブカメラの中心視線とメインカメラの中心視線が交差する２点間の距離[ｍ]（両眼の視線が平行ならば、δ’は常に一定）

w ′: absolute value of the difference between the abscissa [pixel] of the center of the subject on the sub-camera image and the center abscissa [pixel] of the subject on the image of the main camera (| X _L −X _R |)
δ ′: Distance [m] between two points at which the center line of sight of the sub camera and the center line of sight of the main camera intersect at the position of the subject (when both eyes are parallel, δ ′ is always constant)

  In the formula of [Equation 4], the distance d does not depend on δ ′ because it is constant when the central line of sight of both eyes is parallel. Note that d is sufficiently larger than δ ′ (d >> δ ′), and even if the subject 60 exists at the edge of the image, it is approximately established.

  From the above, if two cameras are used, the distance d depends only on w ', as represented by the equation of [Equation 5]. Therefore, since d is calculated from the pixel value and is a discrete value, d (w ′) of each w ′ is calculated in advance and stored in a table, thereby correcting the distance (d) to the subject 60 at high speed. Can be processed.

If the above principle is incorporated in the determination of the effectiveness of the activity area 26 by the CPU 9, the accuracy of the determination can be further improved. In this case, the CPU 9 performs the process of step 57 ′ shown in FIG. 14 in the process of step S57, calculates the center coordinate distance w ′ (| X _L −X _R |), and stores the table data in the memory 10. Is checked to determine the validity / invalidity of the active rectangular area 26 by measuring the exact distance to the subject 60 and determining whether this distance is within the preset allowable depth distance range. Is what you do.

As a result, if each activity rectangular area 26 on the right eye side and each activity rectangle area 26 on the left eye side have the relationship shown in the schematic diagram of FIG. 26, the number “the center coordinate distance is equal to or greater than a predetermined value”. The number “0” is determined to be invalid for the right-eye side active rectangle area (O _R1 ) 26 and the left-eye side active rectangle area (O _L1 ) 26 corresponding to “1”. It is determined that the right-eye active rectangular area (O _R2 ) 26 and the left-eye active rectangular area (O _L2 ) 26 corresponding to 2 ″ are valid.

Next, the right-side active rectangular area (O _R2 ) 26 determined to be valid by the CPU 9 and the left-side active rectangular area (O _L2 ) 26 determined to be valid are previously designated, For example, a virtual cursor activity region image 27 and a virtual button click activity region image in which the activity rectangle region ( _OL2 ) on the left eye side determined to be valid is left and the other activity rectangle regions 26 are deleted are created, This is stored in the image storage area 16 of the hard disk 8 as a virtual cursor activity area image 27 from which the movement of the person in front of the operator and the movement of the person behind are removed by the binocular parallax method, and the virtual button click activity area image. (Step S57).

《Position, size, and color control of virtual cursor by one hand gesture》
Thereafter, as shown in the flowchart of FIG. 7, the CPU 9 starts the virtual cursor control process / screen control process (step S18). This process is the operation as the operation instruction generating means E described in FIG. 1, and detects the movement of the hand or fingertip determined to be the operation of the operator in step S17, and issues an operation instruction according to the detection result. Generate.

  As shown in the flowchart of FIG. 15, each virtual cursor activity area image stored in the image storage area 16 of the hard disk 8 (the movement of the person in front of the operator and the background of the person behind the operator by the binocular parallax method). Among the virtual cursor activity region images 27) from which movement is removed, several frames of virtual cursor activity region images 27 including the latest activity rectangle region 26 are read (step S61), and one or more adjacent activities are read out. It is checked whether an active rectangular area group constituted by the rectangular areas 26 exists in the virtual cursor active area image 27.

  If there are active rectangular area groups in the latest virtual cursor active area image 27, and the number thereof is “1” and approximately rectangular (steps S62 and S63), the CPU 9 causes the active rectangular area groups to be displayed. The size and moving direction are determined, and virtual cursor control is performed so as to correspond to the determination result (step S64).

  For example, as shown in FIG. 27 (a), the operator turns the fingertips largely at the same height and left and right positions as in the previous time, and correspondingly, a large active rectangular area group obtained in the previous process When a large active rectangular area group is obtained at the same position as the position, the CPU 9 determines that it is a virtual cursor display instruction, and the display unit 3 has a large size and white color as shown in FIG. A virtual cursor 25 is displayed.

  When the operator moves the fingertips up and down or left and right, and correspondingly, a large group of active rectangular areas moving from the position obtained in the previous process is obtained, the CPU 9 The virtual cursor movement instruction is determined, and the large size, white virtual cursor 25 displayed on the display unit 3 is moved so as to correspond to the moving direction of the fingertip.

  Further, as shown in FIG. 28 (a), the operator turns the fingertip small at substantially the same height and left and right positions as in the previous time, and correspondingly, the large active rectangular area group obtained in the previous process. When a small group of active rectangular areas is obtained at the same position as the position, the CPU 9 determines that the movement of the virtual cursor is stopped, and the virtual displayed on the display unit 3 as shown in FIG. The movement of the cursor 25 is stopped and the size is reduced.

  If a certain time has passed in this state, the CPU 9 changes the color of the virtual cursor 25 to red and prohibits a large movement, and a cursor movement instruction is issued to the OS side, and the real cursor 28 is placed in the virtual cursor 25. Move.

  Thereafter, when the operator moves the fingertip slightly, this is detected by the CPU 9, and the position of the virtual cursor 25 displayed on the display unit 3 is finely adjusted, and a cursor position adjustment instruction is issued to the OS side. Then, the position of the real cursor 28 is finely adjusted as shown in FIG.

  Next, if the operator stops moving the fingertip, this is detected by the CPU 9, and after a certain time, the position of the virtual cursor 25 displayed on the display unit 3 is fixed as shown in FIG. The color of 25 is changed from red to gray to inform the operator that it can be clicked.

  Even in this state, if the operator turns the fingertip again to a large extent, this is detected by the CPU 9, the color of the virtual cursor 25 displayed on the display unit 3 is returned to white, and the virtual cursor 25 is returned to a movable state. It is.

《Scroll control with one hand gesture》
When it is checked whether or not there is an active rectangular area group in the virtual cursor active area image 27 described above, if the number of active rectangular area groups is “1” as shown in FIG. In S62, S63), the CPU 9 determines which direction is longer than the previous active rectangular area group, and generates a right scroll instruction (or left scroll instruction) corresponding to the longer direction to the application side. The application screen (operation target screen) displayed on the display unit 3 is scrolled rightward (or leftward) (step S64).

  If it is checked whether or not there is an active rectangular area group in the virtual cursor active area image 27 described above, if the number of active rectangular area groups is “1” and is long in the vertical direction (steps S62 and S63), the CPU 9 Is used to determine which direction is longer than the previous active rectangular area group, and an upward scroll instruction (or downward scroll instruction) corresponding to the longer direction is generated and passed to the application side, and the display unit 3 The application screen (operation target screen) displayed on the screen is scrolled upward (or downward) (step S64).

《Real cursor click control by one hand gesture》
Thereafter, the CPU 9 checks whether the color of the virtual cursor 25 is gray. If the color of the virtual cursor 25 is gray, the latest virtual button click activity area image stored in the image storage area 16 of the hard disk 8 is checked. The virtual button click activity region images for several frames including the activity rectangle region are read out (step S66).

  Next, the CPU 9 checks whether there is an active rectangular area group composed of one or more active rectangular areas 26 in the virtual button click active area image, and whether the shape has changed. Is a change in which the active rectangular area group is set in advance, for example, as shown in FIG. 32A, the operator spreads his hand once from the state of pointing, and the active rectangular area group is only once. If it has changed from “small” to “large” (step S67), it is determined that it is a single click, a single click instruction is issued to the OS side, and the virtual cursor 25 is entered as shown in FIG. An icon or the like is single-clicked with a certain real cursor 28 (step S68).

  In addition, if the operator spreads his / her hand more than twice from the pointing state and the activity rectangular area group changes from “large” to “small” and “small” to “large” a plurality of times (step S67). ), It is determined by the CPU 9 that it is a double click, a double click instruction is issued to the OS side, and the icon or the like at the position of the real cursor 28 is double clicked (step S68).

《Screen enlargement and reduction control using two-hand gestures》
In addition, the operator puts out his right hand and left hand at the focus position of each of the webcams 4 and 6, moves these right hand and left hand, and checks whether or not there is an active rectangular area group in the virtual cursor active area image 27 described above. When the number of activity rectangular area groups becomes “2” and each is rectangular (step S63), this is detected by the CPU 9 and is displayed on the display unit 3 according to the movement of each of these activity rectangular area groups. The displayed operation target screen is enlarged, reduced or rotated (step S65).

  For example, the operator puts out his right hand and left hand at the focus positions of the web cameras 4 and 6 and moves the right hand and left hand away from each other. As shown in FIG. 34 (a), when the distance between each of the activity rectangular areas becomes longer than the previous time when moving in the direction wider than the previous time, the CPU 9 determines that the screen enlargement instruction has been input, and the activity A screen enlargement instruction with an enlargement ratio corresponding to the distance change ratio of the rectangular area group is generated and passed to the application side, and the application screen (operation target screen) displayed on the display unit 3 is enlarged.

  Further, the operator puts out his right hand and left hand at the focus positions of the web cameras 4 and 6 and moves the right hand and left hand in a direction approaching each other. Correspondingly, two active rectangular area groups are shown in FIG. ), When the distance between each of the active rectangular area groups is shorter than the previous distance, it is determined that the screen reduction instruction is input by the CPU 9, and the distance change ratio of the active rectangular area groups is determined. A screen reduction instruction with a reduction ratio corresponding to the above is generated and passed to the application side, and the application screen (operation target screen) displayed on the display unit 3 is reduced.

《Screen rotation control using two-hand gestures》
Also, the operator puts his right hand and left hand at the focus positions of the webcams 4 and 6, and moves one of the right hand and left hand up and the other down, corresponding to the two activities shown in FIG. When at least one of the rectangular area groups moves upward (or downward), the CPU 9 determines that a screen rotation instruction has been input and determines the angle of the upper active rectangular area group relative to the lower active rectangular area group. A screen rotation instruction with a corresponding rotation angle is generated and passed to the application side, and the application screen (operation target screen) displayed on the display unit 3 is rotated.

  At this time, as shown in FIG. 36 (a), in the state where the left and right distances of the respective active rectangular area groups are narrow, one of the active rectangular area groups moves upward and the angle of the upper active rectangular area group with respect to the lower active rectangular area group is large. At this time, a screen rotation instruction with a large rotation angle is generated by the CPU 9 and passed to the application side, and the application screen (operation target screen) displayed on the display unit 3 is largely rotated.

  In addition, as shown in FIG. 36B, when the left and right distances of the respective active rectangular area groups are wide, when one of the active rectangular area groups is small and moves upward, the angle of the upper active rectangular area group with respect to the lower active rectangular area group is small. The CPU 9 generates a screen rotation instruction with a small rotation angle and passes it to the application side, and the application screen (operation target screen) displayed on the display unit 3 is rotated small.

  As described above, in the first embodiment of the present invention, a graying process and an image division / binarization process are performed on a low-resolution color image obtained by photographing the operator with the web cameras 4 and 6. , Color filtering processing, frame buffer processing, inter-frame difference processing, histogram processing, active rectangular area extraction processing, active rectangular area selection processing, virtual cursor control processing / screen control processing to detect the movement of the operator's hand, virtual Since the size control, position control, color control, click control, enlargement control, reduction control, rotation control, up / down scroll control, left / right scroll control, etc. of the operation target screen are performed, the following effects are obtained. be able to.

  In addition, a binarized image obtained by performing graying processing, image segmentation / binarization processing, and color filtering processing on a color image obtained by photographing the operator with each of the web cameras 4 and 6 is displayed as an image. Since the data is stored in the storage area 16, the input device 1a can be configured even when the capacity of the hard disk 8 is small, and the cost of the entire device can be kept low.

  Further, a color image obtained by photographing the operator with each of the web cameras 4 and 6 is subjected to a small number of stages of image processing such as graying processing, image division / binarization processing, color filtering processing, etc. Since the binarized image is obtained, it is possible to prevent the CPU 9 from being subjected to a large burden, and to cope with the movement of the operator almost in real time even when the inexpensive CPU 9 whose processing speed is not fast is used. In addition, size control, position control, color control, click control, enlargement control, reduction control, rotation control, up / down scroll control, left / right scroll control, and the like of the operation target screen can be performed. It can be kept low.

  Further, the center coordinate position is determined by the binocular parallax method with respect to each activity rectangular area 26 on the right eye side and each activity rectangle area 26 on the left eye side obtained by photographing the operator with each of the web cameras 4 and 6. After the correction, numbers are assigned in order of size, and the center coordinate positions are compared. Based on the comparison result, the right-side active rectangular area 26 and the left-eye active rectangular area 26 corresponding to the focus position are selected. Therefore, even if there is a person other than the operator's hand at the focus position of each of the webcams 4 and 6, for example, a person behind the operator and moving, the operator's hand is not affected by this. By extracting only the movement of the hand, size control, position control, color control, click control, enlargement control, reduction control, rotation control, up / down scroll control, left / right scroll control, etc. of the operation target screen can be performed. it can.

  In the first embodiment of the present invention, when the operator is moving only one hand, it is determined that the control is a virtual cursor control instruction or a scroll control of the operation target screen, and the size control, position control, Since color control, click control, scroll control of the operation target screen, etc. are performed, the size, position, color, click, scroll of the operation target screen, etc. of the virtual cursor 25 displayed on the display unit 3 with only one hand. Can be operated remotely.

  In the first embodiment of the present invention, when the operator moves both hands, the movement of the right hand and the movement of the left hand are detected, and the enlargement / reduction control instruction of the operation target screen or the rotation of the operation target screen is detected. Since the control instruction is determined, the application screen (operation target screen) displayed on the display unit 3 can be enlarged, reduced, and rotated by simply moving the right hand and the left hand by the operator.

  In the first embodiment of the present invention, a virtual cursor activity region image 27 and a virtual button click activity region image are created from the histogram using the results obtained by statistically processing the histogram in the activity rectangular region extraction process. Therefore, a moving part such as an operator's hand can be accurately detected, and stable virtual cursor control, click control, and operation target screen control can be performed.

  In the first embodiment of the present invention, the multi-step rectangular object extraction process is performed on the virtual cursor activity area image 27 and the virtual button click activity area image in the activity rectangular area extraction process. Malfunction caused by the shadow of the camera can be prevented, and stable virtual cursor control, click control, and operation target screen control can be performed.

  FIG. 40 is a block diagram showing a second embodiment of the input device according to the present invention. An input device 1b shown in this figure is composed of a plastic member or the like formed in a box shape, and is an input device housing (illustrated) disposed in the vicinity of a remote operation target device such as a personal computer, a television, an air conditioner, or a large screen screen device. And a right-eye video camera main body (color camera main body) 30 that is attached to the front left side of the input device casing and that captures the operator and outputs a color image signal. A right image that is arranged in the input device casing and processes a color image taken by the video camera body 30 for the right eye to generate a virtual cursor activity area image on the right eye side and a virtual button click activity area image on the right eye side Left-eye video camera main body, which is an eye-side image processing board 31 and a left-eye imaging device that is attached to the front right side of the input device casing and that captures an operator and outputs a color image signal 2 and a color image placed in the input device casing and photographed by the left-eye video camera body (color camera body) 32, the left-eye side virtual cursor activity region image, the left-eye side virtual button A left-eye image processing board 33 that generates a click activity area image, a right-eye virtual cursor activity area image that is arranged in the input device housing and is output from the right-eye image processing board 31, and a right-eye virtual The button click activity region image, the left eye side virtual cursor activity region image output from the left eye side image processing board 32, and the left eye side virtual button click activity region image are processed to correspond to the movement of the operator's hand. Generate pointing data to be used, and remotely operated devices such as personal computers, televisions, air conditioners, and large screen devices via cables such as USB cables and signal connection cables. And a common substrate 34 supplies.

  Then, by analyzing the color image obtained by photographing the operator, the movement of the operator's hand while removing the influence of shadows, the influence of the person on the front side of the operator, and the influence of the person behind. Pointing data is generated, and the pointing data is supplied to the remote operation target device through the path of the input device 1b → cable → remote operation target device, and the operation of the remote operation target device is controlled.

  The right-eye video camera main body 30 is composed of a color camera having a resolution of about 320 pixels × 240 pixels. When the power supply voltage, the clock signal, etc. are output from the right-eye image processing board 31, the operator is selected. The color video signal obtained by photographing is supplied to the right eye image processing board 31.

  The right eye side image processing board 31 converts the color video signal output from the video camera body for the right eye into a color image in RGB format, and then is specified in advance by the HSV (Hue / Saturation / Brightness) method. Using a color mask necessary for extracting a color image of a color (for example, skin color), the skin color image extracting circuit 35 for extracting the skin color image in the color image and the video camera body 30 for the right eye are output. After the color video signal is converted into a color image in RGB format, a graying processing circuit 36 for converting the color video signal into a monochrome image having a preset gradation, and a monochrome image output from the graying processing circuit 36 are set in advance. The screen is divided by the number of screen divisions (however, this screen division processing is skipped when screen division is not set) and binarized by the maximum likelihood threshold method. The logical division of the image division / binarization processing circuit 37, the binarized image output from the image division / binarization processing circuit 37, and the skin color image output from the skin color image extraction circuit 35 is performed. And a color filtering processing circuit 38 for extracting a skin color portion in the digitized image.

  Further, the right-eye image processing board 31 is stored in the frame buffer circuit 39 and the frame buffer circuit 39 for temporarily storing the binary image output from the color filtering processing circuit 38 for several frames to several tens of frames. The inter-frame difference processing circuit 40 that performs the inter-frame difference processing while sequentially reading the binarized images, and generates the difference image, and the difference images output in units of frames from the inter-frame difference processing circuit 40 are divided into the divided areas. The histogram processing circuit 41 that accumulates each time to generate a histogram, and performs statistical processing on the histogram output from the histogram processing circuit 41, and also uses the statistical processing result to determine virtual cursor activity area determination processing, virtual button click activity Perform area determination processing, multi-step rectangular object extraction processing, etc. Ri has been removed right eye of the virtual cursor activity area image, and a work rectangular area extraction processing circuit 42 for generating a virtual button click activity area image of the right eye side.

  Furthermore, the activity rectangular area extraction processing circuit 42 compares the activity rectangular area extracted from the latest difference image data of the multi-stage difference image data generated by the histogram processing circuit 41 with the virtual button click activity area image. When the extracted activity rectangle area exceeds the range of the virtual button click activity area, the extraction activity rectangle area is determined to be invalid so that the movement that the operator does not intend for the input operation is regarded as noise. Can be ignored.

  For the color video signal output from the right-eye video camera body 30, graying processing, screen division / binarization processing, color filtering processing, frame buffer processing, inter-frame difference processing, histogram processing, active rectangular area Extraction processing is sequentially performed to generate a virtual cursor activity region image on the right eye side and a virtual button click activity region image on the right eye side, which are supplied to the common processing board 34.

  The left-eye video camera main body 32 is composed of a color camera having a resolution of about 320 pixels × 240 pixels, and is operated when a power supply voltage, a clock signal, etc. are output from the left-eye image processing board 33. The person is photographed, and the color video signal obtained thereby is supplied to the left eye side image processing board 33.

  The left-eye-side image processing board 33 converts the color video signal output from the left-eye video camera body 32 into a color image in RGB format, and is preset in the HSV (Hue / Saturation / Brightness) method. A skin color image extraction circuit 43 that extracts a skin color image in a color image using a color mask necessary for extracting a color image of a specific color (for example, skin color) and an output from the left-eye video camera main body 32 After converting the color video signal to be converted into a color image in RGB format, a graying processing circuit 44 for converting to a monochrome image having a preset gradation, and a monochrome image output from the graying processing circuit 44 are set in advance. (However, when screen division is not set, this screen division processing is skipped) and binarization is performed using the maximum likelihood threshold method. Logical sum of the image division / binarization processing circuit 45 to be a binarized image, the binarized image output from the image division / binarization processing circuit 45 and the skin color image output from the skin color image extraction circuit 43 And a color filtering processing circuit 46 for extracting a skin color portion in the binarized image.

  Further, the left eye side image processing board 33 is stored in the frame buffer circuit 47 and the frame buffer circuit 47 for temporarily storing the binarized image output from the color filtering processing circuit 46 for several frames to several tens of frames. The inter-frame difference processing circuit 48 that performs the inter-frame difference processing while sequentially reading the binarized images, and generates the difference image, and the difference images output in units of frames from the inter-frame difference processing circuit 48 are divided into the divided areas. A histogram processing circuit 49 that accumulates each time and generates a histogram, and performs statistical processing on the histogram output from the histogram processing circuit 49, and also uses the statistical processing result to determine virtual cursor activity area determination processing and virtual button click activity Perform area determination processing, multi-step rectangular object extraction processing, etc. Ri except the left eye side of the virtual cursor activity area image, and a work rectangular area extraction processing circuit 50 for generating a virtual button click activity area image of the left eye side.

  Further, the activity rectangular area extraction processing circuit 50 compares the activity rectangular area extracted from the latest difference image data of the multi-stage difference image data generated by the histogram processing circuit 49 with the virtual button click activity area image. As shown in FIG. 38 described above, when the extracted activity rectangular area exceeds the range of the virtual button click activity area, the operator performs an input operation by performing a process of determining invalidity of these extracted activity rectangular areas. Unintentional movement can be ignored as noise.

  For the color video signal output from the left-eye video camera main body 32, graying processing, screen division / binarization processing, color filtering processing, frame buffer processing, inter-frame difference processing, histogram processing, active rectangular area Extraction processing is sequentially performed to generate a virtual cursor activity region image on the left eye side and a virtual button click activity region image on the left eye side, and supply them to the common processing board 34.

  The common processing board 34 includes right-eye web camera 30 and left-eye web camera 32 attachment position data (horizontal distance “B”, vertical distance, etc.) necessary for position correction by the binocular parallax method, right-eye web camera. 30, a shooting condition setting circuit 51 in which shooting condition information such as the focal length “f” of the left-eye web camera 32 is set, and a right-eye virtual cursor activity region image output from the right-eye image processing board 31 The right eye side virtual button click activity region image, the left eye side virtual cursor activity region image output from the left eye side image processing board 33, and the left eye side virtual button click activity region image include an activity rectangular region. A process of correcting the position of each active rectangular area by binocular parallax using the shooting condition information set in the shooting condition setting circuit 51, a process of adding numbers to each active rectangular area in order of size, Processing to calculate the distance between the center coordinates (center coordinate distance) of each activity rectangular area to which the same number is added, processing to select the activity rectangle area corresponding to each center coordinate distance that is less than or equal to the predetermined value, and selected Perform a process such as creating a virtual cursor activity area image on the left eye side that contains only the activity rectangle area and not an unselected activity rectangle area, and a virtual button click activity area image on the left eye side. Virtual cursor activity region image, right eye side virtual button click activity region image, left eye side virtual cursor activity region image, left eye side virtual button click activity region image, a person in front of the operator, and behind After removing the influence of the movement of the person in the area, the active rectangle area that holds several to several tens of virtual cursor activity area images on the left eye side and virtual button click activity area images on the left eye side And a-option processing circuit 52.

  Furthermore, the activity rectangular area selection processing circuit 52 uses the viewing angles of the web cameras 30 and 32 as constants to change the color according to the center coordinate distance in the horizontal direction between the center points of the right eye activity rectangular area and the left eye activity rectangle area. By correcting the distance from the camera to the subject, it is possible to more accurately remove the influence of the movement of the person in front of the operator and the person behind. The principle of the correction and the calculation formula for the correction are as shown in FIG. 39 and [Formula 5]. The active rectangular area selection processing circuit 52 can realize high-speed processing by holding the distance to the subject calculated in advance according to the center coordinate distance as table data.

In the processing operation of the activity rectangular area selection processing circuit 52, the process of calculating the distance (center coordinate distance) between the center coordinates of each activity rectangular area to which the same number is added is the function of the distance measuring means C in FIG. It corresponds to. That is, if the distance between the web cameras 30 and 32 and the focal length f are set in advance, the distance to the subject is correlated with | X _L −X _R |, so the calculation of the center coordinate distance measures the distance to the subject. Will be.

  Further, the process in which the activity rectangular area selection processing circuit 52 removes the influence of the movement of the person in front of the operator and the person behind is equivalent to the action of the subject discrimination means D in FIG. That is, by comparing the distance to the subject measured by the calculation processing of the center coordinate distance and the preset depth distance, and invalidating the active rectangular area when it does not match within the allowable range, A virtual touch screen is formed at the set depth distance, and only the active rectangular area corresponding to the hand on the virtual touch screen by the operator is selected.

  Further, the common processing board 34, when the active rectangle area group exists in the latest virtual cursor activity area image among the virtual cursor activity area images on the left eye side held in the activity rectangle area selection processing circuit 52, Virtual cursor position instruction, virtual cursor shape instruction, virtual cursor color instruction, operation target screen scroll instruction, operation target screen enlargement instruction, operation target screen reduction instruction based on the number of active rectangle areas, shape, movement presence / absence, movement direction, etc. In addition to generating pointing data such as an operation target screen rotation instruction, and when the virtual cursor is in a clickable state, among the virtual button click activity area images held in the activity rectangular area selection processing circuit 52, Checks whether there is an activity rectangle area group in the latest virtual button click activity area image, and the activity rectangle area When the group is present, a work such as a single click instruction based on the rectangular region group shape, the virtual cursor control processing / display control processing circuit 53 for generating a pointing data such as a double-click instruction. The processing for generating pointing data in the virtual cursor control processing / screen control processing circuit 53 corresponds to the operation of the operation instruction generating means E in FIG.

  Then, the right eye side virtual cursor activity region image output from the right eye side image processing substrate 31, the right eye side virtual button click activity region image, and the left eye side virtual output from the left eye side image processing substrate 32. When an active rectangle is included in the cursor activity area image or virtual button click activity area image on the left eye side, noise caused by the movement of the person in front of the operator or the movement of the person behind the operator is removed. While determining how each hand of the operator is moving, according to the determination result, virtual cursor position instruction, virtual cursor shape instruction, virtual cursor color instruction, operation target screen scroll instruction, operation target screen enlargement instruction, Pointing data such as operation target screen reduction instructions and operation target screen rotation instructions are generated, and the PC, TV, air conditioner, large screen Supplied to such lean apparatus.

  As described above, in the second embodiment, the color filtering process, gray level is applied to the low-resolution color image obtained by photographing the operator by the right-eye video camera body 30 and the left-eye video camera body 32. The operator's hand performs image processing / image segmentation / binarization processing, frame buffer processing, inter-frame difference processing, histogram processing, active rectangular region extraction processing, active rectangular region selection processing, virtual cursor control processing / screen control processing, etc. Point data, virtual cursor position instruction, virtual cursor shape instruction, virtual cursor color instruction, operation target screen scroll instruction, operation target screen enlargement instruction, operation target screen reduction instruction, operation target screen rotation instruction, etc. are generated Therefore, the virtual cursor size and virtual cursor on the remote operation target device side are supplied to the remote operation target device. Sol position, the virtual cursor color, click, scroll up and down of the operation target screen, left and right scrolling, enlargement, reduction, rotation, etc., can be remotely operated.

  Further, in the second embodiment, the low-resolution color image obtained by photographing the operator with the right-eye video camera main body 30 and the left-eye video camera main body 32 is subjected to graying processing, image division / Image processing with a small number of stages such as binarization processing, color filtering processing, frame buffer processing, inter-frame difference processing, histogram processing, active rectangular area extraction processing, active rectangular area selection processing, virtual cursor control processing / screen control processing, etc. Since the pointing data is generated, the flesh color image extraction circuits 35 and 43, the graying processing circuits 36 and 44, the image division / binarization processing circuits 37 and 45, the color filtering processing circuits 38 and 46, the frame Buffer processing circuits 36, 47, inter-frame difference processing circuits 40, 48, histogram processing circuits 41, 49, active rectangle As the area extraction processing circuits 42 and 50, the active rectangular area selection circuit 52, and the virtual cursor control processing / screen control processing circuit 53, it is possible to use an element whose processing speed is not so fast and keep the cost of the entire apparatus low. It is possible to detect the movement of the operator in real time and control the remote operation target device.

  In the second embodiment, the right-eye active rectangular area obtained by photographing the operator with the right-eye video camera main body 30 and the left-eye video camera main body 32 and the left-eye side After correcting the center coordinate position with the binocular parallax method for the activity rectangular area, the center coordinate position is compared in the order of size, and each activity on the right eye side corresponding to the focus position based on this comparison result Since the rectangular area and each active rectangular area on the left eye side are selected, the right hand video camera main body 30 and the left eye video camera main body 32 other than the operator's hand at the focus position, For example, even if there is a person behind the operator and moving, only the movement of the operator's hand is extracted without being affected by this, and the size control, position control, color control, click control, and operation target of the virtual cursor are extracted. Screen enlargement control, reduction control, rotation control , Vertical scroll control can be performed such as horizontally scrolling control).

  In the second embodiment, as in the first embodiment described above, when the operator moves only one hand, it is one of the virtual cursor control instruction, the click control instruction, and the scroll control instruction. And pointing data indicating virtual cursor size instructions, virtual cursor position instructions, virtual cursor color instructions, scroll control instructions, click instructions, etc. are generated. The displayed virtual cursor can be remotely operated such as the size, position, color, click operation, and scrolling of the operation target screen.

  Also in the second embodiment, as in the first embodiment described above, when the operator moves both hands, the right hand movement and the left hand movement are detected to control the operation target screen. Since the pointing data indicating the operation target screen enlargement instruction, operation target screen reduction instruction, operation target screen rotation instruction, etc. is generated, the operator can move the right hand and the left hand. The operation target screen displayed on the display on the device side can be enlarged, reduced or rotated.

  Further, in the second embodiment of the present invention, the active rectangular area extraction processing circuits 42 and 50 statistically process the histogram, and using the statistical processing result, the virtual cursor active area image on the right eye side from the histogram, the right eye Side virtual button click activity area image, left virtual cursor activity area image, left virtual button click activity area image are created so that moving parts such as the operator's hand can be accurately detected It is possible to perform stable virtual cursor control, click control, and operation target screen control.

  In the second embodiment of the present invention, in the active rectangle area extraction processing circuits 42 and 50, the right eye side virtual cursor activity area image, the right eye side virtual button click activity area image, and the left virtual cursor activity area Multi-step rectangular object extraction processing is performed on the image and the virtual button click activity area image on the left side to prevent malfunctions caused by the shadow of the operator, stable virtual cursor control, click control, and operation Target screen control can be performed.

  In each of the above-described embodiments, the entire area of the color image obtained by each of the web cameras 4 and 6, the right-eye video camera main body 30, and the left-eye video camera main body 32 is grayed out and binarized. However, as shown in the flowchart of FIG. 41 and the schematic diagram of FIG. 42, the specified enlargement / reduction for the change area rectangle (rectangle including the active rectangle area) 65 obtained by the statistical processing of the histogram and the active rectangle area extraction process is shown. An enlargement / reduction rectangular mask 66 enlarged / reduced at a rate (eg, enlargement rate “10%”) is created (step S71), and the entire color image area of the next frame is enlarged from the monochrome image obtained. / Only the portion corresponding to the reduced rectangular mask 66 (the image 67 of the active region portion included in the monochrome image) may be extracted and binarized (step S72).

  In this way, only the image included in the area slightly wider than the active area from the monochrome image etc. is enabled, the image in the other area is disabled, and the noise existing in the part other than the change area is removed. can do.

  Further, in the first embodiment and the second embodiment described above, the skin color image is extracted by the color filtering processing by the CPU 9 or the color filtering processing circuits 38 and 46. When controlling the position, click, scrolling of the operation target screen, enlargement of the operation target screen, reduction of the operation target screen, rotation of the operation target screen, etc. using a specific color operation device such as a red pen May use a color mask for red extraction and extract a red color image by the color filtering processing by the CPU 9 or the color filtering processing circuits 38 and 46.

  Thereby, even if there are a plurality of people in the shooting range of each of the web cameras 4, 6, the right-eye video camera main body 30, and the left-eye video camera main body 32, a color image corresponding to the color of the operator's operating device is obtained Extraction can be performed to perform size control, position control, click control, scroll control, enlargement control, reduction control, rotation control, and the like of the operation target screen.

  Further, such color filtering processing is performed by moving each web camera 4, 6, the right-eye video camera body 30, and the operator's hand included in the color video signal output from the left-eye video camera body 32. Color filtering when the lighting conditions in the place where the operator is located are good and the contrast between the image where the operator's hand is moving and the background image is large. Processing may be omitted.

  The suitable Example of the touchless interface operation which concerns on the input device of this invention is shown. As described in the first embodiment, the touchless interface is executed by the CPU 9 in the virtual cursor control process / screen control process in step S18 in the flowchart of FIG. That is, the images from the web cameras 4 and 6 are processed to detect the operation of the operator from the shape of the active region group, and the instruction from the operator according to the movement is recognized. A simple operation instruction method will be described.

(Virtual cursor display operation)
As shown in FIG. 43, when the operator performs an operation (tapping operation) to move only the index finger or both the index finger and the middle finger up and down (tapping operation), the CPU 9 performs the process of step S64 (FIG. 15). This operation is detected from the shape and cursor display control is started.

  First, the CPU 9 controls the display unit 3 so as to perform a response display (feedback) of the virtual cursor display start on the screen. For example, as shown in FIGS. 44 (a) and 44 (b), this feedback sequentially displays gray circles on the screen so that ripples spread from small to large.

  After the feedback display, the CPU 9 controls the display unit 3 to display the virtual cursor 25A in the gray first state as shown in FIG. 44 (c).

(Virtual cursor lock operation)
When the CPU 9 detects that the operator stops the tapping operation or reduces the operation while the gray virtual cursor 25A is displayed, the CPU 9 displays the virtual cursor 25A in the red second state. The display unit 3 is controlled to display the actual cursor 28A by switching to the display (FIG. 45).

  In this embodiment, the cursor is displayed in response to the tapping operation by the operator, but in the first embodiment, the cursor is displayed by turning the fingertip. In addition, when a space where the operation by the hand or the fingertip is effective at a position far from the web cameras 4 and 6, a movement of swinging the palm toward the front as shown in FIG. 46 by the tapping operation of the fingertip. It is preferable for recognizing an action of swinging the palm vertically (such as bye-bye) or beckoning.

(Actual cursor movement operation)
When the virtual cursor 25A is displayed in the red second state as shown in FIG. 45, the CPU 9 moves the cursor when it detects that the operator performs the above-described tapping operation at the target position in the vertical and horizontal directions. Take control. At this time, the CPU 9 generates a first signal and controls the display unit 3 to move and display the virtual cursor 25A to that position. Since the virtual cursor 25A is displayed in the red second state, the operator can recognize that the virtual cursor 25A is movable.

  In this way, by pointing an arbitrary coordinate position on the display unit 3 by a tapping operation, the cursor can be moved to that coordinate position.

(Click operation and double click operation)
When the virtual cursor 25A is displayed in the first gray state, the CPU 9 starts from the state where the operator stands only with the index finger as shown in FIG. 47 (part (a) in FIG. 47). The movement of the second form which changes to the state where the finger is raised and the palm is expanded (part (b) in FIG. 47), or the state where only the index finger is raised as shown in FIG. 48 (part (a) in FIG. 48) ) To expand the palm (part (b) of FIG. 48), and click again to detect the movement of the third form that changes to the state where only the index finger is raised (part (c) of FIG. 48). Take control.

  At this time, the CPU 9 recognizes a single click operation when it detects the second form of operation and generates a second signal, and recognizes a double click operation when it detects the operation of the third form. 3 signal is generated. By generating the second signal or the third signal, the CPU 9 can notify the operator that the single click or the double click has been recognized by the color, shape, and sound of the virtual cursor 25A. When the third signal is generated with the real cursor 28A over the link or application selection button on the screen of the display unit 3, the jump to the link destination or the execution of the application is started. In the case of the first embodiment, single click and double click are recognized by the operator performing an operation of expanding the palm from the pointing state with one finger and returning to the pointing state.

  As described above, regarding the display and movement of the cursor and the click operation, as another example, the virtual cursor is displayed by moving the palm or a predetermined recognition object toward the cameras 4 and 6, and the virtual moves as the palm moves. There is also an operation method in which the cursor is moved, the virtual cursor is locked by stopping the movement of the palm at the target position, and then the user clicks by swinging the palm left and right.

(Flip operation)
When the virtual cursor 25A is displayed in the red second state as shown in FIG. 49 (a), the CPU 9 moves from (b) to (c) in FIG. As shown in the figure, when it is detected that the movement of the fourth form in which the palm is swung to the right or left so as to turn the page of the book, screen scroll control is performed. At this time, the CPU 9 generates a fourth signal and controls the scroll operation of the screen display of the display unit 3. In FIG. 49, the operation from the portion (b) to the portion (c) is a flip operation in the right direction, and the screen is scrolled from the left to the right.

  When the virtual cursor 25A is displayed in the gray first state before the flip operation, the operator performs a tapping operation as shown in FIG. 49 (a), and the virtual cursor 25A is red. Scrolling is performed by performing a flip operation after being displayed in the second state.

(Drag and drop operation)
When the operator detects a tapping operation by a fingertip at a target object or screen position on the screen of the display unit 3, the CPU 9 performs drag and drop control. At this time, the CPU 9 displays the virtual cursor 25A in the red second state at that location (part (a) of FIG. 50), and then moves the palm to the screen to the left and right in the fifth form ( When the (b) part of FIG. 50 is detected, the drag operation is recognized. When the CPU 9 further detects that the palm is moved to the target position with the palm facing the screen, the CPU 9 generates a fifth signal, displays the icon 70 and displays the icon 3 so as to follow the movement of the palm. Control (part (c) of FIG. 50). Then, when the CPU 9 detects the movement of the sixth form in which the operator taps with a finger (part (d) in FIG. 50) or stops moving and waits for a few seconds and then releases the hand, the sixth signal And the display unit 3 is controlled to display the drop of the object.

  In this way, the drag-and-drop operation performs a tapping operation on the object, clicks on the object, then shakes the palm to the left and right to catch the object, and if the catch is recognized by the icon 70 display, Is moved to a target position and a tapping operation with a finger is performed.

  The above is the operation instruction given by the movement of the hand or finger of one arm by the operator. The operation instruction given by the movement of the hand or finger by both arms will be described.

(Screen enlarge / reduce operation)
In step S65 (FIG. 15), the CPU 9 displays a two-point cursor 71 when it detects from the shape of the active region group that the operator performs a tapping operation at two points with both hands or fingers of both hands (FIG. 51 (( a) control the display unit 3 to When the CPU 9 detects that the operator performs the tapping operation or the interval between the fingers of both hands is enlarged or reduced, the CPU 9 performs screen enlargement / reduction control.

  At this time, the CPU 9 enlarges or reduces the interval of the two-point cursor 71 (part (b) in FIG. 51) or reduces (reduces) the interval between the two hands or fingers of both hands where the operator performs the tapping operation. The display unit 3 is controlled so as to be displayed as shown in FIG. Then, the CPU 9 generates a seventh signal when the tapping operation at the two points is stopped, and controls the display unit 3 to perform the enlarged display or the reduced display of the screen according to the interval of the two-point cursor 71.

(Screen rotation operation)
On the other hand, the CPU 9 detects the simultaneous tapping operation at two points with both hands of the operator or the fingers of both hands and displays the two-point cursor 71 (part (a) in FIG. 52), and performs the tapping operation. When it is detected that the relative position of both hands or fingers of both hands changes up and down, screen rotation control is performed.

  At this time, the CPU 9 controls the display unit 3 so that the two-point cursor 71 is displayed with a width in the vertical direction in accordance with the relative positional relationship between the upper and lower hands of the two hands or fingers of both hands where the operator performs the tapping operation. Then, the CPU 9 generates an eighth signal when the tapping operation at the two points is stopped, and displays the screen by rotating the screen to the right or left according to the vertical positional relationship of the two-point cursor 71. The display unit 3 is controlled. That is, when the left cursor is at a position higher than the right cursor (part (b) in FIG. 52), it is rotated to the right, and when it is opposite (part (c) in FIG. 52), it is rotated to the left.

  In this embodiment, a screen enlargement / reduction instruction or a screen rotation instruction is performed by a tapping operation with both hands. Compared with the case of the screen enlargement / reduction instruction or the screen rotation instruction of the first embodiment, the tapping operation is incorporated to perform an operation. The recognition rate is further increased.

  As described above, when an operator gives an operation instruction by the movement of a hand or a finger, notifying the operator of the position on the screen leads to an improvement in operability. There are the following two methods for this purpose.

  One is to transparently display the operator himself / herself reflected on the web cameras 4 and 6 on the screen of the display unit 3. The CPU 9 controls the video capture 5 to display the video signal of the operator captured by the display I / F 11. To be output to the display unit 3 and displayed.

  Thus, the operator can intuitively point the screen position. In addition, the operator can see the content on the screen without being disturbed by the transparent display by transparently displaying the operator himself / herself on the screen according to the movement of the operator. For example, when the CPU 9 detects that the operation by the operator is stopped and there is no active area group in the process of step S63, the CPU 9 does not perform the transparent display, and on the other hand, when the operation is performed and the presence of the active area group is detected. The output of the video signal to the display unit 3 is controlled so as to perform transmissive display.

  The other is to detect the movement of the operator's hand and transparently display an effect corresponding to the movement on the screen of the display unit 3. When the CPU 9 recognizes the movement of the operator's hand from the shape of the active area group in the process of step S64 or step S65, the CPU 9 displays an effect corresponding to the operation.

  For example, when the CPU 9 detects a flip movement in which the palm is shaken to the right or left as shown in the (b) to (c) parts of FIG. 49, the flip effect image stored in the image storage area 16 in advance is displayed. The data is output to the display unit 3 through the display I / F 11. By displaying the effect in this way, the operator can know the position currently pointing.

  By using either of these two methods at the same time or selectively using either one according to the contents of the operation instruction operation, the operator can easily operate, and an input device with an excellent user interface is provided. .

  The present invention is used by being connected to an information device such as an information terminal device or a personal computer, capturing an operation on a virtual touch screen from an operation image of an operator by a camera, The present invention relates to an input device that controls selection and execution of an application program, and has industrial applicability.

A imaging device for the right eye B imaging device for the left eye C distance measurement means D subject determination means E operation instruction generation means 1a, 1b input device 2 personal computer 3 display unit 4 web camera (color camera for left eye)
5 Video capture 6 Web camera (color camera for right eye)
9 CPU
25, 25A Virtual cursor 26 Activity rectangle area 27 Virtual cursor activity area image 28, 28A Real cursor 30 Right-eye video camera body (right-eye color camera body)
31 Right-eye image processing board 32 Left-eye video camera body (left-eye color camera body)
33 Left-eye image processing board 34 Common processing board 35 Skin color image extraction circuit 36 Graying processing circuit 37 Image segmentation / binarization processing circuit 38 Color filtering processing circuit 39 Frame buffer circuit 40 Inter-frame difference processing circuit 41 Histogram processing circuit 42 Activity rectangular area extraction processing circuit 43 Skin color image extraction circuit 44 Graying processing circuit 45 Image segmentation / binarization processing circuit 46 Color filtering processing circuit 47 Frame buffer circuit 48 Inter-frame difference processing circuit 49 Histogram processing circuit 50 Activity rectangular area extraction processing Circuit 51 Imaging condition setting circuit 52 Active rectangular area selection processing circuit 53 Virtual cursor control processing / screen control processing circuit 65 Change area rectangle 66 Enlarged / reduced rectangle mask 67 Image after masking

Claims (43)

An input device that processes an operator's image obtained by an imaging device and generates an operation instruction according to the operation content of the operator,
An imaging device for the right eye;
A left-eye imaging device arranged in parallel with the right-eye imaging device at a predetermined interval from the right-eye imaging device;
Distance measuring means for measuring a distance to the subject from the images of the right-eye imaging device and the left-eye imaging device with respect to the subject by a binocular parallax method;
Based on the distance between the imaging device for the right eye and the imaging device for the left eye, a depth distance determined by a binocular parallax method and set in advance, and a distance to the subject measured by the distance measuring unit, A subject discriminating means for selecting the subject in which the distance to the subject and the depth distance match within a predetermined allowable range and discriminating from the operator;
An operation instruction generating means for detecting the movement of the subject determined by the subject determining means and generating an operation instruction according to the detection result;
An input device comprising: A display unit for displaying a recognition result of the operation instruction;
The operation instruction generation means displays the virtual cursor in the first state on the display unit when detecting the movement of the first form by the operator's hand or finger while waiting for the cursor to be displayed. A cursor that displays the virtual cursor in the second state and displays the real cursor when it is detected that the first movement by the hand or finger is stopped or the first movement is reduced during the display in the state The input device according to claim 1, further comprising display control means.   The operation instruction generation means moves the cursor to the position of the hand or finger when the hand or finger moves in the first form when the virtual cursor is displayed in the second state. 3. The input device according to claim 2, further comprising cursor movement control means for generating a first signal for instructing.   The operation instruction generation unit generates a second signal that recognizes a click operation when detecting the movement of the second form by the hand or the finger when the virtual cursor is displayed in the first state. 4. A click control means for generating a third signal for recognizing a double click operation when a movement of the third form following the movement of the second form is detected. The input device described.   When the virtual cursor is displayed in the first state and the movement of the fourth form by the hand or finger is detected when the virtual cursor is displayed in the first state, the operation instruction generation unit instructs the screen scroll of the display unit The input device according to claim 2, further comprising a screen scroll control unit that generates the signal.   The operation instruction generating means detects that the object is a drag target object when detecting the movement of the fifth form by the hand or finger in a state where the real cursor is superimposed on the object displayed on the display unit. Recognizing and generating a fifth signal instructing movement of the object following movement by the hand or finger after recognition, and then detecting movement of the object in the first form by the hand or finger. 6. The input device according to claim 2, further comprising drag and drop control means for generating a sixth signal for instructing dropping.   The operation instruction generating unit displays a two-point cursor when detecting that the two hands or fingers perform the movement of the first form, and the interval between the hands or fingers performing the movement of the first form. The interval between the two-point cursors is enlarged or reduced according to the enlargement or reduction of the image, and the display unit is enlarged or reduced according to the interval between the two-point cursors when the movement of the first mode is stopped. The input device according to claim 2, further comprising a screen enlargement / reduction control unit that generates a seventh signal instructing reduction display.   The operation instruction generating means displays the two-point cursor when detecting that the two hands or fingers respectively perform the movement of the first form, and also displays the two-point cursor and the hand or fingers performing the movement of the first form. The relative position of the two-point cursor is changed up and down according to the up-and-down change in the relative position, and the two-point cursor is moved according to the up-and-down positional relationship when the movement of the first mode is stopped. The input device according to claim 2, further comprising a screen rotation control unit that generates an eighth signal instructing a right rotation or a left rotation of display on the display unit.   The input device according to claim 2, wherein the display unit transparently displays an image of the operator by the imaging device.   The input device according to claim 9, wherein the display unit transparently displays the image when the operator is performing an operation instruction.   The input device according to claim 2, wherein the display unit displays an effect of an operation instruction performed by the operator by the movement of the hand or the finger. An input device that processes an operator's image obtained by a video camera and generates an operation instruction according to the operation content of the operator,
A right-eye color camera,
A left-eye color camera arranged side by side with the right-eye color camera at a position away from the right-eye color camera by a predetermined distance;
The operator performs graying processing, image division / binarization processing, inter-frame difference processing, histogram processing, and active rectangular area extraction processing on the color image output from the right-eye color camera that captures the subject. A right eye side image processing program for extracting a right eye side activity rectangular region of
The color image output from the left-eye color camera capturing the subject is subjected to graying processing, image segmentation / binarization processing, inter-frame difference processing, histogram processing, and active rectangular area extraction processing, and the operation Left eye side image processing program for extracting the left eye side activity rectangular area of the person,
By calculating the center coordinate distance between the coordinate axis of the right eye side active rectangular area obtained by the right eye side image processing program and the coordinate axis of the left eye side active rectangular area obtained by the left eye side image processing program, both are calculated. The distance to the subject is measured by an eye parallax method, and the measured distance to the subject is determined by a binocular parallax method based on the interval between the right eye imaging device and the left eye imaging device. The right eye activity rectangular area and the left eye activity rectangular area in which the distance to the subject and the depth distance match within a predetermined tolerance range are compared with the depth distance set in advance by the operator. An activity area selection processing program to be selected as an activity rectangle area;
A virtual cursor control process / screen control process is performed on the active rectangular area selected by the active area selection processing program to detect the movement of the hand or fingertip of the subject, and an operation instruction corresponding to the detection result An image processing program for generating
An input device comprising:   In the virtual cursor control process / screen control process, when there is one active rectangular area group on the virtual cursor active area image, a Carlsol control instruction or a screen scroll instruction is generated based on the shape and presence / absence of movement. The input device according to claim 12.   The virtual cursor control process / screen control process generates one of a screen rotation instruction, a screen enlargement instruction, and a screen reduction instruction based on the moving direction when there are two active rectangular area groups on the virtual cursor active area image. The input device according to claim 12 or 13, characterized in that:   15. The input according to claim 12, wherein the activity rectangular area extraction processing creates a virtual cursor activity area image from the histogram using a histogram statistical processing result. apparatus.   16. The active rectangle area extracting process performs a multi-stage rectangular object extracting process on the virtual cursor active area image to remove a noise component. Input device.   An enlargement / reduction rectangle mask creation process is added, and an image corresponding to the change area rectangle on the virtual cursor activity area image is selected from the color images obtained by the color camera by the enlargement / reduction rectangle mask creation process. The input device according to any one of claims 12 to 16, wherein a noise component is removed by extracting and cutting other images. A display unit for displaying a recognition result of the operation instruction;
The image processing program displays the virtual cursor in the first state on the display unit when detecting the movement of the first form by the operator's hand or finger while waiting to display the cursor. Cursor display for displaying the virtual cursor in the second state and displaying the real cursor when it is detected that the first movement by the hand or finger is stopped or the first movement is reduced during display The input device according to any one of claims 12 to 17, further comprising a control step.   When the virtual cursor is displayed in the second state and the hand or finger moves in the first form, the image processing program moves the cursor to the position of the hand or finger. 19. The input device according to claim 18, further comprising a cursor movement control step of generating a first signal to be instructed.   The image processing program generates a second signal for recognizing a click operation when detecting the movement of the second form by the hand or finger when the virtual cursor is displayed in the first state, 20. A click control step of generating a third signal for recognizing a double-click operation when a movement of the third form following the movement of the second form is detected is provided. The input device described.   When the virtual cursor is displayed in the first state and the movement of the fourth form by the hand or finger is detected, the image processing program instructs the display unit to flip display. 21. The input device according to claim 18, further comprising a step of screen scroll control for generating the signal.   The image processing program recognizes that the object is the object to be dragged when detecting the movement of the fifth form by the hand or finger in a state where the real cursor is superimposed on the object displayed on the display unit When the fifth signal for instructing the movement of the object following the movement by the hand or finger after recognition is generated, and the movement of the first form by the hand or finger is subsequently detected, the object is dropped. The input device according to any one of claims 18 to 21, further comprising a step of drag-and-drop control for generating a sixth signal instructing the control signal.   When the image processing program detects that the two hands or fingers perform the movement of the first form, a two-point cursor is displayed, and an interval between the hands or fingers performing the movement of the first form is displayed. The interval between the two-point cursors is enlarged or reduced according to the enlargement or reduction, and the display unit is enlarged or reduced according to the interval between the two-point cursors when the movement of the first mode is stopped. The input device according to any one of claims 18 to 22, further comprising a screen enlargement / reduction control step of generating a seventh signal for instructing display.   The image processing program displays the two-point cursor when detecting that the two hands or fingers perform the movement of the first form, respectively, and also displays the two-point cursor of the hand or finger performing the movement of the first form. Displaying the relative position of the two-point cursor up and down in accordance with the up-and-down change in the relative position, and displaying the display in accordance with the up-and-down positional relationship of the two-point cursor when the movement of the first mode is stopped The input device according to any one of claims 18 to 23, further comprising a screen rotation control step of generating an eighth signal instructing a right rotation or a left rotation of the display of the unit.   The input device according to any one of claims 18 to 24, wherein the display unit transparently displays an image of the operator by the color camera.   26. The input device according to claim 25, wherein the display unit transparently displays the image when the operator is performing an operation instruction.   The input device according to any one of claims 18 to 26, wherein the display unit displays an effect of an operation instruction performed by the operator by the movement of the hand or the finger. An input device that processes an image of an operator obtained by a video camera, generates an operation instruction according to the operation content of the operator, and controls the operation of the remote operation target device,
An input device housing formed in a box shape;
A color camera body for the right eye attached to the front left side of the input device housing;
A color camera body for the left eye attached to the front right side of the input device housing;
Output from the right-eye color camera body by the graying processing circuit, the image segmentation / binarization processing circuit, the inter-frame difference processing circuit, the histogram processing circuit, and the active rectangular area extraction processing circuit which are arranged in the input device casing. A right eye side image processing board that processes the color image to be extracted and extracts the right eye side activity rectangular region of the operator;
Output from the left-eye color camera body by the graying processing circuit, image segmentation / binarization processing circuit, inter-frame difference processing circuit, histogram processing circuit, and active rectangular area extraction processing circuit, which is arranged in the input device casing. A left eye side image processing board that processes the color image to be extracted and extracts the left eye side activity rectangular region of the operator;
The right eye side obtained by the right eye side image processing program using the binocular parallax method by the active rectangle area selection processing circuit, the virtual cursor control processing / screen control processing circuit, which is arranged in the input device casing Measure the distance to the subject by binocular parallax method by calculating the distance from the coordinate axis of the activity rectangular area and the coordinate axis of the left eye side activity rectangular area obtained by the left eye side image processing program, The distance to the subject is determined by a binocular parallax method based on the distance between the right-eye imaging device and the left-eye imaging device, and compared with a preset depth distance, The right-eye activity rectangular region and the left-eye activity rectangular region in which the distance and the depth distance coincide within a predetermined allowable range are selected as the activity rectangle region by the operator, and the selected activity rectangle Common processing for performing virtual cursor control processing / screen control processing on the area, detecting movement of the hand or fingertip of the subject, generating pointing data according to the detection result, and controlling the operation of the remote operation target device A substrate,
An input device comprising:   The virtual cursor control processing / screen control processing circuit generates a Carl Sol control instruction or a screen scroll instruction based on the shape and presence / absence of movement when there is one active rectangular area group on the virtual cursor active area image. The input device according to claim 28, characterized in that:   The virtual cursor control processing / screen control processing circuit, when there are two active rectangular area groups on the virtual cursor active area image, selects one of a screen rotation instruction, a screen enlargement instruction, and a screen reduction instruction based on the moving direction. 30. The input device according to claim 28, wherein the input device is generated.   31. The activity rectangular area extraction process according to claim 28, wherein a virtual button click activity area image is created from the histogram using a histogram statistical processing result. Input device.   32. The activity rectangle area extraction process according to claim 28, wherein a noise component is removed by performing a multi-stage rectangle object extraction process on the virtual button click activity area image. The input device described.   An enlargement / reduction rectangular mask creation processing circuit is added, and the virtual button click activity region image is changed from among the color images obtained by the color camera and the color camera body by the enlargement / reduction rectangular mask creation processing circuit. The input device according to any one of claims 28 to 32, wherein an image corresponding to a region rectangle is extracted, and other images are cut to remove noise components. A display unit for displaying a recognition result of the operation instruction;
The common processing board displays the virtual cursor in the first state on the display unit when detecting the movement of the first form by the hand or finger of the operator while waiting for the cursor to be displayed. Cursor display for displaying the virtual cursor in the second state and displaying the real cursor when it is detected that the first movement by the hand or finger is stopped or the first movement is reduced during display 34. The input device according to claim 28, further comprising a control unit.   The common processing board moves the cursor to the position of the hand or finger when the hand or finger moves in the first form when the virtual cursor is displayed in the second state. 35. The input device according to claim 34, further comprising a cursor movement control unit that generates the first pointing data to be instructed.   The second pointing data indicating that the common processing board is a click operation when the movement of the second form by the hand or the finger is detected when the virtual cursor is displayed in the first state. And a click control unit for generating the third pointing data indicating a double click operation when the movement of the third form following the movement of the second form is detected. Item 36. The input device according to item 34 or 35.   The common processing board, when detecting the movement of the fourth form by the hand or the finger when the virtual cursor is displayed in the first state, instructs the display unit to scroll the screen. 37. The input device according to claim 34, further comprising a screen scroll control unit that generates the pointing data.   The common processing board recognizes that the object is the object to be dragged when the movement of the fifth form by the hand or the finger is detected in a state where the real cursor is superimposed on the object displayed on the display unit. When the fifth pointing data for instructing the movement of the object following the movement by the hand or finger after recognition is generated, and the movement of the first form by the hand or finger is subsequently detected, the object is detected. 38. The input device according to claim 34, further comprising a drag-and-drop control unit that generates the sixth pointing data for instructing a drop.   The common processing board displays a two-point cursor when it detects that the two hands or fingers perform the movement of the first form, and the distance between the hands or fingers performing the movement of the first form is The interval between the two-point cursors is enlarged or reduced according to the enlargement or reduction, and the display unit is enlarged or reduced according to the interval between the two-point cursors when the movement of the first mode is stopped. 39. The input apparatus according to claim 34, further comprising a screen enlargement / reduction control unit that generates the seventh pointing data for instructing display.   The common processing board displays the two-point cursor when detecting that the two hands or fingers perform the movement of the first form, respectively, and also displays the two-point cursor of the hands or fingers performing the movement of the first form. Displaying the relative position of the two-point cursor up and down in accordance with the up-and-down change in the relative position, and displaying the display in accordance with the up-and-down positional relationship of the two-point cursor when the movement of the first mode is stopped 40. The input device according to claim 34, further comprising a screen rotation control unit that generates the eighth pointing data for instructing a right rotation or a left rotation of the display of the unit. .   41. The input device according to claim 34, wherein the display unit transparently displays an image of the operator by the imaging device.   42. The input device according to claim 41, wherein the display unit transparently displays the image when the operator is performing an operation instruction.   The input device according to any one of claims 34 to 42, wherein the display unit displays an effect of an operation instruction performed by the operator by the movement of the hand or the finger.

Images