JP2012068854A - Operation input device and operation determination method and program - Google Patents

Abstract

PROBLEM TO BE SOLVED: To accurately determine and infrequently misrecognize a motion input from intuitive and simple operations, without putting much burden on operators even during long-time operations.SOLUTION: A display 111 is placed in front of an operator 102. The operator 102 can perform operations by keeping in mind that shapes of a finger, etc. are relevant to an operation determination, in a range set at fixed positions at the right and the left between the operator 102 and the display 111. The present invention allows input of a position decision and an operation determination into a system by independently moving a right hand 114 and a left hand 115 on right- and left-hand sides of the display 111, viewing from the operator 102. Misrecognition as well as a physical burden accompanying the input can be both reduced by separately recognizing two motions, i.e., selection and determination.

Description

  The present invention relates to an operation input device and an operation determination method, and more particularly to an operation input device and an operation determination method for determining an operation of a measurement target from an image captured by a video camera or the like.

  The operation on the computer is basically performed using a display which is an output device and a keyboard and a mouse which are input devices. That is, the operator moves the mouse to move the pointer to a predetermined position such as a button or an input field displayed on the display, and by clicking or the like, “presses the button” or “input field input” starts. ”Is confirmed. As described above, conventionally, using a mouse, pointing or focusing (selection), clicking a click button once at a predetermined position to select the location, etc., clicking the click button twice in succession. Thus, operations such as a double click for executing various applications associated with the pointed position and a drag for linking the pointed position and the movement destination can be performed. Input keyboards are usually used to input specific information such as numerical values and characters. Therefore, the operator can move the mouse and input using the keyboard to output the desired information or input the necessary information. Will be repeated.

  In addition to these basic operation input methods, in addition to the conventional method of directing the display surface with a touch panel separately from a conventional mouse and keyboard, or inputting data with handwritten characters, the entire operator with a camera Alternatively, a technique has been proposed in which a part is photographed and the operation is performed by determining the will of the operator according to the image. For example, Patent Document 1 includes a host computer that recognizes the shape and movement of an object in an image captured by a CCD camera, and a display that displays the shape and movement of the object recognized by the host computer. If you give an instruction by hand gesture or the like, the given hand gesture will be displayed on the display screen of the display, you can select the virtual switch etc. displayed on the display screen with the arrow cursor icon by hand gesture, and input devices such as a mouse There has been proposed a technique that enables operation of a very simple device without being required. As such an operation input method other than a mouse and a keyboard, various input systems such as recognizing operation contents by recognizing as a kind of gesture from an image obtained by capturing the movement and shape of fingers are proposed. A method for eliminating the need for devices such as a mouse, a keyboard, and a touch panel is being sought.

  In addition, with the aim of further improving convenience related to the operation of the operator without wearing special equipment, the operator is provided with an operation means corresponding to a keyboard, a mouse, etc. For example, by defining a virtual plane G or a virtual keyboard H, an operator can perform a click operation, a drag operation, a double click operation, or the like as a mouse operation on a display image, a desired line, figure, Alternatively, a technique that enables drawing and inputting of characters and the like has been proposed (see, for example, Patent Document 2).

JP 2004-079777 A JP 2004-258714 A

  However, gesture operations, etc. that do not use a device such as a mouse, keyboard, touch panel, etc., are those that require non-intuitive and complicated operations, those that require physical burdens due to large-scale operations, such as the invention of Patent Document 2 Actually existing technologies have not been satisfied due to the fact that the operation target screen is difficult to see with the operating hand, and the problem of misrecognition caused by the ambiguity of the “connection” between gesture operations. In particular, the invention of Patent Document 2 can be input without attaching a special device or the like, and seems to be an effective technique at first glance. However, the implementation is very difficult due to the visibility problem that the operating hand itself covers the target screen and the problem that the two operation steps from “select” to “decision” are not clearly distinguished. It can be said that it is difficult.

  The present invention has been made in view of such problems, and while being intuitive and simple, it can secure a sufficient field of view for the operation target screen, has a low incidence of erroneous recognition, and can be operated even for a long time operation. It is an object of the present invention to provide an operation input device, an operation determination method, and a program that can accurately determine an operation input with less burden on the user.

  In order to achieve such an object, the invention according to claim 1 is an operation input device, which is arranged in the vicinity of a display means for displaying an image to be viewed by an operator and a display means for the operator. Including an imaging unit that generates image data of the operator and a range indicated by a predetermined series of operations of the operator read by the imaging unit, and a part of the operator controls the marking. A control range setting means for preliminarily setting a control range that is a range that does not cover an area defined by connecting the operator's viewpoint and each vertex of the display surface of the display means; When a part of the operator is extracted and the part of the operator is within the set control range, the predetermined sign displayed on the display means is moved according to the positional relationship between the part of the operator and the display means. On the display surface of the control means and the display means of the indication An operation determination unit that determines whether or not an operation related to a position has been performed, and a signal output unit that outputs a predetermined signal when the operation determination unit determines that the operation has been performed. To do.

  According to a second aspect of the present invention, in the operation input device according to the first aspect, the imaging unit is a three-dimensional imaging unit that generates stereoscopic image data of the operator, and the control range setting unit is read by the imaging unit. Display of the display means based on a predetermined series of actions of the given operator so that the control range does not overlap the area defined by connecting the operator's viewpoint and each vertex of the display surface of the display means A position determination surface associated with the surface is set in advance, and the control means moves the sign to a position on the display surface corresponding to a position on the position determination surface that is part of the operator. And

  According to a third aspect of the present invention, in the operation input device according to the second aspect, the position determination surface is formed between the display means and the operator.

  According to a fourth aspect of the present invention, in the operation input device according to any one of the first to third aspects, the operation determination means extracts a part different from the part of the operator from the operator's image, and operates It is characterized in that it is determined that a predetermined operation associated with the series of actions has been performed based on a series of actions by a part different from a part of the person.

  According to a fifth aspect of the present invention, in the operation input device according to the fourth aspect, based on a predetermined series of operations of the operator read by the imaging means, a part different from the part of the operator is used. A determination range setting unit that presets a determination range that is a range in which the determination can be performed is further provided, and the operation determination unit is configured to operate the operator when a part different from a part of the operator is within the set determination range. It is characterized in that it is determined that a predetermined operation associated with the series of operations has been performed based on a series of operations different from a part of the operations.

  According to a sixth aspect of the present invention, in the operation input device according to the fifth aspect, the imaging unit is a three-dimensional imaging unit that generates stereoscopic image data of the operator, and the determination range setting unit is read by the imaging unit. Based on a predetermined series of operations of the given operator, the determination range does not cover the exclusion area determined by connecting the operator's viewpoint and each vertex of the display surface of the display means, and the control range An operation determination surface associated with the display surface of the display unit is set in advance so as not to overlap, and the operation determination unit extracts a part different from the part of the operator from the operator's image, and It is characterized in that it is determined that an operation associated with a series of actions has been performed when a part different from the part is between the set operation determination surface and the display means.

  According to a seventh aspect of the present invention, in the operation input device according to the sixth aspect, the operation determination surface is formed between the display means and the operator.

  The invention according to claim 8 is the range indicated by the operator's predetermined series of operations read by the imaging means in the operation input device according to claim 2 or 3, wherein The range that does not cover the area defined by connecting the viewpoint and each vertex of the display surface of the display means and does not overlap the position determination plane is a range that can be determined by a part different from the part of the operator. The apparatus further comprises a determination range setting means for setting a certain determination range in advance, and the operation determination means extracts a part different from the part of the operator from the operator image and sets a part different from the part of the operator. When it is within the determined determination range, it is determined that a predetermined operation associated with the series of actions is performed based on a series of actions by a part different from a part of the operator.

  According to a ninth aspect of the present invention, in the operation input device according to the eighth aspect, the determination range setting means is configured as a determination range based on a predetermined series of operations of the operator read by the imaging means. The operation determination surface associated with the display surface of the display unit is previously set so as not to cover the exclusion area determined by connecting the operator's viewpoint and each vertex of the display surface of the display unit and to overlap the control range. The operation determining means extracts a part different from the operator part from the operator image, and between the operation determination surface on which the part different from the operator part is set and the display means. In some cases, it is determined that an operation associated with a series of operations has been performed.

  According to a tenth aspect of the present invention, in the operation input device according to the ninth aspect, the operation determination surface is formed between the display means and the operator.

  According to an eleventh aspect of the present invention, in the operation input device according to the second or third aspect, the distance is calculated from a positional relationship between a part of the operator on the operator side and the position determining surface of the position determining surface. The sign is changed according to the distance to indicate the position to move the sign.

  According to a twelfth aspect of the present invention, in the operation input device according to the sixth, seventh, ninth, or tenth aspect, the determination range setting means includes two or more determination ranges that are determined based on a positional relationship with the operation determination plane as a determination range. When a part different from the part of the operator is read in any region of the two or more virtual operation hierarchies, the operation determining means sets the operation previously assigned to the virtual operation hierarchies. The content of the operation is determined based on a series of actions by a type and a part different from a part of the operator in the virtual operation hierarchy.

  The invention according to claim 13 is the operation input device according to any one of claims 1 to 3, further comprising voice recognition means for recognizing an operator's voice, wherein the operation determination means is recognized by the voice means. When the voice is a predetermined voice, it is determined that an operation corresponding to the predetermined voice is performed.

  According to a fourteenth aspect of the present invention, in the operation input device according to any one of the first to thirteenth aspects, the imaging means is constituted by one imaging means of the display means.

  According to a fifteenth aspect of the present invention, in the operation input device according to any one of the first to thirteenth aspects, the imaging means is constituted by imaging means respectively arranged on the left and right sides of the display means, each of which is part of the operator. It is characterized in that images of a part different from the above and a part of the operator are taken.

  According to a sixteenth aspect of the present invention, in the operation input device according to any one of the first to fifteenth aspects, a part of the operator is a hand of the operator.

  The invention according to claim 17 is the operation input device according to any one of claims 4 to 10, wherein a part of the operator is a hand of the operator, and a part different from the part of the operator is It is the other hand of the operator.

  The invention according to claim 18 is a display means for displaying an image viewed by the operator, and is arranged in the vicinity of the display means for the operator of the display means, and generates image data of the operator who views the display means. A method of performing operation determination by an operation input device including an imaging unit, including a range indicated by a predetermined series of operations of an operator read by the imaging unit, and a part of the operator A control range setting step for presetting a control range that is a range in which the marking can be controlled and does not cover an area determined by connecting the viewpoint of the operator and each vertex of the display surface of the display means; When a part of the operator is extracted from the operator's image and the part of the operator is within the set control range, it is displayed on the display means according to the positional relationship between the part of the operator and the display means. Controls that move a given sign And an operation determination step for determining whether or not an operation related to the position of the display means on the display surface has been performed, and if it is determined that the operation has been performed in the operation determination step, a predetermined signal is And a signal output step of outputting.

  According to a nineteenth aspect of the present invention, there is provided display means for displaying an image viewed by the operator, and image data of the operator who is arranged in the vicinity of the display means for the operator of the display means and who views the display means. A program for causing an operation input device including an imaging unit to execute an operation determination method, the operation determination method including a range indicated by a predetermined series of operations of an operator read by the imaging unit, Control in which a part of the operator can control the labeling and sets a control range in advance that does not cover the region defined by connecting the operator's viewpoint and each vertex of the display surface of the display means When a part of the operator is extracted from the range setting step and the operator's image and the part of the operator is within the set control range, depending on the positional relationship between the part of the operator and the display means Displayed on the display means It is determined that an operation has been performed in a control step for moving a fixed sign, an operation determination step for determining whether or not an operation related to the position of the display means on the display surface has been performed, and an operation determination step And a signal output step for outputting a predetermined signal.

It is a figure which shows an example of the operation input system of this embodiment. It is a block diagram which shows typically the relationship with the computer of the operation input system of this embodiment. It is a block diagram which shows an example of the functional module of the program processed within CPU of the computer of this embodiment. It is a flowchart of the process of this embodiment. It is a figure for demonstrating the left and right virtual operation surface concerning one Embodiment of this invention. It is a flowchart which shows the process which extracts the position of the designated operation surface concerning one Embodiment of this invention. It is a side view of the virtual operation surface of one embodiment of the present invention. It is a top view of the virtual operation surface of one embodiment of the present invention. It is a figure for demonstrating the function example of two right and left operation surfaces concerning one Embodiment of this invention. It is a figure for demonstrating another example of a function of two right and left operation surfaces concerning one Embodiment of this invention. It is a figure which shows an example of input operation by the operation input system concerning one Embodiment of this invention. It is a figure which shows typically the function of the screen concerning one Embodiment of this invention, and a corresponding operation surface. It is a figure which shows typically the function of the screen concerning one Embodiment of this invention, and a corresponding operation surface. It is a figure which shows the relationship between the operator's movement concerning one Embodiment of this invention, and the icon displayed on a screen. It is a figure which shows an example of the concrete display of the operation input screen concerning one Embodiment of this invention. It is a figure which shows the example of the various icons which can be used on the operation input screen concerning one Embodiment of this invention. It is a figure which shows the relationship between the operator's movement concerning one Embodiment of this invention, and the icon displayed on a screen. It is a figure which shows another example of the operation input system of this embodiment. It is a figure which shows an example of the operation attitude | position of the operation input system of this embodiment. It is a figure which shows an example of the operation input system of this embodiment.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

(First embodiment)
FIG. 1 is a diagram illustrating an example of an operation input system according to the present embodiment. The display 111 according to the present embodiment is arranged in front of the operator 102, and the operator 102 has a shape such as a finger as an operation determination target within a range set at a certain left and right position with the display 111. The operation can be performed with awareness of In the present invention, as will be described later, position determination and operation determination can be input to the system by moving the right hand 114 and the left hand 115 separately on the left and right of the display 111 as viewed from the operator 102. Here, various images for various applications normally used on a personal computer are displayed on the display 111. In addition to this, operation input is supported, that is, for example, the region of the target operator 102 is displayed on the screen. It can be displayed near the corner to display to the operator 102 what operation the system is currently recognizing.

  Here, in this embodiment, a normal personal computer display can be used as a monitor that the operator views. However, as will be described later, a three-dimensional display is used in order to improve the operability by displaying icons three-dimensionally. You can also. In this case, any three-dimensional display known in the art can also be used. Further, in the present embodiment, the position determination is performed using the right hand and the operation determination is performed using the left hand. However, the present invention is not limited to this, and the left and right can be reversed depending on the dominant hand or the preference of the operator. Further, for operation determination, operation determination or trigger can be realized by other methods without using such hand movement. For example, some switches may be provided to determine the operation after determining the position based on the position determination by hand, or a voice recognition device may be provided to trigger an operation or specify the operation content by the operator's voice. it can. In this embodiment, the operation is determined by moving the left and right hands in a certain range on the left and right sides when viewed from the display. However, the present invention is not limited to this, and the hand is placed in a range where the hand can reach naturally. It can also be used for operation determination by moving. For example, various functions such as using the lower side of the quadrangular pyramid formed by the viewpoint 501 and the display surface 520 shown in FIG. 1 as a space key can be set.

  In this embodiment, as shown in FIG. 5, a camera at any one of the video cameras 511 to 514 is used to acquire an image. Here, although four cameras are shown in FIG. 5, any one or two or more can be provided. When using multiple cameras, the left and right cameras can be used to capture the movements of the left and right hands separately to improve the accuracy of the image. Further, any imaging means known in this technical field such as an infrared camera can be used as the camera, and it is possible to select anywhere near the monitor as an installation place other than the four positions as shown in FIG. . Therefore, the camera 511 or the like may be of any type as long as it can capture the target image, for example, an RGB binocular stereo arrangement method or a monocular three-dimensional measurement camera such as the TOF method. The method is not limited, but a camera that can acquire a wide-angle range is preferable in terms of installation layout.

  In addition, an inexpensive web camera or the like that has been recently used in personal computers can also be used. Instead of an expensive camera, using an inexpensive web camera, each camera is installed at both ends of the target screen like the cameras 512 and 514, and left and right operation sensing is performed on each of the left and right cameras as described above. Sensing accuracy can also be improved by the method of taking charge independently. In this case, it is difficult to obtain the depth information of the target hand, but since two-dimensional image information can be obtained, it is possible to move the cursor with the dominant arm (image recognition of the fingertip), and to recognize the contour image of the gesture of the other arm. By a method known in this technical field such as decision making, a PC operation with two stages of operation becomes possible.

  Next, the operation input method of the present embodiment will be described more specifically with reference to FIG. FIG. 20 is a diagram illustrating an example of the operation input system according to the present embodiment. Referring to FIG. 20, the operation of the present embodiment is set as an operation target only when the hand is moved in the ranges 2001 and 2002 divided into the left and right with the quadrangular pyramid connecting the above-described viewpoint 501 and the four corners of the operation target screen 520. Therefore, even if the operator moves his / her hand in the operation input operation, the screen displayed on the display 111 is not blocked, and visibility during normal personal computer operation can be ensured. Similarly, since the left and right movement ranges 2001 and 2002 are divided and arranged by the above-described quadrangular pyramids, it is possible to accurately determine the movement in each range from the left hand and right hand of the operator. In conventional normal gesture recognition, misrecognition often occurs due to the intersection of the operator's arms or the overlap of the body and arms. It becomes possible to restrict and distinguish the operation areas of the left and right hands divided by the area, and a stable operation can be realized.

  In this embodiment, either the left or right hand is used only for the XY coordinate pointing or focus (equivalent to cursor movement by a conventional mouse) function (virtual trackpad function), and the other hand is determined for the focus. Used only for functions that accept actions, such as click, double-click, and drag triggers. In this way, it is possible to overwhelmly reduce input mistakes and misrecognitions frequently seen in gesture operations by assigning the functions to the left and right completely and judging the operations in two stages. Specific gestures include opening the palm (Janken par) and clicking and double-clicking, raising the index finger and clicking, and double-clicking with two fingers. Can do.

(System configuration of this embodiment)
FIG. 2 is a block diagram schematically showing the structure of the computer 110 of the operation input device of this embodiment. A video camera 511 or the like that is attached to the display 111 and captures the operator 102 or the like is connected to the computer 110, and the captured image is taken into the computer 110. The CPU 210 extracts an image of right and left hands that is a part of the operator, which is a feature of the present embodiment, or a part different from a part of the operation place, from the CPU 210. The computer 110 generally includes a CPU 210, executes a program stored in the ROM 211 or the like on the RAM 212, and outputs a processing result based on an image input from the operation input device to the display 111 or the like. In the present embodiment, the display 111 can output various images provided by various applications that the operator wants to experience, in addition to the icons displayed by the operation of the above-described operator's hand, As will be described later, information that assists operation input is also displayed.

  FIG. 3 is a block diagram illustrating an example of functional modules of a program processed in the CPU 210 of the computer 110 according to the present embodiment. As shown in FIG. 3, the processing in this system is executed by an image reading unit 301, an image extraction unit 302, an image position calculation unit 303, an operation determination unit 304, and a sign control unit 305. In this embodiment, processing from receiving an image from the video camera 511 to outputting data is executed by these five modules. However, the present invention is not limited to this, and other modules can be used. It can be processed with few modules.

  In the present embodiment, a 3D display capable of stereoscopic display is used as the display 111, and an icon is displayed so that the operation surface appears to exist at the set or adjusted position as viewed from the operator 102. it can. As a result, the operator 102 can recognize an icon or the like as a 3D image as if it exists, and can perform an operation on the recognized 3D image using a hand or a finger. The operator 102 looks at an image and an icon displayed on the display 111 with the right eye and the left eye, and the icon or the like is composed of two images including parallax, for example, in order to display the icon in three dimensions. Further, when viewed with the left eye, it is possible to form a surface that is stereoscopically displayed from the display 111 so as to be in front of the operator. Note that the position of the three-dimensionally displayed surface may be displayed at a slightly different position from the position where an icon or the like is actually formed as an operator's operational feeling, but it is not always necessary to match.

(Operation range setting)
In the present embodiment, as a premise for performing the operation range setting process, assuming that the initial setting known in the art, for example, the operation input device of the present embodiment is newly installed, a video camera that is used as a preliminary preparation It is necessary to input information such as the distortion of the used lens 511 and the distance between the display 111 and the lens to the apparatus. Further, the threshold setting or the like is adjusted in advance. When the initial setting of the system is completed, the processing of the present embodiment is performed. With reference to FIG. 1, processing for setting the left and right operation ranges by the operator's gesture according to the present embodiment will be described. As in the example shown in FIG. 1, the position and size of the operation range are determined by the hands 114 and 115 of the operator's image taken by the camera. Specifically, as shown in FIG. 1, a certain range including the position indicated by the operator with the hands 114 and 115 is set as an effective operation range. As examples of the operation range, ranges 2001 and 2002 are shown in FIG. Here, since the left and right effective ranges are divided and arranged by the above-described quadrangular pyramids, a portion where the set range covers the above-described quadrangular pyramids is not regarded as an effective operation, and is therefore excluded from the effective operation range.

  In the present embodiment, such an effective operation range setting process is started after the system is turned on or by a specific operation, for example, an instruction by a remote controller, but is not limited thereto, and is known in the art. It can be started with any method and timing.

  Further, in the present embodiment, gesture input with both elbows as shown in FIG. 19 is a general form in normal PC operation. Therefore, in setting the operation range, such an operation posture is taken into consideration. Thus, it is useful to set the range to a position that fits the operator's body. In addition, according to such an operation posture with both elbows, the muscle load on both shoulders due to the weight of both arms is considerably reduced compared to the conventional mouse and keyboard in PC operation, and the minimum necessary Since only a small gesture operation is required, the physical burden is greatly reduced, and the mouse and keyboard can be replaced with a single camera, which is a great advantage. In addition, by placing both elbows on the desk surface, it is possible to always ensure a stable gesture position even when the cursor is moved (focused) by the right hand 114 or when the operation is determined by the left hand 115.

  Furthermore, as shown in FIG. 18, the desk can be set to an effective operating range. In other words, the icon can be moved by moving the desk with the right hand 114, and the left hand 115 is usually floated on the desk, and the operation can be determined by touching the desk.

(Operation determination process of this embodiment)
In this embodiment, referring to FIGS. 5 and 20, the positions of the hands and fingers 114 and 115 that are part of the operator 102 photographed by the video camera 511 and the like are determined, and the relative positions of these positions and the display are determined. The icon on the display is moved or determined as a certain operation based on the correct position or shape. First, data captured by the video camera 511 or the like is read by the image reading unit 301, and an image of the operator's hand is extracted from the data by the image extracting unit 302.

  As a result of such preparation, the positions and sizes of the effective operation ranges 2001 and 2002 that have been set in advance by the above-described operation range setting and stored in the storage means are read out. Here, the operation start position of the right hand is appropriately determined, and an icon or the like is displayed on the display control unit 305 at a position on the display 111 corresponding to the position of the right hand 114 by any method known in the art. And controlled to move with the movement of the hand. Therefore, the operator can perform an operation such as moving the icon displayed on the display surface by accurately moving a finger or hand within the effective operation range 2001 in accordance with the movement of the displayed icon.

  The operation determination unit 304 determines that a predetermined operation has been performed based on the shape and movement of the left hand 115 within the operation range 2002 of the operator 102, and relates to the shape or movement of the left hand based on the position of the right hand 114. Along with a predetermined operation content, the system is notified that the operation has been executed.

  Here, what kind of shape and movement corresponds to what kind of operation can be determined independently by the system, can be determined by adopting any method known in this technical field, or It can also be determined that the position has been touched by the operator, for example, that an icon has been selected or a button has been pressed. The result of the determination is executed by the computer 110 assuming that such an operation has been input. The determination of the operation content is not limited to the method described here, and any method known in the present embodiment can be used. Although a specific determination method has also been omitted, generally, the shape and movement of the operator's body, such as a predetermined gesture, and the operation content meaning the same are stored in a database or the like, and after image extraction Then, this database is accessed to determine the operation content. Also in this case, of course, it is possible to improve the determination accuracy by using an image recognition technique, artificial intelligence, or the like by a method known in the technical field. Here, if the hand is not in the effective operation range 2001, 2002, it is not determined as an operation. For example, when the right hand is out of the effective range 2001, the icon stops at that position or returns to the initial position. If the left hand is out of the effective range 2002, it is not determined that the operation is performed even if the shape and movement are predetermined.

  As described above, according to the present embodiment, the field of view of the display screen, that is, the left and right operation ranges 2001 and 2002 that are clearly separated by the above-described quadrangular pyramids are defined, and different functions, for example, movement or selection of icons, By performing the operation separately from the operation contents such as click and drag, it is possible to perform an accurate operation input without using a keyboard on a personal computer or the like without burdening the operator, despite the input of only the gesture.

(Second Embodiment)
In the first embodiment described above, the operation input by sharing the function between the operator and the display on the left and right and moving the hand has been described. In this embodiment, basically the same system configuration is used, Furthermore, highly accurate operation input is enabled. That is, in this embodiment, the concept of a virtual operation surface is introduced, and the operation trigger and the operation content are determined using the movement of the hand with respect to the operation surface and the region formed by the operation surface. For this reason, in this embodiment, by using a three-dimensional camera instead of the camera of the first embodiment, the operator himself / herself or the position of the hand or finger in the z direction is also acquired, and the hand on the virtual operation surface is obtained. It is possible to accurately acquire the position of and to determine whether or not the operation surface has been crossed. Accordingly, the system configuration is the same as that of the first embodiment except that a three-dimensional camera is used, and therefore, overlapping points are omitted. The process using the virtual operation surface of the present embodiment can be applied to both the left and right operations. However, the process of the first embodiment described above is applied to either the right or left operation. The processing of this embodiment can also be applied to the other side, and can be selected as appropriate according to the system configuration and usage status.

  FIG. 5 is a diagram for explaining the left and right virtual operation surfaces according to the embodiment of the present invention. Of the video cameras 511 to 514 shown in FIG. 5, when actually used in the system, basically one three-dimensional camera is sufficient. In this embodiment, a three-dimensional (or 3D) camera is used as the video camera 511 so that a stereoscopic image including the operator can be created. Of course, a normal two-dimensional camera can be arranged on the left and right. For example, both the video cameras 512 and 514 may be provided, and in this case, a two-dimensional camera may be used as the camera, and the two video cameras 512 and 514 may generate a stereoscopic image.

  7 and 8 are a view of the operation surface seen from the side and a view seen from the top. As can be understood with reference to FIGS. 7 and 8, the operation surfaces 104 and 105 are arranged on the left and right sides so as to avoid a quadrangular pyramid connecting the operator's viewpoint 501 and the four corners of the operation target screen 520. Here, the positions of the position determination surface 104 and the operation determination surface 105 are set by an operator with both hands extended at an optimal position for the input operation in accordance with the use environment in an initial setting step of the PC installation stage described later. At this time, for example, the camera side supplements the position information of the upper body of the operator's head and shoulders, the position of both eyes, etc., so that the operator facing the desk can easily operate both hands with the elbows on. The arrangement is set at the left and right contrast positions, but is not limited to this, and can be optimally set for the operating environment of the operator. Note that the position determination surface 104 and the operation determination surface 105 shown in FIGS. 7 and 8 are arranged on the left and right sides so as to avoid the above-described quadrangular pyramids. This is to prevent the hands 114 and 115 from being obstructed. Therefore, as long as this purpose can be achieved, the position determination surface 104 and the operation determination surface 105 need only be arranged substantially on the left and right sides. There is no problem even if it takes.

  9 and 10 are diagrams for explaining examples of typical functions of the left and right operation surfaces according to the embodiment of the present invention. 9 and 10, the position determination surface 104 has a function of pointing or focusing. For example, as shown in FIG. 12, an area 1202 corresponding to the position determination surface 104 is displayed at an appropriate place on the display 111. It is possible to change the color of the corresponding part that is focused, or to display an icon (not shown). In this manner, by changing the image on the display 111 in conjunction with the movement of the hand 114 on the position determination surface 104, the operator can focus with the hand 114 while looking at the display 111 intuitively. Become.

  Further, the operation determination surface 105 has a certain area 901 as shown in FIG. 9, and it is determined that the operation is started when the hand 115 exceeds the operation determination surface 105, and the shape of the hand 115 in the area 901 The operation content can be determined based on the gesture. That is, the operation surface of the present embodiment is set based on the height and arm length of the operator 102, or body dimension information such as the height and shoulder width. If the user wants to perform various operations, the user can project the hand 115 to the operation area 901 in front of the operation determination surface 105 to show a gesture.

  Alternatively, as shown in FIG. 10, for example, if the hand 115 has penetrated only the first layer 1001 as the second layer 902, 1001 as the operation determination surface, it is determined that the click has been reached. A click can be determined. In the present embodiment, an example having two hierarchies as an operation determination surface has been described. However, the present invention is not limited to this, and three hierarchies or higher hierarchies can be set to correspond to various functions.

  Further, in the example as shown in FIG. 12, for example, the operation input system of the present embodiment has a simple keyboard function, but has a normal keyboard function as shown in FIG. It can also be done. That is, by assigning the right half of the normal keyboard to the right operation surface 104 and the left half of the normal keyboard to the left operation surface, the operator can focus while looking at the keyboard layouts 1202 and 1301 on the display 111, The operation can be determined. In this case, the operation can be determined as an operation determination surface by setting another operation surface under an operation screen that is not normally used. Further, when an operation surface is provided below such an operation screen, a space key can be simply assigned.

(Operation side settings)
In the present embodiment, when the initial setting of the system is completed, the operation surface setting process according to the present embodiment is performed. As in the example shown in FIG. 1, the shape (including size and angle with respect to the display) of the operation surface is determined by the hands 114 and 115 of the operator's image taken by the three-dimensional camera. The positions of the operation surfaces 104 and 105 are determined based on the distance to the hands 114 and 115. Specifically, as shown in FIG. 1, it is understood that the operator may form the operation surfaces 104 and 105 at the positions indicated by the hands 114 and 115.

  Next, the operation surface extraction process will be described with reference to FIG. FIG. 6 is a flowchart showing processing for extracting the position of the designated operation surface according to the present embodiment. First, when data, that is, the operator's stereoscopic image data is input (S601), the hand shape and head are recognized (S602, S603), and it is determined whether or not an operation surface is to be formed (S604). ). Specifically, for example, when a gesture of holding the elbow on the desk is maintained for a certain period of time, it is determined that the operation surface setting processing is performed, and the operation surface is acquired at the position indicated by the operator by hand. An operation surface is formed by extracting from the three-dimensional image (S605), and when necessary, an operation region is set based on measurement values of the operator's body, for example, arm length, torso and head positions (S606). ) The data is output and the process ends (S609). Here, in the present embodiment, the movement of holding the hand on the desk on the desk is determined to be a setting instruction for the operation surface by resting for a certain period of time, but this is not a limitation, and instructions are given by various gestures, remote control operations, etc. The instruction can be determined by any method known in the art. In the present embodiment, the operation surface is basically formed in the shape and position formed by the hand. However, the operation surface can be adjusted according to the physical characteristics of the operator, the position of the display, the usage situation, and the like.

  If it is determined from the recognition of the head or the gesture of the hand that the operation surface is not set, it is determined whether or not the operation surface is already set (S607). If the operation surface is set, normal operation is performed. And the position of the hand in the operation area is acquired (S608). As described above, the operation surface setting process based on the operator's instruction is performed.

  Here, the shape of the operation surface is a vertically standing rectangle with reference to FIG. 1 and the like, but is not limited to this, and includes various shapes (sizes, angles with respect to the display) according to the operation mode of the operator. ) Operation surface can be formed. For example, as shown in FIG. 18, a position determination surface 1801 and an operation determination surface 1802 can be arranged on a desk, and an operation can be input as if using a keyboard or a mouse. Of course, in order to achieve such an operation surface arrangement, the operation surface may be set by placing a hand on the desk or setting a predetermined gesture when setting the operation surface.

(Operation determination process of this embodiment)
In the present embodiment, the positions of the hands and fingers 114 and 115 that are part of the operator 102 photographed by the video camera 511 and the like are determined, and the positional relationship between the operation surfaces 104 and 105 and the fingers 114 and 115 of the operator 102 is determined. The process which calculates is performed. This process will be described below with reference to FIG.

  FIG. 4 is a flowchart of the operation determination process of the present embodiment. First, data captured by the video camera 511 or the like is read by the image reading unit 301 (S401), and an image of the operator is extracted from the data by the image extracting unit 302 (S402).

  As a result of such preparation, the position and shape of the operation surface preset in the above-described operation surface setting and stored in the storage means are read (S403). As described above, when the shape and position of the operation surface are read out, an icon or the like is displayed and controlled at a position on the display 111 corresponding to the position of the hand 114 on the operation surface 104 by any method known in the art. Displayed by the unit 305 and controlled to move with the movement of the hand. Therefore, the operator can accurately touch the position on the operation surface corresponding to the displayed icon with a finger or a hand, so that the operator performs an operation such as touching the icon displayed on the display surface. Can do.

  The operation determination unit 304 uses a relative positional relationship between the operation surface formed by the operation input system and the operator 102 (S404), and a part of the operator 102 is in front of the operation surface viewed from the video camera 511 or the like. , It is determined that the operation has been started (S405), the position of the hand 114 on the position determination surface 104 is determined, the icon is moved to the corresponding position on the display unit, and the operation determination surface 105 is displayed. Based on the shape of the hand 115 (open hand or two fingers etc.) and movement as a reference, it is determined which operation the shape and movement are assumed in advance (S406). The result of the determination is executed by the computer 110 as if there was an input of such an operation (S407), and if the hand is not released from the operation surface to the near side, it is determined that the operation has not been performed, and the process ends. (S408). The determination of the operation content is not limited to the method described here, and any method known in the present embodiment can be used.

(Operation support on the front side of the operation surface)
In the present embodiment, the operation content is controlled by operating as if there is an input device that can substitute for the mouse on the basis of the operation surface virtually set in the space by the operator. Although it is to be judged reliably, until the hand or finger that is part of the operator reaches the operation surface, that is, after the operator starts moving the hand or finger to perform some operation, By supporting the operation until the button is pressed, the operation input can be performed more easily and with higher accuracy. This function is particularly effective for the pointing or focusing function, but it is often effective for operation determination using the operation determination surface. Explained as an example.

  Basically, the principle of such operation support is a visual indication of what operation the operator is trying to perform on the display 111 in accordance with the movement of the position of the operator, for example, the hand or finger relative to the operation surface. By displaying on the screen, the operator is guided to enable accurate operation input.

  This point will be described with reference to FIG. 14. In this embodiment, the operator sets the operation surface 104 in advance to a position suitable for operation on the operation surface. As shown in FIG. 14, when an operator wants to perform any operation on the system, the position of the hand or finger 114 changes in this example by moving the arm 2401 back and forth with respect to the display 111. When the state is displayed on the display 111, when the protruding finger 114 reaches a certain position, a certain process is performed as a system, for example, an item instructed on the screen of the display 111 is executed. In the example of FIG. 14, the size of the icon changes depending on the position (depth) of the finger 114 with respect to the operation surface 104. It is possible to recognize that focus is on a certain place. Then, the operation is confirmed at the position where the icon becomes the smallest, and processing corresponding to that is executed.

  FIG. 15 shows how the icon changes on the screen 2501 of the display 111 as a result of the above operation. Referring to FIG. 15, for example, a television program guide is displayed on a screen 2501 of the display 111, and an operation regarding a certain program is about to be performed. In such a state, for example, when the operator wants to select the “change setting” menu button, the operator tries to select by pushing the finger 114 toward the display 111 as described above. In this embodiment, an icon 2503 is displayed on the screen 2501 when the finger 114 approaches a certain distance from the television program guide on the operation surface. Since this finger is still far away, a relatively large icon on the right side of the icon shown in FIG. 14 is displayed. When the operator further extends the arm 2401, this icon approaches the target selection item “setting change” and becomes smaller and becomes a special icon when the icon 2502 has a certain size. Here, when it is determined that any operation is performed by the operation determination surface 105 of the present embodiment, the icon can be changed to a different shape or color to indicate that the operation has been determined.

  As described above, in the example of FIG. 15, by changing the size of the icon displayed on the screen 2501 according to the position of the finger 114, the operator can see how his / her movement is recognized by the system. It can be grasped, the position of the operation surface can be recognized more easily, and operations such as menu selection can be performed. Here, the whole operator including the finger 114 and the arm 2401 and the position and size of each part can be extracted by using a three-dimensional camera as in the whole image of the operator. Thereby, since the object in the screen can be grasped including the depth, the distance and the positional relationship with the operation surface can be calculated based on such information. However, since any method known in this technical field can be used for the three-dimensional camera used in the present embodiment, position extraction, distance calculation, and the like, description thereof is omitted here.

  The icons displayed on the screen are circular and change in size according to the operation of the operator. However, the present invention is not limited to this, and various forms of icons can be used as shown in FIG. it can. That is, referring to FIG. 16, (1) is an icon in the form of a finger, which is made smaller as the operation surface is approached as in the above example. Although (2) is circular and gradually smaller, it is indicated that the input has changed to a special shape when input or selection is confirmed by the operation determination surface. In the case of this icon and other icons, the color of the icon can be changed instead of or in combination with the change in shape (including size and angle with respect to the display). For example, by changing from a cold color system to a warm color system such as blue, green, yellow, and red, the operator can intuitively recognize that the operation is focused and determined. (3) is a shape like X, and when it is far away, it is not only large but also blurred. As it gets closer, the size of the icon becomes smaller and the blur disappears and becomes a sharp shape. In (4), the size of the entire icon is not changed, and the figure drawn therein is recognized to change its shape and become focused. In this case, the color of the figure can also be changed. (5) shown in FIG. 16 also changes the shape. In FIG. 23, the shape and color of the icon change according to the movement of the finger, and when the operation surface is exceeded, the shape or color is changed or flashed as shown in a column 2601 at that moment. Thus, the operator can recognize that the operation is determined. In addition, although not shown in the figure, as other icon changes, a change that is transparent at first and becomes opaque as the finger approaches the operation surface is also effective. Although not shown in the drawing, a stereoscopic display function of the display is separately provided as will be described later, and an image can be created so that the icon pops out from the screen when the focus is focused, and the focus status can be notified to the operator. .

  Here, in the case of changing the color and the darkness without changing the shape so much among the variations of the icons, as shown in FIG. 16, if the icon 114 does not move so much and the finger 114 is brought closer, the color becomes a warm color system. The input can be confirmed by becoming darker or darker.

  In the above example, the icon is displayed at the position on the operation surface to check the operation determination status, and the color and shape are changed according to the operation of the operator. If the icon is fixed, even if you do not bother to display the icon, the position that the finger points to is determined by the closest item button on the displayed menu, and the finger movement, especially the distance from the operation surface The position of the operation surface can be made easier to understand and the operation input can be facilitated by changing the color or darkness of filling the item button pointed to.

  As described above, the principle of the present embodiment has been described for the case where the operation surface is formed substantially perpendicular to the front surface in the horizontal direction of the operator. However, this principle is based on the positional relationship and shape between the operator and the display. Various arrangements and configurations are possible without being affected.

  In order to achieve such an object, the invention according to claim 1 is an operation input device, which is arranged in the vicinity of a display means for displaying an image to be viewed by an operator and a display means for the operator. Including an imaging unit that generates image data of the operator and a range indicated by a predetermined series of operations of the operator read by the imaging unit, and a part of the operator controls the marking. A control range setting means for preliminarily setting a control range that is a range that does not cover an area defined by connecting the viewpoint of the operator and each vertex of the display surface of the display means, and a display surface of the display means of the indication A determination range setting means for presetting a determination range that is a range for determining whether or not an operation related to the upper position has been performed, and extracting a part of the operator from the operator's image, If the part is within the set control range A control unit that moves a predetermined mark displayed on the display unit according to a positional relationship between a part of the operator and the display unit, and a part that is different from the part of the operator is extracted from the operator image. When a part different from the part of the operator becomes a predetermined mode within the determination range, an operation related to the position on the display surface of the display means of the sign is performed based on a series of actions of the part of the operator. It is characterized by comprising an operation determining means for determining that the operation has been performed, and a signal output means for outputting a predetermined signal when the operation determining means determines that an operation has been performed.

  According to a second aspect of the present invention, in the operation input device according to the first aspect, the determination range setting means sets the constant surface range so as not to overlap the control range.

  According to a third aspect of the present invention, in the operation input device according to the second aspect, the control range and the determination range are set to either the left or right of the operator.

  According to a fourth aspect of the present invention, in the operation input device according to any one of the first to third aspects, the determination range setting means is based on a predetermined series of operations of the operator read by the imaging means. The determination range is set.

  According to a fifth aspect of the present invention, in the operation input device according to any one of the second to fourth aspects, the imaging unit is a three-dimensional imaging unit that generates stereoscopic image data of the operator, and the control range setting unit is Based on a predetermined series of operations of the operator read by the imaging unit, the control range does not overlap with a region defined by connecting the operator's viewpoint and each vertex of the display surface of the display unit. In addition, a position determination surface associated with the display surface of the display means is set in advance, and the control means displays a sign at a position on the display surface corresponding to a position on the position determination surface of a part of the operator. It is characterized by moving.

  According to a sixth aspect of the present invention, in the operation input device according to the fifth aspect, the position determination surface is formed between the display means and the operator.

  According to a seventh aspect of the present invention, in the operation input device according to any of the first to sixth aspects, the imaging unit is a three-dimensional imaging unit that generates stereoscopic image data of the operator, and the determination range setting unit is Based on a series of predetermined operations of the operator read by the imaging means, the exclusion range determined by connecting the operator's viewpoint and each vertex of the display surface of the display means within the determination range is not covered. In addition, an operation determination surface associated with the display surface of the display unit is set in advance, and the operation determination unit extracts a part different from the part of the operator from the operator image, It is characterized in that it is determined that an operation associated with a series of operations has been performed if the different part is between the set operation determination surface and the display means.

  The invention according to claim 8 is the operation input device according to claim 7, wherein the operation determination surface is formed between the display means and the operator.

  According to a ninth aspect of the present invention, in the operation input device according to the fifth or sixth aspect, the distance is calculated from the positional relationship between a part of the operator on the operator side and the position determining surface of the position determining surface. The sign is changed according to the distance to indicate the position to move the sign.

  According to a tenth aspect of the present invention, in the operation input device according to the seventh or eighth aspect, the determination range setting means includes two or more virtual operation hierarchies determined as a determination range based on a positional relationship with the operation determination plane. The operation determination means reads an operation type and a virtual operation previously assigned to the virtual operation hierarchy when a part different from the operator's part is read in any region of the two or more virtual operation hierarchies. The content of the operation is determined based on a series of actions by a part different from the part of the operator in the hierarchy.

  According to an eleventh aspect of the present invention, in the operation input device according to any one of the first to tenth aspects, the imaging means is constituted by one imaging means of the display means.

  According to a twelfth aspect of the present invention, in the operation input device according to any one of the first to tenth aspects of the present invention, the imaging means is constituted by imaging means respectively arranged on the left and right sides of the display means, each of which is part of the operator. It is characterized in that images of a part different from the above and a part of the operator are taken.

  According to a thirteenth aspect of the present invention, in the operation input device according to any one of the first to twelfth aspects, the part of the operator is the hand of the operator.

  The invention described in claim 14 is the operation input device according to any one of claims 1 to 12, wherein a part of the operator is a hand of the operator, and a part different from the part of the operator is It is the other hand of the operator.

  According to the fifteenth aspect of the present invention, there is provided display means for displaying an image viewed by the operator, and image data of the operator who is disposed in the vicinity of the display means for the operator of the display means and who views the display means. A method for performing an operation determination by an operation input device including an imaging unit, including a range indicated by a predetermined series of operations of an operator read by the imaging unit, wherein a part of the operator is labeled A control range setting step for presetting a control range that is a range that can be controlled and that does not cover a region defined by connecting the operator's viewpoint and each vertex of the display surface of the display means; A determination range setting step for presetting a determination range, which is a range for determining whether or not an operation related to the position on the display surface of the display means has been performed, and extracting a part of the operator from the operator's image One of the operators Is within the set control range, a control step of moving a predetermined mark displayed on the display unit according to the positional relationship between a part of the operator and the display unit, and the operator's image from the operator's image When a part that is different from the part is extracted, and a part that is different from the part of the operator is set in a predetermined mode, the indication is displayed based on a series of actions of the part of the operator. An operation determining step for determining that an operation related to a position on the display surface of the means has been performed, and a signal output step for outputting a predetermined signal when it is determined that an operation has been performed in the operation determining step. It is characterized by.

  According to a sixteenth aspect of the present invention, there is provided display means for displaying an image viewed by an operator, and image data of an operator who is disposed in the vicinity of the display means for the operator of the display means and who views the display means. A program for causing an operation input device including an imaging unit to execute an operation determination method, the operation determination method including a range indicated by a predetermined series of operations of an operator read by the imaging unit, Control in which a part of the operator can control the labeling and sets a control range in advance that does not cover the region defined by connecting the operator's viewpoint and each vertex of the display surface of the display means A range setting step, a determination range setting step for presetting a determination range that is a range for determining whether or not an operation related to the position of the display means on the display surface has been performed, and an operator's image based on the operator's image When a part is extracted and a part of the operator is within the set control range, a predetermined sign displayed on the display means is moved according to the positional relationship between the part of the operator and the display means. When a part different from the operator's part is extracted from the control step and the operator's image, and a part different from the part of the operator is set in a predetermined range, the operator's image An operation determination step for determining that an operation related to the position on the display surface of the sign display means has been performed based on a series of operations of the unit, and an operation related to the position on the display surface of the sign display means. An operation determination step for determining whether or not the operation has been performed, and a signal output step for outputting a predetermined signal when it is determined that the operation has been performed in the operation determination step.

Claims (19)

Display means for displaying an image viewed by the operator;
An imaging unit that is arranged in the vicinity of the display unit toward the operator and generates image data of the operator;
Including a range indicated by a predetermined series of operations of the operator read by the imaging means, a range in which a part of the operator can control the marking, and the viewpoint of the operator And a control range setting means for presetting a control range that is a range that does not cover a region defined by connecting each vertex of the display surface of the display means;
When a part of the operator is extracted from the image of the operator and the part of the operator is within the set control range, depending on the positional relationship between the part of the operator and the display means Control means for moving a predetermined sign displayed on the display means,
Operation determining means for determining whether or not an operation related to the position of the display on the display surface of the display means has been performed;
An operation input device comprising: signal output means for outputting a predetermined signal when the operation determination means determines that an operation has been performed. The imaging means is a three-dimensional imaging means for generating the operator's stereoscopic image data,
The control range setting means determines the operator's viewpoint and each vertex of the display surface of the display means as the control range based on a predetermined series of operations of the operator read by the imaging means. In order not to overlap the area determined by tying, a position determination surface associated with the display surface of the display means is set in advance,
The operation according to claim 1, wherein the control unit moves the sign to a position on the display surface corresponding to a position on the set position determination surface of a part of the operator. Input device.   The operation input device according to claim 2, wherein the position determination surface is formed between the display unit and the operator.   The operation determining means extracts a part different from the part of the operator from the image of the operator, and based on a series of actions by a part different from the part of the operator, The operation input device according to claim 1, wherein the operation input device determines that an associated and predetermined operation has been performed. Determination range setting for presetting a determination range that can be determined by a part different from a part of the operator based on a predetermined series of operations of the operator read by the imaging unit Further comprising means,
The operation determining means, when a part different from the part of the operator is within the set determination range, the series of operations based on a part of the series of operations different from the part of the operator. The operation input device according to claim 4, wherein the operation input device determines that a predetermined operation is performed. The imaging means is a three-dimensional imaging means for generating the operator's stereoscopic image data,
The determination range setting means determines the operator's viewpoint and each vertex of the display surface of the display means as the determination range based on a predetermined series of operations of the operator read by the imaging means. The operation determination surface associated with the display surface of the display means is set in advance so as not to be in the excluded area determined by tying and to overlap the control range,
The operation determining means extracts a part different from the part of the operator from the operator's image, and a part different from the operator's part is set to the operation determination surface and the display means. The operation input device according to claim 5, wherein the operation input device determines that an operation associated with the series of operations has been performed.   The operation input device according to claim 6, wherein the operation determination surface is formed between the display unit and the operator. It is a range indicated by a predetermined series of operations of the operator read by the imaging means, and is in an area determined by connecting the operator's viewpoint and each vertex of the display surface of the display means. And a determination range setting means for presetting a determination range that is a range that can be determined by a part different from a part of the operator, and a range that does not overlap the position determination plane.
The operation determining means extracts a part different from the part of the operator from the image of the operator, and when a part different from the part of the operator is within the set determination range, 4. The operation according to claim 2, wherein it is determined that a predetermined operation associated with the series of operations has been performed based on a series of operations by a part different from a part of the operator. Input device. The determination range setting means determines the operator's viewpoint and each vertex of the display surface of the display means as the determination range based on a predetermined series of operations of the operator read by the imaging means. The operation determination surface associated with the display surface of the display means is set in advance so as not to be in the excluded area determined by tying and to overlap the control range,
The operation determining means extracts a part different from the part of the operator from the operator's image, and a part different from the operator's part is set to the operation determination surface and the display means. The operation input device according to claim 8, wherein the operation input device determines that an operation associated with the series of operations has been performed.   The operation input device according to claim 9, wherein the operation determination surface is formed between the display unit and the operator.   A position of the position determination plane is calculated from a positional relationship between a part of the operator on the operator side and the position determination plane, and the position is changed according to the distance to indicate a position to move the label. The operation input device according to claim 2 or 3, wherein The determination range setting means sets two or more virtual operation hierarchies determined based on a positional relationship with the operation determination surface as the determination range,
When a part different from the part of the operator is read in any region of the two or more virtual operation hierarchies, the operation determining means reads the operation type assigned to the virtual operation hierarchies and the virtual 11. The operation input device according to claim 6, 7, 9 or 10, wherein the content of the operation is determined based on a series of actions by a part different from the part of the operator in the operation hierarchy. Voice recognition means for recognizing the voice of the operator,
The operation determining means determines that an operation corresponding to the predetermined sound is performed when the sound recognized by the sound means is a predetermined sound. An operation input device according to any one of the above.   The operation input device according to claim 1, wherein the imaging unit is configured by one imaging unit of the display unit.   The imaging means includes imaging means arranged on the left and right sides of the display means, respectively, and picks up images of a part different from a part of the operator and a part of the operator. The operation input device according to claim 1.   The operation input device according to claim 1, wherein a part of the operator is a hand of the operator.   The part of the operator is the hand of the operator, and a part different from the part of the operator is the other hand of the operator. An operation input device according to any one of the above. Display means for displaying an image browsed by an operator, and imaging means arranged near the display means for the operator of the display means and generating image data of an operator who views the display means A method for determining operation by an operation input device,
Including a range indicated by a predetermined series of operations of the operator read by the imaging means, a range in which a part of the operator can control the marking, and the viewpoint of the operator And a control range setting step for presetting a control range that is a range that does not cover a region defined by connecting each vertex of the display surface of the display means;
When a part of the operator is extracted from the image of the operator and the part of the operator is within the set control range, depending on the positional relationship between the part of the operator and the display means A control step of moving a predetermined sign displayed on the display means;
An operation determination step for determining whether an operation related to a position on the display surface of the display means of the indication has been performed;
An operation input method comprising: a signal output step of outputting a predetermined signal when it is determined that an operation is performed in the operation determination step. Display means for displaying an image browsed by an operator, and imaging means arranged near the display means for the operator of the display means and generating image data of an operator who views the display means A program for causing an operation input device to execute an operation determination method, wherein the operation determination method includes:
Including a range indicated by a predetermined series of operations of the operator read by the imaging means, a range in which a part of the operator can control the marking, and the viewpoint of the operator And a control range setting step for presetting a control range that is a range that does not cover a region defined by connecting each vertex of the display surface of the display means;
When a part of the operator is extracted from the image of the operator and the part of the operator is within the set control range, depending on the positional relationship between the part of the operator and the display means A control step of moving a predetermined sign displayed on the display means;
An operation determination step for determining whether an operation related to a position on the display surface of the display means of the indication has been performed;
A program comprising: a signal output step of outputting a predetermined signal when it is determined that an operation has been performed in the operation determination step.