JP2005321869A - Operation input device and operation input method - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide an "operation input device and an operation input method" capable of eliminating requirement of moving an indicator such as a finger to a determined space, and capable of enhancing using convenience for a space human interface. <P>SOLUTION: This operation input device is provided with a three-dimensional image sensor 1 for detecting a three-dimensional position of an object, a coordinate detecting part 23 for detecting coordinates of an objective point on a display screen, based on the position of the object within an imaginary pointing space, a motion detecting part 24 for detecting a three-dimensional motion of the object within the pointing space, and an operation input content determination part 25 for determining an operation content, based on the three-dimensional motion of the object or the like. A troublesome operation of of moving the indicator to the planely fixed space is eliminated by setting the pointing space operable with the indicator such as the finger in a conical shape connecting the display screen and a reference point. <P>COPYRIGHT: (C)2006,JPO&NCIPI

Description

  The present invention relates to an operation input device and an operation input method, and is particularly suitable for an operation input device that gives an input command by an operator operation to a computer device or a navigation device.

  2. Description of the Related Art Conventionally, as an operation input device that gives an input command by an operator operation to a computer device, a navigation device, etc., a remote controller (remote controller) including a keyboard, a mouse, a trackball, a joystick, etc., a touch panel, etc. are frequently used. All of these operation input devices need to be operated by actually touching them, and the operation is troublesome.

  For example, when starting up application software on a computer device and performing work, it is necessary to use a keyboard for inputting characters and a mouse and trackball for specifying positions, etc., which are very different. It was troublesome. In addition, when operating an in-vehicle navigation device, the driver must stop the vehicle for safety, release the hand from the steering wheel, and operate the remote control joystick and touch panel one by one. There wasn't.

On the other hand, a technique has been proposed in which operation input is performed by detecting the movement of a finger in space (see, for example, Patent Document 1). In the technique described in Patent Document 1, infrared light is scanned in a predetermined space corresponding to the display screen, and the infrared sensor detects that the pointing object such as a finger placed in the space is irradiated with infrared light. Then, the position on the space designated by the indicator is obtained from the infrared irradiation position at that time, the position on the display screen corresponding to the position on the space is positioned, and a predetermined operation command by the position designation is issued. Output.
JP-A-8-202480

  However, in the technique described in Patent Document 1, since the position on the space where the operation input is to be performed is specified on the infrared scanning surface, it is necessary to intentionally move an indicator such as a finger to the space surface. In particular, although the operator is not always at the same position with respect to the display screen, the space where the operation can be input is always fixed to a predetermined space corresponding to the display screen. For this reason, there is a problem that it is very difficult to use an indicator such as a finger because it is necessary to move an indicator such as a finger to a space surface at a predetermined position regardless of where the operator is.

  Also, the number of operators who use the operation input device is not limited to one, and there is a possibility that an operator with a different physique may perform an operation. The space plane is fixed at a predetermined position. However, the range that the operator can reach varies depending on the physique, and there is a problem that it is difficult for a small operator to perform operation input depending on the position of the space. In addition, since the spatial plane is defined in an absolute positional relationship with respect to the display screen, there is a problem that when the operator's position changes, the angle at which the spatial plane is desired varies, and an error occurs in operation input.

  Furthermore, in the technique described in Patent Document 1, since there is no distinction between the movement corresponding to the operation input of the pointing object and the other movements, it is determined that the operation input is all when the pointing object is put in the space. Therefore, there is a problem that operation input may be performed against the user's intention.

The present invention has been made to solve such problems, and it is an object of the present invention to eliminate the need to move an indicator such as a finger to a fixed space and improve usability.
It is another object of the present invention to allow normal operation input from anywhere regardless of the position of the operator.
It is another object of the present invention to make it difficult for errors to occur regardless of where the operator is located.
Another object of the present invention is to avoid the inconvenience that an operation input contrary to the user's intention is performed.

  In order to solve the above-described problems, the present invention detects a three-dimensional position of an object, and moves the object in a virtual pointing space formed by connecting a display screen and a predetermined reference point. A display screen corresponding to the three-dimensional movement of the object and the position of the object by calculating the coordinates of the target point on the display screen obtained by mapping the position where the object exists in the pointing space to the display screen The operation input content is determined on the basis of the coordinates of the target point.

  Further, it recognizes that the object in the pointing space is a predetermined indicator, and based on the position in the pointing space detected for the object recognized as the indicator, the three-dimensional You may make it detect the coordinate of the target point in a motion and a display screen.

  The pointing space described above is a conical virtual space formed by connecting a display screen and a predetermined reference point located away from the display screen. The predetermined reference point is, for example, an operator's The eye position is preferably set. Further, the predetermined reference point is a position corresponding to the apex of a cone having both a display screen and a plane obtained by connecting the positions of the objects when the four corners of the display screen are respectively pointed in space. The initial value may be used to move following the movement of the operator. If it can be assumed that the operator's movement is small, the initial value may be fixed.

  According to the present invention configured as described above, the pointing space that can be operated with an indicator such as a finger is set in a conical shape that connects the display screen and the reference point. There is no need to perform an operation such as moving a pointing object such as a finger. In addition, if the position of the pointing object is within the pointing space, the pointing object may be located anywhere between the display screen and the operator. Can be done. Thereby, the usability of the operation input device by the human interface in space can be improved.

  Further, according to the present invention, since the coordinates on the display screen are obtained by mapping the position where the indicator is present, an error occurs in the operation input regardless of the position of the operator with respect to the display screen. Can be minimized.

  In addition, according to the present invention, a predetermined indicator is recognized from objects detected in the pointing space, and the operation input content is determined only for this pointer. Even if an object other than the object moves, it is not determined that there has been an operation input. In addition, since it is determined that there is an operation input only when the indicator moves in a predetermined manner, the movement corresponding to the operation input of the indicator is distinguished from other movements, and the indicator is accidentally moved in the pointing space. Even if it moves to, if it is an unintended movement, it will not be determined that there has been an operation input. Thereby, the inconvenience that the operation input is performed against the user's intention can be prevented.

  In addition, when the operator's eye position is set as a predetermined reference point that defines the pointing space, when the operator moves, the pointing space also moves according to the moved eye position. As a result, no matter where the operator moves with respect to the display screen, there is no need to perform an operation such as moving a pointing object such as a finger to a fixed space surface as in the past. Operation input can be performed, and the usability of the human interface in space can be further improved.

  In addition, as a predetermined reference point for defining the pointing space, a cone having a cross section as a cross section of both the display screen and a surface obtained by connecting the four corners of the display screen when the pointing operation is performed on the space. When the position corresponding to the apex of the body is set, the pointing space is always included in the pointing space when the operator normally indicates the position in the display screen while looking at the display screen. Thereby, operation input can be performed in a more natural manner, and the usability of the human interface in the space can be further improved.

(First embodiment)
Hereinafter, an embodiment of the present invention will be described with reference to the drawings. FIG. 1 is a diagram illustrating a configuration example of an operation input device according to the first embodiment. As shown in FIG. 1, the operation input device of the first embodiment includes a microcomputer (hereinafter referred to as a microcomputer) that includes a three-dimensional image sensor 1 as a position detection unit, a CPU 2a, a ROM 2b, a RAM 2c, and the like. ) 2 and a display controller 3 as display control means for performing display control of the display device 20.

  The three-dimensional image sensor 1 detects a three-dimensional position of an object in space. For example, the distance to the object (the relative position of the object) is detected by irradiating the object with a beam and measuring the beam that is reflected and returned. As this three-dimensional image sensor 1, it is possible to apply a known technique as disclosed in, for example, US Pat. No. 6,515,740. By using the technology described in this US patent, it is possible not only to detect the three-dimensional relative position of the object with respect to the position where the three-dimensional image sensor 1 is installed, but also to detect the three-dimensional shape of the object. It is. That is, when the object has irregularities and there is a distance difference from the three-dimensional image sensor 1 on the surface of the object, the position information corresponding to the distance difference is output to thereby obtain the three-dimensional shape of the object. Can be detected.

  The CPU 2a included in the microcomputer 2 controls the entire operation input device. The ROM 2b stores various programs necessary for the operation of the operation input device. The RAM 2c temporarily stores data obtained during various processes by the CPU 2a and data obtained as a result of various processes. That is, the CPU 2a controls the operation of the operation input device while using the RAM 2c as a work memory in accordance with various programs stored in the ROM 2b.

  FIG. 2 is a block diagram showing a functional configuration of the microcomputer 2. FIG. 3 is a conceptual diagram for explaining the pointing space of the present embodiment. As shown in FIG. 2, the microcomputer 2 of the present embodiment includes a pointing space setting unit 21, a pointing object recognition unit 22, a coordinate detection unit 23, a motion detection unit 24, and an operation input content determination unit 25 as its functional configuration. ing.

  As shown in FIG. 3, the pointing space setting unit 21 connects a screen of the display device 20 (hereinafter, referred to as a display screen 30) and a predetermined reference point 31 located away from the display screen 30. A conical virtual pointing space 32 is set.

  Here, the reference point 31 is set to a position where an operator is expected in many cases. For example, when the operation input device of the present embodiment is applied to an in-vehicle navigation device, it is preferable to set the reference point 31 at one point assumed to be the position of the driver's eyes in the space before the headrest of the driver's seat. FIG. 3 shows such a state. That is, in the case of the in-vehicle navigation device, the display screen 30 is installed at the approximate center of the instrument panel, while the driver's seat is located on the right side (or the left side) of the vehicle, the pointing space 32 is slanted as shown in FIG. It becomes the shape of a cone.

  In this case, since the position of the display screen 30 and the position of the reference point 31 are known in advance (for example, stored as information in the RAM 2c or other memory (not shown)), the pointing space 32 is uniquely determined.

  The indicator recognizing unit 22 detects an object in the pointing space 32 by the three-dimensional image sensor 1 and recognizes that the object is a predetermined indicator 34. In the present embodiment, it is assumed that the indicator 34 is a rod-like protrusion. For example, the indicator 34 may be one in which only one operator's finger is raised, a writing instrument or an indicator that the operator has.

  When the 3D image sensor 1 is installed in the vicinity of the display screen 30, the indicator 34 is operated in the direction of the display screen 30, that is, the direction of the 3D image sensor 1. At this time, as described above, the three-dimensional image sensor 1 outputs position information corresponding to the distance to the object. Therefore, if the rod-shaped object is directed toward the three-dimensional image sensor 1, the rod-shaped object is output. Position information that is close only to the tip portion of is output. Therefore, it is possible to detect the portion where the close position information is output as the position where the indicator 34 is present.

  When the 3D image sensor 1 is installed on the side of the pointing space 32 between the display screen 30 and the reference point 31, the indicator 34 is operated in the lateral direction as viewed from the 3D image sensor 1. It will be. As described above, the three-dimensional image sensor 1 can output position information according to the three-dimensional shape of the object. Therefore, the pointing object recognition unit 22 detects that an object having a predetermined length or more and a predetermined thickness or less in the pointing space 32 based on the position information corresponding to the object shape output from the three-dimensional image sensor 1. It is possible to recognize that it is the indicator 34 if it exists.

  The coordinate detection unit 23 detects the coordinates of the target point 36 on the display screen 30 using the mapping relationship based on the position of the indicator 34 in the pointing space 32 detected by the three-dimensional image sensor 1. That is, the coordinate detection unit 23 recognizes and detects in the pointing space 32 using the target point 36 where the line 35 passing through the position of the reference point 31 and the position of the indicator 34 intersects the display screen 30 as the coordinates of the display screen 30. It calculates from the position of the indicator 34 to be displayed. Note that the position of the indicator 34 detected by the three-dimensional image sensor 1 is a relative position as viewed from the three-dimensional image sensor 1. However, since the position of the three-dimensional image sensor 1 itself is known in advance, the indicator 34 Can also be detected.

  The coordinate detection unit 23 outputs the detected coordinate information of the target point 36 to the display controller 3. The display controller 3 adds a predetermined mark (for example, an arrow mark) to the coordinate position of the target point 36 on the display screen 30 as necessary based on the coordinate information of the target point 36 supplied from the coordinate detection unit 23. Control to display.

  The motion detection unit 24 detects a three-dimensional motion of the indicator 34 in the pointing space 32. As described above, the three-dimensional image sensor 1 outputs the three-dimensional position information of the object in the pointing space 32 at any time, and the indicator recognizing unit 22 detects whether the object is the indicator 34 or not. Yes. Therefore, it is possible to detect a three-dimensional movement of the indicator 34 by observing a change in position information regarding the indicator 34.

  The operation input content determination unit 25 operates based on the coordinates of the target point 36 on the display screen 30 detected by the coordinate detection unit 23 and the three-dimensional movement of the indicator 34 detected by the motion detection unit 24. The operation input content by the indicator 34 made by the person is determined and a corresponding operation command is output.

  For example, as shown in FIG. 4, when the pointing object 34 moves close to the display screen 30 at a predetermined speed or more within a predetermined coordinate range of the display screen 30 in the pointing space 32, the operation input content determination unit 25 determines that the pointing start operation has been performed. When the pointing object 34 moves away from the display screen 30 at a predetermined speed or higher within a predetermined coordinate range of the display screen 30 in the pointing space 32, it is determined that the pointing release operation has been performed.

  Further, in the pointing space 32, when the pointing object 34 performs the pointing release operation within a predetermined time after performing the above-described pointing start operation, it is determined that the click operation has been performed. Further, in the pointing space 32, after the pointing object 34 performs the above-described pointing start operation, the coordinates of the target point 36 on the display screen 30 detected by the coordinate detection unit 23 move without performing the pointing release operation. When the movement is performed, it is determined that the drag operation has been performed, and the movement amount on the coordinates on the display screen 30 is detected as the drag amount.

  The operation input content determination unit 25 outputs information indicating the detected operation input content to the display controller 3. The display controller 3 controls display on the display screen 30 based on the information supplied from the operation input content determination unit 25. For example, when it is determined by the operation input content determination unit 25 that the drag operation has been performed, an object at the coordinate position of the target point 36 on the display screen 30 detected by the coordinate detection unit 23 (the position where the arrow mark is present) is displayed. The display is controlled to move in the dragged direction by the drag amount.

  Next, the operation of the operation input device configured as described above according to the present embodiment will be described. FIG. 5 is a flowchart showing the operation of the operation input device according to the present embodiment. In FIG. 5, first, a three-dimensional position of an object in space is detected by the three-dimensional image sensor 1 (step S1). Then, the indicator recognition unit 22 determines whether or not the object is in the pointing space 32 (step S2). When the object is in the pointing space 32, the indicator recognition unit 22 further determines whether or not the object is the indicator 34 (step S3). If the object is not in the pointing space 32, or if the object is in the pointing space 32 but it is not the indicator 34, the process returns to step S1.

  When the pointing object 34 in the pointing space 32 is recognized by the pointing object recognition unit 22, the motion detection unit 24 detects the three-dimensional movement of the pointing object 34 in the pointing space 32 (step S4). In addition, based on the position of the pointing object 34 in the pointing space 32 detected by the three-dimensional image sensor 1, the coordinate detection unit 23 calculates the coordinates of the target point 36 on the display screen 30 using the mapping relationship ( Step S5). Note that the order of the process of step S4 and the process of step S5 may be reversed.

  Finally, the operation input content determination unit 25 is displayed on the display screen 30 corresponding to the three-dimensional movement of the indicator 34 detected by the motion detector 24 and the position of the indicator 34 detected by the coordinate detector 23. The operation input content is determined based on the coordinates of the target point 36, and a corresponding operation command is output (step S6). That is, a click operation or a drag operation is determined based on whether or not there is a predetermined movement, and an operation command corresponding to the movement is output.

  For example, when the operation input content determination unit 25 determines that a click operation has been performed toward an object (for example, an operation button using a GUI) existing at the coordinate position of the target point 36, the process assigned to the operation button is performed. Outputs an operation command for execution. When the operation input content determination unit 25 determines that a drag operation has been performed on an object (for example, an icon) existing at the coordinate position of the target point 36, the operation input content determination unit 25 moves the object in the dragged direction by the drag amount. An operation command is output to the display controller 3. When the operation input device of the present embodiment is applied to a navigation device that displays a map, an operation command for scrolling the entire map so that the point indicated by the pointing start operation moves in the dragged direction by the drag amount is issued. You may make it output.

  As described above in detail, according to the present embodiment, the pointing object 34 such as a finger is not detected at a place fixed in a flat shape as in the prior art, but a cone-shaped pointing space. The operation content is detected in accordance with the position and movement. As a result, it is not necessary to perform the troublesome operation of moving the pointing object 34 such as a finger to the plane surface fixed in a flat shape, and it is possible to perform an operation input at any place within the pointing space 32. It becomes.

  Further, according to the present embodiment, the coordinates of the target point 36 on the display screen 30 are obtained by mapping the position where the indicator 34 is present, so that the operator can position the display screen 30 at any position. However, the inconvenience that an error occurs in the operation input can be minimized.

  Further, according to the present embodiment, since the operation input content is determined only for the objects detected as the predetermined indicator 34 among the objects detected in the pointing space 32, the inside of the pointing space 32 is determined. Even if an object other than the pointing object 34 moves, it is not determined that there has been an operation input. In addition, since the movement of the pointing object 34 corresponding to the operation input content is limited, the movement corresponding to the operation input of the pointing object 34 is distinguished from the other movements, and the pointing object 34 accidentally enters the pointing space 32. Even if it moves to, if it is an unintended movement, it will not be determined that there has been an operation input. Thereby, the inconvenience that the operation input is performed against the user's intention can be prevented.

(Second Embodiment)
Next, a second embodiment of the present invention will be described. FIG. 6 is a diagram illustrating a configuration example of the operation input device according to the second embodiment. In FIG. 6, components having the same reference numerals as those shown in FIG. 1 have the same functions, and thus redundant description is omitted here.

As shown in FIG. 6, the operation input device of the second embodiment, two three-dimensional image sensor 1 _-1 as the position detecting means, and a 1 _-2. One three-dimensional image sensor _1-1 is used to detect the three-dimensional position of the indicator 34 in the pointing space 32, as in the first embodiment. The other 3-dimensional image sensor 1 ₂ must be used to detect the three-dimensional position of the eyes of the operator.

FIG. 7 is a conceptual diagram for explaining the pointing space 32 according to the second embodiment. As shown in FIG. 7, in the second embodiment, which sets a predetermined reference point 31 to the eye position of the operator detected by the other 3-dimensional image sensor 1 _-2. That is, the pointing space 32 according to the second embodiment is a conical space formed by connecting the display screen 30 and the position 31 of the operator's eyes.

  FIG. 8 is a block diagram illustrating a functional configuration example of the microcomputer 2 according to the second embodiment. In FIG. 8, the same reference numerals as those shown in FIG. 2 have the same functions, and therefore redundant description is omitted here. As shown in FIG. 8, in the second embodiment, a reference point recognition unit 26 is provided as a functional configuration of the microcomputer 2. In addition, a pointing space setting unit 27 having a function different from that of the pointing space setting unit 21 shown in FIG. 2 is provided.

Reference point recognition unit 26 based on information output from the other 3-dimensional image sensor 1 _-2, recognizes the operator's eyes to be the reference point 31. The pointing space setting unit 27 sets a pointing space 32 with the recognized eye position of the operator as a reference point 31. The position information of the display screen 30, which is another element that defines the pointing space 32, is registered in advance in a memory such as the RAM 2c. In the present embodiment, an instruction object recognition unit 22 based on the position information of the object to be within the pointing space 32 which is output from one of the 3-dimensional image sensor 1 _-1 recognizes the indicator 34.

As described above, the position of the eyes of the operator detected by the 3-dimensional image sensor 1 _-2 When configured as a reference point 31 of the pointing space 32, when the operator moves, according to the position of the moved eye Therefore, the pointing space 32 also moves. Thus, no matter where the operator moves with respect to the display screen 30, it is necessary to perform a troublesome operation such as moving the pointing object 34 such as a finger to a fixed space surface as in the past. The usability can be further improved. Further, no matter where the operator moves with respect to the display screen 30, the operator always operates while expecting the display screen 30, so that the inconvenience that an error occurs in the operation input is minimized. It can also be suppressed.

  The position of the operator's eye used as the reference point 31 is preferably the position of the effect. When the display screen 30 is viewed with both eyes as shown in FIG. 9A, the indicator 34 between the display screen 30 and the operator looks double. This is because, as shown in FIGS. 9B and 9C, the position where the indicator 34 can be seen differs depending on whether the display screen 30 together with the indicator 34 is viewed with the right eye or the left eye. .

  In general, when the image looks double due to parallax, it is said that the image viewed with effectiveness works on the judgment. For example, when the effect is the right eye, when both eyes are opened and a target is pointed with a fingertip, the fingertip does not deviate from the target even if the left eye is closed, but when the right eye is closed, the fingertip is displaced from the target. Therefore, by setting the cone of the pointing space 32 to be a cone that looks into the display screen 30 from the effect of the operator, even if the indicator 34 looks double, the target point 36 that is indicated is a point as intended. Can be.

The operator's effect may be registered in advance in the RAM 2c of the microcomputer 2 or other memory (not shown) using an input device (not shown) such as a keyboard, mouse, or joystick. . In this case, pointing object recognition unit 22, according to the information of efficacy that was previously registered to recognize the effect based on the information output from the other 3-dimensional image sensor 1 _-2. Then, the pointing space setting unit 27 sets the pointing space 32 with the effect as the reference point 31.

  In the second embodiment, the example in which the position of the operator's eyes is set as the reference point 31 has been described. However, the reference point 31 may be set at the nose position, mouth position, face center, or the like. good.

(Third embodiment)
Next, a third embodiment of the present invention will be described. In the third embodiment, one three-dimensional image sensor 1 is used as the position detection means, as in the first embodiment shown in FIG. The functional configuration of the microcomputer 2 according to the third embodiment is as shown in the block diagram of FIG. In FIG. 10, components having the same reference numerals as those shown in FIG. 2 have the same functions, and thus redundant description is omitted here.

  As shown in FIG. 10, in the third embodiment, as a functional configuration of the microcomputer 2, a pointing space setting unit 28 having a function different from that of the pointing space setting unit 21 shown in FIG. The pointing space setting unit 28 sets the reference point 31 at a position where the pointing space 32 where the operator can easily operate the indicator 34 is set. Specifically, the pointing space setting unit 28 instructs the display controller 3 to display a calibration screen of the reference point 31 as shown in FIG. Predetermined marks 40 are displayed at the four corners of the calibration screen.

  The operator performs a pointing operation (for example, a pointing start operation or a click operation) on the four corner marks 40 in space. When the pointing operation is performed on the mark 40 at the four corners by the operator, the pointing space setting unit 28 determines the position of the indicator 34 when the operator starts or ends the pointing operation for each of the four corner marks. Each plane is detected by the three-dimensional image sensor 1 and a plane formed by connecting these four positions is specified. Further, the pointing space setting unit 28 performs the pointing operation on the display screen 30 (more precisely, a plane formed by connecting the four corner marks 40 on the display screen 30) and the four corner marks 40 on the display screen 30. Assuming a cone having both the cross section of the plane obtained by connecting the position of the indicator 34, the position corresponding to the apex is set as the reference point 31. A cone formed by connecting the reference point 31 thus obtained and the display screen 30 is set as a pointing space 32.

  In the second embodiment described above, an indicator 34 such as a finger is always placed on a straight line 35 connecting the target point 36 to be operated on the display screen 30 and the assumed reference point 31 of the operator's eye position. It is necessary to put and operate. On the other hand, according to the third embodiment, when the pointing operation is performed on the four corners of the display screen 30 using the calibration screen as shown in FIG. When a pointing operation is performed with the pointing object 34 placed, a pointing space 32 that always includes planes defined by the four corners is set. Therefore, it is possible to set the pointing space 32 at a position where the operator can easily perform an operation input according to the individual operator, and the usability of the human interface in the space can be further improved.

  In addition, since the pointing operation of the mark 40 at the four corners is generally performed unconsciously using the effect, it is not necessary to register the effect in the memory in advance using the input device as in the second embodiment. This is equivalent to automatically setting the effect during the pointing operation of the mark 40 at the four corners.

(Fourth embodiment)
Next, a fourth embodiment of the present invention will be described. The fourth embodiment is a combination of the second embodiment and the third embodiment. That is, in the fourth embodiment, using one of the three-dimensional image sensors _1-1 , the movement of the indicator 34 at the time of calibration is detected by the same operation as in the third embodiment, and the reference point 31 is obtained. Set this to the initial value. Also, the operator of the eye by the second embodiment and the other 3-dimensional image sensor 1 _-2 same operation, nose, mouth, detect movement, such as the center of the face, the reference point 31 in accordance with the movement Move from the initial value.

  In this way, when the operator moves, the pointing space 32 also moves in accordance with the position of the moved eye or the like. Thus, no matter where the operator moves with respect to the display screen 30, as long as the movement of the pointing space 32 follows, the operability of the operator does not change, so that the usability can be improved. Further, no matter where the operator moves with respect to the display screen 30, the operator always operates while expecting the display screen 30, so that the inconvenience that an error occurs in the operation input is minimized. It can also be suppressed.

  In the first to fourth embodiments, the example in which the three-dimensional image sensor is used as the position detection unit has been described. However, the present invention is not limited to this example. For example, a stereo camera may be used. FIG. 12 is a diagram for explaining an object position detection method when a stereo camera is used as position detection means.

  As shown in FIG. 12, for example, the reference camera 41 is installed on the right side of the display screen 30 in the direction of the pointing space 32 where the indicator 34 is located, and the indicator 34 is located on the left side of the display screen 30, for example. An auxiliary camera 42 is installed in the direction of the pointing space 32. These two cameras 41 and 42 have the same optical system, and are installed so that the optical axes intersect with a distance d and an angle θ.

Here, the position of the imaging point A1 where the indicator 34 forms an image on the imaging surface of the reference camera 41 is (x1, y1), and the position of the imaging point A2 where the indicator 34 forms an image on the imaging surface of the auxiliary camera 42. Is (x2, y2). When the focal length of the lenses 41a and 42a of each camera 41 and 42 is F, the three-dimensional position (x, y, z) of the indicator 34 viewed from the display screen 30 is obtained by the following equation. be able to.
x = d · x1 · G (x2) / H (x1, x2)
y = d · y1 · G (y2) / H (y1, y2)
z = d · F · G (x1) / H (x1, x2)
Here, G (x1) = F · sinθ · x1 (1-cosθ)
H (x1, x2) = (F + x1 · x2) sinθ + F (x1-x2) cosθ

  In the first to fourth embodiments, the example in which the reference point 31 of the pointing space 32 is provided at one place has been described. However, the reference points 31 are set at a plurality of places and the plurality of pointing spaces 32 are set. Anyway. Further, the position of the reference point 31 may be switched according to the position of the operator. For example, when the operation input device of the first embodiment is applied to an in-vehicle navigation device, the reference point 31 can be switched to any one of predetermined positions provided in the vicinity of a driver seat, a passenger seat, and a rear seat. . Further, when the operation input device of the second embodiment is applied to an in-vehicle navigation device, the reference point 31 can be switched to any of the positions of the eyes of the operator sitting in the driver's seat, front passenger seat, and rear seat. To. Further, when the operation input device of the third embodiment is applied to an in-vehicle navigation device, the reference point 31 is set at any one of positions set by calibration by an operator sitting in a driver's seat, a passenger seat, and a rear seat. Make it switchable. In the fourth embodiment, the reference point 31 can be switched by a combination of the second embodiment and the third embodiment described above.

Further, in the second embodiment and the fourth embodiment, and the indicator 34 one of the three-dimensional image sensor 1 _-1 for detecting the of the other 3-dimensional image sensor 1 _-2 for detecting a reference point 31 Although two are used, the same operation may be performed with one three-dimensional image sensor that can cover both areas where the indicator 34 and the reference point 31 exist.

  In the first to fourth embodiments, the pointing space setting units 21, 27, and 28 set the pointing space 32. However, a pointing surface may be further set in the pointing space 32. . FIG. 13 is a conceptual diagram for explaining the pointing space 32 and the pointing surface 33 in that case.

  For example, the first embodiment will be described as an example. The pointing space setting unit 21 connects the display screen 30 and a predetermined reference point 31 located away from the display screen 30 as shown in FIG. A virtual cone-shaped pointing space 32 is set, and a space surface obtained by mapping the coordinates of the display screen 30 into the pointing space 32 is set as the pointing surface 33.

  Here, the pointing surface 33 is always set between the display screen 30 and the operator (located before the reference point 31). The pointing surface 33 has a tip (such as a fingertip) of the pointing object 34 when the operator performs a pointing operation (including a pointing start operation, a pointing release operation, a click operation, and a drag operation) using the pointing object 34. This is a plane or curved surface that is assumed to form a trajectory of the indicator 34 when the operator naturally moves the indicator 34.

  As described above, in the first embodiment, since the position of the display screen 30 and the position of the reference point 31 are known in advance, the pointing space 32 is uniquely determined. On the other hand, the pointing surface 33 is dynamically set according to the movement of the pointing object 34 in the pointing space 32. In other words, the pointing space setting unit 21 dynamically sets the pointing surface 33 as described above every time a pointing operation is performed.

  In addition, each of the above-described embodiments is merely an example of implementation in carrying out the present invention, and the technical scope of the present invention should not be construed in a limited manner. In other words, the present invention can be implemented in various forms without departing from the spirit or main features thereof.

  The present invention is useful for an operation input device that gives an input command by an operator operation to a computer device, a navigation device, or the like.

It is a figure which shows the structural example of the operation input apparatus by 1st Embodiment. It is a block diagram which shows the function structure which the microcomputer by 1st Embodiment has. It is a conceptual diagram for demonstrating the pointing space by 1st Embodiment. It is a figure for demonstrating the content of the pointing operation | movement by 1st-4th embodiment. It is a flowchart which shows operation | movement of the operation input apparatus by 1st-4th embodiment. It is a figure which shows the structural example of the operation input apparatus by 2nd Embodiment. It is a conceptual diagram for demonstrating the pointing space by 2nd Embodiment. It is a block diagram which shows the function structure which the microcomputer by 2nd Embodiment has. It is a conceptual diagram for demonstrating an effect. It is a block diagram which shows the function structure which the microcomputer by 3rd Embodiment has. It is a figure which shows the calibration screen of a reference point. It is a figure for demonstrating the position detection method of the object in the case of using a stereo camera as a position detection means. It is a conceptual diagram for demonstrating the pointing space and pointing surface which concern on the modification of 1st Embodiment.

Explanation of symbols

DESCRIPTION OF SYMBOLS 1 3D image sensor 2 Microcomputer 3 Display controller 21, 27, 28 Pointing space setting part 22 Pointer recognition part 23 Coordinate detection part 24 Motion detection part 25 Operation input content determination part 26 Reference point recognition part 30 Display screen 31 Reference point 32 Pointing space 33 Pointing surface 34 Pointer 36 Target point

Claims (26)

Position detecting means for detecting a three-dimensional position of an object in space;
A pointing space setting means for setting a conical virtual pointing space formed by connecting a display screen and a predetermined reference point located away from the display screen;
Coordinate detection means for detecting the coordinates of a target point on the display screen using a mapping relationship based on the position of the object in the pointing space detected by the position detection means;
Movement detection means for detecting a three-dimensional movement of the object in the pointing space;
The operation input content is determined based on the coordinates of the target point on the display screen detected by the coordinate detection means and the three-dimensional movement of the object detected by the motion detection means, and a corresponding operation command is output. An operation input device comprising: a determination unit. Comprising an indicator recognition means for recognizing that an object in the pointing space detected by the position detection means is a predetermined indicator;
The coordinate detection means is a coordinate of a target point on the display screen based on a position in the pointing space detected by the position detection means for an object recognized as the indicator by the indicator recognition means. Detect
The operation input device according to claim 1, wherein the movement detection unit detects a three-dimensional movement of an object recognized as the pointing object by the pointing object recognition unit. In the pointing space, when the indicator moves toward the display screen at a predetermined speed or higher within a predetermined coordinate range of the display screen, the determination means determines that a pointing start operation has been performed. The operation input device according to claim 2. In the pointing space, when the pointing object moves away from the display screen at a predetermined speed or more within a predetermined coordinate range of the display screen, the determination unit determines that the pointing release operation has been performed. The operation input device according to claim 2. Within the pointing space, the indicator moves toward the display screen at a predetermined speed or higher within a predetermined coordinate range of the display screen, and then within a predetermined time range of the display screen within a predetermined time. The operation input device according to claim 2, wherein the determination unit determines that a click operation has been performed when the user moves away from the display screen at a predetermined speed or more. In the pointing space, after the indicator moves closer to the display screen at a predetermined speed or more within a predetermined coordinate range of the display screen, the coordinates of the display screen detected by the coordinate detecting means are The operation input device according to claim 2, wherein the determination unit determines that a drag operation has been performed when a movement is made. The operation input device according to claim 1, wherein the reference point is an eye position of an operator detected by the position detection unit. The operation input device according to claim 1, wherein the reference point is a position of an operator's effect detected by the position detection unit. The reference point is a position corresponding to the apex of a cone having both the display screen and a plane obtained by connecting the positions of the indicators when the four corners of the display screen are respectively pointed in space. The operation input device according to claim 2, wherein the operation input device is provided. The pointing space setting means corresponds to the apex of a cone having both the display screen and a plane obtained by connecting the positions of the indicators when the four corners of the display screen are respectively pointed on the space. The position is set as an initial value of the reference point, and the three-dimensional movement of the operator's eyes detected by the movement detecting means based on the position of the operator's eyes detected by the position detecting means. 3. The operation input device according to claim 2, wherein the reference point is moved from the initial value. The operation input device according to claim 2, wherein the indicator recognizing unit recognizes a rod-like protrusion as the indicator. 2. The operation input device according to claim 1, further comprising display control means for controlling to display a predetermined mark at the coordinate position of the target point on the display screen detected by the coordinate detection means. . When it is determined by the determination means that the drag operation has been performed, the object at the coordinate position of the target point on the display screen detected by the coordinate detection means is moved in the dragged direction by the drag amount. The operation input device according to claim 6, further comprising display control means for controlling display. Position detecting means for detecting a three-dimensional position of an object in space;
A virtual cone-shaped pointing space that connects the display screen and a predetermined reference point located away from the display screen is set, and the coordinates of the display screen are set to the object in the pointing space. Pointing space setting means for setting a pointing surface mapped to a position to be
Coordinate detection means for detecting the coordinates of the target point on the display screen using a mapping relationship based on the position of the object on the pointing surface detected by the position detection means;
Movement detection means for detecting a three-dimensional movement of the object in the pointing space;
The operation input content is determined based on the coordinates of the target point on the display screen detected by the coordinate detection means and the three-dimensional movement of the object detected by the motion detection means, and a corresponding operation command is output. An operation input device comprising: a determination unit. The pointing surface is between the display screen and the operator, includes the position of the indicator when the operator performs a pointing operation, and the operator moves the indicator naturally when the operator moves the indicator naturally. It is the surface that the trajectory of the pointing object is supposed to form,
15. The operation input device according to claim 14, wherein the pointing space setting means sets the pointing surface every time a pointing operation is performed. A first step of detecting a three-dimensional position of an object in a virtual cone-shaped pointing space formed by connecting a display screen and a predetermined reference point located away from the display screen;
The coordinates of the target point on the display screen are detected using the mapping relationship based on the position of the object detected in the first step while detecting the three-dimensional movement of the object in the pointing space. A second step of calculating
A third step of determining an operation input content based on the three-dimensional movement of the object and the coordinates of the target point on the display screen corresponding to the position of the object, and outputting a corresponding operation command. Characteristic operation input method. A fourth step of recognizing that the object in the pointing space detected in the first step is a predetermined indicator;
In the second step, the object point on the display screen is determined based on the position in the pointing space detected in the first step for the object recognized as the indicator in the fourth step. The operation input method according to claim 16, wherein the coordinates are calculated. When it is detected in the second step that the indicator moves within the predetermined coordinate range of the display screen and approaches the display screen at a predetermined speed or higher in the pointing space, the third step is performed. 18. The operation input method according to claim 17, wherein it is determined that a pointing start operation has been performed in the step. In the second step, when it is detected that the indicator moves away from the display screen at a predetermined speed or more within a predetermined coordinate range of the display screen in the pointing space, the third step is performed. The operation input method according to claim 17, wherein it is determined that the pointing release operation has been performed in the step. In the second step, the indicator moves within a predetermined coordinate range of the display screen and approaches the display screen at a predetermined speed or higher in the pointing space, and then the display is performed within a predetermined time. 18. When it is detected that a movement away from the display screen at a predetermined speed or more within a predetermined coordinate range of the screen is detected, it is determined in the third step that a click operation has been performed. The operation input method described in 1. In the second step, after the indicator moves closer to the display screen at a predetermined speed or more within a predetermined coordinate range of the display screen in the pointing space, the coordinates of the display screen move. 18. The operation input method according to claim 17, wherein when it is detected that a movement has occurred, it is determined in the third step that a drag operation has been performed. The operation input method according to claim 16, wherein the reference point is set at an eye position of the operator. A fifth step of displaying predetermined marks at the four corners of the display screen to prompt the operator to perform a pointing operation;
A cone having both the plane obtained by connecting the position of the indicator when the operator performs a pointing operation on the space and the display screen as cross sections is set as the pointing space. The operation input method according to claim 17, further comprising the step of: before the first step. A fifth step of displaying predetermined marks at the four corners of the display screen to prompt the operator to perform a pointing operation;
The reference point is a position corresponding to the apex of a cone having both the surface obtained by connecting the position of the indicator when the operator performs a pointing operation on the space and the display screen as cross sections. A sixth step of setting the pointing space by setting an initial value before the first step;
A seventh step of detecting a position of the operator's eyes in the space to detect a three-dimensional movement of the eyes of the operator;
The reference point is moved from the initial value in accordance with the three-dimensional movement of the operator's eyes, and a cone formed by connecting the moved reference point and the display screen is reset as the pointing space. The operation input method according to claim 17, further comprising: an eighth step. A first step of detecting a three-dimensional position of an object in a virtual cone-shaped pointing space formed by connecting a display screen and a predetermined reference point located away from the display screen;
The three-dimensional movement of the object in the pointing space is detected, and the position of the object on the pointing surface obtained by mapping the coordinates of the display screen to the position where the object exists in the pointing space is determined. And a second step of detecting the coordinates of the target point on the display screen using a mapping relationship;
A third step of determining an operation input content based on the three-dimensional movement of the object and the coordinates of the target point on the display screen corresponding to the position of the object, and outputting a corresponding operation command. Characteristic operation input method. The pointing surface is between the display screen and an operator, includes the position of the indicator when the operator performs a pointing operation, and the operator moves the indicator naturally when the operator moves the indicator naturally. It is the surface that the trajectory of the pointing object is supposed to form,
26. The operation input method according to claim 25, wherein, in the second step, the pointing surface is set every time a pointing operation is performed.

Images