JPWO2016051520A1 - Cursor control device and cursor control method - Google Patents

Abstract

An operator detection unit 31 that detects the position of the operator's feature point from the detection result by the motion sensor 2, and among the feature points of the operator detected by the operator detection unit 31, the operation point and the feature point that is the base point A feature point extraction unit 31 that extracts a position, and an angle calculation unit that calculates horizontal and vertical angles formed by a line segment connecting the positions of the operation point and the base point extracted by the feature point extraction unit 31 and a reference axis 32 and a conversion unit 33 that converts the angle calculated by the angle calculation unit 32 into the position of the cursor.

Description

  The present invention relates to a cursor control device and a cursor control method for controlling the position of a cursor on a screen displayed on a monitor using the position of a human feature point in a three-dimensional space detected by a feature point detection sensor. It is.

  2. Description of the Related Art Conventionally, a system is known in which an operator performs a gesture on a screen displayed on a monitor (for example, a large display for an exhibition) to move the cursor on the screen (for example, a patent). Reference 1).

In this system, for example, a motion sensor (feature point detection sensor) is used to detect the position (spatial coordinate value) of an operator's feature point (for example, a hand) in a predetermined three-dimensional space. The position of the cursor is set by associating the spatial coordinate value of this hand with the two-dimensional coordinates (monitor coordinates) on the screen displayed on the monitor. More specifically, as shown in FIG. 8, for the x-coordinate, is associated with the minimum value x ₁ of the spatial coordinate to the minimum value x _min of the monitor coordinate, the maximum value of the monitor coordinate maximum value x ₂ spatial coordinates Correspond to x _max . Similarly, with regard to the y coordinate, the minimum value y ₁ of the spatial coordinate is made to correspond to the minimum value y _min of the monitor coordinate, and the maximum value y ₂ of the spatial coordinate is made to correspond to the maximum value y _max of the monitor coordinate. By setting the cursor position in this manner, the cursor on the screen displayed on the monitor can be moved by the operator moving his / her hand.

JP 2013-143148 A

  As described above, in the prior art, the spatial coordinate value of the operator's hand is associated with the monitor coordinate. Therefore, the cursor on the screen moves according to the amount of movement in the horizontal and vertical directions of the operator's hand. However, in this case, there is a problem that the operation load varies depending on the physique of the operator. That is, in the case of an operator with a small physique, a larger gesture is required for an operator with a large physique, which increases the physical burden.

  The present invention has been made to solve the above-described problems, and provides a cursor control device and a cursor control method capable of controlling the position of a cursor without depending on an operator's physique difference. It is an object.

  The cursor control device according to the present invention includes an operator detection unit that detects a position of the operator's feature point from a detection result by the feature point detection sensor, and an operation among the operator's feature points detected by the operator detection unit. Horizontal and vertical angles between the reference point and the feature point extraction unit that extracts the point and the position of the feature point that is the base point, the line segment that connects the operation point and the position of the base point extracted by the feature point extraction unit An angle calculation unit for calculating the angle, and a conversion unit for converting the angle calculated by the angle calculation unit into the position of the cursor.

  In the cursor control method according to the present invention, the operator detecting unit detects the position of the feature point of the operator from the detection result by the feature point detecting sensor, and the feature point extracting unit is operated by the operator detecting unit. Of the detected feature points of the operator, a feature point extraction step for extracting the position of the feature point as the operation point and the base point, and the angle calculation unit determines the position of the operation point and the base point extracted by the feature point extraction unit. An angle calculating step for calculating the horizontal and vertical angles formed by the connecting line segment and the reference axis, and a converting unit for converting the angle calculated by the angle calculating unit into a cursor position. Is.

  According to this invention, since it comprised as mentioned above, the position of a cursor can be controlled without depending on an operator's physique difference.

It is a figure which shows the structure of the whole system containing the cursor control apparatus which concerns on Embodiment 1 of this invention. It is a flowchart which shows operation | movement of the cursor control apparatus which concerns on Embodiment 1 of this invention. It is a figure explaining operation | movement of the cursor control apparatus which concerns on Embodiment 1 of this invention, (a) It is a figure which shows the case where angle (theta) _x = 0 of horizontal direction, (b) Angle (theta) _y = vertical direction It is a figure which shows the case of 0, and is a figure which shows the cursor position on the screen in the case of (c) (a), (b). It is a figure explaining operation | movement of the cursor control apparatus which concerns on Embodiment 1 of this invention, (a) It is a figure which shows the case of the angle (theta) _x = (theta) _x1 of a horizontal direction, (b) Angle (theta) _{y of a} perpendicular direction It is a figure which shows the case of = 0, and is a figure which shows the cursor position on the screen in the case of (c) (a), (b). It is a figure explaining operation | movement of the cursor control apparatus which concerns on Embodiment 1 of this invention, (a) It is a figure which shows the case of the angle (theta) _x = (theta) _x1 of a horizontal direction, (b) Angle (theta) _{y of a} perpendicular direction It is a figure which shows the case of = -theta _y1 , and is a figure which shows the cursor position on the screen in the case of (c) (a), (b). It is a figure explaining the effect of the cursor control apparatus which concerns on Embodiment 1 of this invention. It is a figure explaining the coordinate offset in the cursor control apparatus which concerns on Embodiment 1 of this invention. It is a figure explaining the conventional cursor control, (a) It is a figure which shows a space coordinate, (b) It is a figure which shows a monitor coordinate.

Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.
Embodiment 1 FIG.
FIG. 1 is a diagram showing a configuration of the entire system including a cursor control device 3 according to Embodiment 1 of the present invention.
In the system shown in FIG. 1, a monitor 1, a motion sensor (feature point detection sensor) 2, and a cursor control device 3 are provided.

The monitor 1 displays various contents (screens).
The motion sensor 2 detects the position (spatial coordinate value) of a human feature point in a three-dimensional space at a predetermined sampling interval, and detects an operator who operates the screen displayed on the monitor 1. It is installed in a place where This motion sensor 2 includes a camera that captures a color image by capturing a predetermined space, and a depth sensor that detects the depth of an object existing in the space. It also has a database that holds a model indicating the position of each feature point (head, shoulders, hands, hips, feet, etc.) with respect to a human silhouette. The motion sensor 2 extracts a silhouette of an object existing in the space from the color image captured by the camera and the detection result of the depth sensor. Then, by matching this silhouette with a model stored in the database, a person is detected, and the position of the feature point of the person is detected.

  The cursor control device 3 controls the position of the cursor on the screen displayed on the monitor 1 using the position of the human feature point in the three-dimensional space detected by the motion sensor 2. As illustrated in FIG. 1, the cursor control device 3 includes an operator detection unit 31, a feature point extraction unit 32, an angle calculation unit 33, and a conversion unit 34. The cursor control device 3 is executed by program processing using a CPU based on software.

  The operator detection unit 31 detects the position of the feature point of the operator from the detection result by the motion sensor 2. Note that the operator detection unit 31 has a function of storing the detected positions of the feature points of the operator one by one.

For example, when this system is used in an exhibition, it is assumed that there are many people in the space where the motion sensor 2 detects. Therefore, the operator detection unit 31 has a function of following only the person recognized as the operator.
In this function, a person who has come to a predetermined position (for example, the middle) in the space is recognized as an operator and follows. Specifically, first, the range in which the operator is detected is limited to a partial range (for example, the middle) of the space. A person who falls within this detection range is recognized as an operator. Thereafter, the detection range is extended to the entire space, and the movement of the operator is continuously followed. Thereafter, even if another person enters the detection range, the operation right is not transferred as long as the operator is within the detection range. Thereafter, when the operator goes out of the detection range, it is considered that the operation right has been abandoned. Then, the detection range is narrowed again to wait for recognition of a new operator.

  Moreover, you may add the function to transfer an operation right to another person within the said detection range. In this case, after recognizing the operator, another person enters the detection range, and both perform a predetermined gesture (shake hand, shake hand, etc.). As a result, the transfer of the operation right is permitted.

  The feature point extraction unit 32 includes the position of the feature point (the feature point corresponding to the cursor on the screen) as the operation point and the feature point as the base point among the feature points of the operator detected by the operator detection unit 31. The position of (a feature point that becomes an axis with respect to the operating point) is extracted.

  The angle calculation unit 33 calculates the horizontal and vertical angles between the line segment connecting the positions of the operation point and the base point extracted by the feature point extraction unit 32 and the reference axis. The reference axis may be set independently of the operating point and the base point, or may be related to each other.

  The conversion unit 34 converts the angle calculated by the angle calculation unit 33 into the position of the cursor.

  Next, the operation of the cursor control device 3 configured as described above will be described with reference to FIGS. In the following description, an example in which the operator moves his / her hand in front of the motion sensor 2 to move the cursor on the screen displayed on the monitor 1 will be described. Further, the motion sensor 2 detects the positions of human feature points in a three-dimensional space at a predetermined sampling interval.

  In the operation of the cursor control device 3, as shown in FIG. 2, first, the operator detection unit 31 detects the position of the feature point of the operator from the detection result by the motion sensor 2 (step ST1, operator detection step). .

  Next, the feature point extraction unit 32 extracts the position of the feature point serving as the operation point and the base point among the feature points of the operator detected by the operator detection unit 31 (step ST2, feature point extraction step). 3 to 5, the operating point is the operator's hand, and the base point is the operator's head.

Next, the angle calculation unit 33 calculates the horizontal and vertical angles formed by the line segment connecting the positions of the operation point and the base point extracted by the feature point extraction unit 32 and the reference axis (steps ST3 to 5). Angle calculation step). For example, in FIGS. 3 to 5, a line (Z axis) connecting the head of the operator and the sensor position of the motion sensor 2 is used as a reference axis. Then, the angle calculation unit 33 calculates a horizontal direction (XZ component) angle θ _x and a vertical direction (YZ component) angle θ _y formed by a line segment connecting the position of the hand and the head and the reference axis.

Next, the conversion unit 34 converts the angle calculated by the angle calculation unit 33 into the position of the cursor (step ST6, conversion step). At this time, as shown in FIGS. 3 to 5, the conversion unit 34 associates, for example, the case of the angle θ _x = 0 in the horizontal direction with the center value x _c of the monitor coordinates, and determines the positive direction (on the right side of the figure). the maximum angle θ _x = θ _x1) of in correspondence to the maximum value _{x max} of the monitor coordinates, a maximum angle theta _x = - [theta] _x1 of a predetermined negative direction (left side in the figure) to the minimum value _{x min} of the monitor coordinates Make it correspond. Similarly, for example, the case where the angle θ _y = 0 in the vertical direction is made to correspond to the center value y _c of the monitor coordinates, and the predetermined maximum angle θ _y = −θ _y1 in the negative direction (the lower side in the figure) is monitored. Corresponding to the maximum value y _{max of} coordinates, a predetermined maximum angle θ _y = θ _y2 in the positive direction (upper side in the figure) is made to correspond to the minimum value y _min of monitor coordinates. When the hand is moved beyond the maximum angle, it corresponds to the extreme value (maximum value, minimum value) of the monitor coordinates. FIG. 3 shows the position of the cursor 101 when the angle between the line segment connecting the hand and the head and the reference axis is θ _x = 0 and θ _y = 0. FIG. 4 shows the position of the cursor 101 when the angles are θ _x = θ _x1 and θ _y = 0. FIG. 5 shows the position of the cursor 101 when the angles are θ _x = θ _x1 and θ _y = −θ _y1 .

  Next, the effect of the present invention will be described using FIG. 6 while comparing with the case of the prior art. In the following, a case where the operator moves his / her hand in the horizontal direction and moves the cursor on the screen displayed on the monitor 1 to the side will be described as an example.

First, in the case of the prior art, the hand space coordinate value detected by the motion sensor 2 is converted into monitor coordinates.
In this method, as shown in FIG. 6, when the length of the arms of the L1 is moving the hand by an angle theta _x, horizontal movement distance of the hand becomes X1, cursor from the following equation (1) The amount of movement is M1. Α is a coefficient.
M1 = αX1 (1)

On the other hand, when the length of the arm of L2 (> L1) is the angle theta _x simply moving the hand, the horizontal movement distance of the hand X2 (> X1), and the following equation from the cursor (2) The amount of movement is M2 (> M1).
M2 = αX2 (2)
Therefore, the amount of movement of the cursor changes depending on the physique of the operator, and the operation burden changes.

On the other hand, in the case of the present invention, the horizontal and vertical angles formed by the line segment connecting the operating point (hand) and the base point (head) detected by the motion sensor 2 and the reference axis are as follows: It has been converted to monitor coordinates.
In this method, as shown in FIG. 6, since the movement amount M3 of the cursor by the angle theta _x moving the hands is determined, the moving amount M3 of the length of the arms cursor even L2 be L1 is unchanged The operation burden is not changed. In this case, the movement amount M3 of the cursor is expressed by the following equation (3). Here, D is the maximum angle for moving the hand, and P is the maximum number of pixels on the screen.
M3 = (θ _x / D) × P (3)

  As described above, according to the first embodiment, the operator detection unit 31 that detects the position of the operator's feature point from the detection result of the motion sensor 2 and the operator detected by the operator detection unit 31. Of the feature points, a feature point extraction unit 32 that extracts the positions of the operation points and the base point feature points, a line segment that connects the operation points and the base point positions extracted by the feature point extraction unit 32, Depends on the physique difference of the operator since the angle calculation unit 33 that calculates the angle in the horizontal direction and the vertical direction to be formed and the conversion unit 34 that converts the angle calculated by the angle calculation unit 33 into the position of the cursor The cursor position on the screen can be controlled without any operation, and any operator can obtain the same operation experience.

  In the above description, the case where the operating point is the operator's hand has been described. However, the operating point is not limited to this, and may be another characteristic point (such as a fingertip) of the operator. In the above description, the base point is the operator's head. However, the present invention is not limited to this, and any point that serves as an axis with respect to the operating point may be used. Elbow, neck base, arm base, etc.).

  For example, when the position of the operator's hand (operation point) and shoulder (base point) is detected by the motion sensor 2, the shoulder may be hidden by the hand depending on the position of the hand, and the position of the shoulder may not be detected. . Therefore, in such a case, the feature point extraction unit 32 may determine that the shoulder has overlapped with the hand and estimate the position of the shoulder. As an estimation method at this time, for example, a method of estimating from the position of the shoulder immediately before the shoulder overlaps with the hand, a method of setting the center position from the position of the shoulder before and after the shoulder overlaps with the hand, an operator not overlapping with the hand A method of estimating the position of the shoulder according to a model held in a database from the position of other feature points (for example, the head) can be considered.

  Further, when the operator is recognized by the operator detection unit 31, the feature point extraction unit 32 is adjacent from the positional relationship of each feature point (head, hand, elbow, shoulder, waist, etc.) of the operator. The length of the line segment connecting the feature points is stored. Thereafter, the movement of each feature point is stored for each process (every frame). If any of the feature points cannot be recognized, the position of the feature point is calculated from the change indicating the position of each feature point in the previous few frames and the length of the line segment connecting each feature point stored first. May be estimated.

In the above description, the feature point extraction unit 32 extracts the position of one base point. However, the present invention is not limited to this, and the positions of a plurality of base points may be extracted. By extracting the positions of a plurality of base points, it is possible to perform cursor control in consideration of an operator's failure, habit, operational fatigue, and the like.
For example, when moving a hand forward, there are people who move their hands around their elbows in addition to those who move their hands around their shoulders. Therefore, when only the positions of the hand and the shoulder are extracted, the detection result may be smaller than the angle change intended by the operator for a person who moves the hand around the elbow. Therefore, the feature point extraction unit 32 may extract the positions of a plurality of base points, and the conversion unit 34 may be configured to use an angle with a large angle change among a plurality of sets of angles calculated by the angle calculation unit 33. Good.

  Further, in the cursor control by gesture, it is difficult to stop the operator's hand, so it is difficult to stop the cursor. Therefore, when a filter unit is added to the cursor control device 3 and the current angle calculated by the angle calculation unit 33 is within the threshold with respect to the previous angle (an angle one to several frames before, etc.), The current angle may be replaced with the previous angle. Thereby, an operation within a threshold value such as a hand shake can be excluded as noise, and the cursor can be stopped.

  In the above description, the motion sensor 2 is used as the feature point detection sensor. However, the present invention is not limited to this, and any sensor that can detect the position of a human feature point in a three-dimensional space may be used.

  Further, in the processing by the angle calculation unit 33, the position of the feature point and the reference axis may be offset. Thereby, for example, an operator's failure, wrinkle, operational fatigue, etc. can be taken into account, and operability is improved. The example of FIG. 7 shows a case where the positions of the base point (head) and the reference axis are offset downward for an operator who cannot move his hand greatly upward due to some factor. Reference numeral 102 indicates the position of the virtual head after the offset, and the axis of the broken line indicates the position of the virtual reference axis after the offset. The offset setting may be set before starting the screen operation or may be changed during the screen operation.

In the above description, the description is given assuming a system in which an operator performs a gesture on the screen of a large display (monitor 1) for an exhibition to move the cursor on the screen. However, the present invention is not limited to this, and the present invention can be applied to any system that operates by moving the cursor on the screen by the operator performing a gesture on the screen displayed on the monitor 1. The same effect can be obtained.
For example, when a worker checks a work procedure manual on a personal computer while wearing work gloves at a factory, the keyboard cannot be touched. Even in such a case, the present invention can be applied. For example, the same effect can be obtained by using the operating point as a fingertip and the base point as a palm.

  Further, in the present invention, any constituent element of the embodiment can be modified or any constituent element of the embodiment can be omitted within the scope of the invention.

  The cursor control device according to the present invention can realize operability that does not depend on the physique difference of the operator, and uses the position of the human feature point in the three-dimensional space detected by the feature point detection sensor as a monitor. It is suitable for use in a cursor control device that controls the position of the cursor on the displayed screen.

  DESCRIPTION OF SYMBOLS 1 Monitor, 2 motion sensor (feature point detection sensor), 3 cursor control apparatus, 31 operator detection part, 32 feature point extraction part, 33 angle calculation part, 34 conversion part.

Claims (5)

In a cursor control device that controls the position of a cursor on a screen displayed on a monitor using the position of a feature point of a person in a three-dimensional space detected by a feature point detection sensor,
An operator detection unit for detecting the position of the operator's feature point from the detection result by the feature point detection sensor;
Among the operator feature points detected by the operator detection unit, a feature point extraction unit that extracts the position of the feature point that is the operation point and the base point;
An angle calculation unit for calculating horizontal and vertical angles formed by a line segment connecting the position of the operation point and the base point extracted by the feature point extraction unit, and a reference axis;
A cursor control device comprising: a conversion unit that converts the angle calculated by the angle calculation unit into the position of the cursor. 2. A filter unit that replaces the current angle with the previous angle when the current angle calculated by the angle calculation unit is within a threshold with respect to the previous angle. The cursor control device described. 2. The cursor control according to claim 1, wherein when the position of the base point cannot be extracted, the feature point extracting unit determines that the base point overlaps the operation point and estimates the position of the base point. apparatus. The feature point extraction unit extracts a plurality of feature point positions serving as base points;
The cursor control device according to claim 1, wherein the conversion unit uses an angle having a large angle change among a plurality of sets of angles calculated by the angle calculation unit. In the cursor control method for controlling the position of the cursor on the screen displayed on the monitor using the position of the feature point of the person in the three-dimensional space detected by the feature point detection sensor,
An operator detection step in which an operator detection unit detects the position of the operator's feature point from the detection result of the feature point detection sensor;
A feature point extraction step for extracting a position of a feature point serving as an operation point and a base point among the feature points of the operator detected by the operator detection unit;
An angle calculation step in which an angle calculation unit calculates an angle in a horizontal direction and a vertical direction formed by a line segment connecting the positions of the operation point and the base point extracted by the feature point extraction unit, and a reference axis;
A conversion unit, wherein the conversion unit includes a conversion step of converting the angle calculated by the angle calculation unit into the position of the cursor.

Images