JP5148002B1 - Information processing apparatus, electronic device, information processing method, and program - Google Patents

Abstract

An object of the present invention is to accurately recognize the position of one end of an object existing in a detection range.
An information processing apparatus according to the present invention detects an object existing in a predetermined detection range, outputs detection data corresponding to the detected object, and one end of the detected object based on the detection data. An area specifying means for specifying an object area including a part having a portion, a reference point setting means for setting a reference point based on the specified object area, and a detection range so that a scanning line surrounding the reference point extends from the reference point The object is detected in the detection range based on the scanning means for scanning and the shape of the scanning line when the condition that the part of the scanning line that overlaps the object region is a part of the scanning line and one continuous line is satisfied And a recognition space setting means for setting a recognition space for doing this.
[Selection] Figure 3

Description

  The present invention relates to a technique for setting a recognition space for recognizing an object.

  When inputting a user instruction to the computer, the user uses an input interface such as a keyboard or a mouse. In recent years, a technique related to a natural interface (sometimes referred to as a natural user interface) has been developed in order to facilitate the input of user instructions. For example, as described in Patent Document 1, a technique for inputting a user instruction by a gesture has been developed. Further, as described in Patent Document 2, a technique has been developed that improves accuracy when a user instruction is input by combining a user's line of sight with a gesture.

US Patent Application Publication No. 2011/0193939 US Patent Application Publication No. 2011/0029918

  When receiving an input based on the shape and motion of an object, an object in a predetermined detection range is detected by an image sensor, a depth sensor, or the like, but the range to be detected does not necessarily have to be the entire detection range. Therefore, it is desirable to analyze the movement limited to a part of the detection range depending on the position of the object to be detected. On the other hand, the method for accurately limiting the detection range differs depending on the shape of the object to be detected. Therefore, further improvement of the method for limiting the detection range is desired.

  An object of this invention is to recognize the position of the one end part of the object which exists in the detection range accurately.

According to an embodiment of the present invention, the detection unit detects a hand and an arm existing in a predetermined detection range, and outputs detection data corresponding to the detected hand and arm , and the detection based on the detection data. Area specifying means for specifying an object area including a hand and an arm , reference point setting means for setting a reference point inside the specified object area , and a scanning line surrounding the reference point so as to extend from the reference point A scanning unit that scans the detection range, and the scanning line when the condition that a portion of the scanning line that overlaps the object region becomes a part of the scanning line and one continuous line is identified and identified The detection based on the position of the line when the amount of change in the length of the one continuous line by translating the scanning line along the direction in which the arm extends exceeds a predetermined threshold Range The information processing apparatus is provided and a recognition space setting means for setting a recognition space for recognizing the hand.

According to one embodiment of the present invention, a detecting means for detecting the hand and arm is present in a predetermined detection range, based on the detection data output in response to the hand and arm is detected, the hand and arm to the detection Identifying an object area including the reference area , setting a reference point inside the specified object area , scanning the detection range so that a scan line surrounding the reference point extends from the reference point, and out of the scan lines, Identifying the scanning line when the condition that the portion overlapping the object region is a part of the scanning line and one continuous line is specified, and translating the specified scanning line along the direction in which the arm extends A recognition space for recognizing the movement of the hand in the detection range is set on the basis of the position of the line when the amount of change in the length of the one continuous line exceeds a predetermined threshold. Information processing There is provided.

Hand According to one embodiment of the present invention, a computer, a detection means for detecting a hand and arm is present in a predetermined detection range, based on the detection data output in response to the hand and arm is detected, that the detection And an area specifying means for specifying an object area including an arm , a reference point setting means for setting a reference point inside the specified object area , and a scanning line surrounding the reference point so that a scanning line extends from the reference point. A scanning unit that scans a detection range; and the scanning line that satisfies a condition that a portion of the scanning line that overlaps the object region is a part of the scanning line and one continuous line is identified, and the identified scanning line with reference to the position of the line when the variation of the length of said one continuous line scanning lines by translating along the extending direction of the arm exceeds a predetermined threshold, the detection range Among programs to function as a recognition space setting means for setting a recognition space for recognizing the operation of the hand it is provided.

  According to the present invention, the position of one end portion of an object existing in the detection range can be accurately recognized.

It is the schematic which shows the structure of the electronic device 1 which concerns on embodiment of this invention. It is a block diagram which shows the function structure of the information processing apparatus 10 which concerns on embodiment of this invention. It is a block diagram which shows the structure of the recognition space setting function 100 which concerns on embodiment of this invention. It is a figure explaining an example of the method of setting the reference point C in the scanning part 130 which concerns on embodiment of this invention. It is a flowchart which shows the process which sets recognition space in the recognition space setting part 140 which concerns on embodiment of this invention. It is a figure explaining an example of the identification method of the scanning line shape in the recognition space setting part 140 which concerns on embodiment of this invention. It is a figure explaining an example of the identification method of the feature position in the recognition space setting part 140 which concerns on embodiment of this invention. It is a figure explaining an example of the adjustment method of the specific shape in the recognition space setting part 140 which concerns on embodiment of this invention. It is a figure explaining an example of the setting method of the recognition space in the recognition space setting part 140 which concerns on embodiment of this invention. It is a block diagram which shows the structure of the gesture input function 200 which concerns on embodiment of this invention. It is a figure explaining the position of the coordinate calculated by the behavior measurement part 210 which concerns on embodiment of this invention. It is a figure explaining the table for collation used by the determination part 220 which concerns on embodiment of this invention. It is a figure explaining the example of the gesture defined in the table for collation which concerns on embodiment of this invention. It is a flowchart which shows the process in the determination part 220 which concerns on embodiment of this invention. It is a block diagram which shows the structure of the gaze authentication function 300 which concerns on embodiment of this invention. It is a figure explaining the example of the confirmation screen displayed on the display apparatus 14 which concerns on embodiment of this invention. It is a flowchart which shows the process in the determination part 330 which concerns on embodiment of this invention.

  Hereinafter, an electronic apparatus according to an embodiment of the present invention will be described in detail with reference to the drawings. In addition, embodiment shown below is an example of embodiment of this invention, and this invention is not limited to these embodiment.

<Embodiment>
An electronic apparatus according to an embodiment of the present invention will be described in detail with reference to the drawings.

[overall structure]
FIG. 1 is a schematic diagram showing a configuration of an electronic apparatus 1 according to an embodiment of the present invention. In this example, the electronic device 1 is a personal computer. The electronic device 1 is a natural interface that can accept an instruction input from the user to the electronic device 1 not only by the keyboard 13a or the mouse 13b but also by a user's gesture (mainly a gesture using a hand) or a line of sight. It is an apparatus having. The electronic device 1 may be any device that accepts an input of a user instruction and executes a process according to the instruction, such as a smartphone, a mobile phone, a television, a game machine, or a security device.

  The electronic device 1 includes an information processing apparatus 10. The electronic device 1 includes an operation unit 13 (in this example, a keyboard 13a and a mouse 13b), a display device 14, an object detection device 20, and a line-of-sight measurement device 30. Each of these components is connected to the information processing apparatus 10 by wire or wirelessly. In addition, some structures may be comprised as an integral apparatus, such as the object detection apparatus 20 and the visual line measuring apparatus 30 being comprised as an integral apparatus, or the whole structure is comprised as an integral apparatus. May be.

  The information processing apparatus 10 includes a CPU (Central Processing Unit) 11 and a memory 12. The CPU 11 implements various functions such as a recognition space setting function 100, a gesture input function 200, and a line-of-sight authentication function 300 (see FIG. 2) described later by executing a program stored in the memory 12. Each component of the electronic device 1 is controlled by various functions realized by the CPU 11.

  The operation unit 13 receives an operation by the user and outputs a signal corresponding to the received operation to the CPU 11. In the display device 14, the display mode of the screen is controlled by the control of the CPU 11.

  The object detection device 20 detects an object by the pattern projection method, measures the shape of the object in three dimensions, and outputs a measurement signal indicating the measurement result. The object detection device 20 includes a detection sensor 21 that detects an object existing in a predetermined detection range. In this example, the detection sensor 21 receives a light-emitting element that irradiates infrared light in a predetermined pattern (dot pattern, mesh pattern, etc.) and infrared light having a pattern distorted by reflection from an object, and receives a light-receiving signal. It is a depth sensor which has an image sensor for infrared light which outputs. The object detection device 20 outputs a measurement signal using the light reception signal output from the infrared image sensor. Therefore, the measurement signal reflects the shape of the object when viewed from the detection sensor 21.

  In this example, the pattern projection method is used to detect an object in the object detection device 20, but a TOF (Time of Flight) method used in a laser interferometer or the like may be used, an image sensor for visible light, or the like. A method of analyzing and detecting the image data obtained as a result of shooting with may be used. Also, a plurality of methods may be used in combination. In any method, the object detection device 20 outputs a measurement signal indicating the result of measuring the shape of an object existing in the detection range with three-dimensional coordinates.

  In this example, as shown in FIG. 1, the object detection device 20 is installed on the left side of the display device 14 as viewed from the user using the electronic device 1, and the detection sensor 21 is directed in the direction of the arrow A <b> 1. Therefore, infrared light is irradiated from the left hand side of the user toward the right hand side, and the space between the user who uses the electronic device 1 and the display device 14 and the surrounding space become the detection range. Therefore, the position of the object detection device 20 and the orientation of the detection sensor 21 are suitable for detecting the user's right hand as an object. In addition, arrangement | positioning of this object detection apparatus 20 is an example, Comprising: It is not limited to this arrangement | positioning, What is necessary is just to be determined according to the object which should be detected.

  The line-of-sight measurement device 30 is a device that performs imaging for measuring the user's line-of-sight direction. The line-of-sight measurement device 30 includes a line-of-sight sensor 31 including a light emitting element that irradiates infrared light toward the user's eye to acquire a Purkinje image, and an image sensor that captures an area including the user's eye. A photographing signal indicating a photographing result by the image sensor is output to the information processing apparatus 10. This imaging signal is used in the information processing apparatus 10 to measure the user's line-of-sight direction. In this example, the line-of-sight measurement device 30 is attached to the upper part of the display device 14 as shown in FIG.

  In this example, for the measurement of the line-of-sight direction, a known method of measuring the line-of-sight direction by the corneal reflection method using a Purkinje image is used, but the sclera tracker method, EOG (Electro-oculography) method, search coil method, etc. Other known methods may be used. When using another method, the line-of-sight measurement device 30 may acquire information on the line of sight of the user with a configuration corresponding to the method used for the measurement of the line-of-sight direction.

[Functional configuration of information processing apparatus 10]
FIG. 2 is a block diagram showing a functional configuration of the information processing apparatus 10 according to the embodiment of the present invention. The information processing apparatus 10 includes a recognition space setting function 100, a gesture input function 200, a line-of-sight authentication determination function 300, a detection unit 500, and an execution unit 700.

  The detection unit 500 outputs detection data corresponding to the detected object based on the measurement signal output from the object detection device 20. The detection data may be any data as long as the shape (for example, contour) of the object detected in the detection range can be recognized from the measurement signal. The detection data may include data indicating an outline of the shape of the object (for example, the axis of the object). Note that the detection unit 500 may be included in the object detection device 20.

  The recognition space setting function 100 has a function of recognizing the position of one end of an object existing in the above-described detection range based on detection data from the detection unit 500 and setting a recognition space around it. In this example, the object is a user's hand and arm, and one end of the object means a hand (including a finger). In the following description, it is assumed that the object is a user's hand and arm. The recognition space is set around the hand in the detection range, and is set to recognize the gesture by paying attention to the movement of the hand.

  The gesture input function 200 recognizes the movement or shape of a hand (particularly a finger) in the recognition space as a gesture based on detection data from the detection unit 500, and uses an instruction corresponding to the gesture as an instruction input to the apparatus. Has a function to determine. In this example, it also has a function of determining an instruction input to the apparatus based on an operation on the operation unit 13.

  The line-of-sight authentication determination function 300 measures the user's line-of-sight direction based on the measurement signal from the line-of-sight measurement device 30 and determines whether the line of sight is facing a predetermined area displayed on the display device 14. Have. Details of the recognition space setting function 100, the gesture input function 200, and the line-of-sight authentication determination function 300 will be described later.

  The execution unit 700 controls the operation of the electronic device 1 by executing processing based on instructions determined by the processing of the gesture input function 200 and the line-of-sight authentication determination function 300. At this time, depending on the result of the process in the gesture input function 200 and the line-of-sight authentication determination function 300, the process based on the instruction may not be executed or the process content may be changed.

[Recognition space setting function 100]
FIG. 3 is a block diagram showing the configuration of the recognition space setting function 100 according to the embodiment of the present invention. The recognition space setting function 100 is realized using each function of the region specifying unit 110, the reference point setting unit 120, the operation unit 130, and the recognition space setting unit 140. In this example, the process using the recognition space setting function 100 described in detail below is executed when a start instruction is input to the operation unit 13 by the user. This process may be executed at regular intervals, may be executed according to a predetermined schedule, or may be executed when a predetermined object such as a hand is detected in the detection range. Good.

  Based on the detection data output from the detection unit 500, the region specification unit 110 specifies a hand and arm region (hereinafter referred to as an object region) from the contour of an object detected in the detection range. In this example, the axis L (see FIG. 4) along the direction in which the arm extends is also specified.

  Based on the identified object region, the reference point setting unit 120 sets a reference point C, which will be a scanning start point described later, as shown in FIG.

  FIG. 4 is a diagram illustrating an example of a method for setting the reference point C in the scanning unit 130 according to the embodiment of the present invention. A center point P shown in FIG. 4 indicates a center position in the entire detection range. The reference point setting unit 120 extends a line from the center point P in the vertical direction, and sets the intersection with the axis L as the reference point C. The reference point C may be a point closest to the center point P on the axis L, or may be set by other methods including a method not using the axis L or the center point P. It is desirable to set the area (hand 1000 and arm 2000), and it is more desirable to set the area obtained by removing the finger from the object area.

  Returning to FIG. 3, the description will be continued. The scanning unit 130 sets a scanning line SL surrounding the reference point C, and scans the detection range so that the scanning line SL extends from the reference point C (see FIG. 6). The shape of the scanning line SL is circular in this example, but may be any shape as long as it surrounds the reference point C, such as an ellipse or a rectangle.

  The recognition space setting unit 140 has a shape of the scanning line SL when the portion where the scanning line SL and the object region overlap is a part of the scanning line SL and one continuous line (hereinafter referred to as a specific condition). Based on the above, a recognition space is set. A method for setting the recognition space will be specifically described below with reference to FIGS.

  FIG. 5 is a flowchart showing processing for setting a recognition space in the recognition space setting unit 140 according to the embodiment of the present invention. First, the recognition space setting unit 140 determines whether or not a portion where the scanning line SL and the object region (hand 1000 and arm 2000) overlap (hereinafter referred to as a superimposed portion) satisfies a specific condition (step S110). If the specific condition is not satisfied (step S110; No), this determination is subsequently performed. Therefore, this determination is executed at regular intervals as the scanning unit 130 enlarges the scanning line SL until it is determined that the specific condition is satisfied (step S110; Yes). When the top portion satisfies the specific condition (step S110; Yes), the recognition space setting unit 140 specifies the shape of the scanning line SL that satisfies the specific condition (step S120). Hereinafter, the shape of the specified scanning line SL is referred to as a specific shape. The processing in steps S110 and S120 will be specifically described with reference to FIG.

  FIG. 6 is a diagram for explaining an example of a scanning line shape specifying method in the recognition space setting unit 140 according to the embodiment of the present invention. The scanning lines SL (1), SL (2), and SL (3) indicate the scanning lines SL extending from the reference point C. In FIG. 6, the scanning lines SL (1), SL (2), and SL (3) are indicated by broken lines, and the overlapping portion is indicated by a thick solid line. When extending to the scanning line SL (3), the scanning line SL (3) overlaps only the portion of the arm 2000 in the object region. That is, the overlapping portion LO (3) satisfies the specific condition, and the recognition space setting unit 140 specifies the shape of the scanning line SL (3) as the specific shape.

  Note that whether or not the specific condition is satisfied is not limited to this method, and other methods may be used. For example, the number of intersections between the scanning line SL and the contour portion of the object region may be extracted, and it may be determined whether or not the specific condition is indirectly satisfied, for example, when the number of intersections becomes two. As described above, the method for determining whether or not the specific condition is satisfied includes not only a direct determination of the state of the superimposed portion but also a determination method indirectly.

  Returning to FIG. Subsequently, the recognition space setting unit 140 specifies a feature position (step S130). In this example, the characteristic position is the position of the wrist between the hand 1000 and the arm 2000. An example of the method for specifying the feature position will be described with reference to FIG.

  FIG. 7 is a diagram for explaining an example of a feature position specifying method in the recognition space setting unit 140 according to the embodiment of the present invention. The recognition space setting unit 140 translates the specific shape (scanning line) SL (3) along the axis L. In this example, SL (3), SL (4), and SL (5) are changed in this order. With this parallel movement, the overlapping portion LO changes in the order of LO (3), LO (4), LO (5), and the length of the overlapping portion LO changes. The recognition space setting unit 140 calculates the change in the length of the overlapped portion LO, and changes the length of the overlap portion LO abruptly (for example, when the amount of change in length exceeds a predetermined threshold). It is specified as (characteristic position). In the example illustrated in FIG. 7, the position corresponding to the overlapped portion LO (4) is specified as the feature position.

  As described above, when the overlapping portion LO is recognized by two intersection points between the scanning line SL and the contour portion of the object region, the distance between the two intersection points is set corresponding to the length of the overlapping portion LO. Use it.

  Returning to FIG. Subsequently, when the feature space is specified, the recognition space setting unit 140 adjusts the specific shape based on the feature position within a range that satisfies the specific condition (step S140). An example of a specific shape adjustment method will be described with reference to FIG.

  FIG. 8 is a diagram illustrating an example of a specific shape adjustment method in the recognition space setting unit 140 according to the embodiment of the present invention. In this example, among the intersections of the specific shape (scan line) SL (4) and the axis L, the intersection D1 on the feature position side is fixed, and the diameter R of the specific shape is adjusted. At this time, the specific shape is adjusted so that the diameter R is minimized within a range that satisfies the specific condition (a predetermined margin may be provided). In the example illustrated in FIG. 8, the specific shape SL (4) is adjusted to the specific shape SL (6) having a diameter R connecting the intersections D1 and D2 with the axis L.

  When adjusting so that the diameter R is minimized, for example, the following method may be used. First, the intersection point D1 is fixed, and the specific shape is reduced until the specific condition is not satisfied (for example, the specific shape overlaps the finger portion of the hand 1000). Then, the specific shape size may be adjusted to the size immediately before the specific condition is not satisfied (for example, the specific shape does not overlap the finger portion of the hand 1000). Note that this adjustment method is an example, and another method may be used as long as the adjustment is performed based on the characteristic position within a range that satisfies the specific condition.

  Returning to FIG. Subsequently, the recognition space setting unit 140 sets a recognition space based on the adjusted specific shape (step S150). An example of a recognition space setting method will be described with reference to FIG.

  FIG. 9 is a diagram illustrating an example of a recognition space setting method in the recognition space setting unit 140 according to the embodiment of the present invention. In this example, the recognition space is set as a sphere RA whose center is the intersection D1 and whose radius is the diameter R of the adjusted specific shape SL (6). In consideration of a margin, this radius may be R × (1 + α) (where α is a positive value representing an allowable error ratio) instead of R. The recognition space may not be a sphere, and may be various shapes such as an ellipsoid, a rectangular parallelepiped, and a hemisphere. In any case, it may be determined based on the adjusted specific shape SL (6) (for example, diameter), and the one end side (hand 1000) of the object is fixed with the characteristic position (wrist) fixed. It is desirable to include a movable range.

  Here, when making a gesture by moving the hand 1000, the hand 1000 moves around the wrist. Therefore, by setting the center of the recognition space near the wrist as a characteristic position, the behavior of the hand 1000 can be measured no matter how the hand 1000 moves. Further, erroneous recognition can be reduced due to the influence of surrounding movements other than the hand 1000. The recognition space set in this way is used in the gesture input function 200 (behavior measuring unit 210).

  There is a limit that the hand 1000 can be bent to the arm side. Therefore, the center of the sphere RA in the recognition space is not limited to D1, but may be a point moved to the fingertip side (D2 side) from D1. This center may be a midpoint between two intersections of the adjusted specific shape SL (6) and the contour of the object region. Thus, the radius when the center is not D1 may be a distance from this center to D2.

[Gesture input function 200]
FIG. 10 is a block diagram showing a configuration of the gesture input function 200 according to the embodiment of the present invention. The gesture input function 200 is realized using the functions of the behavior measurement unit 210 and the determination unit 220.

  The behavior measurement unit 210 measures the behavior of at least a part of the object in the recognition space based on the detection data output from the detection unit 500. In this example, the behavior measurement unit 210 functions as a coordinate calculation unit that calculates the coordinates of at least a part of the object (the tip of five fingers in this example). The fingers whose coordinates are calculated are all five fingers in this example, but may be a part of the fingers or only one finger.

  At this time, the behavior measurement unit 210 may calculate the coordinates by extracting information in the recognition space from the information included in the detection data. Therefore, it is possible to reduce the load of information processing, and it is possible to improve the resolution in calculating coordinates. In order to improve accuracy, the optical system (lens magnification, etc.) of the object detection device 20 may be adjusted so that the detection range is narrowed down to the vicinity of the recognition space, and the density of the pattern irradiated by the detection sensor 21 in the vicinity of the recognition space (Dot pattern density, etc.) may be increased.

  FIG. 11 is a diagram illustrating the position of coordinates calculated by the behavior measuring unit 210 according to the embodiment of the present invention. The behavior measuring unit 210 calculates the coordinate A of the tip of the thumb 1001 of the hand 1000, the coordinate B of the tip of the index finger 1002, the coordinate C of the tip of the middle finger 1003, the coordinate D of the tip of the ring finger 1004, and the coordinate E of the tip of the little finger 1005. To do. The behavior measurement unit 210 continues to calculate coordinates at predetermined intervals following the change in the position of each finger.

  Returning to FIG. The determining unit 220 determines an instruction to the user's device based on the coordinates calculated by the behavior measuring unit 210 (hereinafter referred to as “calculated coordinates”). In this example, the determination unit 220 determines an instruction from the user to the apparatus based on the collation table in which the gesture type and the instruction type are associated with each other and the calculated coordinates. The instructions include, for example, an instruction for performing a specific process (calling a menu screen, executing a program, switching pages, etc.) in addition to an instruction to input a determination such as OK or NG. As will be described later, the determination unit 220 may change the instruction to be determined or may not determine the instruction depending on the determination result in the line-of-sight authentication determination function 300 or the like. In addition, the determination unit 220 may determine an instruction based on an operation on the operation unit 13.

  FIG. 12 is a diagram illustrating a collation table used by the determination unit 220 according to the embodiment of the present invention. As shown in FIG. 12, in the verification table, the type of gesture and the type of instruction are associated with each other. In the example illustrated in FIG. 12, the gesture A and the instruction a are associated with each other, the gesture B and the instruction b are associated with each other, and the instruction c is associated with the change from the gesture A to the gesture B.

  FIG. 13 is a diagram for explaining examples of gestures defined in the collation table according to the embodiment of the present invention. In this example, the gesture A is a gesture shown in FIG. 13A, and the relative relationship between the calculated coordinates of each finger is determined as shown in the figure for the definition of the gesture A. That is, the gesture A is defined as a shape in which coordinates A to E are arranged vertically.

  On the other hand, the gesture B is a gesture shown in FIG. 13B, and the relative relationship of the calculated coordinates of each finger is determined as shown in the figure for the definition of the gesture B. That is, the gesture B is defined as a shape in which the coordinates B and C are separated into information and arranged side by side, and the coordinates A, D, and E are gathered below the coordinates B and C. The case of changing from the gesture A to the gesture B indicates a case of performing an operation of changing from the gesture shown in FIG. 13A to the gesture shown in FIG.

  In addition, although the right hand gesture is shown and described here, the left hand gesture may also be defined. Which hand gesture is applied is determined from the positional relationship between the feature position specified in the recognition space setting unit 140 described above and one end of the object (which is relatively on the left side, etc.). May be.

  FIG. 14 is a flowchart showing processing in the determination unit 220 according to the embodiment of the present invention. The determination unit 220 calculates the change speed of the calculated coordinate corresponding to each finger from the change in the position of the calculated coordinate corresponding to each finger, and sets the speed of the finger with the fastest movement as the change speed Vf. The change speed Vf may be determined based on the change speed of the calculated coordinates of each finger, may be an average value of the change speeds of the calculated coordinates of each finger, or may be the calculated coordinates of a specific finger. The rate of change may be.

  The determination unit 220 determines whether the change speed Vf is equal to or higher than the first speed V1 (in this example, 30 cm / second) (step S210). If the change speed Vf is less than the first speed V1 (step S210; No), this determination is subsequently performed. When the change speed Vf becomes equal to or higher than the first speed V1 (step S210; Yes), the determination unit 220 determines whether or not the first time T1 has elapsed while the change speed Vf is equal to or higher than the second speed V2. However, if it has not elapsed (step S221; No), it is determined whether or not the change speed Vf has decreased to less than the second speed V2 (step S222). The second speed V2 is a speed (0.5 cm / second in this example) slower than the first speed V1. Specifically, it is desirable that the second speed V2 is a speed that can be regarded as a stationary state.

  If the first time T1 (0.5 seconds in this example) has elapsed (step S221; Yes) without the change speed Vf decelerating to less than the second speed V2 (step S222; No), the step Return to S210.

  If the change speed Vf decelerates to less than the second speed V2 (step S222; Yes) before the first time T1 has elapsed (step S221; No), the determination unit 220 determines that the change speed Vf is the first speed Tf. It is determined whether or not the second time T2 (1 second in this example) has elapsed in a state of less than 2 speed V2 (step S223). If the second time T2 has not elapsed, the process returns to step S222.

  When the change speed Vf is less than the second speed V2 and the second time T2 has elapsed (step S223; Yes), the calculated coordinates (first coordinates) of each finger are stored (step S230). Thereafter, the determination unit 220 determines whether the change speed Vf is equal to or higher than the first speed V1 (step S240). When the third time T3 has elapsed (step S250; Yes) at a speed less than the first speed V1 (step S240; No), the determination unit 220 uses the first coordinate stored in step S230 and the verification table gesture. The type of instruction corresponding to the first coordinate is determined (step S280). Then, the process returns to step S210.

  On the other hand, if the change speed Vf becomes equal to or higher than the first speed V1 (step S240; Yes) before the third time T3 (1 second in this example) has elapsed (step S250; No), the step The process proceeds to S261. Steps S261 to S263 are the same as the processes from Step S221 to Step S223, and thus the description thereof is omitted.

  When the change speed Vf is less than the second speed V2 and the second time T2 has elapsed (step S263; Yes), the calculated coordinates (second coordinates) of each finger are stored (step S270). Then, the determination unit 220 collates the change from the first coordinate stored in step S230 to the second coordinate stored in step S270 and the type of gesture in the verification table, and the type of instruction corresponding to this change. Is determined (step S280). Then, the process returns to step S210. The execution unit 700 is executed according to the instruction thus determined. When the instruction determined in step S280 is an instruction to end the gesture input, the instruction may be ended without returning to step S210. The instruction to end the gesture input may be performed by a user operation on the operation unit 13 or may be performed when a hand is not detected in the detection range for a certain period.

  The determination unit 220 recognizes the gesture as a gesture when the movement of the user's finger moves at a certain speed or more and remains almost stationary for a certain period of time by performing the processing in the above-described flow. When the user intends to input a gesture, the inventor moves the finger quickly to stop the finger for a while in a state corresponding to the intended gesture. We focused on the slow movement of fingers until the start of.

For example, assuming that a hand is spread out as a gesture, in general, a user often performs an operation of suddenly spreading a hand after slowly picking up the hand. For example, the flow is as shown in (1) to (3) below.
(1) The hand (finger) moves slowly until a gesture input is intended. Therefore, it is not subject to gesture recognition.
(2) As a preparation for starting a gesture, the hand (finger) moves slowly until the hand is picked up, or there is little rest time after the hand is picked up. Therefore, it is not subject to gesture recognition.
(3) In order to recognize as a gesture, the movement of spreading the hand has a fast movement of the hand (finger), and there is a stationary time after the hand is spread. Therefore, it becomes an object of gesture recognition.

[Gaze authentication determination function 300]
FIG. 15 is a block diagram showing a configuration of the line-of-sight authentication function 300 according to the embodiment of the present invention. The line-of-sight authentication determination function 300 is realized using the functions of the line-of-sight measurement unit 310, the display control unit 320, and the determination unit 330.

  The line-of-sight measurement unit 310 measures the user's line-of-sight direction using the imaging signal output from the line-of-sight measurement device 30 and calculates which part of the display device 14 the user's line of sight is directed to. For the measurement of the line-of-sight direction, as described above, a known method for measuring the line-of-sight direction by a corneal reflection method using a Purkinje image is used, but other known methods may be used.

  The display control unit 320 has a function of controlling display on the display device 14 and causes the display device 14 to display a confirmation screen including an authentication area.

  FIG. 16 is a diagram illustrating an example of a confirmation screen displayed on the display device 14 according to the embodiment of the present invention. The confirmation screen is provided with at least one authentication area. In the example shown in FIG. 16, authentication areas W1, W2, and W3 are provided. Further, VP indicates a position on the display device 14 where the user's line of sight obtained based on the calculation result of the line-of-sight measurement unit 310 is directed (hereinafter referred to as line-of-sight position VP). The line-of-sight position VP may be displayed on the display device 14 or may not be displayed.

  The confirmation screen is a screen that is displayed on the display device 14 as content to be confirmed by the user, such as a screen that is necessary for business confirmation performed in a bank or the like, and a screen that displays software usage rules. Items to be checked by the user, such as items to be confirmed, particularly important items, are displayed in a preset authentication area. As shown in FIG. 16, when there are a plurality of authentication areas, the order (authentication order) to be visually recognized by the user is determined. This may be determined in advance in association with the authentication area, or may be determined based on the positional relationship of the authentication area in the display device 14 (such as the earlier the order is on the screen).

  In this example, the authentication order is determined as the first, second, and third in the order of the authentication areas W1, W2, and W3. In the example shown in FIG. 16, two lines of horizontally written character strings are displayed in the authentication area W1, and two lines of horizontally written character strings are displayed in the authentication area W2, although the number of characters is smaller than that of the authentication area W1. In the authentication area W3, one line of horizontal character string is displayed. Note that the authentication area may be one area for the entire screen of the display device 14.

  Returning to FIG. 15, the description will be continued. The determination unit 330 determines whether the user has confirmed the confirmation screen based on the relationship between the line-of-sight position VP and the authentication area.

  FIG. 17 is a flowchart showing processing in the determination unit 330 according to the embodiment of the present invention. First, the determination unit 330 continues to determine whether or not the line-of-sight position VP exists in the first authentication area (authentication area W1 in the example of FIG. 16) (a predetermined margin may be allowed) (step S310; No). ). When it is detected that the line-of-sight position VP exists in the first authentication area (step S310; Yes), the determination unit 330 determines whether there is an authentication area in the next authentication order (step S320).

If there is an authentication area in the next authentication order (step S320; Yes), the determination unit 330 continues to determine whether or not the line-of-sight position VP exists in the next authentication area (step S3 30 ; No). If it cannot be detected that the line-of-sight position VP exists in the next authentication area even after a predetermined time has elapsed, it is assumed that a series of processes for visually recognizing the authentication area has been interrupted, and the previous processes are reset The process may be started again from step S310.

When it is detected that the line-of-sight position VP exists in the next authentication area (step S3 30 ; Yes), the process returns to step S320 again. If there is no authentication area in the next authentication order (step S320; No), the determination unit 330 determines that the confirmation screen has been confirmed by the user (step S340), and ends the process. Hereinafter, this determination is referred to as user confirmation determination.

  Here, the determination unit 330 may determine that the line-of-sight position VP exists in the authentication area on the condition that the line-of-sight position VP passes through a part of the authentication area when the line-of-sight position VP moves. You may judge. Hereinafter, various conditions that can be taken for the determination are exemplified as the following (a) to (c).

(A) The line-of-sight position VP remains in the authentication area for a predetermined time continuously. Note that the predetermined time may be determined for each authentication area according to the amount of characters displayed in the authentication area.
(B) The line-of-sight position VP is stationary for a predetermined time in the authentication area. Note that the predetermined time may be determined for each authentication area according to the amount of characters displayed in the authentication area.
(C) It has been detected that the line-of-sight position VP has moved along the direction of the character string displayed in the authentication area (which may be shifted within a predetermined range). That is, the line-of-sight position VP moves approximately along the horizontal direction in horizontal writing, and the line-of-sight position VP moves approximately along the vertical direction in vertical writing. In the case where a character string is displayed over a plurality of lines, on the condition that the line-of-sight position VP has been detected in the authentication area at least as many times as the number of lines of the character string in the authentication area. Also good. Further, the amount of movement along the direction of the character string may be determined according to the length of the character string.

  Returning to FIG. 15, the description will be continued. The determination result in the determination unit 330 described above is reflected in the processing in the determination unit 220 and the execution unit 700. For example, when reflected in the processing in the execution unit 700, the execution unit 700 executes a process (such as transition to the next confirmation screen) determined in accordance with the confirmation screen when the user confirmation determination is made. You may make it do. Further, the process based on the instruction determined by the determination unit 220 may not be executed until the user confirmation determination is made.

  When reflected in the processing in the determination unit 220, the determination unit 220 may determine an instruction to the apparatus on the condition that the user confirmation is determined. Thereby, in the state where the user confirmation determination is not made, the input of the instruction by the user is rejected. Further, the type of instruction to be determined may be changed according to the presence / absence of user confirmation determination. For example, the display may be changed to an instruction for transitioning to an error screen, displaying a pop-up indicating an error, or displaying which authentication area the user is not viewing.

  In this way, it is determined whether or not the user has visually recognized the authentication area on the confirmation screen, and the process reflecting the determination result is executed, so that the content is not read without reading the content such as the text to be confirmed by the user. It is possible to prevent the operation from proceeding by performing an operation indicating that it has been confirmed. At this time, when there are a plurality of authentication areas that require confirmation, the user often needs to visually recognize them in a predetermined order. However, when the user visually recognizes in an order different from the order, it may be considered that the line of sight is merely directed to the authentication area and the content is not confirmed. In such a case, since it is possible to prevent the user confirmation determination, it is possible to prevent the user from proceeding without confirming the contents.

[Another processing example in determination unit 330]
The determination unit 330 performs the user confirmation determination because the line-of-sight position VP exists in the authentication area in a predetermined order as described above, but performs the user confirmation determination only while the line-of-sight position VP exists in the authentication area. You may make it do. That is, the determination unit 330 performs user confirmation determination when detecting that the user's line of sight is facing the authentication area (the user is viewing the authentication area).

  In such a case, the determination unit 220 determines an instruction for the apparatus only while the user confirmation determination is being performed (while the user visually recognizes the authentication area), and does not determine otherwise. May be. As a result, the user can input an instruction to the apparatus by operating the gesture or the operation unit 13 only when the user is viewing the authentication area. Note that for some gestures or operations on the operation unit 13, an instruction to the apparatus may be determined regardless of the user confirmation determination.

  In addition, the user may be able to input an instruction to the apparatus for a predetermined time after visually recognizing the authentication area. In this case, the determination unit 330 may perform the user confirmation determination for a certain time after the line-of-sight position VP does not exist in the authentication area in addition to the line-of-sight position VP existing in the authentication area. In this way, after the user visually recognizes the authentication area, it is possible to input an instruction to the apparatus even if the user operates while looking at the operation unit 13.

  When there are a plurality of authentication areas on one screen, the determination unit 220 performs an operation on the input gesture or the operation unit 13 when the user visually confirms each authentication area and makes a user confirmation determination. Based on this, an instruction to the apparatus may be determined. Here, the operation unit 13 that receives an input of an operation from the user and the behavior measurement unit 210 that receives an input of a gesture can also be considered as an input reception unit that receives an input of an instruction from the user.

  Note that an instruction to the apparatus may be determined when the relationship between the recognized authentication area and the input gesture or operation on the operation unit 13 corresponds to a predetermined one. For example, when the user performs the gesture A while visually recognizing the authentication area W1, performs the gesture B while visually recognizing the authentication area W2, and performs the gesture A while visually recognizing the authentication area W3. An instruction to the apparatus may be determined accordingly.

  As described above, when the user's line of sight faces the authentication area, the instruction to the apparatus by the determination unit 200 is determined, so that it is possible to prevent the user from inputting the instruction to the apparatus while looking away. .

<Modification>
As mentioned above, although embodiment and the Example of this invention were described, this invention can be implemented in various aspects as follows.

[Modification 1]
The line-of-sight measurement unit 310 described above measures the user's line-of-sight direction, but further measures the user's point of sight (concept including not only the direction but also the distance) by measuring the line-of-sight direction of the user's eyes. You may measure. In this case, the determination unit 330 further determines that the point of sight exists on the screen plane of the display device 14 (a predetermined margin may be allowed) in determining that the line-of-sight position VP exists in the authentication area. This may be a condition. For example, the intersection of the line of sight of the left eye and the line of sight of the right eye is obtained in space and treated as a gazing point, and it is determined whether or not the intersection exists on the screen plane of the display device 14. . When the obtained intersection does not exist on the screen plane, for example, when it is at the back side of the screen as viewed from the user (when a predetermined margin is allowed, it is shifted by a predetermined distance or more on the back side of the screen) In this case, the determination unit 330 may not perform the user confirmation determination even if the line-of-sight direction exists in the authentication area.

[Modification 2]
The recognition space setting function 100, the gesture input function 200, and the line-of-sight authentication determination function 300 described above have been described as being simultaneously implemented in the information processing apparatus 10, but may function independently of each other.

  For example, the recognition space set by the recognition space setting function 100 is not only used for gesture input, but can also be used for other purposes such as when it is desired to limit the range of image recognition to a specific range. In the gesture input function 200, the behavior of the object may be measured in the entire detection range or a predetermined range of the detection range even when the recognition space is not set. For example, the behavior of the entire arm may be measured, and an instruction (such as a screen scroll instruction corresponding to the swing of the arm) may be determined based on this behavior.

DESCRIPTION OF SYMBOLS 1 ... Electronic device, 10 ... Information processing apparatus, 11 ... CPU, 12 ... Memory, 13 ... Operation part, 13a ... Keyboard, 13b ... Mouse, 14 ... Display apparatus, 20 ... Object detection apparatus, 21 detection sensor, 30 ... Line of sight Measuring device 31 ... Gaze sensor 100 ... Recognition space setting function 110 ... Region specifying unit 120 ... Reference point setting unit 130 ... Scanning unit 140 ... Recognition space setting unit 200 ... Gesture input function 210 ... Behavior measurement , 220 ... determination unit, 300 ... gaze authentication determination function, 310 ... gaze measurement unit, 320 ... display control unit, 330 ... determination unit, 500 ... detection unit, 700 ... execution unit, 1000 ... hand, 2000 ... arm

Claims (8)

Detecting means for detecting a hand and an arm existing in a predetermined detection range, and outputting detection data corresponding to the detected hand and arm ;
An area specifying means for specifying an object area including the detected hand and arm based on the detection data;
Reference point setting means for setting a reference point inside the identified object region;
Scanning means for scanning the detection range such that a scanning line surrounding the reference point extends from the reference point;
The scanning line is specified when the condition that the portion of the scanning line that overlaps the object region is a part of the scanning line and one continuous line is satisfied , and the identified scanning line is extended in the direction in which the arm extends. For recognizing the hand in the detection range on the basis of the position of the line when the amount of change in the length of the one continuous line by translating along a predetermined threshold is exceeded. An information processing apparatus comprising: a recognition space setting means for setting a recognition space. The recognition space setting means includes
Adjusting the position and shape of the scanning line when the amount of change in the length of the one continuous line exceeds the threshold within a range satisfying the condition on the basis of the position of the line ;
The information processing apparatus according to claim 1 for setting a center which is determined from the position with the reference, the interior of solid based on the length determined from the shape of the adjusted the scan lines as the recognition space. The information processing apparatus according to claim 2, wherein the adjustment unit includes a process of reducing the shape of the scanning line, and performs adjustment so that the scanning line and the finger of the specified hand do not overlap each other. When the scanning line and the specified finger of the specified hand overlap with each other by the process of reducing the shape of the scanning line, the adjusting unit increases the shape of the scanning line to increase the scanning line and the specified hand . The information processing apparatus according to claim 3, wherein adjustment is performed so that the finger does not overlap. Behavior measuring means for measuring the behavior of the hand in the recognition space based on the detection data;
On the basis of the behavior of the measured hand information processing apparatus according to any one of claims 1 to 4 further comprising determination means for determining an instruction to the device. An information processing apparatus according to claim 5 ;
An electronic device comprising: execution means for executing processing based on the determined instruction. Based on detection data output according to the detected hand and arm from the detection means for detecting the hand and arm existing in a predetermined detection range, the object region including the detected hand and arm is specified,
Setting a reference point within the identified object region;
Scanning the detection range such that a scanning line surrounding the reference point extends from the reference point;
The scanning line is specified when a condition that a part of the scanning line that overlaps the object region is a part of the scanning line and one continuous line is satisfied , and the specified scanning line is extended in a direction in which the arm extends. The movement of the hand in the detection range is recognized with reference to the position of the line when the amount of change in the length of the one continuous line due to parallel movement exceeds a predetermined threshold. Information processing method for setting a recognition space for Computer
Area specifying means for specifying an object area including the detected hand and arm based on detection data output according to the detected hand and arm from a detecting means for detecting a hand and an arm existing in a predetermined detection range When,
Reference point setting means for setting a reference point inside the identified object region;
Scanning means for scanning the detection range such that a scanning line surrounding the reference point extends from the reference point;
The scanning line is specified when the condition that the portion of the scanning line that overlaps the object region is a part of the scanning line and one continuous line is satisfied , and the identified scanning line is extended in the direction in which the arm extends. The movement of the hand in the detection range is recognized with reference to the position of the line when the amount of change in the length of the one continuous line due to parallel movement exceeds a predetermined threshold. A program for functioning as a recognition space setting means for setting a recognition space.

Images