JP5646532B2 - Operation input device, operation input method, and program - Google Patents

Description

  The present invention relates to an operation input device, an operation input method, and a program.

There is known an operation input device in which a user can operate an operation target device on a virtual plane set to overlap the operation target device between the operation target device and the user (for example, Patent Document 1). 2).
[Patent Literature 1] JP 2011-175617 A [Patent Literature 2] JP 2010-244480 A

  However, in the above apparatus, since the virtual plane and the operation target device overlap with each other when viewed from the user, a part of the body such as the user's hand to be operated overlaps the operation target device, and a part of the operation target device Is obstructed by the body and it is difficult to see the operation target device.

In the first aspect of the present invention, the three-dimensional information indicating the position of the body input from a three-dimensional sensor that detects the position of the user's body in the three-dimensional space, and the operation target device stored in advance From the position information, the user expects the operation target device in a space surrounded by a line segment connecting the position of the user's eye or head and the four corners of the display unit of the operation target device and the display unit. A plane identifying unit that identifies a prospective space that is a space , and that automatically identifies the position of one plane outside the prospective space, and three-dimensional information detected by the three-dimensional sensor, Detecting a change in the position of the user's body based on the specified position on the one plane, specifying a predetermined operation from the detected change, and selecting a selection signal corresponding to the specified operation as the operation target Operation to output to equipment Providing an operation input device and a specific part.

In the second aspect of the present invention, the three-dimensional information indicating the position of the body input from a three-dimensional sensor that detects the position of the user's body in the three-dimensional space, and the operation target device stored in advance From the position information, the user expects the operation target device in a space surrounded by a line segment connecting the position of the user's eye or head and the four corners of the display unit of the operation target device and the display unit. In the plane specifying step, the plane is specified as a prospective space, and a plane specifying step for automatically specifying the position of one plane outside the candidate space, and the three-dimensional information detected by the three-dimensional sensor, Detecting a change in the position of the user's body based on the specified position on the one plane, specifying a predetermined operation from the detected change, and selecting a selection signal corresponding to the specified operation as the operation target Machine Providing an operation input method and an operation specifying step of outputting to.

In the third aspect of the present invention, the three-dimensional information indicating the position of the body input from a three-dimensional sensor that detects the position of the user's body in the three-dimensional space, and the operation target device stored in advance From the position information, the user expects the operation target device in a space surrounded by a line segment connecting the position of the user's eye or head and the four corners of the display unit of the operation target device and the display unit. In the plane specifying procedure, the plane is specified as a prospective space, and a plane specifying procedure for automatically specifying the position of one plane outside the candidate space, and three-dimensional information detected by the three-dimensional sensor, Detecting a change in the position of the user's body based on the specified position on the one plane, specifying a predetermined operation from the detected change, and selecting a selection signal corresponding to the specified operation as the operation target Output to device It provides a program for executing the that operation determination procedure in the computer.

  It should be noted that the above summary of the invention does not enumerate all the necessary features of the present invention. In addition, a sub-combination of these feature groups can also be an invention.

1 is an overall configuration diagram of a system 10 including an operation input device 16. 2 is a block diagram illustrating a control system of the system 10. FIG. 4 is an example of an operation table 54 stored in a storage unit 48. 5 is a flowchart for explaining operation input processing by the operation input device 16. It is a flowchart explaining the plane specific process of step S10 by the plane specific part. 3 is a diagram for explaining a method for specifying a planar region 20. FIG. 1 is an overall side view of a system illustrating a planar area 20. It is a flowchart explaining the operation specific process of step S12 by the operation specific part 44. FIG. 10 is a diagram illustrating an image displayed on a display unit. It is a figure explaining the input method of a selection signal. It is a figure explaining the operation method of the display screen of a bookmark. It is a figure explaining the operation method of the display screen of a bookmark. It is a figure explaining the display method of the menu image.

  Embodiments of the present invention will be described below, but the following embodiments do not limit the invention according to the claims.

  FIG. 1 is an overall configuration diagram of a system 10 including an operation input device 16. As illustrated in FIG. 1, the system 10 includes a display device 12 that is an example of an operation target device, a three-dimensional sensor 14, and an operation input device 16.

  An example of the display device 12 is a liquid crystal display device, an organic EL display device, or the like. The display device 12 acquires images and sounds via television broadcasting and the Internet, and outputs the images and sounds. The display device 12 is provided with an input unit 22 such as a button operated by the user 18 in direct contact.

  The three-dimensional sensor 14 detects the position of the body of the user 18 in the three-dimensional space. An example of the three-dimensional sensor 14 is an image pickup apparatus that has a distance measuring function using phase difference autofocus and picks up an image. The three-dimensional sensor 14 is directed to the image display side of the display device 12. Accordingly, the three-dimensional sensor 14 acquires an image of the user 18 who is viewing the image displayed on the display device 12. The three-dimensional sensor 14 measures the distance to the user 18. The three-dimensional sensor 14 outputs the acquired image of the user 18 and three-dimensional information including the distance to the user 18 to the operation input device 16.

  The operation input device 16 specifies a plane area 20 that is a virtual plane for operation outside the expected space 21 from the image of the user 18 input from the three-dimensional sensor 14 and the distance to the user 18. . The expected space 21 is a space where the user 18 looks into the display device 12. An example of the prospect space 21 is a space surrounded by a line segment connecting the eyes of the user 18 and the four corners of the display unit of the display device 12 and the display unit itself. The operation input device 16 specifies an operation based on a change in the body position of the user 18 within the specified plane region 20 and outputs a selection signal corresponding to the operation to the display device 12.

  FIG. 2 is a block diagram illustrating a control system of the system 10. As shown in FIG. 2, in addition to the input unit 22, the display device 12 includes a control unit 32, an image generation unit 34, a display unit 36 that is an example of a display surface, a communication unit 38, and a sub display unit 40. With.

  When the button or the like is pressed by the user 18, the input unit 22 receives an input such as power on / off switching associated with the button and outputs the input to the control unit 32.

  The control unit 32 governs overall control of the display device 12. For example, the control unit 32 acquires the selection signal output from the operation input device 16 and the input unit 22 and controls the image generation unit 34.

  The communication unit 38 receives a television broadcast. The communication unit 38 further transmits information such as images to and from the server 26 via the Internet 24 and the like.

  The image generation unit 34 outputs the image and sound acquired by the communication unit 38 to the display unit 36. The display unit 36 displays the image input from the image generation unit 34 and outputs sound.

  The sub display unit 40 has a plurality of indicator lights. The sub display unit 40 displays various states of the display device 12, for example, an on / off state of the power of the display device 12, by turning on and off the plurality of indicator lamps.

  The operation input device 16 includes a plane specifying unit 42, an operation specifying unit 44, a notification unit 46, and a storage unit 48. The operation input device 16 may be a computer, and may read the program for executing the above functions from the outside or the storage unit 48 and execute the mechanisms.

  The plane specifying unit 42 specifies the prospective space 21 from the three-dimensional information indicating the position of the body input from the three-dimensional sensor 14 and the device coordinates 56 stored in advance in the storage unit 48. The plane specifying unit 42 further specifies the position of the plane area 20 outside the prospective space 21 when the user 18 detects an operation of designating a plane in the space. The plane specifying unit 42 stores area coordinates 52 indicating the position of the specified plane area 20 in the storage unit 48. The plane specifying unit 42 gives a warning by display or sound when the area coordinates 52 of the plane area 20 specified by the user 18 are within the expected space 21.

  The operation specifying unit 44 detects a change in the position of the body of the user 18 based on the position of the planar area 20 specified by the plane specifying unit 42 from the three-dimensional information detected by the three-dimensional sensor 14. The operation specifying unit 44 further specifies a predetermined operation based on the detected change based on the operation table 54, and outputs a selection signal corresponding to the specified operation to the display device 12.

  In this case, for example, the operation specifying unit 44 specifies an operation based on an operation while the body of the user 18 has the common area with the plane region 20 specified by the plane specifying unit 42. The operation specifying unit 44 determines the presence / absence of a shared point from the region coordinates 52 of the planar region 20 and the body coordinates of the user 18 calculated from the image of the user 18. When the operation specifying unit 44 determines that the shared point exists, the operation specifying unit 44 outputs presence information indicating the fact to the notification unit 46.

  When the presence information is input from the operation specifying unit 44, the notification unit 46 notifies the user 18 that the body is present in the planar area 20. In this case, the notification unit 46 may notify the presence by voice or the like, or may notify by display on the display unit 36 and the sub display unit 40 of the display device 12.

  The storage unit 48 stores region coordinates 52 that are coordinates indicating the position of the planar region 20 specified by the plane specifying unit 42. The storage unit 48 further stores an operation table 54 indicating determination conditions for the operation specifying unit 44 to specify an operation. The storage unit 48 further stores, as an example of position information of the display device 12, device coordinates 56 that are the coordinates of the four corners of the display unit 36 for specifying the prospective space 21.

  FIG. 3 is an example of the operation table 54 stored in the storage unit 48. As shown in FIG. 3, the operation table 54 associates operations with conditions for changing the body position of the user 18. Furthermore, an operation and an output signal when the operation is specified are associated with each other.

  In the example shown in FIG. 3, the pressing operation for pressing the target is associated with a change in the position of the body in which the amount of movement in the normal direction in the plane region 20 is greater than or equal to the amount of movement in the plane. A selection signal is associated as an output signal. In addition, the scroll operation for scrolling the display is associated with a change in the position of the body in which the amount of movement in the normal direction is smaller than the amount of movement in the plane in the plane region 20, and the scroll signal is used as an output signal. It is associated.

  FIG. 4 is a flowchart for explaining operation input processing by the operation input device 16. The operation input process is started when the power-on button of the input unit 22 is pressed. First, the plane specifying unit 42 executes a plane specifying process for specifying the plane area 20 (S10). Next, the operation specifying unit 44 executes an operation specifying process for specifying an operation based on a change in the body of the user 18 (S12).

  FIG. 5 is a flowchart for explaining the plane specifying process in step S10 by the plane specifying unit 42. FIG. 6 is a diagram illustrating a method for specifying the planar region 20. FIG. 7 is an overall side view of the system illustrating the planar area 20.

  The three-dimensional sensor 14 picks up an image at a predetermined frame rate and inputs it to the plane specifying unit 42 as three-dimensional information together with the distance measurement information. The plane specifying unit 42 detects the face of the user 18 from the three-dimensional information. It is determined whether or not it has been detected (S20). Thereby, the plane specifying unit 42 can omit the operation of detecting both hands of the user 18 when the user 18 is not in front of the display unit 36 or when the user 18 is not facing the display screen. The plane identifying unit 42 repeats Step S20 until it is determined that a face has been detected (S20: No).

  When it is determined that a face has been detected (S20: Yes), the plane specifying unit 42 detects the body of the user 18, particularly both hands, on the image (S22). The plane specifying unit 42 determines whether or not the plane area 20 is designated from the detected movements of both hands (S24).

  Whether or not the plane area 20 is designated is determined by the user's 18 hand having a predetermined shape or movement. For example, as shown in FIG. 6, the user 18 stands with the index finger and thumb of the hand 28 being substantially orthogonal, and obtains an image in which the thumb of one hand 28 and the index finger of the other hand 28 are parallel. When it does, the plane specific | specification part 42 determines with the plane area | region 20 having been designated (S24: Yes). The plane identifying unit 42 continues to detect the body of the user 18 until it is determined that a plane is designated (S24: No).

  When the plane specifying unit 42 determines that the plane area 20 is specified (S24: Yes), the plane specifying unit 42 calculates the area coordinates 52 specified by the user 18 (S26). In the example shown in FIG. 6, the plane specifying unit 42 calculates the coordinates of the intersection of the thumb of the hand 28 and the base of the index finger from the image from the position in the image and the distance measurement information, and a rectangle having the intersection as a vertex. Is identified. The plane specifying unit 42 may further be configured to set the thickness of the rectangle to be the plane area 20. An example of the thickness is several centimeters to several tens of centimeters. In other words, the planar region 20 has a thin rectangular parallelepiped shape. The plane specifying unit 42 calculates the coordinates of eight vertices of the rectangular parallelepiped as region coordinates 52.

  The plane specifying unit 42 detects the eyes of the user 18 from the three-dimensional information (S28). The plane specifying unit 42 specifies the prospective space 21 from the detected coordinates of the eyes of the user 18 and the device coordinates 56 stored in the storage unit 48 (S29). Here, not only the configuration for detecting the eyes of the user 18 but also the configuration for specifying the expected space 21 that is the visual field space area that the user 18 should secure in order to visually recognize the entire screen, the user 18's It may be configured to detect the shape of the head (for example, the shape above the neck) and specify the prospective space 21 from the position of the head and the device coordinates 56.

  The plane specifying unit 42 determines whether or not the plane area 20 specified in step 26 is outside the prospective space 21 (S32). In this case, the plane specifying unit 42 determines that the plane area 20 specified by the area coordinates 52 is outside the prospective space 21 when it does not have a common point with the prospective space 21, and if it has a common point, the prospective space 21. It is determined that

  If the plane specifying unit 42 determines that the region coordinates 52 are within the prospective space 21 (S32: No), the plane specifying unit 42 notifies a warning by sound or an image using the sub display unit 40 (S34). Thereafter, step S20 and subsequent steps are repeated. On the other hand, when the plane specifying unit 42 determines that the region coordinates 52 are outside the prospective space 21 (S32: No), the plane specifying unit 42 sets the coordinates of eight vertices of the plane region 20 as the region coordinates 52. Is stored in the storage unit 48 (S38). As described above, as shown in FIG. 7, the planar region 20 having a thickness is specified outside the prospective space 21.

  FIG. 8 is a flowchart for explaining the operation specifying process of step S <b> 12 by the operation specifying unit 44. FIG. 9 is a diagram for explaining an image displayed on the display unit 36. FIG. 10 is a diagram for explaining a selection signal input method. 11 and 12 are diagrams for explaining a method of operating the bookmark display screen.

  The operation specifying unit 44 outputs a menu display signal to the control unit 32 (S40). Accordingly, the control unit 32 outputs a menu display signal to the image generation unit 34, and the image generation unit 34 displays a menu image 60 on the display unit 36 as shown in FIG. The menu image 60 prompts the user 18 to select one of the menus. In an area other than the area where the menu image 60 is displayed on the display unit 36, an image acquired from the Internet or the like is displayed.

  Next, the operation specifying unit 44 determines whether or not the body position of the user 18 is within the planar region 20 based on the region coordinates 52 stored in the storage unit 48 (S42). In this case, the three-dimensional sensor 14 captures an image at a predetermined frame rate and inputs the image together with the distance measurement information to the operation specifying unit 44 as three-dimensional information. The operation specifying unit 44 calculates the body position of the user 18 from the three-dimensional information, and determines whether or not it has a shared point with the planar region 20 specified by the region coordinates 52.

  When the operation specifying unit 44 determines that the body position of the user 18 is not within the plane region 20 (S42: No), the operation specifying unit 44 repeats Step S42 until the end time has elapsed (S44: No). The operation specifying unit 44 ends the operation specifying process when it is determined that the body position of the user 18 is not in the plane area 20 and the end time has elapsed while repeating Step S42 (S44: Yes). To do.

  On the other hand, when the operation specifying unit 44 determines that the body position of the user 18 is within the plane region 20 (S42: Yes), the notification unit 46 outputs a display signal of the cursor 62 to the control unit 32 ( S46). Thereby, the image generation unit 34 displays the cursor 62 on the display unit 36 as shown in the upper diagram of FIG. 10.

  In FIG. 10, the upper diagram shows the display unit 36, and the lower diagram shows the planar area 20. The menu image 60 illustrated in the lower figure is not actually displayed in the planar area 20, but is illustrated for explanation.

  As shown in the upper diagram of FIG. 10, the user 18 moves the index finger 29 of the hand 28, for example, as shown in the lower diagram of FIG. 10, with the operation specifying unit 44 displaying the menu image 60 on the display unit 36. It is assumed that it is placed in the plane area 20. When this image is input from the three-dimensional sensor 14 to the operation specifying unit 44, the operation specifying unit 44 determines that the body of the user 18 exists in the planar area 20 in step S <b> 42.

  When the operation specifying unit 44 outputs the presence information indicating that the body of the user 18 is present in the planar region 20 to the notification unit 46, the notification unit 46 indicates that the body of the user 18 is present in the planar region 20. A cursor display instruction for notifying that this is being performed is output to the control unit 32. Accordingly, the control unit 32 instructs the image generation unit 34 to display a cursor. As a result, as shown in the upper diagram of FIG. 10, the image generation unit 34 displays the cursor 62 at the position of the display unit 36 corresponding to the tip portion of the index finger 29 in the planar region 20.

  In this case, the image generation unit 34 determines the “corresponding position” based on, for example, the size ratio between the planar region 20 and the display unit 36. In this case, the operation specifying unit 44 uses the coordinates of the lower left corner as the origin as a plane ignoring the thickness direction of the plane area 20. The operation specifying unit 44 uses the coordinates of the lower left corner of the display unit 36 as the origin. Further, the operation specifying unit 44 calculates the area ratio of the display unit 36 to the planar region 20. Next, the operation specifying unit 44 multiplies the coordinates in the planar region 20 of the index finger 29 by the square root of the area ratio to obtain a corresponding position in the display unit 36. For example, when the coordinates of the index finger 29 in the plane area 20 are (1, 1) and the area ratio is four times, the operation specifying unit 44 moves the cursor to the position of the coordinates (2, 2) on the display unit 36. indicate. When the plane area 20 is not similar to the display unit 36, the operation specifying unit 44 calculates the ratio of the lengths of corresponding sides of the display unit 36 to the plane area 20 and the coordinates in the plane area 20 Multiply the length ratios to obtain corresponding positions in the display 36.

  Next, the operation specifying unit 44 determines whether or not the body position of the user 18 has changed in the plane area 20 (S48). The change is calculated, for example, by comparing three-dimensional information within a predetermined time in time series. When the operation specifying unit 44 determines that the body position of the user 18 has not changed in the plane region 20 (S48: No), the operation specifying unit 44 executes Step S44.

  On the other hand, if the operation specifying unit 44 determines that the body position of the user 18 has changed in the plane area 20 (S48: Yes), does the change satisfy any condition of the operation table 54? It is determined whether or not (S50). In the example shown in FIG. 3, the condition is alternatively satisfied by comparing the normal movement distance and the in-plane movement distance. In this example, the operation specifying unit 44 determines that the normal of the body position is The moving distance and the in-plane moving distance are calculated and compared (same step). Hereinafter, an example of the conditions shown in FIG. 5 will be described.

  The normal movement distance of the body position is calculated from the components in the normal direction of the planar region 20 out of the movement distance of the body position. Here, since the plane area 20 has a thickness, the operation specifying unit 44 can calculate the normal movement distance by moving the body position in the thickness direction. When the planar area 20 does not have a thickness, the operation specifying unit 44 calculates the normal movement distance by, for example, a change in shape such as a diameter of a finger or a circumference crossing the planar area 20. The in-plane movement distance of the body position is calculated from the in-plane component of the planar region 20 out of the movement distance of the body position.

  If the operation specifying unit 44 determines that the normal movement distance of the body position is equal to or greater than the in-plane movement distance (S50: Yes), the operation specifying unit 44 determines that the operation of the user 18 is a pressing operation with reference to the operation table 54, and is selected. A signal is output to the control part 32 (S52), and it returns to step S42. In this case, the operation specifying unit 44 selects the selection signal of any menu based on the relationship between the in-plane position of the index finger 29 in the planar region 20 and the in-plane position of each menu of the menu image 60 on the display unit 36. To identify.

  For example, it is assumed that the user 18 moves the index finger 29 in the normal direction of the plane region 20 from the state of FIG. When the operation specifying unit 44 acquires an image in which the index finger 29 moves in the normal direction from the three-dimensional sensor 14, the operation specifying unit 44 determines that the bookmark is pressed in the menu and selects a bookmark display command selection signal. 32. The selection signal output by the operation specifying unit 44 includes a command corresponding to the selected menu, for example, a power on / off command, a bookmark display command, and the like. The operation specifying unit 44 may specify the pressing operation regardless of the direction of movement in the normal direction, or may detect the direction of movement in the normal direction and identify the pressing operation and the pull-in operation separately. Also good.

  When acquiring the selection signal, the control unit 32 causes the image generation unit 34 to display an image corresponding to the selection signal. For example, as illustrated in FIG. 10, when a selection signal related to the pressing operation of the button of the bookmark 66 is input to the control unit 32, the image generation unit 34 displays a list of bookmarks 66 as illustrated in FIG. 11. To display.

  On the other hand, when determining that the normal movement distance of the body position is less than the in-plane movement distance (S50: No), the operation specifying unit 44 determines that the operation of the user 18 is a scroll operation and generates a scroll signal including the scroll distance. The data is output to the control unit 32 (S54), and the process returns to step 42. For example, when the bookmark 66 is displayed as shown in FIG. 11 and a vertical scroll signal is input to the control unit 32, the image generation unit 34 associates the list of bookmarks 66 with the scroll distance. Scroll up and down.

  In FIG. 11, the upper diagram shows the display unit 36, and the lower diagram shows the planar area 20. FIG. 11 is a diagram before the operation, and FIG. 12 is a diagram after the operation. As described above, when a bookmark display command selection signal is input to the control unit 32, the control unit 32 outputs a bookmark display command to the image generation unit 34. As a result, the image generation unit 34 displays a list of bookmarks 64 on the display unit 36 as shown in the upper diagram of FIG.

  As shown by the solid line in the lower diagram of FIG. If the operation specifying unit 44 determines that the tip of the index finger 29 is placed in the planar area 20, the operation specifying unit 44 instructs the cursor display together with the coordinates corresponding to the tip of the index finger 29. As a result, the image generation unit 34 displays the cursor 62 at a position corresponding to the tip of the index finger 29 as shown in the upper diagram of FIG.

  Next, it is assumed that the user 18 moves the index finger 29 in the plane direction in the plane area 20 and moves the index finger 29 to a position indicated by a dotted line in the lower diagram of FIG. The operation specifying unit 44 specifies the moving distance and moving direction of the index finger 29. The moving direction here includes two directions along both sides of the planar region 20 and an oblique direction intersecting with both sides. The operation specifying unit 44 determines that the scroll operation is performed when the in-plane moving distance is longer than the moving distance in the normal direction of the planar region 20. The operation specifying unit 44 outputs a scroll signal including the moving distance and the moving direction to the control unit 32.

  Here, the movement distance included in the scroll signal may be equal to the movement distance of the index finger 29 or may be different from the movement distance of the index finger 29. When the movement distance included in the scroll signal is different from the movement distance of the index finger 29, for example, the operation specifying unit 44 uses, as the scroll signal, a value obtained by integrating the magnification of the display unit 36 with respect to the planar area 20 to the movement distance of the index finger 29. The included movement distance. The control unit 32 outputs a scroll signal to the image generation unit 34.

  As a result, as shown in FIG. 12, the image generation unit 34 moves the list of bookmarks 64 within the display unit 36 by a direction and a movement distance corresponding to the movement distance and the movement direction included in the scroll signal. The image generation unit 34 also moves the cursor 62 in accordance with the direction and the movement distance included in the scroll signal.

  FIG. 13 is a diagram for explaining an example of a method for displaying the menu image 60. In FIG. 13, the upper diagram shows the display unit 36, and the lower diagram shows the planar area 20. As illustrated in FIG. 13, when the menu image 60 is unnecessary, the image generation unit 34 may display only the end of the menu image 60 on the display unit 36. Alternatively, the image generation unit 34 may delete the menu image 60 from the display unit 36 when the menu image 60 is unnecessary. The case where the menu image 60 is unnecessary includes a case where the user 18 instructs that the menu image 60 is not necessary, a case where the user 18 does not input an instruction via the menu image 60 for a certain period of time, and the like.

  When the operation specifying unit 44 detects that the user 18 has moved the index finger 29 from the right side to the left side in the plane area 20 as shown in the lower diagram of FIG. The unit 44 determines that the operation is a scroll operation. Thereafter, the operation specifying unit 44 outputs a scroll signal including the moving distance and the moving direction to the control unit 32. As a result, the image generation unit 34 displays the menu image 60 as shown in FIG.

  As described above, according to the present embodiment, the operation input device 16 includes the plane specifying unit 42 that specifies the plane area 20 that is a virtual plane for operation at a position specified by the user 18 outside the prospective space 21. Thus, the user 18 can operate the display device 12 while viewing the display content without observing the hand, without blocking the display unit 36 of the display device 12 with his / her body.

  When the plane area 20 specified by the user 18 is inside the prospective space 21, the plane specifying unit 42 warns. Thereby, the user 18 can reliably set the planar area 20 outside the prospective space 21.

  In the operation input device 16, the operation specifying unit 44 determines whether the operation of the user 18 is a pressing operation or a scrolling operation based on the magnitude relationship between the normal moving distance and the in-plane moving distance. Thereby, the user 18 can operate the display apparatus 12 with the same operation feeling as operating a tablet terminal etc. in direct contact.

  In the operation input device 16, the plane specifying unit 42 specifies the plane area 20 having a thickness. Thereby, the operation specifying unit 44 does not calculate from the shape of the user's body crossing the plane area 20, but from the movement of the body position in the normal direction of the user to the normal movement distance of the user's body position. Can be easily calculated.

  8 to 13, the index finger 29 of the hand 28 is detected as the body of the user 18, but the body is not limited to this. It may be the hand itself, the arm, or another part of the body. In the operation table 54, different operations may be assigned depending on whether a finger is detected or another part of the body is detected.

  Moreover, in the said embodiment, when the plane specific | specification part 42 detects the operation | movement which designates one plane on the space by the user 18, based on the position which the user 18 specified, the position of the plane area | region 20 is specified. To do. Instead of this, the plane specifying unit 42 may automatically specify the plane area 20. In this case, the plane identifying unit 42 identifies the position of the plane area 20 at a predetermined relative position outside the prospective space 21 and with respect to the prospective space 21. In this case, it is preferable that the plane specifying unit 42 specifies the plane region 20 that is non-parallel to the display unit 36. For example, when the display surface of the display unit 36 is vertically upright, the planar region 20 is set to be horizontal.

  In the said embodiment, the plane specific | specification part 42 specified the expectation space 21 based on the position of the user's 18 eye detected from three-dimensional information. Instead, the plane specifying unit 42 may detect the face of the user 18 from the three-dimensional information and specify the prospective space 21 based on the position of the face.

  In the above embodiment, the three-dimensional sensor 14 has a distance measuring function by phase difference autofocus. Instead of this or in addition to this, the three-dimensional sensor 14 may have a distance measuring function by contrast autofocus using image contrast information.

  In the above embodiment, the display device 12 is an example of the operation target device. The operation target device is not limited to the display device 12, and may be another device such as an air conditioner or a lighting device.

  Although the embodiments of the present invention have been described above, the technical scope of the present invention is not limited to the scope described in the above embodiments. It will be apparent to those skilled in the art that various modifications or improvements can be added to the above-described embodiment. It is apparent from the scope of the claims that the embodiments added with such changes or improvements can be included in the technical scope of the present invention.

  The order of execution of each process such as operations, procedures, steps, and stages in the apparatus, system, program, and method shown in the claims, the description, and the drawings is particularly “before” or “prior to”. It should be noted that the output can be realized in any order unless the output of the previous process is used in the subsequent process. Regarding the operation flow in the claims, the description, and the drawings, even if it is described using “first”, “next”, etc. for convenience, it means that it is essential to carry out in this order. It is not a thing.

  10 system, 12 display device, 14 3D sensor, 16 operation input device, 18 user, 20 plane area, 21 prospect space, 22 input section, 24 Internet, 26 server, 28 hands, 29 fingers, 32 control section, 34 Image generation unit, 36 display unit, 38 communication unit, 40 secondary display unit, 42 plane identification unit, 44 operation identification unit, 46 notification unit, 48 storage unit, 52 area coordinate, 54 operation table, 56 device coordinate, 60 menu image , 62 Cursor, 64 Bookmark, 66 Bookmark

Claims (3)

From the three-dimensional information indicating the position of the body input from the three-dimensional sensor that detects the position of the user's body in the three-dimensional space, and the position information of the operation target device stored in advance, the user's eyes Alternatively, a space surrounded by a line segment connecting the position of the head and the four corners of the display unit of the operation target device and the display unit is specified as an expected space that is a space where the user looks at the operation target device, and , a plane surface specifying portion for automatically identifying the location of a plane outside of the prospective space,
From the three-dimensional information detected by the three-dimensional sensor, a change in the position of the user's body with respect to the position of the one plane specified by the plane specifying unit is detected, and predetermined from the detected change. An operation input device comprising: an operation specifying unit that specifies a specified operation and outputs a selection signal corresponding to the specified operation to the operation target device. From the three-dimensional information indicating the position of the body input from the three-dimensional sensor that detects the position of the user's body in the three-dimensional space, and the position information of the operation target device stored in advance, the user's eyes Alternatively, a space surrounded by a line segment connecting the position of the head and the four corners of the display unit of the operation target device and the display unit is specified as an expected space that is a space where the user looks at the operation target device, and , a plane surface specifying step of automatically identify the position of one plane out of the prospective space,
From the three-dimensional information detected by the three-dimensional sensor, a change in the position of the user's body based on the position of the one plane specified in the plane specifying step is detected, and predetermined from the detected change. An operation input method comprising: an operation specifying step of specifying a specified operation and outputting a selection signal corresponding to the specified operation to the operation target device. From the three-dimensional information indicating the position of the body input from the three-dimensional sensor that detects the position of the user's body in the three-dimensional space, and the position information of the operation target device stored in advance, the user's eyes Alternatively, a space surrounded by a line segment connecting the position of the head and the four corners of the display unit of the operation target device and the display unit is specified as an expected space that is a space where the user looks at the operation target device, and , a plane surface specifying procedure automatically identify the position of one plane out of the prospective space,
A change in the position of the user's body based on the position of the one plane specified in the plane specifying procedure is detected from the three-dimensional information detected by the three-dimensional sensor, and is determined in advance from the detected change. A program for causing a computer to execute an operation specifying procedure for specifying a specified operation and outputting a selection signal corresponding to the specified operation to the operation target device.

Images