WO2016121708A1

WO2016121708A1 - Input system, input device, input method, recording medium

Info

Publication number: WO2016121708A1
Application number: PCT/JP2016/052054
Authority: WO
Inventors: 竜太郎谷村
Original assignee: Ｎｅｃソリューションイノベータ株式会社
Priority date: 2015-01-26
Filing date: 2016-01-25
Publication date: 2016-08-04
Also published as: JPWO2016121708A1

Abstract

An input device (1) has: a contact detection unit (12) that detects the contact of a user's hand on the upper surface of a table; an angle detection unit (13) that detects the contact elevation angle and the contact orientation angle of the user's hand contacting the upper surface of the table with respect to the upper surface of the table; and a display control unit (15) that, on the basis of the contact elevation angle and the contact orientation angle detected by the angle detection unit (13), transmits to a projection device (3) instruction information for displaying an operation screen at a prescribed position that uses, as a reference, the contact position with respect to the user's hand on the upper surface of the table. The projection device (3) projects an operation screen on the upper surface of the table in accordance with the instruction information received from the input device (1).

Description

INPUT SYSTEM, INPUT DEVICE, INPUT METHOD, AND RECORDING MEDIUM

The present invention relates to an input system, an input device, an input method, and a recording medium.

In Patent Document 1, data (numerical values and instructions) are displayed according to a user's operation on a monitor device arranged in front of the operator and a virtual plane (operation surface) arranged between the operator and the monitor device. An input system is disclosed that includes an input device for inputting (including) an input device.

In Patent Document 2, an image of an operation screen is displayed on the upper surface of a desk, a distance image including pixel values indicating a distance to a detection target is generated, and an object contacts the upper surface of the desk based on the distance image. An information processing apparatus that determines this and inputs data corresponding to the contact position is disclosed.

Patent Document 3 discloses an input device that displays a virtual operation surface in a three-dimensional manner, specifies a positional relationship between the virtual operation surface and a user's hand or finger, and inputs data based on the specified positional relationship. ing. In this input device, the placement of the virtual operation surface is determined based on the position of a specific part of the operator set in advance, such as the position of the joint of the operator.

Japanese Unexamined Patent Publication No. 2011-039844 JP2011-253255A JP 2014-056462 A

In the technique described in Patent Document 1, the layout of the virtual operation surface is fixed by the layout of the monitor. Also in the technique described in Patent Document 2, the position of the operation screen is fixed to the upper surface of the desk. Therefore, the operator may feel it difficult to operate the operation screen.

In addition, the technique described in Patent Document 3 specifies the operator's posture only by the position of a specific part of the operator, and thus the actual posture of the operator may not be sufficiently specified. For this reason, the arrangement of the operation surface is not suitable for the posture of the operator, and the operator may feel it difficult to operate.

The present invention has been made in view of the above circumstances, and an object thereof is to provide an input system with high operability.

An input system according to a first aspect of the present invention includes:
Contact detection means for detecting contact of an object with a projection target surface of an operation screen on an object, and angle detection means for detecting a contact elevation angle and a contact azimuth angle of the object in contact with the projection target surface with respect to the projection target surface And instruction information for displaying the operation screen at a predetermined position based on the contact position of the projection target surface with the object based on the contact elevation angle and the contact azimuth angle detected by the angle detection means. An input device having display control means for transmitting to the device;
The projection device that projects the operation screen onto the object according to the instruction information received from the input device;
It is characterized by providing.

An input device according to a second aspect of the present invention is:
An input device connected to the projection device,
Contact detection means for detecting contact of an object with a projection target surface of an operation screen on the object;
An angle detecting means for detecting a contact elevation angle and a contact azimuth angle of the object in contact with the projection target surface with respect to the projection target surface;
Based on the contact elevation angle and the contact azimuth angle detected by the angle detection means, instruction information for displaying the operation screen at a predetermined position based on the contact position of the projection target surface with the object is used as the projection device. Display control means for transmitting to,
It is characterized by providing.

An input method according to the third aspect of the present invention includes:
A contact detection step of detecting contact of an object with a projection target surface of an operation screen on an object executed by an input device;
An angle detection step of detecting a contact elevation angle and a contact azimuth angle of the object in contact with the projection target surface with respect to the projection target surface;
Based on the contact elevation angle and the contact azimuth angle detected in the angle detection step, instruction information for displaying the operation screen at a predetermined position based on the contact position of the projection target surface with the object is displayed on the projection device. A display control step to transmit;
Projecting the operation screen onto the object according to the instruction information received from the input device executed by the projection device;
It is characterized by providing.

The recording medium according to the fourth aspect of the present invention is:
A computer connected to the projection device,
Contact detection means for detecting contact of an object with a projection target surface of an operation screen on an object;
An angle detection means for detecting a contact elevation angle and a contact azimuth angle of the object in contact with the projection target surface with respect to the projection target surface;
Based on the contact elevation angle and the contact azimuth angle detected by the angle detection means, instruction information for displaying the operation screen at a predetermined position based on the contact position of the projection target surface with the object is used as the projection device. Display control means for transmitting to
Stores a program characterized by functioning as

According to the present invention, based on the contact elevation angle and the contact azimuth angle, the operation screen is displayed at a predetermined position based on the contact position with the object on the projection target surface (for example, the operator's hand). Accordingly, it is possible to display the operation screen at a predetermined position that is easy for the operator to operate. Thereby, the operativity of an input device can be improved.

It is a block diagram which shows the structure of the input system which concerns on embodiment of this invention. It is a block diagram which shows the function structure of the input device which concerns on embodiment. It is a figure for demonstrating the contact elevation angle and contact azimuth | direction angle which concern on embodiment. It is a figure for demonstrating the contact elevation angle and contact azimuth | direction angle which concern on embodiment. It is a figure for demonstrating the contact elevation angle and contact azimuth | direction angle which concern on embodiment. It is a figure which shows an example of the method of detecting the contact point of the object which contacted the target object concerning embodiment, a contact elevation angle, and a contact azimuth. It is a figure which shows an example of the method of detecting the contact area | region of the object which contacted the target object concerning embodiment, a contact elevation angle, and a contact azimuth. It is a figure which shows the vicinity part of the operator's hand in FIG. It is a figure which shows the vicinity part of the operator's hand in FIG. It is a figure which shows the example of a display of the screen corresponding to the selected selection item which concerns on embodiment. It is a flowchart which shows an example of the operation | movement of the input process which concerns on embodiment. It is a block diagram which shows an example of the hardware constitutions of the input device which concerns on embodiment.

Hereinafter, embodiments of the present invention will be described. In the present embodiment, a case will be described in which an operation screen is projected on the upper surface of the table, and the user performs an input operation by bringing his / her hand into contact with the upper surface of the table.

The input system 100 includes an input device 1, a depth sensor 2, and a projection device 3, as shown in FIG. The input device 1 is connected to the depth sensor 2 and the projection device 3. The depth sensor 2 detects the depth of the object including the upper surface 41 of the table on which the operation screen is projected (distance from the depth sensor 2), generates distribution information of the detected depth value, and transmits it to the input device 1.

The input device 1 analyzes the distribution information of the depth value received from the depth sensor 2, and determines whether or not the user's hand has touched the upper surface 41 of the table. When the input device 1 detects that the hand has touched the upper surface 41, the input device 1 detects a contact point, a contact line, or a contact area. When the input device 1 detects that the tip of one finger contacts the upper surface 41, the input device 1 detects the tip of the finger as a contact point. Further, when the input device 1 detects that the side surface of the hand is in contact with the upper surface 41, the input device 1 detects the side surface of the hand as a contact line. Further, when the input device 1 detects that the palm has contacted the upper surface 41, the input device 1 detects the palm as a contact area. When the input device 1 detects that the hand is in contact with the upper surface 41, the input device 1 detects an elevation angle and an azimuth angle (hereinafter referred to as contact elevation and contact azimuth angle) formed with the upper surface 41 of the finger, palm, or arm. The input device 1 determines a direction in which the user can easily move a finger, palm, or arm (hereinafter referred to as an operation direction) from the detected contact elevation angle and contact azimuth angle. The input device 1 transmits instruction information for instructing to display an operation screen suitable for the determined operation direction at a predetermined position based on the position of the contact point, the contact line, or the contact area.

The operation screen is, for example, a list (menu) screen of selection items (menu items) associated with functions provided in the input device 1. The projection device 3 projects an operation screen (menu) on the upper surface 41 of the table according to the instruction information received from the input device 1. When the operation screen is projected, the user can select (instruct) the selection item by moving the contact point, the contact line, or the contact area on the upper surface 41 of the table. The input device 1 analyzes the distribution information of the depth value received from the depth sensor 2, and specifies the instructed selection item.

The input device 1 transmits to the projection device 3 instruction information that instructs to display a screen corresponding to the selected item at a predetermined position based on the position of the contact point, the contact line, or the contact area. The projection device 3 projects a screen corresponding to the selected item on the upper surface 41 of the table according to the instruction information received from the input device 1. In FIG. 1, the input system 100 includes one depth sensor 2 and one projection device 3. The present invention is not limited to this, and a plurality of depth sensors 2 and a plurality of projection devices 3 may be provided. Further, it may be detected that a plurality of user's hands are in contact with the upper surface of the table.

2, the input device 1 includes a depth acquisition unit 11, a contact detection unit 12, an angle detection unit 13, a storage unit 14, and a display control unit 15.

The depth acquisition unit 11 receives depth value distribution information from the depth sensor 2. The depth acquisition unit 11 sends the received depth value distribution information to the contact detection unit 12.

The contact detection unit 12 analyzes the distribution information of the depth value received from the depth acquisition unit 11, and extracts the user's skeleton from the user's area. The contact detection unit 12 determines whether or not a hand has touched the upper surface 41 of the table from the user's region and skeleton. The contact detection unit 12 determines that the hand has touched the upper surface 41 when the difference between the depth value of the upper surface 41 of the table and the depth value of the contour of the hand is equal to or smaller than the threshold value. This threshold value is set to a value corresponding to the resolution + α of the depth sensor 2, for example. When it is determined that the hand touches the upper surface 41, the contact detection unit 12 detects a contact point, a contact line, or a contact area from the user's area and skeleton (a detection method will be described later). The contact detection unit 12 sends contact position information indicating the position of the detected contact point, contact line, or contact area to the display control unit 15. In addition, the contact detection unit 12 sends analysis result information indicating the analysis result of the distribution information of the depth value to the angle detection unit 13.

The angle detection unit 13 detects the contact elevation angle and the contact azimuth angle from the analysis result information received from the contact detection unit 12 as follows. When the tip (point) of one finger contacts the table, as shown in FIG. 3A, the angle detection unit 13 causes the upper surface 41 (in the direction V1 along the finger with the contact point P1 as a base point) A contact elevation angle φ1 and a contact azimuth angle θ1 with respect to the projection target surface or tangent plane are obtained. Here, the contact elevation angle φ1 and the contact azimuth angle θ1 are angles with respect to the reference direction V0 along the upper surface 41 of the table. When the side surface of the hand contacts the table with a line, the angle detection unit 13, as shown in FIG. 3B, of the normal line V2 of the plane PL2 along the palm of the user with reference to the contact line L1 on the side surface of the hand, A contact elevation angle φ2 and a contact azimuth angle θ2 with respect to the tangent plane (the upper surface 41 of the table) are obtained. When the palm touches the upper surface 41 of the table with the surface, as shown in FIG. 3C, the angle detection unit 13 makes a tangential plane (the upper surface 41 of the table) in the direction V3 along the user's forearm with the contact area as a base point. A contact elevation angle φ3 and a contact azimuth angle θ3 are obtained.

The angle detection unit 13 sends angle information indicating the detected contact elevation angle and contact azimuth angle to the display control unit 15. Based on the angle information received from the angle detection unit 13, the display control unit 15 determines the operation direction by a method described later. The display control unit 15 generates an operation screen suitable for the determined operation direction. The display control unit 15 generates an operation screen that is inclined from the reference direction by an angle between the determined operation direction and a preset reference direction.

The display control unit 15 receives the contact position information from the contact detection unit 12, and at a predetermined position (a position where the user can easily move the hand) based on the position of the contact point, the contact line, or the contact area indicated by the contact position information. Instruction information for displaying the operation screen is generated. The display control unit 15 sets the position of the operation screen to a position that is higher than the height of the contact point, the contact line, or the contact area by a preset distance. Then, the display control unit 15 transmits the generated instruction information to the projection device 3. The projection device 3 projects an operation screen (menu) on the upper surface 41 of the table according to the instruction information received from the input device 1.

When it is determined that the user's hand has touched the upper surface of the table in a state where the operation screen is projected on the upper surface 41 of the table, the contact detection unit 12 indicates the position of the detected contact point, contact line, or contact area. The selection item selected by the user is specified from the contact position information. The contact detection unit 12 sends selection information indicating the selected selection item to the display control unit 15.

The display control unit 15 generates instruction information for instructing to display an operation screen corresponding to the selection item indicated by the selection information received from the contact detection unit 12, and transmits the instruction information to the projection device 3. The projection device 3 projects a screen corresponding to the selected selection item on the upper surface 41 of the table according to the instruction information received from the input device 1. Here, a method for detecting the user's hand that has touched the upper surface of the table from the distribution information of the depth value will be described.

FIG. 4 is a diagram illustrating an example of a method for detecting a contact point, a contact elevation angle, and a contact azimuth angle of an object that has contacted an object according to the embodiment. In the example of FIG. 4, the user performs a touch operation that makes contact with the upper surface 41 of the table with one finger. The depth value distribution information includes a table region T and a floor region F having substantially constant depth values.

First, the contact detection unit 12 extracts points where a difference of a predetermined value or more appears from a substantially constant depth value of the continuous table region T or floor region F from the distribution information of depth values. The contact detection unit 12 connects the extracted points to extract the user's contour. The contact detection unit 12 separates the user area U surrounded by the user's outline. In the example of FIG. 4, the user's head is indicated by a broken line for easy understanding.

In the user area U, the contact detection unit 12 replaces the pixel value with a larger pixel value as the distance to the contour portion of the user area U is shorter. When the point where the pixel value becomes maximum is extracted and connected, the user's skeleton S is extracted. The extraction method of the skeleton S is not limited to this, and thinning or central axis conversion generally used in image processing may be used.

The contact detection unit 12 detects the end points P1 to P4 of the skeleton S. When there is one end point whose difference from the depth value of the table region T is not more than a threshold value among the end points P1 to P4, the contact detection unit 12 detects the position of the end point as a contact point. In the example of FIG. 4, the end point P1 is a user's fingertip that has contacted the upper surface of the table, and the position of the end point P1 is a contact point. The contact detection unit 12 sends analysis result information indicating these analysis results to the angle detection unit 13.

In the example of FIG. 4, the angle detection unit 13 determines the contact elevation angle of the user's finger from the direction from the end point P1 (fingertip) of the skeleton S indicated by the analysis result information received from the contact detection unit 12 to the base end portion of the finger. Detect the contact azimuth. The angle detection unit 13 sends angle information indicating the contact elevation angle and contact azimuth angle of the user's finger to the display control unit 15. The display control unit 15 determines the operation direction based on the angle information received from the angle detection unit 13.

FIG. 5 is a diagram illustrating an example of a method for detecting a contact area, a contact elevation angle, and a contact azimuth angle of an object in contact with an object according to the embodiment. In the example of FIG. 5, parts other than the user's hand and forearm are omitted. As in the case of FIG. 4, the contact detection unit 12 determines the user area U using the depth value distribution information, and extracts the skeleton S. As shown in FIG. 5, when the user performs a lap operation in which the user touches the upper surface of the table with the palm of the hand, the contact detection unit 12 detects the end points P 5 to 9 from the skeleton S.

For example, when there are five end points whose difference from the depth value of the table region T is equal to or less than the threshold value, the contact detection unit 12 determines that the operation is a lapping operation. In the example of FIG. 5, since the difference between the end point P5 to the end point 9 is not more than the threshold value of the table region T, the contact detection unit 12 determines that the lap operation is performed and the difference from the depth value of the table region T. An area of the palm of the user for which is less than or equal to the threshold is detected as a contact area. The contact detection unit 12 sends analysis result information indicating these analysis results to the angle detection unit 13.

The angle detection unit 13 detects the contact elevation angle and contact azimuth angle of the user's forearm from the direction from the wrist to the elbow of the skeleton S indicated by the analysis result information received from the contact detection unit 12. The angle detection unit 13 sends angle information indicating the contact elevation angle and contact azimuth angle of the user's forearm to the display control unit 15. The display control unit 15 determines the operation direction based on the angle information received from the angle detection unit 13. Subsequently, display of the operation screen will be described.

FIG. 6A is a diagram illustrating a display example of the operation screen according to the embodiment. For example, as shown in FIG. 5, when the user performs a lap operation on the upper surface of the table, the display control unit 15 determines the tangent plane of the user's forearm as shown in FIG. 6A based on the angle information received from the angle detection unit 13. The direction toward the top is up, and the top, bottom, left and right are determined. Upon receiving the contact position information from the contact detection unit 12, the display control unit 15 instructs to display the operation screen at a predetermined position based on the position of the contact region (palm region) indicated by the contact position information. Information is generated and transmitted to the projection device 3. For example, the operation screen is displayed at a position 10 cm above the contact area.

The projection device 3 projects an operation screen as shown in FIG. 6A onto a predetermined position on the upper surface 41 of the table according to the instruction information received from the input device 1. In the example of FIG. 6A, the user can select selection items A, B, C, and D associated with each function of the input device 1 on the operation screen.

6B, when the user selects the selection item B, the contact detection unit 12 of the input device 1 specifies that the selection item B has been selected from the contact position information indicating the position of the detected contact point. The contact detection unit 12 sends selection information indicating the selected selection item B to the display control unit 15. The display control unit 15 generates instruction information for instructing to display a screen corresponding to the selection item B indicated by the selection information received from the contact detection unit 12, and transmits the instruction information to the projection device 3. The projection device 3 projects a screen corresponding to the selected selection item on the upper surface of the table according to the instruction information received from the input device 1.

FIG. 7 is a diagram illustrating a display example of a screen corresponding to the selected selection item according to the embodiment. For example, when the function corresponding to the selection item B selected by the user in FIG. 6B is a function for changing the temperature of the air conditioner, the screen shown in FIG. 7 is a screen for changing the temperature of the air conditioner. In this case, the user raises or lowers the temperature of the air conditioner by operating an operation button (open arrow in FIG. 7) displayed on the screen for changing the temperature of the air conditioner.

The contact detection unit 12 of the input device 1 determines the user region U using the depth value distribution information and extracts the skeleton S, as in the case of FIGS. When the side surface of the user's hand contacts the top surface of the table, the contact detection unit 12 detects a contact line on the side surface of the user's hand. For example, the contact detection unit 12 detects, as a contact line, a line connecting points where the difference from the depth value of the table region T is equal to or less than a threshold value in the contour of the user. The contact detection unit 12 sends analysis result information indicating these analysis results to the angle detection unit 13.

The angle detection unit 13 detects the contact elevation angle and the contact azimuth of the palm of the user from the distribution information of the depth values of the palm region of the skeleton S or the user region U indicated by the analysis result information received from the contact detection unit 12. The angle detection unit 13 sends angle information indicating the contact elevation angle and the contact azimuth angle of the palm of the user to the display control unit 15. The display control unit 15 determines the operation direction based on the angle information received from the angle detection unit 13.

Here, the right palm is tilted to the left and the putting operation is performed, and the angle formed with the tangent to the normal line of the plane along the right palm of the user is the left when the angle is obtuse and the right when the angle is acute. It is assumed that which hand is used to tilt the left or right to perform the pad operation is determined in advance. Similarly to the above-described method for determining the operation direction in the lap operation, in the pad operation, the direction toward the tangential plane of the user's forearm may be set upward, and the top, bottom, left, and right may be determined. Alternatively, with the user's palm as a reference, the wrist direction may be down (front), the right may be high (large), and the left may be low (small).

The display control unit 15 generates an operation screen in accordance with the determined operation direction, and transmits to the projection device 3 instruction information for instructing display at a predetermined position based on the position of the contact line. The projection device 3 projects an operation screen according to the instruction information. The operation screen is, for example, a position that is 20 cm away from the position of the contact line and the tangent plane of the normal line of the plane along the right palm of the user, and a plane along the right palm of the user. Is displayed at a position 10 cm away on the side where the angle formed with the tangent plane of the normal line is an acute angle.

In the example of FIG. 7, the left pad operation is an operation to lower the temperature, the right pad operation is an operation to increase the temperature, a left arrow with-displayed on the left side of the contact line, and a + on the right side of the contact line. Project an operation screen that displays a right-pointing arrow. In the pad operation, the side where the normal formed by the plane along the right palm of the user and the tangent plane makes an obtuse angle is easy to make a hand shadow. By projecting the operation screen in such a way, it is possible to prevent the operation screen from becoming invisible as a hand shadow. In addition, since the positive / negative direction by a pad operation is uniquely determined, the display of an operation direction may be one side.

The function of the input device 1 is not limited to the display of the screen corresponding to the selection item selected by the user, and may be any function such as a function of controlling an electric device. The input device 1 may perform a function corresponding to the selected selection item.

FIG. 8 is a flowchart showing an example of the operation of the input process according to the embodiment. The input process of FIG. 8 starts when the power of the depth sensor 2, the input device 1, and the projection device 3 is turned on.

The depth sensor 2 detects the depth of the object (step S11), generates distribution information of the detected depth value, and transmits it to the input device 1 (step S12). If the power is not turned off (step S13; NO), the depth sensor 2 returns to step S11 and repeats steps S11 to S13. The depth sensor 2 ends the process when the power is turned off (step S13; YES).

The depth acquisition unit 11 of the input device 1 receives the distribution information of the depth value from the depth sensor 2 (step S21) and sends it to the contact detection unit 12. The contact detection unit 12 analyzes the depth value distribution information received from the depth acquisition unit 11, and determines whether or not the user's hand has touched the upper surface of the table (step S22).

If it is not determined that the user's hand has touched the upper surface of the table (step S22; NO), the process returns to step S21, and steps S21 and S22 are repeated. When it is determined that the user's hand has touched the upper surface of the table (step S22; YES), the contact detection unit 12 detects a contact point, a contact line, or a contact area (step S23). The contact detection unit 12 sends contact position information indicating the position of the detected contact point, contact line, or contact area to the display control unit 15, and sends depth value distribution information to the angle detection unit 13. Next, the angle detection unit 13 determines whether or not the operation screen is being displayed (step S24).

When the operation screen is not displayed yet (step S24; NO), the angle detection unit 13 analyzes the distribution information of the depth value received from the contact detection unit 12, and the angle with respect to the tangent plane (contact elevation angle and contact azimuth angle). Is detected (step S25). The angle detection unit 13 sends angle information indicating the contact elevation angle and the contact azimuth angle to the display control unit 15. The display control unit 15 determines the operation direction based on the angle information received from the angle detection unit 13 (step S26).

The display control unit 15 generates an operation screen in accordance with the determined operation direction, and operates to a predetermined position based on the position of the contact point, the contact line, or the contact area indicated by the contact position information received from the contact detection unit 12. Instruction information for displaying the screen is generated and transmitted to the projection apparatus 3 (step S27). The process proceeds to step S30.

On the other hand, when the operation screen is being displayed (step S24; YES), the contact detection unit 12 specifies a selection item selected from the contact position information indicating the position of the detected contact point, contact line, or contact area ( Step S28). The input device 1 executes a function corresponding to the selected selection item (step S29). At this time, in the case of displaying a screen corresponding to the selected selection item, the contact detection unit 12 sends selection information indicating the selected selection item to the display control unit 15. The display control unit 15 generates a screen corresponding to the selection item indicated by the selection information received from the contact detection unit 12, and generates instruction information for instructing to display the screen corresponding to the selection item selected at a predetermined position. And transmitted to the projection device 3.

If the power supply is not turned off (step S30; NO), the input device 1 returns to step S21 and repeats steps S21 to S30. When the power is turned off (step S30; YES), the input device 1 ends the process.

Projection device 3 receives instruction information for displaying an operation screen from input device 1 (step S31). The projection device 3 projects the operation screen onto the upper surface 41 of the table according to the instruction information received from the input device 1 (step S32). In the case of displaying a screen corresponding to the selected selection item, the projection device 3 receives instruction information for instructing to display a screen corresponding to the selected selection item from the input device 1. The projection device 3 projects a screen corresponding to the selected selection item on the upper surface of the table according to the instruction information received from the input device 1.

If the projector 3 is not turned off (step S33; NO), the projection apparatus 3 returns to step S31 and repeats steps S31 to S33. When the power is turned off (step S33; YES), the projection device 3 ends the process.

According to the input system 100 of the present embodiment, it becomes possible to display an operation screen at an arbitrary place of an object in accordance with the operation direction of the user (operator), and the operability is improved.

In the above embodiment, the object is the upper surface of the table, and the object that contacts the object is the user's hand, but is not limited thereto. The shape of the object is not limited to a horizontal plane such as the upper surface of the table, but may be any shape that can project an operation screen, such as a vertical plane such as a wall surface, a cylinder, a sphere, or a staircase. In this case, the display control unit 15 of the input device 1 may generate an operation screen that matches the shape of the object. The object that contacts the object may be any object that has a shape that can define the direction extending from the tangent plane, such as a stick or a pen.

In the above embodiment, the depth sensor 2 and the projection device 3 are provided on the ceiling, but the present invention is not limited thereto. Depth sensor 2 should just be provided in the position which can generate distribution information of the depth value which can detect the contact elevation angle and contact azimuth angle of the object which contacted the subject. The projection device 3 may be provided at a position where the operation screen can be projected onto the object. In the above embodiment, the example in which the display position of the operation screen is determined based on the contact elevation angle and the contact azimuth angle is described. However, the present invention is not limited to this, and other parameters other than the contact elevation angle and the contact azimuth angle are used. The display position of the operation screen may be determined based on this. For example, the display position of the operation screen may be determined based on the corner of the elbow or the number of finger contacts.

In the above embodiment, the operation screen is displayed when it is determined that an object (such as a user's hand) has touched the display surface (such as the upper surface of the table) of the object. In addition to this, an operation screen may be displayed, and the direction and position of the operation screen may be changed by detecting that an object (such as a user's hand) approaches the display surface of the object. For example, when the hand is brought close to the display surface, the direction of the operation screen is changed in accordance with the extending direction of the hand and the arm, and the operation screen is brought closer to the tip of the hand. In this case, since the operation screen is displayed, it is easy for the user to understand the operation, and it is easy to start the operation.

FIG. 9 is a block diagram illustrating an example of a hardware configuration of the input device according to the embodiment. The control unit 31 includes a CPU (Central Processing Unit) and the like, and executes each process of the contact detection unit 12, the angle detection unit 13, and the display control unit 15 in accordance with a control program 39 stored in the external storage unit 33. .

The main storage unit 32 is constituted by a RAM (Random-Access Memory) or the like, loads a control program 39 stored in the external storage unit 33, and is used as a work area of the control unit 31.

The external storage unit 33 includes a nonvolatile memory such as a flash memory, a hard disk, a DVD-RAM (Digital Versatile Disc Random-Access Memory), a DVD-RW (Digital Versatile Disc Disc ReWritable), and controls the processing of the input device 1. A program to be executed by the control unit 31 is stored in advance, and data stored by the program is supplied to the control unit 31 in accordance with an instruction from the control unit 31, and the data supplied from the control unit 31 is stored. The storage unit 14 is configured in the external storage unit 33.

The operation unit 34 includes a pointing device such as a keyboard and a mouse, and an interface device that connects the keyboard and the pointing device to the internal bus 30. When the system administrator inputs information to the input device 1, the input information is supplied to the control unit 31 via the operation unit 34.

The display unit 35 is composed of a CRT or LCD. When the system administrator inputs information to the input device 1, the display unit 35 displays an operation screen.

The transmission / reception unit 36 includes a network termination device or a wireless communication device connected to the network, and a serial interface or a LAN (Local Area Network) interface connected to them. The transmission / reception unit 36 functions as the depth acquisition unit 11 and the display control unit 15.

The processing of the depth acquisition unit 11, the contact detection unit 12, the angle detection unit 13, the storage unit 14, and the display control unit 15 illustrated in FIG. 3 includes a control program 39, a control unit 31, a main storage unit 32, an external storage unit 33, The processing is executed by using the operation unit 34, the display unit 35, and the transmission / reception unit 36 as resources.

In addition, the hardware configuration and flowchart described above are merely examples, and can be arbitrarily changed and modified.

The central part that performs control processing including the control unit 31, the main storage unit 32, the external storage unit 33, the internal bus 30 and the like can be realized by using a normal computer system, not a dedicated system. . For example, a computer program for executing the above operation is stored and distributed in a computer-readable recording medium (flexible disk, CD-ROM, DVD-ROM, etc.), and the computer program is installed in the computer. Thus, the input device 1 that executes the above-described processing may be configured. Also, the computer program may be stored in a storage device included in a server device on a communication network such as the Internet, and the input device 1 may be configured by being downloaded by a normal computer system.

Further, when the function of the input device 1 is realized by sharing of the OS and the application program or by cooperation between the OS and the application program, only the application program portion may be stored in the recording medium or the storage device.

Also, it is possible to superimpose a computer program on a carrier wave and distribute it via a communication network. For example, the computer program may be posted on a bulletin board (BBS: Bulletin Board System) on a communication network, and the computer program may be distributed via the network. The computer program may be started and executed in the same manner as other application programs under the control of the OS, so that the above-described processing may be executed.

The present invention is capable of various embodiments and modifications without departing from the broad spirit and scope of the present invention. The above-described embodiments are for explaining the present invention and do not limit the scope of the present invention. That is, the scope of the present invention is shown not by the embodiments but by the claims. Various modifications made within the scope of the claims and the scope of the equivalent invention are considered to be within the scope of the present invention.

Some or all of the above embodiments may be described as in the following supplementary notes, but are not limited to the following.

(Appendix 1)
Contact detection means for detecting contact of an object with a projection target surface of an operation screen on an object, and angle detection means for detecting a contact elevation angle and a contact azimuth angle of the object in contact with the projection target surface with respect to the projection target surface And instruction information for displaying the operation screen at a predetermined position based on the contact position of the projection target surface with the object based on the contact elevation angle and the contact azimuth angle detected by the angle detection means. An input device having display control means for transmitting to the device;
The projection device that projects the operation screen onto the object according to the instruction information received from the input device;
An input system comprising:

(Appendix 2)
When the contact detection means detects a contact point, the angle detection means is the contact elevation angle that is an elevation angle and an azimuth angle in a one-dimensional direction extending with reference to the contact point of an object that has contacted the object. The input system according to claim 1, wherein the contact azimuth angle is detected.

(Appendix 3)
The angle detection means, when the contact detection means detects a contact line, an elevation angle and an azimuth with respect to a tangential plane of a normal in a two-dimensional direction extending with reference to the contact line of an object that has contacted the object The input system according to

appendix

1 or 2, wherein the contact elevation angle and the contact azimuth angle which are angles are detected.

(Appendix 4)
When the contact detection means detects the contact area, the angle detection means is an elevation angle and an azimuth angle of a one-dimensional direction extending with reference to the contact area of the object that has contacted the object. The input system according to any one of appendices 1 to 3, wherein the contact azimuth angle is detected.

(Appendix 5)
The contact detection means detects contact of the object with the projection target surface of the operation screen on the target object by analyzing distribution information of depth values indicating the depth of the object in a region including the target object. The input system according to any one of appendices 1 to 4.

(Appendix 6)
The angle detection unit analyzes the distribution information of depth values indicating the depth of the object in the region including the target object, so that the contact elevation angle of the object in contact with the projection target surface with respect to the projection target surface and The input system according to any one of appendices 1 to 5, which detects the contact azimuth angle.

(Appendix 7)
An input device connected to the projection device,
Contact detection means for detecting contact of an object with a projection target surface of an operation screen on the object;
An angle detecting means for detecting a contact elevation angle and a contact azimuth angle of the object in contact with the projection target surface with respect to the projection target surface;
Based on the contact elevation angle and the contact azimuth angle detected by the angle detection means, instruction information for displaying the operation screen at a predetermined position based on the contact position of the projection target surface with the object is used as the projection device. Display control means for transmitting to,
An input device comprising:

(Appendix 8)
When the contact detection means detects a contact point, the angle detection means is the contact elevation angle that is an elevation angle and an azimuth angle in a one-dimensional direction extending with reference to the contact point of an object that has contacted the object. The input device according to appendix 7, wherein the contact azimuth angle is detected.

(Appendix 9)
The angle detection means, when the contact detection means detects a contact line, an elevation angle and an azimuth with respect to a tangential plane of a normal in a two-dimensional direction extending with reference to the contact line of an object that has contacted the object The input device according to appendix 7 or 8, wherein the contact elevation angle and the contact azimuth angle which are angles are detected.

(Appendix 10)
When the contact detection means detects the contact area, the angle detection means is an elevation angle and an azimuth angle of a one-dimensional direction extending with reference to the contact area of the object that has contacted the object. The input device according to any one of appendices 7 to 9, wherein the contact azimuth angle is detected.

(Appendix 11)
The contact detection means detects contact of the object with the projection target surface of the operation screen on the target object by analyzing distribution information of depth values indicating the depth of the object in a region including the target object. The input device according to any one of appendices 7 to 10.

(Appendix 12)
The angle detection unit analyzes the distribution information of depth values indicating the depth of the object in the region including the target object, so that the contact elevation angle of the object in contact with the projection target surface with respect to the projection target surface and The input device according to any one of appendices 7 to 11, which detects the contact azimuth angle.

(Appendix 13)
A contact detection step of detecting contact of an object with a projection target surface of an operation screen on an object executed by an input device;
An angle detection step of detecting a contact elevation angle and a contact azimuth angle of the object in contact with the projection target surface with respect to the projection target surface;
Based on the contact elevation angle and the contact azimuth angle detected in the angle detection step, instruction information for displaying the operation screen at a predetermined position based on the contact position of the projection target surface with the object is displayed on the projection device. A display control step to transmit;
Projecting the operation screen onto the object according to the instruction information received from the input device executed by the projection device;
An input method comprising:

(Appendix 14)
In the angle detection step, when the contact point is detected in the contact detection step, the contact elevation angle that is an elevation angle and an azimuth angle of a one-dimensional direction extending with reference to the contact point of the object that has contacted the object. The input method according to appendix 13, wherein the contact azimuth angle is detected.

(Appendix 15)
In the angle detection step, when a contact line is detected in the contact detection step, an elevation angle and an azimuth with respect to a tangent plane of a normal line extending in two dimensions with respect to the contact line of the object contacting the object. 15. The input method according to

appendix

13 or 14, wherein the contact elevation angle and the contact azimuth angle which are angles are detected.

(Appendix 16)
In the angle detection step, when the contact area is detected in the contact detection step, the contact elevation angle that is an elevation angle and an azimuth angle of a one-dimensional direction extending with reference to the contact area of the object that has contacted the object. The input method according to any one of appendices 13 to 15, wherein the contact azimuth angle is detected.

(Appendix 17)
In the contact detection step, the contact of the object to the projection target surface of the operation screen on the target object is detected by analyzing distribution information of depth values indicating the depth of the object in the region including the target object. The input method according to any one of appendices 13 to 16.

(Appendix 18)
In the angle detection step, by analyzing distribution information of a depth value indicating the depth of the object in a region including the target object, the contact elevation angle of the object in contact with the projection target surface with respect to the projection target surface and The input method according to any one of appendices 13 to 17, wherein the contact azimuth angle is detected.

(Appendix 19)
A computer connected to the projection device,
Contact detection means for detecting contact of an object with a projection target surface of an operation screen on an object;
An angle detection means for detecting a contact elevation angle and a contact azimuth angle of the object in contact with the projection target surface with respect to the projection target surface;
Based on the contact elevation angle and the contact azimuth angle detected by the angle detection means, instruction information for displaying the operation screen at a predetermined position based on the contact position of the projection target surface with the object is used as the projection device. Display control means for transmitting to
A recording medium for storing a program characterized by functioning as a storage medium.

The present invention is based on Japanese Patent Application No. 2015-012683 filed on Jan. 26, 2015. The specification, claims, and entire drawings of Japanese Patent Application No. 2015-012683 are incorporated herein by reference.

DESCRIPTION OF SYMBOLS 1 Input device 2 Depth sensor 3 Projection apparatus 11 Depth acquisition part 12 Contact detection part 13 Angle detection part 14 Storage part 15 Display control part 30 Internal bus 31 Control part 32 Main storage part 33 External storage part 34 Operation part 35 Display part 36 Transmission / reception Unit 39 Control program 41 Upper surface 100 Input system F Floor area P1 to P9 End point S Skeleton T Table area U User area

Claims

Contact detection means for detecting contact of an object with a projection target surface of an operation screen on an object, and angle detection means for detecting a contact elevation angle and a contact azimuth angle of the object in contact with the projection target surface with respect to the projection target surface And instruction information for displaying the operation screen at a predetermined position based on the contact position of the projection target surface with the object based on the contact elevation angle and the contact azimuth angle detected by the angle detection means. An input device having display control means for transmitting to the device;
The projection device that projects the operation screen onto the object according to the instruction information received from the input device;
An input system comprising:
When the contact detection means detects a contact point, the angle detection means is the contact elevation angle that is an elevation angle and an azimuth angle in a one-dimensional direction extending with reference to the contact point of an object that has contacted the object. 2. The input system according to claim 1, wherein the contact azimuth angle is detected.
The angle detection means, when the contact detection means detects a contact line, an elevation angle and an azimuth with respect to a tangential plane of a normal in a two-dimensional direction extending with reference to the contact line of an object that has contacted the object The input system according to claim 1, wherein the contact elevation angle and the contact azimuth angle which are angles are detected.
When the contact detection means detects the contact area, the angle detection means is an elevation angle and an azimuth angle of a one-dimensional direction extending with reference to the contact area of the object that has contacted the object. The input system according to claim 1, wherein the contact azimuth angle is detected.
The contact detection means detects contact of the object with the projection target surface of the operation screen on the target object by analyzing distribution information of depth values indicating the depth of the object in a region including the target object. ,
The input system according to any one of claims 1 to 4.
The angle detection unit analyzes the distribution information of depth values indicating the depth of the object in the region including the target object, so that the contact elevation angle of the object in contact with the projection target surface with respect to the projection target surface and Detecting the contact azimuth;
The input system according to any one of claims 1 to 5.
An input device connected to the projection device,
Contact detection means for detecting contact of an object with a projection target surface of an operation screen on the object;
An angle detecting means for detecting a contact elevation angle and a contact azimuth angle of the object in contact with the projection target surface with respect to the projection target surface;
Based on the contact elevation angle and the contact azimuth angle detected by the angle detection means, instruction information for displaying the operation screen at a predetermined position based on the contact position of the projection target surface with the object is used as the projection device. Display control means for transmitting to,
An input device comprising:
A contact detection step of detecting contact of an object with a projection target surface of an operation screen on an object executed by an input device;
An angle detection step of detecting a contact elevation angle and a contact azimuth angle of the object in contact with the projection target surface with respect to the projection target surface;
Based on the contact elevation angle and the contact azimuth angle detected in the angle detection step, instruction information for displaying the operation screen at a predetermined position based on the contact position of the projection target surface with the object is displayed on the projection device. A display control step to transmit;
Projecting the operation screen onto the object according to the instruction information received from the input device executed by the projection device;
An input method comprising:
A computer connected to the projection device,
Contact detection means for detecting contact of an object with a projection target surface of an operation screen on an object;
An angle detection means for detecting a contact elevation angle and a contact azimuth angle of the object in contact with the projection target surface with respect to the projection target surface;
Based on the contact elevation angle and the contact azimuth angle detected by the angle detection means, instruction information for displaying the operation screen at a predetermined position based on the contact position of the projection target surface with the object is used as the projection device. Display control means for transmitting to
A recording medium for storing a program characterized by functioning as a storage medium.