JP2014056462A - Operation device - Google Patents

Abstract

PROBLEM TO BE SOLVED: To provide an operation device that is excellent in operability, lessens a hand shake upon operation and hardly generates weariness.SOLUTION: The operation device comprises: a three-dimensional display device 11 that is arranged to be directed at a user and three-dimensionally displays a virtual operation surface used for inputting an order or information to an operation device by the user; an acquisition 12 that is arranged to be directed at the user and acquires a distance image of the user; an operation position determination part 13 that determines an operation position of the user operating the virtual operation surface; a reference position acquisition part 14 that acquires a reference position serving as a reference when arranging the virtual operation surface; a calculation part 15 that calculates an arrangement position of the virtual operation surface; a viewing position determination part 16 that determines a viewing position of the user; a preparation part 17 that prepares display data on the virtual operation surface to be three-dimensionally displayed, and transmits the display data to the three-dimensional display device 11; and an instruction part 18 that, on the basis of operation position data and position data on the virtual operation surface, determines an operation content to the operation device and instructs the determined operation content to an operation device 200.

Description

  Embodiments described herein relate generally to an operation device that reduces camera shake during operation.

  When operating a device, it is indispensable to input commands and information to the device. However, when reaching and operating the control device in a vehicle such as a car, it is difficult to hold the hand in the correct position due to vibrations or camera shake.

  Therefore, the virtual operation surface is displayed in front of the 3D display device so that the operation position of the hand matches the display position so that the operation can be performed intuitively, or when the virtual operation surface is formed, Adjustments are made as appropriate according to the position of the person and the arrangement of the monitor and stereo camera.

  In addition, it is also proposed to set an operation reference plane and a virtual operation plane near the user so that the user's operation position relative to the displayed operation position can be recognized without limiting the user's operation. Has been.

  However, when the virtual operation surface is displayed parallel to the display device and the virtual operation surface is provided perpendicular to the ground, not only is it difficult to see, but the hand must be put forward and the elbow is raised. The limb is susceptible to vibration. There is also a problem that it is difficult to operate because the direction of the operation surface is perpendicular to the user.

JP 2011-175617 A JP 2011-187086 A

  The problem to be solved by the present invention is to provide an operating device that has good operability, reduces camera shake during operation, and is less fatigued.

  The operation device according to the embodiment is arranged to face the user, a three-dimensional display device that stereoscopically displays a virtual operation surface for the user to input commands and information to the operation device, and the user An acquisition unit that is arranged and acquires a distance image of the user, an operation position determination unit that determines an operation position of a user who operates the virtual operation surface based on the distance image data acquired by the acquisition unit, and the acquisition unit On the basis of the distance image data acquired in step (b), based on the reference position acquisition unit that acquires a reference position serving as a reference when placing the virtual operation surface, and the position data of the reference position acquired by the reference position acquisition unit, A calculation unit that calculates an arrangement position of the virtual operation surface; a viewing position determination unit that determines a viewing position of a user based on the distance image data acquired by the acquisition unit; position data of the viewing position; Based on the position data of the original display device and the position data of the virtual operation surface, the display unit generates the display data of the virtual operation surface to be stereoscopically displayed and sends it to the three-dimensional display device, the operation position data, and the And an instruction unit that determines operation contents for the operation device based on position data of the virtual operation surface and instructs the operation device.

It is a block diagram which shows schematic structure of the operating device which concerns on embodiment of this invention. It is a figure which shows the coordinate system of the position of the acquisition part and the distance image to acquire. It is a figure which shows an example of arrangement | positioning of a three-dimensional display apparatus and an acquisition part. It is a figure explaining the relationship between the operating range of a hand and arrangement | positioning of a virtual operation surface. It is a figure which shows the range which can operate a virtual operation surface by wrist fixing. It is a figure which shows the range which can operate a virtual operation surface by wrist fixing. It is a figure which shows the range which can operate a virtual operation surface by wrist fixing. It is a figure which shows the range which can operate a virtual operation surface by wrist fixing. It is a figure which shows the range which can operate a virtual operation surface by wrist fixing. It is a figure which shows the example of arrangement | positioning of a spherical virtual operation surface. It is a figure which shows the example of arrangement | positioning of the virtual operation surface in which the direction of a surface turns into a user's direction. It is a flowchart which shows the flow of the basic process in the operating device which concerns on embodiment. It is a flowchart which shows the flow of the process which calculates the arrangement position of the virtual operation surface in a calculation part. FIG. 5 is a diagram illustrating an example in which a plurality of virtual operation surfaces are prepared in the double spherical surface illustrated in FIG. 4. It is a figure which shows an example which made the virtual solid icon three-dimensionally display in the double spherical surface shown in FIG. 4, and was made into the operation target. It is a figure which shows the example of arrangement | positioning of the virtual operation surface with the same direction of a surface with respect to a user. It is a figure which shows an example which arrange | positions a three-dimensional display apparatus below a normal display apparatus, and an example of arrangement | positioning of an air gun. It is a figure which shows the virtual operation surface by the three-dimensional display apparatus at home. It is a figure which shows an example of arrangement | positioning of the virtual operation surface and air gun by a head mounted display.

  Hereinafter, an embodiment of the present invention will be described with reference to the drawings. In the drawings, the same portions are denoted by the same reference numerals, and redundant description is omitted.

  In the operation device according to the present embodiment, for example, the position of the elbow or the elbow is set as the reference position, and the three-dimensional operation is performed in consideration of the position, orientation, and shape of the virtual operation surface so that the touch operation can be performed without moving the elbow or the elbow. indicate.

  The operation device according to the present embodiment uses an arithmetic processing unit (CPU), a main memory (RAM), a read-only memory (ROM), an input device (for example, an operation panel), a hard disk device, or a flash memory that is a semiconductor memory. A general-purpose computer such as a personal computer equipped with a storage device such as a solid state drive (SSD) device can be used, and can be realized by installing a processing program for stereoscopic display of the virtual operation surface and operation support.

  FIG. 1 is a block diagram showing a schematic configuration of an operating device 100 according to an embodiment of the present invention. The operation device 100 according to the embodiment mainly includes a 3D display device 11, an acquisition unit 12, an operation position determination unit 13, a reference position acquisition unit 14, a calculation unit 15, a viewing position determination unit 16, a creation unit 17, and an instruction unit 18. It is configured. FIG. 2 is a diagram illustrating a coordinate system of an acquisition unit and a position of a distance image to be acquired. FIG. 3 is a diagram illustrating an example of the arrangement of the three-dimensional display device 11 and the acquisition unit 12.

  The operating device 100 operates the operating device 200 by issuing an instruction from the instruction unit 18 to the operating device 200. The operating device 200 is an in-vehicle stereo device, for example. When the instruction unit 18 instructs to increase the volume, the volume is increased.

  The three-dimensional display device 11 is arranged to face the user, and three-dimensionally displays a virtual operation surface for the user to input commands and information to the operation device. As shown in FIG. 3, the three-dimensional display device 11 can be disposed, for example, in front of the driver's seat of the vehicle. The three-dimensional display device 11 can also be disposed on the back side of the passenger seat of the vehicle. In such a case, a seated person in the rear seat of the vehicle operates the operation device 100.

  For example, the user can perform an operation on the virtual operation surface stereoscopically displayed at a certain position with the 3D display device 11 while being aware that the shape of a finger or the like is an operation determination target. The 3D display device 11 can be stereoscopically viewed by the user by presenting separate images to both eyes of the user. In general, it is necessary to present an image with parallax to the left and right eyes and change the appearance of the image in accordance with the user's head movement.

  The three-dimensional display device 11 can be configured with, for example, a head-up display, a projector, a naked-eye stereoscopic display, a display with polarized glasses, a follower, or the like. You can also use the screen and instrument panel of the car navigation system. When the 3D display device 11 is a device that can change the focus, such as a head-mounted display, it is preferable that the display focus position is aligned and displayed so that the eye focus is the same at the operation display position and the finger. is there.

  The acquisition unit 12 acquires distance information with respect to an object including a user facing the controller device 100 in an image format. The acquired distance image data is sent to the operation position determination unit 13, the reference position acquisition unit 14, and the viewing position determination unit 16. The acquisition unit 12 can be configured by a stereo camera or a depth sensor, for example. A stereo camera is a camera that can record information in the depth direction by simultaneously photographing an object from a plurality of different directions. When obtaining a distance from a stereo image, it is usual to use the principle of triangulation. Further, the depth sensor irradiates infrared rays to a certain range to obtain a state where fine dots are distributed, and the pattern of the dots changes depending on the object existing there.

  The operation position determination unit 13 acquires the depth to the object based on the distance image data acquired by the acquisition unit 12, and specifies the three-dimensional position of the hand operating the virtual operation surface. The position of the hand may be identified by recognizing the tip of a rod-shaped object coming out of the body as a hand. The shape is recognized as it is.

  Based on the distance image data acquired by the acquisition unit 12, the reference position acquisition unit 14 specifies a position that serves as a reference when placing the virtual operation surface. In the present embodiment, the “reference position” is specified by three-dimensional information about “the position of the joint” and “the place where the elbow is in contact”, ie, “the position of the elbow”. The “reference position” may be specified using data acquired by a depth sensor, a pressure sensor, or the like without being limited to the distance image acquired by the stereo camera. In order to specify the “elbow position”, for example, a pressure sensor can be installed in an elbow rest provided near the user's seat, and the position where the pressure is highest can be set as the “elbow position”. Further, the position from the wrist to the elbow and the direction extending from the wrist to the opposite side of the finger is extended by the length is the “elbow position”. To determine the length from the wrist to the elbow, place the arm in a position where you can confirm the bending of the elbow as seen from the stereo camera, and extend and bend the elbow to straight line from the shoulder to the elbow and from the elbow to the wrist. What is necessary is just to recognize two of the straight lines and determine the distance between the intersection and the wrist. The identified three-dimensional information such as “reference position” and “arm position” is sent to the calculation unit 15.

  The calculation unit 15 calculates the arrangement position of the “virtual operation surface” based on the obtained three-dimensional information of “arm position” and “reference position”. The “virtual operation surface” may be a plane, but need not be limited to a plane. Hereinafter, a “virtual operation surface” that is a plane and a “virtual operation surface” that is not a plane will be described in detail.

<When the virtual operation surface is a plane>
FIG. 4 is a diagram illustrating the relationship between the operating range of the hand and the placement of the virtual operation surface. A case where the virtual operation surface is a plane will be described with reference to FIG. When “wrist” is fixed, the range of movement of the hand is from the state where the hand is on the straight line from the elbow to the wrist (hereinafter referred to as state A), with the wrist tilted 90 degrees forward (hereinafter referred to as state A). Until state B). Further, the rearward direction is up to a state tilted by 90 degrees (hereinafter referred to as state C). Similarly, the horizontal direction is also tilted by 90 degrees. In addition, in the case of a user whose wrist is flexible and the range in which the hand can be moved is wide, the operating range of the hand may be expanded.

  Next, in a state where the “elbow position” is fixed, the range in which the fingertip position reaches (reachs) in the state A is a sphere that is equidistant from the elbow (the outer circle in FIG. 4). The range in which the fingertip position at C reaches (is reached) is a sphere that is equidistant from the elbow (inner circle in FIG. 4). That is, a virtual operation surface may be disposed between the outer sphere and the inner sphere.

  5 to 9 are diagrams showing a range in which the virtual operation surface can be operated with the wrist fixed. The direction of the virtual operation surface that can be touched by the user is operated from the A plane in FIG. 5 to the C plane in FIG. 7 through the B plane in FIG. 6 through the B plane in FIG. can do. On the contrary, when tilted backward, it is possible to operate from the A plane in FIG. 5 to the E plane in FIG. 9 through the D plane in FIG. 8. In the case of the C plane, it can be operated by bending the hand. When tilted backward, in addition to the method of tilting the hand forward, it is possible to touch the side tilted backward. That is, at a position away from the distance from the elbow to the wrist, the direction of the virtual operation surface is the direction opposite to the elbow (C surface) from the user direction (A surface) to the elbow direction (E surface). Range.

  Although the lower part of the virtual operation surface has been described as the same place, it is not always necessary to have the same place. As long as it is between two spherical surfaces, it may be arranged at any position and in any orientation.

  In addition, the explanation has been made so that the virtual operation surface is positioned directly below the elbow, but the virtual operation surface is placed in front of the explanation position so that the hand does not need to be raised so that the virtual operation surface can be easily viewed. It is desirable that the normal direction of the virtual operation surface is the user's direction as the downward direction.

  In practice, the position of the elbow does not need to be completely fixed, and the orientation, position, and size of the virtual operation surface can be set in a wider range.

    When the 3D display device 11 is in front of the hand operated like a head-mounted display, the virtual operation surface is on the back side with the palm up, and the range is the same. The area touched by the hand may be shifted to the front or may not be displayed. When a stereo camera is installed, even if a head-mounted display is placed immediately in front of the left and right eyes, the actual eye position is slightly shifted, so distance information can be obtained with a depth sensor, etc. It is preferable to correct the position. In the head-mounted display, the position of the eyes is fixed, but it is preferable to display a more accurate display by mounting a stereo camera capable of acquiring the position of the eyes and correcting the mounting displacement.

<When the virtual operation surface is not a plane>
Next, a case where the virtual operation surface is not a plane will be described. FIG. 10 is a diagram illustrating an arrangement example of a spherical virtual operation surface. FIG. 11 is a diagram illustrating an arrangement example of the virtual operation surface in which the direction of the surface is the direction of the user.

  When the elbow is fixed and a plane is set within the operating range of the hand, it is necessary to arrange the plane between the two spherical surfaces, so that the arrangement range is narrowed. Therefore, the virtual operation surface may not be a flat surface but may be arranged on a spherical surface within the operating range of the hand. By doing so, the virtual operation surface can be installed over a wide range without moving the elbow. For example, the virtual operation surface is arranged spherically slightly inside the outer spherical surface in FIG. For example, as shown in FIG. 10, the arrangement is such that a spherical surface divided within a certain range can be seen. A button or icon may be placed on the spherical surface.

  Further, as shown in FIG. 11, the virtual operation surface may be arranged so that the normal direction of the surface becomes the direction of the user. By doing so, it becomes easier to push than arranging the surface on the spherical surface. Further, the virtual operation surface may be directed slightly upward from the user direction so as to be easier to press.

  Since the virtual operation surface may be installed between the two spherical surfaces, the virtual operation surface may be bent and arranged regardless of the above arrangement and shape. Moreover, you may arrange | position a solid object instead of a virtual operation surface. For example, a display object such as a square cube or a sphere may be arranged, and the operation content may be changed depending on the touch surface. Further, the virtual display may be displayed at a position where the virtual display is recessed by the touched area and depth, and the touch may be felt with the eyes.

  The easy-to-operate range from the “elbow position” is shown above, but the reference direction as the reference to the wrist with respect to the “elbow position” is determined in advance, and depending on the distance from the elbow to the wrist depending on the user Then, the position and orientation of the virtual operation surface that is easy to operate in the reference direction are specified. For example, as shown in FIG. 16, when the reference direction is determined, the virtual operation surface is arranged in a direction that is easy to operate slightly below.

  Since the “elbow position” moves somewhat during operation, the position of the virtual operation surface may be fixed based on the position specified by the “elbow” first, or linked to the “elbow position”. May be. Further, the position of the virtual operation surface may be moved in conjunction with the position of the body that does not move, such as the shoulder when the “elbow position” is first identified. The reference direction may be determined in advance, but when the operation reception is started with the hand raised and the finger extended only by the index finger, the direction to the wrist at that time may be set as the reference direction.

  The virtual operation surface may be set in advance so as to have a relative position and orientation with respect to the reference direction. You may make it become the relative position and direction by the angle of a fingertip from a wrist when only an index finger is extended. At this time, the relative relationship is set in advance.

  Further, a virtual operation surface position may be set in advance, and the user may move to match that position. For example, an elbow rest is prepared, and the position is set so as to be a virtual operation surface when the elbow is placed on the elbow rest.

  The viewing position determination unit 16 specifies the viewing position of the user based on the distance image acquired by the acquisition unit 12. The viewing position is specified by the three-dimensional position data of the user's eyes.

  The creation unit 17 creates a display of the virtual operation surface using the 3D display device 11 based on the relationship between the viewing position, the position of the 3D display device 11 and the virtual operation surface. For example, in the case of a two-parallax display device, display is performed from the relationship between the position and orientation of the 3D display device 11 with respect to the acquisition unit 12 so that the position of the virtual operation surface is determined for each eye. Calculate and create When the acquisition unit 12 is attached to the 3D display device 11, the relative relationship between the acquisition unit 12 and the 3D display device 11 is fixed, so the relative relationship between the position and orientation of the 3D display device 11 is calculated in advance. Keep it. When the acquisition unit 12 is not attached to the 3D display device 11, the relative relationship of the 3D display device 11 is calculated by specifying the position of the 3D display device 11 by the acquisition unit 12.

  The instruction unit 18 specifies an operation performed based on the virtual operation surface and the operation position, and transmits an operation instruction to the operation device 200. The operation performed by the virtual operation surface and the operation position is specified, and an operation instruction is transmitted to the operation device 200. For example, a pause mark is displayed on the virtual operation surface, and when the hand presses the mark, the pause is transmitted to the operating device and paused. At this time, in order to allow a slight deviation, when the finger moves in the direction of quick touch, it may be touched even if it is a little far away.

  Next, a basic processing flow by the operation device 100 configured as described above will be described.

  FIG. 12 is a flowchart showing a basic processing flow in the controller device 100.

  First, a distance image representing the depth of an object is acquired by a depth sensor or a stereo camera constituting the acquisition unit 12 (step S1201). A part of the body of the user who operates the operation device 100 is captured.

  Next, a reference position is acquired based on data obtained with a depth sensor or a stereo camera (step S1202). For example, the reference position is the “elbow position” of the user.

  Next, the placement position of the virtual operation surface to be displayed on the controller device 100 is calculated based on the “elbow position” (step S1203). The flow of calculation processing of the arrangement position will be described later.

  Next, the operation position on the virtual operation surface by the user is determined (step S1204). For example, the position of the user's hand on the virtual operation surface is the operation position.

  Next, it is determined whether or not the user has operated the operation device 200 (step S1205).

  If it is determined that the user is operating (Yes in step S1205), the operation content is instructed to the operating device 200 (step S1206), and the process proceeds to step S1207. On the other hand, if it is determined that no operation has been performed (No in step S1205), the process immediately proceeds to step S1207.

  In step S1207, the position where the user is viewing the virtual operation surface is determined based on the data obtained by the depth sensor or the stereo camera.

  Next, the creation unit 17 generates a virtual operation surface to be displayed (step S1208).

  Next, the virtual operation surface generated by the creation unit 17 is sent to the three-dimensional display device 11 to be displayed (step S1209). The three-dimensional display device 11 displays the virtual operation surface so that it looks as if the virtual operation surface is present at the set or adjusted position when viewed from the user. Accordingly, the user can perform an operation on the virtual operation surface, which is a recognized stereoscopic image, using a hand or a finger.

  The present invention is not limited to the above, and the start of accepting operations on the controller device 100 may be determined by the user's voice or gesture. The virtual operation surface may be displayed after the reception is started, or the reception start may be notified by sound.

  On the other hand, the end of the operation in the controller device 100 may be triggered by the user's finger slid or hand down. Furthermore, the operation reception is limited to the fingertip only, and the operation may be ended when a finger other than the fingertip touches.

  FIG. 13 is a flowchart showing a flow of processing for calculating the placement position of the virtual operation surface in the calculation unit 15.

  First, the distance from the reference position, for example, “elbow position”, to the virtual operation surface is specified by the distance from the “elbow position” to the operation position, for example, “hand” (step S1301).

  Next, the orientation and the arrangement position of the virtual operation surface are rotated so as to become a reference direction, for example, “the direction from the elbow to the hand”, starting from the reference position, for example, “elbow position” (step S1302).

  Next, the orientation and position of the virtual operation surface are finely adjusted slightly below the reference direction (step S1303). In some cases, it is preferable to display the virtual operation surface at a position slightly different from the position where the virtual operation surface is actually formed as the user's operational feeling. The formation position of the virtual operation surface and the stereoscopic display position of the virtual operation surface image do not necessarily match.

  As described above, the placement position of the virtual operation surface is calculated.

(Modification)
About this embodiment, the following modifications are possible.

<Virtual operation surface>
Since the virtual operation surface is a base for displaying icons and the like, it is only necessary to arrange icons and buttons to be operated on the surface, and it is not necessary to display the surface.

  FIG. 14 is a diagram showing an example in which a plurality of virtual operation surfaces are prepared in the double spherical surface shown in FIG. In such a case, the hierarchy may be switched according to the height of the finger. In that case, it is only necessary to quickly raise the finger once to recognize the touch. Alternatively, a gap may be provided between the virtual operation surfaces so that the hand does not move and touch the gap. The number of operable icons can be increased by the hierarchical virtual operation surface. This is effective when the area of the operation surface must be reduced with the elbow fixed.

  In order to make it easy for the user to understand, it is preferable that the display of the layer currently being operated is darkened and the different layers are displayed lightly.

  FIG. 15 is a diagram illustrating an example in which a virtual solid icon is stereoscopically displayed on the virtual operation surface as illustrated in FIG. 10 as an operation target in the double spherical surface illustrated in FIG.

  In that case, it is only necessary to quickly raise the finger once to touch. Or you may enable it to prevent a hand moving and touching the clearance gap between solid icons. By doing so, the number of icons that can be selected even when the elbow is fixed can be increased. In addition, since icons that can be operated at all times can be continuously displayed, it is easy to select an icon to be selected. In addition, the solid icon is not limited to a square, but may be a circle or triangle, or a solid shape suitable for the meaning, such as a telephone shape in the case of a telephone.

<About the reference position>
Although the “elbow position” has been described as a fixed reference, it may be any joint of a person, for example, a wrist reference, a finger joint reference, a shoulder reference, or the like. When the operation is performed with the foot, the waist reference, the base of the foot, the knee reference, the ankle reference, and the joint reference of the toes may be used. Moreover, what is necessary is just to set it as a neck reference | standard when operating with a forehead or a tongue. The virtual operation surface, which is an operable range, is arranged within a range in which the previous joint bends from the reference joint.

  The position of the virtual operation surface may be shifted in conjunction with the movement of the “joint position” as a reference. At this time, since the operating range changes depending on the position of the joint on the body side and the orientation of the body relative to the reference joint, the position and orientation of the virtual operation surface may be changed in consideration thereof. For example, when the elbow is attached to the body, the hand is difficult to move to the outside of the body, so the virtual operation surface is set to the inside. When the elbow is away from the body, the hand easily moves to the outside of the body, so the virtual operation surface is on the outside.

  When the reference joint is too far away from the 3D display device, the virtual operation surface cannot be displayed beyond the range of the 3D display device. Even in this case, the operation may be accepted on the assumption that there is a virtual operation surface that is not displayed.

  Instead of the joint reference, a position where a part of the body is applied to something like an elbow rest may be used as a reference. In addition, the base of both palms may be placed on both knees, and the virtual operation display of the keyboard, numeric keys, and buttons may be displayed based on the position of the palm. In that case, the display may be divided into left and right instead of a single keyboard. Further, when the car is being driven, the palm may be placed on the handle. In that case, the 3D display device may be a HUD (head-up display). Further, since the handle rotates in a circular shape, not only the position but also the virtual operation surface may be rotated in accordance therewith. The virtual operation surface, which is an operable range, has a degree of freedom from the reference contact point to the next joint, and an arrangement in a range where the previous joint bends.

  When the wrist joint is used as a reference, the position of the virtual operation surface is linked to the wrist position so that the orientation of the display surface from the elbow position does not change. Good.

<Arrangement of 3D display device>
In order to display the virtual operation surface, the three-dimensional display device is lengthened vertically as shown in FIG. 3, or the virtual operation surface display 3 is displayed below the normal display device as shown in FIG. The dimension display device 11 may be separately arranged. When the virtual operation screen is displayed at home or the like, as shown in FIG. 18, the virtual operation screen is placed under the front of a display device such as a TV or the like so that it faces upward on the floor or is easy to see for viewers. It may be installed at an angle so that it can be operated without raising its hand.

<Accepting operations>
It is preferable to prevent the malfunction by accepting the operation only when the user is looking at the virtual operation surface.

<About touch feeling>
It is preferable to install an air gun that emits an air flow in the vicinity of the virtual operation surface so that the user can know that the virtual operation surface has been touched.

  Recognizing that the virtual operation surface has been touched, airflow is applied to the hand from the air gun. One air gun may cause the direction of the air gun to follow the position of the finger, or a transparent air gun may be spread over the surface of the three-dimensional display device and air may be fired only in the touched area. When air guns are spread, they may be arranged on a surface of a three-dimensional display device and irradiated in parallel.

  FIG. 17 is a diagram illustrating an example of the arrangement of air guns. As shown in FIG. 17, a radiation may be irradiated in the vertical direction in a line in a horizontally long gap such as between a display device and a virtual operation surface display three-dimensional display device. Moreover, it may arrange | position to a point and may irradiate to the radial form of a point. Further, as shown in FIG. 19, in a head mounted display or the like, air guns may be arranged on the back side of the device with a large number of air guns disposed on the surface, and air may be emitted only in an area touching the virtual operation surface. A finger may be caused to follow each of the air guns, and may be fired from a closer air gun. Moreover, you may enlarge the intensity | strength of an air gun, so that it is near. Further, when the finger comes to the area where the button is located, the air gun may be weakly irradiated, and when the button is pressed, the air gun may be strongly irradiated to feel the button being pressed. The temperature may be adjusted using a heater so that the air gun is close to the temperature of the finger. In that case, the temperature of the finger may be read and the temperature of the air to be fired may be changed according to the temperature.

  According to the present embodiment, since it is possible to operate in a state in which hand shaking is unlikely to occur, operability is remarkably improved by reducing hand movement and hand shake due to vibration of a vehicle such as a car. In addition, fatigue can be reduced because the elbows do not need to be moved much.

  Although several embodiments of the present invention have been described, these embodiments are presented by way of example and are not intended to limit the scope of the invention. These novel embodiments can be implemented in various other forms, and various omissions, replacements, and changes can be made without departing from the scope of the invention. These embodiments and modifications thereof are included in the scope and gist of the invention, and are included in the invention described in the claims and the equivalents thereof.

DESCRIPTION OF SYMBOLS 100 ... Operation apparatus 11 ... Three-dimensional display apparatus 12 ... Acquisition part 13 ... Operation position determination part 14 ... Reference position acquisition part 15 ... Calculation part 16 ... Viewing position determination part 17 ... Creation unit 18 ... Instruction unit 200 ... Operating device

Claims (16)

A three-dimensional display device that is arranged facing the user and displays a three-dimensional virtual operation surface for the user to input commands and information to the operation device;
An acquisition unit arranged to face the user and acquiring a distance image of the user;
An operation position determination unit that determines an operation position of a user operating the virtual operation surface based on the distance image data acquired by the acquisition unit;
Based on the distance image data acquired by the acquisition unit, a reference position acquisition unit that acquires a reference position serving as a reference when placing the virtual operation surface;
Based on the position data of the reference position acquired by the reference position acquisition unit, a calculation unit that calculates the placement position of the virtual operation surface;
A viewing position determination unit that determines the viewing position of the user based on the distance image data acquired by the acquisition unit;
Creation of display data of the virtual operation surface to be stereoscopically displayed based on the position data of the viewing position, the position data of the 3D display device, and the position data of the virtual operation surface, and sending the display data to the 3D display device And
An operating device comprising: an instruction unit that determines an operation content for the operation device based on the operation position data and the position data of the virtual operation surface and instructs the operation device.   The operating device according to claim 1, wherein the three-dimensional display device is a head-up display.   The operating device according to claim 1, wherein the acquisition unit is a stereo camera.   The operating device according to claim 1, wherein the acquisition unit is a depth sensor.   The operation device according to claim 1, wherein the operation position determination unit determines a three-dimensional position of a user's hand as an operation position.   The operating device according to claim 1, wherein the reference position is a position of a user's elbow or an arm.   The operating device according to any one of claims 1 to 5, wherein the reference position is specified by using data acquired by a depth sensor or a pressure sensor.   The operation device according to claim 6, wherein the position of the elbow is specified by installing a pressure sensor in an elbow rest provided in the vicinity of the user's seating and setting the position at the highest pressure.   7. The position of the elbow is specified in advance by determining the length from the wrist to the elbow and extending the direction extending from the wrist position to the opposite side of the finger by the length. Operating device.   The operating device according to any one of claims 1 to 4, wherein the viewing position determination unit determines a three-dimensional position of a user's eyes as a viewing position.   The operation device according to claim 1, wherein the virtual operation surface is a flat surface.   The operation device according to claim 1, wherein the virtual operation surface is a curved surface.   The position and orientation of the virtual operation surface is determined in advance as a reference direction from the elbow position to the wrist, and the position and orientation are specified in the reference direction according to the distance from the elbow to the wrist. 6. The operating device according to 6.   14. The operating device according to claim 13, wherein the reference direction is not determined in advance, and is determined to be the direction to the wrist at that time when the operation reception is started with the hand raised and the index finger extended.   The operation device according to claim 13, wherein the virtual operation surface is set in advance so as to have a relative position and orientation with respect to the reference direction.   The air gun for emitting an air flow is installed in the vicinity of the virtual operation surface, and when the user recognizes that the virtual operation surface is touched, the air flow is discharged from the air gun. The operating device according to any one of the above.

Images