JP2007094454A - Depth sensing system and interfacing device - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To display an image having depth information on a flat screen, and to enable a user to recognize the depth, and to much more accurately recognize a display object. <P>SOLUTION: In a depth sensing system, software 14 of a computer 10 displays the image of image data 12 with depth information on the screen of a monitor 16. A mouse device 30 is provided with a grip 34 which can be lifted up, a height sensor for detecting the height and z axial brakes 44 and 46 for regulating movement. A base 32 is provided with electromagnets 48 and 54 for regulating horizontal movement. When a mouse pointer 18 is pointed to pixels with depth, the software 14 operates the electromagnets 48 and 54 to regulate its horizontal movement. When the user lifts up the grip 34 to have height equivalent to depth, the software 14 operates the z axial brakes. <P>COPYRIGHT: (C)2007,JPO&INPIT

Description

  The present invention relates to a depth sensing system and an interface device, and more specifically to a depth sensing system and an interface device used in cooperation with an image display device that displays an image having depth or thickness information.

  Patent Documents 1 and 2 describe a mouse that is prevented from dropping by electromagnetic force even on an inclined mouse pad.

Patent Document 3 and Non-Patent Document 1 describe a mouse that exhibits a feeling of resistance in the front-rear and left-right directions by frictional force and electromagnetic force. The edge part of the figure drawn on the image display device is regarded as a step, and a braking force is generated when the mouse pointer moves on that part. Thereby, it is possible to cause the user to feel a collision and to give the user a feeling that there is a step.
JP 2001-185184 A JP 05-279596 A JP 05-035398 A Proceedings of Human Interface Symposium 2004, pp. 1179-1187

  By displaying an image having depth information on a flat screen and allowing the user to recognize the depth, the displayed object can be recognized more accurately.

  An object of the present invention is to present a depth sensing system and an interface device having a simple configuration that allows a user to recognize depth.

  The depth sensing system according to the present invention is a depth sensing system including a control device that controls an interface device capable of moving up and down a contact portion with which an operator's hand contacts according to image data having depth information. The interface device includes a guide mechanism that guides the contact portion up and down on the base, a lateral movement detecting unit that detects lateral movement of the base, a lateral movement regulating unit that regulates lateral movement of the base, and the contact. A vertical movement restricting means for restricting the vertical movement of the portion and a height sensor for detecting the height of the contact portion. The control device compares the depth information of the pixel corresponding to the contact portion of the image data with the height value of the height sensor according to the position information indicated by the lateral movement detection unit, and the depth When the value indicated by the information is higher than the height value, the lateral movement control means for controlling the lateral movement restriction means to regulate the lateral movement of the base, and the lateral movement restriction means by the lateral movement restriction means, When the height value becomes equal to the value indicated by the depth information, the vertical movement control means is provided for controlling the vertical movement restriction means to restrict the vertical movement of the contact portion.

  An interface device according to the present invention includes a contact portion that comes into contact with an operator's hand, a guide mechanism that guides the contact portion up and down on the base, a lateral movement detection unit that detects lateral movement of the base, It is characterized by comprising lateral movement restricting means for restricting lateral movement, vertical movement restricting means for restricting vertical movement of the contact portion, and a height sensor for detecting the height of the contact portion.

  According to the present invention, it is possible to make the user sense the depth with a simple configuration.

  Hereinafter, embodiments of the present invention will be described in detail with reference to the drawings.

  FIG. 1 shows a schematic configuration diagram of an embodiment of the present invention, and FIG. 2 shows an external perspective view of the embodiment. In the present embodiment, “depth information” is information that expresses the position in the direction perpendicular to the image plane as a numerical value.

  Image data 12 with depth information is stored in the hard disk of the computer 10, and the software 14 including the operating system and application software of the computer 10 displays the image of the image data 12 with depth information on the screen of the monitor 16. . The images with depth information are images having different depths in the x direction, the y direction, or both on the screen of the monitor 16.

  Such image data 12 with depth information can be obtained by the following method. For example, a method of capturing a real object using two cameras, a method of projecting a fringe pattern onto the real object, obtaining a depth distribution from the distortion of the fringe pattern, and a laser scanning the real object, There is a method of obtaining the depth distribution from the change. Furthermore, there is a method of extracting depth information from what is already a two-dimensional image. Image information with depth information can be obtained by giving arbitrary height information to an arbitrary pixel in an arbitrarily drawn image.

  Also connected to the computer 10 is a mouse device 30 as an interface device that allows the user to sense the depth. As is well known, the operating system of the software 14 of the computer 10 acquires information indicating the moving direction and moving amount of the vertically movable mouse 30 and information indicating ON / OFF of one or a plurality of mouse buttons from the mouse device 30. Then, the mouse cursor 18 displayed on the screen of the monitor 16 is moved according to the movement of the mouse device 30. An example of depth information of an image displayed in the vicinity of the mouse cursor 18 is shown in Table 20. The depth information is given in units of n × n pixels of the image 12, for example. n is an integer of 1 or more.

  The transmission / reception device 22 of the computer 10 acquires the movement information from the mouse device 30 and transmits a control signal for depth output to the mouse device 30.

  FIG. 1 shows a side view of the mouse device 30 and FIG. 2 shows a perspective view. The configuration and function of the mouse device 30 will be described with reference to FIGS. 1 and 2. The mouse device 30 includes a base 32 mounted on a mouse pad 90 and a grip portion 34 that can be moved up and down from the base 32 and is gripped by the user so as to be wrapped in the palm of the user.

  Two hollow cylinders 36 and 38 are erected on the base 32, and correspondingly, guide bars 40 and 42 extend vertically downward from the lower surface of the grip portion 34. The guide bar 40 is inserted through the cylinder 36, and the guide bar 42 is inserted through the cylinder 38. The guide bar 40 can move up and down in the cylinder 36, and the guide bar 42 can move up and down in the cylinder 38. The guide bar 40 is located at substantially the center or the center of gravity of the grip portion 34. With this mechanism, the grip portion 34 can move up and down in the vertical direction with respect to the base 32. The hollow cylinder 36 and the guide rod 40 are paired to hold the grip portion 34 on the base 32 so as to be movable up and down. By adding a similar hollow cylinder 38 and guide rod 42, the gripping portion 34 prevents lateral shaking. In this embodiment, the hollow cylinders 36 and 38 and the guide rods 40 and 42 constitute a guide mechanism that guides the grip portion 34 up and down with respect to the base 32.

  A side surface of the cylinder 36 is equipped with a z-axis brake that restricts the movement of the guide rod 40. That is, the pin 46 driven by the solenoid 44 is movably inserted in the side surface of the cylinder 36. 3 and 4 show cross-sectional views of a portion of the z-axis brake. FIG. 3 shows a brake-off state, and FIG. 4 shows a brake-on state. The pin 46 is always urged by a spring 45 in a direction toward the guide rod 40. However, the solenoid 44 is energized and biases the pin 46 against the spring 45 in a direction away from the guide rod 40. When the current to the solenoid 44 is cut off, the pin 46 is pressed against the guide bar 40 by the spring 45, and the guide bar 40 cannot move up and down. As a result, the user moves the grip part 34 upward and downward. Cannot be moved. In this manner, the on / off of the z-axis brake can be controlled by turning on / off the energization of the solenoid 44.

  In addition, a height sensor that detects the height of the grip portion 34 is incorporated in the inner surface of the cylinder 38. FIG. 5 shows a configuration diagram of the height sensor. A light emitting element 47 a is disposed on the lower end surface of the guide bar 42, and a light receiving element 47 b is disposed on the bottom inside the cylinder 38. The light receiving power of the light receiving element 47b varies depending on the height of the guide rod 42. Therefore, the height of the guide rod 42, that is, the height of the grip portion 34 can be measured from the output current of the light receiving element 47b.

  In addition, for example, the microswitches closed by the side surfaces of the guide rods 42 are arranged in an array in the height direction, and the height of the gripping portion 34 is determined depending on which position of the microswitch is open. Can be detected. As another configuration, even if photocouplers whose optical paths are shielded by the guide rods 42 are arranged on the inner surface of the cylinder 38 in the height direction, the height can be similarly detected. The detection result of the height sensor is transmitted to the OS of the software 14 via the transmission / reception devices 58 and 22.

  Further, electromagnets 48, 50, 52, 54 as brakes on the xy plane are arranged at the four corners of the base 32. The mouse pad 90 is made of a magnetic metal such as an iron plate. When the electromagnets 48 to 54 are energized, the electromagnets 48 to 54 are fixed to the mouse pad 90, and the mouse device 30 cannot move laterally.

  On the base 32, a movement detecting device 56 for optically measuring the lateral movement of the base 32 with respect to the mouse pad 90 is disposed. The configuration of the movement detection device 56 is the same as that of a normal optical mouse.

  A transmission / reception device 58 connected to the transmission / reception device 22 of the computer 10 is further arranged on the base 32. The transmission / reception device 58 transmits a signal indicating horizontal movement by the movement detection device 56 to the transmission / reception device 22, and controls (on / off) the current to the electromagnetic actuator 44 and the electromagnets 48 to 54 according to the control signal from the transmission / reception device 22. To do.

  The grip portion 34 is provided with click buttons 60 and 62 and is provided with dents 64 for easy lifting on both side surfaces. The click buttons 60 and 62 are connected to the transmission / reception device 58, and the transmission / reception device 58 transmits the operation of the click buttons 60 and 62 to the transmission / reception device 22. The gripping part 34 further incorporates a rise detection sensor (not shown) for detecting the lifting of the gripping part 34, and the detection result is transmitted to the software 14 via the transmission / reception devices 58 and 22.

  FIG. 6 shows a state transition diagram of this embodiment, and FIGS. 7 to 10 show operation flowcharts of this embodiment. The operation of this embodiment will be described with reference to FIGS.

  This embodiment has four states, an initial state, a steady state, an aerial state, and an invitation state, and starts from the initial state. The steady state is a state in which the depth value D and the height h of the mouse device 30 coincide with each other. As will be described later, the grip portion 34 does not fall even when the hand is released. The airborne state indicates a state where the mouse height h is larger than the depth value D, that is, a state where the mouse device 30 is floating. In other words, the airborne state is a state in which the operator lifts the grip portion 34 of the mouse device 30, and when the hand is released, the grip portion 34 naturally falls. The collision state indicates a state in which the mouse height h is smaller than the depth information D, that is, a state in which the mouse device 30 collides with a step and stops.

  In an initial state, the operator raises or lowers the grip portion 34 of the mouse device 30, and the depth value D of the pixel (pixel) pointed to by the mouse pointer 18 and the height h of the grip portion 34, that is, the mouse device 30. When the heights match, the steady state is entered. When the difference d (= D−h) becomes positive in the steady state, the state shifts from the steady state to the collision state. When the difference d becomes negative in the steady state or the rise detection sensor detects the rise, the steady state shifts to the air state. In the aerial state, when the difference d = 0, it shifts to a steady state, and when d becomes positive, it shifts to a collision state. When d becomes 0 in the collision state, the state shifts to a steady state.

  FIG. 7 shows an operation flowchart in the initial state. The z-axis brake and the xy plane brake are turned on (S1). The computer 10 acquires the mouse height h from the height sensor of the mouse device 30 (S2), and acquires the depth value D of the pixel indicated by the mouse pointer (S3). The difference d = D−h is calculated (S4). If the difference D is 0, the xy-plane brake is turned off, the mouse device 30 can be moved freely in the xy plane (S7), and a transition is made to a steady state. When the difference d is not 0 (S5), it waits for the upper and lower sides of the mouse device 30 (S6) and repeats the step S2.

  FIG. 8 shows a steady state operation flowchart. At the start, the z-axis brake is on and the xy-plane brake is off. The pixel where the mouse cursor is located is P0, and its depth value is D0.

  The mouse device 30 can be moved (S11, S12, S14), without moving horizontally (S14), and when the mouse button is pressed (S12), the z-axis brake is turned off (S13). As a result, the user can lift the grip portion 34, d becomes positive, and the state shifts to an airborne state.

  When horizontal movement is detected (S14), the moving direction is transmitted from the mouse device 30 to the computer 10 (S15), and the computer 10 moves the pointer to the pixel P1 in the moving direction (S16). The depth value D1 of the new pixel P1 is taken in (S17), and the difference d between the depth value D1 and the height h is calculated (S18). If the difference d is 0 (S19), the pixel and the depth value are updated (S20), and the movement of the mouse device 30 is waited (S11). If the difference d is positive, it shifts to a collision state, and if it is negative, it shifts to an airborne state (S21).

  FIG. 9 shows an operation flowchart of the collision state. At the start, the z-axis brake is off and the xy in-plane brake is on. The pixel where the mouse cursor is located is P0, and its depth value is D0.

  Waiting for the gripping portion 34 of the mouse device 30 to rise (S31), the height h is acquired (S32). A difference d between the depth D0 and the height h is calculated (S33). If the difference d is positive (S34), Step S31 and the subsequent steps are repeated, and if the difference d is 0 or negative (S34), the xy-plane brake is turned off, that is, horizontal movement is enabled, and a transition is made to a steady state. (S35).

  FIG. 10 shows an operation flowchart in the air state. At the start, the pixel where the mouse cursor is located is P0 and its depth value is D0. When starting, first, the z-axis brake is turned off and the xy in-plane brake is turned off (S41).

  The mouse device 30 can move horizontally (S42), and if the horizontal movement is not detected (S43), the height h is acquired (S44). A difference d between the depth D0 and the height h is calculated (S45). If the difference d is negative (S46), step S41 and subsequent steps are repeated. If the difference d is 0 or positive (S46), it shifts to a steady state.

  When the horizontal movement of the mouse device 30 is detected (S43), the movement direction is transmitted from the mouse device 30 to the computer 10 (S47), and the computer 10 moves the pointer to the pixel P1 in the movement direction (S48). The depth value D1 of the new pixel P1 is taken in (S49), and the difference d between the depth value D1 and the height h is calculated (S49). If the difference d is 0 (S51), the process proceeds to a steady state. If the difference d is negative (S52), the pixel and the depth value are updated (S54), and the process returns to step S41, and the difference d is positive. If this is the case (S52), the xy-plane brake is turned on, that is, the mouse device 30 cannot be moved in the horizontal direction (S53), and a transition is made to the collision state.

  FIG. 11 shows the relationship between the horizontal movement of the present embodiment and the height of the grip portion 34 according to the depth. (A) shows a display image, (b) shows the change of the depth value of the display image in the horizontal direction. (C) shows the state transition when the mouse device 30 is moved laterally from A to A ', changes in h, D, d, and the state of the brake.

  At the time point (1) when the state is shifted from the initial state to the steady state, the depth value D is 6, the height h of the mouse is 6, and the difference d is 0. In the state (2) in which the operator has moved the mouse device 30 to the right, since the depth value D of the pixel after the movement is 6, d = 0 and remains in a steady state.

  Further, in the state (3) in which the mouse device 30 is moved to the right, the depth value D of the pixel after the movement is 8, so the difference d is 2. As a result, a transition is made to a collision state. That is, the xy plane brake is turned on, and the mouse device 30 cannot be moved in the horizontal plane. The operator feels that he has hit a step, and in this state (4), the operator tries to lift the grip 34. The rise detection sensor detects the rise, and the z-axis brake is turned off, so that the operator can actually move the grip portion 34 upward.

  In the state (5) in which the grip portion 34 is lifted and the height h of the grip portion 34 is equivalent to 8, the depth values D and h match. Then, it shifts to a steady state, the z-axis brake is turned on, and the xy plane brake is turned off. Since the xy plane brake is turned off, the mouse device 30 can be moved in a horizontal plane.

  In the state (6) in which the mouse device 30 is moved further to the right, the depth value D is 8 and the height h is 8, so that the mouse device 30 remains in a steady state.

  In this way, in this embodiment, the operator can obtain a feeling that actually exceeds the step due to the transition from the collision state (3) to the steady upper stage (5).

  An embodiment in which the operator senses the depth of a plurality of points will be described. FIG. 12 is a perspective view of the mouse device 110 according to the second embodiment. The mouse device 110 has a vertical movement mechanism (columns 36 and 38, guide bars 40 and 42, a z-axis brake mechanism, a height sensor, and a height sensor) on the base 112 with respect to the palm and each finger. And a vertical movement mechanism 114, 116, 118, 120, 122, 124 having the same configuration as that of the ascending sensor). That is, the vertical movement mechanism 114 is for the palm, the vertical movement mechanism 116 is for the thumb, the vertical movement mechanism 118 is for the index finger, the vertical movement mechanism 120 is for the middle finger, the vertical movement mechanism 122 is for the ring finger, and the vertical movement mechanism 124 is for the little finger. It is. A stage 114a on which a palm is placed is provided on the upper surface of the vertical movement mechanism 114, and a belt 114b through which the palm passes is further hung on the stage 114a. Similarly, each of the vertical movement mechanisms 116 to 124 is provided with stages 116 a to 124 a on which the belly of the finger is placed, and belts 116 b to 124 b through which the fingers can pass through the stages 116 a to 124 a are hung. The stages 114a to 124a correspond to the grip portion 34 of the embodiment shown in FIG. Each stage 116a-124a can be moved up and down by the belts 116b-124.

  Further, electromagnets 126, 128, 130, 132 as xy plane brakes are arranged at the four corners of the base 112. The mouse device 110 is also used on a mouse pad made of magnetic metal. Further, on the base 112, a movement detecting device 134 that optically measures the lateral movement of the base 112 is disposed, and a transmitting / receiving device 136 that communicates with the transmitting / receiving device 22 of the computer 10 is disposed. The transmission / reception device 136 transmits data indicating various states of the mouse device 110 to the computer 10, receives control signals from the computer 10, and controls corresponding portions.

  When the mouse device 110 shown in FIG. 12 is connected to the computer 10, the computer assigns the pixel depth of the mouse cursor indicated by the mouse device 110 to the palm vertical movement mechanism 114. Then, the depths of pixels corresponding to the relative positions of the vertical movement mechanisms 116, 118, 120, 122, 124 with respect to the vertical movement mechanism 114 are assigned to the vertical movement mechanisms 116, 118, 120, 122, 124. The control mode of each vertical movement mechanism 114 to 124 is the same as that of the embodiment shown in FIG. 1, and the computer 10 controls each vertical movement mechanism independently. In this way, the operator can sense the depth with the palm and each finger.

  In the embodiment described above, electromagnetic force is used for the brake mechanism, but electrostatic force may be used. For the xy-plane brake, a method may be used in which a cloth is provided on one of the mouse pad and the back of the mouse, and the friction force is controlled by adjusting the degree of adhesion of the cloth to the other.

  Although the invention has been described with reference to specific illustrative embodiments, various modifications and alterations may be made to the above-described embodiments without departing from the scope of the invention as defined in the claims. This is obvious to an engineer in the field to which the present invention belongs, and such changes and modifications are also included in the technical scope of the present invention.

The schematic block diagram of one Example of this invention is shown. The external appearance perspective view of a present Example is shown. A cross-sectional view of the z-axis brake portion shows a brake-off state. A cross-sectional view of the z-axis brake portion shows the brake-on state. The block diagram of a height sensor is shown. The state transition diagram of a present Example is shown. The operation | movement flowchart of an initial state is shown. The operation | movement flowchart of a steady state is shown. The operation | movement flowchart of a collision state is shown. An operation flowchart in the air state is shown. The relationship between the horizontal movement of a present Example and the height of the holding part 34 according to the depth is shown. The perspective view of 2nd Example of this invention is shown.

Explanation of symbols

10: Computer 12: Image data with depth information 14: Software 16: Monitor 18: Mouse cursor 20: Depth information table 22: Transmitter / receiver 30: Mouse device 32: Base 34: Grasping part 36, 38: Hollow cylinders 40, 42 : Guide rod 44: Solenoid 45: Spring 46: Pin 47 a: Light emitting element 47 b: Light receiving element 48, 50, 52, 54: Electromagnet 56: Movement detector 58: Transceiver 60, 62: Click button 64: Recess 90: Mouse Pad 110: Mouse device 112: Base 114, 116, 118, 120, 122, 124: Vertical movement mechanism 114a, 116a, 118a, 120a, 122a, 124a: Stage 114b, 116b, 118b, 120b, 122b, 124b: Belt 126 , 128, 130, 132; electromagnets 34: movement detection unit 136: transmitting and receiving device

Claims (2)

A depth sensing system comprising a control device (10) for controlling an interface device (30, 110) capable of moving up and down a contact portion (34, 114a to 124a) with which an operator's hand is in contact according to image data having depth information. And
The interface device is
A guide mechanism (36, 38, 40, 42) for vertically guiding the contact portion (34, 114a to 124a) on the base (32, 112);
Lateral movement detecting means (56, 134) for detecting lateral movement of the base (32, 112);
Lateral movement regulating means (48 to 54, 126 to 132) for regulating lateral movement of the base (32, 112);
Vertical movement restricting means (44) for restricting vertical movement of the contact portions (34, 114a to 124a);
Height sensors (47a, 47b) for detecting the height of the contact portions (34, 114a to 124a)
And
The control device
Comparing means for comparing pixel depth information corresponding to the contact portion of the image data and the height value of the height sensor according to the position information indicated by the lateral movement detecting means;
When the value indicated by the depth information is higher than the height value, the lateral movement control means for controlling the lateral movement regulating means to regulate the lateral movement of the base;
When the height value becomes equal to the value indicated by the depth information in the lateral movement restricted state by the lateral movement restricting means, the vertical movement control is performed to control the vertical movement restricting means to restrict the vertical movement of the contact portion. And a depth sensing system. Contact portions (34, 114a to 124a) with which the operator's hand contacts;
A guide mechanism (36, 38, 40, 42) for vertically guiding the contact portion (34, 114a to 124a) on the base (32, 112);
Lateral movement detecting means (56, 134) for detecting lateral movement of the base (32, 112);
Lateral movement regulating means (48 to 54, 126 to 132) for regulating lateral movement of the base (32, 112);
Vertical movement restricting means (44) for restricting vertical movement of the contact portions (34, 114a to 124a);
Height sensors (47a, 47b) for detecting the height of the contact portions (34, 114a to 124a)
An interface device comprising:

Images