JP2003337961A - Device and method for image processing, image processing program, and recording medium therefor - Google Patents

Abstract

<P>PROBLEM TO BE SOLVED: To provide a device and a method for image processing, and an image processing program which can move an object such as a book by an action of throwing, etc., and rearrange and display it at a specified position on a wall, etc., in an easy-to-see state in a virtual space and to provide a recording medium therefor. <P>SOLUTION: When it is judged that coordinates of a virtual book and fingers deviate in the virtual space, the image processor computes the position of the book after the book leaves the fingers and the moving speed and flying direction from the position (step S207) to generate and display successive images wherein the book is flying (step S208). Then it is judged whether coordinates of the virtual book and a virtual wall match each other (step S209) and when it is judged that the coordinates of the book and wall (obstacle) match each other, the virtual book is displayed on the virtual wall so as to face the eyes of a user (S210). <P>COPYRIGHT: (C)2004,JPO

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to an image processing apparatus and an image processing method, and more particularly, to an image processing apparatus and an image processing method for displaying an object in a virtual space image so as to face the user. The present invention also relates to an image processing program and its recording medium.

[0002]

2. Description of the Related Art In Japanese Unexamined Patent Publication No. 6-282371, a virtual space image of a book or the like is displayed, and when a user (user) performs an operation of turning over the pages of the displayed virtual book, the operation is performed according to the operation. , A virtual space desktop device configured to turn over the pages of the book is disclosed.

However, in the above-mentioned conventional apparatus, the user cannot feel the texture of the paper of the book or the touch when touching, for example, the edge of the paper with the hands and fingers, and the user cannot use the hands and fingers. I feel a sense of incongruity because I cannot feel the reaction force when I hold a book.

When turning pages of a book, the edge of the book sheet is recognized by the feel of a finger, and the sheet (page) is detected.
It is a natural movement to flip the pages by pinching and lifting them with your fingers, but since the conventional device does not give you the feel of the book with your hands or fingers, you can flip the pages of the book every time. It is necessary to pay attention to the edge of the paper of the book and to move the finger to the edge, and it takes time and time to turn the pages of the book.

By the way, after the operation of reading a virtual book in a virtual space, it is considered that the books should be placed side by side in an easy-to-read position so that they can be easily searched and viewed again the next time they want to see them. To be In this case, as a way for human beings to understand intuitively and to remember the place,
There is a method of displaying books side by side like a bookshelf.

[0006] It is convenient if the virtual books are displayed on the wall where the virtual books are thrown so that they are lined up on the bookshelf by the action of the user throwing the virtual books toward the wall by making the best use of the points in the virtual space. .

[0007]

SUMMARY OF THE INVENTION According to the present invention, an image processing capable of moving an object such as a book by a motion such as throwing it in a virtual space and organizing and displaying it at a predetermined position such as a wall for easy viewing. An object is to provide an apparatus, an image processing method, an image processing program, and a recording medium thereof.

[0008]

The above object is achieved by the present invention described in (1) to (32) below.

(1) Image generation means for generating a virtual space image, display means for displaying the virtual space image generated by the image generation means, and moving speed and movement of a virtual object in the virtual space image. Comprising a calculating means for calculating a direction and a collision determining means for determining whether or not the moving object collides with a virtual obstacle, and the collision determining means causes the object to collide with the obstacle. If it is determined that the target object is displayed, the object is displayed so as to face the moving direction substantially.

(2) The image generating means updates and generates a virtual space image at a predetermined time interval based on the moving speed and moving direction of the object calculated by the calculating means. The image processing device according to item 1.

(3) The image processing apparatus according to (2), wherein the display means continuously displays virtual space images updated as the virtual object is moved.

(4) A tactile sense / force sense presenting means having a finger position detecting means for detecting the position of the finger and a tactile sense / force sense generating means for generating a tactile sense and a force sense applied to the finger,
The image generation unit generates a virtual finger image at the position of the finger detected by the finger position detection unit in the virtual space image, and the display unit displays a virtual space image including the virtual finger image. The image processing apparatus according to any one of (1) to (3) above.

(5) The finger position detecting means is an orthogonal coil type position detecting means having a first coil, a second coil and a third coil whose central axes are substantially orthogonal to each other. The image processing device according to 4).

(6) The image processing apparatus according to (4), wherein the finger position detecting means has a gyro sensor for detecting angular velocities around X, Y and Z axes which are substantially orthogonal to each other.

(7) When the position of the finger detected by the finger position detecting means coincides with the position of the object,
The image processing apparatus according to any one of (4) to (6), further including a control unit that controls driving of the tactile sensation / force sense generation unit so that a sensation equivalent to the contact is obtained on the finger pad.

(8) When the position of the finger detected by the finger position detecting unit and the position of the object are deviated, the control unit prevents the feeling of the contact from being obtained on the finger pad. The image processing apparatus according to (7) above, which controls driving of the tactile sense / force sense generating means.

(9) The calculating means calculates from a moving distance and a time required therefor in a minute time before the position of the finger detected by the finger position detecting means and the position of the object deviate from each other, and a moving direction of the object. The image processing device according to any one of (4) to (8) above, which calculates a moving speed and a moving direction after the object is separated from a finger.

(10) The tactile / force sense presenting means is in the shape of a glove worn from a finger to an arm, receives control data regarding driving from the controlling means, and detects a finger detected by the finger position detecting means. The image processing apparatus according to any one of (4) to (9) above, including a communication unit that transmits position data to the calculation unit.

(11) The image processing apparatus according to any one of (1) to (10), wherein the display means is mounted on a face for use.

(12) The image processing apparatus according to any one of (1) to (11), wherein the display means includes a communication means for receiving the virtual space image data generated by the image generation means.

(13) The image generation means, the calculation means and the collision determination means, and / or the control means are configured as one unit, and the one unit is the display means and / or the tactile force. The image processing apparatus according to any one of (1) to (12) above, further including a communication unit that transmits and receives data to and from the haptic presentation unit.

(14) The virtual space image data generated by the image generation means, the calculation result obtained by the calculation means, the collision determination result obtained by the collision determination means, the finger position of the tactile / force sense presentation means The above-mentioned (1) to (1) further comprising storage means for storing any or all of the finger position data obtained from the detection means.
The image processing device according to any one of 3).

(15) The collision determining means determines whether or not an object moving in the virtual space image has collided with a predetermined plane in the virtual space image which is the obstacle (1). The image processing device according to any one of (1) to (13).

(16) Generating a virtual space image, displaying the generated virtual space image,
Calculating a moving speed and a moving direction of the virtual object in the virtual space image, and updating the virtual space image at a predetermined time interval based on the calculation result to show the moving state of the virtual object. A step of displaying, a step of determining whether or not the moving object has collided with a virtual obstacle, and if it is determined that the object has collided with the obstacle, And a step of displaying the image so as to substantially face the moving direction.

(17) The image processing method according to (16), further including the step of detecting the position of the finger, wherein the finger is displayed on the generated virtual space image.

(18) Determining whether or not the detected finger position and the position of the object match.
(16) or (17) further comprising: providing tactile sensation and force sense to the finger pad so that a touch corresponding to the contact is obtained when the positions match.
The image processing method described in.

(19) When the position of the finger deviates from the position of the object after the step of providing the tactile sensation and the force sensation, the tactile sensation and force are applied to the finger pad so that a sensation corresponding to the contact is not obtained. The image processing method according to (18) above, further including a step of stopping giving a sensation.

(20) In the calculation step, the object is separated from the finger based on a moving distance and a time required for the minute moving time before the position of the finger and the position of the object deviate from each other and a moving direction of the object. The image processing method according to any one of (16) to (19), wherein the moving speed and the moving direction after calculation are calculated.

(21) The image processing method according to any one of (16) to (20), further including a step of transmitting the generated virtual space image data to a virtual space display device.

(22) A step of storing the generated virtual space image data and the updated virtual space image data, a step of storing the calculation result obtained in the calculation step, and a step of storing in the collision determination step. The image processing method according to any one of (16) to (21) above, further comprising the step of storing the obtained collision determination result.

(23) The image processing method according to any one of (17) to (22), further including a step of storing the position coordinate data of the finger obtained in the finger position detecting step.

(24) An image generating means for generating a virtual space image, which is a computer of an information processing device for generating a virtual space image and displaying the virtual space image for the user to visually recognize.
Display means for displaying the virtual space image generated by the image generating means, calculation means for calculating the moving speed and moving direction of the virtual object in the virtual space image, and the moving object is a virtual obstacle. Collision determining means for determining whether or not the object has collided, when the collision determining means determines that the object has collided with the obstacle, the object is generally directly opposed to the moving direction. An image processing program of an information processing device for causing the display unit to display as described above.

(25) The computer of the information processing apparatus has a finger position detection means for detecting the position of the finger, and a tactile force sense presentation means for generating a tactile force and a force sense applied to the finger. Further comprising a program code for functioning as means, causing the image generating means to generate a virtual finger image at the position of the finger detected by the finger position detecting means in the virtual space image, and displaying the display means. The image processing program of the information processing apparatus according to (24), which causes the virtual space image including the virtual finger image to be displayed.

(26) In the computer of the information processing device, when the position of the finger detected by the finger position detecting means and the position of the object match, a feeling corresponding to the contact is obtained on the finger pad. The image processing program of the information processing apparatus according to (25) above, further including a program code for causing the tactile sensation / force sense generating means to function as control means for controlling driving.

(27) The control means obtains a feeling corresponding to the touch on the finger pad based on the match / mismatch between the position of the finger detected by the finger position detection means and the position of the object. The tactile sensation
The image processing program of the information processing device according to (26) above, including a program code for causing the force sense generating means to be controlled to be driven.

(28) The computer of the information processing apparatus transmits / receives control data relating to driving between the tactile / force sense presenting means and the control means, so that the tactile / force sense presenting means and the computing means are connected. The image processing program of the information processing apparatus according to any one of (25) to (27), further including a program code for functioning as a communication unit that transmits and receives finger position data detected by the finger position detection unit. .

(29) A program code for causing a computer of the information processing apparatus to function as a communication means for transmitting and receiving the generated virtual space image data between the display means and the image generation means. An image processing program for an information processing apparatus according to any one of (24) to (28).

(30) A computer of the information processing device is provided between at least one of the image generation means, the calculation means, the collision determination means and the control means and the display means and / or the tactile / force sense presentation means. (24) to (2), further including a program code for functioning as communication means for transmitting and receiving data to and from each other.
An image processing program for an information processing device according to any one of 9).

(31) In the computer of the information processing device, the virtual space image data generated by the image generation means, the calculation result obtained by the calculation means, the collision determination result obtained by the collision determination means, the tactile sense
The method according to any one of (24) to (30) above, further comprising a program code for functioning as a storage unit that stores any or all of the finger position data obtained from the finger position detection unit of the force sense presentation unit. Image processing program of the information processing apparatus of.

(32) A computer-readable recording medium recording the image processing program according to any one of (24) to (31) to be executed by a computer.

[0041]

BEST MODE FOR CARRYING OUT THE INVENTION Preferred embodiments of an image processing apparatus and an image processing method according to the present invention will be described in detail below with reference to FIGS. It should be noted that this embodiment is given as an example, and the present invention should not be limitedly interpreted thereby.

FIG. 1 is a conceptual diagram showing a configuration example of a first embodiment of an image processing apparatus including a tactile / force sense presentation device of the present invention, and FIG. 2 is a circuit configuration of the real glove 20 shown in FIG. It is a block diagram which shows an example. 3 is an external view (partially omitted from illustration) showing an example of the external configuration of the real glove 20 shown in FIG.

As shown in FIG. 1, the image processing apparatus 1 is attached to the hand (fingers) of a user (user), and tactile sensation (feeling of touch, for example, texture, shape,
A pair of real gloves (data gloves) 20 and 20 for giving a distribution of contact strength such as strength) and a force sense (a sense of force applied when touched), and a virtual space for displaying a virtual space image (virtual space image) Various processes such as drive control of the display device (display means) 5 and each of the real globes 20 and the virtual space display device 5 (in the present invention, tactile / force sensation generation process of FIG. 19 and virtual process of FIG. An information processing device 8 that performs the main alignment processing) and a base station 9 that creates a virtual space image and provides various services using the virtual space image are provided.

Each of the real globe 20, the virtual space display device 5, the information processing device 8 and the base station 9 is wireless (for example, wireless LAN (Local Area Network), infrared (IR) data communication (IrDA: Infrared Data Association).
ation), Bluetooth, etc.) has a wireless communication function (wireless communication means) capable of transmitting and receiving signals (wireless communication) by means of information processing device 8, each real globe 20, and virtual space display device. 5 and the base station 9,
Each is able to communicate.

The image processing device 1 is created (generated) by the information processing device 8 based on the information supplied from the base station 9, the virtual space display device 5 and each real globe 20.
When the virtual space image thus displayed is displayed on the virtual space display device 5 and the hand or finger wearing the real glove 20 is operated, the hand or finger virtually contacts an object displayed in the virtual space. The state is displayed on the virtual space display device 5, and each process for giving a tactile sensation or a force sensation upon contact to the hand or fingers wearing the real glove 20 is performed.

The information processing device 8 is, for example, a mobile phone terminal, a personal computer having a wireless communication function, or P
It is composed of various devices having a wireless communication function (communication means) and a control function (control means) such as portable information equipment called DA (Personal Digital Assistants).

The base station 9 is composed of various devices such as various server computers connected to the Internet and mobile phone base stations that wirelessly communicate with mobile phone terminals, which provide information about virtual space images. To be done.

In this embodiment, each real globe 20, the virtual space display device 5, the information processing device 8 and the base station 9 have a wireless communication function. However, for example, some or all of them are wired communication. It may be configured to perform.

The information processing device 8 may have some or all of the functions of the base station 9. In the tactile sensation / force sensation generation process and the virtual main alignment process described later, the information processing device 8 is described as including all the functions of the base station 9 for convenience.

The real globe 20 may have some or all of the functions of the virtual space display device 5, the information processing device 8 and the base station 9.

Here, in the present embodiment, the tactile / force sense presentation device 2 is equipped with a pair of real gloves 20, 20, that is, the real glove 20 for the left hand worn on the left hand of the user and the real glove 20 for the right hand of the user. It is configured with the right-handed real glove 20. In the following description, one of the real gloves 20 will be representatively described.

In the present invention, the tactile / force sense presentation device 2 may be composed of only one of the real gloves 20.

In the present invention, the tactile / force sense presentation device 2
In addition to the real globe 20, the virtual space display device 5,
Any one of the information processing device 8 and the base station 9
It may have one, two, or all. In this case, the tactile sense / force sense presentation device 2 may have some or all of the functions thereof.

As shown in FIG. 2, the real gloves 20
Is the sense (eg, texture, shape, etc.) of a finger touching an object (object) appearing in the virtual space (on the virtual space image).
A tactile generation unit (tactile generation means) 21 for generating a distribution of contact strength such as strength) and a sense of the strength of force applied to a hand or finger when touching an object appearing in a virtual space. A force sense generating unit (force sense generating means) 22 for causing it and a position / orientation detecting unit for detecting the position of each part of the real glove 20 (the position of each part such as the hand, wrist, arm, finger, etc. wearing the real glove 20). 23, a control unit 24 that controls each unit of the real globe 20, a signal processing unit 25 that performs signal processing such as amplification of a detection data signal transmitted to the information processing apparatus 8, and wireless communication between the information processing apparatus 8. And a signal transmission / reception unit (communication means) 26 for performing.

Further, as shown in FIG. 3, the overall shape of the real glove 20 is a substantially long glove shape (glove shape). That is, the real glove 20 is a glove-like wearing unit that is worn from the wearer's (finger) 's finger to the vicinity of the elbow of the arm (from finger to arm), and is a tubular portion 20a covering the arm.
Has a relatively long glove 2a.

The user can easily and surely wear and remove the real glove 20 with the gloves 2a.

Further, since the glove 2a is worn from the user's hand to the arm, the operation (operation) in the virtual space can be performed easily and reliably.

In this glove 2a, a play absorbing part 27 for suppressing or preventing play of a part corresponding to a fingertip, a wrist fixing part 28 for fixing the real glove 20 to the wrist, and the real glove 20 for fixing to the arm. The arm fixing portion 2d and the twist absorbing portion 29 that absorbs the twist of the portion corresponding to the arm,
Each is provided.

The play absorbing portion 27 is provided at a portion corresponding to the vicinity of the wrist of the glove 2a. This play absorbing part 27
Is formed of an elastic body such as various rubbers.

When the user puts on the real glove 20, the restoring force (elastic force) of the play absorbing portion 27 causes the glove 2 to move.
“A” is pulled in the elbow direction (base end side) and held in that state. As a result, it is possible to prevent or suppress play at the fingertips.

Also, when used by a plurality of users having different hand sizes such as finger lengths, the play absorbing portion 27 prevents or suppresses the play of the fingertips of each user. be able to.

The wrist fixing portion 28 is provided continuously to the play absorbing portion 27 on the elbow side (base end side) of the play absorbing portion 27 of the glove 2a. The wrist fixing portion 28 is formed of, for example, an elastic body such as various kinds of rubber.

When the user wears the real glove 20, the glove 2 is restored by the restoring force (elastic force) of the wrist fixing portion 28.
a is fixed to the wrist. Thereby, it is possible to prevent the real gloves 20 from being displaced during use.

Further, by the wrist fixing portion 28 and the play absorbing portion 27, the bundle 212a of the actuator wires 212 of the tactile sense generating portion 21, which will be described later, has a predetermined length so that each actuator wire 212 can move in its longitudinal direction. Held in position.

The arm fixing portion 2d is the tubular portion 20a of the glove 2a.
It is provided at the end of the elbow side (base end side). The arm fixing portion 2d is formed of, for example, an elastic body such as various rubbers.

When the user wears the real glove 20, the restoring force (elastic force) of the arm fixing portion 2d causes the glove 2a to move.
Is fixed to the arm. Thereby, it is possible to prevent the real gloves 20 from being displaced during use.

The twist absorbing portion 29 is the tubular portion 2 of the glove 2a.
It is provided in the middle of 0a, that is, between the wrist fixing portion 28 and the arm fixing portion 2d.

The twist absorbing portion 29 is composed of three strip portions 291 obtained by forming three openings 292 in a part of the tubular portion 20a. Each strip-shaped portion 291 is formed of a stretchable member such as cloth made of a stretchable material.

When the user wears the real glove 20 and twists his / her wrist, for example, each strip-shaped portion 291 expands / contracts and deforms, whereby the twist of the tubular portion 20a is absorbed.

As described above, the twist absorbing portion 29 can prevent the real glove 20 from being twisted. Therefore, the user can easily and smoothly move his / her hand or arm.

The number of the strip-shaped portions 291 of the twist absorbing portion 29 may be 2 or less, or 4 or more.

Further, openings 21a are formed in the portions of the gloves 2a corresponding to the joints on the back side of the fingers.

These openings 21a allow the user to easily and smoothly move each joint of each finger.

The real glove 20 has a portion corresponding to the arm of the glove 2a, which is the twist absorbing portion 2
The control unit 24 is provided at a portion located in the elbow direction with respect to 9;
A power supply unit 30, a power generation unit 211 and the like are provided.

By arranging the control unit 24, the power supply unit 30, and the power generation unit 211 as described above, the weight of the fingertip side (tip side) of the real glove 20 can be reduced (inertia reduced). Therefore, the user can easily perform the operation (operation).

The main part of the tactile sensation generator 21 is the first part of each finger of the user when the user wears the real glove 20.
It is provided at a position that covers the fingertip portion ahead of the joint (the joint between the distal phalanx and the middle phalanx). Since the configuration and operation of each tactile sense generating unit 21 are almost the same, one of them will be described representatively below.

FIG. 4 is a vertical sectional view showing an example of the structure of the tactile sense generating section 21. In order to avoid making the figure complicated,
In the drawing, the thickness of some members is omitted.

As shown in FIG. 4A, the tactile sense generating section 21 includes a power generating section (moving means) 211 having a motor (driving source) 211a and an actuator wire (movable in the longitudinal direction). Flexible linear body) 212
And a wire holding portion (biasing means) 213 that holds one end of the actuator wire 212, a tactile presentation pin (pressing portion) 214 that presses the finger pad to generate a tactile sense, and a tactile presentation pin 214 while supporting the tactile presentation pin 214. Tactile presentation pin 21
4 has a plurality of unit tactile sense generating units (actuators) 210 configured with tactile sense pin support units (moving direction regulating means) 215 that regulate the moving direction of the No. Each unit tactile sensation generator 210 is independent of each other.

The power generation section 211, the actuator wire 212, and the tactile sense presentation pin support section 215 constitute a drive mechanism for driving the tactile sense presentation pin 214.

The power generation section 211 is, for example, the control section 24.
The motor 211a provided therein is rotationally driven in accordance with the control signal from, and the process of winding the actuator wire 212 by the rotational driving of the motor 211a (the process of moving the actuator wire 212 in the longitudinal direction of the actuator wire 212) is performed. The control unit 24 controls the power generation unit 211 according to an instruction from the information processing device 8 so that the motor 211a is rotationally driven with a predetermined torque for a predetermined time. The power generation unit 211 performs a process of winding the actuator wire 212 for a predetermined length by applying rotational power to the motor 211a according to an instruction from the information processing device 8.

As shown in FIG. 4A, the wire holding portion 213 holds one end (end) of the actuator wire 212 in the vicinity of the first joint of the finger in this embodiment. The wire holding portion 213 is made of an elastic member (elastic body) such as rubber or spring.

Therefore, the motor 211 of the power generation unit 211
When the actuator wire 212 is wound by driving a, and the actuator wire 212 moves clockwise in FIG. 4 (A), the wire holding portion 21 is correspondingly moved.
3 expands, and the actuator wire 212 moves in the counterclockwise direction in FIG. 4A (movement of the actuator wire 212 when the tactile sense presenting pin 214 is projected) due to the restoring force (elastic force) of the wire holding portion 213. (Direction opposite to direction)
Be urged by. Then, the motor 21 of the power generation unit 211
When the driving of the actuator 1a is stopped and the winding state of the actuator wire 212 is released, the wire holding portion 213 contracts by the restoring force (elastic force), whereby the wound actuator wire 212 is pulled out, Figure 4
Move counterclockwise in (A).

The power generator 211 is located on the other end (starting end) side of the actuator wire 212.

The tactile sense presenting pin 214 is used to give a touch sensation (presence or absence of contact, strength, etc.) to the finger pad of the user, and is a minute contact plate (contact portion) which is a portion contacting the finger pad.
214a and a supporting portion 214b for supporting the contact plate 214a
Composed of and. The support portion 214b has one end fixed to the middle of the actuator wire 212 and the other end provided with a contact plate 214a.

In this embodiment, as shown in FIG. 4 (A), the glove 2a of the real glove 20 has a double mounting structure for mounting (covering) a fingertip (for example, two members such as cloth are superposed). A structure in which some member can be housed between the two stacked members), and one of the actuator wires 212 is placed inside the portion that covers the fingertip (inside the double structure). A wire holding portion 213, a support portion 214b, and a tactile sense pin support portion 215 for restricting movement of the tactile sense pin 214 in the movement direction of the actuator wire 212. Then, one end side of the support portion 214b projects and the contact plate 214a is positioned inside the glove 2a in which the fingers are stored.

In this embodiment, the contact plate 214a thereof is
Is always in contact with the fingertips of the hand wearing the real glove 20. The contact plate 2 is not limited to this.
14a may be configured to be in a state of being separated from the fingertip (a state where the contact plate 214a and the fingertip are not in contact with each other).

In FIG. 4A, in order to simplify the explanation,
One unit tactile sense generation unit 210 (one tactile sense presentation pin 21
4) is shown as a representative, as described above, in reality, the tactile sense generating unit 21 includes a plurality of unit tactile sense generating units 210.
The plurality of tactile sense presenting pins 214 are provided at positions corresponding to the finger pad of the user's fingertip when the user wears the real glove 20, for example, along the finger pad and on the finger pad. They are arranged in a matrix.

Before and after each tactile sense presenting pin 214, a tactile sense presenting pin support part 215 for restricting the movement of the supporting part 214b of the tactile sense presenting pin 214 in the movement direction of the actuator wire 212 is provided.

The tactile sense presenting pins 214 may be regularly arranged or may be irregularly arranged.

Next, a tactile sense generating operation by the tactile sense generating section 21 will be described. When the user puts on the real glove 20 and moves his hand or finger, the fingertip (finger pad) virtually touches an object (virtual object) appearing in the virtual space, that is, an object, the information processing device 8 Calculates the pressing force when there is an actual contact, and converts it into PWM data of the power generation unit 211 (for example, data indicating an excitation pattern for rotating the motor 211a) based on the calculation result.

In this case, in the virtual space (three-dimensional space), X-axis, Y-axis and Z-axis which are orthogonal to each other are previously set.
That is, X-Y-Z coordinates (three-dimensional coordinates) are assumed, and a match between the coordinates of the object and the coordinates of the fingertip of the user is detected, and when the match is detected, the coordinates of the object are detected. It is determined that the corresponding part and the part corresponding to the coordinates of the fingertip of the user have come into contact with each other. The coordinates of the user's fingertip are derived based on a signal (information) transmitted from the position / orientation detection unit 23, which will be described later, to the information processing device 8 via the signal transmission / reception unit 26.

Next, the information processing device 8 determines the derived PW.
The M data and the data for specifying (designating) the power generation unit 211 that operates the contact plate 214a that applies the pressing force to the contact position when it is actually contacted are transmitted to the real glove 20. The real globe 20 has a power generation unit 21 designated according to the received PWM data.
1 to wind up the actuator wire 212.

When the actuator wire 212 is wound up, it is placed on the finger pad.
Moves along the finger pad toward the tip of the finger, as shown in FIG.
As shown in (B), the tactile sense presenting pin 214 fixedly attached to the actuator wire 212 also moves along the finger pad toward the tip direction of the finger.

At this time, as shown in FIG. 4C, the tactile sense presenting pin 214 is restricted by the tactile sense presenting pin support portion 215 from moving in the tip direction of the finger, and is guided by the tactile presenting pin support portion 215. Then, it moves to the upper side in FIG. 4C (projects toward the finger pad). That is, the contact plate 214a of the tactile sense providing pin 214 moves substantially vertically upward with respect to the surface of the finger pad. Support part 214 of tactile presentation pin 214
In the standby state (in the initial state), b is inclined from the vertical direction to the tip direction of the finger as shown in FIG. 4 (B), and the contact plate 214a acts so as to move almost vertically upward. As a result, the contact plate 214a presses the finger pad in a substantially vertical direction, thereby applying a pressing force to the finger pad of the user's hand.

This operation is performed by the plurality of unit tactile sense generating units 210.
Of the designated unit tactile sensation generating unit 210, and the tactile sensation is presented (applied) to the finger pad of the user's hand. Therefore, the user can obtain, on the finger pad, a feeling corresponding to the contact of the finger pad with the object (object) in the virtual space image.

In the designated unit tactile sensation generating section 210, when the driving of the motor 211a of the power generating section 211 is stopped and the winding state of the actuator wire 212 is released, the wire holding section 213 restores the elastic force (elastic force). ), The actuator wire 212 that has been wound is pulled out.

When the actuator wire 212 is pulled out, it is placed on the finger pad.
Moves toward the base end direction of the finger (right side in FIG. 4C) along the finger pad, and the tactile sense providing pin 214 fixedly attached to the actuator wire 212 also moves to the base end of the finger along the finger pad. Moving in the direction, the tactile presentation pin 214 is shown in FIG.
It returns to the initial state shown in (B). As a result, the pressing force applied from the tactile sense providing pin 214 to the finger pad of the user's hand is substantially eliminated.

According to the tactile sensation generating section 21, the actuator wire 212 is arranged substantially parallel to the finger pad (along the finger pad), and the actuator wire 212 is arranged.
Since the force is applied almost perpendicularly to the finger pad by winding, the mechanism for providing the tactile sensation can be made thin, which allows the real glove 20 to be used.
The thickness on the finger pad side of can be suppressed as much as possible.

Further, it is possible to prevent the slack of the actuator wire 212, and it is possible to more accurately and reliably apply a large target pressing force from the tactile sense providing pin 214 to the finger pad of the user.

FIG. 5 and FIG. 6 are external views showing an example of the structure of the force sense generator 22. Note that FIG. 5 shows a state in which the frame cover is removed.

As shown in FIG. 5, the force sense generator 22 is provided on the back side of each finger of the user when the real glove 20 is worn. Since the configuration and operation of the force sense generator 22 are almost the same, one of them will be representatively described below.

The force sense generator 22 includes a link mechanism having a plurality of (six in this embodiment) frames 221 that are rotatably connected, and a plurality (seven in this embodiment) that rotate each frame 221. Link motor (driving source) 222
And a plurality of gloves provided on the glove 2a (four in this embodiment)
Frame holder 2b.

When the user wears the real glove 20, the frame holder 2b has a mounting portion located on the tip side of the first joint of the user's finger, the first joint and the second joint (the middle phalanx and the proximal joint). (A joint between the bone) and a mounting portion located between the second joint and the third joint,
It is provided in each of the mounting portions located on the proximal end side of the third joint (the joint between the proximal phalanx and the metacarpal bone).

The link mechanism is attached to the back of the user's finger when the real glove 20 is worn on the glove 2a by the frame holder 2b.

That is, when the user wears the real glove 20, the frame 221 at the tip side (right side in FIG. 5) of the link mechanism is located at the tip side of the first joint of the user's finger. The frame 221 is rotatably installed with respect to the frame holder 2b.

The base end side of the link mechanism (left side in FIG. 5)
The frame 221 at the end of the
When the user wears 0, the user's finger is arranged such that its end portion is located at a portion closer to the base end side than the third joint, and the base end portion of this frame 221 is rotatable with respect to the frame holder 2b. It is installed in.

Further, each of the remaining frames 22 of the link mechanism is
1 is when the user wears the real glove 20,
A portion of the user's finger between the first joint and the second joint, and the second joint
A frame connecting portion is provided at a portion between the joint and the third joint.
It is arranged so as to be positioned every other turn, and is installed so as to be rotatable with respect to the frame holder 2b at every other frame connecting portion.

Driving force (rotational force) of each link motor 222
Are transmitted to the corresponding frames 221 via power transmission units (not shown). That is, when the link motor 222 is rotationally driven in the predetermined direction, the corresponding frame 221 is rotated in the predetermined direction, whereby the angle of the frame 221 is changed (adjusted).

Note that FIG. 5 shows the bundle 212a of the actuator wires 212 described above.

As shown in FIG. 6, each frame 221 and each link motor 222 of the force sense generator 22 are housed in a frame cover 2c made of cloth, for example.

Further, as shown in FIG. 6, inside the frame cover 2c, a plurality of (four in this embodiment) position / orientation detecting units (positions) for detecting the position and attitude of a predetermined part of a hand or finger are positioned. The detection means) 23 is installed. Each of the position / orientation detection units 23 is provided so as to be able to maintain a fixed positional relationship with respect to a portion (detection position) for detecting the position and orientation.

That is, each position / orientation detector 23 is attached and fixed to each link motor 222 fixed to the position where the frame holder 2b is provided in this embodiment. Therefore, the positional relationship between the part where each position / orientation detection unit 23 is provided and the predetermined part on the palm side of the user is kept constant. Therefore, the position and orientation detection unit 23 can easily and accurately derive the position and orientation of a predetermined part on the palm side of the user by detecting the position and orientation of the user.

Further, since each position / orientation detecting unit 23 is arranged on the back side of the hand, the real glove 2
The thickness on the finger pad side of 0 can be suppressed as much as possible.

Further, as shown in FIG. 3, the position / orientation detecting unit 23, when the user wears the real glove 20,
It is also installed at a position corresponding to the base end side portion of the arm of the real glove 20.

The position / orientation detecting unit 23 can surely grasp the position and orientation of each part such as each finger, hand, wrist, arm, etc. on which the real glove 20 is worn.

Next, the force sense generating operation by the force sense generating section 22 will be described. When the user wears the real glove 20 and moves his or her hand or finger, the hand or finger virtually touches an object (virtual object) appearing in the virtual space, that is, an object. The calculation processing unit 81 described later calculates the reaction force from the object to the finger (each joint of the finger) in the case of actual contact, and based on the calculation result, the PWM data of each link motor 222 (for example, , Data indicating an excitation pattern for rotating the link motor 222).

The determination of the contact between the object and the user's hand or finger is the same as in the case of the tactile sense generating section 21.

The information processing device 8 transmits the derived PWM data and the data indicating each link motor 222 to be driven to the real globe 20. Real glove 20
Adjusts the angle of each frame 221 by rotating the designated link motor 222 in a predetermined direction and rotating the corresponding frame 221 in a predetermined direction in accordance with the received data. Frames 22 on both ends
The angle of 1 and the angle of each frame connection part are adjusted.

By adjusting the angle of the frame 221, a predetermined amount of force is applied to each of the predetermined frame holders 2b, whereby a force corresponding to the reaction force from the object appearing in the virtual space is applied by the user. Given to each joint of the finger. That is, a force sensation is presented (applied) to each joint of the finger.

Here, the force-sense generating section 22 has a function of adjusting the intervals between a plurality of mounting sections to be mounted on a plurality of parts of a finger (also serves as adjusting means).

For example, as shown in FIG. 5, by rotating a predetermined frame 221 and adjusting the angle of the frame 221, the intervals between the frame holders 2b can be lengthened or shortened. You can

FIG. 7 is a block diagram showing an example of the configuration of the position / orientation detection unit 23 of the real glove 20. In this embodiment, a quadrature coil type (orthogonal coil type) position sensor is used for the position / orientation detection unit 23. That is, as shown in FIG. 7, the position / orientation detection unit 23 includes an X-direction detection coil 231, a Y-direction detection coil 232, and a Z-direction detection coil 233. In addition, X direction, Y direction,
The Z direction is orthogonal to each other.

On the other hand, the information processing device 8 is provided with a magnetic field generator 88, which will be described later, for generating a magnetic field. The magnetic field generation unit 88 has, for example, a configuration substantially similar to that of the position / orientation detection unit 23, that is, an orthogonal coil type (orthogonal coil type) magnetic field generator (X-direction coil, Y-direction coil, Z).
Directional coil) is used.

The magnetic field generated by this magnetic field generating section is detected by the position / orientation detecting section 23. In this case, X of the magnetic field generator
Directional coils, Y-direction coils, and Z-direction coils generate time-divided magnetic fields in order, and the magnetic fields generated by the coils are respectively detected by the X-direction detection coils 231 and Y of the position / orientation detection unit 23.
Three coils, the direction detection coil 232 and the Z direction detection coil 233, are used for detection.

The signals (detection data) detected in the XYZ directions by the position / orientation detection unit 23 are amplified by the amplification unit 251 of the signal processing unit 25, respectively, and are A /
After being converted into a digital signal by the D conversion unit 252, FIG.
Is output to the signal transmitting / receiving unit 26, and is transmitted to the information processing device 8 by the signal transmitting / receiving unit 26.

The information processing device 8 receives the signal,
Based on the signal (information), the position and posture of each part such as each finger, hand, wrist, and arm wearing the real glove 20 is derived, and the information is used in each predetermined process.

In this embodiment, the virtual space display device 5 is of a form to be mounted on the face for use, that is, H
A spectacle-shaped device called MD (head mounted display) is used.

FIG. 8 is a perspective view showing an example of the external configuration of the virtual space display device 5, and FIG. 9 is a block diagram showing an example of the circuit configuration of the virtual space display device 5.

The virtual space display device 5 is a wearer (user).
The image display portions 51a, 5a for displaying the virtual space image with respect to
1b and line-of-sight detection units 52a, 5 for detecting the line of sight of the wearer
2b and the concave / convex input unit 53 that captures the surrounding image (video)
a, 53b, a control unit 54 that controls the entire apparatus, a signal transmission / reception unit 55 that transmits / receives signals to / from the information processing device 8,
A position / orientation detection unit (position detection unit) 56 that detects a user's viewpoint, a mounting unit 57 that hooks and holds the user's ears, and a signal processing unit 58 that processes various signals are provided.

FIG. 10 is a diagram (block diagram, schematic diagram) showing a configuration example of the unevenness input section 53a of the virtual space display device 5. As shown in FIG. 10, the concave / convex input unit 53a has a lens 531, a CCD (imaging device) 532, a CCD control unit 533, and a CCD output amplification unit 534.

The light (light flux) incident on the lens 531 is guided by the lens 531 onto the light receiving surface (imaging surface) of the CCD 532 to form an image, and the subject image (surrounding image).
Are imaged by the CCD 532. This CCD532
The drive of is controlled by the CCD control unit 533.

The signal output from the CCD 532 is CC
It is amplified by the D output amplifying section 534, and then the unevenness input section 5
3a is output and input to the signal processing unit 58 shown in FIG. Then, the signal processing unit 58 performs predetermined signal processing, and the signal transmitting / receiving unit 55 transmits the signal to the information processing device 8.

Since the structure and operation of the unevenness input section 53b are the same as those of the unevenness input section 53a, the description thereof will be omitted.

FIG. 11 is a diagram (block diagram, schematic diagram) showing a configuration example of the image display section 51a of the virtual space display device 5. In FIG. 11, the image display unit 51 a includes a lens 511 and a lens moving mechanism 5 that moves the lens 511.
12, stepping motor 513, and motor controller 5
14, a liquid crystal display panel 515 having a color filter (not shown), a liquid crystal drive unit 516, and a backlight 51.
7, the backlight control unit 518, and the transistor 51
9 and 9.

When the virtual space image is displayed on the image display section 51a, the transistor 519 is turned on by the control of the backlight control section 518 to turn on the backlight 517.
Is driven, and the backlight 517 drives the liquid crystal display panel 51.
5 is irradiated with light.

Then, the liquid crystal drive section 516 is connected to the control section 54.
The drive of the liquid crystal display panel 515 is controlled based on the display signal (image data) input from the liquid crystal display panel 515, whereby the virtual space image is displayed on the liquid crystal display panel 515.

The wearer (user) can visually recognize the virtual space image displayed on the liquid crystal display panel 515 through the lens 511 and the half mirror 521 of the line-of-sight detecting unit 52a described later.

The driving of the stepping motor 513 is controlled by the motor control section 514. When the stepping motor 513 rotates in a predetermined direction, the lens moving mechanism 51 is moved.
2, the lens 511 moves in a direction approaching the liquid crystal display panel 515, that is, a direction away from the wearer's eyes.

When the stepping motor 513 rotates in the opposite direction to the above, the lens moving mechanism 512 moves the lens 511 away from the liquid crystal display panel 515, that is, toward the wearer's eyes.

By changing the position of the lens 511, the perspective of the virtual space image can be changed.

The adjustment of the position of the lens 511, that is, the adjustment of the perspective of the virtual space image is performed based on the information on the line of sight of the wearer detected by the line-of-sight detection units 52a and 52b described later.

Since the structure and operation of the image display section 51b are the same as those of the image display section 51a, the description thereof will be omitted.

The line-of-sight detection units 52a and 52b have a function of detecting in which direction the line of sight of the wearer is (where in the virtual space image the focus is intended), and the wearer's line of sight for use as security information. It has the function of reading the iris pattern of the eye.

FIG. 12 is a diagram (block diagram, schematic diagram) showing an example of the configuration of the line-of-sight detection unit 52a of the virtual space display device 5. 12, the line-of-sight detection unit 52a includes a half mirror 521, a filter 522 that removes visible light and transmits infrared light, a lens 523, a CCD (imaging device) 524, and a CCD control unit 525. CCD output amplifier 526 and LED (light emitting diode) that emits infrared light
527, LED control unit 528, and transistor 529.
And have.

The LED 527 is driven by the LED control unit 52.
When the transistor 529 is turned on under the control of the LED control unit 528, the LED 527 is driven and infrared light is emitted from the LED 527.

The infrared light emitted from the LED 527 is transmitted through the filter 522, part of it is reflected by the half mirror 521, and is radiated to the eyes of the wearer.

Then, reflected light (light flux) from the wearer's eyes
Is partially reflected by the half mirror 521 and transmitted through the filter 522, and the lens 523 causes the CCD 524 to pass through.
The image of the subject (the image of the wearer's eyes) is guided and formed on the light receiving surface (image pickup surface) of the CCD 524. The driving of the CCD 524 is controlled by the CCD controller 525.

The signal output from the CCD 524 is CC
It is amplified by the D output amplification unit 526, and then the line-of-sight detection unit 5
It is output from 2a, is input to the signal processing unit 58 shown in FIG. 9, is subjected to predetermined signal processing, and is transmitted to the information processing device 8 by the signal transmitting / receiving unit 55.

In the information processing device 8, the virtual space display device 5
From the signal (information) from, the direction of the wearer's line of sight is detected (where in the virtual space image to focus), and
The iris pattern of the wearer's eye is read, analyzed, and used as security information.

On the other hand, from the lens 523 side to the filter 522
The visible light incident on the filter 522 is removed by the filter 522, and the visible light can be prevented from being emitted to the eyes of the wearer. As a result, it is possible to prevent the visibility of the virtual space image displayed on the liquid crystal display panel 515 from being degraded.

Since the structure and operation of the line-of-sight detection unit 52b are the same as those of the line-of-sight detection unit 52a, description thereof will be omitted.

The virtual space display device 5 includes two image capturing devices (imaging means), that is, the unevenness input section 53.
The surrounding image is captured and captured by a and 53b, and the captured image data (video data) is transmitted to the information processing device 8.

The information processing device 8 includes the concave / convex input units 53a, 5a and 5b.
The visual angle difference is derived from the image captured in 3b, that is, the captured image data, and the virtual space image, that is, the image data of the virtual space is created based on the surrounding irregularities obtained from the visual angle difference and the like. Then, the information processing device 8 arranges the created image (video) of the virtual space and a predetermined virtual object such as a virtual wallpaper, a virtual personal computer, or a virtual book in the virtual space so as to match the surrounding unevenness. Image data (video data) indicating a composite image (composite video) obtained by combining the image and the image is transmitted to the virtual space display device 5.

The virtual space display device 5 displays the image based on the image data from the information processing device 8 in the image display section 51.
It is displayed using a and 51b.

In addition, the virtual space display device 5 is designed so that when the worn real glove 20, that is, the user's hand or arm is within the area of the image displayed by the image display portions 51a and 51b. The images corresponding to the hands and arms are also displayed on the image display units 51a and 51b.

In this case, as described above, the detection data of the plurality of position / orientation detecting units 23 provided in the real glove 20 are transmitted to the information processing device 8, whereby the position and the posture of the real glove 20, that is, the posture. The user's fingers, hands, wrists, arms, etc. are recognized by the information processing device 8.
That is, the information processing device 8 is step by step in the real globe 2
Each finger of the user based on the detection data from 0,
Positions and postures of hands, wrists, arms, etc. are specified, and when they are within the area of the image displayed by the image display units 51a and 51b, a virtual space image showing them is created, and The image is combined with the basic image to obtain a combined image (image data of the combined image), and the image data is transmitted to the virtual space display device 5.

Therefore, the user's hand or arm is displayed on the image display unit 51.
When it is within the area of the image displayed by a and 51b, the image corresponding to the hand or arm is displayed on the image display units 51a and 51b.

Then, in conjunction with the actual movement of the hand or finger wearing the real glove 20, the image display portions 51a and 51b are linked.
The hands and fingers move in the virtual space image displayed in.

The position / orientation detector 56 determines the position and orientation of the virtual space display device 5, that is, the face (head) of the wearer.
Is a sensor for detecting the position and orientation of the. In this embodiment, a quadrature coil type position sensor, that is, the same as the position / orientation detecting unit 23 is used as the position / orientation detecting unit 56.

The signal (detection data) from the position / orientation detecting unit 56 is input to the signal processing unit 58, subjected to predetermined signal processing, and transmitted to the information processing device 8 one by one by the signal transmitting / receiving unit 55. The information processing device 8 updates the virtual space image one by one at a predetermined time interval based on the detection data.

Therefore, for example, when the user wearing the virtual space display device 5 moves his / her neck, the image display section 51
The virtual space images displayed on a and 51b change so as to be an image that matches the direction in which the user is facing.

FIG. 13 is a block diagram showing a circuit configuration example of the information processing device 8 shown in FIG. In FIG. 13, the information processing device 8 includes an arithmetic processing unit (arithmetic means) 81.
A collision determination unit (collision determination unit) 82, an image generation unit (image generation unit) 83, a control unit (control unit) 84,
A storage unit (storage unit) 85, a signal processing unit 86, a signal transmission / reception unit (communication unit) 87, and a magnetic field generation unit 88 are provided.

When the finger or hand of the above-mentioned real glove 20 comes into contact with an object in the virtual space, the arithmetic processing section 81 determines what the object is like,
Then, the reaction force to the finger (each joint of the finger) is calculated, and the calculation result is converted into PWM data of each link motor 222 of the real glove 20, and the conversion data is converted into the signal processing unit 8.
6 and the signal transmitting / receiving unit 87, the real glove 20
Send to.

Further, the arithmetic processing section 81 uses the control section 84 to generate the image generation section 8 in the virtual main alignment processing described later.
The virtual space image generated by updating in 3 is input, the moving speed and the moving direction when the virtual book moves are calculated based on the image data, and the calculation result is output to the control unit 84.

In the virtual book alignment process described later, the collision determination unit 82 determines that an object (a virtual book in the virtual book alignment process of FIG. 17) moving in the virtual space is an obstacle (a virtual book of FIG. 17). In this alignment processing, it is determined whether or not a collision has occurred with a virtual wall. Actually, the collision determination unit 82
Specifies the current coordinate position of the virtual book from the moving speed and moving direction of the virtual book calculated by the arithmetic processing unit 81, and the current coordinate position of the virtual book is the planar coordinate of the obstacle (usually, the obstacle is planar). It is determined whether or not Then, the collision determination result is output to the control unit 84.

The image generation unit 83 is imaged by the virtual space image previously received from the base station 9 or the unevenness input units 53a and 53b of the virtual space display device 5 according to the instruction of the control unit 84, and is converted from the captured image. The image (for example, a virtual book) specified by the user and the image of the arm or finger of the user are superimposed on the virtual space image to generate a combined virtual space image, and the combined virtual space image is generated by the signal processing unit 86. Output to. Further, the image generation unit 83 updates and generates the virtual space image at predetermined time intervals with the movement of the virtual space display device 5, and outputs it to the signal processing unit 86.

Further, the image generation section 83 has a collision determination section 82.
Based on the collision determination result output from the control unit 84, the target to be moved by the instruction of the control unit 84 as if facing the user's standing position, that is, the direction of the virtual space display device 5 (HMD). Generate an image that displays an object in front of a wall (eg, a bookshelf, etc.). The generated image is transmitted to the virtual space display device 5 via the signal processing unit 86 and the signal transmitting / receiving unit 87.

The control unit 84 controls the operations of the arithmetic processing unit 81, the collision determination unit 82, and the image generation unit 83,
The calculation result, the collision determination result, the generated image data, and the like are saved in a predetermined storage area of the storage unit 85, and
They are transmitted to the corresponding devices via the signal processing unit 86 and the signal transmitting / receiving unit 87.

The storage unit 85 has a computer-readable storage medium (also referred to as a recording medium, which is also referred to as a recording medium in the claims) in which programs and data are stored in advance. The storage medium is, for example, a RAM (Random Access Memory: volatile,
Including non-volatile), FD (Floppy Disk (Flo
ppy is a registered trademark)), HD (Hard Disk), CD-ROM
It is composed of a magnetic or optical recording medium such as (Compact Disc Read-Only Memory) or a semiconductor memory. This storage medium is fixedly provided in the storage unit 85 or is detachably mounted. The storage medium includes the virtual space display device 5, the information processing device 8, and the tactile / force sense presentation. Various application programs corresponding to the device 2, a tactile / force sense generation processing program, a virtual main alignment processing program, data processed by each processing program, document data, and the like are stored.

Further, a part or all of the programs, data, etc. stored in this storage medium are transmitted from other devices such as servers and clients (base station 9 in the present embodiment) to network lines (eg, the Internet). ,
Signal transmission / reception unit 87 via a transmission medium such as a telephone line)
It may be configured to be received and stored from the server, and the storage medium may be the storage medium of the server of the base station 9 constructed on the network. Furthermore, the program may be transmitted from a server or a client via a transmission medium such as a network line and installed in the information processing device 8. Further, the program may be a program for realizing some of the functions described above, and may be a program that can realize the functions described above in combination with a program already recorded in the computer system.

The signal processing unit 86 performs signal processing such as amplification of image data and control signals transmitted to the virtual space display device 5 and the tactile / force sense presentation device 2. The signal transmitting / receiving unit 87 is
Control is performed to communicate various signals and data processed by the signal processing unit 86 with external devices such as the virtual space display device 5 and the tactile / force sense presentation device 2.

Although the magnetic field generator 88 is not shown in detail, for example, an orthogonal coil type (orthogonal coil type) magnetic field generator (X-direction coil, Y-direction coil) corresponding to the position / orientation detector 23 of FIG. 7 is used. , Z direction coil). The magnetic field generation unit 88 receives X, Y, Z
A magnetic field is generated in the order of the direction coils, and the detection coils 231 and 23 in each direction of the position / orientation detection unit 23 of the real globe 20.
2 and 233 to detect the magnetic field generated,
The position / orientation detection unit 56 of the virtual space display device 5 also detects the magnetic field.

The magnetic field generation unit 88 may be stored in the information processing device 8 as shown in the figure, but may be configured separately from the information processing device 8 as a magnetic field generation device. In particular,
When a mobile phone or PDA is used as the information processing device 8, it is not possible to provide a magnetic field generator inside the mobile phone or PDA, so it is useful to use a different configuration. Even if a magnetic field generator (or magnetic field generator) is separately provided in this way,
The control is performed by a control signal from the information processing device 8.

Next, the image processing apparatus 1 according to the first embodiment
The operation (action) of will be described. Here, a case where the user operates the virtual personal computer appearing in the virtual space by using the image processing apparatus 1 will be described as an example.

The user turns on the power of the respective parts, puts on the real gloves 20 on both hands, and puts on the virtual space display device 5 in the form of glasses (worn on the face). Then, the virtual space image including the virtual personal computer is displayed on the image display units 51a and 51b of the virtual space display device 5.

Also, the user's arm is placed on the image display portions 51a, 5a and 5b.
If the image is displayed within the virtual space image displayed in 1b, the image of the hand or arm created by the image generation unit 83 of the information processing device 8 is displayed as a virtual object in the virtual space image. Then, when the user moves the hand or finger, the displayed hand or finger moves in the virtual space image in conjunction with the actual movement of the hand or finger.

In the information processing apparatus 8, the collision determining unit 82 causes the object in the virtual space image, that is, in the present embodiment, for example, a part of the virtual personal computer (eg, keyboard) and a part of the user's finger to be the same. When it is detected that the coordinates have been reached, an instruction for generating a tactile sensation and a force sensation is issued to the real glove 20 based on the calculation result of the calculation processing unit 81. Specifically, the information processing device 8 is configured such that, with respect to the real globe 20, the arithmetic processing unit 8
1. The power generation unit 211 to be driven obtained from the calculation result of the reaction force calculated by 1 and PWM data used for driving the same,
The link motor 222 to be driven, PWM data used for driving the link motor 222, and the like are transmitted via the signal transmitting / receiving unit 87.

According to the instruction from the information processing device 8, the real globe 20 operates the designated motion generating sections 211,
PW for each specified torque and specified time
It is driven by M control. As a result, the contact plate 214a of each tactile sense providing pin 214, which is arranged at the same coordinate as a part of the virtual personal computer, projects,
A tactile sensation is given to each fingertip of the user having the same coordinates as a part of the virtual personal computer. That is,
Each fingertip is given a feeling of touch, texture, shape, strength, and the like.

At the same time, the real globe 20 drives each designated link motor 222 by PWM control for a designated time with a designated torque in accordance with an instruction from the information processing device 8. As a result, the force applied to each joint of each finger when touching a keyboard of an existing personal computer, that is,
A force corresponding to the pressing force (reaction force) applied to each joint of each finger is applied to each joint of each finger of the user.

When the user further moves the finger in the direction of pressing the key of the virtual personal computer, a virtual space image showing how the key is pressed is displayed on the image display portions 51a and 51b in accordance with the coordinate change of the finger. To be done.

In this case, the information processing device 8 generates image data of the virtual space image for reproducing the change of the display state based on the pressed key in the virtual space image in the image generation unit 83, and generates it in the virtual space. The virtual space display device 5 performs a process of transmitting to the display device 5, and based on the image data of the virtual space image received from the information processing device 8, the virtual space display device displays a display in which a key press is reflected. Display the images that are made.

As described above, according to the tactile / force sense presentation device 2 and the image processing device 1, the real glove 20 can be easily and surely attached and detached, and the operation in the virtual space ( Operation) can be performed easily and reliably.

Further, according to the virtual space image, the tactile sensation and the force sensation can be presented to each finger accurately and surely.

As a result, the user can comfortably, easily and surely perform the operation (operation) in the virtual space.

Particularly, in the tactile sense generating section 21, it is possible to prevent the slack of the actuator wire 212, and more accurately and reliably apply a target large pressing force from each tactile sense presenting pin 214 to each finger pad of the user. Can be given.

Thus, the user can more accurately and surely feel the sensation corresponding to the contact of each finger pad with the object (object) in the virtual space image.

Further, in the tactile sense generating section 21, the actuator wires 2 are arranged substantially parallel to the finger pads (along the finger pads).
12 is arranged, and by winding up its actuator wire 212, a force almost perpendicular to each finger pad is applied, so that the mechanism for giving a tactile sensation can be made thin, which makes it possible to realize a realistic structure. The thickness of the glove 20 on the finger pad side can be suppressed as much as possible.

Further, since the force sense generating section 22 has the link mechanism, it can be dealt with both the state where the finger is extended and the state where the finger is bent, thereby more reliably. Force can be given to each finger.

In the above-described embodiment, the case where the personal computer displayed in the virtual space image is virtually operated has been described as an example of use. However, the present invention is not limited to this and, for example, a virtual book is used. It can also be used to handle other items such as. When handling a virtual book, for example, when touching the edges of a virtual book or turning a page, feel the touch or force applied to a finger when touching the edge of a real book or turning a page. You can

Further, in the present invention, the virtual space image may not be displayed. That is, there may be predetermined data corresponding to the virtual space image, and a tactile sensation or a force sensation may be presented to the finger based on the data.

Next, a second embodiment of the tactile / force sense presentation device and the image processing device of the present invention will be described. 14
FIG. 8 is a vertical cross-sectional view showing a configuration example of a tactile sense generating unit of the tactile sense / force sense presentation device of the second embodiment.

Hereinafter, the tactile and force sense presentation device 2 and the image processing device 1 of the second embodiment will be described focusing on the differences from the first embodiment described above, and the same matters will be described.
The description is omitted.

As shown in FIG. 14 (A), in the tactile force sense presentation device 2 of the second embodiment, each unit tactile sense generating section (actuator) 210 of the tactile sense generating section 21 has a motor (driving source). ) 211 a including a power generating unit (moving means) 211 and an actuator wire (flexible linear body) 212 movably installed in the longitudinal direction.
And a tactile sense presentation unit 216 that presses the finger pad to generate a tactile sense.

The tactile sense presentation unit 216 is used to give a touch feeling (presence or absence of contact, strength, etc.) to the finger pad of the user.
For example, it is formed of an elastic member (elastic body) such as a leaf spring. In this case, the tactile sense presentation unit 216 preferably has appropriate rigidity, and can be formed of, for example, an elastic body made of various metals.

The tactile sense presenting section 216 includes a first plate piece (pressing section) 216a and a second plate piece (sliding section) 216b.
It consists of and. The first plate piece 216a and the second plate piece 216b are joined at one end side, and the other end side of the first plate piece 216a has a minute contact portion 216c that is a portion that comes into contact with the finger pad. Is provided. This contact part 2
16c can be formed, for example, by bending the end portion of the first plate piece 216a toward the second plate piece 216b side.

Further, the one end side of the tactile sense presentation section 216, that is, the left end in FIG. 14A, is curved so as to be convex downward in FIG. 14A.

The first plate piece 216a and the second plate piece 216
A columnar guide part (moving direction restricting means) 217 for restricting the moving direction of the first plate piece 216a is installed between it and b.

A holding plate 218 is installed on the lower side of the tactile sense presenting section 216 in FIG. 14 (A), and the tactile sense presenting section 216 moves (slides) along the holding plate 218.
Is ready to go. In this case, the tactile sense presentation unit 216
14 has a curved left end in FIG. 14 (A), it can move smoothly and reliably.

The guide portion 217 is fixed to the pressing plate 218, and the pressing plate 218 is fixed to the glove 2a.
That is, both the guide portion 217 and the pressing plate 218 are fixedly installed on the glove 2a.

Also, the second plate piece 21 of the tactile sense providing section 216
One end of the actuator wire 212 is fixed to the end portion of 6b on the right side in FIG. 14 (A).

When the actuator wire 212 is wound up by driving the motor 211a of the power generating section 211,
The actuator wire 212 is shown in FIG.
4 (A) to the right (in the direction of the proximal end of the finger), and along with this, the tactile sense presentation unit 216 attached and fixed to the actuator wire 212 also moves to the right in FIG. 14 (A) along the finger pad.

At this time, as shown in FIG. 14 (B), the first plate piece 216a of the tactile sense providing section 216 has the guide section 217.
14B, the movement to the right side in FIG. 14B is restricted, and the guide portion 217 guides the movement to the upper side in FIG. 14B (projects toward the finger pad). That is,
The tactile sense providing portion 216 is elastically deformed, and the first plate piece 216 thereof is
By changing the posture of “a” to the posture shown in FIG. 14B, the contact portion 216c moves (is pushed up) substantially vertically to the surface of the finger pad.

On the other hand, the actuator wire 212 is moved by the restoring force (elastic force) of the tactile sense presenting section 216 as shown in FIG.
(B) It is urged toward the middle left side (the direction opposite to the moving direction of the actuator wire 212 when the first plate piece 216a is projected). That is, the tactile sense presentation section 216 (first plate piece 216a) also serves as the urging means.

The contact portion 216c presses the finger pad in a substantially vertical direction by the force of the contact portion 216c attempting to move substantially vertically upward, thereby applying a pressing force to the finger pad of the user's hand. give.

This operation is performed by the plurality of unit tactile sense generating units 210.
Of the designated unit tactile sensation generating unit 210, and the tactile sensation is presented (applied) to the finger pad of the user's hand. Therefore, the user can obtain, on the finger pad, a feeling corresponding to the contact of the finger pad with the object (object) in the virtual space image.

In the designated unit tactile sense generator 210, when the driving of the motor 211a of the power generator 211 is stopped and the winding state of the actuator wire 212 is released, the restoring force (elastic force) of the tactile sense presenter 216 is restored. ) And the pressure from the user's finger, the tactile sense presentation unit 216 returns to the initial state shown in FIG. As a result, the pressing force applied from the tactile sense presentation unit 216 to the finger pad of the user is substantially eliminated.

According to the tactile / force sense presentation device 2 and the image processing device 1, the same effect as that of the above-described first embodiment can be obtained.

Next, a third embodiment of the tactile / force sense presentation device and the image processing device of the present invention will be described.

FIG. 15 is an external view (a part of which is not shown) showing an external configuration example of the real glove of the tactile / force sense presentation device of the third embodiment. In addition, in FIG. 15, the tactile sense generating unit 21, the power generating unit 211, the force sense generating unit 22 shown in FIG.
Position / orientation detection unit 23, control unit 24, and power supply unit 30
Etc. are not shown.

Hereinafter, the tactile / force sense presentation device 2 and the image processing device 1 of the third embodiment will be described focusing on the differences from the above-mentioned first embodiment, and the same matters will be described.
The description is omitted.

As shown in FIG. 15, in the tactile / force sense presentation device 2 of the third embodiment, the glove 2a of the real glove 20 is used.
An arm fixing portion 2d having a form (air injection method) of fixing to the arm by air pressure (pressure) by injecting air (air) is provided at an end portion of the tubular portion 20a on the elbow side (base end side). .

The arm fixing portion 2d is composed of a bag-shaped arm fixing portion main body 21d into which air is injected and an air injecting portion 22d provided in the arm fixing portion main body 21d. The outer shape of the arm fixing portion main body 21d is tubular.

The hollow portion of the air injecting portion 22d communicates with the inside of the arm fixing portion main body 21d.
A lid 23d is provided at the tip of the.

When the lid 23d is attached to the tip of the air injecting portion 22d, the airtightness inside the arm fixing portion main body 21d is maintained, and when the lid 23d is removed, the air injecting portion 22d is released from the arm fixing portion main body 21d. Air can be injected and exhausted via.

The arm fixing portion 2d as described above is molded to fit the shape of the user's arm by using a mold such as a rubber mold according to each user.

When wearing the real glove 20, the user first wears the glove 2a and then the arm fixing portion 2
Air is injected into the arm fixing portion main body 21d of d.

In this air injection, for example, the lid 23d is removed from the tip of the air injection part 22d, and air is blown into the arm fixing part main body 21d from the air injection part 22d. Then, after injecting a predetermined amount of air, the lid 23d is attached to the tip of the air injecting portion 22d.

In this way, the real glove 20 is
The air is fixed to the user's arm by the air pressure of the arm fixing portion 2d, and is also fixed to the user's wrist by the restoring force (elastic force) of the wrist fixing portion 28.

When the user removes the real glove 20, first, the arm fixing portion main body 21d of the arm fixing portion 2d.
Evacuate the air from inside and then remove the real glove 20 from your hand.

According to the tactile / force sense presentation device 2 and the image processing device 1, the same effect as that of the above-described first embodiment can be obtained.

Then, in this tactile / force sense presentation device 2,
Since the arm fixing portion 2d is of the air injection type, the real gloves 20 can be easily and quickly attached and detached.

In the present invention, the unit tactile sense generating section 210 of the tactile sense generating section 21 may be configured as in the second embodiment described above.

Next, a fourth embodiment of the tactile / force sense presentation device and the image processing device of the present invention will be described.

FIG. 16 is a block diagram showing an example of the arrangement of a position / orientation detection unit of the tactile / force sense presentation device of the fourth embodiment, and FIG.
FIG. 7 is an external view (partially omitted from illustration) showing a configuration example of a position / orientation detection unit of the tactile / force sense presentation device of the fourth embodiment, and FIG.
FIG. 18 is an external view showing a configuration example of a Z direction detection sensor of the position / orientation detection unit shown in FIG. 17. Note that FIG. 17 and FIG.
In the description, an X axis, a Y axis, and a Z axis that are orthogonal to each other are assumed.

Hereinafter, the tactile / force sense presentation device 2 and the image processing device 1 according to the fourth embodiment will be described focusing on the differences from the first embodiment described above, and the same matters will be described.
The description is omitted.

In the haptic / force sense presentation device 2 of the fourth embodiment, as the position / orientation detecting section (position detecting means) 23, for example, the X-axis, the Y-axis, and the Z-axis which are orthogonal to each other are utilized by utilizing Coriolis force. A gyro sensor (gyroscope) that detects each angular velocity of is used.

Then, in this tactile / force sense presentation device 2,
When the surroundings are imaged by the unevenness input units (imaging means) 53a and 53b and the imaged surrounding image includes a portion corresponding to a finger, the image data (video image) from the unevenness input units 53a and 53b. The position of the finger is obtained based on the data), and when the captured surrounding image does not include the part corresponding to the finger, the position and orientation detection unit 23 detects the position of the finger, and the position is detected. The position of the finger is obtained based on the information from the posture detection unit 23.

As shown in FIG. 17, the position / orientation detecting unit 23
Is provided in the casing 60 and the casing 60,
An X-direction detection sensor (first sensor for detecting an angular velocity around the X-axis)
Sensor 61, Y for detecting the angular velocity around the Y axis
It has a direction detection sensor (second detection sensor) 62 and a Z direction detection sensor (third detection sensor) 63 that detects an angular velocity around the Z axis.

As shown in FIG. 18, the Z-direction detection sensor 6
3 includes an elinvar (base) 64, a piezoelectric element for vibration (piezoelectric element for vibration) 65 and a piezoelectric element for detection (piezoelectric element for detection) 66 installed on the elinvar 64. The Z direction detection sensor 63 is supported by two wires 67 and 68.

The shape of the elinvar 64 is a substantially rectangular parallelepiped elongated in the Z-axis direction. The vibrating piezoelectric element 65 is installed on one of the two surfaces substantially orthogonal to each other, and the detecting piezoelectric element 66 is installed on the other side. Has been done. In this case, the vibration piezoelectric element 65 is arranged on the surface perpendicular to the Y-axis, and the detection piezoelectric element 66 is arranged on the surface perpendicular to the X-axis, but the arrangement is reversed. Good.

When an AC voltage is applied from the AC power supply 69 shown in FIG. 16 to the vibrating piezoelectric element 65 of the Z-direction detecting sensor 63, the vibrating piezoelectric element 65 vibrates in the X-axis direction (flexural vibration). As a result, the elinvar 64 also vibrates in the X-axis direction. At this time, when rotation (angular velocity) occurs around the Z axis, a voltage (signal) of a level corresponding to the angular velocity is generated.
Is detected by the detecting piezoelectric element 66.

The constructions of the X-direction detecting sensor 61 and the Y-direction detecting sensor 62 are similar to those of the Z-direction detecting sensor 63, and therefore their explanations are omitted. When the rotation (angular velocity) is generated around the X-axis, a voltage (signal) of a level corresponding to the angular velocity is detected by the detecting piezoelectric element 66 of the X-direction detection sensor 61, and these actions act around the Y-axis. When rotation (angular velocity) occurs, a voltage (signal) of a level corresponding to the angular velocity is detected by the detection piezoelectric element 66 of the Y-direction detection sensor 62.

As shown in FIG. 16, the position / orientation detector 23
A signal (detection data) corresponding to the angular velocity around the X axis detected by the X direction detection sensor 61, and a signal (detection data) corresponding to the angular velocity around the Y axis detected by the Y direction detection sensor 62. , A signal (detection data) corresponding to the angular velocity around the Z-axis detected by the Z-direction detection sensor 63 is amplified by the amplification unit 251 of the signal processing unit 25, and is amplified by the A / D conversion unit 252. After being converted into a digital signal, the signal is output to the signal transmitting / receiving unit 26 shown in FIG.
Sent to.

The information processing device 8 receives the signal,
Based on the signal (information), the position and orientation of each part such as each finger, hand, wrist, arm, etc., wearing the real glove 20, that is, the coordinates of each part are derived, and the information is used in each predetermined process. In order to do so, it is saved in a predetermined storage area of the storage unit 85.

Next, the operation (action) of the image processing apparatus 1 according to the fourth embodiment will be described. Here, except for the method of obtaining the position and posture of each part such as each finger, hand, wrist, arm, etc. wearing the real gloves 20, the above-mentioned first
The description is omitted because it is the same as the embodiment.

Prior to use, the user first makes the surrounding image picked up by the concave / convex input units 53a and 53b include a portion corresponding to a detected portion such as a finger of the real glove 20. As a result, the image processing apparatus 1 is initialized.

In this initial setting, the virtual space display device 5
Is an image data (video data) imaged by the information processing device 8 by capturing and capturing images of the surroundings by the two image capturing devices (image capturing means), that is, the concave-convex input units (image capturing means) 53a and 53b. ) Is sent.

In this case, since the images picked up by the concave / convex input sections 53a and 53b include a portion corresponding to a portion such as a finger of the real glove 20 to be detected, the information processing apparatus 8 has the concave / convex input section. The position and orientation (coordinates) of the finger or the like are obtained based on the images captured by 53a and 53b, that is, the captured image data. The obtained position and posture are the reference position and the reference posture, respectively.
That is, the reference state is set, and the information of the reference position and the reference posture is stored in a predetermined area of the storage unit 85 of the information processing device 8.

Upon completion of the initial setting, the virtual space display device 5 is ready for use, and the virtual space display device 5 captures and captures the surrounding image by the concave / convex input portions (imaging means) 53a and 53b, and captures the image on the information processing device 8. Send the image data.

[0240] The control unit 84 (discriminating means) of the information processing device 8
Is based on the images captured by the unevenness input units 53a and 53b, that is, the imaged image data.
Real gloves 20 are added to the images captured by 3a and 53b.
It is determined whether or not the part corresponding to the part to be detected such as the finger is included.

If the captured image includes a portion corresponding to the part to be detected such as the finger,
The information processing device 8 obtains the position and orientation (coordinates) of the finger or the like based on the captured image data. The obtained position and posture are set as the reference position and the reference posture, respectively. That is, the information on the reference position and the reference attitude stored in the storage unit 85 is rewritten with the information on the new reference position and the reference attitude.

On the contrary, when the captured image does not include the part corresponding to the part to be detected such as the finger,
The information processing device 8 obtains the position and orientation (coordinates) of the finger or the like based on the signal (information) from each of the position and orientation detection units 23. In this case, the relative position and orientation with respect to the reference position and the reference orientation are obtained from the information from each position and orientation detection unit 23, and the information about the relative position and orientation and the information about the reference position and the reference orientation. The position and posture of the finger or the like are obtained based on

According to the tactile / force sense presentation device 2 and the image processing device 1, the same effect as that of the first embodiment described above can be obtained.

In this tactile / force sense presentation device 2,
The position and orientation of a predetermined part of the hand and each finger can be accurately and surely obtained, and the position and orientation detection unit 2
By using a gyro sensor as 3, the cost can be reduced.

In the present embodiment, the position / orientation detecting section (position detecting means) 23 is composed of a gyro sensor, but the present invention is not limited to this.

Further, in the present invention, the unit tactile-sense generating section 210 of the tactile-sense generating section 21 may be configured as in the second embodiment described above.

Further, in the present invention, the arm fixing portion 2d is of a form (air injecting method) in which it is fixed to the arm by air pressure (pressure) by injecting air as in the third embodiment described above. May be.

Although not described in detail, the above-described configuration can be applied to the position / orientation detection unit 56 of the virtual space display device 5. Also in this case, the same effect as the first embodiment described above can be obtained.

Next, with reference to the flow charts of FIGS. 19 and 20, each configuration of the above-described embodiment, particularly the first
The operation of the image processing apparatus of the present invention will be described based on the configuration of the embodiment. Here, the program for realizing each function described in these flowcharts is
It is stored in the recording medium of the storage unit 85 in the form of a computer-readable program code, and the control unit 84
Sequentially executes the operations according to this program code. In addition, the control unit 84 can also sequentially execute the operation according to the above-described program code transmitted via the transmission medium, as described above. That is, in addition to the recording medium, the program / externally supplied via the transmission medium /
The data can also be used to perform operations specific to this embodiment.

FIG. 19 is a flow chart showing the tactile / force sense generating process in the image processing apparatus of the present invention.
0 is a flow chart showing a virtual main alignment process in the image processing apparatus of the present invention. It should be noted that these flowcharts will be described based on each component in the first embodiment.

First, the tactile sense / force sense generating process will be described.
The image processing device 1 according to the present invention, while the power is turned on for updating the virtual space image and generating tactile and force senses,
This tactile / force sense generation processing is executed. That is, the position and orientation detection unit 23 of the real glove 20 detects the position of the user's finger (and hand) (step S10).
1) After the signal processing unit 25 performs predetermined signal processing, the position signal detected via the signal transmitting / receiving unit 26 is transmitted to the information processing device 8 (step S102).

The information processing device 8 receives this detection signal via the signal transmitting / receiving unit 87. The control unit 84 of the information processing device 8 detects the position of the finger, the position of the finger in the current virtual space image generated in the image generation unit 83 and already transmitted to the virtual space display device 5 (stored in the storage unit 85). It has been moved) (step S103). When the movement of the finger is not detected, the control unit 8
The image processing apparatus 1 returns to step S101 and repeats the same processing without issuing any instruction.

When the movement of the finger is detected in step S103, the control unit 84 causes the image generation unit 83 to update and generate the virtual space image based on the position data of the finger (step S104), and move the virtual space image. The finger position data and the updated virtual space image are stored in a predetermined storage area of the storage unit 85, and the updated image data is transferred to the virtual space display device (via the signal processing unit 86 and the signal transmitting / receiving unit 87). HMD) 5. The HMD 5 that has received this image data displays the image data on the image display units 51a and 51b according to an instruction from the control unit 54 (step S105).

At the same time, in the information processing apparatus 8, the control unit 84 causes the arithmetic processing unit 81 to display the virtual space in accordance with the finger position data from the position / orientation detection unit 23 of the real glove 20. It is determined whether or not the coordinates of the existing virtual book and the coordinates of the moved finger, for example, the index finger and the thumb are overlapped (step S10).
6). Here, "the coordinates of the finger and the virtual book match" means
A state in which a virtual book is held with multiple fingers including a thumb. If they do not match, the process returns to step S101 again, and the image processing apparatus 1 repeats the same processing. When the coordinates of the both match, the arithmetic processing unit 81 calculates the reaction force from the book to the finger based on the object (here, the book),
The calculation data is transmitted to the real globe 20 via the signal processing unit 86 and the signal transmitting / receiving unit 87 (step S
107). Here, the calculated reaction force data is the control unit 8
4 is stored in a predetermined storage area of the storage unit 85.

In the real glove 20 having received the reaction force calculation data, the control unit 24 drives the tactile sense generating unit 21 and the force sense generating unit 22 to generate the tactile sense and the force sense based on the calculated data ( In step S108), the tactile / force sense generation process ends. FIG. 21A shows a display example of a hand and a book in the virtual space in this state. HM
Images corresponding to this display example are displayed on the image display portions 51a and 51b of D5, and the user sees the image of holding a book through the HMD5 along with the feel of holding a book.

Next, the virtual main alignment processing will be described. This virtual main alignment process is executed in the state where the tactile sensation and the force sensation are generated on the user's finger in step S108 of the tactile sensation / force sensation generation process described above.

As in the case of the above tactile / force sense generation processing,
The movement of the finger in the real glove 20 is detected, the image data in the virtual space is updated and generated in the information processing device 8, and the virtual space image is displayed on the HMD 5 (steps S201 to S205). The updated image data is stored in a predetermined storage area of the storage unit 85, as in the tactile / force sense generation process.

Since the coordinates of the book and the finger have already coincided with each other this time, the control unit 84 of the information processing device 8 determines whether these coordinates have deviated (step S20).
6). If the coordinates remain the same, step S2
Returning to 01, the image processing apparatus 1 repeats similar processing.

If it is determined in step S206 that these coordinates have deviated, the control unit 84 causes the arithmetic processing unit 81 to operate based on the finger position data from the position / orientation detection unit 23 of the real glove 20. , The position of the book after being separated from the finger (hand), the moving speed and the flight direction from the position are calculated (step S207). Specifically, the control unit 8
4 detects the moment when the coordinate of the finger deviates from the coordinate of the book, and causes the arithmetic processing unit 81 to determine the movement distance of the finger of the real glove 20 from that moment to a minute time before and the movement time and the movement direction. Then, the moving speed and the moving direction of the book after being separated from the finger are calculated. The control unit 84 saves the calculated data in a predetermined storage area of the storage unit 85.

Then, the control unit 84, on the basis of the calculation result, generates the continuous images in which the book is flying by the image generation unit 8
3 to generate the image data, store the image data in a predetermined storage area of the storage unit 85, and transmit the data to the HMD 5. In the HMD5 that receives the image data, the control unit 5
4 displays the image data on the image display units 51a and 51b.
Is displayed (step S208). FIG. 21B shows a display example of the hand and the book in the virtual space at the moment when the book is separated from the finger. Images corresponding to this display example are displayed on the image display units 51a and 51b of the HMD 5, and the user feels that the book has been released from the hand and the HMD
Through 5 you will see the image of a book flying.

After that, the information processing device 8 determines whether or not the coordinates of the virtual book and the coordinates of the virtual wall match (step S).
209). That is, the information processing device 8 includes the collision determination unit 8
The wall that is 2 and the book flying in the virtual space is an obstacle in the virtual space (forms a specific plane in the virtual space)
To determine if the vehicle has collided with. Here, the obstacle with which the book collides may be any obstacle that is incorporated in the virtual space image, and for example, considering that the book is thrown, it may be a bookshelf or a simple organizer.

If it is determined that the coordinates of the book and the coordinates of the obstacle do not match, the process returns to step S208, and the flying image of the book is continuously displayed. In this case, the position of the book may be calculated again by returning to step S207, and the image of the book may be displayed based on the calculation result. If it is determined that the coordinates of the book and the wall (obstacle) match, the control unit 8
4 is an image generating unit 83 based on the calculation result of the moving direction of the virtual book by the arithmetic processing unit 81, and an image in which the book is directly faced to the line of sight of the user of the HMD 5 on the virtual wall (that is, , An image with the spine cover facing the user side) is generated, the image data is stored in a predetermined storage area of the storage unit 85, and the data is transmitted to the HMD 5. The HMD 5 that has received this image data displays the video on the image display units 51a and 51b based on the data (step S210). FIG. 22 shows a display example of a hand and a wall (bookshelf) in the virtual space at the moment of this collision.
Images corresponding to this display example are displayed on the image display units 51a and 51b of the HMD 5, and the user sees an image in which the virtual books are aligned facing himself through the HMD 5.

As described above, the image processing apparatus 1 of the present invention
Detects the movement of the finger by the position / orientation detection unit 23, and when the virtual book displayed in the virtual space image and the coordinates of the finger match, the tactile / force sense presentation device 2 generates the tactile / force sense. After that, when the coordinates of the virtual book and the finger are deviated from each other, the arithmetic processing unit 81 calculates and displays the moving speed and the moving direction of the book, and the collision determination unit 82 determines that the virtual wall collides with the virtual wall. , And decided to display it facing the user.

Therefore, according to the image processing apparatus and the image processing method of the present invention, an object such as a virtual book that has been read can be moved by an operation such as throwing in a virtual space image so that it can be easily seen at a predetermined position such as a bookshelf or a wall. Since it can be organized and displayed as described above, the user of the image processing device can easily find an object such as a virtual book visually, and the position of the virtual book is intuitive and memorable. Produce the effect of.

The configuration of the image processing apparatus of the present invention has been described above based on the illustrated embodiment, but the present invention is not limited to this, and the configuration of each unit may be any function having the same function. It can be replaced with that of the configuration.

Further, in the present invention, any two or more configurations (features) of the above embodiments may be combined appropriately.

In the above-described embodiments of the image processing apparatus and the image processing method of the present invention, the case where the user throws a virtual book in the virtual space and aligns it on a wall or the like has been described. It is not limited to this. For example, it can be applied to various virtual objects in the virtual space, such as boxes and personal computers, which are thrown at obstacles such as shelves and walls to be organized.

The operation of the user in the virtual space is
It is not limited to the throwing operation, and for example, when pushing or rolling an object, the present invention can be applied to arrange and display the object.

[0269]

As described above, according to the present invention,
Objects such as books can be moved in a virtual space by throwing them, and can be arranged and displayed in a predetermined position such as a wall for easy viewing. It has the unique effect of being easy to search, intuitive, and easy to remember.

[Brief description of drawings]

FIG. 1 is a conceptual diagram showing a configuration example of a first embodiment of a tactile / force sense presentation device of the present invention and an image processing apparatus having the tactile / force sense presentation device.

FIG. 2 is a block diagram showing a circuit configuration example of the real glove shown in FIG.

FIG. 3 is an external view (partially omitted from illustration) showing an external configuration example of the real glove shown in FIG. 1.

FIG. 4 is a vertical cross-sectional view showing a configuration example of a tactile sense generating unit of the real glove shown in FIG.

5 is an external view showing a configuration example of a force sense generation unit of the real glove shown in FIG.

6 is an external view showing a configuration example of a force sense generation unit of the real glove shown in FIG.

7 is a block diagram showing a configuration example of a position / orientation detection unit of the real glove shown in FIG.

8 is a perspective view showing an example of an external configuration of the virtual space display device shown in FIG.

9 is a block diagram showing a circuit configuration example of the virtual space display device shown in FIG. 1. FIG.

10 is a diagram (block diagram, schematic diagram) showing a configuration example of an unevenness input unit of the virtual space display device shown in FIG. 1. FIG.

11 is a diagram (block diagram, schematic diagram) showing a configuration example of an image display unit of the virtual space display device shown in FIG. 1. FIG.

12 is a diagram (block diagram, schematic diagram) showing a configuration example of a line-of-sight detection unit of the virtual space display device shown in FIG. 1. FIG.

13 is a block diagram showing a circuit configuration example of the information processing apparatus shown in FIG.

FIG. 14 is a vertical cross-sectional view showing a configuration example of a tactile sense generating unit of the tactile sense / force sense presentation device of the second embodiment.

FIG. 15 is an external view (partially omitted from illustration) showing an external configuration example of a real glove of the tactile / force feedback device of the third embodiment.

FIG. 16 is a block diagram showing a configuration example of a position / orientation detection unit of the tactile / force sense presentation device of the fourth embodiment.

FIG. 17 is an external view (partially omitted from illustration) showing a configuration example of a position / orientation detection unit of the tactile / force sense presentation device of the fourth embodiment.

18 is an external view showing a configuration example of a Z-direction detection sensor of the position / orientation detection unit shown in FIG.

FIG. 19 is a flowchart showing tactile / force sense generation processing in the image processing apparatus of the present invention.

FIG. 20 is a flowchart showing virtual main alignment processing in the image processing apparatus of the present invention.

FIG. 21 is an example of a conceptual diagram corresponding to step S208 of virtual main alignment processing in the image processing apparatus of the present invention.

FIG. 22 is an example of a conceptual diagram corresponding to step S210 of virtual main alignment processing in the image processing apparatus of the present invention.

[Explanation of symbols]

1 ... Image processing device 2 ... Tactile / force sense presentation device 2a
…… Gloves 2b …… Frame holder 2c …… Frame cover 2d …… Arm fixing part 20 …… Real gloves
20a ... Cylindrical part 21 ... Tactile generating part 21a ... Opening 21d ... Arm fixing part main body 210 ... Unit tactile generating part 211 ... Power generating part 211a ... Motor 21
2 ... Actuator wire 212a ... Bundle 213
...... Wire holding part 214 …… Tactile presentation pin 214a
...... Contact plate 214b ...... Support portion 215 ...... Tactile presentation pin support portion 216 ...... Tactile presentation portion 216a ...... First plate piece 216b ...... Second plate piece
216c …… Contact part 217 …… Guide part 218 ……
Presser plate 22 ... Force-sense generating section 22d ... Air injecting section 221 ... Frame 222 ... Link motor 23
...... Position / orientation detector 23d ...... Lid 231 ...... X direction detection coil 232 ...... Y direction detection coil 233 ...... Z direction detection coil 24 ...... Control unit
25 ... Signal processing unit 251 ... Amplifying unit 252 ... A
/ D conversion unit 26 …… Signal transmission / reception unit 27 …… Play absorption unit 28 …… Wrist fixing unit 29 …… Twist absorption unit 29
1 ... Band-shaped portion 292 ... Aperture 30 ... Power supply section
5 ... Virtual space display device 51a, 51b ... Image display unit 511 ... Lens 512 ... Lens moving mechanism 5
13: Stepping motor 514: Motor control section 515: Liquid crystal display panel 516: Liquid crystal drive section
517 Backlight 518 Backlight control unit 519 Transistors 52a, 52b Line-of-sight detection unit 521 Half mirror 522 Filter 523 Lens 524
... CCD 525 ... CCD control unit 526 ... CCD
Output amplification section 527 ... LED 528 ... LED control section 529 ... Transistors 53a and 53b ... Convex / convex input section 531 ... Lens 532 ... CCD 533
...... CCD control unit 534 ...... CCD output amplification unit 54
...... Control unit 55 …… Signal transmission / reception unit 56 …… Position and orientation detection unit 57 …… Mounting unit 58 …… Signal processing unit 60 ……
… Casing 61 …… X direction detection sensor 62 …… Y
Direction detection sensor 63 …… Z direction detection sensor 64 ……
Elinver 65 ... Piezoelectric element for vibration 66 ... Piezoelectric element for detection 67, 68 ... Wire 69 ... AC power supply
8 ... Information processing device 81 ... Arithmetic processing unit 82 ... Collision determination unit 83 ... Image generation unit 84 ... Control unit 85
...... Memory unit 86 ...... Signal processing unit 87 ...... Signal transmission / reception unit 88 ...... Magnetic field generation unit 9 ...... Base station

   ─────────────────────────────────────────────────── ─── Continued front page    F-term (reference) 5B050 BA07 BA08 BA09 CA07 EA04                       EA24 FA02 GA02                 5B087 AA09 CC21 CC32 DD12 DE05                 5E501 AA02 AA04 AB03 BA05 CA02                       CC08 CC09 CC14 FA14

Claims (32)

[Claims] 1. An image generation unit that generates a virtual space image, a display unit that displays the virtual space image generated by the image generation unit, and a moving speed and a moving direction of a virtual object in the virtual space image. And a collision determination unit that determines whether or not the moving object collides with a virtual obstacle, and the object has collided with the obstacle by the collision determination unit. If it is determined that the target object is displayed, the object is displayed so as to face the moving direction substantially. 2. The image generation means updates and generates a virtual space image at predetermined time intervals based on the moving speed and the moving direction of the object calculated by the calculating means. Image processing device. 3. The image processing apparatus according to claim 2, wherein the display unit continuously displays virtual space images updated as the virtual object moves. 4. Finger position detecting means for detecting the position of a finger,
The image generation unit further includes a tactile force / force sense presenting unit having a tactile force / force sense generating unit that generates a tactile force and a force sense given to a finger, and the image generating unit is detected by the finger position detecting unit in the virtual space image. The image processing device according to claim 1, wherein a virtual finger image is generated at a position of a finger, and the display unit displays a virtual space image including the virtual finger image. 5. The finger position detecting means is an orthogonal coil type position detecting means having a first coil, a second coil and a third coil whose central axes are substantially orthogonal to each other. The image processing device according to item 1. 6. The image processing apparatus according to claim 4, wherein the finger position detection means includes a gyro sensor that detects angular velocities around X-axis, Y-axis, and Z-axis that are substantially orthogonal to each other. 7. The tactile / force sense generating means so that when the position of the finger detected by the finger position detecting means and the position of the object coincide with each other, a sensation corresponding to the contact is obtained on the finger pad. 7. The image processing apparatus according to claim 4, further comprising control means for controlling the driving of the. 8. The control means, when the position of the finger detected by the finger position detection means and the position of the object are deviated so that the feeling corresponding to the contact is not obtained on the finger pad. The drive of a tactile sense / force sense generating means is controlled.
The image processing device according to item 1. 9. The calculating means, based on a moving distance and a time required therefor in a minute time before the position of the finger detected by the finger position detecting means and the position of the object are displaced, and a moving direction of the object, 5. The moving speed and the moving direction after the object is separated from the finger are calculated.
9. The image processing device according to any one of 8 to 8. 10. The tactile / force sense presenting means is glove-like to be worn from a finger to an arm, receives control data regarding driving from the control means, and detects a finger position detected by the finger position detecting means. The image processing apparatus according to claim 4, further comprising a communication unit that transmits data to the calculation unit. 11. The image processing apparatus according to claim 1, wherein the display means is mounted on a face for use. 12. The image processing apparatus according to claim 1, wherein the display unit includes a communication unit that receives the virtual space image data generated by the image generation unit. 13. The image generation means, the calculation means and the collision determination means, and / or the control means are configured as one unit, and the one unit is
13. The image processing apparatus according to claim 1, further comprising a communication unit that transmits and receives data to and from the display unit and / or the tactile sense / force sense presenting unit. 14. The virtual space image data generated by the image generating means, the calculation result obtained by the calculating means, the collision determination result obtained by the collision determining means, and the finger position detection of the tactile / force sense presenting means. 14. The image processing apparatus according to claim 1, further comprising a storage unit that stores any or all of the finger position data obtained from the unit. 15. The collision determination means determines whether or not an object moving in the virtual space image has collided with a predetermined plane in the virtual space image which is the obstacle. The image processing device according to any one of 1. 16. A step of generating a virtual space image, a step of displaying the generated virtual space image, a step of calculating a moving speed and a moving direction of a virtual object in the virtual space image, and the calculation. Based on the result, updating the virtual space image at a predetermined time interval to display the moving state of the virtual object, and whether the moving object collides with a virtual obstacle. An image comprising: a determining step; and a step of displaying the object so as to substantially face the moving direction when it is determined that the object collides with the obstacle. Processing method. 17. The image processing method according to claim 16, further comprising the step of detecting the position of a finger, wherein the finger is displayed in the generated virtual space image. 18. A step of determining whether or not the detected position of the finger and the position of the target object match, and if the positions match, a touch corresponding to the contact is obtained on the finger pad. The image processing method according to claim 16, further comprising: providing a tactile sensation and a force sensation to the finger pad. 19. When the position of the finger and the position of the object are deviated after the step of providing the tactile sensation and the force sensation, the tactile sensation and the force sensation are applied to the finger pad so that a feeling equivalent to the contact is not obtained. 19. The image processing method according to claim 18, further comprising the step of: 20. In the calculating step, the object is separated from the finger based on a moving distance and a time required for the minute moving time before the position of the finger and the position of the object are displaced from each other and a moving direction of the object. 20. The image processing method according to claim 16, wherein the subsequent moving speed and moving direction are calculated. 21. The image processing method according to claim 16, further comprising a step of transmitting the generated virtual space image data to a virtual space display device. 22. A step of storing the generated virtual space image data and the updated virtual space image data, a step of storing the calculation result obtained in the calculation step, and a step of storing in the collision determination step. 22. The image processing method according to claim 16, further comprising: storing the collision determination result. 23. The image processing method according to claim 17, further comprising a step of storing the position coordinate data of the finger obtained in the finger position detecting step. 24. A computer of an information processing device for generating a virtual space image and displaying the virtual space image so that it can be visually recognized by a user, image generating means for generating the virtual space image, and the virtual space image generated by the image generating means. A display unit for displaying a moving speed and a moving direction of a virtual object in the virtual space image, and a collision for determining whether or not the moving object collides with a virtual obstacle. Determination means, when the collision determination means determines that the object has collided with the obstacle, display means for displaying the object so as to face the movement direction in general. Image processing program of the information processing apparatus of. 25. A computer of the information processing device,
Further comprising a program code for functioning as a tactile sense / force sense presenting means having a finger position detecting means for detecting the position of the finger, and a tactile sense / force sense generating means for generating a tactile sense and a force sense given to the finger, The image generation means generates a virtual finger image at the position of the finger detected by the finger position detection means in the virtual space image, and the display means displays the virtual space image including the virtual finger image. Claim 2 for functioning as
4. An image processing program of the information processing device according to item 4. 26. A computer of the information processing device,
When the position of the finger detected by the finger position detection unit coincides with the position of the object, the drive of the tactile sensation / force sense generation unit is controlled so that a sensation corresponding to the contact is obtained on the finger pad. The image processing program of the information processing apparatus according to claim 25, further comprising a program code for causing it to function as control means. 27. A sensation corresponding to the contact is obtained in the finger pad based on the match / mismatch between the position of the finger detected by the finger position detection unit and the position of the object in the control unit. 27. The image processing program of the information processing apparatus according to claim 26, further comprising a program code for causing a function of controlling driving of the tactile sensation / force sense generating means so as not to be obtained. 28. A computer of the information processing device,
Finger position data detected by the finger position detecting means between the tactile / force sense presenting means and the control means, and control data relating to driving is transmitted / received between the tactile / force sense presenting means and the computing means. 26. The system further comprises a program code for causing it to function as a communication means for transmitting and receiving a message.
28. An image processing program for an information processing device according to any one of 1 to 27. 29. A computer of the information processing device,
29. The information processing apparatus according to claim 24, further comprising a program code for functioning as communication means for transmitting and receiving the generated virtual space image data between the display means and the image generation means. Image processing program. 30. A computer of an information processing device, at least one of the image generation means, the calculation means, the collision determination means and the control means, and the display means and / or
30. The image processing program for an information processing apparatus according to claim 24, further comprising a program code for causing the tactile sense / force sense presenting unit to function as a communication unit that transmits and receives data to and from each other. 31. A computer of an information processing apparatus, the virtual space image data generated by the image generation means, the calculation result obtained by the calculation means, the collision determination result obtained by the collision determination means, the tactile sense / force sense. 25. A program code for causing a storage unit to store any or all of the finger position data obtained from the finger position detecting unit of the presenting unit.
31. An image processing program for an information processing device according to any one of 1 to 30. 32. A computer-readable recording medium recording the image processing program according to claim 24, which is to be executed by a computer.