JP3392080B2 - User interface method, user interface device, space drawing device, workstation device, and program storage medium - Google Patents

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a user interface method and apparatus for arranging objects in a virtual space, and further to a workstation apparatus and the like.

[0002]

2. Description of the Related Art When arranging a virtual object in a virtual space or a mixed reality space (hereinafter, both spaces are collectively referred to as "virtual space" in the present specification), an arrangement destination must be designated.

In a conventional computer system, a pointing device such as a mouse is used to specify a target object and an arrangement destination.

[0004]

However, since a pointing device such as a mouse operates in a two-dimensional space, one target object is designated from a plurality of overlapping objects, The operability when designating the placement destination three-dimensionally was poor.

Various methods and devices for accurately inputting coordinate values in a three-dimensional space have been proposed, however,
Immerse yourself in a virtual space (especially,
In order to use such a device (HMD), the HMD
If you take off and return to the real world again, the operability will be destroyed.

[0006] One of the conventional techniques is a technique of designating an object to be placed in a space for an agent (humanoid appearing in a virtual space) (for example, "Gandalf: An Em
bodied Humanoid Capable of Real-Time Multimodal Di
alogue with People) has been proposed, but how to specify the placement destination is not pointed out.

The object of the present invention is to make ambiguous designations
We propose a user interface method and apparatus that allow objects to be placed in a virtual space.

[0008]

To achieve the above object, a user interface method for arranging an object in a three-dimensional virtual space by a user command for the three-dimensional space drawing system is a user for the three-dimensional space drawing system. A user interface method for arranging an object in a three-dimensional virtual space by a command, which is a command for arranging an object in the virtual space by user input, and target object information designating an object to be arranged, An input step of inputting a command including arrangement direction information indicating an arrangement direction relative to an arbitrary object, and recognition for recognizing a pre-arrangement position of the target object in the virtual space based on the target object information of the input command. Process and user
Detected viewpoint position and line-of-sight direction, detected viewpoint position
And a detection step of obtaining the viewing direction, wherein the obtained <br/> viewpoint position and viewing direction, and recognized the predeployment position, on the basis of said orientation information, the placement destination of the target object And a generation step of generating the arrangement location information.

The placement destination position is determined based on the user's viewpoint position and line-of-sight direction, the object position before placement, and the placement direction. This allows ambiguous placement instructions.

[0010]

The user interface itself may draw the virtual object arrangement. Therefore, according to claim 2 , there is further provided a drawing step of drawing the virtual image of the target object so that the target object is arranged in the virtual space in accordance with the arrangement destination information generated in the generating step. To have.

When immersing in the virtual space through the field of view,
When it is necessary to interface with the real world through the field of view for command input, the immersion must be interrupted. Therefore, according to claim 3 which is a preferable aspect of the present invention, the input step includes a step of inputting a voice representing the command, and a step of recognizing the input voice. Prevent interruptions.

According to claim 4 which is a preferable aspect of the present invention, the input step is performed by using a pointing device (for example, an arrow key of a keyboard).

When the user goes to the virtual space, especially when giving a command to the agent, the user thinks the direction as self-centered. Therefore, according to claim 5 which is a preferred aspect of the present invention, the arbitrary object is a user, and the arrangement direction information included in the command is leftward or rightward in the real space for the user, or It includes information indicating either the front direction or the depth direction. Similarly, according to claim 6 , the arrangement direction information included in the command includes information indicating either an upward direction or a downward direction in the physical space for this user.

The direction may be defined with the virtual space as the center. Therefore, according to claim 7 , the arbitrary object is an arbitrary object in a virtual space, and the arrangement direction information included in the command is any one of a left direction, a right direction, a front direction, and a depth direction with respect to the arbitrary object. Including information indicating Similarly, according to claim 8 , the arbitrary object is an arbitrary object in a virtual space, and the arrangement direction information included in the command includes information indicating either an upward direction or a downward direction with respect to the arbitrary object. It is characterized by

When the system gives the movement amount,
There may be a contradiction with the property of the virtual space (gravity relation or interference relation). Therefore, according to claim 9 which is a preferable aspect of the present invention, the generation step checks the validity of the placement destination position information according to the spatial rule of the virtual image, and if it is not valid, the placement is performed. The placement destination information represented by the destination position information is corrected.

The interactive placement instruction is preferably given to the agent. Therefore, claim 10
According to the method, the step of detecting that the command is sent to the agent existing in the virtual space, and the agent placing the target object in the placement destination position,
Draw a virtual image of the agent.

Furthermore, the above object can be achieved by a user interface device or a space drawing device.

[0019]

BEST MODE FOR CARRYING OUT THE INVENTION An embodiment in which the present invention is applied to a virtual space experience system using an agent will be described below.

FIG. 1 shows a block diagram of the virtual space experience system. In the figure, reference numeral 101 denotes an image input unit for capturing a real image group or inputting a CG image that has already been created from the outside. Instead of the image input unit 101, a database that stores a large number of pre-captured images may be used. A CPU 102 performs processing according to the processing procedure stored in the storage unit 103. A storage unit 103 stores image data of the real image group, ray space data generated from multi-viewpoint images, a program showing a processing procedure of the CPU, and the like, and uses a part of the data as a work memory. . Reference numeral 104 denotes a viewpoint / line-of-sight detection unit for detecting the viewpoint position / line-of-sight direction of the observer. An input device such as a keyboard or mouse is used for the viewpoint / gaze detection unit, or HMD (head mounted display)
A device with a sensor or the like can be used. Reference numeral 105 denotes an image output unit for displaying an image generated according to the viewpoint position / line-of-sight direction of the observer. As the image output unit, a general two-dimensional display such as a CRT or a liquid crystal may be used, or a three-dimensional display such as a lenticular or HMD may be used. The program is recorded in a storage medium such as an FD (floppy disk), CD-ROM, magnetic tape, etc., and the storage medium reading unit 106
Alternatively, it may be read from the storage unit 103 and stored in the storage unit 103.

Reference numeral 107 denotes a command input section, which uses a microphone in this embodiment. The voice of the command acquired by the microphone is converted into a digital format and sent to the voice recognition unit. The voice recognition unit performs well-known voice recognition processing on voice data of a command. As will be described later, in this embodiment, since the command for spatial arrangement has a predetermined format, natural language recognition is not required.

FIG. 2 is a block diagram showing software functions in the embodiment system shown in FIG.
The command conversion unit 300 converts the voice signal picked up by the microphone into
It is converted into a command and sent to the agent process 200. The agent process 200 removes the ambiguity existing in the received command in consideration of a predetermined rule provided for this virtual space, and passes it to the virtual space / mixed reality space system 400. The input command is not limited to voice input, and may be input from a keyboard as shown in FIG.

FIG. 3 shows an example of the virtual space of this embodiment. In this embodiment, it is assumed that the virtual space is one room, in which a desk 202 and a chair 201 are placed inside, and a rectangular box 203 to be placed is placed on the desk. . The agent is supposed to move this box 203 into the virtual space.

FIG. 4 shows the form of commands prepared in this embodiment. The virtual space command is divided into four parts, and the first operand is the "instruction" operand, which indicates the operation for placement. In this example,

[Move, Shift, Swap, Rotate

The command is provided. The second operand specifies the object A to be placed. In this embodiment, the name of an object in the virtual space, for example, “chair”, “tsukue”, “box” is inserted as the object A.
The third and fourth operands indicate the position of the movement destination by these two. For example, MOVE AB right (left, front, back, up, down) puts object A to the right of object B (or left, front, back, top,
Command to move to the bottom). SHIFT A (A) right (left, front, back, up, down) means that the object A is to the right of the object A (or left, front, back, top,
It is a command to move (that is, shift) downward.

SWAP, A, B, exchange the positions of object A and object B with each other. ROTATE, A, (A), and right (left, front, back) are for rotating the object A around the object A to the right (to the left, to the front, to the back).

MOVE, A and away move the object A to a specific position. The destination is a fixed position, so there is no need to specify it in the command. All the above commands are classified as type I. Type I
That is, the criterion for moving the object of operand 2 is the position in virtual space (specified by operand 3). In addition, the above-mentioned command means that the reference of the arrangement direction is the user. That is, to the right of object A (or
Left, front, rear, up, down) refers to the direction viewed from the user side. Therefore, the right side (or left, front, rear, top, bottom) of the object A as seen from the user side is the left side (or right, rear, front, top, bottom) of the object A toward the user in the system. Is.

By using the format of the command as shown in FIG. 4, although the command is not entered in the natural language format, it becomes easy to recognize the command from the input voice data.

FIG. 5 shows a definition table for each virtual object in the virtual space of FIG. This table is used when searching which object in the virtual space the object in operand 1 and operand 2 in the voice input command corresponds to. That is, if the object name in the operand recognized by voice recognition is, for example, “Tsukue”, the object is identified as the object 1 in the virtual space.

The feature of this embodiment is that even if the voice input command is in an ambiguous form as shown in FIG. 4, the agent process 200 removes the ambiguity and passes it to the spatial process section 400. is there. Agent process 2
00 and the space system unit 400 are separated so that the space system unit 400 can use existing application programs as they are. That is, if the command is simplified, a process for removing the ambiguity is essential, but by making the process independent of the spatial processing application program, the usability of the existing application program, that is, the compatibility is improved. This is because the advantage of maintaining is born.

In this embodiment, how the ambiguity is removed will be described with reference to FIG.

FIG. 6 shows the Tsukue 202 placed in the virtual space of FIG. The width of the Tsukue in the left-right direction is W. It is assumed that a command to move the object 203 placed on this is moved to the right, for example. In the present embodiment, the command allows ambiguity about the movement amount and the movement direction. For this reason, it has a default value (default algorithm) for the movement amount. That is, when the object A (box 203) placed on a certain object B (tsukue 202) is moved, the amount of movement is set to 1/2 of the width W of the object B. Of course, the value of 1/2 may be any 1 / k.

The feature of this embodiment is that the above-mentioned movement amount is corrected according to the characteristics of the virtual space.

Now, as shown in FIG. 6, it is assumed that the box 203 is located at a distance x (x <W / 2) from the edge of the Tsukue 202. If the box 203 is moved by W / 2 in this state, the box 203 will be displaced from the edge of the Tsukue 202. In other words, the moved box 203 appears to be floating in the air. Whether or not it is a problem for an object to float in the air depends on the nature of this virtual space defined by the space system 400. If this virtual space prohibits an object from floating in the air, the movement amount is changed (corrected) from W / 2 to x.

As a modified example, if the agent process 200 is informed of data relating to the property of the virtual space defined by the system 400, then according to the rule concerning the property, for example, as shown in FIG. And reconfigure the command for that trajectory,
It may be modified to pass to the spatial system 400.

As a property of the space, if the virtual object does not allow the virtual object to sink into the virtual wall, the moving distance is changed to the distance y to the wall, for example, as shown in FIG.

The control procedure of FIG. 9 shows the control procedure of the agent process section.

First, a voice signal is input in step S102, and voice recognition is performed in step S104. Step S1
At 06, the recognized voice is coded and decomposed into operands. In step S108, the target object operand (operands 2 and 3) is specified in the command. In step S110, the identifier of the specified target object is sent to the system 400, and its spatial data (positional information) is received. In step S112, the line-of-sight direction of the user is acquired. In step S114, the visual line direction of the user and the spatial direction are compared. This is because if the user faces the front of the virtual space, the left-right direction and the previous direction in the virtual space are opposite to those of the user. In step S116, the movement amount is added to the coordinate value of the object to be moved. In step S118, the property data of the virtual space is read. This property data has been established in advance, made into a rule, and stored in the storage unit 103. These properties are properties related to the gravity relationship and the interference relationship. The gravity relationship is whether or not an object is allowed to float in the air. The interference relation is whether to allow an object to enter another object. If there is a contradiction, it is corrected in step S122. In step S124, the target object is moved.

In the above embodiment, the front-rear relationship and the left-right system are determined based on the user, but may be determined based on the virtual space. In this case, when the user faces the front of the virtual space and moves the object A to the right, the object A is moved to the left when viewed from the user.

Further, the present invention may be applied to a system composed of a plurality of devices or an apparatus composed of a single device. Further, it goes without saying that the present invention is also applied to the case of being implemented by supplying a program to a system or an apparatus. In this case, the storage medium storing the program according to the present invention constitutes the present invention. Then, by reading the program from the storage medium to the system or device, the system or device operates in a predetermined method.

[0042]

As described in detail above, according to the present invention,
A user interface method and apparatus capable of arranging an object in a virtual space even with ambiguous designation could be provided.

[Brief description of drawings]

FIG. 1 is a block diagram illustrating a hardware configuration of a virtual space system according to an embodiment.

FIG. 2 is a block diagram illustrating a software configuration of the virtual space system according to the embodiment.

FIG. 3 is a diagram illustrating a configuration of a target virtual space.

FIG. 4 is a diagram illustrating formats of various commands set in the user interface device according to the embodiment.

FIG. 5 is a diagram illustrating a method of linking a command and an object in a virtual space in the embodiment.

FIG. 6 is a diagram illustrating a state in which a contradiction occurs due to gravity in an example of a virtual space.

FIG. 7 is a diagram illustrating avoidance when a contradiction occurs due to gravity.

FIG. 8 is a diagram illustrating a situation in which a contradiction occurs due to an interference relationship.

FIG. 9 is a flowchart showing a processing procedure of the embodiment.

Continuation of the front page (56) Reference JP-A-7-85303 (JP, A) JP-A-8-101758 (JP, A) JP-A-9-97355 (JP, A) JP-A-7-271546 (JP , A) JP-A-11-18025 (JP, A) JP-A-11-195131 (JP, A) Kenji Mochizuki, 1 person, "Access to 3D image by natural language", IEICE technical research Report PRU92-35 to 44 Pattern recognition and understanding, The Institute of Electronics, Information and Communication Engineers, October 15, 1992, Volume 92, No. 255, p. 69-75 (58) Fields investigated (Int.Cl. ⁷ , DB name) G06T 17/40 G06F 3/00 G06F 3/033 CSDB (Japan Patent Office)

Claims (23)

(57) [Claims] 1. A user interface method for arranging an object in a three-dimensional virtual space by a user command to a three-dimensional space drawing system, which is a command for arranging an object in the virtual space by user input, An input step of inputting a command including target object information designating an object to be arranged and arrangement direction information indicating a relative arrangement direction with respect to an arbitrary object, and the target object based on the target object information of the input command The recognition step of recognizing the pre-placement position in the virtual space and the user's viewpoint position and line-of-sight direction are detected
A detection step of acquiring a viewpoint position and a line-of-sight direction, an arrangement indicating an arrangement destination of the target object based on the acquired viewpoint position and the line-of-sight direction, the recognized pre-arrangement position, and the arrangement direction information. And a generating step of generating the previous position information. 2. The method further comprises a drawing step of drawing a virtual image of the target object so that the target object is arranged in the virtual space according to the arrangement destination information generated in the generating step. The user interface method according to claim 1 , wherein the method is a user interface method. Wherein said input step includes a user interface method according to claim 1 or 2, characterized the steps of: inputting a voice representing the command, that includes the step of recognizing the input speech. Wherein said input step includes a user interface method according to claim 1 or 2, characterized in that input using the pointing device. 5. The arbitrary object is a user, and the arrangement direction information included in the command includes information indicating one of a left direction, a right direction, a front direction, and a depth direction in the physical space for the user. The interface method according to any one of claims 1 to 4 , characterized in that: 6. The arbitrary object is a user, and the arrangement direction information included in the command includes information indicating either an upward direction or a downward direction in the physical space for the user. 5. The user interface method according to any one of Items 1 to 4 . 7. The arbitrary object is an arbitrary object in a virtual space, and the arrangement direction information included in the command indicates any one of left direction, right direction, front direction, and depth direction with respect to the arbitrary object. the user interface method according to any one of claims 1 to 6, characterized in that it comprises a. 8. The arbitrary object is an arbitrary object in a virtual space, and the arrangement direction information included in the command includes information indicating either an upward direction or a downward direction with respect to the arbitrary object. The user interface method according to any one of claims 1 to 6 . 9. The generating step checks the validity of the placement destination position information according to the spatial rule of the virtual image, and corrects the placement destination information represented by the placement destination position information if it is not valid. The user interface method according to any one of claims 1 to 8 , which is characterized. 10. A step of detecting that the command is sent to an agent existing in a virtual space, and a virtual image of the agent is drawn so that the agent arranges the target object at the arrangement position. the user interface method according to any one of claims 1 to 9, characterized by including the step. 11. A user interface device for arranging an object in a three-dimensional virtual space by a user command for a three-dimensional space drawing system, which is a command for arranging an object in the virtual space by user input, Input means for inputting a command including target object information designating an object to be arranged and arrangement direction information indicating an arrangement direction relative to an arbitrary object, and the target object based on the target object information of the input command A recognition unit that recognizes a pre-arrangement position in the virtual space, a position of the target object based on the user's viewpoint position and line-of-sight direction, the recognized pre-arrangement position, and the arrangement direction information. a generating means for generating a placement location information indicating detects the viewpoint position and the sight line direction of the user, detection of
Detection for passing the generated viewpoint position and line-of-sight direction to the generation means
User interface apparatus characterized by and means. 12. The image forming apparatus further comprises drawing means for drawing a virtual image of the target object so that the target object is arranged in the virtual space according to the arrangement destination information generated by the generating means. The user interface device according to claim 11 , characterized in that 13. The user interface device according to claim 11 , wherein the input unit includes a unit for inputting a voice representing the command and a unit for recognizing the input voice. 14. The input means, to claim 11, characterized in that inputting using the pointing device
13. The user interface device according to any one of 13 . 15. The arbitrary object is a user, and the arrangement direction information included in the command includes information indicating one of a left direction, a right direction, a front direction, and a depth direction in the physical space for the user. The user interface device according to any one of claims 11 to 14 , characterized in that: 16. The arbitrary object is a user, and the arrangement direction information included in the command includes information indicating either an upward direction or a downward direction in the physical space for the user. Item 15. The user interface device according to any one of items 11 to 14 . 17. The arbitrary object is an arbitrary object in a virtual space, and the arrangement direction information included in the command is
The user interface device according to any one of claims 11 to 16 , further comprising information indicating one of a left direction, a right direction, a front direction, and a depth direction with respect to the arbitrary object. 18. The arbitrary object is an arbitrary object in a virtual space, and the arrangement direction information included in the command is
17. The user interface device according to claim 11 , wherein the user interface device includes information indicating either an upward direction or a downward direction with respect to the arbitrary object. 19. The generating means checks the validity of the placement destination position information according to the space rule of the virtual image, and corrects the placement destination information represented by the placement destination position information if it is not valid. The user interface device according to any one of claims 11 to 18 , characterized by the above-mentioned. 20. A means for detecting that the command is sent to an agent existing in a virtual space, and a virtual image of the agent is drawn so that the agent arranges the target object at the arrangement position. 20. The user interface device according to claim 11 , further comprising means. 21. A program storage medium storing a computer program for realizing the user interface method according to any one of claims 1 to 10. 22. A space drawing device incorporating the user interface device according to claim 11 . 23. A workstation device incorporating the user interface device according to claim 11 .