JPH08101758A - Data operation device for virtual space - Google Patents

Abstract

PURPOSE: To provide the data operation device for virtual space which operates objects in a virtual space more naturally and more easily. CONSTITUTION: An operation means 150 whose position and direction can be operated, a detection means 140 which detects the position and the direction of the operation means 150, a storage means 100 where data of components constituting the virtual space is stored, a display part 160 on which the virtual space is displayed based on stored data, a correspondence part 110 which makes the operation means 150 and one component of the virtual space displayed on the display device 160 correspond to each other and switches this correspondence to another component of the virtual space, and a control part 120 which controls the relative position and direction of this corresponding component of the virtual space based on the detection result of the detection means 140 are provided.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a data manipulating device in a virtual space, and more particularly to a device for manipulating data in a remote place or a device for simulating its operation,
The present invention relates to various systems using virtual reality such as a communication system in which multiple members participate in a virtual space.

[0002]

2. Description of the Related Art A technique called "virtual reality" or "artificial reality" has attracted attention from various aspects. In general, this technology uses new input / output devices and the latest signal processing technology such as computer graphics and sound field processing to work on the five senses of human sight, hearing, force, etc. It aims to realize the feeling of being in a space. Using this technology, applications in various fields such as remote control of robots, games that pursue reality, and games that enhance the reality of video conferencing called tele-existence are being studied (for example, Katsura Hattori, “ World of Artificial Reality ”, Industrial Research Committee).

Conventionally known virtual reality techniques often use an input / output device such as a head mounted display or a data glove to be worn on the body in order to cover up the reality with virtual reality. The data glove is a device for attaching an optical fiber for measuring the bending angle of the joint of the glove and a magnetic sensor for detecting the position of the hand to the glove and transmitting the movement and position of the hand to the computer. There have been three-dimensional pointing devices as input devices for three-dimensional CAD, etc.
It is epoch-making in that it allows the user to input the orientation and the degree of grip, and that the operation corresponds directly to the operation by hand. The head mounted display is a color liquid crystal display that displays images on the left and right eyes and has a wide viewing angle so that it can be worn on the head.Normally, the view of the real space is blocked, and the left and right sides are separated. Since it is possible to provide the image of, the three-dimensional display can be performed.
These are the current status of typical devices for virtual reality.

On the other hand, computers are becoming more and more popular due to the decrease in the price of personal computers, and it is expected that they will become more and more familiar in the future due to the widespread use of broadband networks such as optical fibers. It will be much easier to understand. For example, as an example of future realization, a realization form of using a personal computer to realize tele-shopping using the CATV network can be considered. In this case, it is possible to arrange a menu as a simple and easy-to-understand user interface, but another method is to incorporate a moving image into the user interface. That is, for example, in the case of tele-shopping, an image of a shopping center in a town is provided, and it is possible to shop by searching for an image, but the shopping center in this case is a virtual one. Space. However, unlike the above-mentioned actual example performed by the virtual reality, the realization means is CATV, so the display means is a normal TV monitor, and if the input means is possible, it is lightweight and inexpensive like a remote controller. Is desirable. On the contrary, cost performance is required rather than unnecessarily high reality.

[0005]

The prior art will be considered in consideration of the use of a popular virtual space represented by the above-mentioned teleshopping. As described above, the typical input means in the conventional virtual reality is the data globe. Usually, a combination of this and a head-mounted display realizes operation in a virtual space. For example, when grasping an object in a virtual space with a hand, the positional relationship between the virtual object and the hand is virtual and cannot be perceived, so as an example, an image of a virtual hand is displayed together in the virtual space. That is done.

In another example, a state in which the object can be grasped by simulating a virtual three-dimensional position and the position of the hand in the real space is simulated by combining with a stereoscopic image using glasses having a liquid crystal shutter. However, even in this case, since the virtual object does not actually exist, it is not possible to obtain the feeling of holding it by hand, and it is difficult to confirm whether or not the virtual object is grabbed. It is possible to confirm whether or not you have grasped it for the first time by moving it, so there is a possibility that the operation will be incorrect. In order to improve this, there is an example of using a force feedback device for reproducing a reaction force when grasped by hand, but there is a problem that the device becomes complicated.

The data manipulating device in the virtual space of the present invention has been made in view of such a problem, and its object is to associate an object in the virtual space with an operating means in the real space. In addition to operating the operation means in the real space as if it were an object in the virtual space, and by arbitrarily switching the above correspondence, the object in the virtual space can be operated more naturally and easily. It is to provide a data operation device in a virtual space.

[0008]

In order to achieve the above-mentioned object, a data operating device in a virtual space of the present invention detects an operating means capable of operating a position and an orientation and a position and an orientation of this operating means. Detecting means, storage means for storing the data of the constituent elements that make up the virtual space, a display device for displaying the virtual space based on the data stored in the storage means, and the operating means. In accordance with an operation, the operation means is associated with one component of the virtual space displayed on the display device, and this association is switched to another component of the virtual space. It is provided with associating means and control means for controlling the relative positions and orientations of the associated components of the virtual space based on the detection result of the detecting means.

[0009]

That is, the data operating device in the virtual space of the present invention detects the position and orientation of the operating means by the detecting means, and accumulates the data of the constituent elements forming the virtual space by the accumulating means, and accumulates the data. The virtual space is displayed on the display device based on the generated data. Next, according to the operation of the operating means, the operating means and one component of the virtual space displayed on the display device are associated with each other, and this association is performed on the virtual space. Switch to other components of. Then, the relative position and orientation of the components of the virtual space associated with the above,
The control is performed based on the detection result of the detection means.

[0010]

Embodiments First, the outline of embodiments of the present invention will be described. One difference between the operating means in the present embodiment and the data glove used in the conventional technique is that the data glove virtually realizes a hand, whereas the operating means in the present embodiment is a virtual object. It is a point that virtually realizes. That is, since the operation means actually acts as an object in the virtual space to operate the object in the virtual space, the operation target actually exists and there is a reaction from the operation target. By doing this, you will be able to perform more natural operations. Of course, since the reaction in this case comes from the operating means itself, it is not necessary to prepare a large device to obtain it. Further, since the operating means is present in front of the user, it is not necessary to display the positional relationship between the object and the hand in the virtual space by displaying the image of the hand in the virtual space, which is also natural. .

Further, even when an object in the virtual space is thrown, rolled, or otherwise moved away from the hand, natural handling is possible because the operation means is present in the real space. In addition, since the operation of the operation means is exactly the same as the operation when the object in the virtual space is in the real space, the operation can be very simple and comfortable.

That is, in FIG. 1, the user is
While observing the image of the three-dimensional space as the virtual space displayed by 160, the operation means 150 is moved and the desired constituent element is designated from the image. Each of the constituent elements in the video has 3D data 101 consisting of parameters representing the shape and pattern accumulated in the accumulating section 100. By combining and combining 3D data for each constituent element,
An image of the dimensional space can be obtained. The associating unit 110 associates the constituent elements in the three-dimensional space designated by the user by operating the operating means 150 with the operating means 150 itself. As a result, the user can handle the operation means 150 and the designated component as one. 3 of the specified components
The dimension data is the control unit as the corresponding component data 102.
The other component data 103 is sent to the synthesis unit 16
Sent to 0. In the control unit 120, the specified component
Make sure that the relative position and orientation in the image in the three-dimensional space match the relative position and orientation of the user's operation means 150 itself.
Based on the detection result 104 obtained by the operating means 150, the corresponding component data 102 is processed.

More specifically, when the operating means 150 is moved and rotated, its coordinates and orientation are obtained as a detection result, and the designated component is moved accordingly in the image in the three-dimensional space. The component data 102 is processed so as to rotate. The processed data 105 is input to the combining unit 130, combined with the other component element data 103, output as combined data 106, and provided to the user as a three-dimensional space image on the display unit 160.

Examples of the present invention will be described below. FIG.
FIG. 3 is a block diagram showing the configuration of an embodiment according to the present invention. In the figure, the user operates the operation means 150 while watching the image 108 in the three-dimensional space displayed on the display unit 160, and acts on the virtual space data shown in the image. In the figure, the operating principle by which the position and direction are detected by the operating means 150 and the detecting means 140 will be explained in an embodiment to be described later, so that the explanation is omitted here, but the operating means
It is assumed that, for example, a button for instruction is attached to 150 and the signal thereof is input to the associating unit 110.

Further, the data 104 sent from the detecting means 140 to the control unit 120 is a parameter necessary for defining the position and the rotation angle corresponding to the three-dimensional space, for example, 3
It is the position coordinates of three points in the dimensional space or the position coordinates of one point and the rotation angle around three axes passing through the points. The three-dimensional data storage unit 100 stores three-dimensional data including parameters representing shapes, patterns, etc. corresponding to the constituent elements in the video.
101 have been accumulated. Specifically, the shape parameters are functions that form a curved surface and the coordinates of each patch vertex that forms a wireframe model that approximates the curved surface with a set of triangular patches, and the pattern parameters are texture mapping parameters. Pattern data, a function for generating the pattern, color or brightness data, and the like.
The three-dimensional virtual space image is created by performing position calculation, deformation, movement calculation, and brightness calculation on the model of each element, and then combining them.

In the virtual space, the relative position between the user and the entire virtual space can be changed by the button operation of the user. This corresponds to the user's sense of movement in the virtual space. In addition to the movement, the user can operate the cursor (having the position and direction as an attribute) on the corresponding screen by moving the operation means. The cursor also occupies the coordinate position in the virtual space, and the detection result data 104 from the detecting means 140 is stored in the associating unit 110.
The positional relationship between the cursor and the constituent elements in the virtual space is always determined. Here, when the control information 109 is input by a button operation from the operation means 150 while the cursor is in contact with a certain component, the associating unit
110 interprets that the user has designated the component, and thereafter operates to simulate the movement of the operating means by the movement of the designated component.

That is, first, 3 of the designated components
The dimensional data is sent to the control unit 120 as the corresponding component data 102, and the other component data 103 are combined.
Sent to 130. In the control unit 120, the relative position and orientation in the image of the three-dimensional space of the designated component is adjusted by applying an appropriate reduction ratio so that the user's operating unit 150
The image data 105 of the corresponding component is created by using the detection result 104 obtained by the operation unit 150 in the above-described calculation at the time of image formation so as to match the relative position and orientation of itself. In the synthesizing unit 130, first, an image corresponding to another component is generated from the other component data 103. This calculation only creates a still image, and does not require a large amount of calculation. Next, the image data 105 of the designated component is combined with the other component data 103 and output as composite image data 106.
This image is provided to the user on the display unit 160 as the image 108 in the three-dimensional space.

The designation of the constituent elements is canceled in this mode by inputting a control signal from the operating means 150 to the associating section 110 by a button operation. In the normal movement mode, the data of all the constituent elements are stored as the data 102 in the control unit.
It is sent to 120, where the deformation calculation associated with the movement described above is performed, and the image 105 is created. In this case, this image passes through the combining unit 130 and is output to the display unit 160.

Next, an example in which the present invention is applied to teleshopping using a virtual space will be described as a specific use example. FIG. 2 shows an image of a specific usage environment corresponding to FIG. In the figure, 1001 serves both as an operating unit and a detecting unit, and also serves as a remote controller for the TV. 1
002 is a TV monitor, which operates as a display device. Reference numeral 1003 denotes a CATV adapter (actually a computer) which executes creation and management of three-dimensional data, control, association with a designated area, and the like. The adapter includes a memory device in which original data for creating three-dimensional data is stored. Part of the data or part of the information for controlling the data may come from the CATV center through a line. The remote controller 1001 and the adapter 1003 exchange necessary data such as detection result parameters by infrared rays, and transfer the data to the TV monitor 1002 as necessary.

Next, a specific operation example will be described. What is assumed here is the image that shopping can be performed by searching in a virtual shopping center as described above. The operations required for this are as follows. (1) Move (2) Search for an object (or place) (3) Point to an object (4) Look at an object = Grab + Move + Release (5) Call a clerk How these actions are realized Here is an example.
FIG. 3 is a state transition diagram after the start. After starting, first select one of the two modes. One is an access mode for specifying a desired store by gradually limiting it to a narrow range from a sketch of the center. Another one
One is a shopping mode in which you can move from the entrance of the center to find what you want to buy. In the access mode, the floor plan is hierarchically structured, for example, center = floor = store = sales floor, and while moving the cursor and specifying the location with the switch, the sequence proceeds (or skips halfway) to the lower layers, and finally You can reach the desired sales floor (Fig. 4). By performing a search using a floor plan, there is an effect that the time required for the search can be significantly shortened when the purchase is clear.

Next, the operation in the shopping mode will be described with reference to FIG. In shopping mode, first move in the virtual space to find the item you want to buy. The movement is performed, for example, by operating the “accelerator” button and the “brake” button on the remote controller. The direction of travel is controlled by the direction of the remote control. On the screen, a cursor indicating the direction is displayed so as to be multiplexed with the data in the virtual space (a). When you find a product, put it in a state where the cursor and the product are in contact (the virtual space is three-dimensional, so you can distinguish between the state where the cursor is in contact with the product in the virtual space and the state where it is not). (B). In order to make it easy to see whether or not the cursor is in contact with the product, it is advisable to make some changes such as the brightness of the cursor appear on the screen when touching.

When the "grab" button is pressed in this state, the product can be moved. The feature of this embodiment is that in this mode, the movement of the product is exactly the same as the movement of the remote controller by the user (c). That is, by moving the remote control as if it were the product, the same movement as the product is reproduced on the screen. This allows the user to see the product from any angle of 360 degrees and make a decision. When the "release" button is pressed from this state, the moving mode is released, and when the contact state is released, the original movement mode is restored. Here, since the button operation is unique to each mode and does not duplicate in functionality, the number of buttons can be suppressed to the minimum necessary number by sharing each mode. The transition between the access mode and the shopping mode can be performed from any hierarchy at any timing.

From the above, among the necessary operations described above (1)
To (4) are supported. With respect to the correspondence with FIG. 1, the part that calculates and determines the contact between the cursor and the product and specifies and releases the product by button operation corresponds to the association unit 110 in FIG. 1.

Next, the function (5) will be described.
As the function of (5), for example, a store clerk is arranged in a virtual space, and access to the data of the store clerk has a meaning different from that of the product. This is shown in FIG. In this example, when the cursor is in contact with the store clerk (a), a mode for requesting a product description is entered. When the item is brought into contact with again in this mode and the (b) button is pressed, the video screen for explaining the item is switched (c), and when it is finished, the original moving mode is automatically restored. If you press another button in the product description mode to automatically call a TV phone and connect to the actual product shipping center (d), you can get more detailed product description and advice as needed. Can be beneficial. Such a thing is possible if an upstream line for a TV phone is set for the CATV line.

FIG. 7 is a block diagram of an apparatus for enabling this operation. In FIG. 7, the operation of the block corresponding to the embodiment of FIG. 1 is the same as that of FIG. However, the associating unit 1110 specially handles only the operation related to the data of the clerk among the three-dimensional data. The video phone 115 shown in FIG.
When the flow corresponding to the automatic call of 0 is passed, that is, when the control signal by the button operation comes after the contact between the clerk and the cursor is detected, the associating unit 1110 sends the control signal to the switching circuit 1140 and the videophone 1150. To send. The switching circuit 1140 has a normal display unit 1130 and a control unit 11.
Although the image (and sound) of the virtual space created in 20 is output, the image and sound 1107 from the TV phone 1150 are switched by the control signal. The videophone 1150 automatically calls a predetermined number by a control signal. When the videophone 1150 call is completed, the switching circuit 11 is activated by the control signal 1108 which notifies the end of the call.
40 is again switched to the image (and audio) side of the virtual space created by the control unit 1120.

Next, as a concrete example of the operating means capable of operating the three-dimensional position and orientation and the detecting means for detecting the position and orientation of the operating means, an embodiment utilizing ultrasonic waves is described. Two are shown.

First, a first structural example using ultrasonic waves will be described. The operating means shown in FIG. 8 includes an ultrasonic pulse generator 2001 and ultrasonic output devices 2002 and 200.
It consists of 3. The ultrasonic pulse generator 2001 is
The pulse signals of different frequencies f and f '(f <f') are output intermittently at regular intervals T. The ultrasonic wave output device 2002 outputs a pulse signal of frequency f as an ultrasonic wave. Similarly, the ultrasonic wave output device 2003 outputs a pulse signal of frequency f ′ as an ultrasonic wave. Next, the detecting means shown in FIG. 9 is the ultrasonic detecting device 2004, 2005, 20.
06, low-pass filter (LPF) 2007/2009
・ 2011, High-pass filter (HPF) 2008 ・ 2
010/2012, a time detection device 2013, and a position calculation device 2014. Ultrasonic detector 2004
2005 and 2006 convert ultrasonic waves into electric signals.
The low-pass filters 2007, 2009, and 2011 are low-pass filters with a cutoff frequency fc (f <fc <f '), and extract only the signal of frequency f from the input signal.
The high-pass filters 2008, 2010, 2012 are high-pass filters with a cutoff frequency fc, and extract only the signal of frequency f'from the input signal. The time detection device 2013 uses the ultrasonic detection device 2004 for frequency f.
And the time t1 when the ultrasonic pulse of f'is detected and
t1 'and the frequencies f and f'at the ultrasonic detector 2005
The time t2 and t2 'at which the ultrasonic pulse is detected and the times t3 and t3' at which the ultrasonic detecting device 2006 detects the ultrasonic pulse of the frequencies f and f'are detected. The position calculation device 2014 is a position P (x, y, z) in the three-dimensional space where the ultrasonic pulses of frequencies f and f'are output.
And compute P '(x', y ', z').

Here, as shown in FIG. 8, the ultrasonic output devices 2002 and 2003 are attached to both ends of the operating means, and the distance between them is l. In addition, the ultrasonic detection devices 2004 and 20 shown in FIG.
The coordinates in the three-dimensional space of 05 ・ 2006 are A1 (x
1, y1, z1), A2 (x2, y2, z2), A3 (x3, y3, z3), which are not on a straight line. At this time, the coordinates P (x, in the three-dimensional space of the ultrasonic output devices 2002 and 2003, P (x,
The following relationship is established between y, z), P '(x', y ', z') and the time t when the ultrasonic pulse is output from the ultrasonic output devices 2002 and 2003.

[0029]

[Equation 1] Where v is the ultrasonic velocity. Position calculation device 201
By solving these equations as simultaneous equations in 4, the coordinates of the upper and lower ends of the operating means in the three-dimensional space can be obtained. If the point P (x, y, z), P '(x', y ', z') is the solution of this equation, the point A1 (x1, y1, z1), A
2 (x2, y2, z2), A3 (x3, y3, z3) points P to the plane
The points symmetric to (x, y, z) and P '(x', y ', z') are also clearly solutions of this equation. However, if the ultrasonic detecting devices 2004, 2005, 2006 are arranged so as to contact the ground as shown in FIG. 10, the point A1 (x1, y
The plane defined by (1,0), A2 (x2, y2,0), A3 (x3, y3,0) is equal to the ground, and the points P (x, y, z), P '(x', y ', The range in which z ') exists is z> 0, z'> 0, so the solution is uniquely determined. The temporal resolution of the operating means and the detecting means described above is determined by the interval between the pulse signals output from the ultrasonic pulse generator 2001.
Determined by T The pulse at time t and the pulse at time t + T must be distinguished in order to detect the position and orientation correctly. To do so, T should satisfy the following equation.

[0030]

[Equation 2] For example, if the distance between the ultrasonic detecting devices 2004, 2005 and 2006 is set to 3 m or less, the left side of the above formula is at most 0.01 seconds and T is about 0.03 seconds. Therefore, in this case, the position and orientation can be detected about 30 times per second. In addition, it is possible that an error may occur between the actual position and the detected position due to fluctuations in the sound velocity v due to changes in temperature. However, even in that case, there is no problem in practical use because the correspondence between the actual movement (movement and rotation) of the operating means and the detected movement of the operating means is maintained correctly.

As described above, since the three-dimensional positions of the upper and lower ends of the operating means can be detected, the correspondence between the upper and lower ends of the operating means and two points on the object in the virtual space is designated in advance. By doing so, the position and orientation of the object in the virtual space can be controlled as desired.

Next, a second structural example using ultrasonic waves will be described. The operating means shown in FIG. 11 includes an ultrasonic pulse generator 2015 and ultrasonic output devices 2016 and 2017.
・ It is composed of 2018. Ultrasonic pulse generator 20
Reference numeral 15 intermittently outputs pulse signals of different frequencies f, f ', f''(f<f'<f'') at regular intervals T. The ultrasonic wave output device 2016 outputs a pulse signal of frequency f as an ultrasonic wave. Similarly, the ultrasonic wave output devices 2017 and 2018 output pulse signals of frequencies f ′ and f ″ as ultrasonic waves, respectively. Next, the detecting means shown in FIG. 12 is the ultrasonic detecting device 2019.2020.2021,
It is composed of low pass filters 2022, 2025, 2028, band pass filters 2023, 2026, 2029, high pass filters 2024, 2027, 2030, a time detection device 2031 and a position calculation device 2032.

Ultrasonic detector 2019/2020/20
Reference numeral 21 converts ultrasonic waves into electric signals. The low-pass filters 2022, 2025, 2028 are low-pass filters with a cutoff frequency fc1 (f <fc1 <f '), and extract only the signal of frequency f from the input signal. The bandpass filters 2023, 2026, 2029 are bandpass filters having cutoff frequencies fc1 and fc2 (f '<fc2 <f''), and extract only the signal having the frequency f'from the input signals. High pass filter 2024/2027/20
Reference numeral 30 denotes a high-pass filter having a cutoff frequency fc2, which extracts only the signal having the frequency f ″ from the input signals. The time detection device 2031 is the ultrasonic detection device 201.
The time t1, t1 ', t1''when the ultrasonic pulse of frequency f, f', f '' was detected at 9 and the frequency at the ultrasonic detector 2020
Time t2, t2 ', t when the ultrasonic pulse of f, f', f '' is detected
2 '' and the times t3, t3 ', t3''at which ultrasonic pulses of frequencies f, f', f '' are detected by the ultrasonic detecting device 2021 are detected. The position calculation device 2032 has frequencies f, f ′, f ″.
Position in the three-dimensional space P (x,
y, z), P '(x', y ', z'), P '' (x '', y '', z '') are calculated.

Here, as shown in FIG. 13, the ultrasonic output devices 2016, 2017, 2018 are attached to three points P, P ', P''which are not on a straight line on the operating means, Let the distance between PP 'be l1 and the distance between PP''be l2. In addition, the ultrasonic detection devices 2019 and 2020
・ Set the coordinates of 2021 in the three-dimensional space to A1 (x1, y1,
z1), A2 (x2, y2, z2), A3 (x3, y3, z3), which are not on a straight line. At this time, the ultrasonic output device 2
016/2017/2018 Coordinate P in 3D space
(x, y, z), P '(x', y ', z'), P '' (x '', y '', z '') and ultrasonic wave from ultrasonic output device 2016/2017/2018 The following relation holds during the time t when the pulse is output.

[0035]

(Equation 3) Where v is the ultrasonic velocity. Position calculation device 203
In step 2, the equations (10) to (19) are solved as simultaneous equations to obtain the coordinates of the three points on the operating means in the three-dimensional space. Where the points P (x, y, z), P '(x', y ', z'), P '' (x '', y
'', z '') is the solution of this equation, the point A1
(x1, y1, z1), A2 (x2, y2, z2), A3 (x3, y3, z3) points P (x, y, z), P '(x', y ', z '), P''(x'',y'',z'
Points that are symmetric to ') are also clearly solutions of this equation. However, for example, the ultrasonic detection device 201
If 9,2020,2021 are arranged so as to contact the ground as shown in FIG. 13, points A1 (x1, y1,0), A2 (x2, y2,0),
The plane spanned by A3 (x3, y3,0) becomes equal to the ground, and the point P
The range of (x, y, z), P '(x', y ', z'), P '' (x '', y '', z '') is z> 0, z '> 0 , z ''> 0, so the solution is uniquely determined.

The temporal resolution of the operating means and the detecting means described above is determined by the interval T of the pulse signals output from the ultrasonic pulse generator 2015. In order to detect the position and orientation correctly, the pulse at time t and the time
The t + T pulses must be distinguished. To do so, T should satisfy the following equation.

[0037]

[Equation 4] For example, if the distance between the ultrasonic detecting devices 2019, 2020 and 2021 is set to 3 m or less, the left side of the above equation is at most 0.01 seconds and T is about 0.03 seconds. Therefore, in this case, the position and orientation can be detected about 30 times per second.

Further, it is conceivable that an error may occur between the actual position and the detected position due to fluctuations in the sound velocity v due to changes in temperature. However, even in that case, there is no problem in practical use because the correspondence between the actual movement (movement and rotation) of the operating means and the detected movement of the operating means is maintained correctly. Alternatively, if more accurate position and orientation detection is required, the effect of fluctuations in sound velocity v can be removed by solving sound velocity v as an unknown and solving equations (1) to (3) as simultaneous equations. It is possible.

As described above, since the three-dimensional positions of the three points of the operating means can be detected, how the three points on the operating means are associated with the three points on the object in the virtual space. However, it is possible to arbitrarily control the position and orientation of the object in the virtual space.

Next, in the case where the operating means capable of operating the three-dimensional position and orientation and the detecting means for detecting the position and orientation of the operating means are configured to have the detecting means inside the operating means. An example of is shown.

FIG. 14 is a diagram showing the configuration of this embodiment. The operation means 3101 comprises a video input means 3102, a relative position / orientation detection means 3103, and a transmission / reception means 3104.
The video signal input by the video input means 3101 is input to the relative position / direction detection means 3102. The relative position / orientation detecting means 3102 detects a motion vector and a rotation angle with respect to the video signal one frame before based on the input video signal,
Detected relative position / direction information (ΔX, ΔY, ΔZ, ΔθX, Δ
θY, ΔθZ) is sent to the transmitting means 3104. Transmission means 31
04 indicates the relative position and the relative position of the operating means.
The orientation information (ΔX, ΔY, ΔZ, ΔθX, ΔθY, ΔθZ) is transmitted to the control means for controlling the positions and orientations of the constituent elements in the virtual space.

FIG. 15 is a block diagram of the image input means 3102. The image input means 3201 includes a mirror 3202 and an image pickup circuit 3203.
It is composed of an A / D conversion circuit 3204.

The mirror is configured as shown in FIG. 16 so as to capture images in the three directions of the x, y and z axes, and the image pickup circuit 3203 is mounted in front of FIG.

FIG. 17 is a block diagram of the relative position / direction detecting means 3103. The relative position / direction detecting means 3401 is composed of a frame memory 3402, a blocking circuit 3403, a motion vector detecting circuit 3404 and a frame memory 3405.

The video input signal sent from the video input means is once written in the frame memory 3402. The blocking circuit 3403 blocks the video signal of m × n pixels near the center of the video frame of each axis as shown in FIG. 18, and the motion vector detection circuit 3404 converts the m × n block into the following equation (23). Block matching is performed between the affine-transformed image signal and the video signal one frame before on the frame memory 3405. At the time of matching, if there is no corresponding pixel, the video signal of one frame before on the frame memory 3405 is subjected to co-linear interpolation represented by the following equation (24) to generate a pixel value. As a result of matching, the sum of absolute differences of pixel values in the block, or the motion vector and rotation angle (Δt1, Δt2, Δ
θ) is the motion vector and rotation angle of the plane, the motion vector and rotation angle of the yz plane (Δx1, Δx2, Δθx3) The motion vector and rotation angle of the zx plane (Δy1, Δy2, Δθy3) xy The motion vector and rotation angle of the plane Find (Δz1, Δz2, Δθz3).

From the motion vector and the rotation angle, the relative position / direction information (ΔX, ΔY, ΔZ, ΔθX, ΔθY, Δθ of the operating means is obtained.
Z,) is calculated by the equation (25).

[0047]

(Equation 5)

[0048]

(Equation 6)

[0049]

(Equation 7) In the frame memory 3405, the video input signal of the frame memory 3402 is written after detecting the relative position / orientation information.

In this embodiment, the block of the current video input signal is affine-transformed, but there is also a method of affine-transforming the video signal one frame before and performing block matching. Further, as the block matching method in the present embodiment, when a full search is performed considering all possible parameters, the amount of calculation becomes large. Therefore, as a method of reducing the calculation amount, a method of limiting the parameter values (c1, c2, Δt1, Δt2, Δθ) in advance and reducing the calculation accuracy of interpolation, m ×
The n block was subdivided into smaller blocks, and from some of these smaller blocks, the 1990 Fall of Science, D-224
There is a method of estimating the parameters of the above equation (23) by a method as shown in.

FIG. 19 is a diagram showing, as a modification of this embodiment, the configuration of the relative position / orientation detecting means 3501 for the rotation angles (ΔθX, ΔθY, ΔθZ) obtained using a gyro. The relative position / orientation detecting means 3501 is a frame memory.
3502, blocking circuit 3503, and motion vector detection circuit 3504
And a gyro circuit 3505 and a frame memory 3506. The difference from the relative position / orientation detecting means 3401 is that the rotation angle (ΔθX, ΔθY, ΔθZ) is the gyro circuit 340.
5, the motion vector detection circuit 3504 detects only the motion vectors (Δt1, Δt2) according to the above equation (23) based on the value, and the motion vector of each plane yz The motion vector of the plane yz Δx1, Δx2) zx plane motion vector (Δy1, Δy2) xy plane motion vector (Δz1, Δz2) is obtained, and (ΔX, ΔY, ΔZ) is calculated by the following equation (26).

[0052]

(Equation 8) The gyro circuit 3505 is equipped with three gyros for the x-axis, y-axis, and z-axis, and the angular velocities (ωX, ωY,
ωZ) is integrated in a frame unit time of the video input signal, the rotation angle (ΔθX, ΔθY, ΔθZ) of each axis is calculated, and the result is sent to the motion vector detection circuit. For other operations,
This is the same as the relative position / direction detecting means 3401.

The feature of the operating means of this embodiment is that it has a position and orientation detecting means inside the operating means, and can detect an accurate position and orientation without requiring a special device outside the operating means. It is a point that can be done.

Next, an embodiment will be described with reference to the drawings, in which the data portion forming the virtual space is made to have a data structure having a hierarchical structure with respect to reality in the above-mentioned embodiment.

When it is desired to see (display) the components associated with the operating means in a more realistic manner, all the data forming the virtual space are natural image signals or data obtained by highly efficient encoding the natural image signals. Then, the amount of data becomes enormous, which makes it difficult to store in the database. Further, in consideration of the storage cost and the cost for generating a highly realistic natural image, there may be a case where it is not necessary to increase the reality of the components other than the components associated with the operating means in the virtual space. Alternatively, it may not be necessary to display a highly realistic natural image until the desired component is searched.

Therefore, the data forming the virtual space is efficiently stored in the database as computer graphics (CG) data by sacrificing the reality, and only the components corresponding to the operating means are natural images. By accumulating data that can also be displayed as data, the efficiency of accumulating data that constitutes a virtual space is improved.

FIG. 20 is a diagram for explaining another embodiment of the present invention. FIG. 20A shows a first realization method using texture mapping.

The CG model database 5000 stores data representing the shapes of constituent elements in the virtual space. The texture database 5010 stores texture data of the constituent elements in the virtual space associated with the operating means. As texture data,
Natural image signals obtained by photographing the constituent elements from a plurality of directions, or data obtained by highly efficiently encoding the natural image signals are stored. The decoder 5020 decodes and outputs the high-efficiency coded data stored in the texture database 5010. If the image signal itself is stored in the texture database 5010, it is output as it is.

In the texture mapping device 5030,
When the output of the decoder 5020 is selected in the switching device 5040, the texture is mapped to the CG model to generate a pseudo natural image, and the display device 505 is selected.
Supply 0. If the output of the decoder 5020 is not selected in the switching device 5040, C
The structural model image of G (for example, a wire frame image) is supplied to the display device 5050.

The switching device 5040 is controlled by the operating means. For example, the switch (SW) may be switched to the output side of the decoder 5020 when a signal that associates the operating means with a component in the virtual space is detected. Alternatively, the switch (SW) may be switched to the output side of the decoder 5020 by bringing the operating means close to the constituent element in the virtual space.

FIG. 20B shows the second realization method. C
The G database 5100 stores data necessary for generating CG image data of constituent elements in the virtual space. That is, the CG model database 5000 is
Included in this CG database. CG image generation device 5
At 110, a CG image signal is generated from the CG data supplied from the CG database 5100, and the switching device 5
Supply to 140.

The natural image database 5120 stores the natural image signals of the constituent elements in the virtual space associated with the operating means or the data obtained by highly efficient encoding the natural image. In the decoder 5130, when the high-efficiency coded data is stored in the natural image database 5120, this data is decoded and a natural image signal is output. If the natural image signal itself is stored in the natural image database 5120, it is output as it is.

The switching device 5140 selects the output of the CG image generating device 5110 or the output of the decoder 5130 and supplies the image to the display device 5150. The switching device 5140 is controlled by the operating means. For example, a switch (SW) may be switched to the output side of the decoder 5130 when a signal that associates the operating means with a component in the virtual space is detected.

FIG. 20C shows the third method of realization. C
The G database 5200 stores data necessary for generating CG image data of constituent elements in the virtual space. That is, the CG model database 5000 is
Included in this CG database. CG image generation device 5
At 210, a CG image signal is generated from the CG data supplied from the CG database 5200, and the addition circuit 525 is generated.
Supply 0.

The natural image database 5220 stores high-efficiency coded difference signals obtained by subtracting the CG image signals from the natural image signals of the constituent elements in the virtual space associated with the operating means. The decoder 5230 decodes the high-efficiency coded data supplied from the natural image database 5220 and outputs a differential signal.

The switching device 5240 is controlled by the operating means. For example, a switch (SW) may be switched to the output side of the decoder 5230 when associating the operating means with the constituent elements in the virtual space. In the addition circuit 5250, when the output of the decoder 5230 is selected by the switching device 5240, a difference signal is added to the output of the CG image generation device 5210 to generate a natural image and supply it to the display device 5260. When the output of the decoder 5230 is not selected by the switching device 5240, the output of the CG image generation device 5210 is supplied to the display device 5260.

According to the present embodiment, the data structure of the virtual space is a data structure having a hierarchical structure with respect to reality, so that the virtual space can be stored without significantly increasing the amount of data stored in the virtual space. It is possible to increase the reality of a desired component in the inside.

Next, in the control means, when the result of associating the operation means with one component by the means for determining the correspondence is input, the voice recognition means for inputting voice and the voice recognition means. When the result of voice recognition from is input, the correction means for correcting the correspondence between the operation means and one component in the upper virtual space, and the corrected result output from the correction means are input. An example will be described below.

FIG. 21 is a block diagram showing the configuration of this embodiment. The voice 6211 is recognized by the voice recognition means 6201, and the recognition result 6212, the data 6102 that associates the operation means with the constituent elements are input to the correction unit 6202. The correction unit 6202 corrects the associated data 6102 based on the voice recognition result 6212,
The corrected correspondence data 6213 is output. The corrected association data 6213 and the detection result 6104 are input to the control unit 6120.

The control unit 6120 controls the constituent elements of the virtual space based on the detection result 6104 and the corrected correspondence data 6213, and the control result 6105 is displayed on the display unit 6130.
Is input to User is displayed on display 6160
While watching the image 6108 in the three-dimensional space, the operation means 6150 is operated and the voice 6211 is called. The voice 6211 is input as a signal from a microphone, and is analyzed using the input voice signal and the data 6102 associated with the above-mentioned virtual reality constituent elements. For example, in the voice recognition unit 6201, the voice signal is decomposed into vowels and consonants to be sentenced, and intelligent processing for recognizing the meaning is performed. The operation is specified by the constituent element or situation to be operated, and the operation to be actuated can be limited by the specified constituent element or situation. Therefore, the operation for voice input is also limited, which facilitates intelligent processing. Further, it is also possible to prepare the data of the voice signal shape of the operation that can be performed in advance for each component and to determine the operation by the table lookup from the input signal. In the correction unit 6202, the voice input result 6212 is added to the result 6102 detected by the operation means to create data 6213 corresponding to the desired operation.

An example of control based on the obtained detection result and voice input result will be described below. A case where a plurality of components exist in the virtual space will be described. For example, the rearmost component from a very large number of components, or the component that overlaps and is difficult to identify are identified by both the voice input and the operating means. After specifying a representative one of the plurality of constituent elements by the operating means, by voice, "rear" and "right 4
Enter "Tsumu". Similarly, it can be used as a supplementary means for switching to the operation of another component.

Further, when a plurality of constituent elements having common characteristics are operated simultaneously, even if these constituent elements are scattered, the characteristics are first inputted by voice to collectively specify, and thereafter, The operation means operates on all of these constituent elements. Further, when a component to be operated is erroneously detected, it is immediately corrected by voice input. Therefore, from the situation where one constituent element is specified, in this example, the input operation can be limited to the relative position with respect to the constituent element to be truly specified and one feature of the specified constituent element. This voice operation is easier than adding other complementary operation methods. Also, an example will be described in which the component to be operated is a person. Use voice input to call people, talk, and move. It is more natural than other methods of operation and is necessary for communication. It is especially effective in the case of a communication system using virtual space.

Below, for the operation of the constituent elements in the virtual space,
An example of adding a complicated operation beyond the position and orientation will be described. When the material structure of the component and the operating means are different, it is necessary to bend the component, divide it in half, see the contents of the component, smell the smell, operate a part, etc. Need to add a new operation method. In this case, if you add a voice operation method,
Operation is easier than adding other operating methods. Also,
Since it can be used for various operations, it is highly versatile.

Furthermore, even if an operation by voice input is added,
It does not give the user the trouble of operability. Rather, it is a natural operation method for the user. As described above, since the input method of the voice is easy, if it is used as one of the methods for operating the object in the virtual space, a good human interface can be realized. Therefore, the operation by voice input is effective as a means to supplement the above-mentioned operation means.

As described above, by adding the detection result of the voice input to the detection result input by the control means, the correspondence between the operation means and one constituent element in the virtual space is obtained. The effect is that the decision can be made accurately and quickly, and that the human interface is improved.

[0076]

According to the present invention, the object in the virtual space is associated with the operating means in the real space, and the operating means in the real space is operated as if it were an object in the virtual space. At the same time, by arbitrarily switching the association, it becomes possible to operate the object in the virtual space more naturally and easily.

[Brief description of drawings]

FIG. 1 is a diagram illustrating an embodiment of the present invention.

FIG. 2 is a diagram illustrating a more specific embodiment of the present invention.

FIG. 3 is a diagram illustrating state transitions in the embodiment of FIG.

FIG. 4 is a diagram showing a configuration of a sketch in the embodiment of FIG.

FIG. 5 is a diagram for explaining the operation of the embodiment of FIG.

FIG. 6 is a diagram for explaining the operation of the embodiment of FIG.

FIG. 7 is a diagram showing a configuration example of an apparatus that realizes one function of FIG.

FIG. 8 is a diagram showing a configuration of an embodiment of an operating device according to the present invention.

FIG. 9 is a diagram showing a configuration of an embodiment of a detection device according to the present invention.

FIG. 10 is a diagram showing the relationship between the operating device and the detection device of FIGS.

FIG. 11 is a diagram showing a configuration of an embodiment of an operating device according to the present invention.

FIG. 12 is a diagram showing a configuration of an embodiment of a detection device according to the present invention.

FIG. 13 is a diagram showing the relationship between the operating device and the detection device of FIGS.

FIG. 14 is a diagram showing a configuration of another operation device and a detection device according to the present invention.

15 is a diagram showing the configuration of the video input means of FIG.

16 is a diagram showing a configuration of the mirror of FIG.

17 is a diagram showing an example of the configuration of the detection means shown in FIG.

FIG. 18 is a diagram showing an example of the relationship between screens and blocks in the embodiment of FIG.

19 is a diagram showing another configuration example of the detecting means in FIG.

FIG. 20 is a diagram showing the configuration of an embodiment incorporating the concept of reality scalability in the present invention.

FIG. 21 is a diagram showing a configuration of an example in which a detection result by voice is added in the present invention.

[Explanation of symbols]

100 ... Accumulation part, 110 ... Correlation part, 120 ... Control part, 130 ... Compositing part, 140 ... Detection means, 150 ... Operation means, 160 ... Display part.

 ─────────────────────────────────────────────────── ─── Continuation of the front page (72) Inventor Emperor Amada 1 Komukai Toshiba-cho, Sachi-ku, Kawasaki-shi, Kanagawa Prefecture, Corporate Research & Development Center, Toshiba Corporation (72) Yoko Miyakawa Komukai-Toshiba, Kawasaki-shi, Kanagawa Town No. 1 Incorporated company Toshiba Research and Development Center (72) Inventor Takehiko Kagoshima Komukai Toshiba Town No. 1 in Komukai Toshiba Town, Kawasaki City, Kanagawa

Claims (1)

[Claims] 1. An operating means capable of operating a position and an orientation, a detecting means for detecting the position and the orientation of the operating means, an accumulating means for accumulating data of components constituting a virtual space, and an accumulating means. A display device for displaying the virtual space based on the data accumulated in the means, and a configuration of the operation means and the virtual space displayed on the display device according to the operation of the operation means. Corresponding means for performing correspondence with elements and switching the correspondence to other constituent elements of the virtual space, and the relative position and orientation of the constituent elements of the corresponding virtual space, And a control unit that controls based on the detection result of the detection unit.