JPH08315175A - Virtual reality device - Google Patents

Abstract

PURPOSE: To provide a virtual reality device by which a feeling of visually and auditorily deeper immersion can be obtained. CONSTITUTION: While spatial data read out of a spatial data storage part 107 are displayed on a goggle type stereoscopic display device 101, a person who is experiencing the virtual reality looks at a virtual entity and then shape data corresponding to the head height and moving speed of the person at the time are superimposed on the spatial data by using shape data on the virtual entity which are read out of an entity still shape data storage part 108 and an entity operation shape data storage part 109 and stored in an entity shape temporary storage part 110.

Description

Detailed Description of the Invention

[0001]

BACKGROUND OF THE INVENTION 1. Field of the Invention The present invention relates to a simulated experience device.

[0002]

2. Description of the Related Art Recently, a simulated experience device based on so-called virtual reality technology has been in the limelight and is used for designing houses and selling house equipment. For example, in designing a house, the designed house is realized in a virtual space, and while evaluating the designed house while getting the feeling of being in the space, that is, immersive feeling, and selling the housing equipment, for example, In the case of selling system kitchens, the prospective purchaser can experience multiple system kitchens in a virtual space and select the desired one from them.

In the conventional simulated experience device, a goggle type stereoscopic display device attached to the head of the experience person has a line of sight of the experience person detected by a sensor integrally attached to the stereoscopic display device. According to the drawing, the surrounding virtual space was drawn.

[0004]

However, in the above-described conventional pseudo experience device, when the experience person turns his or her gaze toward his or her feet, the surrounding scene is drawn according to the gaze,
Since the person's feet, which should be seen there, are not drawn, the experience person feels as if they were floating in the air, which impairs the immersive feeling.

In view of the above problems, it is a first object of the present invention to provide a simulated experience device capable of visually obtaining a higher sense of immersion. The second object of the present invention is to
It is an object of the present invention to provide a simulated experience device that can provide a higher sense of immersion.

[0006]

In order to achieve the first object, a simulated experience device according to claim 1 is a stereoscopic display device mounted on a head of a user and used, and a stereoscopic display device of the stereoscopic display device. The spatial data storage means for storing the three-dimensional data of the virtual space to be displayed, the detection means for detecting the position and orientation of the head of the experiencer, and the viewpoint position of the experiencer determined from the detection result are also included. A first display control means for selecting a predetermined space data and displaying it on a display device; and a main body shape data storage means for storing the shape data of a virtual main body simulating the body shape of the experiencer. According to the detection result of the detection means, when the sight line of the experience person is directed to the body direction of the virtual subject, the main body shape data is read from the main body shape data storage means and superposed on the spatial data, thereby providing a stereoscopic display device. And second display control means for displaying It is characterized.

Further, in order to achieve the first object, the simulated experience device according to claim 2 is different from the simulated experience device according to claim 1 in that the main shape data storage means has a type of animal, Shape data of a virtual subject imitating a plurality of body shapes such as male and female are also stored, and one of a plurality of types of shape data stored in the subject shape data storage means is stored. The second display control means is provided with a selecting means for selecting, and according to the detection result of the detecting means, when the sight line of the experience person is directed to the body direction of the virtual subject, the subject shape data selected by the selecting means is displayed. Spatial data is read from the main shape data storage means.
The feature is that it is displayed on a stereoscopic display device in a manner superimposed on the data.

Further, in order to achieve the first object, the simulated experience device according to claim 3 is the claim 1 or claim 2.
In the simulated experience device, the subject shape data storage means stores a plurality of posture data representing the posture state determined by the height of the head of the experience person for each shape data type. Cage, second
According to the detection result of the detecting means, when the sight line of the experience person is directed toward the body of the virtual subject, the display control means of the subject displays posture data corresponding to the head height at that time from the subject shape data storing means. It is characterized in that it is read out and superposed on the spatial data and displayed on a stereoscopic display device.

Further, in order to achieve the first object, the simulated experience device according to claim 4 is claim 1 or claim 2.
In the simulated experience device described above, the main shape data storage means stores a series of movement motion data that changes with the movement motion for each shape data type, and the space of the experience person is stored. In accordance with the speed detecting means for detecting the moving speed in the inside and the detection result of the detecting means, when the sight line of the experience person is directed to the body direction of the virtual subject, depending on the movement speed of the experience person detected by the speed detecting means And a third display control means for sequentially superimposing the moving operation data on the spatial data and displaying the same on the stereoscopic display device.

In order to achieve the second object, the simulated experience device according to claim 5 is a stereoscopic display device that is used by being attached to the head of the experience person, and is a display target of the stereoscopic display device. Spatial data for storing three-dimensional data in virtual space
Data storage means, detection means for detecting the position and orientation of the head of the experience person, and display control for selecting and displaying predetermined spatial data on the display device based on the viewpoint position of the experience person determined from the detection result. Means, a sound field reproducing device used by being attached to the head of an experience person, and an eco-mode for each frequency band when a plurality of positions in the space are sound source positions and arbitrary positions are sound receiving points. Band response storage means for storing the response for each frequency band obtained by convolving the band pass filter for each frequency band in the time pattern, head-related transfer coefficient storage means for storing the head-related transfer coefficient, and anechoic Sound data of sound,
A dry source storing means for storing a plurality of dry sources corresponding to the sound source positions, and for each sound source, the position of the head of the experience person determined from the detection result of the detecting means is the sound receiving point in the space. Then, the response for each frequency band of each sound source is calculated by convolving the dry source and the head-related transfer coefficient to the band response corresponding to that position, and the response for each frequency band of each sound source is calculated for each same frequency band. After that, the reproduced sound data is generated by synthesizing over the entire frequency band, and the reproduced sound data is converted into a sound and output to the sound field reproducing device.

Further, in order to achieve the second object, the simulated experience device according to claim 6 is a stereoscopic display device which is used by being attached to a head of an experience person, and a display target of the stereoscopic display device. Spatial data for storing three-dimensional data in virtual space
Data storage means, a first detection means for detecting the position and orientation of the head of the experience person, and a predetermined space data is selected and displayed on the display device based on the viewpoint position of the experience person determined from the detection result. Display control means, a sound field reproducing device attached to the head of the user, and a sound source position at a predetermined position on the surface of a virtual object such as a wall, furniture, or window existing in the background of the display target space. Band response storage means for storing the response for each frequency band obtained by convolving the band pass filter for each frequency band in the eco-time pattern for each frequency band when the position of is the sound receiving point, Head-related transfer coefficient storage means for storing the head-related transfer coefficient, dry sound storage means for storing a plurality of dry sources corresponding to the virtual object, which is acoustic data of anechoic sound, and a body of an experience person. The second part to detect the position of The first means when the position data detected by the second detection means and the position data of the virtual object coincide with each other when the experience person moves in the virtual space. By convolving a head-related transfer function with a dry source corresponding to a virtual object whose position data matches the band response corresponding to the position of the head of the experience person determined by the detection result of The reproduced sound data is generated by calculating the response for each frequency and synthesizing the response for each frequency band over the entire frequency band.
It is characterized in that the data is converted into voice and output to the sound field reproducing device.

Further, in order to achieve the second object, the simulated experience device according to claim 7 is a stereoscopic display device which is attached to a head of a user and used, and a display target of the stereoscopic display device. Spatial data for storing three-dimensional data in virtual space
Data storage means, detection means for detecting the position and orientation of the head of the experience person, and display control for selecting and displaying predetermined spatial data on the display device based on the viewpoint position of the experience person determined from the detection result. Means, a sound field reproducing device used by being attached to the head of the experience person, voice input means for inputting a sound near the mouth of the experience person, and the mouth of the experience person as the sound source position in the space. A band that stores the response for each frequency band obtained by convolving the bandpass filter for each frequency band into the eco-time pattern for each frequency band when the vicinity of both ears of the experienced person is the sound receiving point. A response storage means and a head-related transfer coefficient storage means for storing a head-related transfer coefficient are provided, and the band response corresponding to the position of the experiencer's head determined from the detection result of the detection means is input from the voice input means. Voice and head communication Generates data, - calculating a response for each frequency band by convoluting number, playback sound de by the response for each frequency band is synthesized over the entire frequency band
It is characterized in that the reproduced sound data is converted into voice and outputted to the sound field reproducing device.

In order to achieve the first object, the simulated experience device according to claim 8 is different from the simulated experience device according to claim 1 in that a plurality of voice waveforms and instruction information corresponding to each of them are provided. And a reference voice pattern storage unit for storing the reference voice pattern.
It is characterized by a detection means for detecting the instruction information from the content stored in the storage unit.

[0014]

According to the configuration of the simulated experience device according to claim 1,
When the viewer's line of sight is directed toward the body of the virtual subject while the spatial data is displayed on the stereoscopic display device, the shape data of the virtual subject that imitates the body shape of the experience subject is displayed as the spatial data. Is superimposed and displayed on the stereoscopic display device. Here, in the present specification, "space" means not only a literal space with a considerable space in which no object exists, but also a wall that can be seen as the background,
It is used as a concept that includes floors and devices and equipment provided on them.

According to the structure of the virtual experience device according to the second aspect, one of the shape data of the virtual subject imitating a plurality of body shapes such as the type of animal and male and female is selected, and the stereoscopic display device is selected. When the viewer's line of sight is directed toward the body of the virtual subject while the spatial data is displayed on the screen, the selected shape data is displayed.
The data is superimposed on the spatial data and displayed on the stereoscopic display device. According to the structure of the simulated experience device according to claim 3, when the viewer's line of sight is directed to the body direction of the virtual subject while the spatial data is displayed on the stereoscopic display device, the viewer's head at that time Posture data according to the height of the section is superimposed on the space data and displayed on the stereoscopic display device.

According to the structure of the simulated experience device according to the fourth aspect, when the sight line of the experience person is directed toward the body of the virtual subject while moving while the spatial data is displayed on the stereoscopic display device, A series of moving motion data is sequentially superimposed on the spatial data according to the moving speed and displayed on the stereoscopic display device. Claim 5
According to the configuration of the simulated experience device described above, the dry source corresponding to each of the plurality of sound source positions and the head-related transfer coefficient are convoluted into the band response of each sound source corresponding to the head position of the person who experiences the sound source. After calculating the response for each frequency of the sound source and synthesizing the response for each frequency of each sound source for each identical frequency band, the synthesized sound data is generated by synthesizing over all frequency bands, and the reproduced sound data is obtained. It is converted into voice and output to the sound field reproducing device.

According to the structure of the simulated experience device described in claim 6, when the position of a part of the body of the experience person and the position of the surface of the virtual object match, the dry source corresponding to the matched virtual object.
And the head-related transfer coefficient are convoluted into the band response for each sound source corresponding to the head position of the experience person to calculate the response for each frequency, and the response for each frequency is synthesized and played over all frequency bands. Sound data is generated, the reproduced sound data is converted into voice, and the sound is output to the sound field reproducing device.

According to the structure of the simulated experience device according to claim 7, each frequency is obtained by convolving the voice input from the voice input means and the head-related transfer coefficient into the band response corresponding to the head position of the experiencer. The response for each frequency is calculated, and the reproduced sound data is generated by synthesizing the response for each frequency over the entire frequency band.
The reproduced sound data is converted into voice and output to the sound field reproducing device.

According to the structure of the simulated experience device according to the eighth aspect, when the experience person gives an instruction by voice to the body direction of the virtual subject while the spatial data is displayed on the stereoscopic display device, Shape data of a virtual subject imitating the body shape of the experience person is superimposed on the space data and displayed on the stereoscopic display device.

[0020]

DESCRIPTION OF THE PREFERRED EMBODIMENTS Embodiments of the present invention will be described with reference to the drawings. (First Embodiment) FIG. 1 is an overall configuration diagram of a simulated experience device according to a first embodiment of the present invention. As shown in the figure, the simulated experience device includes a goggle-type stereoscopic display device 101, a magnetic field generator 102, a head magnetic sensor 103, a subject selection menu storage unit 104, a hand input device 105, and a magnetic sensor.
Unit 106, spatial data storage unit 107, main stationary shape data storage unit 108, main motion shape data storage unit 10
9, a main body shape temporary storage unit 110, a display control unit 111, a program storage unit 112, and a CPU 113.

The goggle-type stereoscopic display device 101 is mounted on the head of the experience person M, and the position and direction of the head constantly detected by the head magnetic sensor 103, that is, a virtual position corresponding to the sight line of the experience person M. A stereoscopic interior scene or the like is stereoscopically displayed by utilizing the parallax of the left and right eyes, and two liquid crystal color displays are built in for the left and right eyes. The magnetic field generation device 102 is a device that generates a magnetic field in a space where an experiencer moves around, and has a quadrature coil and the like built therein.

The head magnetic sensor 103 is integrally attached to the goggle type stereoscopic display device 101 and detects the position and direction of the head of the experience person M in the magnetic field. Etc. are built in. Main subject selection menu
The storage unit 104 stores a menu whose display mode is shown in FIG. The subject selection menu is used to select the type of subject to be displayed in the virtual space.

The hand input device 105 has a shape shown in FIG. 3 and is a device which the experience person M holds in one hand and operates, and has a right button 500, a center button 510, a left button 520 and the like. The right button 500 is for displaying the subject selection menu on the goggle type stereoscopic display device 101, and when the right button is pressed, the subject selection menu is displayed. The center button 510 is for moving the pointer 400 of the subject selection menu, and each time the center button 510 is pressed, the pointer 4 is moved.
00 sequentially moves to items 420, 430, ... The left button 520 is used to select a menu item on the subject selection menu, and the pointer 4
Used to select an item by pressing when 00 is on the menu item (subject type) to be selected. In addition, when the left button 530 is pressed, the menu is erased.

The magnetic sensor unit 106 processes the magnitude of the electric current induced in the head magnetic sensor 103 placed in the magnetic field generated by the magnetic field generating device 102, so that This is a device for measuring the position and direction of the head and the moving speed of the experience person M, and outputting the measurement result to the CPU. The space data storage unit 108 stores three-dimensional shape data of a space including a room wall, a ceiling, a floor, a window, a table, and the like which are display targets.

As shown in FIG. 4, the subject stationary shape data storage unit 108 stores the subject shapes displayed in the virtual space, such as the experience subject, adult / male, adult / female, and the display state thereof. It is stored in a place specified by the address for each standing state and sitting state). The subject motion shape data storage unit 109 stores, as shown in FIG. 5, a series of shape data indicating a walking state that changes according to the movement of the experience person M in the virtual space, by address for each subject type. Stored in a specified location. A series of operations is completed with the end numbers 1 to 12 of each address as one cycle, and each shape data is sequentially displayed according to the moving speed of the experience person M. In the present embodiment, one cycle of a series of operations is represented by 12 patterns of shape data, but the number of patterns is not limited and may be increased or decreased as appropriate.

As shown in FIG. 6, the main body shape temporary storage unit 110 stores the main body still shape data storage area 800, the main body motion shape data storage area 900, and the pointer position management area 30.
It has 0 and is composed of RAM and the like. Main subject stationary shape data storage area 800 and main subject motion shape data storage area 90
Reference numeral 0 denotes main body shape data corresponding to the main body type selected in the main body selection menu, and the main body static shape data storage unit 108 and the main body motion shape data storage unit 10 are stored.
The shape data called from 9 are stored in the locations specified by the respective addresses. The pointer position management area 300 is a place for selectively managing the display of the main body shape data, and according to the head height, line of sight, and moving speed of the experience person M detected by the head magnetic sensor 103, The pointer 310 is switched to display the shape data where the pointer is located. Reference numeral 320 in FIG. 6 denotes a pointer waiting area when no main body shape data is displayed.

The display control unit 111 creates drawing data according to the line of sight of the experience person M based on the data stored in the space data storage unit 107, and from the drawing data. A video signal is generated, and the video signal is used as the goggle-type stereoscopic display device 10.
1 or a video signal is similarly generated from the shape data stored in the main shape temporary storage unit 110, and is superimposed on the video signal from the spatial data and output. .

The program storage unit 112 stores a program corresponding to the flowchart shown in FIG.
It consists of OM. The CPU 113 performs processing according to the program stored in the program storage unit 112, and includes a microprocessor and the like. Hereinafter, the control content of the present apparatus will be described based on the flowchart shown in FIG. 7.

With the space data displayed (step S
100), the experience person moves the line of sight (step S1)
10) Display the spatial data corresponding to the line of sight at that time (step S111), while continuing to display the same spatial data if the line of sight is not moved. Then, step S12
If there is a menu display instruction from the experience-maker at 0, the process proceeds to step S121, and the same subject selection menu as shown in FIG. 2 is displayed.

Looking at the display screen, the experience person operates the hand input switch 105 to select one subject who wants to be in the virtual space (step S122). Here, "adult / male"
If is selected, the process proceeds to step S123,
Shape data corresponding to the selected subject type "Adult / Male"
The data is read from the main body still shape data storage unit 108 and the main body motion shape data storage unit 109 and stored in the main body shape temporary storage unit 110. The storage result is shown in FIG. continue,
The process proceeds to step S124, and the height of the viewpoint of the experience person in the virtual space is changed to the height of the viewpoint according to the selected subject type. By doing so, for example, when the experience person is an adult and the subject selected in step S122 is a child, the experience person can experience a simulated experience from the viewpoint (height) of the child. Subsequently, the process is step S12.
5, the subject selection menu display is erased, and the process proceeds to step S130.

In step S130, the head magnetic sensor--
Based on the head height detected by 103, it is determined whether the experiencer is standing or sitting. Here, if the experience person is sitting, the process proceeds to step S140, and if the experience person turns his or her gaze toward the body of the virtual subject, the process proceeds to step S141, and the selected subject type "adult / male" The main body still shape data in the sitting state is read out from the main body shape temporary storage unit 110 and is displayed by being superimposed on the space data. An example of the display result is shown in FIG.

On the other hand, if it is determined in step S130 that the experiencer is standing, the process proceeds to step S130.
Proceeding to 131, it is judged whether or not the sight line of the experience person is facing the body direction of the virtual subject, and if so, the process proceeds to step S132, and it is further determined whether the experience person is moving or stationary. to decide. The determination is made based on the value V of the moving speed of the experiencer output by the magnetic sensor unit 106. If V is less than or equal to a preset value V1, it is determined that the experiencer is stationary, while V If exceeds V1, the experience person determines that the user is moving. If it is determined that the subject is still, the process proceeds to step S133, and the subject still shape data in the standing state of the selected subject type “adult / male” is read from the subject shape temporary storage unit 110. And is displayed by superimposing it on the spatial data. An example of the display result is shown in FIG. On the other hand, when it is determined that the subject is moving, the process proceeds to step S134, and the main motion shape data stored in the main shape temporary storage unit shown in FIG. ,
AM2, ..., AM12, AM1, ... Are sequentially read and displayed by superimposing them on the spatial data.

Also, in step S122, "Cat" is selected as another subject, and in a standing state (step S13).
0) When the experience person turns around, the eyes turn to the body direction of the virtual subject "cat" (step S133), and it is assumed that the experience person is still at that time (step S132),
The process of step S133 is performed, and the subject type "cat"
The main body static shape data in the standing state is displayed by being superimposed on the space data. FIG. 11 shows an example of the display result.

As described above, according to the simulated experience device of the first embodiment, when the observer turns his or her line of sight in the body direction of the virtual subject, the shape of the virtual subject according to the head height and the moving speed of the experience subject at that time. Since the data is displayed so as to be superimposed on the spatial data, a higher sense of immersion can be visually obtained. (Second Embodiment) FIG. 12 is an overall configuration diagram of a simulated experience device according to a second embodiment of the present invention. As shown in the figure, the simulated experience device includes a goggle type stereoscopic display device 201, a magnetic field generation device 202, a head magnetic sensor 203, headphones 204, a sound source selection menu storage unit 205, a hand input device 206, and a magnetic field. Sensor unit 207, spatial data storage unit 208, right-hand shape data storage unit 209, contact area storage unit 210, continuous sound dry source storage unit 211, impact sound dry source storage unit 212, microphone 213, Band response storage unit 214, head related transfer function storage unit 215, voice control unit 216, display control unit 217, program storage unit 218,
It is composed of a CPU 219.

The goggle type stereoscopic display device 201 is mounted on the head of the experience person M, and the position and direction of the head constantly detected by the head magnetic sensor 203, that is, the virtual line corresponding to the line of sight of the experience person M. A stereoscopic interior scene or the like is stereoscopically displayed by utilizing the parallax of the left and right eyes, and two liquid crystal color displays are built in for the left and right eyes. The magnetic field generation device 202 is a device that generates a magnetic field in a space in which an experiencer moves, and has a quadrature coil and the like built therein.

The head magnetic sensor 203 is integrally attached to the goggle type stereoscopic display device 201 and detects the position and direction of the head of the experience person M in the magnetic field. Etc. are built in. Headphones 20
4 is attached to the head of the experience person M, and the head magnetic sensor-2
This is for reproducing the sound in the three-dimensional sound field according to the position and direction of the head constantly detected by 03, that is, the position and direction of both ears of the experiencer M.

The sound source selection menu storage unit 205 stores a menu whose display mode is shown in FIG.
The sound source selection menu is used to select the kind of sound source desired to be heard among sound sources of continuously audible sounds existing in the virtual space. As shown in FIG. 3, the hand-held input device 206 has a shape similar to that of the hand-held input device 105 of the first embodiment, and is a device that the experience person M normally operates with his / her right hand. It has a right button 600, a center button 610, a left button 620, etc., and a magnetic sensor at hand (not shown).
Built in. The right button 600 is the sound source menu
The display / non-display is switched to the goggle type stereoscopic display device 201, and the display / non-display is repeated each time the right button 600 is pressed. Center button 610
Is for moving the pointer 50 of the sound source selection menu, and each time the center button 610 is pressed, the pointer 50 moves over the item indicating the sound source type 51, 5 or 5.
2, 53, 51, ... Left button 620
Is for selecting or canceling a menu item on the sound source selection menu, and when the pointer 50 is on a menu item (sound source type) to be selected or canceled, Used to select or cancel that item. The hand magnetic sensor detects a position in the magnetic field generated by the magnetic field generator 202, and is composed of a quadrature coil and the like. When the experience person M holds the hand input device 206 in the right hand, the position of the right hand of the experience person M in the virtual space is detected by the hand magnetic sensor.

The magnetic sensor unit 207 processes the magnitude of the current induced in the head magnetic sensor 203 and the hand magnetic sensor placed in the magnetic field generated by the magnetic field generator 202. Is a device for measuring the position and direction of the head and right hand of the experience person M and outputting the measurement result to the CPU 219. The space data storage unit 208 includes an interior wall, a ceiling, a floor, which are display targets.
It stores the three-dimensional shape data of the space consisting of windows, tables and the like.

The right-hand shape data storage unit 209 stores shape data imitating the right-hand shape of the experience person, and the right-hand shape data is in the hand input device 20 in the sight line of the experience person M.
When 6 is entered, the magnetic sensor at hand detects the position,
It is displayed by being superimposed on the space data. As shown in FIG. 14, the contact portion area storage unit 210 stores, for each virtual object type displayed as a virtual space, the contact portion area and the dry source NO. And are stored.

The continuous sound source device 211 is a continuous sound,
A device for outputting to the CPU 219 an audio signal (dry source) which is not influenced by reflection from a wall surface such as an output sound of an audio device such as a stereo, a playing sound of a musical instrument such as a piano, a running sound of an automobile, etc. Yes, it is composed of a plurality of CD (compact disc) players and the like. For each CD player,
A CD in which one type of continuous sound dry source is recorded is mounted, and each CD player is associated with the menu item of the sound source selection menu depending on the type of continuous sound stored therein. . Then, the handheld input device 20
Along with the selection or cancellation of the menu item of the sound source selection menu by 6, the corresponding CD player is started (output of audio signal) or stopped.

The impact sound dry source storage unit 212 is shown in FIG.
As shown in FIG. 5, the dry source NO. Corresponding to, a dry source of an impact sound generated when a human hand comes into contact with an actual object corresponding to the virtual object type name is stored, and is composed of a ROM or the like. The microphone 213 is a device that converts the voice of the experience person M into a voice signal and outputs the voice signal to the CPU 219, and further includes a switch (not shown) that switches between output and non-output. Since the output signal of the microphone 213 is a dry source, it is preferable that the microphone 213 has a high directivity that does not detect the background sound and the reflected sound of the voice of the person M as much as possible.

The band response storage unit 214 stores 2 in the virtual space.
By setting points, and convolving the bandpass filter for each frequency band into the eco-time pattern for each frequency band when one of the points is the sound source position and the other point is the sound receiving point A band response which is a response for each obtained frequency band is stored, and a band response storage area for continuous sound 214a, a band response storage area for impact sound 214b, and a band response storage area for experience voice 214c are stored depending on the type of sound source. Have The band response storage area 214a for continuous sound is shown in FIG.
As shown in FIG. 6, the band response specified from each sound source position and the sound receiving point position is stored for each sound source whose sound source position is a fixed point in the virtual space. Also,
The continuous sound receiving point area in FIG. 16 is position information of a space centered on the position of the receiving point and having a certain spread. As shown in FIG. 17, the impact sound band response storage area 214b stores the band response specified by the sound source position and the sound receiving point position for each sound source type of the impact sound. In FIG. 17, the impulsive sound source region is position information of a space having a certain extent around the sound source position, and the impulsive sound receiving point region is centered on the position of the sound receiving point. This is position information of a space having a spread. Experienced voice band response storage area 214c
18 shows the microphone 2 in the virtual space as shown in FIG.
The band response specified by the sound source position which is the position of 13 and the sound receiving point position which is the midpoint between the ears of the experiencer is stored.
The experience-sound-voice receiving point area in FIG. 18 is position information of a space having a certain space centered on the position of the receiving point.

The head-related transfer function storage unit 215 is a function representing a state in which a sound incident from the surroundings toward the head changes depending on conditions such as the shape of the human head and the position of the ear, and is calculated in advance. It stores things. Voice control unit 2
Reference numeral 16 is a portion for converting the reproduced sound data output from the CPU 219 into an audio signal and outputting a realistic three-dimensional sound field to the headphone 204.

The display control unit 217 creates drawing data according to the line of sight of the experience person M based on the data stored in the space data storage unit 208, and from the drawing data. A video signal is generated, and the video signal is used as the goggle type stereoscopic display device 20.
1, or a video signal is similarly generated from the shape data stored in the right-hand shape data storage unit 209 and is superimposed on the video signal from the spatial data and output. By the way.

The program storage section 218 is shown in FIGS.
The program corresponding to the flowchart shown in FIG. CPU 219
The processing is performed according to the program stored in the program storage unit 218, and includes a microprocessor or the like. The control contents of this apparatus will be described below based on the flowcharts shown in FIGS.

FIG. 19 is a flow chart showing the outline of the processing procedure. When there is a menu display instruction while displaying the space data (step S200) (step S200)
210) and a sound source selection menu similar to that shown in FIG. 13 is displayed (step S211). In step S211,
When the user gives an instruction to select a sound source by looking at the display screen, the CD player corresponding to the selected sound source is activated (step S213). On the other hand, if the user gives an instruction to erase the sound source, the CD player corresponding to the erased sound source is stopped (step S214). In addition, if the user gives an instruction to delete the menu display, the menu is displayed.
The display is erased (step S215).

When the processing in the sound source selection menu display state is completed or when there is no menu display instruction in step S210, the processing proceeds to step S220.
If no continuous sound has been selected, step S230.
If there is a shift in the line of sight of the experiencer, the spatial data according to the line of sight at that time is displayed (step S23).
1) If the line of sight does not move, the same spatial data is continuously displayed. On the other hand, if it is determined in step S220 that the continuous sound is selected, the process proceeds to step S221 to reproduce the continuous sound, and then the process proceeds to step S222 to move the sight line of the experiencer. In this case, a continuous sound is reproduced according to the line of sight (head position and orientation) at that time (step S223), and spatial data is displayed (step S).
224).

Step S231 or step S224
If it is determined that there is no movement of the line of sight in step S230 or step S222, the process proceeds to step S240, and the position data detected by the magnetic sensor at hand and the contact portion storage area 210 are stored. It is determined whether or not the position data indicating the present contact area is matched, and if there is a match (contact is made), the impact sound corresponding to the corresponding contact area is reproduced (step S24).
1).

On the other hand, if there is no match (no contact) in step S240, the process proceeds to step S250. If there is a voice input from the microphone 213, the input voice is reproduced, and if there is no voice input, the step is performed. Return to S210.
Next, details of the processing procedure in the continuous sound reproduction step S221, the continuous sound reproduction update step S223, the impact sound reproduction step S241, and the audio reproduction step S251 will be described.

FIG. 20 shows a continuous sound reproducing step S221,
It is a flow chart which shows details of a processing procedure of update step S223 of continuous sound reproduction. First, the band response (see FIG. 16) corresponding to the continuous sound receiving point area including the head position detected by the head position magnetic sensor 203 is read for each selected sound source type (step S2210). The process proceeds to step S2211, the dry source and the head related transfer function are convoluted into the band response for each sound source, and then the process proceeds to step S2212 for each frequency band obtained for each sound source in the previous step. Are combined for each identical frequency band, and then the combined response for each identical frequency band is combined over all frequency bands to generate reproduced sound data (step S2213). Then, the process is step S
In step 2214, the reproduced sound data generated in the previous step is converted into sound and output to the headphone (step S
2215).

FIG. 21 is a flow chart showing details of the processing procedure of the impact sound reproduction step S241. First, a corresponding band response is obtained from the impact sound source area including the coincident position detected in the previous step S240 and the impact sound receiving point area including the head position detected by the head position magnetic sensor-203 at that time (see FIG. 17). ) Is read (step S241)
1), the process proceeds to step S2412, and the dry source corresponding to the contacted virtual object (see FIGS. 14 and 15) and the head related transfer function are convoluted with the read band response, and then the process proceeds to step S2413. Proceeding, the response for each frequency band obtained in the previous step is synthesized over all frequency bands to generate reproduced sound data. Subsequently, the process proceeds to step S2414, the reproduced sound data generated in the previous step is converted into a sound, and the sound is output to the headphones (step S2415).

FIG. 22 is a flow chart showing details of the processing procedure of the audio reproducing step S251. First, the corresponding band response from the experience-sound receiving point area including the head position detected by the head position magnetic sensor-203 (see FIG. 18)
(Step S2511), the process proceeds to step S2512, and the microphone 2 is added to the read band response.
The voice signal from 13 and the head related transfer function are convoluted, and then the process proceeds to step S2513, and the response for each frequency band obtained in the previous step is synthesized over all frequency bands to generate reproduced sound data. To do. Subsequently, the process proceeds to step S2514, the reproduced sound data generated in the previous step is converted into a sound, and the sound is output to the headphones (step S2515).

In this embodiment, when the experience person holds the hand input device having the hand magnetic sensor in his / her right hand and moves in the virtual space, when the experience person's right hand comes into contact with the virtual object. Although the impulsive sound is reproduced, the magnetic sensor may be attached to another part of the body of the experience person, and the content of the impulsive sound dry source storage unit may be appropriately changed. By doing so, it becomes possible to reproduce the impact sound when another part of the experienceer's body comes into contact with the virtual object. Needless to say, a plurality of magnetic sensors may be attached to the body of the experiencer.

As described above, according to the simulated experience device of the second embodiment, the sounds from a plurality of sound sources that can be heard in the real space,
The impulsive sound when a part of the body comes into contact with the object and the voice uttered by the user can be heard as the sound in the virtual space, so that a higher immersive feeling can be auditorily obtained. (Third Embodiment) FIG. 23 is an overall configuration diagram of a simulated experience device according to a third embodiment of the present invention.

The third embodiment differs from the first embodiment in that the experience person does not move and gives an instruction by voice,
It is devised so that the spatial data displayed on the goggle type stereoscopic display device can be changed. Therefore, the third embodiment is different from the first embodiment in that the microphone 301,
The A / D converter 302, the input voice pattern temporary storage unit 303, the reference voice pattern storage unit 304, the instruction code temporary storage unit 305, and the instruction amount storage unit 306 are provided. From the above, the magnetic generator 102, the head magnetic sensor 103, and the magnetic sensor unit 106 are omitted. In the drawings and the like used for the description of the third embodiment, the same components as those in the first embodiment are designated by the same reference numerals, and the description thereof will be omitted.

The microphone 301 is a device for converting the voice of the experience person M into a voice signal and outputting it to the A / D converter 302. The A / D converter 302 converts an analog audio signal from the microphone 301 into a digital signal and converts it into the CPU 113.
Is a device for outputting to. Input voice pattern temporary storage unit 3
Reference numeral 03 is a place for storing a voice waveform for one word input from the microphone 301, and is composed of a RAM or the like.

The reference voice pattern storage unit 304 is shown in FIG.
As shown in FIG. 5, a reference voice pattern storage area 242 for storing a waveform of a voice used for voice instruction, and an instruction code storage area 243 for storing an instruction code corresponding to each voice waveform. have. For example, in 242a, a waveform of a voice pronounced as "mae" is stored.
The instruction code "D1" corresponding to that is stored in a.

As shown in FIG. 25, the instruction code temporary storage section 305 stores the instruction codes corresponding to the voice waveforms stored in the input voice pattern temporary storage section 303. The storage area is determined by the type of instruction. 26, the instruction amount storage unit 306 stores the instruction amount corresponding to the instruction code or the combination thereof. For example, for the instruction code V1 (fast), the speed v1 is
The distance m1 is stored for the instruction code T1 (slight) and A1 (moving), and the angle a2 is stored for the instruction code T2 (many) and A2 (facing). The instruction codes D1 to D1 regarding the instruction type "direction"
Since D6 has nothing to do with the quantity, no corresponding data is stored in this storage unit.

The control contents of this apparatus will be described below with reference to the flow chart shown in FIG. First, go
When spatial data is displayed on the glu-type stereoscopic display device 101 (step S300) and the experience person M inputs a voice (step S310), the process proceeds to step S320, and the waveform of the input voice is input. The waveform is stored in the pattern temporary storage unit 303, and then the process proceeds to step S330, and a waveform corresponding to the stored waveform is used as a reference voice pattern.
Search from the storage unit 304.

If there is no corresponding waveform pattern (step S340), the process goes to step S31.
It returns to the front of 0 and enters the voice input waiting state. On the other hand, if there is a corresponding waveform, the process proceeds to step S350, and the instruction code of the waveform is changed to the instruction code temporary storage unit 3
05 is stored in the storage location of the corresponding instruction type. Subsequently, the process proceeds to step S360, and it is determined whether or not all the instruction codes corresponding to the instruction type are stored in the instruction code temporary storage unit 305. If not, go to the step S310. It returns and enters the voice input waiting state again. On the other hand, if all the codes are stored, the process proceeds to step S370, and the space data corresponding to the stored instruction code and the instruction amount corresponding to the instruction amount storage unit 306 is displayed. Will be updated.

For example, if the experiencer M inputs voices "front", "slowly", "slightly" and "how", steps S310 to S360 are repeated, and the instruction code temporary storage section 305 is repeated. As shown in FIG. 28, since the instruction code is stored, and all the codes are stored in this state, the goggle-type stereoscopic display device 101 displays "Before, Spatial data when moving a distance m1 "at a speed v1 is displayed.

As described above, according to the simulated experience device of the third embodiment, the experienced person can perform the simulated experience without moving around, for example, while sitting on a chair.

[0063]

As described above, according to the simulated experience device according to the invention of claim 1, when the sight line of the experience person is directed in the body direction of the virtual subject, the shape data of the virtual subject imitating the body shape of the experience person is displayed. Since the data is superimposed on the spatial data and displayed on the stereoscopic display device, a higher immersive feeling can be visually obtained. Further, according to the simulated experience device according to the invention of claim 2, the line of sight of the experience person is virtual. When directed toward the body of the subject, the shape data selected from the shape data of the virtual subject imitating a plurality of body shapes such as the type of animal and male and female are superimposed on the spatial data. Since it is displayed on the stereoscopic display device, the same effect as the effect of the simulated experience device according to the first aspect can be obtained.

According to the simulated experience device according to the third aspect of the present invention, when the sight line of the experience person is directed toward the body of the virtual subject, the posture data corresponding to the head height of the experience person at that time. Is displayed on the stereoscopic display device while being superimposed on the spatial data, the same effect as that of the simulated experience device according to the first aspect can be obtained. Further, according to the simulated experience device according to the invention of claim 4, when the line of sight of the experience person is directed toward the body of the virtual subject during movement, a series of movement motion data is superimposed on the spatial data. Since it is displayed on the stereoscopic display device, the same effect as the effect of the simulated experience device according to claim 1 can be obtained.

According to the simulated experience device of the fifth aspect of the present invention, the reproduced sound data generated from the dry source corresponding to each of the plurality of sound source positions is converted into sound and output to the sound field reproducing device. As a result, a higher level of immersive feeling can be obtained auditorily. According to the simulated experience device according to the invention of claim 6, when the position of a part of the body of the experience person and the position of the surface of the virtual object match, a dry source corresponding to the matched virtual object.
Since the reproduced sound data generated from the audio data is converted into voice and output to the sound field reproducing device, a higher immersive feeling can be obtained auditorily.

According to the simulated experience device according to the invention of claim 7, the reproduced sound data generated from the sound inputted from the sound input means is converted into the sound and outputted to the sound field reproducing device. Higher immersive feeling is obtained. Also,
According to the simulated experience device according to the invention of claim 8, when the experience person gives a voice instruction to face the body of the virtual subject, the shape data of the virtual subject that imitates the body shape of the experience person is spatial data. -Because it is displayed on the stereoscopic display device in a superimposed manner,
The same effect as the effect of the simulated experience device according to the first aspect is obtained, and further, the effect that the experienced person can perform the simulated experience without moving around, for example, while sitting on a chair.

[Brief description of drawings]

FIG. 1 is a diagram showing an overall configuration of a simulated experience device according to a first embodiment.

FIG. 2 is a diagram showing a main body selection menu in the first embodiment.

FIG. 3 is a diagram showing the external shape of the hand-held input device according to the first and second embodiments.

FIG. 4 is a diagram showing the contents of a stationary stationary shape data storage unit in the first embodiment.

FIG. 5 is a diagram showing the contents of a main motion shape data storage unit in the first embodiment.

FIG. 6 is a diagram showing the contents of a main body shape temporary storage unit in the first embodiment.

FIG. 7 is a flowchart showing a processing procedure of the present apparatus in the first embodiment.

FIG. 8 is a diagram showing contents of a storage state of a main body shape temporary storage unit in the first embodiment.

FIG. 9 is a diagram showing an example of a screen display of the goggle type stereoscopic display device in the first embodiment.

FIG. 10 is a diagram showing an example of a screen display of the goggle type stereoscopic display device in the first embodiment.

FIG. 11 is a diagram showing an example of a screen display of the goggle type stereoscopic display device in the first embodiment.

FIG. 12 is a diagram showing an overall configuration of a simulated experience device according to a second embodiment.

FIG. 13 is a diagram showing a sound source selection menu in the second embodiment.

FIG. 14 is a diagram showing the contents of a contact area storage unit in the second embodiment.

FIG. 15 is a diagram showing the contents of an impact sound dry source storage unit in the second embodiment.

FIG. 16 is a diagram showing the contents of a continuous sound band response storage area in the second embodiment.

FIG. 17 is a diagram showing the contents of an impact sound band response storage area in the second embodiment.

FIG. 18 is a diagram showing the contents of a bandwidth response storage area for experience voice in the second embodiment.

FIG. 19 is a flowchart showing a processing procedure of the present apparatus in the second embodiment.

FIG. 20 is a flowchart showing a processing procedure of the present apparatus in the second embodiment.

FIG. 21 is a flowchart showing a processing procedure of the present apparatus in the second embodiment.

FIG. 22 is a flowchart showing a processing procedure of the present apparatus in the second embodiment.

FIG. 23 is a diagram showing an overall configuration of a simulated experience device according to a third embodiment.

FIG. 24 is a diagram showing the contents of a reference voice pattern storage unit 304 in the third embodiment.

FIG. 25 is a diagram showing the contents of an instruction code temporary storage unit 305 in the third embodiment.

FIG. 26 is a diagram showing the contents of an instruction amount storage unit 306 in the third embodiment.

FIG. 27 is a flowchart showing a processing procedure of the present device in the third embodiment.

FIG. 28 is a diagram showing an example of a storage state of an instruction code temporary storage unit 305 in the third embodiment.

[Explanation of symbols]

 107 Spatial data storage unit 108 Main stationary shape data storage unit 109 Main motion shape data storage unit 110 Main shape temporary storage unit 210 Contact area storage unit 211 Continuous sound source device 212 Impact sound dry source storage unit 214 Band response storage unit 215 Head related transfer function storage unit 303 Input voice pattern temporary storage unit 304 Reference voice pattern storage unit

 ─────────────────────────────────────────────────── ─── Continuation of front page (72) Inventor Hisashi Kodama 1006 Kadoma, Kadoma City, Osaka Prefecture Matsushita Electric Industrial Co., Ltd.

Claims (8)

[Claims] 1. A stereoscopic display device attached to a head of a user to be used, spatial data storage means for storing three-dimensional data of a virtual space to be displayed by the stereoscopic display device, and an experience. Detection means for detecting the position and orientation of the person's head; first display control means for selecting predetermined spatial data based on the viewpoint position of the experience person determined from the detection result and displaying it on the display device; The subject shape data storage means for storing the shape data of the virtual subject imitating the experience subject's body shape, and, according to the detection result of the detection means, when the sight line of the experience person is directed to the body direction of the virtual subject, A simulated experience device comprising: second display control means for reading the shape data from the main shape data storage means and superimposing the spatial data on the stereoscopic display device. 2. The main body shape data storage means also stores shape data of a virtual main body imitating a plurality of body shapes such as animal types and sexes, and the main body shape data storage means. The second display control means includes a selection means for selecting one of a plurality of types of shape data stored in the means, and the second display control means determines that the line of sight of the experience person is the virtual subject. The main shape data selected by the selecting means is read from the main shape data storing means, and is superimposed on the space data and displayed on the stereoscopic display device. The simulated experience device according to claim 1. 3. The main body shape data storage means stores a plurality of posture data representing a posture state determined by a head height of an experience person for each shape data type. According to the detection result of the detection means, the second display control means, when the sight line of the experience person is directed toward the body of the virtual subject, the posture data corresponding to the head height at that time is used as the subject shape data. The simulated experience device according to claim 1 or 2, wherein the simulated experience device is read out from a storage means, and is superimposed on the spatial data and displayed on a stereoscopic display device. 4. The main body shape data storage means stores a series of moving motion data that changes with a moving motion for each shape data type, and in the space of the experience person. According to the detection result of the detection means of the moving speed of, the sight line of the experience person is directed to the body direction of the virtual subject, according to the movement speed of the experience person detected by the speed detection means. 3. A third display control means for sequentially superimposing the movement data on the spatial data and displaying the spatial movement data on a stereoscopic display device. Simulated experience device. 5. A stereoscopic display device attached to a head of a user for use, spatial data storage means for storing three-dimensional data of a virtual space to be displayed by the stereoscopic display device, and an experience. Detection means for detecting the position and orientation of the person's head, display control means for selecting and displaying predetermined spatial data on the display device based on the viewpoint position of the experience person determined from the detection result, and A sound field reproducing device attached to the head and used, and each frequency in an eco-time pattern for each frequency band when a plurality of positions in the space are sound source positions and arbitrary positions are sound receiving points. Band response storage means for storing a response for each frequency band obtained by convolving band-pass filters for each band, head-related transfer function storage means for storing a head-related transfer function, and acoustic data of anechoic sound. , Which corresponds to the sound source position A dry source storage means for storing a plurality of sources, and the position of the head of the experience person, which is determined from the detection result of the detection means, is set as a sound receiving point in the space for each sound source, and corresponds to the position. The response for each frequency band of each sound source is calculated by convolving the dry source and the head related transfer function to the band response, and after the responses for each frequency band of each sound source are combined for each same frequency band, A simulated experience device, wherein reproduced sound data is generated by synthesizing over a frequency band, the reproduced sound data is converted into a sound, and the sound is output to the sound field reproducing device. 6. A stereoscopic display device attached to a head of a user to be used, spatial data storage means for storing three-dimensional data of a virtual space to be displayed by the stereoscopic display device, and an experience. First detection means for detecting the position and orientation of the person's head; display control means for selecting predetermined space data based on the viewpoint position of the experience person determined from the detection result and displaying it on the display device; A sound field playback device that is used by being attached to the user's head and a sound source position at a predetermined position on the surface of a virtual object such as a wall, furniture, or window in the background of the display target space, and an arbitrary position as a sound receiving point And the head-related transfer function, and the band response storage means for storing the response for each frequency band obtained by convolving the band-pass filter for each frequency band into the eco-time pattern for each frequency band. Head-related transfer function storage means Dry sound storage means for storing a plurality of dry sources corresponding to the virtual object, and second detection means for detecting the position of a part of the body of the experiencer. When the experience person moves in the virtual space so that the position data detected by the second detection means and the position data of the virtual object coincide with each other, the first detection means is provided. Response for each frequency band by convolving the dry source and the head related transfer function corresponding to the virtual object whose position data coincides with the band response corresponding to the position of the head of the experience person determined from the detection result of Is calculated, reproduced sound data is generated by synthesizing responses for each frequency band over all frequency bands, and the reproduced sound data is converted into voice and output to the sound field reproducing device. Characteristic simulated experience device. 7. A stereoscopic display device attached to a head of a user for use, spatial data storage means for storing three-dimensional data of a virtual space to be displayed by the stereoscopic display device, and an experience. Detection means for detecting the position and orientation of the person's head, display control means for selecting and displaying predetermined spatial data on the display device based on the viewpoint position of the experience person determined from the detection result, and A sound field reproduction device attached to the head and used, a voice input means for inputting a sound near the mouth of the experience person, and a mouth of the experience person as a sound source position in the space of the experience person Band response storage means for storing the response for each frequency band obtained by convolving the bandpass filter for each frequency band into the eco-time pattern for each frequency band when the neighborhood is the sound receiving point, and Head transfer storing partial transfer functions A number storage unit, and convolves the voice input from the voice input unit and the head related transfer function into the band response corresponding to the position of the head of the experience person determined from the detection result of the detection unit, The response for each frequency is calculated, and the reproduced sound data is generated by synthesizing the response for each frequency band over the entire frequency band, and the reproduced sound data is converted into voice and output to the sound field reproducing device. A simulated experience device that is characterized by: 8. A reference voice pattern storage unit for storing a plurality of voice waveforms and instruction information corresponding to each of the plurality of voice waveforms, wherein the detection means stores a waveform of a voice uttered by a user and a reference voice pattern storage. The simulated experience device according to claim 1, wherein the simulated experience device is a detection unit that detects the instruction information from the stored content of the unit.