JP2020020966A - Virtual space and sound providing system - Google Patents

Abstract

To provide a system which can provide a sense of visual immersion by an image projected on a HMD, and can further enhance immersive feeling by auditory stimulation by outputting sound corresponding to this image.SOLUTION: The system includes an information processing device 1, a head-mounted display 2 for incorporating and projecting video data sent from this information processing device 1 into video data in a virtual space 20, an additional position sensor 3 which a user 8 wearing the head-mounted display 2 carries for making capable of acquiring three-dimensional coordinates and sends a signal of position information to the information processing device 1, a video/audio output device 4 which outputs video data to the information processing device 1, a first speaker 5 for outputting sound from this video/audio output device 4, and a second speaker 6 for outputting sound recorded in the information processing device 1. The head-mounted display 2 includes a head position sensor for acquiring three-dimensional coordinates and a camera angle.SELECTED DRAWING: Figure 1

Description

  The present invention relates to a virtual space and sound providing system using a head mounted display.

  Conventionally, a head-mounted display (hereinafter, referred to as “HMD”) is mounted as shown in Patent Document 1, and a video such as a game or a movie is projected on the HMD, thereby enhancing the immersion feeling for the wearer. A system aiming at improving the entertainment property of a computer has already been known.

JP 2013-258614 A

  However, since the system as in Patent Document 1 deals only with images, it does not cover other senses such as hearing, so that the wearer can enjoy a sense of immersion only with vision. However, its use is limited, and there is also a limit on the improvement of entertainment properties.

  Accordingly, the present invention is intended to solve the above-described problems, and it is possible to enjoy a visual immersion feeling by an image projected on the HMD, and to output a sound corresponding to the image to thereby achieve an auditory sense. It is an object of the present invention to obtain a system that can further enhance the immersion feeling by the stimulus from the user.

  In order to solve the above-described problems, the present invention is directed to an information processing apparatus, an HMD for incorporating video data transmitted from the information processing apparatus into video data in a virtual space, and a user carrying the HMD to carry the information. An additional position sensor that sends a signal of position information to the information processing device so that dimensional coordinates can be acquired, a video / audio output device that outputs video data to the information processing device, and a video / audio output device that outputs audio from the video / audio output device The HMD includes one speaker and a second speaker that outputs sound recorded in the information processing device, and the HMD includes a head position sensor that acquires three-dimensional coordinates and a camera angle.

  Further, the information processing device includes a video acquisition unit that acquires video data from the video / audio output device, a storage unit that stores video data in a virtual space and audio data that accompanies the video data, and a virtual storage that is stored in the storage unit. A video synthesizing unit that incorporates video data obtained by the video obtaining unit into space, a position information obtaining unit that obtains position information from the HMD and the additional position sensor, and a position information obtained by the position information obtaining unit is used as the virtual information. A video / audio processing unit for reflecting the video and audio in the space, a video output unit for outputting the information obtained by the video / audio processing unit and the video synthesis unit to the HMD, and a second speaker for outputting the voice stored in the storage unit And an audio output unit that outputs the audio signal to the computer.

  Further, the video and audio output device may include a vibration device that generates vibration under the feet of the user by receiving the audio data from the video and audio output device. By providing the vibration device in this manner, not only sight and hearing but also a bodily sensation felt by the vibration can be obtained, so that the immersion feeling can be further enhanced.

  Further, the additional position sensor may be mounted on a microphone used by the user. By attaching the additional position sensor to the microphone in this way, the additional position sensor responds to the user's operation of raising and lowering the microphone, so that the user can arbitrarily change the video and audio of the HMD.

  Further, the first speakers may be arranged corresponding to the arrangement of the speakers in the virtual space. In this way, by actually arranging the speakers corresponding to the arrangement in the virtual space, the user can feel as if he or she is in the virtual space, and it is possible to further enhance the sense of reality. Become.

  Since the present invention is configured as described above, the user wearing the HMD and the additional position sensor can actually experience the virtual space with a sense of reality by actually moving the HMD and the additional position sensor. In addition, by outputting sound with the first speaker and the second speaker, it is possible to receive a stimulus from the auditory sense, so that it is possible to further enhance the immersive feeling and improve the entertainment property.

FIG. 1 is a block diagram showing a first embodiment of the present invention. FIG. 3 is a conceptual diagram illustrating a user when the user wears the HMD in the first embodiment. The conceptual diagram which shows the karaoke microphone provided with the additional position sensor. The conceptual diagram which shows the karaoke room of a present Example. The flowchart of a drawing operation. FIG. 3 is a conceptual diagram showing a virtual space. Connection diagram regarding speakers and amplifier. FIG. 4 is a flowchart of a speaker and an amplifier. The flowchart of an additional position sensor.

  First Embodiment A first embodiment of the present invention will be described with reference to FIGS. The system according to the present embodiment aims to enhance the entertainment performance when the user (8) enjoys karaoke in the karaoke room (13). As shown in FIG. 1, the system according to the present embodiment includes an information processing device (1), a head mounted display (hereinafter, referred to as “HMD”) (1), an additional position sensor (3), and a video / audio output. The apparatus includes a device (4), first and second speakers (5) and (6), and a vibration device (7).

  The HMD (2) is used by being mounted on the head of a user (8) who uses this system, and has a camera (10) at the front center as shown in FIG. The HMD (2) further includes a head position sensor (not shown) for acquiring three-dimensional coordinates and a camera angle. As shown in FIG. 3, an additional position sensor (3) capable of acquiring three-dimensional coordinates is connected and fixed to the karaoke microphone (11) held by the user (8).

  In the karaoke room (13) of the present embodiment, as shown in FIG. 4, a vibration device (7) is arranged on the floor of a stage provided behind the room, and a karaoke room is provided behind the vibration device (7). A large-screen display (15) that can be viewed by anyone other than the user (8) in the karaoke room (13) in the room (13) is provided. A pair of first speakers (5) are arranged on both sides of the display (15). A handrail (16) is arranged in front of the vibration device (7), and a second speaker (6) is arranged further in front of the handrail (16). Further, a projector (17) is arranged further ahead of the second speaker (6), and an image output from the projector (17) is projected on a wall (18) located further forward than the projector (17). It is possible.

  Next, the information processing apparatus (1) according to the present embodiment will be described. As shown in FIG. 1, the information processing apparatus (1) includes a video acquisition section, a storage section, a video / audio processing section, a video synthesis section, a position information acquisition section. , A video output unit and an audio output unit.

  The video acquisition unit acquires video data from the video / audio output device (4). The storage unit stores video data of the virtual space (20) projected on the HMD (2) and audio data accompanying the video data. The position information acquisition unit acquires position information from the HMD (2) and the additional position sensor (3). The video / audio processing unit reflects the information obtained by the position information obtaining unit in the virtual space (20).

  The video synthesizing unit incorporates the video data obtained by the video obtaining unit into the virtual space (20) processed by the video / audio processing unit. The video output unit outputs information obtained by the video synthesis unit to the HMD (2), the display (15), and the projector (17). The audio output unit outputs the audio information in the storage unit to the second speaker (6) via the second amplifier (30).

  Further, as described above, in the karaoke room (13) of the present embodiment, as shown in FIG. 4, the vibration device (7) is disposed behind the room. As shown in FIG. 1, the vibration device (7) is connected to the video / audio output device (4) of the present embodiment, and as shown in FIGS. The vibration is transmitted through a third amplifier (27) for the device (7), and the vibration device (7) generates vibration under the feet of the user (8).

  Also, as shown in FIG. 6, the virtual space (20) stored in the storage unit is made the same space as the space in the karaoke room (13) shown in FIG. Is increasing. That is, in the virtual space (20), the position corresponding to the vibration device (7) of the karaoke room (13) in FIG. 4 is set as the stage (14), and the virtual handrail (16) is positioned at the position of the handrail (16) of the karaoke room (13). 22) is displayed. Further, in front of the virtual handrail (22), a video display unit (23) for displaying a video from the video / audio output device (4) as it is is arranged. In order to further enhance the presence of the stage (14), a plurality of staffs (24) are provided in front of the video display section (23), and an audience (25) is further provided in front of the staff (24). it's shown. Further, a pair of rear speakers (26) are displayed at positions corresponding to the positions of the first speakers (5) in the karaoke room (13) behind the stage (14).

  Next, video processing in the present embodiment will be described with reference to FIGS. First, as shown in FIG. 1, video data of the video / audio output device (4) is output to the information processing device (1) via the capture unit (28). In this embodiment, a karaoke device is used as the video / audio output device (4). The video / audio output device (4) of the present invention is not limited to this, and a game machine, a television, or the like can be used.

  Further, the position information acquisition unit of the information processing device (1) acquires the position information of the head position sensor and the additional position sensor (3) of the HMD (2), and the position information is acquired by the video / audio processing unit. This is reflected on the video data of the virtual space (20) stored in the storage unit in advance. Further, in the video synthesizing unit, as shown in FIG. 1, the video data from the video / audio output device (4) sent via the capture unit (28) is added to the virtual space (20) processed by the video / audio processing unit. ) Is synthesized. Then, the video data processed by the video synthesizing unit is output to the HMD (2), the display (15), and the projector (17) as shown in FIG.

  The video data sent to the HMD (2), the display (15), and the projector (17) are displayed on the screens of the HMD (2) and the display (15), and at the projection destination of the projector (17). You. The HMD (2) and the display (15) output the same video, but the projector (17) outputs a video from the audience (25) side of the virtual space (20).

  In the flowchart of FIG. 5, "START" means the time when the information processing apparatus (1) is started. The video and audio processed in the drawing processing flow are from the virtual space (20) and are stored in the storage unit of the information processing device (1) as needed. The “update of additional position sensor input”, “update of HMD input”, and “update of capture board input” performed in “multi-thread processing” are performed by the video acquisition unit of the information processing device (1). Further, “update of information such as coordinates” is performed by the position information acquisition unit. Further, the “object operation determination” for determining whether each element arranged in the virtual space (20) is operating and the “object operation processing” are performed by the video / audio processing unit. In addition, “drawing process start point”, “HMD drawing”, and “display and projector drawing” for performing drawing in accordance with the operation in the virtual space (20) are performed in the video synthesizing unit and the video output unit. "Sound reproduction and fade processing" is performed in the audio output unit.

  Next, the audio processing in the present embodiment will be described with reference to FIGS. 7 and 8. First, the information processing apparatus (1) transmits the audio information stored in the storage unit of the information processing apparatus (1). The information is sent to the audio output unit of the information processing device (1) shown in FIG. 1, and is output from the audio output unit to the second speaker (6) through the second amplifier (30). The audio of the video / audio output device (4) is output to a pair of left and right first speakers (5) via the first amplifier (31).

  The sound flow of the additional position sensor (3) fixed to the karaoke microphone (11) shown in FIG. 3 will be described. As shown in FIG. 1, the position information of the additional position sensor (3) is transmitted to the information processing device (1). Is transmitted to the video / audio processing unit via the position information obtaining unit. The video and audio processing unit updates information and the like as shown in the flowchart of FIG. 9, and reproduces audio through the second speaker (6) under certain conditions.

  The case where the virtual space (20) and the voice providing system of the present embodiment configured as described above are used will be described below. As shown in FIG. 2, the user (8) wears the HMD (2) on the head and holds the microphone provided with the additional position sensor (3). Then, the video and audio data is output from the video and audio output device (4) to the information processing device (1) by streaming the karaoke video and audio, and the audio data is transmitted to the vibration device (7) and the first speaker. Output to (5).

  As a result, sound is output from the first speaker (5) into the karaoke room (13), and vibration is generated at the feet of the user (8) in accordance with the sound from the vibration device (7). The video data is acquired by a video acquisition unit in the information processing device (1), and is incorporated into a video display unit (23) of the virtual space (20) by a video synthesis unit. Then, the video data synthesized by the video synthesis unit is output from the video output unit to the display (15) and the projector (17) installed in the HMD (2) and the karaoke room (13). The audio data stored in the storage unit in the information processing device (1) is output from the audio output unit to the second speaker (6). That is, regarding the audio, the audio of the video / audio output device (4) is output from the first speaker (5), and the audio of the virtual space (20) is output from the second speaker (6).

  When the user (8) moves his / her head, position information from a head position sensor in the HMD (2) attached to the head is sent to the position information acquisition unit of the information processing device (1). Then, the position information obtained by the position information obtaining unit is reflected on the video and audio of the virtual space (20) by the video and audio processing unit of the information processing device (1). Then, the video data and the audio data processed by the video / audio processing unit are sent again from the video output unit and the audio output unit to the HMD (2) and the second speaker (6) so that the user (8) can view the data. Become.

  The video data processed as described above is sent from the video output unit to the display (15) and the projector (17), and is displayed on the display (15) and the projector (17). People other than the user (8) existing in (13) can also view the same video as the HMD (2). The display (15) displays the same image as the HMD (2), but the projector (17) displays an image when the stage (14) is viewed from the audience side of the virtual space (20). It is configured to be.

  When the user (8) moves the karaoke microphone (11), a signal from the additional position sensor (3) provided on the karaoke microphone (11) obtains the position information of the information processing device (1). Unit, and reflects the position information obtained by the video and audio processing unit of the information processing device (1) on the video and audio of the virtual space (20), and converts the video data through the video output unit and the audio output unit. The sound is sent to the HMD (2), the display (15), and the projector (17), and the sound is output to the second speaker (6). Therefore, by intentionally moving the microphone, the video of the virtual space (20) can be changed, and the sound can be changed to further enhance the entertainment property.

  Referring to the flowchart of this additional position sensor (3) shown in FIG. 9, "START" means when the information processing apparatus (1) is started. Then, the signal from the additional position sensor (3) is transmitted to the position information acquisition unit of the information processing device (1), and “update of the additional position sensor input” is performed. Information such as the position and angle of the karaoke microphone (11) to which the additional position sensor (3) is assembled is detected.

  When the position of the karaoke microphone (11) is higher than the specified value, "sound reproduction" is performed, and when the position is smaller than the specified value, "fade-out processing" is performed. In the present embodiment, when the user (8) raises the karaoke microphone (11) higher than 160 cm from the floor, a cheering sound flows from the second speaker (6), while the sound rises from 160 cm from the floor. If it is lowered, the sound from the second speaker (6) is muted.

1 Information processing device 2 Head mounted display (HMD)
3 Additional position sensor 4 Video / audio output device 5 First speaker 6 Second speaker 7 Vibration device 8 User 11 Microphone for karaoke 20 Virtual space

Claims (5)

  An information processing device, a head-mounted display for incorporating video data sent from the information processing device into video data in a virtual space, and a user carrying the head-mounted display to carry the information so that three-dimensional coordinates can be acquired. An additional position sensor for sending a position information signal to the processing device, a video / audio output device for outputting video data to the information processing device, a first speaker for outputting audio from the video / audio output device, and the information processing device And a second speaker for outputting a voice recorded in the virtual space, and the head mounted display includes a head position sensor for obtaining three-dimensional coordinates and a camera angle.   The information processing apparatus includes: a video acquisition unit that acquires video data from the video / audio output device; a storage unit that stores video data of a virtual space and audio data accompanying the video data; and a virtual space that is stored in the storage unit. A video synthesizing unit that incorporates video data obtained by the video obtaining unit, a position information obtaining unit that obtains position information from a head-mounted display and an additional position sensor, and the position information obtained by the position information obtaining unit. A video / audio processing unit that reflects the video and audio in the virtual space; a video output unit that outputs information obtained by the video / audio processing unit and the video synthesis unit to a head-mounted display; and a voice stored in the storage unit. A virtual space and sound providing system, comprising: a sound output unit that outputs the sound to a second speaker.   2. The virtual space and audio providing system according to claim 1, wherein the video / audio output device includes a vibration device that generates vibration under the user's feet by receiving audio data from the video / audio output device. .   The virtual space and voice providing system according to claim 1, wherein the additional position sensor is mounted on a microphone used by the user.   2. The virtual space and sound providing system according to claim 1, wherein the first speaker is arranged corresponding to a speaker arrangement in the virtual space.