WO2018174500A1

WO2018174500A1 - System and program for implementing augmented reality three-dimensional sound reflecting real-life sound

Info

Publication number: WO2018174500A1
Application number: PCT/KR2018/003189
Authority: WO
Inventors: 이승학
Original assignee: 주식회사 라이커스게임
Priority date: 2017-03-20
Filing date: 2018-03-19
Publication date: 2018-09-27

Abstract

Provided is an augmented reality sound implementation system for executing an augmented reality sound implementation method. The system includes: a first computing device of a first user; and a first sound device which is worn by the first user so that the first user can receive a three-dimensional augmented reality sound, is connected to the first computing device in a wired or wireless manner, and includes a sound recording function. The method includes: a step in which the first sound device acquires real-life sound information which indicates a real-life sound, and transmits the real-life sound information to the first computing device; a step in which the first computing device acquires a first virtual sound which indicates a sound generated from a virtual reality game executed in the first computing device; a step in which the first computing device generates a three-dimensional augmented reality sound on the basis of the real-life sound information and the first virtual sound; and a step in which the first computing device provides the three-dimensional augmented reality sound to the first user through the first sound device.

Description

Augmented Reality 3D Sound Implementation System and Program Reflecting Real Sound

The present invention relates to an augmented reality three-dimensional sound implementation system and program that reflects reality sound.

Augmented reality refers to computer graphics technology that blends real world images and virtual images that users see with one image. Augmented reality synthesizes an image relating to a virtual object or information to a specific object of the real world image.

Three-dimensional sound refers to a technology that provides three-dimensional sound to feel a sense of reality. In the field of virtual reality, 3D sound is implemented by providing sound along a path transmitted from a location of sound generation to a user using vector values of a virtual reality image.

However, in augmented reality, since it is impossible to grasp the direction of the real sound in advance, it is difficult to apply the 3D sound realization technology of the virtual reality field as it is.

For example, in a virtual reality, when a plurality of users are inside a building, a sound is ringing, so there is a problem in that the location of another user cannot be predicted intuitively.

Therefore, in the field of augmented reality, there is a need for a method and program capable of realizing 3D sound by reflecting real sound in real time.

The problem to be solved by the present invention is to provide a system and a program that can implement a three-dimensional augmented reality sound reflecting the real sound in real time.

Problems to be solved by the present invention are not limited to the above-mentioned problems, and other problems not mentioned will be clearly understood by those skilled in the art from the following description.

An augmented reality sound implementation system according to an embodiment of the present invention for solving the above problems, the first computing device of the first user; And a first sound device that is worn by the first user so that the first user can be provided with 3D augmented reality sound, and is wired or wirelessly connected to the first computing device, and includes a sound recording function. The first sound device obtaining real sound information indicating real sound and transmitting the real sound information to the first computing device; Acquiring, by the first computing device, a first virtual sound indicative of a sound generated in a virtual reality game played on the first computing device; Generating, by the first computing device, a 3D augmented reality sound based on the reality sound information and the first virtual sound; And providing, by the first computing device, the three-dimensional augmented reality sound to the first user through the first acoustic device.

According to the present invention, since the three-dimensional augmented reality sound is provided by an appropriate method of binaural method or positioning method according to the distance between users using the augmented reality game 3 reflects the real sound and virtual sound in real time more realistically The augmented reality sound of the dimension can be realized.

In addition, when a difference occurs in the position or movement of the user's position using the augmented reality game and the user's avatar, it is possible to implement a three-dimensional augmented reality sound in consideration of the difference.

Effects of the present invention are not limited to the effects mentioned above, and other effects not mentioned will be clearly understood by those skilled in the art from the following description.

1 is a schematic conceptual view for explaining a method of implementing augmented reality sound.

2 is a block diagram illustrating an apparatus for implementing augmented reality sound.

3 is a flowchart illustrating a first embodiment of a method for implementing augmented reality sound.

4 is a flowchart illustrating a second embodiment of a method of implementing augmented reality sound.

An augmented reality sound implementation system according to an embodiment of the present invention for solving the above problems, the first computing device of the first user; And a first sound device that is worn by the first user so that the first user can be provided with 3D augmented reality sound, and is wired or wirelessly connected to the first computing device and includes an audio recording function. The first sound device obtaining real sound information indicating real sound and transmitting the real sound information to the first computing device; Acquiring, by the first computing device, a first virtual sound indicative of a sound generated in a virtual reality game played on the first computing device; Generating, by the first computing device, a 3D augmented reality sound based on the reality sound information and the first virtual sound; And providing, by the first computing device, the three-dimensional augmented reality sound to the first user through the first acoustic device.

The first computing device may include obtaining direction characteristic information indicating a location of generation of the real sound; And the first computing device further considering the direction characteristic information in generating the 3D augmented reality sound.

Here, the first computing device determines whether the first user and the second user spaced apart from the first user are closer than a preset distance, and when the first user and the second user are closer than the preset distance, The direction characteristic information of the reality sound may be obtained based on the reality sound information, and the reality sound information may be a binaural type measured by a plurality of microphones in the first sound device.

Here, the first computing device determines whether the first user and the second user spaced apart from the first user are closer than a preset distance, and the first user and the second user are not closer than the preset distance. In this case, the direction characteristic information of the real sound may be obtained based on the location information of the first user and the second user.

Here, the first computing device is a relative position of the first user and the second user spaced apart from the real space and the avatar of the virtual space of the first user and the second user When a position difference that does not correspond to a position occurs, the 3D augmented reality sound may be generated based on the position difference.

Here, the position difference may be a case where the second user uses a skill to the first user, and the second user and the avatar of the second user are separated.

Here, the position difference may be a case where the second user uses a skill to the first user, and the movement of the avatar of the second user is greater or shorter than the movement of the second user.

The 3D augmented reality sound may be generated through blending of the second virtual sound and the reality sound generated to correspond to the position of the avatar.

Here, the computer-readable medium may include a program recording a program for executing the augmented reality sound implementation method performed by the augmented reality sound implementation system described above.

Here, an application for a terminal device stored in a medium may be included in order to execute the augmented reality sound implementation method performed by the augmented reality sound implementation system described above in combination with the computing device that is hardware.

Other specific details of the invention are included in the detailed description and drawings.

Advantages and features of the present invention and methods for achieving them will be apparent with reference to the embodiments described below in detail with the accompanying drawings. However, the present invention is not limited to the embodiments disclosed below, but can be embodied in various different forms, and the present embodiments only make the disclosure of the present invention complete, and those of ordinary skill in the art to which the present invention belongs. It is provided to fully inform the skilled worker of the scope of the invention, which is defined only by the scope of the claims.

The terminology used herein is for the purpose of describing particular embodiments only and is not intended to be limiting of the invention. In this specification, the singular also includes the plural unless specifically stated otherwise in the phrase. As used herein, "comprises" and / or "comprising" does not exclude the presence or addition of one or more other components in addition to the mentioned components. Like reference numerals refer to like elements throughout, and "and / or" includes each and all combinations of one or more of the mentioned components. Although "first", "second", etc. are used to describe various components, these components are of course not limited by these terms. These terms are only used to distinguish one component from another. Therefore, of course, the first component mentioned below may be a second component within the technical spirit of the present invention.

Unless otherwise defined, all terms used in the present specification (including technical and scientific terms) may be used in a sense that can be commonly understood by those skilled in the art. In addition, terms that are defined in a commonly used dictionary are not ideally or excessively interpreted unless they are specifically defined clearly.

Hereinafter, exemplary embodiments of the present invention will be described in detail with reference to the accompanying drawings.

An augmented reality 3D sound implementation method reflecting reality sound according to an embodiment of the present invention is realized by the computing device 200. The augmented reality sound implementation method may be implemented as an application, stored in the computing device 200, and performed by the computing device 200.

For example, the computing device 200 may be provided as a mobile device such as a smartphone or a tablet, but is not limited thereto. The computing device 200 may include a camera, output a sound, and process and store data. 200) is enough. That is, the computing device 200 may include a camera and may be provided as a wearable device such as glasses or a band that outputs sound. Any computing device 200 that is not illustrated may be provided.

Referring to FIG. 1, a plurality of

users

10 and 20 carry sound devices 100-1 and 100-2 and computing devices 200-1 and 200-2 to experience augmented reality content. For example, although only two

users

10 and 20 are illustrated, the present invention is not limited thereto, and the method of implementing augmented reality sound may be substantially the same for two or more user environments.

For example, the acoustic device 100 may be provided in the form of headphones, a headset, earphones, and the like. The sound device 100 may not only output a sound including a speaker, but also obtain and record a surrounding sound including a microphone. The acoustic device 100 may be provided in a binaural type to increase the sense of reality. By using the binaural effect, by recording the left-right sound separately, the sound including the direction characteristic information may be obtained. In some embodiments, the sound device 100 may be provided as a separate device as the sound output device and the sound recording device.

The sound device 100 may obtain reality sound information generated by the

users

10 and 20. Alternatively, the acoustic apparatus 100 may obtain reality acoustic information generated around the

users

10 and 20. That is, the sound source may be a location where the real sound occurs. The sound source is not limited to sound generated by the plurality of

users

10 and 20.

In this case, the real acoustic information may indicate the acoustic information actually occurring in the reality. For example, when the second user 20 yells to the first user 10 while playing an augmented reality game, the first sound device 100-1 of the first user 10 may transmit the second user 20. Real sound information (sound) generated from the Here, the second user 20 may be a user located at a position spaced apart from the first user 10.

The acoustic device 100-1 of the first user 10 may also acquire direction characteristic information of the real sound generated by the second user 20. The first computing device 200-1 of the first user 10 may store the virtual sound information and the virtual sound information of the first user 10 based on the direction characteristic information of the real sound obtained from the first sound device 100-1. The 3D augmented reality sound for the first user 10 may be generated by synthesizing the first virtual sound information indicating the sound generated in the real game (eg, background sound, effect sound, etc.).

The first computing device 200-1 when the sound device 100 does not support the binaural type of sound or the distance between the first user 10 and the second user 20 is not closer than the preset distance. May obtain direction characteristic information of the real sound based on the relative position information of the first user 10 and the second user 20.

The plurality of computing devices 200-1 and 200-2 or the server may acquire positions of the first user 10 and the second user 20, and generate relative position information by comparing the positions of the first user 10 and the second user 20. . For location acquisition of a plurality of

users

10, 20, any well known positioning system can be used, including, for example, a GPS system. The plurality of computing devices 200-1 and 200-2 or the server acquires a three-dimensional position of the first user 10 and the second user 20, and compares the three-dimensional positions of each other to the relative three-dimensional position. Information can be generated.

For example, as shown in FIG. 1, relative location information indicating that the second user 20 is located at an 8 o'clock direction, a 50m distance, and a 5m low altitude based on the first user 10 may be generated. have. Here, the second user 20 may generate a realistic sound.

Then, the direction characteristic information of the real sound acquired by the first user 10 is determined based on the relative position information. The 3D augmented reality sound for the first user 10 is synthesized by synthesizing the real sound information obtained by the first user 10 from the second user 20, the direction characteristic information of the real sound, and the first virtual sound information. Can be implemented. According to the determined direction characteristic information of the real sound, elements such as amplitude, phase or frequency of the real sound may be adjusted.

In the augmented reality sound implementation method according to an embodiment of the present invention, by using the relative position information of the binaural type sound device 100 or a plurality of users (10, 20), the three-dimensional by reflecting the real sound in real time Can implement the augmented reality sound.

According to an exemplary embodiment, the above-described binaural type sound device 100 and the relative position information of the first user 10 and the second user 20 may be used together.

Referring to FIG. 2, the sound apparatus 100 may include at least one control unit 110, a storage unit 120, an input unit 130, an output unit 140, a transceiver 150, a GPS unit 160, and the like. It may include.

Each component included in the acoustic device 100 may be connected by a bus to communicate with each other.

The controller 110 may execute a program command stored in the storage 120. The controller 110 may refer to a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. have.

The storage unit 120 may be configured of at least one of a volatile storage medium and a nonvolatile storage medium. For example, the storage unit 120 may be configured as at least one of a read only memory (ROM) and a random access memory (RAM).

The input unit 130 may be a recording device capable of recognizing and recording voice. For example, the input unit 130 may be a microphone. The output unit 140 may be an output device capable of outputting voice. The output device may include a speaker or the like.

The transceiver 150 may be connected to the computing device 200 or a server to perform communication. The GPS unit 160 may track the location of the sound device 100.

The computing device 200 may include at least one control unit 210, a storage unit 220, an input unit 230, an output unit 240, a transceiver unit 250, a GPS unit 260, a camera unit 270, and the like. It may include.

Each component included in the computing 200 may be connected by a bus to communicate with each other. The output unit 240 may be an output device capable of outputting a screen. The output device may include a display or the like.

The controller 210 may execute a program command stored in the storage 220. The controller 210 may mean a central processing unit (CPU), a graphics processing unit (GPU), or a dedicated processor on which methods according to embodiments of the present invention are performed. have. The storage unit 220 may be configured of at least one of a volatile storage medium and a nonvolatile storage medium. For example, the storage unit 220 may be configured as at least one of a read only memory (ROM) and a random access memory (RAM).

The transceiver 250 may be connected to another computing device 200, an acoustic device 110, or a server to perform communication. The GPS unit 260 may track the location of the computing device 200. The camera unit 270 may acquire a real image.

In some embodiments, the augmented reality sound implementation method may be realized by linking the computing device with another computing device or server.

Referring to FIG. 3, the first sound device 100-1 of the first user 10 may obtain reality sound information. In this case, the real sound information may be a real sound generated from the second user 20 or a real sound generated around the first user 10.

The first sound device 100-1 may transmit reality sound information to the first computing device 200-1 of the first user 10. The first computing device 200-1 may obtain realistic acoustic information from the first acoustic device 100-1 (S300).

The first computing device 200-1 may determine whether another user (eg, the second user 20) exists at a distance from the first user 10 (S310). Herein, the close distance may be a preset distance.

When the first user 10 and the second user 20 are closer than the preset distance, the first computing device 200-1 may acquire direction characteristic information of the real sound based on the real sound information (S320). ). Here, the real sound information may be binaural type sound information measured by the plurality of input units 130 of the first sound device 100-1.

The first computing device 200-1 may acquire first virtual sound information indicating sound generated in the virtual reality game (S321).

The first computing device 200-1 may generate the 3D augmented reality sound based on at least one of the real sound information, the direction characteristic information, and the first virtual sound information (S322). In detail, the first computing device 200-1 may generate three-dimensional augmented reality sound by blending at least one of real acoustic information, direction characteristic information, and first virtual acoustic information.

For example, if the first user 10 obtains the sound source of the national anthem verse 1, "God bows so that the sea and the Baekdusan dry and wear out in the north direction of the first user 10", the first computing device. The 200-1 blends the direction characteristic information, the real sound information, and the first virtual sound information indicating the sound generated in the virtual reality game so that the sound source may sound as if the sound is generated in the north, thereby generating 3D augmented reality sound. Can be generated.

The first computing device 200-1 may provide 3D augmented reality sound to the first user 10 through the first acoustic device 100-1 (S323).

When the first user 10 and the second user 20 are not closer than the preset distance, the second computing device 200-2 may determine location information of the first user 10 and the second user 20. Can be obtained (S330).

The first computing device 200-1 may acquire direction characteristic information of the real sound based on the location information of the first user 10 and the second user 20 (S331). The first computing device 200-1 may acquire first virtual sound information indicating sound generated in the virtual reality game (S332).

The first computing device 200-1 may generate the 3D augmented reality sound based on at least one of the real sound information, the direction characteristic information, and the first virtual sound information (S333). In detail, the first computing device 200-1 may generate three-dimensional augmented reality sound by blending at least two or more pieces of information among the real sound information, the direction characteristic information, and the first virtual sound information.

For example, if the sound source of the national verse 1, "God bows to dry and wear Donghae and Mt. Baekdu, live in our country," originated from the second user 20, the first computing device 200-1 may be a second user ( 20, the first user 10 located at the right side may generate the 3D augmented reality sound in consideration of the direction characteristic information so that the sound source may sound as if it is generated at the left side.

The first computing device 200-1 may provide a 3D augmented reality sound to the first user 10 through the first acoustic device 100-1 (S334).

Referring to FIG. 4, the first sound device 100-1 of the first user 10 may obtain reality sound information. In this case, the real sound information may be a real sound generated from the first user 10 or a real sound generated around the first user 10.

The first computing device 200-1 has a relative position between a plurality of

users

10 and 20 in a real space and a relative position of avatars of a plurality of

users

10 and 20 in a virtual space in an augmented reality game. It is possible to determine whether a position difference that does not correspond to (S301).

The position difference is a case where the second user 20 uses the skill to the first user 10, and may be a case where the avatars of the second user 20 and the second user 20 are separated. A specific example of the case where the avatar is separated may be as follows. The second user 20 may use the skill with respect to the first user 10. In this case, the avatar of the second user 20 may be separated and moved to the avatar of the first user 10 to use the skill.

In addition, the position difference may be a case where the second user 20 uses a skill and the avatar of the second user 20 moves at a moment. Teleportation is commonly referred to in the game as teleport. Teleportation (or teleportation) can mean moving to a space in an instant. It can usually be used to move very far away.

For example, when the avatar of the second user 20 is located to the west of the avatar of the first user 10 by teleporting while being located east of the avatar of the first user 10, the first computing is performed. The device 200-1 may generate a 3D augmented reality sound in consideration of a position of the avatar of the second user 20 and a position difference of the second user 20.

In addition, the position difference is a case where the second user 20 uses the skill to the first user 10, and the movement of the avatar of the second user 20 is greater than that of the second user 20. Or short.

When the movement of the avatar of the second user 20 is greater or shorter than the movement of the second user 20. For example, when the second user 20 moves rapidly due to the use of the skill of the second user 20. There can be. In this case, the first computing device 200-1 may consider sounds that may occur while the avatar of the second user 20 moves rapidly.

When a location difference occurs, the first computing device 200-1 may acquire location information of the first user 10 and the second user 20 (S302). The first computing device 200-1 may generate a 3D augmented reality sound based on the position difference (S303).

When the location difference occurs, the first computing device 200-1 may generate the 3D augmented reality sound by blending the second virtual sound and the real sound generated to correspond to the positions of the avatars of the plurality of

users

10 and 20. Can be generated. When the first user 10 or the second user 20 uses the skill, the first computing device 200-1 may perform acoustic blending according to the first-person or third-person situation.

For example, while playing a game in a first-person situation, the first user 10 or the second user 20 uses the skill, so that the position of the avatar of the first user 10 or the second user 20 If different, the first computing device 200-1 may generate a virtual sound and blend it with the real sound according to the third-person situation.

The first computing device 200-1 may provide the 3D augmented reality sound to the first user through the first acoustic device 100-1 (S304).

If the location difference does not occur, the first computing device 200-1 may determine whether another user (eg, the second user 20) exists at a distance from the first user 10 (S310). ). Herein, the close distance may be a preset distance.

In the above description of the method for implementing augmented reality sound, the person skilled in the art to which the present invention pertains is not limited to the implementation of the augmented reality sound. It will be appreciated that the same may be implemented for a mixed reality sound implementation including the same.

The steps of a method or algorithm described in connection with an embodiment of the present invention may be implemented directly in hardware, in a software module executed by hardware, or by a combination thereof. The software module may include random access memory (RAM), read only memory (ROM), erasable programmable ROM (EPROM), electrically erasable programmable ROM (EEPROM), flash memory, hard disk, removable disk, CD-ROM, or It may reside in any form of computer readable recording medium well known in the art.

In the above, embodiments of the present invention have been described with reference to the accompanying drawings, but those skilled in the art to which the present invention pertains may implement the present invention in other specific forms without changing the technical spirit or essential features. I can understand that. Therefore, it should be understood that the embodiments described above are exemplary in all respects and not restrictive.

Claims

An augmented reality sound implementation system performing the augmented reality sound implementation method,

A first computing device of a first user; And

And a first sound device that is worn by the first user so that the first user can be provided with 3D augmented reality sound, is connected to the first computing device by wire or wirelessly, and includes a sound recording function. ,

Acquiring, by the first acoustic device, real acoustic information indicating real sound and transmitting the real acoustic information to the first computing device;

Acquiring, by the first computing device, a first virtual sound indicative of a sound generated in a virtual reality game played on the first computing device;

Generating, by the first computing device, a 3D augmented reality sound based on the reality sound information and the first virtual sound; And

And providing, by the first computing device, the three-dimensional augmented reality sound to the first user via the first acoustic device.
The method according to claim 1,

Acquiring, by the first computing device, direction characteristic information indicating a location of generation of the real sound; And

And the first computing device further considering the direction characteristic information in the generation of the 3D augmented reality sound.
The method according to claim 2,

The first computing device determines whether the first user and the second user spaced apart from the first user are closer than a preset distance, and when the first user and the second user are closer than a preset distance, the reality Obtaining direction characteristic information of the real sound based on sound information, wherein the real sound information is a binaural type measured by a plurality of microphones in the first acoustic device.
The method according to claim 2,

The first computing device determines whether the first user and the second user spaced apart from the first user are closer than a preset distance, and when the first user and the second user are not closer than the preset distance, And obtaining direction characteristic information of the real sound based on location information of the first user and the second user.
The method according to claim 1,

The first computing device has a relative position in a real space of the first user and a second user spaced apart from the first user, and a relative position of an avatar of the virtual space of the first user and the second user. If an inconsistent position difference occurs,

And generating the three-dimensional augmented reality sound based on the position difference.
The method according to claim 5,

The location difference is a case where the second user uses the skill to the first user, and the avatar of the second user and the second user are separated.
The method according to claim 5,

The location difference is a case where the second user uses a skill to the first user, and the motion of the avatar of the second user is greater or shorter than the movement of the second user. system.
The method according to claim 5,

And the 3D augmented reality sound is generated through blending the second virtual sound and the real sound generated to correspond to the position of the avatar.
A computer readable medium having recorded thereon a program for executing an augmented reality sound implementation method performed by the augmented reality sound implementation system according to any one of claims 1 to 8.
An application for a terminal device, which is stored in a medium for executing an augmented reality sound implementation method performed by the augmented reality sound implementation system according to any one of claims 1 to 8 in combination with the computing device, which is hardware.