WO2022070319A1 - 情報処理装置、制御方法、非一時的なコンピュータ可読媒体、及び情報処理システム - Google Patents

情報処理装置、制御方法、非一時的なコンピュータ可読媒体、及び情報処理システム Download PDF

Info

Publication number
WO2022070319A1
WO2022070319A1 PCT/JP2020/037184 JP2020037184W WO2022070319A1 WO 2022070319 A1 WO2022070319 A1 WO 2022070319A1 JP 2020037184 W JP2020037184 W JP 2020037184W WO 2022070319 A1 WO2022070319 A1 WO 2022070319A1
Authority
WO
WIPO (PCT)
Prior art keywords
information
sound image
user terminal
image localization
voice
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Ceased
Application number
PCT/JP2020/037184
Other languages
English (en)
French (fr)
Japanese (ja)
Inventor
悠志 中島
俊一 丸山
紫水子 鐘ヶ江
健臣 村岸
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
NEC Corp
Original Assignee
NEC Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by NEC Corp filed Critical NEC Corp
Priority to US18/021,899 priority Critical patent/US20230370798A1/en
Priority to PCT/JP2020/037184 priority patent/WO2022070319A1/ja
Priority to JP2022553313A priority patent/JP7509220B2/ja
Publication of WO2022070319A1 publication Critical patent/WO2022070319A1/ja
Anticipated expiration legal-status Critical
Ceased legal-status Critical Current

Links

Images

Classifications

    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30—Control circuits for electronic adaptation of the sound field
    • H04S7/302—Electronic adaptation of stereophonic sound system to listener position or orientation
    • H04S7/303—Tracking of listener position or orientation
    • H04S7/304—For headphones
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S7/00—Indicating arrangements; Control arrangements, e.g. balance control
    • H04S7/30—Control circuits for electronic adaptation of the sound field
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04R—LOUDSPEAKERS, MICROPHONES, GRAMOPHONE PICK-UPS OR LIKE ACOUSTIC ELECTROMECHANICAL TRANSDUCERS; ELECTRIC HEARING AIDS; PUBLIC ADDRESS SYSTEMS
    • H04R2460/00—Details of hearing devices, i.e. of ear- or headphones covered by H04R1/10 or H04R5/033 but not provided for in any of their subgroups, or of hearing aids covered by H04R25/00 but not provided for in any of its subgroups
    • H04R2460/07—Use of position data from wide-area or local-area positioning systems in hearing devices, e.g. program or information selection
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04S—STEREOPHONIC SYSTEMS 
    • H04S2400/00—Details of stereophonic systems covered by H04S but not provided for in its groups
    • H04S2400/11—Positioning of individual sound objects, e.g. moving airplane, within a sound field

Definitions

  • This disclosure relates to information processing devices, control methods, non-temporary computer-readable media, and information processing systems.
  • Patent Document 1 There is known a technique in which a user virtually installs voice data at a position designated by the user and outputs the virtually installed voice data to another user (for example, Patent Document 1). .. Further, there is also known a technique of outputting sound from a virtual sound source position in order to draw attention to the information provided to the user (for example, Patent Document 2).
  • Patent Document 1 when a user selects a position on a map, registers voice data for the selected position, and another user goes to the selected position, the registered voice data is reproduced.
  • the audio reproduction system to be performed is disclosed.
  • Patent Document 2 discloses a voice presentation system that presents voice according to a provided information position and a sound source relative position to a user wearing a wearable terminal.
  • Patent Document 1 discloses that voice data is virtually installed and provided to a user at that position. Therefore, even when Patent Document 1 is used, it may not be possible to satisfy the above requirements. Further, Patent Document 2 does not disclose that voice data is virtually installed. Therefore, even when Patent Document 2 is used, it may not be possible to satisfy the above requirements.
  • One of the purposes of the present disclosure is to solve the above-mentioned problems, and is an information processing device, a control method, and a non-temporary computer capable of outputting voice information localized in a sound image to a position desired by a user.
  • the purpose is to provide a readable medium and an information processing system.
  • the information processing device related to this disclosure is Voice information, first position information of the first user terminal, first direction information of the first user terminal, and distance information from the position indicated by the first position information to the installation position where the voice information is virtually installed.
  • the receiving means for receiving from the first user terminal The sound image localization position is determined based on the first position information, the first direction information, and the distance information, and the first position information, the sound image localization position information about the sound image localization position, and the audio information It is provided with a registration means for registering the above in the storage means in association with each other.
  • the control method is as follows. Voice information, first position information of the first user terminal, first direction information of the first user terminal, and distance information from the position indicated by the first position information to the installation position where the voice information is virtually installed. Is received from the first user terminal, The sound image localization position is determined based on the first position information, the first direction information, and the distance information. This is a control method in which the first position information, the sound image localization position information about the sound image localization position, and the voice information are associated and registered in the storage means.
  • Non-temporary computer-readable media relating to this disclosure may be referred to as.
  • Voice information first position information of the first user terminal, first direction information of the first user terminal, and distance information from the position indicated by the first position information to the installation position where the voice information is virtually installed.
  • the sound image localization position is determined based on the first position information, the first direction information, and the distance information.
  • a non-temporary computer in which a control program for causing a computer to execute a process of registering the first position information, the sound image localization position information about the sound image localization position, and the voice information in a storage means in association with each other is stored. It is a readable medium.
  • the information processing system related to this disclosure is A first user terminal and a server device that communicates with the first user terminal are provided.
  • the first user terminal is The distance from the position indicated by the voice information, the first position information of the first user terminal, the first direction information of the first user terminal, and the first position information to the installation position where the voice information is virtually installed.
  • Get information The server device is The voice information, the first position information, the first direction information, and the distance information are received from the first user terminal, and the voice information, the first position information, the first direction information, and the distance information are received from the first user terminal.
  • the sound image localization position is determined based on the first position information, the first direction information, and the distance information, and the first position information, the sound image localization position information about the sound image localization position, and the voice information It is an information processing system that associates and registers with a storage means.
  • an information processing device a control method, a non-temporary computer-readable medium, and an information processing system capable of outputting voice information localized in a sound image to a position desired by a user.
  • FIG. It is a block diagram which shows the structural example of the information processing apparatus which concerns on Embodiment 1.
  • FIG. It is a flowchart which shows the operation example of the information processing apparatus which concerns on Embodiment 1.
  • FIG. It is a figure for demonstrating the outline of Embodiment 2.
  • FIG. It is a figure which shows the structural example of the information processing system which concerns on Embodiment 2.
  • FIG. It is a figure which shows the structural example of the information processing system which concerns on Embodiment 4.
  • It is a flowchart which shows the operation example of the server apparatus which concerns on Embodiment 4.
  • FIG. 1 is a block diagram showing a configuration example of the information processing apparatus according to the first embodiment.
  • the information processing device 1 is, for example, a server device and communicates with a user terminal (not shown).
  • the user terminal is a communication terminal used by the user, and may be configured to include at least one communication terminal.
  • the information processing apparatus 1 includes a receiving unit 2 and a registration unit 3.
  • the receiving unit 2 receives voice information, position information of the user terminal, direction information of the user terminal, and distance information from the position indicated by the position information of the user terminal to the installation position where the voice information is virtually installed from the user terminal. do.
  • Audio information is audio content that is virtually installed at the above installation position.
  • the position information of the user terminal is the position information when the user gives an instruction to register the voice information, which indicates that the voice information is virtually installed at the installation position.
  • the position indicated by the position information of the user terminal may be different from the installation position where the voice information is virtually installed, or may be the same. That is, the user may virtually install the voice information at a position different from the position where the user is, or may virtually install the voice information at the position where the user is.
  • the position information of the user terminal may be used as the position information of the user.
  • the direction information of the user terminal is direction information when the user gives an instruction to register voice information.
  • the direction information of the user terminal may be used as the direction information of the user.
  • the direction information of the user terminal may be information indicating the face direction in which the user's face is facing, or may be information on the posture of the user.
  • the direction information of the user terminal includes the direction in which the user's face is facing and the elevation angle indicating the angle formed by the horizontal plane and the direction of the user's line of sight.
  • the elevation angle may indicate the angle formed by the ground surface and the direction of the user's line of sight.
  • the registration unit 3 determines the sound image localization position based on the position information, the direction information, and the distance information.
  • the registration unit 3 specifies an installation position where voice information is virtually installed based on position information, direction information, and distance information.
  • the registration unit 3 determines the installation position as the sound image localization position. That is, the registration unit 3 is installed so that the voice information is virtually installed by using the position information, the direction information, and the distance information so that the voice information can be virtually installed at a position different from the position where the user is.
  • the position is specified, and the specified installation position is determined as the sound image localization position.
  • the registration unit 3 registers the position information, the sound image localization position information about the sound image localization position, and the voice information in association with each other in the storage unit (not shown).
  • FIG. 2 is a flowchart showing an operation example of the information processing apparatus according to the first embodiment.
  • the receiving unit 2 receives the distance information about the voice information, the position information of the user terminal, the direction information of the user terminal, and the distance from the position indicated by the position information of the user terminal to the installation position where the voice information is virtually installed. Receive from the terminal (step S1).
  • the registration unit 3 determines the sound image localization position based on the position information, the direction information, and the distance information (step S2).
  • the registration unit 3 specifies an installation position where voice information is virtually installed based on position information, direction information, and distance information.
  • the registration unit 3 determines the installation position as the sound image localization position.
  • the registration unit 3 registers the position information, the sound image localization position information about the sound image localization position, and the voice information in the storage unit (not shown) in association with each other (step S3).
  • the receiving unit 2 receives not only the position information but also the direction information and the distance information, and the registration unit 3 determines the sound image localization position based on the position information, the direction information, and the distance information. Then, the position information, the sound image localization position information, and the voice information are registered in association with each other. In this way, since the registration unit 3 determines the sound image localization position using the direction information and the distance information in addition to the position information, the sound image localization position can be determined at the position desired by the user. Therefore, according to the information processing apparatus 1 according to the first embodiment, it is possible to output the audio information localized in the sound image at a position desired by the user.
  • the second embodiment is an embodiment that embodies the first embodiment.
  • an outline of the second embodiment will be described before explaining a specific configuration example of the second embodiment.
  • FIG. 3 is a diagram for explaining the outline of the second embodiment.
  • FIG. 3 is a diagram schematically showing, for example, a scene of inspecting a road.
  • the user U1 who is an inspector inspects a road
  • the user U1 inspects whether or not there is a repaired portion on the road, the side wall of the road, the ceiling of the tunnel, or the like.
  • the user U1 discovers the repaired portion R1
  • the user U1 informs the user U2 who is a repairer of the position of the repaired portion R1 and the status of the repaired portion R1.
  • the user U2 finds the repaired portion R1 and repairs the repaired portion R1 according to the situation of the repaired portion R1.
  • the user U1 informs the user U2 of the position of the repaired portion R1
  • the user U1 records the position of the repaired portion R1 on a map and records the situation of the repaired portion R1 as a memo.
  • the user U2 finds the repaired portion R1 based on the map recorded by the user U1, and repairs the repaired portion R1.
  • the user U1 finds a repair point R1 on the side wall of the road, for example.
  • the user U2 can reach the vicinity of the repair point R1 based on the map recorded by the user U1.
  • the user U2 may not be able to easily identify which part of the side wall is the repaired portion R1 only from the map recorded by the user U1.
  • the user U2 finds the repaired portion R1 while comparing the map and the memo recorded by the user U1 with the side wall. Therefore, the user U2 may take time to find the repaired portion R1, and as a result, the repaired portion R1 may not be repaired as planned.
  • the user U2 realizes a configuration in which the repaired portion R1 can be easily found.
  • information that shows the status of the repaired portion R1 and the detailed position of the repaired portion R1 is recorded as voice information (sound A).
  • Voice information is virtually installed at the repair location R1.
  • the user U2 comes to the vicinity of the repaired portion R1, the audio information in which the sound image is localized at the position of the repaired portion R1 is output to the user U2.
  • the user U2 can easily identify the position of the repaired portion R1 by virtually installing the voice information (sound A) and moving it so as to approach the position of the virtual sound source of the voice information.
  • the user U1 includes, for example, a communication terminal 20 which is a hearable device including a left unit 20L mounted on the left ear and a right unit 20R mounted on the right ear.
  • the information processing system virtually installs the voice information at the repair location R1 by using the direction information of the user U1 in response to the instruction of the user U1.
  • the user U2 wears, for example, a communication terminal 40, which is a hearable device, including a left unit 40L worn on the left ear and a right unit 40R worn on the right ear.
  • the information processing system uses the direction information of the user U2 to localize the installation position to the sound image localization position, and the voice information for which the sound image localization process is executed is the user U2. Control so that it is output to.
  • FIG. 3 illustrates a scene in which a road is inspected, but since this embodiment is an example of a scene to which the present embodiment is applied, the present embodiment may be applied to other scenes.
  • a user virtually installs voice information on an exhibit displayed on a wall surface, and outputs the installed voice information to another user.
  • the present embodiment including the above-mentioned scenes can be applied to a case where an arbitrary position is set as a sound image localization position, such as a scene where audio information is virtually installed at an arbitrary position.
  • FIG. 4 is a diagram showing a configuration example of the information processing system according to the second embodiment.
  • the information processing system 100 includes a server device 60, a user terminal 110 used by the user U1, and a user terminal 120 used by the user U2.
  • User U1 and user U2 may be different users or the same user.
  • the user terminal 110 is configured to include a function included in the user terminal 120.
  • the server device 60 is shown as a device different from the user terminal 120 in FIG. 4, the server device 60 may be incorporated in the user terminal 120, and the user terminal 120 has a configuration included in the server device 60. You may be prepared.
  • the user terminal 110 is a communication terminal used by the user U1, and includes the communication terminals 20 and 30.
  • the user terminal 120 is a communication terminal used by the user U2, and includes communication terminals 40 and 50.
  • the communication terminals 20 and 40 correspond to the communication terminals 20 and 40 of FIG. 3, and are, for example, hearable devices.
  • the communication terminals 30 and 50 are, for example, smartphone terminals, tablet terminals, mobile phones, and personal computer devices.
  • the user terminals 110 and 120 are configured to include two communication terminals, respectively, but for example, a communication terminal in which two communication terminals such as a head-mounted display are integrated may be used. That is, it may be composed of user terminals 110 and 120, and one communication terminal.
  • the user terminal 110 is configured to include the configuration included in the communication terminal 20 and the communication terminal 30
  • the user terminal 120 is configured to include the configuration included in the communication terminal 40 and the communication terminal 50.
  • the communication terminal 20 is a communication terminal used by the user U1 and is a communication terminal attached to the user U1.
  • the communication terminal 20 is a communication terminal attached to each of both ears of the user U1, and includes a left unit 20L attached to the left ear of the user U1 and a right unit 20R attached to the right ear of the user U1. ..
  • the communication terminal 20 may be a communication terminal in which the left unit 20L and the right unit 20R are integrated.
  • the communication terminal 20 is, for example, a communication terminal capable of wireless communication provided by a communication carrier, and communicates with a server device 60 via a network provided by the communication carrier.
  • the communication terminal 20 acquires the voice information.
  • the audio information may be audio content recorded by the user U1 or may be audio content held in the communication terminal 20.
  • the communication terminal 20 transmits the acquired voice information to the server device 60.
  • the communication terminal 20 left unit 20L and right unit 20R
  • the communication terminal 20 is described as directly communicating with the server device 60, it may be configured to communicate with the server device 60 via the communication terminal 30.
  • the communication terminal 20 acquires the direction information of the communication terminal 20 and transmits the acquired direction information to the server device 60.
  • the direction information of the communication terminal 20 is treated by the server device 60 as the direction information of the user terminal 110.
  • the communication terminal 20 may use the direction information of the communication terminal 20 as the direction information of the user U1.
  • the communication terminal 30 is a communication terminal used by the user U1.
  • the communication terminal 30 connects and communicates with the communication terminal 20 by, for example, wireless communication such as Bluetooth (registered trademark) and WiFi. Further, the communication terminal 30 communicates with the server device 60 via, for example, a network provided by a communication carrier.
  • the communication terminal 30 acquires the position information of the communication terminal 30 and transmits the acquired position information to the server device 60.
  • the position information of the communication terminal 30 is treated as the position information of the user terminal 110 by the server device 60.
  • the communication terminal 20 may use the direction information of the communication terminal 20 as the direction information of the user U1.
  • the communication terminal 30 may use the position information of the communication terminal 30 as the position information of the user U1.
  • the communication terminal 30 may acquire the position information of the left unit 20L and the right unit 20R based on the position information of the communication terminal 30 and the distances to the left unit 20L and the right unit 20R.
  • the communication terminal 30 acquires distance information from the position indicated by the position information of the communication terminal 30 to the installation position where the voice information is virtually installed.
  • the communication terminal 30 transmits the acquired distance information to the server device 60.
  • the installation position is a position where the user U1 virtually installs the voice information, and corresponds to the repair location R1 in the example shown in FIG.
  • the communication terminal 40 is a communication terminal used by the user U2 and is a communication terminal attached to the user U2.
  • the communication terminal 40 is a communication terminal attached to each of both ears of the user U2, and includes a left unit 40L attached to the left ear of the user U2 and a right unit 40R attached to the right ear of the user U2. ..
  • the communication terminal 40 may be a communication terminal in which the left unit 40L and the right unit 40R are integrated.
  • the communication terminal 40 is, for example, a communication terminal capable of wireless communication provided by a communication carrier, and communicates with a server device 60 via a network provided by the communication carrier.
  • the communication terminal 40 acquires the direction information of the communication terminal 40 and transmits the acquired direction information to the server device 60.
  • the direction information of the communication terminal 40 is treated by the server device 60 as the direction information of the user terminal 120.
  • the communication terminal 40 may use the direction information of the communication terminal 40 as the direction information of the user U2.
  • the communication terminal 40 outputs the voice information on which the sound image localization process is executed to each of the user's ears.
  • the communication terminal 40 (left unit 40L and right unit 40R) is described as directly communicating with the server device 60, it may be configured to communicate with the server device 60 via the communication terminal 50.
  • the communication terminal 50 is a communication terminal used by the user U2.
  • the communication terminal 50 connects and communicates with the communication terminal 40 by, for example, wireless communication such as Bluetooth and WiFi. Further, the communication terminal 50 communicates with the server device 60 via, for example, a network provided by a communication carrier.
  • the communication terminal 50 acquires the position information of the communication terminal 50 and transmits the acquired position information to the server device 60.
  • the position information of the communication terminal 50 is treated as the position information of the user terminal 120 by the server device 60.
  • the communication terminal 50 may use the position information of the communication terminal 50 as the position information of the user U2.
  • the communication terminal 50 may acquire the position information of the left unit 40L and the right unit 40R based on the position information of the communication terminal 50 and the distances to the left unit 40L and the right unit 40R.
  • the server device 60 corresponds to the information processing device 1 in the first embodiment.
  • the server device 60 communicates with the communication terminals 20, 30, 40, and 50 via, for example, a network provided by a communication carrier.
  • the server device 60 transmits the position information of the user terminal 110, the direction information of the user terminal 110, and the distance information from the position indicated by the position information of the user terminal 110 to the installation position where the voice information is virtually installed from the user terminal 110. Receive.
  • the server device 60 receives voice information and direction information of the communication terminal 20 from the communication terminal 20.
  • the server device 60 receives the position information and the distance information of the communication terminal 30 from the communication terminal 30.
  • the server device 60 determines the sound image localization position based on the position information, the direction information, and the distance information received from the user terminal 110.
  • the server device 60 registers the position information of the user terminal 110, the sound image localization position information about the sound image localization position, and the voice information in association with each other in the server device 60.
  • the sound image localization position information may be, for example, coordinates specified in a three-dimensional Cartesian coordinate system whose origin coordinate is a predetermined reference position, or position information represented by a direction and height with respect to a predetermined reference position. good.
  • the server device 60 generates area information that specifies an area based on the position information of the user terminal 110.
  • the server device 60 registers the generated area information in the server device 60 in association with the sound image localization position information.
  • the area corresponds to the predetermined area A1 shown in FIG. 3, and is a virtually set area, and may be referred to as a geo-fence.
  • the server device 60 may register the position information of the user terminal 110, the sound image localization position information, the voice information, and the area information in a storage device provided externally or internally.
  • the server device 60 receives the position information of the user terminal 120 and the direction information of the user terminal 120 from the user terminal 120.
  • the server device 60 receives the direction information of the communication terminal 40 from the communication terminal 40.
  • the server device 60 receives the position information of the communication terminal 50 from the communication terminal 50.
  • the server device 60 When the position indicated by the position information received from the user terminal 120 is included in the area indicated by the area information, the server device 60 is based on the position information, the direction information, and the determined sound image localization position information received from the user terminal 120. And generate information for sound image localization. The server device 60 controls so that the voice information corrected based on the sound image localization information is output to the left unit 40L and the right unit 40R of the communication terminal 40.
  • the communication terminal 20 includes a voice information acquisition unit 21 and a direction information acquisition unit 22. Since the communication terminal 20 includes the left unit 20L and the right unit 20R, both the left unit 20L and the right unit 20R include a voice information acquisition unit 21 and a direction information acquisition unit 22. Since it is assumed that the direction and height of the user U1 are substantially the same for the left and right ears, either the left unit 20L or the right unit 20R is the voice information acquisition unit 21 or the right unit 20R.
  • the configuration may include a direction information acquisition unit 22.
  • the voice information acquisition unit 21 includes, for example, a microphone and the like, and is configured to be capable of voice recognition of input voice.
  • the voice information acquisition unit 21 inputs a voice information registration instruction from the user U1.
  • the voice information registration instruction is an instruction to register the voice information so that the voice information is virtually installed at a position designated by the user.
  • the voice information acquisition unit 21 records the content uttered by the user U1 and generates the recorded content as voice information.
  • the voice information acquisition unit 21 transmits the generated voice information to the server device 60.
  • the voice information registration instruction from the user U1 includes information for designating the voice information
  • the voice information acquisition unit 21 acquires the designated voice information from the voice information stored in the communication terminal 20. , The acquired voice information may be transmitted to the server device 60.
  • the direction information acquisition unit 22 is configured to include, for example, a 9-axis sensor (3-axis acceleration sensor, 3-axis gyro sensor, 3-axis compass sensor) and the like.
  • the direction information acquisition unit 22 acquires the direction information of the communication terminal 20 by the 9-axis sensor.
  • the direction information acquisition unit 22 acquires the direction in which the communication terminal 20 is facing and the elevation angle of the communication terminal 20. Since the communication terminal 20 is attached to both ears of the user U1, the orientation in which the communication terminal 20 is facing can be said to be information indicating the face direction in which the face of the user U1 is facing.
  • the direction information acquisition unit 22 may acquire the inclination of the communication terminal 20 with respect to the ground surface or the horizontal plane as the elevation angle of the communication terminal 20.
  • the direction information acquisition unit 22 generates direction information of the communication terminal 20 including the acquired direction and the acquired elevation angle.
  • the direction information acquisition unit 22 may use the direction information of the communication terminal 20 as the direction information of the user U1. For example, when the user U1 inputs an instruction to register voice information, the direction information acquisition unit 22 acquires the direction information of the communication terminal 20 and transmits the acquired direction information of the communication terminal 20 to the server device 60.
  • the direction information acquisition unit 22 includes the 9-axis sensor, the posture of the user U1 can also be acquired. Therefore, the direction information may include posture information about the posture of the user U1. Further, since the direction information is data acquired by the 9-axis sensor, it may be referred to as sensing data.
  • the communication terminal 30 includes a position information acquisition unit 31 and a distance information acquisition unit 32.
  • the position information acquisition unit 31 is configured to include, for example, a GPS (Global Positioning System) receiver.
  • the position information acquisition unit 31 receives the GPS signal, acquires the latitude / longitude information of the communication terminal 30 based on the GPS signal, and uses the acquired latitude / longitude information as the position information of the communication terminal 30.
  • the position information acquisition unit 31 may use the position information of the communication terminal 30 as the position information of the user U1.
  • the position information acquisition unit 31 receives the instruction from the communication terminal 20, acquires the position information of the communication terminal 30, and uses the position information of the communication terminal 30 as the server device. Send to 60. That is, the position information acquisition unit 31 acquires the position information when the user U1 gives an instruction to register the voice information, which indicates that the voice information is virtually installed at the installation position, and the acquired position information is used as the server. It is transmitted to the device 60.
  • the distance information acquisition unit 32 acquires the distance information from the position of the user U1 to the installation position where the voice information is virtually installed.
  • the distance information acquisition unit 32 receives the instruction from the communication terminal 20, acquires the distance information, and transmits the distance information to the server device 60.
  • the distance information acquisition unit 32 also functions as, for example, an image pickup unit for a camera or the like, and generates an captured image when the user U1 captures an installation position where voice information is virtually installed.
  • the distance information acquisition unit 32 may acquire distance information by estimating the distance from the position of the user U1 to the installation position based on the captured image, for example, using the zoom degree of the camera.
  • the distance information acquisition unit 32 may acquire distance information by detecting an object included in the captured image and estimating the distance to the detected object.
  • the distance information acquisition unit 32 is configured to be able to acquire the line of sight of the user U1, and estimates the distance to the pointer by indicating the line of sight of the user U1 as a pointer in the direction indicated by the direction information of the user U1. You may get the distance information with.
  • the distance information acquisition unit 32 when the distance information acquisition unit 32 is configured to be able to recognize the voice of the user U1 and the voice emitted by the user U1 includes information related to the distance, the distance information acquisition unit 32 obtains the distance information based on the voice emitted by the user U1. You may get it.
  • the distance information acquisition unit 32 uses an infrared sensor or a laser configured to include an infrared sensor or a laser for distance measurement to measure the distance to an object included in the captured image to obtain distance information. May be obtained.
  • the communication terminal 40 includes a direction information acquisition unit 41 and an output unit 42. Since the communication terminal 40 includes the left unit 40L and the right unit 40R, both the left unit 40L and the right unit 40R may include a direction information acquisition unit 41 and an output unit 42. Since it is assumed that the direction and height of the user U2 are substantially the same for the left and right ears, either the left unit 40L or the right unit 40R includes the direction information acquisition unit 41. It may be configured.
  • the direction information acquisition unit 41 is configured to include, for example, a 9-axis sensor (3-axis acceleration sensor, 3-axis gyro sensor, 3-axis compass sensor) and the like.
  • the direction information acquisition unit 41 acquires the direction information of the communication terminal 40 by the 9-axis sensor.
  • the direction information acquisition unit 41 acquires the direction in which the communication terminal 40 is facing and the elevation angle of the communication terminal 40. Since the communication terminal 40 is attached to both ears of the user U2, the orientation in which the communication terminal 40 is facing can be said to be information indicating the face direction in which the face of the user U2 is facing.
  • the direction information acquisition unit 41 may acquire the inclination of the communication terminal 40 with respect to the horizontal plane as the elevation angle of the communication terminal 40.
  • the direction information acquisition unit 41 generates the acquired direction information including the direction and the elevation angle.
  • the direction information acquisition unit 41 may use the direction information of the communication terminal 40 as the direction information of the user U2.
  • the direction information acquisition unit 41 acquires the direction information of the communication terminal 40 periodically or aperiodically.
  • the direction information acquisition unit 41 transmits the acquired direction information to the server device 60.
  • the direction information acquisition unit 41 includes the 9-axis sensor, the posture of the user U2 can also be acquired. Therefore, the direction information may include posture information about the posture of the user U2. Further, since the direction information is data acquired by the 9-axis sensor, it may be referred to as sensing data.
  • the output unit 42 is configured to include, for example, a stereo speaker or the like.
  • the output unit 42 also functions as a communication unit, receives voice information for which sound image localization processing has been executed from the server device 60, and outputs the received voice information to the user's ear.
  • the output unit 42 switches from the output audio information to the received audio information and outputs it at a predetermined timing.
  • the audio information for which the sound image localization process has been executed includes the audio information for the left ear for the left unit 40L and the audio information for the right ear for the right unit 40R.
  • the output unit 42 of the left unit 40L outputs the voice information for the left ear
  • the output unit 42 of the right unit 40R outputs the voice information for the right ear.
  • the communication terminal 50 includes a position information acquisition unit 51.
  • the position information acquisition unit 51 is configured to include, for example, a GPS receiver.
  • the position information acquisition unit 51 receives a GPS signal, acquires latitude / longitude information of the communication terminal 50 based on the GPS signal, and uses the acquired latitude / longitude information as the position information of the communication terminal 50.
  • the position information acquisition unit 51 transmits the position information of the communication terminal 50 to the server device 60.
  • the position information acquisition unit 51 acquires the position information of the communication terminal 50 periodically or aperiodically.
  • the position information acquisition unit 51 transmits the acquired position information to the server device 60.
  • the position information acquisition unit 51 may acquire the latitude / longitude information of the communication terminal 50 as the position information of the user U2.
  • the server device 60 includes a receiving unit 61, a registration unit 62, an output unit 63, a control unit 64, and a storage unit 65.
  • the receiving unit 61 corresponds to the receiving unit 2 in the first embodiment.
  • the receiving unit 61 receives voice information, position information of the user terminal 110, direction information of the user terminal 110, and distance information from the user terminal 110.
  • the distance information is distance information from the position indicated by the position information of the user terminal 110 to the installation position where the voice information is virtually installed.
  • the receiving unit 61 receives voice information and direction information of the communication terminal 20 from the communication terminal 20, and receives position information and distance information of the communication terminal 30 from the communication terminal 30.
  • the receiving unit 61 outputs the direction information of the communication terminal 20 to the registration unit 62 as the direction information of the user terminal 110.
  • the receiving unit 61 outputs the position information of the communication terminal 30 to the registration unit 62 as the position information of the user terminal 110.
  • the receiving unit 61 may further receive the voice information registration instruction from the communication terminal 20.
  • the receiving unit 61 receives the position information of the user terminal 120 and the direction information of the user terminal 120 from the user terminal 120. Specifically, the receiving unit 61 receives the direction information of the communication terminal 40 from the communication terminal 40, and receives the position information of the communication terminal 50 from the communication terminal 50. The receiving unit 61 outputs the direction information of the communication terminal 40 to the output unit 63 as the direction information of the user terminal 120. The receiving unit 61 outputs the position information of the communication terminal 50 to the output unit 63 as the position information of the user terminal 120.
  • the registration unit 62 corresponds to the registration unit 3 in the first embodiment.
  • the registration unit 62 determines the sound image localization position based on the position information of the user terminal 110, the direction information of the user terminal 110, and the distance information.
  • the registration unit 62 determines the installation position where the voice information is virtually installed based on the position information of the user terminal 110, the direction information of the user terminal 110, and the distance information.
  • the direction information includes the direction in which the user terminal 110 is facing and the elevation angle of the user terminal 110. Therefore, the registration unit 62 can specify the installation position by using the position information of the user terminal 110, the azimuth and elevation angle included in the direction information of the user terminal 110, and the distance information.
  • the registration unit 62 determines the determined installation position as the sound image localization position.
  • the registration unit 62 registers the position information of the user terminal 110, the sound image localization position information about the sound image localization position, and the voice information in association with each other in the storage unit 65.
  • the registration unit 62 generates area information that specifies an area based on the position indicated by the position information of the user terminal 110.
  • the registration unit 62 generates, for example, area information for designating an area centered on the position indicated by the position information of the user terminal 110.
  • the registration unit 62 registers the generated area information in the storage unit 65 in association with the sound image localization position information.
  • the area corresponds to the predetermined area A1 shown in FIG. 3, and is a virtually set area, and may be referred to as a geo-fence. In the following description, the area may be described as a geofence.
  • the geo-fence can have any shape such as a circle, a sphere, a rectangle, a polygon, etc., and is specified based on the area information.
  • the area information includes, for example, size information that specifies the size of the geo-fence, and the geo-fence is designated by the size information.
  • the size information may be, for example, the radius of the geofence if the geofence is circular or spherical. Further, the size information may be, for example, the distance from the center of the polygon (sound image localization position) to each vertex constituting the polygon when the geo-fence is a polygon including a rectangle.
  • the geo-fence is a circle set on a plane parallel to the horizontal plane based on the position indicated by the position information of the user terminal 110, and the size information is the radius of the circle. do.
  • the size information may be referred to as length information that designates a geofence, or may be referred to as area distance information that designates a geofence.
  • the registration unit 62 generates a circular geo-fence with a predetermined radius. In other words, the registration unit 62 generates area information whose size information is a predetermined distance.
  • the radius of the geo-fence may be arbitrarily set by the user U1.
  • the registration unit 62 adjusts the size information according to the distance information received from the user terminal 110.
  • the registration unit 62 updates the area information registered in the storage unit 65 based on the changed size information.
  • the registration unit 62 adjusts the size information according to the distance information received from the user terminal 110 of the user U1, but the size information may not be adjusted.
  • the registration unit 62 may adjust the size information so as to reduce the size of the geo-fence. For example, when the distance indicated by the distance information is equal to or greater than a predetermined value, the registration unit 62 may change the size information so that the distance is halved from the distance indicated by the distance information.
  • the registration unit 62 may adjust the size information so as to increase the size of the geo-fence. For example, when the distance indicated by the distance information is less than a predetermined value, the registration unit 62 may change the size information so as to be equal to the distance indicated by the distance information.
  • the output unit 63 determines whether or not the position indicated by the position information of the user terminal 120 is included in the geofence designated by the area information. Specifically, the output unit 63 determines whether or not the position information of the user terminal 120 is included in the area information. When the position information of the user terminal 120 is included in the area information, the output unit 63 determines that the position indicated by the position information of the user terminal 120 is included in the geofence designated by the area information.
  • the output unit 63 is based on the position information of the user terminal 120, the direction information of the user terminal 120, the altitude information of the user terminal 120, and the sound image localization position information. , Generates sound image localization information based on the sound image localization position.
  • the sound image localization information is a parameter used to execute the sound image localization process for the audio information.
  • the sound image localization information is a parameter for correcting the audio information so that it can be heard as a sound from the sound image localization position.
  • the output unit 63 has left ear sound image localization information for the left unit 40L and right ear sound image for the right unit 40R based on the position information of the user terminal 120, the direction information of the user terminal 120, and the sound image localization position information. Generate localization information.
  • the output unit 63 outputs the sound image localization information including the left ear sound image localization information and the right ear sound image localization information, and the sound image localization position information to the control unit 64.
  • the control unit 64 acquires the audio information associated with the sound image localization position information output from the output unit 63 from the storage unit 65.
  • the control unit 64 executes the sound image localization process based on the sound image localization information generated by the output unit 63 for the acquired audio information. In other words, the control unit 64 corrects the acquired audio information based on the sound image localization information.
  • the control unit 64 corrects the audio information based on the sound image localization information for the left ear, and generates the audio information for the left ear.
  • the control unit 64 corrects the audio information based on the sound image localization information for the right ear, and generates the audio information for the right ear.
  • the control unit 64 also functions as a communication unit, and transmits voice information for the left ear and voice information for the right ear to the left unit 40L and the right unit 40R of the communication terminal 40, respectively. Each time the output unit 63 generates sound image localization information, the control unit 64 generates audio information for the left ear and audio information for the right ear based on the latest information for sound image localization, and the left unit 40L and the right unit. Send to 40R. The control unit 64 controls to output the voice information for the left ear and the voice information for the right ear to the output unit 43 of the left unit 40L and the right unit 40R of the communication terminal 40.
  • the storage unit 65 stores the position information, the voice information, the sound image localization position information, and the area information of the user terminal 110 in association with each other according to the control of the registration unit 62. Further, the storage unit 65 updates the area information based on the changed size information according to the control of the registration unit 62.
  • FIG. 5 is a flowchart showing an operation example of the server device according to the second embodiment.
  • the flowchart shown in FIG. 5 is broadly divided into a voice information registration process executed in steps S11 to S14 and a voice information output process executed in steps S15 to S19.
  • the voice information registration process is executed when the user U1 gives an instruction to register the voice information by virtually installing the voice information at the installation position.
  • the voice information output process is repeatedly executed every time the server device 60 acquires the position information and the direction information of the user U2.
  • the receiving unit 61 receives voice information, position information of the user terminal 110, direction information of the user terminal 110, and distance information from the user terminal 110 (step S11).
  • the receiving unit 61 receives the voice information and the direction information of the communication terminal 20 from the communication terminal 20, and receives the position information and the distance information of the communication terminal 30 from the communication terminal 30.
  • the receiving unit 61 outputs the direction information of the communication terminal 20 to the registration unit 62 as the direction information of the user terminal 110.
  • the receiving unit 61 outputs the position information of the communication terminal 30 to the registration unit 62 as the position information of the user terminal 110.
  • the registration unit 62 determines the sound image localization position based on the position information of the user terminal 110, the direction information of the user terminal 110, and the distance information (step S12).
  • the registration unit 62 determines the installation position where the voice information is virtually installed based on the position information of the user terminal 110, the direction information of the user terminal 110, and the distance information.
  • the registration unit 62 determines the determined installation position as the sound image localization position.
  • the registration unit 62 registers the sound image localization position information about the sound image localization position of the user terminal 110 and the voice information in the storage unit 65 in association with each other (step S13).
  • the registration unit 62 generates and adjusts the area information for designating the geo-fence based on the position indicated by the position information of the user terminal 110 (step S14).
  • the registration unit 62 generates area information that designates a geo-fence based on the position indicated by the position information of the user terminal 110.
  • the registration unit 62 registers the generated area information in the storage unit 65 in association with the sound image localization position information.
  • the registration unit 62 adjusts the size information included in the area information according to the distance information.
  • the receiving unit 61 receives the position information of the user terminal 120 and the direction information of the user terminal 120 from the user terminal 120 (step S15).
  • the receiving unit 61 receives the direction information of the communication terminal 40 from the communication terminal 40, and receives the position information of the communication terminal 50 from the communication terminal 50.
  • the receiving unit 61 outputs the direction information of the communication terminal 40 to the output unit 63 as the direction information of the user terminal 120.
  • the receiving unit 61 outputs the position information of the communication terminal 50 to the output unit 63 as the position information of the user terminal 120.
  • the output unit 63 determines whether or not the position indicated by the position information of the user terminal 120 is included in the geo-fence (step S16). The output unit 63 determines whether or not the position information of the user terminal 120 is included in the area information. When the position information of the user terminal 120 is included in the area information, the output unit 63 determines that the position indicated by the position information of the user terminal 120 is included in the geofence designated by the area information.
  • the output unit 63 is for sound image localization based on the position information, direction information, and sound image localization position information of the user terminal 120. Generate information (step S17). The output unit 63 generates information for sound image localization based on the determined sound image localization position. The output unit 63 is based on the position information of the user terminal 120, the direction information of the user terminal 120, and the sound image localization position information, and the left ear sound image localization information for the left unit 40L and the right ear for the right unit 40R. Generates sound image localization information. The output unit 63 outputs the sound image localization information including the left ear sound image localization information and the right ear sound image localization information, and the sound image localization position information to the control unit 64.
  • step S16 when the position indicated by the position information of the user terminal 120 is not included in the geofence (NO in step S16), the server device 60 returns to step S15 and executes step S15.
  • step S18 the output unit 63 outputs the sound image localization information to the control unit 64 (step S18).
  • the output unit 63 outputs the sound image localization information including the left ear sound image localization information and the right ear sound image localization information, and the sound image localization position information to the control unit 64.
  • the control unit 64 corrects the voice information and transmits the corrected voice information to the output unit 43 of the communication terminal 40 (step S19).
  • the control unit 64 acquires the audio information associated with the sound image localization position information output from the output unit 63 from the storage unit 65.
  • the control unit 64 corrects the acquired audio information based on the sound image localization information.
  • the control unit 64 corrects the audio information based on the sound image localization information for the left ear to generate the audio information for the left ear, and corrects the audio information based on the sound image localization information for the right ear. Generate voice information for the right ear.
  • the control unit 64 transmits the voice information for the left ear and the voice information for the right ear to the left unit 40L and the right unit 40R of the communication terminal 40, respectively.
  • the receiving unit 61 receives not only the position information of the user terminal 110 but also the direction information and the distance information of the user terminal 110.
  • the registration unit 62 determines the sound image localization position based on the position information of the user terminal 110, the direction information of the user terminal 110, and the distance information. In this way, since the registration unit 62 determines the sound image localization position using the direction information and the distance information of the user terminal 110, the user U1 virtually places the audio information at a position different from the position where the user U1 is located. Even if you want to install it, you can set the sound image localization position to the desired position.
  • the sound image localization position can be set at a position desired by the user. Therefore, according to the information processing system 100 according to the second embodiment, it is possible to output audio information in which the sound image is localized at a position desired by the user.
  • the server device 60 includes an output unit 63 and a control unit 64.
  • the server device 60 outputs to the user U2 the voice information in which the sound image localization process is executed based on the position information of the user terminal 120, the direction information of the user terminal 120, and the sound image localization position information. do.
  • the server device 60 corrects the voice information from the installation position where the voice information is installed by the user U1 so that the voice information is output, and outputs the corrected voice information to the user U2.
  • the user U2 can easily specify the installation position by moving so as to approach the direction of the virtual sound source of the voice information. Using the example shown in FIG.
  • the user U2 who is a repairman moves so as to approach the direction of the virtual sound source of the voice information, so that the repair point R1 found by the user U1 who is an inspector can be easily accessed. Can be specified in. Therefore, according to the information processing system 100 according to the second embodiment, it is possible to contribute to the improvement of work efficiency. Further, according to the information processing system 100 according to the second embodiment, the user U2 can be guided in the direction of the virtual sound source of the voice information.
  • the registration unit 62 adjusts the size information according to the distance information received from the user U1, but adjusts the size information by using the size of the object at the sound image localization position. May be good.
  • the second embodiment can be realized by modifying it as follows.
  • the distance information acquisition unit 32 also functions as, for example, an image pickup unit of a camera or the like, captures an image pickup region including a sound image localization position, and generates an image capture image.
  • the distance information acquisition unit 32 transmits the captured image and the zoom degree of the camera to the server device 60.
  • the receiving unit 61 receives the captured image and the zoom degree from the distance information acquisition unit 32.
  • the registration unit 62 is configured to be capable of performing image analysis, analyzes the captured image, and identifies an object included in the captured image.
  • the registration unit 62 specifies the size of the object based on the zoom degree of the camera and the captured image. Since the user generally captures an object (object) of interest as the center of the imaging region, the registration unit 62 sets the object in the center of the captured image as the object and targets the object based on the zoom degree of the camera and the captured image. Identify the size of the object.
  • the registration unit 62 adjusts the size information based on the size of the object.
  • the registration unit 62 sets the size of the object included in the captured image to the size of the object whose size is known, without using the zoom degree of the camera.
  • the size of the object may be specified based on the above.
  • the registration unit 62 may change the size information so that the size of the geo-fence becomes the first size.
  • the registration unit 62 does not have to change the size information, and changes the size information so that the distance is shorter than the predetermined distance. You may.
  • the registration unit 62 changes the size information so that the size of the geo-fence becomes a second size smaller than the first size when the size of the specified object is less than a predetermined value. You may. When the size of the specified object is less than a predetermined value, the registration unit 62 may change the size information so that the distance is shorter than the predetermined distance. Alternatively, the registration unit 62 may change the size information so that the size of the specified object is smaller than the distance when the size is equal to or larger than a predetermined value.
  • the registration unit 62 determines the size of the object based on the captured image, but based on the object information about the object and the object position information of the object, the registration unit 62 determines the size of the object.
  • the size of the object may be specified.
  • the second embodiment can be realized by modifying it as follows.
  • the storage unit 65 stores the object information about the plurality of objects in association with the object position information of the plurality of objects.
  • the object information may be information in which the object identification information for identifying the object and the size of the object are associated with each other.
  • the registration unit 62 specifies the object position information including the sound image localization position information among the object position information stored in the storage unit 65. Then, the registration unit 62 specifies the size of the object based on the object information associated with the specified object position information. The registration unit 62 adjusts the size information based on the size of the object. The size information may be adjusted by the registration unit 62 in the same manner as in the first modification.
  • FIG. 6 is a diagram showing a configuration example of the information processing system according to the third embodiment.
  • the information processing system 200 includes a user terminal 110, a user terminal 120, and a server device 80.
  • the user terminal 110 includes communication terminals 20 and 30.
  • the user terminal 120 includes communication terminals 50 and 70.
  • the information processing system 200 has a configuration in which the communication terminal 40 according to the second embodiment is replaced with the communication terminal 70, and the server device 60 is replaced with the server device 80. Since the configuration examples and operation examples of the communication terminals 20, 30 and 50 are the same as those in the second embodiment, the description thereof will be omitted as appropriate.
  • the communication terminal 70 includes a direction information acquisition unit 41, a control unit 71, and an output unit 42.
  • the communication terminal 70 has a configuration in which a control unit 71 is added to the configuration of the communication terminal 40 according to the second embodiment. Since the configurations of the direction information acquisition unit 41 and the output unit 42 are basically the same as those of the second embodiment, the description thereof will be omitted as appropriate.
  • the communication terminal 70 is provided with the control unit 71, but the communication terminal 50 may be provided with the control unit 71 and the communication terminal 70 may not be provided with the control unit 71.
  • the control unit 71 also functions as a communication unit and communicates with the server device 80.
  • the control unit 71 receives audio information and sound image localization information from the output unit 81 of the server device 80.
  • the control unit 71 executes the sound image localization process for the audio information based on the sound image localization information. In other words, the control unit 71 corrects the audio information based on the sound image localization information.
  • the sound image localization information includes the sound image localization information for the left ear and the sound image localization information for the right ear, as in the second embodiment.
  • the control unit 71 corrects the audio information based on the sound image localization information for the left ear to generate the audio information for the left ear, corrects the audio information based on the sound image localization information for the right ear, and corrects the audio information for the right ear. Generate voice information for.
  • the control unit 71 outputs the voice information for the left ear and the voice information for the right ear to the output unit 42. Each time the control unit 71 generates sound image localization information from the output unit 81, the control unit 71 generates audio information for the left ear and audio information for the right ear based on the latest information for sound image localization, and the audio information for the left ear and the audio information for the right ear. The voice information for the right ear is output to the output unit 42.
  • the output unit 42 inputs the voice information for which the sound image localization process has been executed by the control unit 71, and outputs the input voice information to the user's ear.
  • the output unit 42 of the left unit 40L outputs the voice information for the left ear
  • the output unit 42 of the right unit 40R outputs the voice information for the right ear.
  • the output unit 42 switches from the output audio information to the received audio information and outputs it at a predetermined timing.
  • the server device 80 includes a receiving unit 61, a registration unit 62, an output unit 81, and a storage unit 65.
  • the server device 80 does not include the control unit 64 according to the second embodiment, and the output unit 63 is replaced with the output unit 81. Since the configurations of the receiving unit 61, the registration unit 62, and the storage unit 65 are basically the same as those in the second embodiment, the description thereof will be omitted as appropriate.
  • the output unit 81 also functions as a communication unit, and transmits (outputs) sound image localization information including left ear sound image localization information and right ear sound image localization information generated by the output unit 81 to the control unit 71. )do.
  • the output unit 81 transmits the sound image localization information to the control unit 71 each time the sound image localization information is generated.
  • the output unit 81 controls the control unit 71 so as to perform the sound image localization process with the latest sound image localization information.
  • the output unit 81 acquires the audio information associated with the sound image localization position information used for generating the sound image localization information from the storage unit 65.
  • the output unit 81 transmits (outputs) the acquired voice information to the control unit 71.
  • the output unit 81 does not transmit the audio information to the control unit 71 again when the sound image localization information is generated and the audio information has already been transmitted to the control unit 71.
  • the output unit 81 outputs (transmits) the sound image localization information to the control unit 71 (step S18).
  • the output unit 81 transmits the generated sound image localization information to the control unit 71.
  • the output unit 81 acquires the audio information associated with the sound image localization position information used for generating the sound image localization information from the storage unit 65.
  • the output unit 81 transmits (outputs) the acquired voice information to the control unit 71.
  • the control unit 71 corrects the voice information and transmits (outputs) the corrected voice information to the output unit 42 (step S19).
  • the control unit 71 receives audio information and sound image localization information from the output unit 81.
  • the control unit 71 corrects the voice information based on the sound image localization information, and transmits (outputs) the corrected voice information to the output unit 42.
  • the third embodiment is configured in which the communication terminal 70 executes the sound image localization process for the voice information.
  • the server device 80 performs sound image localization processing on audio information output to all communication terminals as in the second embodiment
  • the processing load of the server device 80 increases as the number of communication terminals increases. .. Therefore, it becomes necessary to increase the number of server devices according to the number of communication terminals.
  • the server device 80 does not execute the sound image localization process for the voice information, but the communication terminal 70 executes it, so that the processing load of the server device 80 can be reduced. Therefore, according to the information processing system 200 according to the third embodiment, it is possible to suppress the equipment cost associated with the expansion of the server.
  • the network load can be suppressed.
  • the corrected audio information must be transmitted every time the sound image localization information is updated.
  • the output unit 81 when the output unit 81 has already transmitted the audio information to the control unit 71, the output unit 81 does not need to transmit the audio information again, but may transmit only the sound image localization information. Therefore, according to the information processing system 200 according to the third embodiment, the network load can be suppressed.
  • the fourth embodiment is an improved example of the second and third embodiments.
  • the present embodiment will be described using the second embodiment while appropriately omitting the same configuration examples and operation examples as those of the second embodiment.
  • FIG. 7 is a diagram showing a configuration example of the information processing system according to the fourth embodiment.
  • the information processing system 300 includes a user terminal 110, a user terminal 120, and a server device 160.
  • the user terminal 110 includes communication terminals 140 and 30.
  • the user terminal 120 includes communication terminals 150 and 50.
  • the information processing system 300 has a configuration in which the communication terminal 20 is replaced with the communication terminal 140, the communication terminal 40 is replaced with the communication terminal 150, and the server device 60 is replaced with the server device 160 in the second embodiment. Since the configuration examples and operation examples of the communication terminals 30 and 50 are the same as those in the second embodiment, the description thereof will be omitted as appropriate.
  • the communication terminal 140 includes a voice information acquisition unit 21, a direction information acquisition unit 22, and an altitude information acquisition unit 141.
  • the communication terminal 140 is a configuration in which an advanced information acquisition unit 141 is added to the configuration of the communication terminal 20 according to the second embodiment. Since the configurations of the voice information acquisition unit 21 and the direction information acquisition unit 22 are basically the same as those of the second embodiment, the description thereof will be omitted as appropriate.
  • both the left unit 20L and the right unit 20R are provided with the altitude information acquisition unit 141. Since it is assumed that the height of the user U1 is substantially the same for the left and right ears, either the left unit 20L or the right unit 20R may be configured to include the altitude information acquisition unit 141.
  • the altitude information acquisition unit 141 is configured to include, for example, an altitude sensor.
  • the altitude information acquisition unit 141 acquires the altitude information of the communication terminal 140 by the altitude sensor and transmits the acquired altitude information to the server device 160.
  • the altitude information is information indicating the height of the communication terminal 140 with respect to the ground surface or the horizontal plane.
  • the communication terminal 150 includes a direction information acquisition unit 41, an altitude information acquisition unit 151, and an output unit 42.
  • the communication terminal 150 is a configuration in which an advanced information acquisition unit 151 is added to the configuration of the communication terminal 40 according to the second embodiment. Since the configurations of the direction information acquisition unit 41 and the output unit 42 are basically the same as those of the second embodiment, the description thereof will be omitted as appropriate.
  • both the left unit 40L and the right unit 40R are provided with the altitude information acquisition unit 151. Since it is assumed that the height of the user U2 is substantially the same for the left and right ears, one of the left unit 40L and the right unit 40R may be configured to include the altitude information acquisition unit 151.
  • the altitude information acquisition unit 151 is configured to include, for example, an altitude sensor.
  • the altitude information acquisition unit 151 acquires the altitude information of the communication terminal 150 by the altitude sensor at the timing when the direction information acquisition unit 41 acquires the direction information, and transmits the acquired altitude information to the server device 160.
  • the altitude information is information indicating the height of the communication terminal 150 with respect to the ground surface or the horizontal plane.
  • the server device 160 includes a receiving unit 161, a registration unit 162, an output unit 163, a control unit 64, and a storage unit 65.
  • the server device 160 has a configuration in which the receiving unit 61, the registration unit 62, and the output unit 63 in the second embodiment are replaced with the receiving unit 161, the registration unit 162, and the output unit 163, respectively. Since the configurations of the control unit 64 and the storage unit 65 are basically the same as those of the second embodiment, the description thereof will be omitted as appropriate.
  • the receiving unit 161 further receives the altitude information of the user terminal 110 from the user terminal 110.
  • the receiving unit 161 receives the altitude information of the communication terminal 140 from the communication terminal 140, and outputs the altitude information of the communication terminal 140 to the registration unit 162 as the altitude information of the user terminal 110.
  • the receiving unit 161 receives the altitude information of the user terminal 120 from the user terminal 120.
  • the receiving unit 161 receives the altitude information of the communication terminal 150 from the communication terminal 150, and outputs the altitude information of the communication terminal 150 to the output unit 63 as the altitude information of the user terminal 120.
  • the receiving unit 61 outputs the position information of the communication terminal 50 to the output unit 163 as the position information of the user terminal 120.
  • the registration unit 162 further uses the altitude information of the user terminal 110 to determine the sound image localization position.
  • the registration unit 162 determines an installation position where voice information is virtually installed based on the position information of the user terminal 110, the direction information of the user terminal 110, the altitude information of the user terminal 110, and the distance information.
  • the registration unit 162 determines the determined installation position as the sound image localization position.
  • the registration unit 162 registers the position information of the user terminal 110, the sound image localization position information about the sound image localization position, and the voice information in association with each other in the storage unit 65.
  • the output unit 163 is based on the position information of the user terminal 120, the direction information of the user terminal 120, the altitude information of the user terminal 120, and the sound image localization position information. Generates sound image localization information based on the sound image localization position.
  • the output unit 163 localizes the sound image without using the altitude information of the user terminal 120. Information may be generated. Alternatively, when the sound image localization position is the same as the height of the user U1 and the user U2, the output unit 163 may generate the sound image localization information without using the altitude information of the user terminal 120.
  • the output unit 163 has left ear sound image localization information and a right unit for the left unit 40L based on the position information of the user terminal 120, the direction information of the user terminal 120, the altitude information of the user terminal 120, and the sound image localization position information. Generates sound image localization information for the right ear for 40R.
  • the output unit 163 outputs the sound image localization information including the left ear sound image localization information and the right ear sound image localization information, and the sound image localization position information to the control unit 64.
  • FIG. 8 is a flowchart showing an operation example of the server device according to the fourth embodiment.
  • FIG. 8 corresponds to FIG. 5, where steps S11, S12, S15 and S17 of FIG. 5 are replaced by steps S21, S22, 23 and S24, respectively. Since steps S13, S14, S16, S18 and S19 in FIG. 8 are basically the same as those in FIG. 5, description thereof will be omitted as appropriate.
  • the receiving unit 161 receives voice information, position information of the user terminal 110, direction information of the user terminal 110, altitude information of the user terminal 110, and distance information from the user terminal 110 (step S21).
  • the receiving unit 161 receives voice information, direction information of the communication terminal 140, and altitude information of the communication terminal 140 from the communication terminal 140, and receives position information and distance information of the communication terminal 30 from the communication terminal 30.
  • the receiving unit 161 outputs the direction information of the communication terminal 140 and the altitude information of the communication terminal 140 to the registration unit 162 as the direction information of the user terminal 110 and the altitude information of the user terminal 110.
  • the receiving unit 161 outputs the position information of the communication terminal 30 to the registration unit 162 as the position information of the user terminal 110.
  • the registration unit 162 determines the sound image localization position based on the position information of the user terminal 110, the direction information of the user terminal 110, the altitude information of the user terminal 110, and the distance information (step S22).
  • the registration unit 162 determines an installation position where voice information is virtually installed based on the position information of the user terminal 110, the direction information of the user terminal 110, the altitude information of the user terminal 110, and the distance information.
  • the registration unit 162 determines the determined installation position as the sound image localization position.
  • the receiving unit 161 receives the position information of the user terminal 120, the direction information of the user terminal 120, and the altitude information of the user terminal 120 from the user terminal 120 (step S23).
  • the receiving unit 161 receives the direction information of the communication terminal 150 and the altitude information of the communication terminal 150 from the communication terminal 150, and receives the position information of the communication terminal 50 from the communication terminal 50.
  • the receiving unit 161 outputs the direction information of the communication terminal 150 and the altitude information of the communication terminal 150 to the output unit 163 as the direction information of the user terminal 120 and the altitude information of the user terminal 120.
  • the receiving unit 161 outputs the position information of the communication terminal 50 to the output unit 163 as the position information of the user terminal 120.
  • the output unit 163 uses the sound image based on the position information, direction information, altitude information, and sound image localization position information of the user terminal 120. Generate localization information (step S24).
  • the output unit 163 provides left ear sound image localization information for the left unit 40L and sound image localization information for the left unit 40L based on the position information of the user terminal 120, the direction information of the user terminal 120, the altitude information of the user terminal 120, and the sound image localization position information.
  • the output unit 163 outputs the sound image localization information including the left ear sound image localization information and the right ear sound image localization information, and the sound image localization position information to the control unit 64.
  • the server device 160 further uses the altitude information of the user terminal 110 and the user terminal 120 to determine the sound image localization position and generate the sound image localization information. Therefore, according to the information processing system 300 according to the fourth embodiment, the sound image localization position can be set at a position closer to the position where the user U1 virtually wants to install the voice information, as compared with the second and third embodiments.
  • FIG. 9 shows the information processing device 1, the communication terminals 20, 30, 40, 50, 70, 140, 150 and the server devices 60, 80, 160 (hereinafter referred to as information processing device 1 and the like) described in the above-described embodiment. ) Is a diagram showing a hardware configuration example.
  • the information processing apparatus 1 and the like include a network interface 1201, a processor 1202, and a memory 1203.
  • the network interface 1201 is used to communicate with other devices included in the information processing system.
  • the processor 1202 reads software (computer program) from the memory 1203 and executes it to perform processing of the information processing apparatus 1 and the like described by using the flowchart in the above-described embodiment.
  • the processor 1202 may be, for example, a microprocessor, an MPU (MicroProcessingUnit), or a CPU (CentralProcessingUnit).
  • Processor 1202 may include a plurality of processors.
  • Memory 1203 is composed of a combination of volatile memory and non-volatile memory. Memory 1203 may include storage located away from processor 1202. In this case, processor 1202 may access memory 1203 via an I / O interface (not shown).
  • the memory 1203 is used to store the software module group. By reading these software modules from the memory 1203 and executing the processor 1202, the processor 1202 can perform the processing of the information processing apparatus 1 and the like described in the above-described embodiment.
  • each of the processors included in the information processing apparatus 1 and the like executes one or a plurality of programs including a set of instructions for causing a computer to perform the algorithm described with reference to the drawings.
  • Non-temporary computer-readable media include various types of tangible storage mediums.
  • Examples of non-temporary computer-readable media include magnetic recording media (eg, flexible disks, magnetic tapes, hard disk drives), magneto-optical recording media (eg, magneto-optical disks).
  • examples of non-temporary computer-readable media include CD-ROM (Read Only Memory), CD-R, and CD-R / W.
  • examples of non-temporary computer readable media include semiconductor memory.
  • the semiconductor memory includes, for example, a mask ROM, a PROM (Programmable ROM), an EPROM (Erasable PROM), a flash ROM, and a RAM (RandomAccessMemory).
  • the program may also be supplied to the computer by various types of transient computer readable medium. Examples of temporary computer readable media include electrical, optical, and electromagnetic waves.
  • the temporary computer-readable medium can supply the program to the computer via a wired communication path such as an electric wire and an optical fiber, or a wireless communication path.
  • Appendix 1 Voice information, first position information of the first user terminal, first direction information of the first user terminal, and distance information from the position indicated by the first position information to the installation position where the voice information is virtually installed.
  • the receiving means for receiving from the first user terminal The sound image localization position is determined based on the first position information, the first direction information, and the distance information, and the first position information, the sound image localization position information about the sound image localization position, and the voice information
  • An information processing device including a registration means for associating and registering in a storage means.
  • the registration means generates area information that specifies an area based on the position indicated by the first position information, and registers the generated area information in the storage means in association with the sound image localization position information.
  • the information processing apparatus according to Appendix 1.
  • the receiving means receives the second position information of the second user terminal and the second direction information of the second user terminal from the second user terminal. When the position indicated by the second position information is included in the region, sound image localization information is generated based on the sound image localization position information, the second position information, and the second direction information, and the audio information and the audio information are used.
  • the information processing apparatus according to Appendix 2 further comprising an output means for outputting the sound image localization information.
  • the receiving means further receives the first altitude information of the first user terminal and the second altitude information of the second user terminal.
  • the registration means further uses the first altitude information to determine the sound image localization position.
  • the output means has the sound image based on the sound image localization position information, the second position information, the second direction information, and the second altitude information.
  • the information processing apparatus according to Appendix 3, which generates localization information.
  • a control unit that executes sound image localization processing on the voice information based on the voice information and the sound image localization information, and transmits the voice information on which the sound image localization processing is executed to the second user terminal.
  • the information processing apparatus according to Appendix 3 or 4, further comprising.
  • (Appendix 6) The information processing device according to Appendix 3 or 4, wherein the output means transmits the voice information and the sound image localization information to the second user terminal.
  • (Appendix 7) The area information includes size information that specifies the size of the area.
  • the information processing apparatus according to any one of Supplementary note 2 to 6, wherein the registration means adjusts the size information according to the distance information.
  • the information processing apparatus according to Appendix 8) The information processing apparatus according to Appendix 7, wherein the registration means changes the size information so as to reduce the size of the region when the distance indicated by the distance information is equal to or larger than a predetermined value.
  • the registration means When the size of the object is less than a predetermined value, the registration means changes the size information so that the size of the region becomes a second size smaller than the first size.
  • the information processing apparatus according to Appendix 11.
  • the receiving means receives an image of an imaging region including the installation position from the first user terminal, and receives the image.
  • the information processing apparatus according to any one of Supplementary note 10 to 12, wherein the registration means specifies the size of the object at the sound image localization position based on the image.
  • the storage means stores the object information about the plurality of objects in association with the object position information of the plurality of objects.
  • the registration means specifies the size of the object based on the object information associated with the object position information including the first position information among the object position information, Appendix 10 to 12.
  • the information processing apparatus according to any one of the above items.
  • (Appendix 15) Voice information, first position information of the first user terminal, first direction information of the first user terminal, and distance information from the position indicated by the first position information to the installation position where the voice information is virtually installed. Is received from the first user terminal, The sound image localization position is determined based on the first position information, the first direction information, and the distance information.
  • a non-temporary computer in which a control program for causing a computer to execute a process of registering the first position information, the sound image localization position information about the sound image localization position, and the voice information in a storage means in association with each other is stored. Readable medium.
  • a first user terminal and a server device that communicates with the first user terminal are provided.
  • the first user terminal is The distance from the position indicated by the voice information, the first position information of the first user terminal, the first direction information of the first user terminal, and the first position information to the installation position where the voice information is virtually installed.
  • Get information The server device is The voice information, the first position information, the first direction information, and the distance information are received from the first user terminal, and the voice information, the first position information, the first direction information, and the distance information are received from the first user terminal.
  • the sound image localization position is determined based on the first position information, the first direction information, and the distance information, and the first position information, the sound image localization position information about the sound image localization position, and the voice information
  • An information processing system that associates and registers with storage means.
  • the server device is 17. The appendix 17, wherein the area information for designating the area with reference to the position indicated by the first position information is generated, and the generated area information is registered in the storage means in association with the sound image localization position information.
  • Information processing system is used to generate information for designating
  • Information processing device 2 61, 161 Receiver 3, 62, 162 Registration unit 20, 30, 40, 50, 70, 140, 150 Communication terminal 21 Voice information acquisition unit 22, 41 Direction information acquisition unit 31, 51 Position information Acquisition unit 32 Distance information acquisition unit 42, 63, 81, 163 Output unit 60, 80, 160 Server device 64, 71 Control unit 65 Storage unit 100, 200, 300 Information processing system 110, 120 User terminal 141, 151 Advanced information acquisition Department

Landscapes

  • Physics & Mathematics (AREA)
  • Engineering & Computer Science (AREA)
  • Acoustics & Sound (AREA)
  • Signal Processing (AREA)
  • Stereophonic System (AREA)
  • Circuit For Audible Band Transducer (AREA)
  • Telephone Function (AREA)
  • Telephonic Communication Services (AREA)
PCT/JP2020/037184 2020-09-30 2020-09-30 情報処理装置、制御方法、非一時的なコンピュータ可読媒体、及び情報処理システム Ceased WO2022070319A1 (ja)

Priority Applications (3)

Application Number Priority Date Filing Date Title
US18/021,899 US20230370798A1 (en) 2020-09-30 2020-09-30 Information processing device, control method, non-transitory computer-readable medium, and information processing system
PCT/JP2020/037184 WO2022070319A1 (ja) 2020-09-30 2020-09-30 情報処理装置、制御方法、非一時的なコンピュータ可読媒体、及び情報処理システム
JP2022553313A JP7509220B2 (ja) 2020-09-30 2020-09-30 情報処理装置、制御方法、制御プログラム、及び情報処理システム

Applications Claiming Priority (1)

Application Number Priority Date Filing Date Title
PCT/JP2020/037184 WO2022070319A1 (ja) 2020-09-30 2020-09-30 情報処理装置、制御方法、非一時的なコンピュータ可読媒体、及び情報処理システム

Publications (1)

Publication Number Publication Date
WO2022070319A1 true WO2022070319A1 (ja) 2022-04-07

Family

ID=80949925

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2020/037184 Ceased WO2022070319A1 (ja) 2020-09-30 2020-09-30 情報処理装置、制御方法、非一時的なコンピュータ可読媒体、及び情報処理システム

Country Status (3)

Country Link
US (1) US20230370798A1 (https=)
JP (1) JP7509220B2 (https=)
WO (1) WO2022070319A1 (https=)

Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002148070A (ja) * 2000-11-14 2002-05-22 Yamaha Motor Co Ltd 位置案内システム、広告システム、音声データ登録端末、位置案内端末及び記憶媒体
JP2004147283A (ja) * 2002-08-27 2004-05-20 Yamaha Corp 音像定位装置、音像定位方法、サウンドデータ配信システム、サウンドデータ配信方法およびプログラム
JP2005311974A (ja) * 2004-04-26 2005-11-04 Yamaha Corp 音像定位端末

Family Cites Families (4)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP6263098B2 (ja) 2014-07-15 2018-01-17 Kddi株式会社 仮想音源を提供情報位置に配置する携帯端末、音声提示プログラム及び音声提示方法
JP6206537B2 (ja) 2016-05-25 2017-10-04 ソニー株式会社 携帯端末、情報処理装置、およびプログラム
JP6674557B2 (ja) 2016-11-08 2020-04-01 ヤマハ株式会社 音声再生装置及び音声再生方法
JP2019126033A (ja) 2018-01-18 2019-07-25 株式会社電通ライブ 音声情報提供システム、音声情報提供装置、及びプログラム

Patent Citations (3)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JP2002148070A (ja) * 2000-11-14 2002-05-22 Yamaha Motor Co Ltd 位置案内システム、広告システム、音声データ登録端末、位置案内端末及び記憶媒体
JP2004147283A (ja) * 2002-08-27 2004-05-20 Yamaha Corp 音像定位装置、音像定位方法、サウンドデータ配信システム、サウンドデータ配信方法およびプログラム
JP2005311974A (ja) * 2004-04-26 2005-11-04 Yamaha Corp 音像定位端末

Also Published As

Publication number Publication date
JPWO2022070319A1 (https=) 2022-04-07
JP7509220B2 (ja) 2024-07-02
US20230370798A1 (en) 2023-11-16

Similar Documents

Publication Publication Date Title
US11647352B2 (en) Head to headset rotation transform estimation for head pose tracking in spatial audio applications
US10429644B2 (en) Data processing
KR102037412B1 (ko) 이동 단말기와 연결된 보청기를 피팅(ｆｉｔｔｉｎｇ) 하는 방법 및 이를 수행하는 이동 단말기
US9554229B2 (en) Amplifying audio-visual data based on user's head orientation
US11586280B2 (en) Head motion prediction for spatial audio applications
US9774978B2 (en) Position determination apparatus, audio apparatus, position determination method, and program
US10051453B2 (en) Wearable and/or hand-held mobile electronic device and method for directional proximity detection
CN110463165B (zh) 信息处理装置、信息处理方法和记录介质
US20190235622A1 (en) Augmented Reality Display Method and Head-Mounted Display Device
US20130055103A1 (en) Apparatus and method for controlling three-dimensional graphical user interface (3d gui)
KR102546949B1 (ko) 전자 장치, 서버 장치 및 전자 장치의 위치를 결정하는 방법
KR20190038034A (ko) 전자 장치 및 그의 지오펜스 관리 방법
US10222876B2 (en) Display control system and method
KR20200144843A (ko) 가변적 지오펜싱 방법 및 이를 위한 장치
JP6816492B2 (ja) 表示制御プログラム、表示制御方法および表示制御装置
JP7509220B2 (ja) 情報処理装置、制御方法、制御プログラム、及び情報処理システム
JP7384222B2 (ja) 情報処理装置、制御方法及びプログラム
JP5811534B2 (ja) 画像表示制御サーバ
JP7501652B2 (ja) 情報処理装置、制御方法、制御プログラム、及び情報処理システム
US12445794B2 (en) Signal processing apparatus and method, acoustic reproduction apparatus, and program
JP7428189B2 (ja) 情報処理装置、制御方法及び制御プログラム
WO2022239638A1 (ja) 副端末と同一の仮想空間へ参入する主端末、プログラム及びシステム
JP2017034479A (ja) 制御装置、再生システム、補正方法、及び、コンピュータプログラム
KR20190100672A (ko) 전자 장치 및 전자 장치 제어 방법
JP2020041897A (ja) 再生装置

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 20956247

Country of ref document: EP

Kind code of ref document: A1

ENP Entry into the national phase

Ref document number: 2022553313

Country of ref document: JP

Kind code of ref document: A

NENP Non-entry into the national phase

Ref country code: DE

122 Ep: pct application non-entry in european phase

Ref document number: 20956247

Country of ref document: EP

Kind code of ref document: A1