US11244682B2 - Information processing device and information processing method - Google Patents

Information processing device and information processing method Download PDF

Info

Publication number
US11244682B2
US11244682B2 US16/631,889 US201816631889A US11244682B2 US 11244682 B2 US11244682 B2 US 11244682B2 US 201816631889 A US201816631889 A US 201816631889A US 11244682 B2 US11244682 B2 US 11244682B2
Authority
US
United States
Prior art keywords
user
information
information processing
output
spoken utterance
Prior art date
Legal status (The legal status is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the status listed.)
Active, expires
Application number
US16/631,889
Other languages
English (en)
Other versions
US20200143813A1 (en
Inventor
Ayumi NAKAGAWA
Takanobu Omata
Current Assignee (The listed assignees may be inaccurate. Google has not performed a legal analysis and makes no representation or warranty as to the accuracy of the list.)
Sony Corp
Original Assignee
Sony Corp
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Application filed by Sony Corp filed Critical Sony Corp
Assigned to SONY CORPORATION reassignment SONY CORPORATION ASSIGNMENT OF ASSIGNORS INTEREST (SEE DOCUMENT FOR DETAILS). Assignors: OMATA, Takanobu, NAKAGAWA, Ayumi
Publication of US20200143813A1 publication Critical patent/US20200143813A1/en
Application granted granted Critical
Publication of US11244682B2 publication Critical patent/US11244682B2/en
Active legal-status Critical Current
Adjusted expiration legal-status Critical

Links

Images

Classifications

    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L21/00—Speech or voice signal processing techniques to produce another audible or non-audible signal, e.g. visual or tactile, in order to modify its quality or its intelligibility
    • G10L21/06—Transformation of speech into a non-audible representation, e.g. speech visualisation or speech processing for tactile aids
    • G10L21/10—Transforming into visible information
    • G10L21/12—Transforming into visible information by displaying time domain information
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G06F3/167—Audio in a user interface, e.g. using voice commands for navigating, audio feedback
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L13/00—Speech synthesis; Text to speech systems
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G10L15/222—Barge in, i.e. overridable guidance for interrupting prompts
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L25/00—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00
    • G10L25/48—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use
    • G10L25/51—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination
    • G10L25/57—Speech or voice analysis techniques not restricted to a single one of groups G10L15/00 - G10L21/00 specially adapted for particular use for comparison or discrimination for processing of video signals
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • G10L2015/223—Execution procedure of a spoken command

Definitions

  • the present disclosure relates to an information processing device, an information processing method, and a computer program.
  • Patent Literature 1 discloses a voice synthesis device for displaying an utterance time related to synthesized voice.
  • Patent Literature 1 Japanese Utility Model Application Laid-open No. S60-3898
  • the voice synthesis device disclosed in Patent Literature 1 enables a user to grasp the length of output voice. However, it is difficult for the technology disclosed in Patent Literature 1 to cause the user to perceive when voice corresponding to information desired by the user is output.
  • the present disclosure proposes a novel and improved information processing device, information processing method, and computer program capable of causing a user to perceive an output position of an important part in information presentation by a spoken utterance.
  • an information processing device includes an output control unit that controls output of a spoken utterance related to information presentation, wherein the output control unit outputs the spoken utterance, and visually displays an output position of an important part of the spoken utterance.
  • an information processing method includes controlling, by a processor, output of a spoken utterance related to information presentation, wherein the controlling further includes outputting the spoken utterance and visually displaying an output position of an important part of the spoken utterance.
  • a computer program causes a computer to function as an information processing device comprising an output control unit that controls output of a spoken utterance related to information presentation, wherein the output control unit outputs the spoken utterance, and visually displays an output position of an important part of the spoken utterance.
  • the present disclosure enables a user to perceive an output position of an important part in information presentation by a spoken utterance.
  • FIG. 1 is a diagram for describing the outline of one embodiment of the present disclosure.
  • FIG. 2 is a block diagram illustrating a system configuration example of an information processing system according to the embodiment.
  • FIG. 3 is a block diagram illustrating a functional configuration example of an information processing terminal 10 according to the embodiment.
  • FIG. 4 is a block diagram illustrating a functional configuration example of an information processing server 20 according to the embodiment.
  • FIG. 5 is a diagram for describing generation of an individual model based on an inquiry utterance according to the embodiment.
  • FIG. 6A is a diagram for describing generation of an individual model based on a response utterance of a user according to the embodiment.
  • FIG. 6B is a diagram for describing generation of an individual model based on a response utterance of a user according to the embodiment.
  • FIG. 7 is a diagram for describing generation of an individual model based on reaction of a user to information presentation according to the embodiment.
  • FIG. 8 is a diagram for describing output control based on a common model according to the embodiment.
  • FIG. 9 is a diagram for describing output control using a common model corresponding to an attribute of a user according to the embodiment.
  • FIG. 10 is a diagram for describing output control corresponding to a plurality of users according to the embodiment.
  • FIG. 11 is a diagram for describing control as to whether operation input can be received according to the embodiment.
  • FIG. 12 is a diagram for describing control as to whether operation input can be received based on the degree of concentration of a user according to the embodiment.
  • FIG. 13 is a diagram for describing display control of the degree of concentration according to the embodiment.
  • FIG. 14 is a flowchart illustrating the flow of processing by the information processing server according to the embodiment.
  • FIG. 15 is a hardware configuration example common to the information processing terminal and the information processing server according to one embodiment of the present disclosure.
  • the outline of one embodiment of the present disclosure is described.
  • various devices for presenting information to users by spoken utterances have become widespread.
  • the devices as described above can present an answer to an inquiry made by an utterance of a user to the user by using voice or visual information.
  • the devices as described above can transmit various kinds of information to users in addition to answers to inquiries.
  • the devices as described above may present recommendation information corresponding to learned preferences of users to the users by spoken utterances and visual information.
  • one feature of an information processing device, an information processing method, and a computer program according to one embodiment of the present disclosure is to output a spoken utterance and visually display an output position of an important part of the spoken utterance.
  • FIG. 1 is a diagram for describing the outline of one embodiment of the present disclosure.
  • FIG. 1 illustrates an example where an information processing terminal 10 presents restaurant information to a user U 1 by using a spoken utterance SO 1 and visual information VI 1 .
  • the information processing terminal 10 may execute the above-mentioned processing based on control by an information processing server 20 described later.
  • the information processing server 20 according to one embodiment of the present disclosure can output information on a restaurant A to the information processing terminal 10 as an answer to an inquiry from the user U 1 .
  • the information processing server 20 may output an output position of an important part of the spoken utterance to the information processing terminal 10 as the visual information VI 1 . More specifically, the information processing server 20 according to the present embodiment outputs the visual information VI 1 including a bar B indicating the entire output length of the spoken utterance SO 1 and a pointer P indicating the current position of the output of the spoken utterance SO 1 to the information processing terminal 10 .
  • the pointer P is information indicating progress of the output of the spoken utterance SO 1 .
  • the information processing server 20 can display an output position of the important part IP of the spoken utterance SO 1 on the bar B.
  • the above-mentioned important part IP may be a section including information that is estimated to have a higher priority for a user in the spoken utterance.
  • Examples of information presentation related to the restaurant A include various kinds of information such as the location, budget, atmosphere, and words of mouth related to the restaurant A.
  • the information processing server 20 estimates information having a higher priority for the user U 1 , and sets a section including information having the high priority in a spoken utterance corresponding to the information presentation as an important part IP.
  • the information processing server 20 can display an output position of the set important part IP on the bar B.
  • the information processing server 20 sets a section including money information that is estimated to have a higher priority for the user U 1 as an important part IP, and sets sections including information having priority that is lower than the money information, such as the location and atmosphere, as non-important parts.
  • the information processing server 20 controls the information processing terminal 10 to output a spoken utterance including the important part IP and the non-important parts, and displays an output position of the important part IP of the spoken utterance.
  • the information processing server 20 can set the priority and the important part based on preferences, characteristics, and attributes of users. For example, the information processing server 20 may calculate the priority for each category of presented information based on preferences, characteristics, and attributes of users, and set a section including information having priority that is equal to or higher than a threshold as an important part. The information processing server 20 may set a section including information having a higher priority in presented information as an important part.
  • the information processing server 20 can set a plurality of important parts. For example, when the priorities of money information and word-of-mouth information are high in the information presentation related to the restaurant A, the information processing server 20 may set section including the money information and the word-of-mouth information in the spoken utterance as important parts.
  • the information processing server 20 enables the user U 1 to visually recognize an output position of the important part IP of the spoken utterance SO 1 . Consequently, the user U 1 can moderately pretend to listen to the spoken utterance SO 1 until the important part IP is output, and perform operation input such as stop processing and a barge-in utterance on the spoken utterance SO 1 after the important part IP is output, thereby more effectively using time.
  • operation input such as stop processing and a barge-in utterance on the spoken utterance SO 1 after the important part IP is output, thereby more effectively using time.
  • FIG. 2 is a block diagram illustrating a system configuration example of the information processing system according to the present embodiment.
  • the information processing system according to the present embodiment includes an information processing terminal 10 and an information processing server 20 .
  • the information processing terminal 10 and the information processing server 20 are connected via a network 30 so as to communicate with each other.
  • the information processing terminal 10 is an information processing device for presenting information using spoken utterances and visual information to a user based on control by the information processing server 20 .
  • one feature of the information processing terminal 10 according to the present embodiment is to visually display an output position of an important part of spoken utterances.
  • the information processing terminal 10 according to the present embodiment can be implemented as various kinds of devices having a voice output function and a display function.
  • the information processing terminal 10 according to the present embodiment may be, for example, a mobile phone, a smartphone, a table, a wearable device, a computer, or a stationary or autonomous dedicated device.
  • the information processing server 20 is an information processing device having a function of controlling output of spoken utterances and visual information by the information processing terminal 10 .
  • one feature of the information processing server 20 according to the present embodiment is to visually display an output position of an important part of a spoken utterance on the information processing terminal 10 .
  • the network 30 has a function of connecting the information processing terminal 10 and the information processing server 20 .
  • the network 30 may include a public line network such as the Internet, a telephone network, or a satellite communication network, and various kinds of local area networks (LANs) and wide area networks (WANs) including Ethernet (registered trademark).
  • the network 30 may include a dedicated line network such as an Internet protocol-virtual private network (IP-VPN).
  • IP-VPN Internet protocol-virtual private network
  • the network 30 may include wireless communication networks such as Wi-Fi (registered trademark) and Bluetooth (registered trademark).
  • the system configuration example of the information processing system according to the present embodiment has been described.
  • the above-mentioned configuration described above with reference to FIG. 2 is merely an example, and the configuration of the information processing system according to the present embodiment is not limited to the example.
  • the functions of the information processing terminal 10 and the information processing server 20 according to the present embodiment may be implemented by a single device.
  • the configuration of the information processing system according to the present embodiment can be flexibly modified depending on specification and operation.
  • FIG. 3 is a block diagram illustrating a functional configuration example of the information processing terminal 10 according to the present embodiment.
  • the information processing terminal 10 according to the present embodiment includes a display unit 110 , a voice output unit 120 , a voice input unit 130 , an imaging unit 140 , a sensor unit 150 , a control unit 160 , and a server communication unit 170 .
  • the display unit 110 according to the present embodiment has a function of outputting visual information such as images and texts.
  • the display unit 110 according to the present embodiment can visually display an output position of an important part of a spoken utterance based on control by the information processing server 20 .
  • the display unit 110 includes a display device for presenting visual information.
  • the display device include a liquid crystal display (LCD) device, an organic light emitting diode (OLED) device, and a touch panel.
  • the display unit 110 according to the present embodiment may output visual information by a projection function.
  • the voice output unit 120 has a function of outputting auditory information including spoken utterances.
  • the voice output unit 120 according to the present embodiment can output an answer to an inquiry from a user by a spoken utterance based on control by the information processing server 20 .
  • the voice output unit 120 according to the present embodiment includes a voice output device such as a speaker and an amplifier.
  • the voice input unit 130 has a function of collecting sound information such as utterances by users and background noise.
  • the sound information collected by the voice input unit 130 is used for voice recognition and behavior recognition by the information processing server 20 .
  • the voice input unit 130 according to the embodiment includes a microphone for collecting sound information.
  • the imaging unit 140 has a function of taking images including users and ambient environments.
  • the images taken by the imaging unit 140 are used for user recognition and behavior recognition by the information processing server 20 .
  • the imaging unit 140 according to the present embodiment includes an imaging device capable of taking images.
  • the above-mentioned images include still images and moving images.
  • the sensor unit 150 has a function of collecting various kinds of sensor information on the behavior of users.
  • the sensor information collected by the sensor unit 150 is used for user state recognition and behavior recognition by the information processing server 20 .
  • the sensor unit 150 includes an acceleration sensor, a gyro sensor, a geomagnetic sensor, a heat sensor, an optical sensor, a vibration sensor, or a global navigation satellite system (GNSS) signal reception device.
  • GNSS global navigation satellite system
  • the control unit 160 has a function of controlling the configurations in the information processing terminal 10 .
  • the control unit 160 controls the start and stop of each configuration.
  • the control unit 160 can input control signals generated by the information processing server 20 to the display unit 110 and the voice output unit 120 .
  • the control unit 160 according to the present embodiment may have the same function as an output control unit 230 as the information processing server 20 described later.
  • the server communication unit 170 has a function of communicating information to the information processing server 20 via the network 30 . Specifically, the server communication unit 170 transmits sound information collected by the voice input unit 130 , image information taken by the imaging unit 140 , and sensor information collected by the sensor unit 150 to the information processing server 20 . The server communication unit 170 receives control signals related to output of visual information and spoken utterances and artificial voice from the information processing server 20 .
  • the functional configuration example of the information processing terminal 10 according to the present embodiment has been described above.
  • the above-mentioned configurations described with reference to FIG. 3 are merely an example, and the functional configurations of the information processing terminal 10 according to the present embodiment are not limited to the example.
  • the information processing terminal 10 according to the present embodiment is not necessarily required to include all the configurations illustrated in FIG. 3 .
  • the information processing terminal 10 may exclude the imaging unit 140 and the sensor unit 150 .
  • the control unit 160 according to the present embodiment may have the same function as the output control unit 230 in the information processing server 20 .
  • the functional configuration of the information processing terminal 10 according to the present embodiment can be flexibly modified depending on specifications and operation.
  • FIG. 4 is a block diagram illustrating a functional configuration example of the information processing server 20 according to the present embodiment.
  • the information processing server 20 according to the present embodiment includes a recognition unit 210 , a setting unit 220 , an output control unit 230 , a voice synthesis unit 240 , a storage unit 250 , and a terminal communication unit 260 .
  • the storage unit 250 includes a user DB 252 , a model DB 254 , and a content DB 256 .
  • the recognition unit 210 has a function of performing various kinds of recognition of users.
  • the recognition unit 210 can recognize users by comparing utterances and images of users collected by the information processing terminal 10 with voice features and images of the users stored in the user DB 252 in advance.
  • the recognition unit 210 can recognize the behavior and state of users based on sound information, image, and sensor information collected by the information processing terminal 10 .
  • the recognition unit 210 can recognize voice based on utterances of users collected by the information processing terminal 10 , and detect inquiries and barge-in utterances of users.
  • the recognition unit 210 can recognize the line of sight, expression, gesture, and behavior of users based on images and sensor information collected by the information processing terminal 10 .
  • the setting unit 220 has a function of setting an important part of a spoken utterance.
  • the setting unit 220 sets a section including information that is estimated to have a higher priority for a user in a spoken utterance as an important part.
  • the setting unit 220 according to the present embodiment may set the priority and the important part based on an individual model set for each user or a common model set for a plurality of users in common.
  • the setting unit 220 can acquire an individual model corresponding to a user recognized by the recognition unit 210 from the model DB 254 described later, and set the priority and the important part.
  • the setting unit 220 may set an important part based on a common model that is common to all users.
  • the setting unit 220 may acquire, based on an attribute of a user recognized by the recognition unit 210 , a common model corresponding to the attribute from a plurality of common models, and set an important part.
  • the setting unit 220 can acquire the common model based on the sex, age, and use language of the user recognized by the recognition unit 210 .
  • the setting unit 220 has a function of generating an individual model based on a response utterance or reaction of a user to a spoken utterance. Details of the function of the setting unit 220 are additionally described later.
  • the output control unit 230 according to the present embodiment has a function of controlling output of spoken utterances related to information presentation.
  • the output control unit 230 according to the present embodiment has a function of outputting a spoken utterance and visually displaying an output position of an important part of the spoken utterance.
  • the output control unit 230 according to the present embodiment may use a bar B and a pointer P to display progress related to the output of the spoken utterance in association with the output position of the important part.
  • the output control unit 230 has a function of controlling whether operation input can be received during the output of the spoken utterance. Details of the function of the output control unit 230 according to the present embodiment are additionally described later.
  • the voice synthesis unit 240 has a function of synthesizing artificial voice output from the information processing terminal 10 based on control by the output control unit 230 .
  • the storage unit 250 includes a user DB 252 , a model DB 254 , and a content DB 256 .
  • the user DB 252 stores therein various kinds of information on users.
  • the user DB 252 stores face images and voice features of users therein.
  • the user DB 252 may store information such as the sex, age, preferences, and tendency of users therein.
  • the model DB 254 stores therein an individual model set for each user and a common model common to a plurality of users.
  • the above-mentioned common model may be a model common to all users, or may be a model set for each attribute of users.
  • the setting unit 220 can acquire a corresponding model from the model DB 254 based on a recognition result of a user by the recognition unit 210 , and set an important part.
  • the content DB 256 stores therein various kinds of contents such as information on restaurants.
  • the output control unit 230 according to the present embodiment can use information stored in the content DB 256 to output an answer to an inquiry from a user, recommendation information, and advertisements by using spoken utterances and visual information.
  • the contents according to the present embodiment are not necessarily stored in the content DB 256 .
  • the output control unit 230 according to the present embodiment may acquire contents from another device through the network 30 .
  • Terminal communication unit 260 Terminal communication unit 260
  • the terminal communication unit 260 has a function of communicating information to the information processing terminal 10 via the network 30 . Specifically, the terminal communication unit 260 receives sound information such as utterances, image information, and sensor information from the information processing terminal 10 . The terminal communication unit 260 transmits control signals generated by the output control unit 230 and artificial voice synthesized by the voice synthesis unit 240 to the information processing terminal 10 .
  • the functional configuration example of the information processing server 20 according to the present embodiment has been described above.
  • the above-mentioned functional configurations described with reference to FIG. 4 are merely an example, and the functional configuration of the information processing server 20 according to the present embodiment is not limited to the example.
  • the information processing server 20 is not necessarily required to include all the configurations illustrated in FIG. 4 .
  • the recognition unit 210 , the setting unit 220 , the voice synthesis unit 240 , and the storage unit 250 may be included in another device different from the information processing server 20 .
  • the functional configuration of the information processing server 20 according to the present embodiment can be flexibly modified depending on specifications and operation.
  • one feature of the information processing server 20 according to the present embodiment is to output a spoken utterance including an important part and a non-important part and visually display an output position of the important part of the spoken utterance.
  • the above-mentioned feature of the information processing server 20 enables a user to clearly perceive an output position of an important part of a spoken utterance to greatly improve the convenience of information presentation using the spoken utterance.
  • the information processing server 20 can use a model corresponding to a recognized user to set the priority and an important part. More specifically, the setting unit 220 according to the present embodiment can acquire, based on the result of recognition by the recognition unit 210 , an individual model set for each user or a common model set for a plurality of users in common, and set the priority and an important part.
  • an important part according to the present embodiment is a section including information that is estimated to have a higher priority for a user in information presented by a spoken utterance.
  • important information that is, information having a high priority
  • one user may be interested in price information but another user may give importance to the atmosphere or location.
  • the information processing server 20 generates an individual model for each user, and sets an important part based on the individual model, thereby being capable of implementing output control corresponding to the need for each user.
  • the setting unit 220 may generate an individual model based on an utterance of a user. Examples of the utterance of the user include an utterance related to an inquiry.
  • FIG. 5 is a diagram for describing the generation of an individual model based on an inquiry utterance according to the present embodiment.
  • FIG. 5 illustrates a situation where a user U 1 makes an inquiry to the information processing terminal 10 by utterances UO 1 and UO 2 . Both the utterances UO 1 and UO 2 related to the inquiry may be requests for information presentation of restaurants.
  • the setting unit 220 can generate an individual model corresponding to the user U 1 based on vocabularies included in the utterances UO 1 and UO 2 of the user U 1 recognized by the recognition unit 210 .
  • the setting unit 220 may estimate that the user U 1 tends to give importance to the price based on a vocabulary “inexpensive store” included in the utterance UO 1 or a vocabulary “budget is 3,000 yen” included in the utterance UO 2 , and generate an individual model reflecting the estimation result.
  • the above-mentioned utterance of the user is not limited to an inquiry.
  • the setting unit 220 may generate an individual model based on a response utterance of the user to an output spoken utterance.
  • FIGS. 6A and 6B are diagrams for describing the generation of an individual model based on a response utterance of a user.
  • FIG. 6A illustrates a situation where the user U 1 makes an utterance UO 3 indicating a response to a spoken utterance SO 2 output by the information processing terminal 10 .
  • the user U 1 makes the utterance UO 3 at a timing at which price information included in the spoken utterance SO 2 is output, and the utterance UO 3 is a barge-in utterance that instructs output of the next restaurant information.
  • the setting unit 220 may estimate that the user U 1 tends to give importance to the price based on the fact that information that has been output when the utterance UO 3 indicating a response is detected is price information and the fact that the utterance UO 3 is a barge-in utterance that instructs output of the next restaurant information, and generate an individual model reflecting the estimation result.
  • FIG. 6B illustrates a situation where the user U 1 makes an utterance UO 4 indicating a response to a spoken utterance SO 3 output by the information processing terminal 10 .
  • the user U 1 makes the utterance UO 4 at a timing at which price information included in the spoken utterance SO 3 is output, and the utterance UO 4 is a barge-in utterance that instructs output of detailed information.
  • the setting unit 220 can estimate that the user U 1 tends to give importance to the price based on the fact that information that has been output when the utterance UO 4 indicating a response is detected is price information and the fact that the utterance UO 4 is a barge-in utterance that instructs output of detailed information, and generate an individual model reflecting the estimation result.
  • the setting unit 220 can estimate an item important for a user and generate an individual model based on a response utterance such as a barge-in utterance and a spoken utterance that is being output.
  • a response utterance of a user to an output spoken utterance can be monitored to estimate an item important for a user with high accuracy.
  • the setting unit 220 may generate an individual model based on a reaction of a user independent from an utterance.
  • FIG. 7 is a diagram for describing the generation of an individual model based on a reaction of a user to information presentation.
  • FIG. 7 illustrates a reaction of the user U 1 to a spoken utterance SO 4 and visual information VI 4 output by the information processing terminal 10 .
  • Examples of the reaction include the expression, line of sight, gesture, and behavior of the user.
  • One example illustrated in FIG. 7 indicates a situation where the user U 1 gazes at price information included in the visual information VI 4 .
  • the setting unit 220 can estimate that the user U 1 tends to give importance to the price based on the fact that the user U 1 gazes at the displayed price information, and generate an individual model reflecting the estimation result.
  • the setting unit 220 may estimate that the user U 1 does not give importance to the location or atmosphere, and reflect the estimation result to an individual model. In this manner, the setting unit 220 according to the present embodiment can generate highly accurate individual models based on various reactions of users.
  • the information processing server 20 may display an output position of an important part based on a common model set for a plurality of users in common.
  • FIG. 8 is a diagram for describing output control based on a common model according to the present embodiment.
  • FIG. 8 illustrates an utterance UO 5 of the user U 1 related to an inquiry about weather, and a spoken utterance SO 5 and visual information VI 5 output by the information processing terminal 10 in response to the utterance UO 5 .
  • the setting unit 220 can set an important part in accordance with a common model common to users.
  • the setting unit 220 sets, as an important part, information corresponding to an answer related to the inquiry from the user U 1 in information included in the spoken utterance SO 5 .
  • the setting unit 220 may set an answer part “10 degrees” corresponding to the utterance UO 5 related to the inquiry in the information included in the spoken utterance SO 5 as an important part.
  • an answer part for an inquiry from a user is set as an important part, thus enabling the user to perceive the output position of information having a higher priority for the user.
  • the information processing server 20 may display an output position of an important part based on a common model corresponding to an attribute of a user. For example, it is assumed that a male user in his 50s and a female user in 20s tend to give importance to different items.
  • the setting unit 220 sets an important part by using a common model corresponding to an attribute of a user recognized by the recognition unit 210 , thus being capable of implementing the highly accurate setting of the important part.
  • FIG. 9 is a diagram for describing output control using a common model corresponding to an attribute of a user according to the present embodiment.
  • the setting unit 220 can estimate that a user with the attribute corresponding to the user U 1 tends to give importance to price information based on a common model acquired based on the sex and age of the user U 1 recognized by the recognition unit 210 , and set the price information as an important part.
  • the above-mentioned function of the setting unit 220 according to the present embodiment enables highly accurate estimation of important parts to be implemented to enhance the convenience for users even when user's personal data is insufficient or absent.
  • the common model corresponding to the attribute of the user may be set in advance or may be generated by diverting the individual model.
  • the setting unit 220 may generate a common model corresponding to the attribute by averaging a plurality of generated individual models for each attribute.
  • the output control based on the individual model and the common model according to the present embodiment has been described above.
  • examples where the output control unit 230 uses a bar B and a pointer P to visually display an output position of an important part have been mainly described.
  • the output control unit 230 according to the present embodiment can perform various kind of output control without being limited to the above-mentioned examples.
  • the output control unit 230 may display a countdown C to more explicitly present a time until voice of an important part IP is output to a user.
  • the output control unit 230 can visually present an important part to a user by including price information “3,000 yen” in the visual information IV 6 in advance.
  • the output control unit 230 may control an output form of a spoken utterance of an important part. For example, in one example illustrated in FIG. 9 , the output control unit 230 emphasizes that an important part is output to the user U 1 by outputting a spoken utterance SO 6 including an emphasized phrase “Listen, deeply”. The output control unit 230 may cause the user U 1 to pay attention by controlling the volume, tone, and rhythm related to the voice output SO 6 .
  • FIG. 10 is a diagram for describing output control corresponding to users according to the present embodiment.
  • FIG. 10 illustrates users U 1 and U 2 and visual information IV 7 and spoken utterances SO 7 and SO 8 output by the information processing terminal 10 .
  • the setting unit 220 acquires models corresponding to the users U 1 and U 2 recognized by the recognition unit 210 from the model DB 254 , and individually sets important parts to the users U 1 and U 2 .
  • the setting unit 220 sets price information as an important part for the user U 1 , and sets location information as an important part for the user U 2 .
  • the output control unit 230 displays output positions of important parts IP 1 and IP 2 corresponding to the users U 1 and U 2 , respectively, based on the degrees of importance set by the setting unit 220 .
  • the information processing server 20 can display the output positions of the important parts corresponding to the respective users based on the individual models corresponding to the users.
  • the above-mentioned function of the information processing server 20 according to the present embodiment enables respective users to grasp when desired information is output, thus being capable of implementing more convenient information presentation.
  • the output control unit 230 may control the output of spoken utterances and visual information depending on the position of a recognized user. For example, in one example illustrated in FIG. 10 , the output control unit 230 outputs the spoken utterance SO 7 including price information in a direction in which the user U 1 is located, and outputs the spoken utterance SO 8 including location information in a direction in which the user U 2 is located.
  • the output control unit 230 can implement the above-mentioned processing by controlling a beamforming function of the voice output unit 120 .
  • the output control unit 230 may display the price information, which is important for the user U 1 , at a position that is easily visually recognized by the user U 1 , and display the location information, which is important for the user U 2 , at a position that is easily visually recognized by the user U 2 .
  • the above-mentioned function of the output control unit 230 according to the present embodiment enables a user to more easily perceive information related to an important part to enhance the convenience of information presentation.
  • the output control unit 230 may control whether operation input can be received during output of a spoken utterance depending on output positions of important parts related to users.
  • FIG. 11 is a diagram for describing control as to whether operation input can be received according to the present embodiment. Similarly to the case in FIG. 10 , FIG. 11 illustrates the output positions of the important parts IP 1 and IP 2 corresponding to the users U 1 and U 2 , respectively, as visual information V 18 .
  • the user U 1 makes an utterance UO 6 that instructs output of the next restaurant information at a timing at which the output of the important part IP 1 corresponding to price information that is important for the user U 1 is finished.
  • voice output of the important part IP 2 corresponding to the location information that is important for the user U 2 has not been completed at the above-mentioned timing.
  • the output control unit 230 may control not to receive operation input made by the user U 1 until the voice output corresponding to the important part IP 2 corresponding to the user U 2 is completed. Specifically, the output control unit 230 according to the present embodiment refuses receiving the operation input made by a second user (user U 1 ) before or during output of an important part corresponding to a first user (user U 2 ), so that the first user can be prevented from failing to hear a spoken utterance corresponding to the important part.
  • the output control unit 230 may display an icon I 1 to explicitly indicate that operation input cannot be received.
  • the above-mentioned operation input includes a barge-in utterance as illustrated in FIG. 11 and stop processing of information output made by a button operation.
  • the above-mentioned function of the output control unit 230 according to the present embodiment can effectively prevent cut-in processing by another user before voice output of an important part is completed.
  • the output control unit 230 may control output of spoken utterances and visual information based on operation input detected before an important part is output. For example, in the case where the user U 1 makes the utterance UO 6 , which is a barge-in utterance, at the timing illustrated in FIG. 11 , the output control unit 230 may shift to presentation of the next restaurant information after voice output corresponding to the important part IP 2 is completed based on the fact that the voice output of the important part IP 2 has not been completed. The output control unit 230 may shift to presentation of the next restaurant information after location information corresponding to the important part IP 2 is displayed as the visual information V 18 .
  • the output control unit 230 may control as to whether the operation input can be received based on the degree of concentration of the first user.
  • FIG. 12 is a diagram for describing control as whether operation input can be received based on the degree of concentration of a user according to the present embodiment.
  • the user U 1 similarly to the case in FIG. 11 , the user U 1 makes an utterance UO 7 that instructs output of the next restaurant information at the timing at which the voice output of the important part IP 1 corresponding to price information that is important for the user U 1 is finished.
  • the output control unit 230 may receive operation input made by the user U 1 and shift to presentation of the next restaurant information based on the fact that it is detected that the degree of concentration of the user U 2 is low during the voice output of the important part IP 2 .
  • the above-mentioned function of the output control unit 230 according to the present embodiment can eliminate the influence of a user who is not concentrated in a spoken utterance and preferentially receive an instruction from another user to efficiently improve the overall convenience.
  • the output control unit 230 may visually display the degree of concentration of a user detected by the recognition unit 210 .
  • FIG. 13 is a diagram for describing display control of the degree of concentration according to the present embodiment.
  • FIG. 13 illustrates visual information VI 10 displayed in a virtual space shared by users. In this manner, for example, the output control unit 230 according to the present embodiment can implement output control of the visual information VI 10 displayed by a head-mount display information processing terminal 10 .
  • the output control unit 230 may control output of information presentation by a virtual character C. Specifically, the output control unit 230 controls visual information related to the virtual character C and output of a spoken utterance SO 11 corresponding to a script of the virtual character C. The output control unit 230 displays output positions of the important parts IP 1 and IP 2 in the spoken utterance SO 11 by using the bar B and the pointer P.
  • the output control unit 230 displays an avatar A in association with an icon 12 indicating the degree of concentration of a user corresponding to the avatar A, so that other users can be caused to perceive the degree of concentration corresponding to the spoken utterance SO 11 of the user corresponding to the avatar A.
  • the user U 2 can grasp that the user U 1 is concentrated in the voice output corresponding to the important part IP 1 by visually recognizing the icon 12 .
  • the above-mentioned function of the output control unit 230 can cause a user to visually grasp the degree of concentration of another user to prevent unintentional hindrance of listening behavior of the other user.
  • FIG. 14 is a flowchart illustrating the flow of processing by the information processing server 20 according to the present embodiment.
  • the terminal communication unit 260 in the information processing server 20 receives collection information collected by the information processing terminal 10 (S 1101 ).
  • the above-mentioned collection information includes sound information including utterances of a user, image information including the user, and sensor information related to the user.
  • the recognition unit 210 recognizes the user based on the collection information received at Step S 1101 (S 1102 ).
  • the recognition unit 210 may continuously recognize the state or behavior of the user to calculate the degree of concentration.
  • the setting unit 220 acquires a model corresponding to the user recognized at Step S 1102 from the model DB 254 (S 1103 ).
  • the setting unit 220 may acquire an individual model corresponding to the user specified at Step S 1102 , or may acquire a common model corresponding to an attribute of the recognized user.
  • the setting unit 220 sets an important part of a spoken utterance based on the model acquired at Step S 1103 (S 1104 ).
  • the output control unit 230 controls the voice synthesis unit 240 to synthesize artificial voice corresponding to the spoken utterance including the important part set at Step S 1104 (S 1105 ).
  • the output control unit 230 controls the output of the spoken utterance, and displays an output position of the important part calculated based on the important part set at Step S 1103 and the artificial voice synthesized at Step S 1105 on the display unit 110 (S 1106 ).
  • the output control unit 230 controls whether operation input by the user can be received (S 1107 ).
  • the setting unit 220 updates the corresponding model based on the response utterance or the reaction (S 1108 ).
  • FIG. 15 is a block diagram illustrating a hardware configuration example of the information processing terminal 10 and the information processing server 20 according to one embodiment of the present disclosure.
  • the information processing terminal 10 and the information processing server 20 include, for example, a CPU 871 , a ROM 872 , a RAM 873 , a host bus 874 , a bridge 875 , an external bus 876 , an interface 877 , an input device 878 , an output device 879 , storage 880 , a drive 881 , a connection port 882 , and a communication device 883 .
  • the illustrated hardware configuration is an example, and a part of the components may be omitted.
  • the hardware configuration may further include a component other than the illustrated components.
  • the CPU 871 functions as, for example, an arithmetic processing unit or a control device, and controls the overall or partial operation of each component based on various control programs recorded in the ROM 872 , the RAM 873 , the storage 880 , or a removable recording medium 901 .
  • the ROM 872 is means for storing therein computer programs read onto the CPU 871 and data used for calculation.
  • computer programs read onto the CPU 871 and various parameters that change as appropriate when the computer programs are executed are temporarily or permanently stored.
  • the CPU 871 , the ROM 872 , and the RAM 873 are mutually connected through the host bus 874 capable of high-speed data transmission.
  • the host bus 874 is connected to the external bus 876 the data transmission speed of which is relatively low through the bridge 875 .
  • the external bus 876 is connected to various components through the interface 877 .
  • the input device 878 for example, a mouse, a keyboard, a touch panel, a button, a switch, and a lever are used.
  • a remote controller capable of transmitting control signals by using infrared rays or other radio waves may be used.
  • the input device 878 includes a voice input device such as a microphone.
  • the output device 879 is a device capable of visually or aurally notifying a user of acquired information, for example, a display device such as a cathode ray tube (CRT), an LCD, or an organic EL, an audio output device such as a speaker or headphones, a printer, a mobile phone, or a facsimile machine.
  • a display device such as a cathode ray tube (CRT), an LCD, or an organic EL
  • an audio output device such as a speaker or headphones
  • printer a printer
  • a mobile phone or a facsimile machine.
  • the output device 879 according to the present disclosure includes various vibration devices capable of outputting tactile stimulation.
  • the storage 880 is a device for storing various kinds of data therein.
  • a magnetic storage device such as a hard disk drive (HDD), a semiconductor storage device, an optical storage device, or a magnetooptical storage device is used.
  • the drive 881 is, for example, a device for reading information recorded in the removable recording medium 901 such as a magnetic disk, an optical disc, a magnetooptical disk, or a semiconductor memory, or writing information into the removable recording medium 901 .
  • the removable recording medium 901 is, for example, a DVD medium, a Blu-ray (registered trademark) medium, an HD DVD medium, or various kinds of semiconductor storage media. It should be understood that the removable recording medium 901 may be, for example, an IC card having a noncontact IC chip mounted thereon, or an electronic device.
  • connection port 882 is, for example, a port for connecting the external connection device 902 , such as a universal serial bus (USB) port, an IEEE 1394 port, a small computer system interface (SCSI), an RS-232C port, or an optical audio terminal.
  • USB universal serial bus
  • SCSI small computer system interface
  • RS-232C small computer system interface
  • optical audio terminal optical audio terminal
  • the external connection device 902 is, for example, a printer, a portable music player, a digital camera, a digital video camera, or an IC recorder.
  • the communication device 883 is a communication device for connection to a network, and is, for example, a communication card for wired or wireless LAN, Bluetooth (registered trademark), or wireless USB (WUSB), a router for optical communication, a router for asymmetric digital subscriber line (ADSL), or a modem for various kinds of communication.
  • a communication card for wired or wireless LAN Bluetooth (registered trademark), or wireless USB (WUSB)
  • WUSB wireless USB
  • a router for optical communication a router for asymmetric digital subscriber line (ADSL), or a modem for various kinds of communication.
  • ADSL asymmetric digital subscriber line
  • the information processing server 20 can output a spoken utterance to the information processing terminal 10 , and visually display an output position of an important part of the spoken utterance. This configuration enables a user to perceive the output position of the important part in information presentation made by the spoken utterance.
  • the steps related to the processing of the information processing server 20 herein are not necessarily required to be processed in chronological order described in the flowchart.
  • the steps related to the processing of the information processing server 20 may be processed in an order different from the order described in the flowchart, or may be processed in parallel.
  • An information processing device comprising an output control unit that controls output of a spoken utterance related to information presentation, wherein
  • the output control unit outputs the spoken utterance, and visually displays an output position of an important part of the spoken utterance.
  • the information processing device wherein the spoken utterance includes:
  • a non-important part including information having priority that is lower than the priority of the important part.
  • the information processing device according to (1) or (2), wherein the output control unit outputs progress related to the output of the spoken utterance in association with the output position of the important part.
  • the information processing device according to any one of (1) to (3), wherein the output control unit displays the output position of the important part based on an individual model set for each of users.
  • the information processing device according to (4), wherein the output control unit displays the output position of the important part corresponding to each of the users based on the individual model related to each of the users.
  • the information processing device according to any one of (1) to (3), wherein the output control unit displays the output position of the important part based on a common model set for a plurality of users in common.
  • the information processing device according to (6), wherein the output control unit displays the output position of the important part based on the common model corresponding to an attribute of the user.
  • the information processing device according to any one of (1) to (7), wherein the output control unit controls whether to receive an operation input during the output of the spoken utterance.
  • the information processing device wherein the output control unit refuses receiving the operation input made by a second user before or during output of the important part corresponding to a first user.
  • the information processing device wherein, when the output control unit detects the operation input made by a second user during output of the important part corresponding to a first user, the output control unit controls whether to receive the operation input based on a degree of concentration of the first user.
  • the information processing device according to any one of (8) to (10), wherein the output control unit controls output of at least one of the spoken utterance and visual information based on the operation input detected before or during output of the important part.
  • the information processing device according to any one of (8) to (11), wherein the operation input includes a barge-in utterance.
  • the information processing device according to (4) or (5), wherein the individual model is generated based on an utterance of the user.
  • the information processing device according to (4), 5, or 13, wherein the individual model is generated based on reaction of the user to the information presentation.
  • the information processing device according to any one of (1) to (14), further comprising a setting unit that sets the important part based on a recognized user.
  • the information processing device wherein the setting unit generates an individual model corresponding to each of the users.
  • the information processing device according to any one of (1) to (16), further comprising a display unit that displays the output position of the important part based on control by the output control unit.
  • the information processing device according to any one of (1) to (17), further comprising a voice output unit that outputs the spoken utterance based on control by the output control unit.
  • An information processing method comprising controlling, by a processor, output of a spoken utterance related to information presentation, wherein
  • the controlling further includes outputting the spoken utterance and visually displaying an output position of an important part of the spoken utterance.
  • a computer program for causing a computer to function as an information processing device comprising an output control unit that controls output of a spoken utterance related to information presentation, wherein
  • the output control unit outputs the spoken utterance, and visually displays an output position of an important part of the spoken utterance.

Landscapes

  • Engineering & Computer Science (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Human Computer Interaction (AREA)
  • Physics & Mathematics (AREA)
  • Multimedia (AREA)
  • Computational Linguistics (AREA)
  • Acoustics & Sound (AREA)
  • Theoretical Computer Science (AREA)
  • Signal Processing (AREA)
  • General Physics & Mathematics (AREA)
  • General Engineering & Computer Science (AREA)
  • General Health & Medical Sciences (AREA)
  • Data Mining & Analysis (AREA)
  • Quality & Reliability (AREA)
  • User Interface Of Digital Computer (AREA)
  • Information Retrieval, Db Structures And Fs Structures Therefor (AREA)
US16/631,889 2017-07-26 2018-04-23 Information processing device and information processing method Active 2038-07-20 US11244682B2 (en)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
JP2017-144362 2017-07-26
JP2017144362 2017-07-26
JPJP2017-144362 2017-07-26
PCT/JP2018/016400 WO2019021553A1 (ja) 2017-07-26 2018-04-23 情報処理装置、情報処理方法、およびプログラム

Publications (2)

Publication Number Publication Date
US20200143813A1 US20200143813A1 (en) 2020-05-07
US11244682B2 true US11244682B2 (en) 2022-02-08

Family

ID=65039515

Family Applications (1)

Application Number Title Priority Date Filing Date
US16/631,889 Active 2038-07-20 US11244682B2 (en) 2017-07-26 2018-04-23 Information processing device and information processing method

Country Status (3)

Country Link
US (1) US11244682B2 (de)
EP (1) EP3660838A4 (de)
WO (1) WO2019021553A1 (de)

Families Citing this family (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
CN110959174A (zh) * 2017-08-01 2020-04-03 索尼公司 信息处理装置、信息处理方法以及程序
US11004449B2 (en) * 2018-11-29 2021-05-11 International Business Machines Corporation Vocal utterance based item inventory actions
US11520079B2 (en) 2019-02-01 2022-12-06 Vidubly Ltd Personalizing weather forecast
US11202131B2 (en) * 2019-03-10 2021-12-14 Vidubly Ltd Maintaining original volume changes of a character in revoiced media stream
CN110125946B (zh) * 2019-04-23 2021-08-27 北京淇瑀信息科技有限公司 自动通话方法、装置、电子设备及计算机可读介质

Citations (14)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US20050033582A1 (en) * 2001-02-28 2005-02-10 Michael Gadd Spoken language interface
US7412389B2 (en) * 2005-03-02 2008-08-12 Yang George L Document animation system
US7743340B2 (en) * 2000-03-16 2010-06-22 Microsoft Corporation Positioning and rendering notification heralds based on user's focus of attention and activity
EP2442243A1 (de) 2010-10-18 2012-04-18 Samsung Electronics Co., Ltd. Verfahren zur Bereitstellung eines Suchdienstes und Anzeigevorrichtung damit
US20120197645A1 (en) 2011-01-31 2012-08-02 Midori Nakamae Electronic Apparatus
WO2013044071A1 (en) 2011-09-23 2013-03-28 Amazon Technologies Inc. Visual representation of supplemental information for a digital work
US20140051042A1 (en) 2012-08-16 2014-02-20 Kabushiki Kaisha Toshiba Speech learning apparatus and method
EP2874404A1 (de) 2013-11-15 2015-05-20 LG Electronics, Inc. Videoanzeigevorrichtung und Betriebsverfahren dafür
US9449526B1 (en) 2011-09-23 2016-09-20 Amazon Technologies, Inc. Generating a game related to a digital work
US9613003B1 (en) 2011-09-23 2017-04-04 Amazon Technologies, Inc. Identifying topics in a digital work
US9639518B1 (en) 2011-09-23 2017-05-02 Amazon Technologies, Inc. Identifying entities in a digital work
US20190108897A1 (en) * 2016-03-02 2019-04-11 Ikoob Co., Ltd Apparatus and method for assisting medical consultation
US10489449B2 (en) * 2002-05-23 2019-11-26 Gula Consulting Limited Liability Company Computer accepting voice input and/or generating audible output
US11120797B2 (en) * 2017-11-15 2021-09-14 Sony Corporation Information processing terminal, information processing apparatus, and information processing method

Family Cites Families (1)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
JPS603898U (ja) 1983-06-20 1985-01-12 三洋電機株式会社 音声合成装置

Patent Citations (30)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
US7743340B2 (en) * 2000-03-16 2010-06-22 Microsoft Corporation Positioning and rendering notification heralds based on user's focus of attention and activity
US20050033582A1 (en) * 2001-02-28 2005-02-10 Michael Gadd Spoken language interface
US10489449B2 (en) * 2002-05-23 2019-11-26 Gula Consulting Limited Liability Company Computer accepting voice input and/or generating audible output
US7412389B2 (en) * 2005-03-02 2008-08-12 Yang George L Document animation system
EP2442243A1 (de) 2010-10-18 2012-04-18 Samsung Electronics Co., Ltd. Verfahren zur Bereitstellung eines Suchdienstes und Anzeigevorrichtung damit
US20120095983A1 (en) 2010-10-18 2012-04-19 Samsung Electronics Co., Ltd. Method of providing search service and display device applying the same
KR20120039921A (ko) 2010-10-18 2012-04-26 삼성전자주식회사 검색 서비스 제공방법 및 이를 적용한 디스플레이 장치
US20130311187A1 (en) 2011-01-31 2013-11-21 Kabushiki Kaisha Toshiba Electronic Apparatus
US20120197645A1 (en) 2011-01-31 2012-08-02 Midori Nakamae Electronic Apparatus
JP2012159683A (ja) 2011-01-31 2012-08-23 Toshiba Corp 電子機器
CN103999075A (zh) 2011-09-23 2014-08-20 亚马逊技术股份有限公司 数字作品的补充信息的视觉表示
US9449526B1 (en) 2011-09-23 2016-09-20 Amazon Technologies, Inc. Generating a game related to a digital work
WO2013044071A1 (en) 2011-09-23 2013-03-28 Amazon Technologies Inc. Visual representation of supplemental information for a digital work
EP2758894A1 (de) 2011-09-23 2014-07-30 Amazon Technologies, Inc. Visuelle darstellung von ergänzenden informationen für eine digitale arbeit
US20130080881A1 (en) 2011-09-23 2013-03-28 Joshua M. Goodspeed Visual representation of supplemental information for a digital work
US8842085B1 (en) 2011-09-23 2014-09-23 Amazon Technologies, Inc. Providing supplemental information for a digital work
JP2014531671A (ja) 2011-09-23 2014-11-27 アマゾン テクノロジーズ インコーポレイテッド デジタル作品のための補足情報の視覚表現
US9639518B1 (en) 2011-09-23 2017-05-02 Amazon Technologies, Inc. Identifying entities in a digital work
US9613003B1 (en) 2011-09-23 2017-04-04 Amazon Technologies, Inc. Identifying topics in a digital work
US9471547B1 (en) 2011-09-23 2016-10-18 Amazon Technologies, Inc. Navigating supplemental information for a digital work
US9128581B1 (en) 2011-09-23 2015-09-08 Amazon Technologies, Inc. Providing supplemental information for a digital work in a user interface
US20140051042A1 (en) 2012-08-16 2014-02-20 Kabushiki Kaisha Toshiba Speech learning apparatus and method
JP5634455B2 (ja) 2012-08-16 2014-12-03 株式会社東芝 音声学習装置、方法およびプログラム
JP2014038209A (ja) 2012-08-16 2014-02-27 Toshiba Corp 音声学習装置、方法およびプログラム
CN104661078A (zh) 2013-11-15 2015-05-27 Lg电子株式会社 视频显示装置及其操作方法
KR20150056394A (ko) 2013-11-15 2015-05-26 엘지전자 주식회사 영상 표시 장치 및 그 동작 방법
US20150139606A1 (en) 2013-11-15 2015-05-21 Lg Electronics Inc. Video display device and method for operating the same
EP2874404A1 (de) 2013-11-15 2015-05-20 LG Electronics, Inc. Videoanzeigevorrichtung und Betriebsverfahren dafür
US20190108897A1 (en) * 2016-03-02 2019-04-11 Ikoob Co., Ltd Apparatus and method for assisting medical consultation
US11120797B2 (en) * 2017-11-15 2021-09-14 Sony Corporation Information processing terminal, information processing apparatus, and information processing method

Non-Patent Citations (3)

* Cited by examiner, † Cited by third party
Title
Extended European Search Report of EP Application No. 18839071.0, dated Aug. 10, 2020, 08 pages.
International Search Report and Written Opinion of PCT Application No. PCT/JP2018/016400, dated Jul. 3, 2018, 10 pages of ISRWO.
Takemura, et al., "Ambient Interface, It's Goal and Strategies for the Realization", Journal of Human Interface Society: human interface, vol. 11, No. 4, Nov. 25, 2009, pp. 15-20.

Also Published As

Publication number Publication date
EP3660838A4 (de) 2020-09-09
EP3660838A1 (de) 2020-06-03
WO2019021553A1 (ja) 2019-01-31
US20200143813A1 (en) 2020-05-07

Similar Documents

Publication Publication Date Title
US11244682B2 (en) Information processing device and information processing method
AU2018241137B2 (en) Dynamic thresholds for always listening speech trigger
KR102089487B1 (ko) 디지털 어시스턴트 서비스의 원거리 확장
CN105320736B (zh) 用于提供信息的装置和方法
KR101617665B1 (ko) 핸즈-프리 상호작용을 위한 자동 적응식 사용자 인터페이스
JP2019197499A (ja) プログラム、記録媒体、拡張現実感提示装置及び拡張現実感提示方法
JPWO2019098038A1 (ja) 情報処理装置、及び情報処理方法
US20130159853A1 (en) Managing playback of supplemental information
CN111919248B (zh) 用于处理用户发声的系统及其控制方法
JP6973380B2 (ja) 情報処理装置、および情報処理方法
WO2015104883A1 (ja) 情報処理装置、情報処理方法およびプログラム
WO2018139036A1 (ja) 情報処理装置、情報処理方法およびプログラム
WO2016052501A1 (ja) ユーザインタフェース装置、プログラム及びコンテンツの通知方法
US11688268B2 (en) Information processing apparatus and information processing method
US20200335087A1 (en) Information processing apparatus, information processing method, and program
US20200234187A1 (en) Information processing apparatus, information processing method, and program
US11238846B2 (en) Information processing device and information processing method
US20210224066A1 (en) Information processing device and information processing method
CN116437155A (zh) 一种直播互动方法、装置、计算机设备和存储介质
JP6950708B2 (ja) 情報処理装置、情報処理方法、および情報処理システム
US11430429B2 (en) Information processing apparatus and information processing method
US11935449B2 (en) Information processing apparatus and information processing method
WO2018139050A1 (ja) 情報処理装置、情報処理方法およびプログラム
US11487409B2 (en) Appearance configuration of information processing terminal
WO2019054009A1 (ja) 情報処理装置、情報処理方法、およびプログラム

Legal Events

Date Code Title Description
FEPP Fee payment procedure

Free format text: ENTITY STATUS SET TO UNDISCOUNTED (ORIGINAL EVENT CODE: BIG.); ENTITY STATUS OF PATENT OWNER: LARGE ENTITY

STPP Information on status: patent application and granting procedure in general

Free format text: DOCKETED NEW CASE - READY FOR EXAMINATION

AS Assignment

Owner name: SONY CORPORATION, JAPAN

Free format text: ASSIGNMENT OF ASSIGNORS INTEREST;ASSIGNORS:NAKAGAWA, AYUMI;OMATA, TAKANOBU;SIGNING DATES FROM 20200114 TO 20200120;REEL/FRAME:052259/0295

STPP Information on status: patent application and granting procedure in general

Free format text: NON FINAL ACTION MAILED

STPP Information on status: patent application and granting procedure in general

Free format text: RESPONSE TO NON-FINAL OFFICE ACTION ENTERED AND FORWARDED TO EXAMINER

STPP Information on status: patent application and granting procedure in general

Free format text: NOTICE OF ALLOWANCE MAILED -- APPLICATION RECEIVED IN OFFICE OF PUBLICATIONS

STPP Information on status: patent application and granting procedure in general

Free format text: PUBLICATIONS -- ISSUE FEE PAYMENT VERIFIED

STCF Information on status: patent grant

Free format text: PATENTED CASE