WO2023248677A1 - Dispositif de commande d'affichage d'informations et procédé d'affichage d'informations - Google Patents

Dispositif de commande d'affichage d'informations et procédé d'affichage d'informations Download PDF

Info

Publication number
WO2023248677A1
WO2023248677A1 PCT/JP2023/019082 JP2023019082W WO2023248677A1 WO 2023248677 A1 WO2023248677 A1 WO 2023248677A1 JP 2023019082 W JP2023019082 W JP 2023019082W WO 2023248677 A1 WO2023248677 A1 WO 2023248677A1
Authority
WO
WIPO (PCT)
Prior art keywords
display
unit
information
reference information
displayed
Prior art date
Application number
PCT/JP2023/019082
Other languages
English (en)
Japanese (ja)
Inventor
功大 中村
廉 渡辺
博一 小池
純一 糟谷
麻梨子 楠美
早紀 内田
雄大 櫻田
Original Assignee
株式会社Ｊｖｃケンウッド
Priority date (The priority date is an assumption and is not a legal conclusion. Google has not performed a legal analysis and makes no representation as to the accuracy of the date listed.)
Filing date
Publication date
Priority claimed from JP2022100400A external-priority patent/JP2024001624A/ja
Priority claimed from JP2022114215A external-priority patent/JP2024011896A/ja
Application filed by 株式会社Ｊｖｃケンウッド filed Critical 株式会社Ｊｖｃケンウッド
Publication of WO2023248677A1 publication Critical patent/WO2023248677A1/fr

Links

Images

Classifications

    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F16/00—Information retrieval; Database structures therefor; File system structures therefor
    • G06F16/90—Details of database functions independent of the retrieved data types
    • G06F16/903—Querying
    • G06F16/9038—Presentation of query results
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/01—Input arrangements or combined input and output arrangements for interaction between user and computer
    • G06F3/048—Interaction techniques based on graphical user interfaces [GUI]
    • G06F3/0484—Interaction techniques based on graphical user interfaces [GUI] for the control of specific functions or operations, e.g. selecting or manipulating an object, an image or a displayed text element, setting a parameter value or selecting a range
    • G—PHYSICS
    • G06—COMPUTING; CALCULATING OR COUNTING
    • G06F—ELECTRIC DIGITAL DATA PROCESSING
    • G06F3/00—Input arrangements for transferring data to be processed into a form capable of being handled by the computer; Output arrangements for transferring data from processing unit to output unit, e.g. interface arrangements
    • G06F3/16—Sound input; Sound output
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/08—Speech classification or search
    • G10L15/10—Speech classification or search using distance or distortion measures between unknown speech and reference templates
    • G—PHYSICS
    • G10—MUSICAL INSTRUMENTS; ACOUSTICS
    • G10L—SPEECH ANALYSIS TECHNIQUES OR SPEECH SYNTHESIS; SPEECH RECOGNITION; SPEECH OR VOICE PROCESSING TECHNIQUES; SPEECH OR AUDIO CODING OR DECODING
    • G10L15/00—Speech recognition
    • G10L15/22—Procedures used during a speech recognition process, e.g. man-machine dialogue
    • H—ELECTRICITY
    • H04—ELECTRIC COMMUNICATION TECHNIQUE
    • H04N—PICTORIAL COMMUNICATION, e.g. TELEVISION
    • H04N7/00—Television systems
    • H04N7/14—Systems for two-way working
    • H04N7/15—Conference systems

Definitions

  • the present invention relates to an information display control device and an information display method.
  • Patent Document 1 There is a known technology for displaying additional information on words and phrases uttered by the other party in a meeting or conversation (for example, see Patent Document 1).
  • the additional information to be displayed is information that provides explanations for abbreviations and difficult keywords. However, depending on the level of understanding of the person receiving the additional information, it may be necessary to explain keywords for which additional information is not displayed. In this case, it may be difficult for the receiver to understand the information. Alternatively, a keyword for which additional information is displayed may not require an explanation. In this case, additional information is displayed even for keywords that do not require additional information on the receiving side, which may impede the flow of communication.
  • the present disclosure has been made in view of the above, and aims to appropriately display reference information from keywords based on utterance content.
  • an information display control device that controls an information display device to display information on a display unit, and which controls the information display device to display information on a display unit, and an operation control unit that detects an operation on the display screen displayed on the display unit; an extraction unit that extracts a keyword requiring reference information from the utterance recognized by the speech recognition unit;
  • the display control unit includes: an information acquisition unit that acquires the reference information for the keyword extracted by the extraction unit; and a display control unit that displays the reference information acquired by the information acquisition unit on the display unit.
  • An information display method is an information display method executed by an information display device that displays information on a display unit, and includes an utterance recognition step of recognizing an utterance of a speaker, and a display screen displayed on the display unit.
  • a display control step of displaying the reference information acquired in the information acquisition step on the display section, and the display control step includes a display range related to the utterance displayed on the display section that is controlled by the operation control step. If it is detected that the keyword has been detected, reference information for the keyword that requires the reference information extracted in the extraction step immediately before the time of detection is displayed on the display unit.
  • reference information can be appropriately displayed from a keyword based on the content of the utterance.
  • FIG. 1 is a block diagram showing a configuration example of an information display device having an information display control device according to the first embodiment.
  • FIG. 2 is a schematic diagram showing an example of how the information display device is used.
  • FIG. 3 is a schematic diagram showing another example of how the information display device is used.
  • FIG. 4 is a flowchart showing an example of the flow of processing in the information display control device according to the first embodiment.
  • FIG. 5 is a diagram showing an example in which the speaker display range is selected.
  • FIG. 6 is a flowchart illustrating an example of the flow of processing in the information display control device according to the second embodiment.
  • FIG. 7 is a diagram showing an example in which the speaker display range is divided into a plurality of sections.
  • FIG. 1 is a block diagram showing a configuration example of an information display device having an information display control device according to the first embodiment.
  • FIG. 2 is a schematic diagram showing an example of how the information display device is used.
  • FIG. 3 is a schematic diagram showing another example of
  • FIG. 8 is a diagram showing an example in which the reference information display range is divided into a plurality of sections.
  • FIG. 9 is a schematic diagram illustrating an example of keywords extracted immediately before the operation time.
  • FIG. 10 is a flowchart illustrating an example of the flow of processing in the information display control device according to the third embodiment.
  • FIG. 11 is a flowchart illustrating an example of the flow of processing in the information display control device according to the fourth embodiment.
  • FIG. 12 is a schematic diagram showing an example of how the information display device is used.
  • FIG. 13 is a schematic diagram showing another example of how the information display device is used.
  • FIG. 14 is a block diagram showing a configuration example of an information display device having an information display control device according to the fifth embodiment.
  • FIG. 15 is a diagram showing an example of keyword information.
  • FIG. 16 is a flowchart illustrating an example of the flow of processing in the information display control device according to the fifth embodiment.
  • FIG. 17 is a flowchart illustrating an example of the flow of processing in the information display control device according to the sixth embodiment.
  • FIG. 18 is a flowchart illustrating an example of the flow of processing in the information display control device according to the seventh embodiment.
  • FIG. 1 is a block diagram showing a configuration example of an information display device 10 having an information display control device (hereinafter referred to as "control device") 20 according to the first embodiment.
  • the information display device 10 displays reference information of keywords included in utterances of participants in at least utterance communication such as a meeting or conversation.
  • the information display device 10 is used in a conference held online (hereinafter referred to as a "WEB conference") or a conversation in which a plurality of participants participate.
  • the information display device 10 used in this case is, for example, an information terminal device such as a notebook personal computer or a smartphone used when participating in an online web conference or conversation via a web conference system (not shown). It may be implemented as one of the functions.
  • the information display device 10 displays keyword reference information based on the content of the participants' utterances during the conference.
  • the information display device 10 when it is detected that the display range related to utterances displayed on the display unit 12 has been manipulated in a meeting, the information display device 10 provides reference information for the keyword extracted immediately before the detected manipulation time. Display.
  • Immediately before the operation time refers to a predetermined retrospective period from the operation time, such as within 5 seconds from the operation time.
  • the information display device 10 includes a microphone 11, a display section 12, an operation section 17, a keyword information storage section 18, a communication section 19, and a control device 20.
  • the information display device 10 used for the web conference further includes a camera (not shown).
  • the microphone 11 is a microphone that picks up the voices of participants in the conference who are using the information display device 10.
  • the microphone 11 outputs the collected audio data to the speech recognition unit 21 of the control device 20 .
  • the microphone 11 picks up the voice of a user who is a participant and uses the information display device 10 .
  • FIG. 2 is a schematic diagram showing an example of how the information display device is used.
  • FIG. 3 is a schematic diagram showing another example of how the information display device is used.
  • the display unit 12 is a display including, for example, a liquid crystal display or an organic EL (Electro-Luminescence) display.
  • the display unit 12 displays video based on the video signal output from the display control unit 28 of the control device 20.
  • the display unit 12 displays reference information corresponding to keywords included in the utterances of conference participants.
  • the range in which reference information is displayed on the display unit 12 is referred to as a reference information display range 110.
  • the reference information display range 110 may be displayed on a part of the display unit 12, for example, at the bottom, top, left side, or right side.
  • the reference information display range 110 may be displayed all the time even when reference information is not displayed.
  • the reference information display range 110 may be displayed as a pop-up on the display unit 12.
  • the reference information display range 110 is, for example, a range in which about three lines or less of characters are displayed.
  • the display unit 12 displays videos of participants, screen-shared materials, and the like.
  • the operation unit 17 is an input device through which various operations on the information display device 10 can be input.
  • the operation unit 17 is, for example, a keyboard, a mouse, a touch panel placed over the display unit 12, or the like.
  • the operation unit 17 can accept, for example, a key input operation, a click operation, or a touch operation.
  • the operation unit 17 can accept operations within the display range related to the utterances displayed on the display unit 12.
  • the operation unit 17 outputs operation information indicating the accepted operation to the operation control unit 32.
  • the display range related to speech is, for example, the speaker display range 120, which is the display range that displays the speaker, or the reference information display range 110, which displays reference information.
  • the operation unit 17 can accept operations at any position in the speaker display range 130120 displayed on the display unit 12.
  • the operation unit 17 is not limited to the range where the speaker's video is displayed in the speaker display range 120, and can accept operations within the range where any participant's video is displayed. be.
  • the speaker display range 120 excludes the positions of operation icons for other commands.
  • the operation unit 17 can accept an operation at any position in the reference information display range 110 displayed on the display unit 12.
  • the keyword information storage unit 18 stores reference information for each keyword as keyword information.
  • the keyword information storage unit 18 is, for example, a recording medium such as a semiconductor memory device such as a RAM (Random Access Memory) or a flash memory.
  • the keyword information storage section 18 may be an external recording section wirelessly connected via the communication section 19.
  • Keywords are words that require supplementary explanation. Keywords include, for example, words that are not generally known, technical terms, abbreviations, difficult words, and buzzwords.
  • the reference information is, for example, the meaning and specific explanation of the keyword. If the keyword is an abbreviation, the reference information may be the full, abbreviated name. Note that the explanation may be, for example, a photograph or a diagram in addition to text. The reference information may be, for example, additional information that is additional information about the keyword.
  • the reference information may have a number of characters, for example, about 100 characters or less, which can be read by a person in about a few seconds.
  • reference information When reference information is searched on the web by the communication unit 19, for example, information from a specific web media such as Wikipedia (registered trademark) for a keyword, the latest news regarding the keyword, etc. are searched. Furthermore, if the keyword is a product name or service name, the website of the provider that provides the product or service is searched. Furthermore, the content displayed at the top of the results searched by a search engine on the web, excluding advertisements, may be used as reference information.
  • a specific web media such as Wikipedia (registered trademark) for a keyword, the latest news regarding the keyword, etc.
  • the keyword is a product name or service name
  • the website of the provider that provides the product or service is searched.
  • the content displayed at the top of the results searched by a search engine on the web excluding advertisements, may be used as reference information.
  • the communication section 19 is a communication unit for performing wireless communication and the like.
  • the communication unit 19 communicates using a communication method using, for example, Wi-Fi (registered trademark) or a mobile phone line, and is connected to an intranet or the Internet. Communication of the communication unit 19 is controlled by a communication control unit 29 of the control device 20 .
  • the communication unit 19 mutually communicates the participant's voice and video with the information display device 10 used by other participants, and searches for reference information. More specifically, the communication unit 19 transmits the participant's voice captured by the microphone 11 and the participant's video captured by the camera to the information display device 10 used by other participants.
  • the communication unit 19 receives audio and video of other participants from the information display device 10 used by the other participants.
  • the input audio refers to the audio collected by the microphone 11 and the audio of other participants acquired via the communication unit 19 from the information display device 10 used by the other participants. It's audio.
  • the control device 20 controls each part of the information display device 10.
  • the control device 20 controls the information display device 10 to display information on the display unit 12.
  • the control device 20 is, for example, an arithmetic processing device (control device) configured with a CPU (Central Processing Unit), an MPU (Micro Processing Unit), etc., and has a storage device such as a RAM or a ROM (Read Only Memory).
  • the control device 20 loads a stored program into memory and executes instructions included in the program.
  • the control device 20 includes an internal memory such as the above-mentioned RAM, and the internal memory is used for temporary storage of data in the control device 20.
  • the control device 20 includes a speech recognition unit 21, an operation control unit 32, an extraction unit 26, an information acquisition unit 27, a display control unit 28, and a communication unit as functional blocks and other components realized by executing a program. It has a control section 29.
  • the utterance recognition unit 21 recognizes at least the utterances of participants participating in communication through speech.
  • the speech recognition unit 21 analyzes the input speech and recognizes the content of the speech contained in the speech.
  • the utterance recognition unit 21 performs acoustic model analysis for each phoneme or word on input speech, and recognizes the content of the utterance by comparing the acoustic model with a phoneme model and a language model.
  • the operation control unit 32 acquires operation information for operations on the operation unit 17. More specifically, the operation control unit 32 acquires operation information of operations on the operation unit 17 in the display range related to the utterance displayed on the display unit 12.
  • the operation control unit 32 receives, for example, a key input operation, a click operation, or a touch operation on the speaker display range 120 or the reference information display range 110.
  • the extraction unit 26 extracts keywords that require reference information from the utterance recognized by the utterance recognition unit 21.
  • the information acquisition unit 27 acquires reference information for the keyword extracted by the extraction unit 26, that is, the keyword for which the extraction unit 26 determines that reference information is required.
  • the information acquisition unit 27 acquires reference information for keywords based on information on the web.
  • the display control unit 28 controls the display of images on the display unit 12.
  • the display control unit 28 outputs a video signal of a video to the display unit 12.
  • the display control unit 28 controls the display of the reference information acquired by the information acquisition unit 27 on the display unit 12.
  • the display control unit 28 outputs a video signal for displaying the reference information corresponding to the keyword extracted by the extraction unit 26 in the reference information display range 110 of the display unit 12.
  • the display control unit 28 displays the reference information for a predetermined period of time, for example, about 5 seconds to 10 seconds.
  • the display control unit 28 may change the display to the next reference information even before the predetermined period has elapsed.
  • the display control unit 28 controls the display of images of the web conference.
  • the display control unit 28 outputs a video signal of the web conference video including the video captured by the camera.
  • the display control unit 28 causes the display unit 12 to display reference information for the keyword that requires the reference information extracted by the extraction unit 26 immediately before the operation control unit 32 detects an operation on the display screen.
  • the display control unit 28 requires the reference information extracted by the extraction unit 26 immediately before the detection.
  • Reference information for the keyword is displayed on the display unit 12.
  • the display control unit 28 displays reference information for the keyword extracted immediately before the operation was detected in a pop-up display.
  • the display control unit 28 may change the display position of the pop-up display according to the position where the operation is detected on the display screen. For example, the display control unit 28 may set the position where the operation is detected to be the center, lower end, upper end, left end, or right end of the pop-up display.
  • the display control unit 28 displays the reference information for the keyword extracted immediately before the detection of the operation in the reference information display range. 110.
  • the communication control unit 29 controls communication by the communication unit 19.
  • the communication control unit 29 controls the communication unit 19 to mutually communicate the participant's audio and video with the information display device 10 used by other participants, and to search for reference information. do. More specifically, the communication control unit 29 transmits, via the communication unit 19, the participant's voice captured by the microphone 11 and the participant's video captured by the camera to an information display device used by other participants. 10.
  • the communication control unit 29 controls the communication unit 19 to receive audio and video of other participants from the information display device 10 used by the other participants.
  • FIG. 4 is a flowchart showing an example of the flow of processing in the information display control device according to the first embodiment.
  • the process shown in FIG. 4 is started by launching an application that conducts a web conference or by starting a web conference.
  • the application for conducting the web conference may include a function for displaying reference information, and the function for displaying reference information may be turned on or off by user operation.
  • the processing of the flowchart shown in FIG. 4 is started.
  • the control device 20 starts speech recognition (step S101). More specifically, the control device 20 uses the speech recognition unit 21 to analyze the input speech and recognize the speech content included in the speech. The control device 20 proceeds to step S102.
  • the control device 20 determines whether the display range related to speech has been manipulated (step S102).
  • the control device 20 determines whether or not the operation control unit 32 receives an operation on the operation unit 17 in the display range related to the utterance displayed on the display unit 12 .
  • the control device 20 determines whether a key input operation, a click operation, or a touch operation in the speaker display range 130 or the reference information display range 110 is received by the operation control unit 32, for example.
  • the process proceeds to step S103. If the control device 20 does not determine that the display range related to speech has been manipulated (No in step S102), the process proceeds to step S105.
  • step S103 determines whether a keyword requiring reference information was extracted immediately before. More specifically, the control device 20 determines whether the extraction unit 26 has extracted a keyword requiring reference information from the utterance recognized by the utterance recognition unit 21 at the time of the operation on the display range related to the utterance.
  • step S104 determines that a keyword requiring reference information has been extracted immediately before (Yes in step S103)
  • step S105 If the control device 20 does not determine that a keyword requiring reference information has been extracted immediately before (No in step S103), the process proceeds to step S105.
  • step S104 the control device 20 acquires the reference information of the keyword and displays it for a predetermined period of time. More specifically, the control device 20 uses the information acquisition unit 27 to acquire reference information for the keyword extracted immediately before. The control device 20 causes the display control unit 28 to output a video signal of reference information corresponding to the keyword extracted by the extraction unit 26 for a predetermined period of time. The control device 20 proceeds to step S105.
  • the control device 20 determines whether to end the process (step S105). For example, the power of the information display device 10 has been turned off, an operation unit (not shown) has been operated, or an operation has been performed to terminate the application that triggered the start of the flowchart shown in FIG. 4. Due to this, it is determined that the process is to be terminated. When the control device 20 determines to end the process (Yes in step S105), it ends the process. If the control device 20 does not determine to end the process (No in step S105), it executes the process in step S102 again.
  • a user of the information display device 10 who is a participant in a web conference checks reference information for an unknown word (keyword) or the like in response to comments made by other participants during the web conference.
  • the user of the information display device 10 wants to check the reference information of an unknown keyword with respect to the utterances of other participants, the user operates the speaker display range 130 or the reference information display range 110 displayed on the display unit 12. conduct.
  • the control device 20 detects an operation on the speaker display range 130 or the reference information display range 110 displayed on the display unit 12.
  • step S104 the control device 20 determines whether the keyword was uttered within a predetermined retrospective period t1, for example, 5 seconds from the time of the operation.
  • the reference information of the immediately preceding keyword is displayed on the display unit 12.
  • the user of the information display device 10 confirms the reference information displayed on the display unit 12.
  • the reference information for the keyword that requires the reference information extracted by the extraction unit 26 immediately before the detection is displayed on the display unit 12.
  • Display the user's operation on the speaker display range 130 or the reference information display range 110 can be used as a trigger to appropriately display the reference information of the keyword.
  • This embodiment can appropriately support the user's understanding.
  • This embodiment can suppress unnecessary keyword reference information from being displayed to the user.
  • This embodiment can assist in smoothing the flow of communication. In this way, the present embodiment can appropriately display reference information from keywords based on the content of the utterance.
  • FIG. 5 is a diagram showing an example in which the speaker display range is selected.
  • FIG. 6 is a flowchart illustrating an example of the flow of processing in the information display control device according to the second embodiment.
  • the basic configuration of the information display device 10 is the same as that of the information display device 10 of the first embodiment.
  • components similar to those of the information display device 10 are denoted by the same or corresponding symbols, and detailed description thereof will be omitted.
  • the information display device 10 displays information immediately before the time of the manipulation, corresponding to the manipulated range. Display reference information for the extracted keywords.
  • processing in the operation unit 17, speech recognition unit 21, operation control unit 32, and display control unit 28 is different from the first embodiment.
  • the operation unit 17 can accept operations on the speaker display range 130 displayed on the display unit 12, in which the image of the speaker is displayed.
  • the operation unit 17 can accept operations on six speaker display ranges 130 in which images of participants A to F are displayed.
  • a participant who is a speaker is associated with each speaker display range 130 and stored in advance in a storage unit (not shown).
  • the utterance recognition unit 21 identifies the speaker and recognizes the utterance.
  • the utterance recognition unit 21 may identify the speaker from participant information obtained from the information display device 10 used by the other participants together with the voices of other participants, for example, via the communication unit 19.
  • the participant information is, for example, information that identifies a participant or information that identifies the information display device 10.
  • the operation control unit 32 acquires operation information for operations on the operation unit 17 in the speaker display range 130 displayed on the display unit 12.
  • the operation control unit 32 can detect and accept the operated speaker.
  • the display control unit 28 selects an extraction unit immediately before the operation point, corresponding to the manipulated range.
  • the reference information for the keyword that requires the reference information extracted by 26 is displayed on the display unit 12. More specifically, when the operation control unit 32 detects that the speaker display range 130 displayed on the display unit 12 has been operated, the display control unit 28 controls the display control unit 28 to display a display area corresponding to the operated speaker display range 130. Reference information for a keyword that is uttered by the speaker and that requires reference information extracted by the extraction unit 26 immediately before the operation time is displayed on the display unit 12.
  • the display control unit 28 may change the display position of the pop-up display according to the operated speaker display range 130.
  • the display control unit 28 may set the operated position in the speaker display range 130 to be the center, lower end, upper end, left end, or right end of the pop-up display.
  • step S111, step S114, and step S115 shown in FIG. 6 are similar to steps S101, step S104, and step S105 of the flowchart shown in FIG.
  • the control device 20 determines whether the speaker display range 130 has been operated (step S112).
  • the control device 20 determines whether the operation control unit 32 has received an operation on the operation unit 17 in the speaker display range 130 displayed on the display unit 12 .
  • the control device 20 determines whether the operation control unit 32 has received a key input operation, click operation, or touch operation on any of the speaker display ranges 130 set corresponding to a plurality of speakers.
  • the control device 20 proceeds to step S113. If the control device 20 does not determine that the speaker display range 130 has been operated (No in step S112), the process proceeds to step S115.
  • the control device 20 determines whether the operated speaker has uttered a keyword that requires reference information immediately before (step S113). ). More specifically, the control device 20 uses the extraction unit 26 to recognize the utterance of the operated speaker by the utterance recognition unit 21 at the time of the operation on the speaker display range 130, and extracts a reference from the recognized utterance. It is determined whether a keyword requiring information has been extracted. If the control device 20 determines that the operated speaker has uttered a keyword that requires reference information immediately before (Yes in step S113), the process proceeds to step S114. If the control device 20 does not determine that the operated speaker has uttered a keyword that requires reference information immediately before (No in step S113), the process proceeds to step S115.
  • the operated range is the speaker display range 130 that displays an image of participant D
  • participant D was speaking immediately before the time of the operation, in other words, participant D was speaking.
  • reference information regarding the keyword is displayed in a pop-up starting from, for example, the speaker display range 130.
  • a case will be described in which a user of the information display device 10 who is a participant in a web conference checks reference information for an unknown phrase (keyword) or the like in response to comments made by other participants during the web conference.
  • the user of the information display device 10 operates the speaker display range 130 of the speaker who uttered the keyword.
  • the control device 20 detects an operation on the speaker display range 130 displayed on the display unit 12 in step S112. If the control device 20 determines in step S113 that the utterance of the speaker operated at the time of the operation has been recognized and that a keyword requiring reference information has been extracted from the utterance, in step S114, a predetermined retrospective period t1 from the time of the operation is performed.
  • the reference information of the last keyword uttered within 5 seconds, for example, is displayed on the display unit 12.
  • the user of the information display device 10 confirms the reference information displayed on the display unit 12.
  • ⁇ Effect> As described above, in this embodiment, when it is detected that the display range related to utterances has been manipulated, keywords requiring reference information extracted by the extracting unit 26 immediately before the operation point are extracted in response to the manipulated range.
  • the reference information for is displayed on the display unit 12. According to the present embodiment, for example, reference information for a keyword that is uttered by a speaker corresponding to the operated speaker display range 130 and that requires reference information extracted by the extraction unit 26 immediately before the operation time is displayed on the display unit. 12 can be displayed.
  • FIG. 7 is a diagram showing an example in which the speaker display range is divided into a plurality of sections.
  • FIG. 7 is a diagram showing an example in which the reference information display range is divided into a plurality of sections.
  • FIG. 8 is a diagram showing an example in which the reference information display range is divided into a plurality of sections.
  • FIG. 9 is a schematic diagram illustrating an example of keywords extracted immediately before the operation time.
  • FIG. 10 is a flowchart illustrating an example of the flow of processing in the information display control device according to the third embodiment.
  • the information display device 10 has the same basic configuration as the information display device 10 of the second embodiment.
  • the information display device 10 displays information immediately before the time of the manipulation, corresponding to the manipulated range. Display reference information for the extracted keywords.
  • processing in the operation section 17, operation control section 32, and display control section 28 is different from the second embodiment.
  • the operation unit 17 can accept operations on sections obtained by dividing the display range related to utterances displayed on the display unit 12.
  • the operation unit 17 can accept operations on the divided sections of the speaker display range 130 displayed on the display unit 12.
  • the speaker display range 130 is divided into a central range 130a located at the center and an outer range 130b located outside the central range 130a.
  • the operation unit 17 can accept operations on the central range 130a and the outer range 130b of the speaker display range 130.
  • the operation unit 17 can accept operations on the divisions of the reference information display range 110 displayed on the display unit 12.
  • the reference information display range 110 is divided into a left range 110a located on the left side and a right range 110b located on the right side of the left range 110a.
  • the operation unit 17 can accept operations on the left range 110a and the right range 110b of the reference information display range 110.
  • the operation control unit 32 acquires operation information for operations on sections obtained by dividing the display range related to utterances displayed on the display unit 12.
  • the display control unit 28 controls the display control unit 28 to display the display area immediately before the operation time, corresponding to the operation position in the operated range. Then, the reference information for the keyword that requires the reference information extracted by the extraction unit 26 is weighted and displayed on the display unit 12.
  • the display control unit 28 when operating the speaker display range 130, the display control unit 28 displays, on the display unit 12, reference information based on utterances that have gone back a longer period of time from the time of operation, the closer the operating position is to the center of the speaker display range 130.
  • the display control unit 28 displays utterances that have been made for a longer period of time from the time of operation than when the operation position is in the outer range 130b.
  • the reference information based on the reference information is displayed on the display unit 12.
  • the display control unit 28 causes the display unit 12 to display the reference information based on the most recent comment if the operating position is the right range 110b of the reference information display range 110, and displays the reference information based on the most recent comment if the operating position is in the left range 110a.
  • Reference information of a plurality of extracted keywords from 2 to 3 previous keywords is displayed on the display unit 12.
  • the display control unit 28 displays reference information of the keyword immediately before uttered within a predetermined retrospective period t2 (t2>t1) from the time of the operation. It is displayed on the display unit 12. If a plurality of keywords uttered immediately before a predetermined retrospective period t2 (t2>t1) from the time of operation are extracted, a plurality of keywords may be displayed, or the following may be performed.
  • the display control unit 28 may selectively display the most recently uttered keyword within a predetermined retrospective period t2 from the time of the operation. For example, when it is detected that the central range 130a of the speaker display range 130 has been operated, the display control unit 28 selects and displays the most recently uttered keyword within a predetermined retrospective period t2 from the time of the operation. For example, when it is detected that the left side range 110a of the reference information display range 110 has been operated, the display control unit 28 selectively displays the most recent keyword uttered within a predetermined retrospective period t2 from the time of the operation.
  • the keyword selection display means for example, displaying a selection screen for selecting a plurality of keywords extracted in a predetermined period.
  • the display control unit 28 acquires and displays reference information for the most recently uttered keyword within a predetermined retrospective period t1 from the time of the operation. .
  • the display control unit 28 acquires reference information for the most recent keyword uttered within a predetermined retrospective period t1 from the time of the operation. indicate.
  • the display control unit 28 acquires reference information for the immediately preceding keyword uttered within a predetermined retrospective period t2 from the time of the operation. indicate.
  • the display control unit 28 may display reference information of the plurality of keywords, or select and display the plurality of keywords. Reference information for the selected keyword may be displayed.
  • FIG. 9 is a schematic diagram illustrating an example of keywords extracted immediately before the operation time.
  • the keyword "word n" is extracted within a predetermined retrospective period t1 from the time of the operation.
  • the keyword "word n” and the keyword “word n-1" are extracted within a predetermined retrospective period t2 from the time of the operation.
  • step S121 to step S123 and step S128 shown in FIG. 10 is the same as the processing from step S111 to step S113 and step S115 of the flowchart shown in FIG.
  • step S124 determines whether the operating position is in the center range 130a of the speaker display range 130. is determined (step S124).
  • the control device 20 uses the operation control unit 32 to determine whether the operation position is in the center range 130a of the speaker display range 130 based on the operation information.
  • step S125 determines whether the control device 20 does not determine that the operating position is in the center range 130a of the speaker display range 130 (No in step S124). the process proceeds to step S127.
  • the control device 20 selects and displays the most recent keyword uttered within a predetermined retrospective period t2 from the time of the operation. (Step S125). More specifically, the control device 20 causes the display control unit 28 to display, on the display unit 12, the reference information of the last keyword uttered within a predetermined retrospective period t2 (t2>t1) from the time of the operation. The control device 20 may cause the display control unit 28 to display reference information of the plurality of keywords, for example, when there are a plurality of keywords immediately before uttered within a predetermined retrospective period t2 from the time of operation, or display the reference information of the plurality of keywords. Keywords may be selected and displayed, and reference information for the selected keywords may be displayed. The control device 20 proceeds to step S126.
  • the control device 20 acquires the keyword reference information and displays it for a predetermined period of time (step S126). More specifically, the control device 20 uses the information acquisition unit 27 to acquire reference information for the selected keyword. The control device 20 causes the display control unit 28 to output a video signal of reference information corresponding to the keyword extracted by the extraction unit 26 for a predetermined period of time. The control device 20 proceeds to step S128.
  • the control device 20 causes the display control unit 28 to extract the reference information extracted by the extraction unit 26 immediately before the operation time.
  • Reference information for the required keyword is displayed on the display unit 12 (step S127). More specifically, the control device 20 uses the information acquisition unit 27 to acquire reference information for the keyword extracted immediately before.
  • the control device 20 causes the display control unit 28 to output a video signal of reference information corresponding to the keyword extracted by the extraction unit 26 for a predetermined period of time. The control device 20 proceeds to step S128.
  • step S124 is replaced with "Is the operation position the left range of the reference information display range?"
  • the user's operation is an operation on the outer range 130b of the speaker display range 130, as shown by the pointer image Pb in FIG.
  • reference information of the most recent keyword uttered within a period t2 for example, 15 seconds from the time of the operation.
  • reference information of the plurality of keywords may be displayed, or the plurality of keywords may be selectively displayed.
  • Reference information for the selected keyword may be displayed. In the example shown in FIG. 9, reference information for both the keyword "word n" and the keyword "word n-1" may be displayed.
  • the keyword "word n” and the keyword “word n-1” may be selectively displayed.
  • reference information for the keyword “word n-1” is displayed. If the keyword “word n” and the keyword “word n-1" are the same word or phrase, selection display may not be performed.
  • the reference information extracted by the extracting unit 26 immediately before the operation time point is The reference information for keywords that require the following are weighted and displayed on the display unit 12. According to the present embodiment, a plurality of keywords can be displayed or selectively displayed retroactively from the time of operation depending on the operation position. According to this embodiment, reference information of keywords required by the user can be appropriately displayed.
  • FIG. 11 is a flowchart illustrating an example of the flow of processing in the information display control device according to the fourth embodiment.
  • the information display device 10 has the same basic configuration as the information display device 10 of the second embodiment.
  • the information display device 10 displays information immediately before the time of the manipulation, corresponding to the manipulated range. Display reference information for the extracted keywords.
  • processing in the operation section 17, operation control section 32, and display control section 28 is different from the second embodiment.
  • the display control unit 28 controls the extraction unit 26 at the time of detection, corresponding to the period of operation.
  • the reference information for keywords that require reference information extracted by is weighted and displayed on the display unit 12.
  • the display control unit 28 when it is detected that the display range related to utterances has been manipulated, the display control unit 28 causes the display unit 12 to display reference information based on utterances that go back a longer period of time from the time of the operation, the longer the operation period is. .
  • the display control unit 28 when it is detected that the display range related to utterances has been manipulated, the display control unit 28 causes the display unit 12 to display reference information based on utterances that go back a shorter period of time from the time of the operation, the shorter the period of operation. .
  • the display control unit 28 displays reference information based on the most recent utterance, For example, as shown in FIG. 9, the reference information of the last keyword uttered within the period t1 from the time of operation is displayed.
  • the display control unit 28 controls the display control unit 28 to Before the word, for example, as shown in FIG. 9, reference information from the time of operation to the keyword uttered within the period t2 is displayed. If there are multiple extracted keywords, they may be selectively displayed.
  • the operation point in this case is preferably the operation start point.
  • step S131 to step S133 and from step S135 to step S138 shown in FIG. 11 are the same as those from step S121 to step S123 and from step S125 to step S128 of the flowchart shown in FIG. 10.
  • step S134 determines whether the operation has been performed for a period of time ta or longer. More specifically, the control device 20 uses the operation control unit 32 to determine whether the operation has been performed for a period of time ta or longer based on the operation information. When the control device 20 determines that the operation has been performed for the period ta or longer (Yes in step S134), the process proceeds to step S135. The control device 20 uses the operation control unit 32 to determine whether the operation has been performed for a period of time ta or longer based on the operation information. If it is determined that the operation has been performed for the period ta or longer (No in step S134), the control device 20 proceeds to step S137.
  • the reference information for the keyword that requires the reference information extracted by the extraction unit 26 at the time of detection is weighted in accordance with the period of manipulation. and display it on the display unit 12.
  • a plurality of keywords can be displayed or selectively displayed retroactively from the time of operation, depending on the period of operation.
  • reference information of keywords required by the user can be appropriately displayed.
  • FIG. 12 is a schematic diagram showing an example of how the information display device is used.
  • FIG. 13 is a schematic diagram showing another example of how the information display device is used.
  • FIG. 14 is a block diagram showing a configuration example of an information display device 10 having an information display control device according to the fifth embodiment.
  • the basic configuration of the information display device 10 is the same as that of the information display device 10 of the first embodiment. This embodiment differs from the first embodiment in that it includes a camera and the processing in the control device 20.
  • the present embodiment includes a line-of-sight detection unit 24 that functions as an operation control unit 32 that detects an operation on a display screen displayed on the display unit 12.
  • the information display device 10 is used in a conference (hereinafter referred to as a "real conference") or conversation in which a plurality of participants gather in one place.
  • the information display device 10 used in this case may be implemented as one of the functions of an information terminal device such as a smartphone used by the participant, for example.
  • the real conference shown in FIG. 12 includes a presenter M11, another participant M12, another participant M13, another participant M14, and another participant M15. Another participant M15 is using the information display device 10.
  • the information display device 10 is used in online communication such as a web conference or conversation with a plurality of participants participating.
  • the information display device 10 When the information display device 10 detects that the direction of the user's line of sight is directed toward the display unit 12 during a meeting, the information display device 10 requires reference information (hereinafter referred to as “recent keyword”) extracted immediately before the time of detection. Reference information for the keyword is displayed on the display unit 12.
  • the information display device 10 includes a microphone 11, a display section 12, a camera 14, a keyword information storage section 18, a communication section 19, and a control device 20.
  • the microphone 11 picks up the voice of the speaker. As shown in FIG. 13, when the information display device 10 is used for a web conference, the microphone 11 picks up the voice of the user who is the participant using the information display device 10.
  • the reference information display range 110 may be displayed on the entire surface of the display unit 12. As shown in FIG. 13, in the case of the information display device 10 used for a web conference, the reference information display range 110 is displayed on a part of the display section 12, for example, at the bottom, top, left side, or right side. Good too. In the case of the information display device 10 used for a web conference, the reference information display range 110 may be displayed as a pop-up on the display unit 12.
  • the display unit 12 displays images of participants, screen-shared materials, and the like.
  • the camera 14 is placed at a position where it can photograph the face of the user of the information display device 10. More specifically, the camera 14 captures an image including the user's face and eyes in order to detect the line of sight of the user of the information display device 10 .
  • the camera 14 is arranged around the display section 12. The camera 14 outputs the captured image to the line of sight detection section 24.
  • FIG. 15 is a diagram showing an example of keyword information stored in the keyword information storage section 18.
  • the keyword information shown in FIG. 15 stores reference information for each keyword. For example, for the keyword “ ⁇ ”, the reference information " ⁇ ... ⁇ ” is stored. For example, for the keyword “ ⁇ ”, the reference information " ⁇ ... ⁇ ” is stored.
  • the communication section 19 acquires keyword information from the keyword information storage section 18, which is an external recording section, and searches for reference information. As shown in FIG. 13, when the information display device 10 is used for a web conference, the communication unit 19 communicates the audio and video of participants with the information display device 10 used by other participants. and searches for reference information.
  • the input audio is the audio collected by the microphone 11 in the case of the information display device 10 used for a real conference.
  • Inputted audio includes, in the case of the information display device 10 used for the web conference, the audio collected by the microphone 11 and the information display device used by other participants acquired via the communication unit 19. This is the voice of another participant obtained from 10.
  • the control device 20 includes a speech recognition unit 21, a line of sight detection unit (operation control unit) 24, an extraction unit 26, an information acquisition unit 27, and a display control unit, as components such as functional blocks realized by executing a program. 28 and a communication control section 29.
  • the line of sight detection unit 24 detects the direction of the line of sight of the user of the information display device 10 based on the photographic data taken by the camera 14.
  • the method of detecting the line of sight is not limited, in this embodiment, the line of sight is detected by corneal reflection.
  • the line of sight detection unit 24 detects, for example, that the direction of the user's line of sight is directed toward the display unit 12. This is because, for example, when a user wants to refer to a keyword during a real conference, it is predicted that the user will direct his/her line of sight to the display unit 12. Therefore, the line of sight detection unit 24 detects that the direction of the user's line of sight is directed toward the display unit 12, for example, during a real conference.
  • the line of sight detection unit 24 detects, for example, that the direction of the user's line of sight is directed toward a predetermined range of the display unit 12. This is because, for example, when a user wants to refer to a keyword during a web conference, it is predicted that the user will direct his/her line of sight to a predetermined range of the screen of the display unit 12.
  • the predetermined range of the display unit 12 is, for example, the reference information display range 110 of the display unit 12 or a predetermined range of the screen that is set in advance.
  • the line of sight detection unit 24 detects, for example, that the direction of the user's line of sight has changed from facing the speaker to facing the display unit 12. This is because, for example, when a user wants to refer to a keyword during a real conference or a web conference, it is predicted that the user will move his/her line of sight from the speaker to the display unit 12.
  • the direction of the speech sound is detected from the sound picked up by the microphone 11.
  • the direction in which the speaker or the material by the speaker is displayed on the screen is detected from the video shot by the camera 14.
  • a state in which the direction of the user's line of sight matches the direction of the speaker is a state in which the direction of the user's line of sight is directed toward the speaker.
  • the extraction unit 26 starts extracting keywords at the same time as the utterance recognition unit 21 starts speech recognition.
  • the information acquisition unit 27 acquires reference information for the keyword extracted by the extraction unit 26.
  • the information acquisition unit 27 acquires reference information for a keyword from the keyword information stored in the keyword information storage unit 18.
  • the information acquisition unit 27 acquires reference information for the most recent keyword when the gaze detection unit 24 detects that the direction of the user's gaze is directed toward the display unit 12.
  • the display control unit 28 causes the display unit 12 to display reference information for the most recent keyword. For example, during a real conference, when the user's line of sight is directed toward the display unit 12 of the smartphone, which is the information display device 10, the reference information of the most recent keyword is displayed on the display unit 12.
  • the display control unit 28 causes the reference information acquired by the information acquisition unit 27 to be displayed in a predetermined range of the display unit 12, and the direction of the user's line of sight detected by the line of sight detection unit 24 is set in the predetermined range of the display unit 12.
  • reference information for the most recent keyword may be displayed in a predetermined range of the display unit 12.
  • the display control unit 28 controls the display control unit 28 to prevent the user from looking at the bottom of the screen. , the reference information for the most recent keywords will be displayed.
  • the display control unit 28 displays reference information for the most recent keyword. It may be displayed on the display unit 12. For example, in the information display device 10 used for a web conference, if the direction of the line of sight is moved from the range where images of participants or materials are displayed on the display section 12, the display control section 28 may cause the reference information of the most recent keyword to change. Is displayed.
  • the display control unit 28 improves the visibility of the display of reference information for the most recent keyword. You can.
  • Improving the visibility of the display of reference information means displaying it in a way that makes it stand out, for example, by increasing the number of characters, making the characters larger, changing the color of the characters, changing the thickness of the characters, etc.
  • the display control unit 28 displays the number of characters of the reference information of the latest keyword in a smaller number, and when the user's line of sight turns to the reference information, displays the number of characters in a larger number.
  • the display control unit 28 displays the text color of the reference information of the latest keyword in a light color or with transparency, and when the user's line of sight turns, the text color is displayed in a dark color or as non-transparent text.
  • the display control unit 28 controls the display of images of the web conference.
  • the communication control unit 29 receives keyword information from the keyword information storage unit 18, which is an external recording unit, via the communication unit 19. , controls to search for reference information.
  • the communication control unit 29 communicates with the information display device 10 used by other participants via the communication unit 19. It communicates the audio and video of the participants with each other and controls the search for reference information.
  • Step ST105 shown in FIG. 16 performs the same process as step S105 of the flowchart shown in FIG. 4.
  • the control device 20 starts speech recognition (step ST101). More specifically, the control device 20 uses the speech recognition unit 21 to analyze the input speech and recognize the speech content included in the speech. Then, the control device 20 uses the extraction unit 26 to extract keywords requiring reference information from the utterance recognized by the utterance recognition unit 21. The control device 20 proceeds to step ST102.
  • the control device 20 determines whether the user's line of sight is directed toward the display unit 12 (step ST102). The control device 20 determines whether the direction of the user's line of sight is directed toward the display unit 12 using the line of sight detection unit 24 . When the control device 20 determines that the user's line of sight is directed toward the display unit 12 (Yes in step ST102), the control device 20 proceeds to step ST103. If the control device 20 does not determine that the user's line of sight is directed toward the display unit 12 (No in step ST102), the control device 20 proceeds to step ST105.
  • the control device 20 determines whether a keyword requiring reference information has been extracted immediately before (step ST103). More specifically, when the extraction unit 26 determines that the most recent keyword has been extracted within, for example, two seconds (Yes in step ST103), the control device 20 proceeds to step ST104. If the extraction unit 26 does not determine that the most recent keyword has been extracted within, for example, 2 seconds (No in step ST103), the control device 20 proceeds to step ST105.
  • the control device 20 acquires the reference information of the most recent keyword and displays it for a predetermined period of time (step ST104). More specifically, the control device 20 uses the information acquisition unit 27 to acquire, for example, reference information for the latest keyword. The control device 20 causes the display control unit 28 to output a video signal of reference information corresponding to the most recent keyword extracted by the extraction unit 26 for a predetermined period of time. The control device 20 proceeds to step ST105.
  • reference information for the most recent keyword is displayed on the display unit 12.
  • reference information for keywords can be displayed.
  • This embodiment can appropriately support the user's understanding when the user needs it.
  • This embodiment can suppress unnecessary keyword reference information from being displayed to the user.
  • This embodiment can assist in smoothing the flow of communication. In this manner, according to the present embodiment, reference information can be appropriately displayed based on keywords based on the content of the utterance.
  • the reference information is displayed in a predetermined range of the display unit 12, and when it is detected that the user's line of sight is directed to the predetermined range of the display unit 12, the reference information for the latest keyword is displayed on the display unit 12. Display it in a predetermined range. According to this embodiment, when the user is looking at a predetermined range of the display unit 12, reference information of keywords can be displayed.
  • reference information for the most recent keyword is displayed on the display unit 12. According to this embodiment, when the user moves his/her line of sight from the speaker to the display unit 12, the reference information of the keyword can be displayed.
  • FIG. 17 is a flowchart illustrating an example of the flow of processing in the information display control device according to the sixth embodiment.
  • the information display device 10 has the same basic configuration as the information display device 10 of the fifth embodiment.
  • the information display device 10 detects that the direction of the user's line of sight continues to be directed toward the display unit 12 for a predetermined period or more in a meeting, the information display device 10 provides reference information for a keyword that requires reference information. is displayed on the display unit 12 by changing the display mode.
  • the processing in the line of sight detection section 24 and the display control section 28 is different from the fifth embodiment.
  • the display control unit 28 causes the display unit 12 to display the reference information in the first display format or the second display format.
  • the first display form and the second display form differ in the amount of displayed reference information.
  • the first display mode and the second display mode when the display control unit 28 displays the reference information are stored in the keyword information storage unit 18, the reference information of the information amount lv1 corresponding to the first display mode, and the second display mode.
  • Reference information of the corresponding information amount lv2 may be stored.
  • the relationship between the amount of reference information of the amount of information lv1 and the amount of reference information of the amount of information lv2 is the amount of information lv1 ⁇ the amount of information lv2. For example, reference information with an amount of information lv1 is displayed in about 1 to 2 lines, and reference information with an amount of information lv2 is displayed in about 3 to 5 lines.
  • the information acquisition unit 27 searches for reference information on the WEB
  • the first display that satisfies the amount of information lv1 ⁇ the amount of information lv2 is based on the reference information acquired from the WEB search or a specific WEB media.
  • the reference information of the form and the reference information of the second display form are obtained.
  • the information acquisition unit sets different information sources for the web media that acquires the reference information for displaying in the first display format and the web media that acquires the reference information for displaying in the second display format.
  • the reference information for displaying in the first display format is obtained from web media that has a relatively small amount of information explaining the keyword
  • the reference information for displaying in the second display format is obtained from the keyword. It is obtained from web media that has a relatively large amount of information explaining the information, in other words, provides detailed explanations.
  • the line of sight detection unit 24 detects, for example, that the user's line of sight continues to face a predetermined range of the display unit 12 for a predetermined period or longer. For example, during a meeting, when a user feels that the reference information of the displayed keyword is insufficiently understood, it is expected that the user will continue to look at the display unit 12 for a long time. .
  • the display control unit 28 When the display control unit 28 detects that a predetermined period of time or more has elapsed with the user's line of sight detected by the line of sight detection unit 24 facing the display unit 12, the display control unit 28 displays reference information for the most recent keyword on the display unit 12. Display by changing the display mode. For example, when the user's line of sight is directed toward the display unit 12 for 5 seconds or more, the display control unit 28 changes the display mode of the reference information for the most recent keyword from the first display mode to the second display mode. and display it.
  • step ST111 to step ST113 and step ST118 shown in FIG. 17 are the same as steps ST101 to step ST103 and step ST105 of the flowchart shown in FIG. 16.
  • step ST114 the control device 20 causes the display control unit 28 to display the reference information of the information amount lv1 corresponding to the most recent keyword extracted by the extraction unit 26 for a predetermined period.
  • the control device 20 determines whether a predetermined period of time has elapsed (step ST115). If the control device 20 determines that the predetermined period has elapsed (Yes in step ST115), it proceeds to step ST116. If the control device 20 does not determine that the predetermined period has elapsed (No in step ST115), it executes the process of step ST115 again.
  • the control device 20 determines whether the user's line of sight remains directed toward the display unit 12 (step ST116).
  • the control device 20 uses the line of sight detection unit 24 to determine whether the direction of the user's line of sight remains facing the display unit 12 or not. If the control device 20 determines that the user's line of sight remains directed toward the display unit 12 (Yes in step ST116), the control device 20 proceeds to step ST117. If the control device 20 does not determine that the user's line of sight remains directed toward the display unit 12 (No in step ST116), the process proceeds to step ST118.
  • control device 20 displays reference information with an amount of information lv2 corresponding to the most recent keyword for a predetermined period of time (step ST117). More specifically, the control device 20 causes the display control unit 28 to increase and display the amount of reference information corresponding to the most recent keyword extracted by the extraction unit 26. The control device 20 proceeds to step ST118.
  • the display control unit 28 starts displaying the reference information with settings such that the reference information is displayed for a predetermined period, such as 5 seconds.
  • a predetermined period of time such as 5 seconds
  • the reference information displayed in the first display format is further displayed on the display unit 12 in the second display format for a predetermined period such as 5 seconds.
  • the display unit 12 displays reference information for the most recent keyword in a different display manner. According to the present embodiment, when the user does not fully understand the keyword from the displayed reference information, the present embodiment can appropriately support the user's understanding.
  • FIG. 18 is a flowchart illustrating an example of the flow of processing in the information display control device according to the seventh embodiment.
  • the information display device 10 has the same basic configuration as the information display device 10 of the fifth embodiment. In the following description, components similar to those of the information display device 10 are denoted by the same or corresponding symbols, and detailed description thereof will be omitted.
  • the information display device 10 displays the reference information on the display unit 12 so that the user's line of sight is directed toward the display unit 12. If detected, the display mode of the reference information is changed and displayed on the display unit 12.
  • the processing in the display control unit 28 is different from the fifth embodiment.
  • the display control unit 28 causes the display unit 12 to display the reference information in the first display format or the second display format.
  • the first display form and the second display form differ in the amount of display information of the reference information or the display visibility of the reference information.
  • the first display form and the second display form when the display control unit 28 displays the reference information are the amount of display information of the reference information
  • the first display form and the second display form are the same as those in the sixth embodiment. This is the same as the first display form and the second display form.
  • the display control unit 28 changes the display visibility of the reference information.
  • the display visibility of the reference information can be changed by, for example, changing the font size, font color, font transmittance, or the like.
  • the display control unit 28 changes the display visibility of reference information. If the change in display visibility of the reference information is a change in font size, the display control unit 28 changes the display mode to a second display mode in which the font size is larger than the font size displayed as the first display mode. This enables display with good visibility. Therefore, the display control unit 28 changes the display mode from the first display mode, which has relatively low visibility, to the second display mode, which has relatively high visibility.
  • the area of the reference information display range 110 or the like may be increased as the font size is changed.
  • the display control unit 28 changes the font color or the transmittance of the characters to be more conspicuous than the characters displayed as the first display form.
  • the font color of the first display mode is, for example, black
  • the font color of the second display mode is, for example, red.
  • the transmittance of the characters in the first display mode so-called transparency
  • the transmittance of the characters in the second display mode is, for example, around 50%, which is enough to see through the background information
  • the transmittance of the characters in the second display mode is 10%, which is such that the background information is hardly visible. less than %.
  • the thickness of characters may also be changed between the first display form and the second display form.
  • step ST121 and step ST126 shown in FIG. 18 are similar to steps ST101 and step ST105 in the flowchart shown in FIG. 16.
  • step ST122 the control device 20 determines whether a keyword requiring reference information has been extracted. More specifically, in the control device 20, the extraction unit 26 determines whether a keyword requiring reference information is extracted from the content of the utterance, based on the recognition result of the utterance by the utterance recognition unit 21. If it is determined in step ST122 that a keyword requiring reference information has been extracted (Yes in step ST122), the process advances to step ST123. If the control device 20 does not determine that a keyword requiring reference information has been extracted (No in step ST122), the process proceeds to step ST126.
  • step ST123 the reference information of the keyword that requires reference information extracted in step ST122 is acquired and displayed in the first display form for a predetermined period of time. More specifically, the control device 20 uses the information obtaining section 27 to obtain reference information of a keyword that requires reference information, and the display control section 28 causes the control device 20 to display the extracted reference information in the first display form, that is, A display format with relatively low visibility is displayed on the display unit 12 for a predetermined period of time.
  • the control device 20 determines whether the user's line of sight is directed toward the display unit 12 (step ST124). The control device 20 determines whether the direction of the user's line of sight is directed toward the display unit 12 using the line of sight detection unit 24 . When the control device 20 determines that the user's line of sight is directed toward the display unit 12 (Yes in step ST124), the control device 20 proceeds to step ST125. If the control device 20 does not determine that the user's line of sight is directed toward the display unit 12 (No in step ST124), the process proceeds to step ST126.
  • step ST125 the reference information displayed in the first display format in step ST123 is displayed in the second display format for a predetermined period of time.
  • the display mode of the reference information displayed in the first display mode in step ST123 is changed to the second display mode. That is, in step ST123, the display mode of the reference information displayed in a display mode with relatively low visibility is changed to a display mode with relatively high visibility.
  • the reference information when a keyword requiring reference information is extracted, the reference information is displayed in the first display form with relatively low visibility, and the direction of the user's line of sight is directed toward the display unit 12. If it is detected that the reference information is displayed, the display format of the reference information is changed to a second display format with relatively high visibility. According to this embodiment, by looking at the display unit 12, the user can clearly see the reference information of the keyword.
  • This embodiment can appropriately support the user's understanding when the user needs it.
  • reference information is displayed with good visibility for keywords for which reference information is unnecessary for the user, so that the user becomes conscious of the reference information and the flow of communication in meetings etc. is obstructed. can be suppressed. In this manner, according to the present embodiment, reference information can be appropriately displayed based on keywords based on the content of the utterance.
  • the information display device may be implemented in various different forms other than the embodiments described above.
  • Each component of the illustrated information display device is functionally conceptual, and does not necessarily have to be physically configured as illustrated.
  • the specific form of each device is not limited to what is shown in the diagram, and all or part of it may be functionally or physically distributed or integrated into arbitrary units depending on the processing load and usage status of each device. You can.
  • the configuration of the information display device is realized by, for example, a program loaded into a memory as software.
  • the above embodiments have been described as functional blocks realized by cooperation of these hardware or software. That is, these functional blocks can be realized in various forms using only hardware, only software, or a combination thereof.
  • the line of sight detection unit 24 in the embodiment described above can also obtain the same effect by detecting that the user is facing the direction of the display unit 12. For this reason, it is also possible to replace it with a face orientation detection section that detects the orientation of the user's face based on the video captured by the camera 14. Based on the relative positional relationship between the position of the camera 14 and the display unit 12, the face direction detection unit detects the direction of the user's face using a known method from the image of the user's face taken by the camera 14. It is detected that the user is facing the direction of the display unit 12. In this case, when the face orientation detection unit detects that the user's face is facing the display unit 12, the display control unit 28 extracts the reference information extracted by the extraction unit 26 immediately before the detection.
  • Reference information for the required keyword is displayed on the display unit 12.
  • the display control unit 28 causes the display unit 12 to change the display mode of the reference information displayed on the display unit 12.
  • the present disclosure can be used, for example, in an information display device used for communication through at least speech, such as a meeting or conversation.

Landscapes

  • Engineering & Computer Science (AREA)
  • Theoretical Computer Science (AREA)
  • General Engineering & Computer Science (AREA)
  • Physics & Mathematics (AREA)
  • Human Computer Interaction (AREA)
  • General Physics & Mathematics (AREA)
  • Health & Medical Sciences (AREA)
  • Audiology, Speech & Language Pathology (AREA)
  • Multimedia (AREA)
  • Computational Linguistics (AREA)
  • Acoustics & Sound (AREA)
  • Databases & Information Systems (AREA)
  • Data Mining & Analysis (AREA)
  • General Health & Medical Sciences (AREA)
  • Signal Processing (AREA)
  • User Interface Of Digital Computer (AREA)

Abstract

L'invention concerne un dispositif de commande d'affichage d'informations (20) qui commande un dispositif d'affichage d'informations (10) pour afficher des informations sur une unité d'affichage (12), le dispositif de commande d'affichage d'informations (20) comprenant : une unité de reconnaissance d'énoncé (21) qui reconnaît un énoncé par un haut-parleur ; une unité de commande d'opération (32) qui détecte une opération par rapport à un écran d'affichage affiché sur l'unité d'affichage (12) ; une unité d'extraction (26) qui extrait des mots-clés nécessitant des informations de référence à partir de l'énoncé reconnu par l'unité de reconnaissance d'énoncé (21) ; une unité d'acquisition d'informations (27) qui acquiert des informations de référence concernant les mots-clés extraits par l'unité d'extraction (26) ; et une unité de commande d'affichage (28) qui affiche les informations de référence acquises par l'unité d'acquisition d'informations (27) sur l'unité d'affichage (12), lorsque l'unité de commande d'opération (32) détecte qu'une plage d'affichage associée à un énoncé affichée sur l'unité d'affichage (12) a été actionnée, l'unité de commande d'affichage (28) amène l'unité d'affichage (12) à afficher des informations de référence concernant les mots-clés nécessitant des informations de référence que l'unité d'extraction (26) a extraits immédiatement avant le temps de détection.
PCT/JP2023/019082 2022-06-22 2023-05-23 Dispositif de commande d'affichage d'informations et procédé d'affichage d'informations WO2023248677A1 (fr)

Applications Claiming Priority (4)

Application Number Priority Date Filing Date Title
JP2022100400A JP2024001624A (ja) 2022-06-22 2022-06-22 情報表示制御装置および情報表示方法
JP2022-100400 2022-06-22
JP2022-114215 2022-07-15
JP2022114215A JP2024011896A (ja) 2022-07-15 2022-07-15 情報表示制御装置および情報表示方法

Publications (1)

Publication Number Publication Date
WO2023248677A1 true WO2023248677A1 (fr) 2023-12-28

Family

ID=89379755

Family Applications (1)

Application Number Title Priority Date Filing Date
PCT/JP2023/019082 WO2023248677A1 (fr) 2022-06-22 2023-05-23 Dispositif de commande d'affichage d'informations et procédé d'affichage d'informations

Country Status (1)

Country Link
WO (1) WO2023248677A1 (fr)

Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2006085565A1 (fr) * 2005-02-08 2006-08-17 Nippon Telegraph And Telephone Corporation Terminal de communication d’information, système de communication d’information, méthode de communication d’information, programme de communication d’information et support d’enregistrement sur lequel le programme est enregistré
JP2012208630A (ja) * 2011-03-29 2012-10-25 Mizuho Information & Research Institute Inc 発言管理システム、発言管理方法及び発言管理プログラム
JP2017004193A (ja) * 2015-06-09 2017-01-05 凸版印刷株式会社 情報処理装置、情報処理方法、及びプログラム
WO2019130817A1 (fr) * 2017-12-25 2019-07-04 京セラドキュメントソリューションズ株式会社 Dispositif de traitement d'informations et procédé d'analyse de la parole
JP2019531538A (ja) * 2016-08-12 2019-10-31 マジック リープ， インコーポレイテッドＭａｇｉｃ Ｌｅａｐ，Ｉｎｃ． ワードフロー注釈

Patent Citations (5)

* Cited by examiner, † Cited by third party
Publication number Priority date Publication date Assignee Title
WO2006085565A1 (fr) * 2005-02-08 2006-08-17 Nippon Telegraph And Telephone Corporation Terminal de communication d’information, système de communication d’information, méthode de communication d’information, programme de communication d’information et support d’enregistrement sur lequel le programme est enregistré
JP2012208630A (ja) * 2011-03-29 2012-10-25 Mizuho Information & Research Institute Inc 発言管理システム、発言管理方法及び発言管理プログラム
JP2017004193A (ja) * 2015-06-09 2017-01-05 凸版印刷株式会社 情報処理装置、情報処理方法、及びプログラム
JP2019531538A (ja) * 2016-08-12 2019-10-31 マジック リープ， インコーポレイテッドＭａｇｉｃ Ｌｅａｐ，Ｉｎｃ． ワードフロー注釈
WO2019130817A1 (fr) * 2017-12-25 2019-07-04 京セラドキュメントソリューションズ株式会社 Dispositif de traitement d'informations et procédé d'analyse de la parole

Similar Documents

Publication Publication Date Title
CN108874356B (zh) 语音播报方法、装置、移动终端和存储介质
KR102222421B1 (ko) 캡처된 이미지들과 관련된 메타 데이터 저장
US10387570B2 (en) Enhanced e-reader experience
JP6570651B2 (ja) 音声対話装置および音声対話方法
EP0986809B1 (fr) Methode de reconnaissance de parole avec plusieurs programmes d'application
US10811005B2 (en) Adapting voice input processing based on voice input characteristics
US11176944B2 (en) Transcription summary presentation
RU2733816C1 (ru) Способ обработки речевой информации, устройство и запоминающий носитель информации
US20230169976A1 (en) Streaming Action Fulfillment Based on Partial Hypotheses
US20160294892A1 (en) Storage Medium Storing Program, Server Apparatus, and Method of Controlling Server Apparatus
CN110544473A (zh) 语音交互方法和装置
CN109032554B (zh) 一种音频处理方法和电子设备
CN112084478A (zh) 多用户账户的切换方法、装置、电子设备以及存储介质
WO2023248677A1 (fr) Dispositif de commande d'affichage d'informations et procédé d'affichage d'informations
US20210103635A1 (en) Speaking technique improvement assistant
CN103428339A (zh) 通过语音输入控制手机进行操作的方法
CA3097683A1 (fr) Systeme d'analyse de la parole
CN111312241A (zh) 一种无人导购方法、终端以及存储介质
JP2024011896A (ja) 情報表示制御装置および情報表示方法
JP2024001624A (ja) 情報表示制御装置および情報表示方法
US11895269B2 (en) Determination and visual display of spoken menus for calls
JP2019203998A (ja) 会話装置、ロボット、会話装置制御方法及びプログラム
CN114999482A (zh) 基于视线的语音识别方法、装置、设备及存储介质
US11238863B2 (en) Query disambiguation using environmental audio
JP2023181704A (ja) 情報表示制御装置および情報表示方法

Legal Events

Date Code Title Description
121 Ep: the epo has been informed by wipo that ep was designated in this application

Ref document number: 23826865

Country of ref document: EP

Kind code of ref document: A1