JP6690442B2 - Presentation support device, presentation support system, presentation support method, and presentation support program - Google Patents

Description

  The present invention relates to a presentation support device, a presentation support system, a presentation support method, and a presentation support program.

  In a scene such as a presentation at a conference or a lecture, introduction of a pamphlet, etc., a plurality of people may communicate by conversation while sharing a document having the same content, for example, a document relating to a progress agenda or a slide material.

  As an example of a technique that supports such communication by conversation, there is a technique that utilizes voice recognition to change the display state of the portion corresponding to the uttered portion of the content included in the shared document.

  In addition, a synchronized content information generation device described below has been proposed. This synchronous content information generation device is a document block in which the document information is divided into N pieces, for example, one page or the like, based on audio / video information in which a state of a meeting using documents is recorded / imaged by a video camera device. Generates synchronized content information for displaying document information in time-sequential synchronization with audio / video information by measuring the distribution of the appearance keywords in the order in which the keywords extracted in units such as one paragraph appear. To do.

JP, 2004-7358, A JP, 2009-272814, A JP, 7-334075, A JP, 2013-83897, A

  However, in the above technique, there may be a case where the display state of the uttered portion is not changed.

  That is, in the above-described synchronized content information generation device, the display state of the document block in which the keyword has a high utterance frequency is changed. However, when the absolute number of keywords contained in the document block is small, even if the keyword in the document block is uttered, if the utterance frequency of the keyword in another document block is high, the other document block is uttered. The display state of is changed. As a result, the display state of a document block having a small number of keywords may be skipped without being changed.

  In one aspect, it is an object of the present invention to provide a presentation support device, a presentation support system, a presentation support method, and a presentation support program that can prevent the omission of the change in the display state of the uttered portion.

  In one aspect, the presentation support device includes a recognition unit that performs voice recognition on voice data by using a word extracted from a character string included in each region into which the display content of the document file is divided; A display control unit that changes a display state of an area including a subsequently recognized one of the two recognized words when two recognized words that are continuously recognized by recognition belong to different areas; Have.

  It is possible to suppress the omission of the change in the display state of the uttered portion.

FIG. 1 is a block diagram illustrating the functional configuration of the presentation support device 10 according to the first embodiment. FIG. 2 is a diagram showing an example of a slide. FIG. 3 is a flowchart illustrating the procedure of the extraction word data generation process according to the first embodiment. FIG. 4 is a flowchart illustrating the procedure of the voice recognition process according to the first embodiment. FIG. 5 is a flowchart illustrating the procedure of the display control process according to the first embodiment. FIG. 6 is a block diagram of the functional configuration of the presentation support device 20 according to the second embodiment. FIG. 7 is a diagram illustrating a configuration example of the presentation support system 3 according to the third embodiment. FIG. 8 is a diagram illustrating a configuration example of the presentation support system 4 according to the third embodiment. FIG. 9 is a diagram showing an application example to an electronic conference system. FIG. 10 is a diagram showing an application example to an electronic conference system. FIG. 11 is a diagram showing an example of implementation in the presentation support system. FIG. 12 is a diagram illustrating a hardware configuration example of a computer that executes the presentation support program according to the first to third embodiments.

  Hereinafter, a presentation support device, a presentation support system, a presentation support method, and a presentation support program according to the present application will be described with reference to the accompanying drawings. Note that this embodiment does not limit the disclosed technology. Then, the respective embodiments can be appropriately combined within the range in which the processing contents are not inconsistent.

[One side of the function of the presentation support device]
FIG. 1 is a block diagram of the functional configuration of the presentation support device according to the first embodiment. The presentation support device 10 illustrated in FIG. 1 is a page screen included in a document having the same content, for example, a document related to a progress agenda or slide materials, shared by a plurality of people, and a speaker on a page screen included in the document. A presentation support service that highlights a portion corresponding to a word recognized from a spoken voice.

  Here, in the following description, as an example, assuming that the above-mentioned highlight display function is added to the presentation software, one or a plurality of slides included in the document file created using the presentation software are displayed on the display device. It is assumed that the presentation is advanced by displaying it on the display 5. You can import text and graphics, as well as content created by other application programs, into this slide. For example, you can import documents created with word processing software, tables and graphs created with spreadsheet software, images and videos taken with an imaging device, and images and videos edited with image editing software. It can be imported.

  The presentation support device 10 is a computer that executes the above-mentioned presentation support service.

  As one embodiment, the information processing device such as a desktop or notebook personal computer can be adopted as the presentation support device 10. In addition, the presentation support device 10 can employ not only the above-mentioned stationary terminal such as a personal computer but also various mobile terminal devices. For example, examples of the mobile terminal device include a mobile communication terminal such as a smartphone, a mobile phone or a PHS (Personal Handyphone System), and a slate terminal such as a PDA (Personal Digital Assistants) in its category.

  In addition, in the present embodiment, as an example, it is assumed that the presentation support apparatus 10 provides the above-mentioned presentation support service as a stand-alone program that executes the above-mentioned presentation software independently without depending on external resources. Although the details will be described later, the above-mentioned presentation support service is not limited to the implementation provided as a stand-alone. For example, it is possible to construct a client-server system by providing a client device that executes presentation software with a server device that provides the above-mentioned presentation support service. In addition, it is also possible to construct a thin client system in which the presentation software is executed by the server device and the execution result is transmitted to the client terminal for display.

  Here, as a part of the above-mentioned presentation support service, the above-mentioned presentation support device 10 recognizes the following recognition of two recognition words when the two recognition words recognized by speech from the utterance extend over a plurality of areas on the slide. Change the display state of the area where the word appears. Therefore, when a word in a region with a small absolute number of words is uttered, even if the utterance frequency of words in other regions is high, the display state of the region can be changed to highlight display, etc. It is possible to suppress skipping without changing the display state of a region with a small number. Therefore, it is possible to suppress the omission of the change in the display state of the uttered portion.

[Peripheral equipment]
As shown in FIG. 1, the presentation support device 10 is connected to a microphone 1, a display device 5, and an input device 7. Peripheral devices such as the microphone 1, the display device 5, and the input device 7 are connected to the presentation support device 10 by wire or wirelessly.

  The microphone 1 is a device that converts voice into an electric signal. The term “microphone” used here is an abbreviation for microphone.

  For example, the microphone 1 can be attached to a speaker, for example, a presenter who gives a presentation. In this case, a headset type or tie pin type microphone can be attached to a predetermined position of the presenter's body or clothes, or a hand type microphone can be carried by the presenter. The microphone 1 can also be installed at a predetermined position in a range where the presenter's speech can be collected. In this case, an attached or stationary microphone can be adopted as the microphone 1. In any of these cases, a microphone having any type of directivity can be adopted as the microphone 1, but voices other than the utterance of the presenter, such as utterances of listeners or noise such as noise, are collected. In order to suppress the noise, the sensitivity of the microphone can be limited to the presenter's speaking direction. It should be noted that the microphone 1 can adopt any conversion method such as a dynamic type, an electret condenser type, or a condenser type. An analog signal obtained by collecting a voice in the microphone 1 is converted into a digital signal and then input to the presentation support device 10.

  The display device 5 is a device that displays various types of information.

  For example, the display device 5 may be a liquid crystal display or an organic EL (electroluminescence) display that realizes display by light emission, or a projector that realizes display by projection. Further, the number of display devices 5 installed is not necessarily limited to one, and a plurality of display devices 5 may be installed. In the following, as an example, it is assumed that a projector and a screen for displaying an image projected by the projector are mounted as a shared display device viewed by both presenters and listeners who are participants of the presentation.

  The display device 5 displays the presentation screen according to an instruction from the presentation support device 10, for example. For example, the display device 5 displays a slide of a document file opened by the presentation software operating on the processor of the presentation support device 10. At this time, the display device 5 can switch and display the slides included in the document file automatically or manually. For example, an arbitrary slide designated by the presenter via the input device 7 can be displayed, or when the slide show function of the presentation software is set to the ON state, the slides are created in the document file in the order of the pages created. The included slides can be switched and displayed.

  The input device 7 is a device that receives an instruction input for various information.

  For example, when the display device 5 is implemented as a projector, a laser pointer for pointing a position on the slide projected on the screen can be implemented as the input device 7. That is, among laser pointers, there is also a laser pointer with a remote control function, which is provided with an operation unit such as various buttons for advancing and returning a slide page. The operation unit included in the laser pointer with the remote control function can be used as the input device 7. In addition, a mouse or a keyboard is adopted as the input device 7, and a screen or a predetermined part of the presenter is imaged in order to perform sensing of the position of the pointer pointed by the laser pointer, detection of the line of sight of the presenter, and gesture recognition. An image sensor for inputting an image may be adopted as the input device 7. When the display device 5 is implemented as a liquid crystal display, the input device 7 may be a touch sensor attached on the liquid crystal display.

  As an example, the input device 7 receives designation of a document file to be executed by the presentation software on the processor of the presentation support device 10, operation of advancing a slide page, operation of returning a slide page, and the like. Thus, the operation accepted via the input device 7 is output to the presentation support device 10.

[Configuration of presentation support device 10]
Subsequently, a functional configuration of the presentation support device 10 according to the present embodiment will be described. As shown in FIG. 1, the presentation support device 10 includes an input / output I / F (InterFace) unit 11, a storage unit 13, and a control unit 15. Note that FIG. 1 shows a solid line representing a data input / output relationship, but FIG. 1 shows only a minimum part for convenience of description. That is, the input / output of data relating to each processing unit is not limited to the illustrated example, and the input / output of data other than the illustrated units, for example, between the processing units and the processing units, between the processing units and the data, and between the processing units and the outside. Data may be input / output between devices.

  The input / output I / F unit 11 is an interface that performs input / output with peripheral devices such as the microphone 1, the display device 5, and the input device 7.

  As one aspect, the input / output I / F unit 11 outputs various operations input from the input device 7 to the control unit 15. Further, the input / output I / F unit 11 outputs the image data of the slide output from the control unit 15 to the display device 5, and outputs a highlight instruction for the area included in the slide or an instruction to cancel the same to the display device 5. To do The input / output I / F unit 11 also outputs the audio data input from the microphone 1 to the control unit 15.

  The storage unit 13 is a device that stores data used for various programs such as an application program including an OS (Operating System) executed by the control unit 15 and presentation software.

  As an embodiment, the storage unit 13 is implemented as a main storage device in the presentation support device 10. For example, the storage unit 13 can employ various semiconductor memory elements such as a RAM (Random Access Memory) and a flash memory. The storage unit 13 can also be implemented as an auxiliary storage device. In this case, an HDD (Hard Disk Drive), an optical disk, an SSD (Solid State Drive), or the like can be adopted.

  The storage unit 13 stores document data 13a, extracted word data 13b, and recognition word data 13c as an example of data used in the program executed by the control unit 15. In addition to these data, the storage unit 13 can also store other electronic data, for example, definition data regarding display state change control. The extracted word data 13b and the recognized word data 13c other than the document data 13a will be described in accordance with the description of the processing unit that registers or refers to each data.

  The document data 13a is data relating to a document.

  As one embodiment, a document file in which one or a plurality of slides are created by using presentation software can be adopted as the document data 13a. Text and graphics, as well as content created by other application programs, can be imported into such slides. For example, you can import documents created with word processing software, tables and graphs created with spreadsheet software, images and videos taken with an imaging device, and images and videos edited with image editing software. It can be imported. As described above, in order to realize a keyword search by voice recognition, content other than text should be provided with meta-information including a character string such as a description phrase or description of the content before the presentation starts. You can

  The control unit 15 has an internal memory that stores various programs and control data, and executes various processes by these.

  As one embodiment, the control unit 15 is implemented as a central processing unit, so-called CPU (Central Processing Unit). The control unit 15 does not necessarily have to be mounted as a central processing unit, and may be mounted as an MPU (Micro Processing Unit) or a DSP (Digital Signal Processor). As described above, the control unit 15 may be implemented as a processor, and it does not matter whether the type is a general-purpose type or a specialized type. The control unit 15 can also be realized by a hard-wired logic such as an ASIC (Application Specific Integrated Circuit) or an FPGA (Field Programmable Gate Array).

  The control unit 15 virtually implements the following processing unit by executing various programs. For example, as shown in FIG. 1, the control unit 15 includes an extraction unit 15a, a recognition unit 15b, a calculation unit 15c, an estimation unit 15d, and a display control unit 15e.

  The extraction unit 15a is a processing unit that extracts, from the slides included in the document file, words to be registered in the dictionary data used for voice recognition as the extracted word data 13b.

  As one embodiment, the extraction unit 15a can automatically start the process of extracting the extracted word data 13b, or can manually start the process. For example, when automatically starting, when the presentation software saves the document file in the storage unit 13 and then closes it, or when the document file is saved in the storage unit 13 while editing the document file through the presentation, Can be activated. Further, in the case of starting by manual setting, the process can be activated when an instruction to execute the pre-process of the presentation is received via the input device 7. In any case, the process is started by reading the document file corresponding to the save or execution instruction among the document files included in the document data 13a stored in the storage unit 13.

  Explaining the generation of the extracted word data 13b, the extracting unit 15a receives the instruction to execute the preprocessing of the document file in which the saving is executed among the document files included in the document data 13a stored in the storage unit 13 or the document. Read the file. Here, as an example, the case where the extraction unit 15a reads the document file from the storage unit 13 is illustrated, but the acquisition route of the document file is not limited to this. For example, the extraction unit 15a can also obtain the document file from an auxiliary storage device such as a hard disk or an optical disk, or a removable medium such as a memory card or a USB (Universal Serial Bus) memory. The extraction unit 15a can also acquire a document file by receiving it from an external device via a network.

  Subsequently, the extraction unit 15a divides the slide included in the previously read document file into a plurality of areas. For example, the extraction unit 15a divides the slide in units of one sentence, line, paragraph, or the like. In this case, the extraction unit 15a scans the character string included in the slide, detects a delimiter character corresponding to a space, a punctuation mark, or a line feed, and sets the delimiter character at the boundary of the area. The extraction unit 15a divides the character string included in the slide before and after the boundary. As a result, the slide is divided into a plurality of areas for each delimiter. Then, the extraction unit 15a assigns an index for identifying the area to the area obtained by dividing the slide. In addition, although the case where the slide is automatically divided is illustrated here, the slide may be manually divided by designating the boundary of the area through the input device 7 or the like.

  After dividing the slide, the extraction unit 15a selects one of the plurality of areas included in the slide. Subsequently, the extraction unit 15a extracts a word by performing natural language processing on the character string included in the previously selected area. For example, the extraction unit 15a extracts a word whose part of speech is a noun, a word forming a phrase, or the like among the morphemes obtained by performing morphological analysis on the character strings in the area. Then, the extraction unit 15a assigns, to each of the previously extracted words, an index assigned to a region including the word. After that, the extraction unit 15a repeatedly executes the above-mentioned word extraction and the above-mentioned index assignment until all the areas included in the slide are selected.

  After words have been extracted from all areas in this way, the extraction unit 15a registers in the storage unit 13 the extracted word data 13b in which the reading of the word k and the index idx are associated with each word included in the slide. To do.

  The recognition unit 15b is a processing unit that executes voice recognition.

  As one embodiment, the recognition unit 15b is activated when the presentation software receives a presentation start instruction with the document file opened, and waits until a voice signal of a predetermined time length is input from the microphone 1. For example, it waits for an audio signal of a time length of at least one frame, for example, 10 msec, to be input. Then, each time the recognition unit 15b inputs a voice signal of a predetermined time length from the microphone 1, the recognition unit 15b performs voice recognition such as word spotting on the voice signal in a past fixed period of time dating back from the time when the voice signal was input. To do. Note that word spotting is a general term for a method of registering necessary words in advance and extracting the registered words from a voice signal. At this time, the recognition unit 15b extracts the extracted word data regarding the slide included in the document file in which the presentation software is being executed among the extracted word data 13b stored in the storage unit 13 and on the display device 5. Apply 13b to word spotting. Accordingly, the recognition unit 15b recognizes whether or not there is a word extracted from each area included in the slide being displayed in the utterance by the speaker such as the presenter. Then, when the word reading is recognized from the audio signal, the recognition unit 15b registers the recognition word data 13c associated with the word and the time at which the word was recognized in the storage unit 13. In addition, when the same word is recognized a plurality of times over time, the last, that is, the latest recognized time is registered in the storage unit 13.

  Then, the recognition unit 15b determines whether or not there is a word in the recognized word data 13c stored in the storage unit 13 for which a predetermined period has elapsed since the word was registered in the storage unit 13. For example, the recognition unit 15b determines, for each word included in the recognition word data 13c, the difference between the time registered in association with the word and the time at which the recognition unit 15b refers to the recognition word data 13c, that is, the current time. It is determined whether or not a predetermined threshold is exceeded. At this time, the recognition unit 15b can change the threshold value used for the above determination according to the unit in which the slide is divided, for example, one sentence, line or paragraph. For example, when the slide is divided into rows, it can be assumed that the number of characters read in one area is approximately 20 to 30. In this case, as an example of the threshold value, 3 seconds can be used by calculating the time required for reading from the average reading speed of the explanation voice, which is 7 beats / second to 8 beats / second. In addition, when the slide is divided into paragraphs, it can be assumed that a longer time than line units is spent reading aloud. In this case, the number of rows × 3 seconds can be used as an example of the above threshold.

  Here, if there is a word that has been stored in the storage unit 13 for a predetermined period, for example, the number of lines × 3 seconds, there is a high possibility that the description of the slide region including the word has been completed. If such a word is left, there is a high possibility that the area where the explanation is completed is highlighted. Therefore, the recognition unit 15b deletes the record related to the word from the recognized word data 13c stored in the storage unit 13. On the other hand, when there is no word for which a predetermined period has elapsed since being registered in the storage unit 13, there is a high possibility that the explanation regarding the slide region in which the word included in the recognized word data 13c appears has not been completed. Therefore, the recognition unit 15b leaves the words included in the recognized word data 13c stored in the storage unit 13 as they are without deleting them.

  The recognition unit 15b also determines whether the slide page displayed on the display device 5 has been changed. For example, the recognition unit 15b determines whether the slides have been switched by the slide show, or the operation of advancing the slide page or the operation of returning the slide page has been accepted via the input device 7. At this time, when the page of the slide displayed on the display device 5 is changed, it is highly possible that the explanation by the speaker such as the presenter is also switched from the slide of the page before the change to the slide of the page after the change. In this case, the recognition unit 15b deletes the recognized word data 13c stored in the storage unit 13. On the other hand, if the slide page displayed on the display device 5 has not been changed, there is a high possibility that the page explained by the speaker will not change. In this case, the recognition unit 15b leaves the words included in the recognized word data 13c stored in the storage unit 13 as they are without deleting them.

  Through a series of these operations, the recognition unit 15b recognizes a word that is likely to be explained by the presenter in the displayed slide. Hereinafter, a word included in the extracted word data 13b will be referred to as an “extracted word”, and a word included in the recognized word data 13c will be referred to as a “recognized word” to distinguish the labels from each other. is there.

  The calculation unit 15c is a processing unit that calculates the position of the recognized word in the area.

  As one embodiment, the calculation unit 15c reads out, from the recognized word data 13c stored in the storage unit 13, records related to two recognized words that have been continuously speech-recognized. For example, a record of the recognized word having the latest entry time and the recognized word recognized immediately before the recognized word is read. In the following, the former recognized word of the two recognized words may be referred to as a “first recognized word” and the latter recognized word may be referred to as a “second recognized word”. Furthermore, the index associated with the first recognition word, that is, the area where the first recognition word appears on the slide is described as “first area” and is associated with the second recognition word. The index, that is, the area where the second recognized word appears on the slide may be referred to as the "second area".

  Then, the calculation unit 15c determines whether or not the index associated with the first recognized word and the index associated with the second recognized word are different. In other words, the calculation unit 15c determines whether the first area and the second area are different. At this time, if the first area and the second area are the same, it is possible that the presentation has not transitioned from the area in which the explanation has been given up to the new area in which the description regarding the next explanation has been given. Can be estimated to be high. On the other hand, when the first area and the second area are different, it is more likely that the presentation has transitioned from the area where the explanation was given up to that time to a new area where there is a description regarding the next explanation. It can be estimated that In this case, the calculation unit 15c further determines whether or not the first recognized word is a word that appears in a plurality of areas in the slide being displayed on the display device 5. For example, the calculation unit 15c compares the index associated with the extracted word that matches the first recognized word among the extracted words included in the extracted word data 13b with the index of the area included in the slide being displayed. , It is determined whether the index associated with the extracted word that matches the first recognized word matches a plurality of indexes of the region included in the slide being displayed.

  Here, if the first recognized word is a word that does not appear in a plurality of areas in the slide being displayed, the presentation is changed from the area in which the explanation was given up to the new area in which the description about the next explanation is given. It is more likely that this is a transitional stage. In this case, the calculation unit 15c determines whether or not the number of extracted words associated with the index of the first region in the extracted word data 13b stored in the storage unit 13 is a predetermined value, for example, “2” or more. To judge. That is, it is determined whether or not the absolute number of words included in the first area on the slide is small. At this time, if the absolute number of words included in the first area is small, highlighting of the first area is omitted unless highlighting is performed at the stage when the first recognized word is obtained by voice recognition. The possibility of doing so increases. In this case, the estimation unit 15d, which will be described later, estimates that the first region is the utterance location. On the other hand, when the absolute number of words included in the first area is not small, the calculation unit 15c uses the first value as a parameter for more finely determining whether or not the progress of the presentation has reached the stage of transition between areas. The position on the area of the recognition word and the position on the area of the second recognition word are calculated.

  For example, the calculation unit 15c calculates the position t1 on the area of the first recognition word and the position t2 on the area of the second recognition word according to the following equations (1) and (2). . Here, “N” in the following formula (1) and the following formula (2) indicates a region, and “K” indicates a recognition word. Then, “INDEX1 (N, K)” in the following expression (1) indicates the index number of the first character of the recognition word K included at the beginning of the area N. On the other hand, “INDEX2 (N, K)” in the following formula (2) indicates the index number of the last character of the recognition word K included at the end of the area N. Note that the following description will be given here, assuming that the index number starts from 0 as an example.

t1 = INDEX1 (N, K) / (number of N characters-1) ... Equation (1)
t2 = INDEX2 (N, K) / (number of N characters-1) ... Equation (2)

  FIG. 2 is a diagram showing an example of a slide. FIG. 2 shows a slide S1 including four areas E1, E2, E3 and E4. Under the situation where the slide S1 shown in FIG. 2 is displayed on the display device 5, when the speaker utters “Tomorrow's weather. It's Kanto ...”, the recognition word is “tomorrow”, Obtained in order of "weather" and "Kanto". In this case, "Kanto" corresponds to the first recognition word, and "weather" corresponds to the second recognition word. Then, the first area E2 including "Kanto" is different from the second area E1 including "weather", and "Kanto" does not appear in other areas on the slide S1. Furthermore, since the first area E2 includes two or more extracted words “Kanto” and “region”, the absolute number of words in the first area E2 is identified as not being a small number.

  By satisfying these conditions, the position t1 of the first recognized word on the region is calculated according to the above equation (1). Specifically, the first character of the recognition word "Kanto" included at the beginning of the first area E2 is "Kan", and the first character "Kan" is the character string "Kanto region" included in the first area E2. Since it is also the head of “”, the index number INDEX1 (E2, Kanto) is calculated as “0”. Therefore, t1 is calculated as "0" by calculating "0 / (4-1)". On the other hand, the final character of the recognition word “weather” included at the end of the second area E1 is “ki”, and this final character “ki” is the same as that of the character string “tomorrow weather” included in the second area E1. Since it corresponds to the fifth character, that is, the last character, the index number is the fifth integer "4" counting from 0, which is calculated as INDEX2 (E1, weather). Therefore, t2 is calculated as "1" by the calculation of "4 / (5-1)".

  By calculating such t1 and t2, the following evaluation can be performed from t1 and t2. For example, the position t1 on the area of the first recognition word calculated as described above has a value that increases as the first recognition word “Kanto” is closer to the beginning on the first area “E2”. While the value is calculated to be low, the value is calculated to be higher as the position is farther from the beginning. Therefore, it is possible to evaluate how close the first recognition word is to the beginning on the first area "E2" by the position t1 on the area of the first recognition word. Further, the position t2 on the area of the second recognition word calculated as described above has a value as the second recognition word “weather” is closer to the end on the second area “E1”. While the value is calculated to be high, the value is calculated to be lower as the position is farther from the end. Therefore, the position t2 on the area of the second recognition word makes it possible to evaluate how close the second recognition word is to the end on the second area "E1".

  The estimation unit 15d is a processing unit that estimates a region corresponding to the utterance portion among the regions included in the slide being displayed.

  As one embodiment, the estimation unit 15d is a word in which the first recognized region and the second region are different from each other, and the first recognized word is a word that does not appear in a plurality of regions in the slide being displayed, and is included in the first region. When the number of extracted words to be extracted is not equal to or larger than the predetermined value, the first area is estimated to be the uttered portion. On the other hand, the estimation unit 15d is a word in which the first region and the second region are different from each other, and the first recognized word is a word that does not appear in a plurality of regions in the currently displayed slide, and is an extracted word included in the first region. When the number of is equal to or larger than a predetermined value, it is determined by the following determination whether the first region is estimated as the utterance portion or the region having the largest number of recognized words is estimated as the utterance portion. That is, the estimation unit 15d determines that the position of the first recognition word is within a predetermined range from the beginning of the first area and the position of the second recognition word is within a predetermined range from the end of the second area. By determining whether or not there is, it is determined in more detail whether or not the progress of the presentation has reached the stage of transition between regions.

  Specifically, the estimation unit 15d determines whether or not the position t1 on the region of the first recognized word is equal to or less than a predetermined threshold Th1, eg, “0.2”. At this time, when the position t1 on the area of the first recognition word is equal to or less than the threshold Th1, the estimating unit 15d determines that the position t2 on the area of the second recognition word is a predetermined threshold Th2, for example, “0.8”. It is further determined whether or not the above. At this time, if the position t1 on the area of the first recognized word is less than or equal to the threshold Th1 and the position t2 on the area of the second recognized word is greater than or equal to the threshold Th2, the presentation follows the description content of the slide. It can be inferred that there is a high possibility that it has just progressed and transitioned between regions. In this case, the estimation unit 15d estimates the first area as the utterance location. On the other hand, if the position t1 of the first recognized word on the region is not less than or equal to the threshold Th1 or the position t2 of the second recognized word on the region is not greater than or equal to the threshold Th2, it may not be immediately after the transition between the regions. . In this case, the region having the largest number of recognized words is estimated as the uttered portion. For example, the estimation unit 15d counts the number of recognition words associated with the index of each area included in the displayed slide, and estimates the area having the largest number of recognition words as the utterance location.

  In this way, by estimating the region corresponding to the uttered portion based on the positions of the first recognized word and the second recognized word, it is possible to prevent the display state of the uttered portion from being omitted. For example, in the example of FIG. 2, when the slide S1 is displayed on the display device 5 and the speaker utters “Tomorrow's weather. Kanto ... The number of recognized words in E1 is "2", and the number of recognized words in the area E2 is "1". Therefore, if the region having the largest number of recognized words is uniformly estimated as the uttered portion, the region E1 is highlighted even though the presentation progresses to the region E2. In this case, when the presentation proceeds to the area E3 without uttering the extracted word “region” included in the area E2, it is difficult for the situation in which the number of recognized words in the area E2 is the maximum to occur, and the highlight of the area E2 is generated. Display leakage may occur. On the other hand, in the present embodiment, the position t1 on the area of the first recognition word is “0” and the position t2 on the area of the second recognition word is “1”, so that the threshold value Th1 is “0.2”. ”Or less and the threshold value Th2 is“ 0.8 ”or more. As a result, the first area E2 is estimated to be the utterance portion, so that the highlight display omission in the area E2 can be suppressed.

  In addition, here, the case where the region having the largest number of recognized words is estimated as the uttered portion is illustrated, but any known method other than this can be used. For example, a score may be calculated for each region by weighting according to an arbitrary parameter such as the appearance frequency of the recognized word on the slide, and the region having the highest score may be estimated as the uttered portion.

  Furthermore, the estimation unit 15d further adds whether or not the distance between the first region and the second region is within a predetermined threshold, for example, α rows, as an additional important matter, and when the distance between the two is within the threshold, For the first time, the first area can be estimated as the utterance location. The reason for adding such a determination is that it can be determined that the possibility that the presentation is proceeding according to the description content of the slide is higher when the distance between the regions is shorter.

  The display control unit 15e is a processing unit that executes display control on the display device 5. Here, of the display controls executed by the display controller 15e, one aspect of display control regarding slides and display control regarding highlights will be described.

[Slide display control]
As one aspect, when the document file is opened by the presentation software, the display control unit 15e causes the display device 5 to display the slide included in the document file. At this time, the display control unit 15e may display the slide of the first page among the slides included in the document file, or may display the slide of the last edited page. After that, the display control unit 15e receives the instruction to start the presentation, and then causes the display device 5 to display the slide related to the region corresponding to the utterance location estimated by the estimation unit 15d. Further, when the display control unit 15e receives a page switching instruction via the input device 7, the display control unit 15e changes the slide to be displayed on the display device 5. For example, when the operation of advancing a page is received, the display control unit 15e causes the display device 5 to display the slide of the next page of the slide being displayed. When the operation of returning the page is accepted, the display control unit 15e causes the display device 5 to display the slide of the previous page of the slide being displayed.

[Highlight display control]
As another aspect, the display control unit 15e repeatedly executes the following processing from the reception of the presentation start instruction to the reception of the presentation end instruction. That is, the display control unit 15e performs highlight display of the area of the utterance location estimated by the estimation unit 15d. The “highlight display” referred to here means not only the highlight display in a narrow sense, that is, the display control of brightening or inverting the background color, but also the highlight display in a broad sense. For example, general highlighting can be arbitrarily executed, such as enclosing the explanation part, emphasizing the filling of the explanation part, and emphasizing the font (font size, underline or italic). The highlighted display may be returned to the normal display when a cancel operation is accepted via the input device 7. Further, as a matter of course, when neither of the regions is output as the explanation part by the estimation unit 15d, for example, when the recognized word does not exist, the highlight display is not executed on the slide being displayed.

[Process flow]
Next, a processing flow of the presentation support device 10 according to the present embodiment will be described. Note that, here, the description will be given in order of (1) extraction word data generation processing, (2) voice recognition processing, and (3) display control processing executed by the presentation support apparatus 10.

(1) Extracted Word Data Generation Processing FIG. 3 is a flowchart showing the procedure of the extracted word data generation processing according to the first embodiment. This process can be started automatically or manually. For example, when automatically starting, when the presentation software saves the document file in the storage unit 13 and then closes it, or when the document file is saved in the storage unit 13 while editing the document file through the presentation, Can be activated. Further, in the case of starting by manual setting, the process can be activated when an instruction to execute the pre-process of the presentation is received via the input device 7. In any case, the process is started by reading the document file corresponding to the instruction to execute the save or pre-process from the document files included in the document data 13a stored in the storage unit 13.

  As shown in FIG. 3, the extraction unit 15a divides the slide included in the document file into a plurality of areas in units of one sentence, line, paragraph, or the like (step S101). Subsequently, the extraction unit 15a assigns an index for identifying each area to the area obtained in step S101 (step S102).

  Then, the extraction unit 15a selects one index from the indexes assigned in step S102 (step S103). Subsequently, the extraction unit 15a extracts a word whose part of speech is a noun from the morphemes obtained by performing morphological analysis on the character string in the area of the index selected in step S103 (step S104). After that, the extraction unit 15a gives each word extracted in step S104 an index assigned to a region including the word (step S105).

  Then, the extraction unit 15a repeatedly executes the above-described steps S103 to S105 until all the indexes assigned in step S102 are selected (No in step S106).

  After that, when all the indexes assigned in step S102 are selected (Yes in step S106), the extraction unit 15a extracts the extracted word data 13b associated with the reading of the word k and the index idx for each word included in the slide. It is registered in the storage unit 13 (step S107), and the process ends.

(2) Voice Recognition Processing FIG. 4 is a flowchart showing the procedure of the voice recognition processing according to the first embodiment. This process is started when the presentation software receives a presentation start instruction with the document file opened, and is repeatedly executed until a presentation end instruction is received.

  As shown in FIG. 4, the recognition unit 15b waits until a voice signal having a predetermined time length is input from the microphone 1, for example, a voice signal having a time length of at least one frame, for example, 10 msec is input (step). S301).

  Then, when a voice signal of a predetermined time length is input from the microphone 1 (Yes in step S301), the recognition unit 15b executes voice recognition such as word spotting on the voice signal (step S302). When word spotting is performed in step S302, the slide is included in the document file in which the presentation software is being executed among the extracted word data 13b stored in the storage unit 13, and is being displayed on the display device 5. Extracted word data regarding a certain slide is applied as dictionary data for voice recognition.

  At this time, when the word is recognized from the voice signal (Yes in step S303), the recognition unit 15b stores the recognized word data 13c in which the word recognized in step S302 and the time at which the word is recognized are associated with each other. Is registered (step S304), and the process proceeds to step S305.

  On the other hand, when the voice signal of the predetermined time length is not input from the microphone 1 or when the word is not recognized from the voice signal (No in step S301 or No in step S303), the subsequent process is skipped and the process proceeds to step S305. .

  Here, the recognition unit 15b determines whether or not there is a word in the recognized word data 13c stored in the storage unit 13 for which a predetermined period has elapsed since being registered in the storage unit 13 (step S305). Then, when there is a word for which a predetermined period has elapsed since being registered in the storage unit 13 (Yes in step S305), the recognition unit 15b deletes the record related to the word from the recognized word data 13c stored in the storage unit 13. (Step S306). If there is no word for which a predetermined period has elapsed since being registered in the storage unit 13 (No in step S305), the process of step S306 is skipped and the process proceeds to step S307.

  After that, the recognition unit 15b determines whether the page of the slide displayed on the display device 5 has been changed (step S307). At this time, if the page of the slide displayed on the display device 5 is changed (Yes in step S307), the recognition unit 15b deletes the recognition word data 13c stored in the storage unit 13 (step S308), and in step S301. Returning to the processing, the processing from step S301 above is repeatedly executed. If the slide page displayed on the display device 5 has not been changed (No in step S307), the process returns to step S301 without executing the process of step S308.

(3) Display Control Process FIG. 5 is a flowchart showing the procedure of the display control process according to the first embodiment. This process is, for example, a process that is executed in parallel with the voice recognition process shown in FIG. 4, and is started when the presentation software receives a presentation start instruction with the document file open, and It is repeatedly executed until the end instruction is received. Note that the cycle in which the execution of the processing is repeated may be the same as or different from the voice recognition processing shown in FIG. 4, and is assumed to be executed in synchronization with the voice recognition processing shown in FIG. It can be done asynchronously.

  As shown in FIG. 5, the calculation unit 15c recognizes the first recognized word having the latest entry time among the recognized word data 13c stored in the storage unit 13 and the recognized word immediately before the first recognized word. A record with the second recognized word is read (step S501).

  Then, the calculation unit 15c determines whether or not the index associated with the first recognized word is different from the index associated with the second recognized word, that is, whether the first region and the second region are It is determined whether they are different (step S502).

  At this time, if the index associated with the first recognized word and the index associated with the second recognized word are different (Yes in step S502), the presentation starts from the area in which the explanation is given until the next. It can be inferred that there is a higher possibility that it is at the stage where the description has changed to a new area. In this case, the calculation unit 15c further determines whether or not the first recognized word is a word that appears in a plurality of regions in the slide being displayed on the display device 5 (step S503).

  Here, when the first recognized word is a word that does not appear in a plurality of regions in the slide being displayed (No in step S503), there is a description regarding the next explanation from the region where the presentation has been explained up to that point. It is more likely that this is a stage where a transition has been made to a new area. In this case, the calculation unit 15c determines whether or not the number of extracted words associated with the index of the first region in the extracted word data 13b stored in the storage unit 13 is a predetermined value, for example, “2” or more. Is determined (step S504).

  Then, when the number of extracted words included in the first area is equal to or larger than a predetermined value (Yes in step S504), calculation is performed as a parameter for more finely determining whether or not the progress of the presentation has reached the stage of transition between areas. The unit 15c calculates the position of the first recognized word on the region and the position of the second recognized word on the region (step S505).

  Then, the estimation unit 15d determines whether or not the position t1 on the region of the first recognized word is equal to or less than a predetermined threshold Th1, eg, “0.2” (step S506). At this time, when the position t1 on the area of the first recognition word is less than or equal to the threshold Th1 (step S506 Yes), the position t2 on the area of the second recognition word is equal to or larger than a predetermined threshold Th2, for example, “0.8” or more. Is further determined (step S507).

  Here, when the position t1 on the area of the first recognition word is less than or equal to the threshold value Th1 and the position t2 on the area of the second recognition word is greater than or equal to the threshold value Th2, the presentation follows the description content of the slide. It can be inferred that there is a high possibility that it has just progressed and transitioned between regions. In this case, the estimation unit 15d further determines whether or not the distance between the first region and the second region is within a predetermined threshold value, for example, α rows (step S508). At this time, when the distance between the first area and the second area is within the threshold value (step S508 Yes), the estimation unit 15d estimates the first area as the utterance location (step S509).

  If the number of extracted words included in the first area is not equal to or larger than the predetermined value (No in step S504), highlight display is not performed when the first recognized word is obtained by voice recognition. The possibility of highlight display omission increases. Also in this case, the estimation unit 15d estimates the first region as the utterance location (step S509).

  On the other hand, when the index associated with the first recognized word and the index associated with the second recognized word are the same, the first recognized word appears in a plurality of regions in the slide being displayed. In the case of a word, if the position t1 on the area of the first recognized word is not less than or equal to the threshold Th1, the position t2 on the area of the second recognized word is not greater than or equal to the threshold Th2, or if the first area and the second area When the distance of the area is not within the threshold (step S502 No, step S503 Yes, step S506 No, step S507 No or step S508 No), the estimation unit 15d estimates the area having the largest number of recognized words as the utterance location (step S510).

  After that, the display control unit 15e performs highlight display regarding the area estimated as the uttered portion in step S509 or step S510 (step S511), and ends the processing.

  The determinations of step S502, step S503, and step S506 to step S508 shown in FIG. 5 do not have to be performed in the order shown in the figure, and may be performed in any order, or may be performed by parallel processing. it can.

[One side of effect]
As described above, in the presentation support device 10 according to the present embodiment, when the two recognized words that are speech-recognized from the utterance span a plurality of areas on the slide, the succeeding recognized word of the two recognized words appears. Change the display status of the area to be edited. Therefore, when a word in a region with a small absolute number of words is uttered, even if the utterance frequency of words in other regions is high, the display state of the region can be changed to highlight display, etc. It is possible to suppress skipping without changing the display state of a region with a small number. Therefore, according to the presentation support device 10 according to the present embodiment, it is possible to prevent the omission of the change in the display state of the uttered portion.

  Further, in the presentation support apparatus 10 according to the present embodiment, when the number of extracted words included in the first area is equal to or larger than a predetermined value, the position of the first recognized word is within a predetermined range from the beginning of the first area. And the position of the second recognized word is within a predetermined range from the end of the second area. Therefore, according to the presentation support device 10 according to the present embodiment, it is possible to quickly switch the highlight display from the first area to the second area.

  In the above-described first embodiment, a conference, a lecture, etc. are assumed as an example of the presentation, but the presentation is not limited to the scenes of the conference, the lecture, etc. That is, a presentation includes a scene in which audio such as a play or a movie is output together with a video according to a predetermined scenario.

  Therefore, in the present embodiment, the scenario data such as plays and movies, for example, the document data 13a, the extracted word data 13b, and the recognition word data 13c related to the dialogue are stored, and the contents such as plays and movies are reproduced on the display device 5. Under such a situation, an embodiment in which an area corresponding to an utterance portion such as a dialogue among the slides included in the document data 13a is picked up and displayed will be described.

  FIG. 6 is a block diagram of the functional configuration of the presentation support device 20 according to the second embodiment. Compared with the presentation support device 10 shown in FIG. 1, the presentation support device 20 shown in FIG. 6 stores video data 21a in the storage unit 21, and the control unit 23 has the same function as the display control unit 15e. The difference is that the units have different display control units 23a. In the following, the parts having the same functions as those of the presentation support device 10 shown in FIG. 1 are designated by the same reference numerals, and the description thereof will be omitted.

  The video data 21a shown in FIG. 6 is data relating to video content such as a moving image. As an example of this video content, content such as a play or movie can be adopted. In connection with this, as the dialogue included in the video content, a document such as a script or script or a document in which the dialogue is extracted is stored in the storage unit 21 as the document data 13a.

  The display control unit 23a illustrated in FIG. 6 is similar to the display control unit 15e illustrated in FIG. 1 in that the display control unit 23a executes display control on the display device 5, but the display control contents are different. That is, the display control unit 23a reproduces the video content included in the video data 21a on the display device 5, instead of displaying the slide of the document data 13a. Then, the display control unit 23a superimposes the dialogue corresponding to the region of the document data 13a estimated by the estimation unit 15d as the uttered portion on the video content to be displayed. In other words, the display control unit 23a realizes the change of the display state of the area estimated as the utterance location by extracting and displaying the dialogue corresponding to the area estimated as the utterance location by the estimation unit 15d. In this way, when the dialogue is superimposed and displayed on the video content, the position and size of the dialogue displayed as a character string can be set arbitrarily, but as an example, a display method similar to that of the subtitle supermarket can be adopted.

  By mounting the video data 21a and the display control unit 23a as described above, the presentation support apparatus 20 according to the present embodiment can realize a presentation in which an area corresponding to an utterance portion such as a dialogue of a play or a movie is picked up and displayed. Also in this presentation support device 20, it is possible to suppress the omission of the change in the display state of the uttered portion, as in the first embodiment.

  Although the embodiments of the disclosed device have been described so far, the present invention may be implemented in various different forms other than the embodiments described above. Therefore, other embodiments included in the present invention will be described below.

[Application example of document file]
In the above-described first embodiment, the case where the document created by the presentation software is used is illustrated, but the document created by another application program can also be used. For example, the processing shown in FIGS. 3 to 5 can be similarly applied by replacing the page included in the document file of the word processing software with a slide or the sheet included in the document file of the spreadsheet software with a slide.

Distributed and integrated
Further, each constituent element of each illustrated device may not necessarily be physically configured as shown in FIG. 1 or FIG. That is, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or part of the device may be functionally or physically distributed / arranged in arbitrary units according to various loads and usage conditions. It can be integrated and configured. For example, the extraction unit 15a, the recognition unit 15b, the calculation unit 15c, the estimation unit 15d, the display control unit 15e, or the display control unit 23a may be connected via a network as an external device of the presentation support device 10 or the presentation support device 20. Good. Further, another device has the extraction unit 15a, the recognition unit 15b, the calculation unit 15c, the estimation unit 15d, the display control unit 15e, or the display control unit 23a, and each device has a network connection to cooperate with each other to support the above presentation. The functions of the device 10 or the presentation support device 20 may be realized.

[Other implementation example]
In the above-described first embodiment, the case where the presentation support device 10 or the presentation support device 20 independently executes the above-mentioned presentation software without depending on external resources and executes the processing relating to FIGS. 3 to 5 is illustrated. However, other implementations can also be adopted. For example, a client server system can be constructed by providing a client that executes the presentation software with a server that executes a part or all of the processes shown in FIGS. In this case, the server device can be mounted by installing the presentation support program that realizes the above-mentioned presentation support service as package software or online software. For example, the server device may be implemented as a Web server that provides the above-mentioned presentation support service, or may be implemented as a cloud that provides the above-mentioned presentation support service by outsourcing. In this case, the client uploads the highlight display start instruction, for example, at least the information designating the document file used for the presentation, to the server device, and then the presentation is started. When the presentation is started, the client uploads the audio signal collected from the microphone 1 or the result of the voice recognition process, and uploads the page information of the slide every time the page of the slide being displayed on the display device 5 is switched. That is, the extraction word data generation processing and the voice recognition processing can be executed either on the client side or on the server side. As a result, the server device can execute at least the processing shown in FIG. Furthermore, the client can be constructed as a thin client system by transmitting operation information regarding an input device (not shown) to the server and displaying only the processing result transmitted from the server on the display device 5. In this case, various resources such as document data are held by the server, and the presentation software is also installed as a virtual machine by the server. For example, when the presentation software is executed on the client side, the identification information of the area to be highlighted, such as the index of the above area, may be transmitted from the server to the client, and when it is implemented as a thin client system. The display data of the slide in which the highlighted portion is displayed or the difference data from the screen before the highlighted display may be transmitted from the server to the client. In the first embodiment described above, it is assumed that the presentation software incorporating the above-mentioned presentation support processing is executed. It is also possible to plug the presentation support program into the presentation software.

[Application example 1 to thin client system]
FIG. 7 is a diagram illustrating a configuration example of the presentation support system 3 according to the third embodiment. FIG. 7 shows an example in which the presentation support device 10 shown in FIG. 1 is implemented as a thin client system. As an example, the presentation support system 3 illustrated in FIG. 7 has the client terminal 30 with a minimum function, and the server device 300 manages resources such as applications and files. Although a thin client system is illustrated as an example of the presentation support system 3 here, it should be additionally noted that the above-described presentation support service can be applied to a general-purpose client server system as described later.

  As shown in FIG. 7, the presentation support system 3 includes a client terminal 30 and a server device 300.

  The client terminal 30 may be an information processing device such as a desktop or notebook personal computer. In addition, as the client terminal 30, not only the above-mentioned stationary terminal such as a personal computer but also various mobile terminal devices can be adopted. For example, examples of mobile terminal devices include smartphones, mobile communication terminals such as mobile phones and PHS, and slate terminals such as PDA in its category.

  The server device 300 is a computer that provides the above-mentioned presentation support service.

  As an embodiment, the server device 300 can be installed by installing a presentation support program that realizes the above-mentioned presentation support service as package software or online software. For example, the server device 300 may be implemented as a Web server that provides the above presentation support service, or may be implemented as a cloud that provides the above presentation support service by outsourcing.

  The client terminal 30 and the server device 300 are connected to each other via the network NW in a communicable state. As an example of the network NW, any type of communication network such as LAN, VPN (Virtual Private Network) including the Internet can be adopted regardless of wired or wireless.

  As shown in FIG. 7, the client terminal 30 includes a microphone 1, a display device 5, an input device 7, and a data transfer unit 24. Note that, in FIG. 7, the same reference numerals are given to the functional units that perform the same functions as the functional units illustrated in FIG. 1, for example, the microphone, the display device, and the input device, and the description thereof will be omitted.

  The data transfer unit 34 is a processing unit that controls transfer of various data to and from the server device 300.

  As an embodiment, the data transfer unit 34 is virtually realized by, for example, a processor such as a CPU included in the client terminal 30 executing a program for a client of the thin client system.

  For example, the data transfer unit 34 transmits the voice data input by the microphone 1, the operation information received by the input device 7, and the like to the server device 300. The data transfer unit 34 also receives display data to be displayed on the desktop screen including the execution result of the presentation software executed by the server device 300, that is, the screen of the display device 5. For example, when a document file is displayed as a slide show by the presentation software, the window generated by the presentation software is displayed on the full screen, so that the desktop screen and the window screen have the same display content. Here, the data transfer unit 34 can receive the display data of the desktop screen transmitted by the server device 300 at an arbitrary frame rate, and displays the desktop screen only when there is a difference in the display data of the desktop screen. It can also receive data. At this time, the display data of the desktop screen transmitted from the server device 300 may be the entire desktop screen or a part of the desktop screen, for example, display data of a difference between frames.

  As described above, various types of data exchanged between the client terminal 30 and the server device 300 may be compression-encoded from the viewpoint of suppressing traffic, and various types of data from the viewpoint of security. The encryption may be performed.

  As shown in FIG. 7, the server device 300 has a storage unit 320 and a control unit 340. In addition to the functional units shown in FIG. 7, the server device 300 has various functional units included in a known computer, for example, a functional unit such as a communication I / F unit that controls communication with other devices. It doesn't matter.

  The storage unit 320 is a device that stores data used for various programs such as an application program including an OS and presentation software executed by the control unit 340.

  As one embodiment, the storage unit 320 is implemented as a main storage device in the server device 300. For example, the storage unit 320 can employ various semiconductor memory devices, such as a RAM or a flash memory. The storage unit 320 can also be implemented as an auxiliary storage device. In this case, HDD, optical disk, SSD, etc. can be adopted.

  For example, the storage unit 320 stores the document data 321, the extracted word data 322, and the recognition word data 323 illustrated in FIG. 7 as an example of the data used in the program executed by the control unit 340. The document data 321, the extracted word data 322, and the recognition word data 323 are document data so that the server device 300 can identify which of the client terminals 30 is connected to the server device 300. 321, the storage area in which the extracted word data 322 and the recognized word data 323 are stored is distinguished according to the identification information of the client terminal 30, or the document data 321, the extracted word data 322, and the recognized word data 323 identify the client terminal 30. It is the same data as the document data 13a, the extracted word data 13b, and the recognition word data 13c shown in FIG. 1 except that it is further associated with information.

  The control unit 340 has an internal memory that stores various programs and control data, and executes various processes by these.

  In one embodiment, the control unit 340 is implemented as a central processing unit, so-called CPU. The control unit 340 does not necessarily have to be implemented as a central processing unit, and may be implemented as an MPU or DSP. The control unit 340 can also be realized by hard-wired logic such as ASIC or FPGA.

  The control unit 340 virtually implements the following processing units by executing various programs. For example, the control unit 340 includes an extraction unit 341, a recognition unit 342, a calculation unit 343, an estimation unit 344, and a display control unit 345, as illustrated in FIG. 7.

  The extraction unit 341, the recognition unit 342, the calculation unit 343, and the estimation unit 344 illustrated in FIG. 7 perform the same processing as the extraction unit 15a, the recognition unit 15b, the calculation unit 15c, and the estimation unit 15d illustrated in FIG. Is.

  The display control unit 345 is a processing unit that executes display control of the display device 5 of the client terminal 30.

  Here, the display control unit 345 transmits display data to be displayed on the desktop screen of the client terminal 30, that is, the screen of the display device 5 at a predetermined frame rate or when the desktop screen is updated. At this time, the display control unit 345 may not necessarily transmit the display data of the desktop screen to the client terminal 30 when the desktop screen is not updated. Furthermore, the display control unit 345 may transmit display data of the entire desktop screen, or may transmit a part of the desktop screen, for example, display data of a difference between frames. In parallel with the transmission of such a desktop screen, the display control unit 345, similar to the display control unit 15e shown in FIG. 1, displays the above slide according to the operation information of the input device 7 transmitted from the client terminal 30. The display data of the window screen generated by the presentation software is updated by executing the display control, the display control of the highlight, and the like. In this way, when the desktop screen is transmitted, the updated contents of the window screen are transmitted from the server device 300 to the client terminal 30.

  As described above, when the presentation support system 3 according to the present embodiment is implemented as a thin client system, the extraction unit 341 of the server device 300 executes the processing illustrated in FIG. 3 and the recognition unit 342 illustrated in FIG. It is possible to execute a voice recognition process. In this voice recognition process, there is no difference in the processing content except that voice data transmitted from the client terminal 30 to the server device 300 is obtained instead of directly obtaining voice data from the microphone 1 in step S301. Furthermore, the calculation unit 343, the estimation unit 344, and the display control unit 345 of the server device 300 can execute the display control process illustrated in FIG.

[Application example 2 to thin client system]
FIG. 8 is a diagram illustrating a configuration example of the presentation support system 4 according to the third embodiment. FIG. 8 shows an example in which the presentation support device 20 shown in FIG. 6 is implemented as a thin client system. As an example, the presentation support system 4 shown in FIG. 8 has the client terminal 40 having a minimum function, and the server device 400 manages resources such as applications and files. Although a thin client system is illustrated here as an example of the presentation support system 4, it should be additionally noted that the above-mentioned presentation support service can be applied to a general-purpose client server system as described later.

  As shown in FIG. 8, the presentation support system 4 includes a client terminal 40 and a server device 400.

  As the client terminal 40, an information processing device such as a desktop or notebook personal computer can be adopted. In addition, the client terminal 40 may be not only the above-mentioned stationary terminal such as a personal computer but also various mobile terminal devices. For example, examples of mobile terminal devices include smartphones, mobile communication terminals such as mobile phones and PHS, and slate terminals such as PDA in its category.

  The server device 400 is a computer that provides the above-mentioned presentation support service.

  As one embodiment, the server apparatus 400 can be implemented by installing a presentation support program that realizes the above-mentioned presentation support service as package software or online software. For example, the server device 400 may be implemented as a Web server that provides the above-mentioned presentation support service, or may be implemented as a cloud that provides the above-mentioned presentation support service by outsourcing.

  The client terminal 40 and the server device 400 are connected to each other via the network NW so that they can communicate with each other. As an example of the network NW, any type of communication network such as the Internet, LAN, VPN, etc. can be adopted regardless of wired or wireless.

  As shown in FIG. 8, the client terminal 40 further includes a video input device 8 in addition to the microphone 1, the display device 5, the input device 7, and the data transfer unit 34 shown in FIG. 7. As the video input device 8, for example, an image pickup device equipped with an image pickup device such as a CCD (Charge Coupled Device) or a CMOS (Complementary Metal Oxide Semiconductor) can be adopted. Accordingly, the video content such as a play imaged by the video input device 8 can be reproduced on the display device 5 in real time without holding the video data 21a in advance. Further, the video data 21a may be held in the server device 400 and the video content may be reproduced by the display control unit 445. Note that, in FIG. 8, the same reference numerals are given to the functional units that exhibit the same functions as the functional units illustrated in FIG. 1, for example, the microphone, the display device, and the input device, and the description thereof will be omitted.

  The data transfer unit 44 is a processing unit that controls transfer of various data to and from the server device 400.

  As one embodiment, the data transfer unit 44 is virtually realized by, for example, a processor such as a CPU included in the client terminal 40 executing a client program of the thin client system. For example, the data transfer unit 44 transmits to the server device 400 the voice data input by the microphone 1 and the operation information received by the input device 7. Further, the data transfer unit 44 receives the display data regarding the dialogue corresponding to the uttered portion, and further the attribute information such as the size and the position where the dialogue is displayed.

  As described above, various types of data exchanged between the client terminal 40 and the server device 400 may be compression-encoded from the viewpoint of suppressing traffic, and various types of data from the viewpoint of security. The encryption may be performed.

  As shown in FIG. 8, the server device 400 has a storage unit 420 and a control unit 440. It should be noted that the server device 400 has various functional units included in a known computer other than the functional units illustrated in FIG. 8, for example, a functional unit such as a communication I / F unit that controls communication with another device. It doesn't matter.

  The storage unit 420 is a device that stores data used for various programs such as an application program including an OS and presentation software executed by the control unit 440.

  As one embodiment, the storage unit 420 is implemented as a main storage device in the server device 400. For example, various semiconductor memory devices such as a RAM and a flash memory can be used as the storage unit 420. The storage unit 420 can also be implemented as an auxiliary storage device. In this case, HDD, optical disk, SSD, etc. can be adopted.

  For example, the storage unit 420 stores the document data 421, the extracted word data 422, and the recognition word data 423 illustrated in FIG. 8 as an example of data used in the program executed by the control unit 440. The document data 421, the extracted word data 422, and the recognition word data 423 are document data so that the server device 400 can identify which client terminal 40 among the client terminals 40 connected to the server device 400. The storage area in which 421, the extracted word data 422, and the recognized word data 423 are stored is distinguished for each identification information of the client terminal 40, or the document data 421, the extracted word data 422, and the recognized word data 423 identify the client terminal 40. The data is the same as the document data 13a, the extracted word data 13b, and the recognition word data 13c shown in FIG. 6 except that it is further associated with information. The video data 21a shown in FIG. 6 can be further stored in the storage unit 420.

  The control unit 440 has an internal memory that stores various programs and control data, and executes various processes by these.

  In one embodiment, the controller 440 is implemented as a central processing unit, a so-called CPU. The control unit 440 does not necessarily have to be implemented as a central processing unit, and may be implemented as an MPU or DSP. The control unit 440 can also be realized by hard-wired logic such as ASIC or FPGA.

  The control unit 440 virtually realizes the following processing units by executing various programs. For example, the control unit 440 includes an extraction unit 441, a recognition unit 442, a calculation unit 443, an estimation unit 444, and a display control unit 445, as illustrated in FIG. 8. The extraction unit 441, the recognition unit 442, the calculation unit 443, the estimation unit 444, and the display control unit 445 illustrated in FIG. 8 are the extraction unit 15a, the recognition unit 15b, the calculation unit 15c, the estimation unit 15d, and the display control unit illustrated in FIG. 23 a is a processing unit that executes the same processing.

  As described above, when the presentation support system 4 according to the present embodiment is implemented as a thin client system, the extraction unit 441 of the server device 400 executes the processing illustrated in FIG. 3 and the recognition unit 442 illustrated in FIG. It is possible to execute a voice recognition process. In this voice recognition process, there is no difference in the processing content except that voice data transmitted from the client terminal 30 to the server device 300 is obtained instead of directly obtaining voice data from the microphone 1 in step S301. Furthermore, the calculation unit 443, the estimation unit 444, and the display control unit 445 of the server device 400 can execute the display control process illustrated in FIG.

[Application example to a general-purpose client-server system]
7 and 8 exemplify the case where the presentation support system 3 or 4 is implemented as a thin client system, but it does not necessarily have to be implemented as a thin client system, and may be implemented as a general-purpose client server system. it can.

  For example, the presentation support apparatus 10 or 20 shown in FIG. 1 or 6 is used as a client terminal, and a server unit (not shown) accommodating the client terminal includes a calculation unit 15c among processing units included in the presentation support apparatus 10 or 20. A processing unit such as the estimation unit 15d and the display control unit 15e or 23a may be mounted. In this case, the presentation support device 10 or 20 which is a client terminal executes the voice recognition process shown in FIG. 4, and every time a recognition word is obtained, the additional recognition word or the entire recognition word data is transmitted to a server device (not shown). By doing so, the recognition word data is stored for each client terminal on the server device (not shown). This prevents voice data from being transmitted between the client and the server.

  As described above, the above presentation support service can be applied to a general-purpose client server system.

[Example of application to electronic conference system]
For example, in the above-described first embodiment, the case where the speaker and the listener share one display device 5 has been illustrated, but the speaker and the listener do not necessarily share one display device. The above-mentioned presentation support service can be applied to a situation where the same display content is shared between display devices. For example, in a communication such as an electronic conference, there is a situation in which each participant participates as at least one or both of a speaker and a listener. In this case, if the computers connected to the respective display devices are connected to each other via the network, they may exist in remote places.

  FIG. 9 is a diagram showing an application example to an electronic conference system. For example, as shown in FIG. 9, client terminals 10A and 10B having the same functions as those of the presentation support device 10 shown in FIG. 1 are connected via a network NW, and communication tools such as a communication tool on the client terminals 10A and 10B are used. It can be applied to the scene where the screen sharing application program is executed. As a result, the same display content, for example, a document file for presentation software is shared between the display devices of the client terminals 10A and 10B. Under such a situation, at least one of the client terminals 10A and 10B executes the processing shown in FIGS. 3 to 5 to utilize the utterance and line of sight of the user of the client terminal 10A or 10B. It is possible to highlight the area corresponding to the explanation part among the slides included in the document file.

  FIG. 10 is a diagram showing an application example to an electronic conference system. For example, as shown in FIG. 10, client terminals 30A and 30B having the same functions as the client terminal 30 shown in FIG. 7 and the server device 300 shown in FIG. 7 are connected via the network NW, and The present invention can be applied to a situation in which a communication tool such as a screen sharing application program is executed on the server device 300. As a result, the same display content, for example, a document file for presentation software is shared between the display devices of the client terminals 30A and 30B. Under such circumstances, the server device 300 executes the processing shown in FIGS. 3 to 5 to use the utterance of the user of the client terminal 30A or 30B to explain the slides included in the document file. The area corresponding to the location can be highlighted.

[How to change the display status]
The presentation support apparatus 10 shown in FIG. 1 and the presentation support system 3 shown in FIG. 7 exemplify the case where the slide is always displayed. However, the area corresponding to the uttered portion is set on condition that the uttered portion is estimated. By excerpting, the display state of the area may be changed to support the presentation. FIG. 11 is a diagram showing an example of implementation in the presentation support system. FIG. 11 shows a case where the presentation support system 4 shown in FIG. 8 is applied to a presentation such as a conference. As shown in FIG. 11, the video input device 8 included in the client terminal 40 illustrated in FIG. 8 is installed at a position where an image of a speaker such as a presenter can be captured, and the display device 5 included in the client terminal 40 is a listener. Will be installed in a viewable state. Although not shown in FIG. 11, the server device 400 is connected to the client terminal 40. The display device 5 may be installed at a remote place as long as it can communicate with the client terminal 40. Under the configuration shown in FIG. 11, it is possible to change the display state of the area by extracting the area corresponding to the uttered portion on the condition that the uttered portion is estimated, thereby supporting the presentation. For example, a region corresponding to the uttered portion is displayed as a subtitle supermarket 5a in a superimposed manner on the video input by the video input device 8.

[Presentation support program]
Further, the various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer or a workstation. Therefore, in the following, an example of a computer that executes a presentation support program having the same functions as those in the above embodiments will be described with reference to FIG.

  FIG. 12 is a diagram illustrating a hardware configuration example of a computer that executes the presentation support program according to the first to third embodiments. As illustrated in FIG. 12, the computer 100 includes an operation unit 110a, a speaker 110b, a camera 110c, a display 120, and a communication unit 130. Further, the computer 100 has a CPU 150, a ROM 160, an HDD 170, and a RAM 180. Each unit of these 110 to 180 is connected via a bus 140.

  As shown in FIG. 12, the HDD 170 has a presentation support program 170a that exhibits the same functions as the extraction unit 15a, the recognition unit 15b, the calculation unit 15c, the estimation unit 15d, and the display control unit 15e described in the first embodiment. Is memorized. Further, the HDD 170 stores the presentation support program 170a that exhibits the same functions as the extraction unit 15a, the recognition unit 15b, the calculation unit 15c, the estimation unit 15d, and the display control unit 23a described in the second embodiment. It doesn't matter. The presentation support program 170a may be integrated or separated similarly to each component of the extraction unit 15a, the recognition unit 15b, the calculation unit 15c, the estimation unit 15d, and the display control unit 15e illustrated in FIG. That is, the HDD 170 does not necessarily need to store all the data shown in the first embodiment or the second embodiment, and the data used for the processing may be stored in the HDD 170.

  Under such an environment, the CPU 150 reads out the presentation support program 170a from the HDD 170 and expands it in the RAM 180. As a result, the presentation support program 170a functions as a presentation support process 180a, as shown in FIG. The presentation support process 180a expands various data read from the HDD 170 in the area allocated to the presentation support process 180a in the storage area of the RAM 180, and executes various processes using the expanded data. For example, the processing shown in FIGS. 3 to 5 is included as an example of the processing executed by the presentation support process 180a. Note that in the CPU 150, not all the processing units described in the above-described first embodiment may operate, and the processing unit corresponding to the processing to be executed may be virtually realized.

  The presentation support program 170a does not have to be stored in the HDD 170 or the ROM 160 from the beginning. For example, the presentation support program 170a is stored in a “portable physical medium” such as a flexible disk, a so-called FD, a CD-ROM, a DVD disk, a magneto-optical disk, an IC card, which is inserted into the computer 100. Then, the computer 100 may acquire and execute the presentation support program 170a from these portable physical media. Further, the presentation support program 170a is stored in another computer or a server device connected to the computer 100 via a public line, the Internet, a LAN, a WAN, etc., and the computer 100 acquires the presentation support program 170a from these. You may make it execute.

1 microphone 5 display device 7 input device 10 presentation support device 11 input / output I / F unit 13 storage unit 13a document data 13b extracted word data 13c recognized word data 15 control unit 15a extraction unit 15b recognition unit 15c calculation unit 15d estimation unit 15e display Control unit

Claims (8)

A recognition unit that performs voice recognition on voice data using a word extracted from a character string included in the display content of the document file for each divided region,
A display control unit that changes a display state of an area including a subsequently recognized recognition word of the two recognition words when two recognition words continuously recognized by the voice recognition belong to different areas. When,
A presentation support device comprising:   The two recognition words are a first recognition word recognized at the latest time by the voice recognition and a second recognition word recognized immediately before the first recognition word. 1. The presentation support device according to 1. When the word extracted from the character string included in the area to which the first recognition word belongs is equal to or larger than a predetermined value, the position on the area of the first recognition word and the position on the area of the second recognition word are determined. Further has a calculation unit for calculating,
When the position on the area of the first recognition word is within a predetermined range from the beginning and the position on the area of the second recognition word is within a predetermined range from the end, the display control unit may: The presentation support device according to claim 2, wherein a display state of an area including the first recognized word is changed.   If the position of the first recognition word on the region is not within a predetermined range from the beginning, or if the position of the second recognition word on the region is not within the predetermined range from the end, it is obtained by the voice recognition. The presentation support device according to claim 3, wherein the display state of the region having the larger number of recognition words is changed.   When the distance between the area to which the first recognition word belongs and the area to which the second recognition word belongs is within a predetermined threshold, the display control unit displays the display state of the area including the first recognition word. The presentation support device according to claim 2, 3 or 4, wherein A presentation support system having a first device and a second device,
The first device is
A display device for displaying,
A microphone for inputting voice,
A transmitter for transmitting voice data input by the microphone to the second device,
The second device is
A recognition unit that performs voice recognition on the voice data by using a word extracted from a character string included in the region in which the display content of the document file is divided,
When two recognized words continuously recognized by the voice recognition belong to different areas, the control for changing the display state of the area including the subsequently recognized one of the two recognized words is performed. A display control unit for the display device,
A presentation support system comprising: For each area into which the display content of the document file is divided, the words extracted from the character string included in the area are used to perform voice recognition on the voice data,
When two recognized words continuously recognized by the voice recognition belong to different areas, a display state of an area including a subsequently recognized one of the two recognized words is changed.
A presentation support method characterized in that processing is executed by a computer. For each area into which the display content of the document file is divided, the words extracted from the character string included in the area are used to perform voice recognition on the voice data,
When two recognized words continuously recognized by the voice recognition belong to different areas, a display state of an area including a subsequently recognized one of the two recognized words is changed.
A presentation support program that causes a computer to execute processing.

Images