JP2016194877A - Explanation support device, explanation support method, and explanation support program - Google Patents

Abstract

An object of the present invention is to suppress a decrease in the estimation accuracy of an explanation location. An explanation support apparatus performs line-of-sight detection on a display unit, performs voice recognition, and determines an explanation portion corresponding to a result of voice recognition among pages of a document displayed on the display unit. The first state where the explanation location corresponding to the speech recognition and the explanation location corresponding to the line of sight detection match, the second state or the line of sight where the explanation location corresponding to the speech recognition and the description location corresponding to the line of sight detection do not match It is determined which of the third states in which the position of the line of sight by detection is not detected in the screen of the display unit 5, and an explanation location for performing highlighting is estimated based on the determined explanation state. . [Selection] Figure 1

Description

  The present invention relates to an explanation support apparatus, an explanation support method, and an explanation support program.

  Pointing devices such as laser pointers and mouse cursors have been used in remote conferences and presentations. When such a pointing device is operated by a presenter such as a presenter, labor is devoted to other than the explanation. For this reason, a technique has also been proposed in which speech presenting is used to present to the presenter or the audience the location that the presenter is explaining.

JP 2005-338173 A JP 2004-246398 A JP 2004-7358 A

  However, in order to increase the response to present the explanation location, it may be necessary to associate the speech recognition result with the explanation location even when a sufficient number of words are not recognized. Are inherently limited in their accuracy. Therefore, if an error is included in the result of speech recognition, the explanation location may not be estimated correctly.

  For this reason, it is conceivable to use both speech recognition and line-of-sight detection in order to suppress a decrease in the estimation accuracy of the explanation location. For example, when the explanation location estimated from the speech recognition result matches the explanation location estimated from the line-of-sight detection result, the explanation location may be highlighted.

  However, even when voice recognition and line-of-sight detection are used together, it may not be possible to suppress a decrease in the estimation accuracy of the explanation location. This is because the presenter does not always read out while watching the portion to be explained.

  For example, there may be a case where the presenter reads and explains the previous part while visually confirming the part to be described next in advance. In such a case, when the explanation location estimated from the result of speech recognition and the explanation location estimated from the result of line-of-sight detection change from a matching state to a mismatching state, the explanation is still continued. The highlighting of the explanation part is canceled by mistake.

  In addition, when the explanation location estimated from the result of speech recognition does not match the explanation location estimated from the result of eye gaze detection, the same word as the current explanation location accidentally exists ahead of the line of sight that moved in advance. In this case, there is a case where the point of the line of sight that has moved in advance is erroneously highlighted as the explanation part, even though the explanation of the current explanation part is continued.

  In one aspect, an object of the present invention is to provide an explanation support device, an explanation support method, and an explanation support program that can suppress a decrease in estimation accuracy of an explanation location.

  An explanation support apparatus according to an aspect includes a line-of-sight detection unit that performs line-of-sight detection on a predetermined display unit, a voice recognition unit that executes voice recognition, and a result of the voice recognition among document pages displayed on the display unit A recognition result determination unit for determining an explanation location corresponding to the first state where the explanation location corresponding to the speech recognition and the explanation location corresponding to the line-of-sight detection match, the explanation location corresponding to the speech recognition, and the Explanation for determining which explanation state is in a second state where the explanation location corresponding to the gaze detection does not match or a third state where the position of the gaze by the gaze detection is not detected in the screen of the display unit It has a state determination part and an estimation part which estimates the description location which performs highlighting based on the determined description state.

  It is possible to suppress a decrease in estimation accuracy of the explanation location.

FIG. 1 is a diagram illustrating a functional configuration of the explanation support apparatus according to the first embodiment. FIG. 2 is a diagram illustrating an example of an explanation state. FIG. 3 is a diagram illustrating an example of the correspondence between the continuity of the explanation state and the explanation location. FIG. 4 is a flowchart illustrating the procedure of the explanation support process according to the first embodiment. FIG. 5 is a diagram illustrating a functional configuration of the explanation support apparatus according to the second embodiment. FIG. 6 is a diagram illustrating an example of movement of a stop point. FIG. 7 is a diagram illustrating an example of a correspondence relationship between the explanation state and the reading aloud state and the explanation portion. FIG. 8 is a flowchart illustrating the procedure of the explanation support process according to the second embodiment. FIG. 9 is a diagram illustrating a hardware configuration example of a computer that executes the explanation support program according to the first embodiment and the second embodiment.

  Hereinafter, an explanation support apparatus, an explanation support method, and an explanation support program according to the present application will be described with reference to the accompanying drawings. Note that this embodiment does not limit the disclosed technology. Each embodiment can be appropriately combined within a range in which processing contents are not contradictory.

  FIG. 1 is a diagram illustrating a functional configuration of the explanation support apparatus according to the first embodiment. The explanation support apparatus 10 shown in FIG. 1 provides an explanation support service that highlights explanation parts in a page displayed on the display unit 5 among pages included in a document by using both voice recognition and line-of-sight detection. To do.

  As illustrated in FIG. 1, a camera 1, a microphone 3, and a display unit 5 are connected to the explanation support apparatus 10. In the following, as an example, a case where a presenter and a listener share a document by browsing the display unit 5 on which a slide created by presentation software is displayed is illustrated. Needless to say, the scope of application extends even when documents are shared by displaying the same document on a computer.

  The camera 1 is an imaging device that includes an imaging device such as a charge coupled device (CCD) or a complementary metal oxide semiconductor (CMOS).

  For example, the camera 1 is disposed at a position facing the presenter together with the screen of the display unit 5. As an example, the camera 1 is arranged so that the presenter's face is included in the angle of view of the camera 1 by arranging the positions of the presenter and the presenter at positions calibrated between the screen and the presenter. Can be used for detecting the line of sight. With such an arrangement, the camera 1 controls a light source (not shown) arranged in the vicinity of the camera 1 to irradiate the subject with infrared rays, receives reflected light from the subject, and then converts the subject into a digital signal. Are output to the line-of-sight detection unit 11. At this time, when a presenter's eyeball is included in the subject, reflection of light on the retina is captured by the camera 1, and an image in which the pupil part of the eyeball appears brighter than other parts can be output to the line-of-sight detection unit 11. .

  The microphone 3 is a device that converts sound into an electrical signal, and is sometimes called a microphone. For example, the microphone 3 can be attached to a presenter who performs a presentation. In this case, a headset-type or tie-pin type microphone can be attached to a predetermined position of the presenter's body or clothes, or a hand-type microphone can be carried by the presenter. The microphone 3 can also be installed at a predetermined position in a range where the utterance of the presenter can be collected. In this case, the microphone 3 may be an attachment type or a stationary type microphone. In any of these cases, a microphone having any type of directivity can be adopted as the microphone 3, but sounds other than the presenter's utterance, for example, the utterance of the listener and the noise such as noise are collected. In order to suppress this, the sensitivity of the microphone can be limited to the speaking direction of the presenter. The microphone 3 can employ any conversion method such as a dynamic type, an electret capacitor type, or a capacitor type.

  The analog signal obtained by collecting the sound in the microphone 3 is converted into a digital signal and then input to the explanation support apparatus 10.

  The display unit 5 is a device that displays various types of information. For example, the display unit 5 may be a liquid crystal display or an organic EL (electroluminescence) display that realizes display by light emission, or a projector that realizes display by projection.

  For example, the display unit 5 displays a presentation screen according to an instruction from the explanation support apparatus 10. Specifically, the display unit 5 displays a slide of a document opened by presentation software that operates on the explanation support apparatus 10. In this case, the display unit 5 can display any slide specified by the presenter via a pointing device such as a laser pointer or a mouse cursor, which is not shown by the presenter, among slides included in the document. When the slide show function is set to ON, the slides included in the document file can be switched and displayed in the order in which the slides are created.

  As illustrated in FIG. 1, the explanation support device 10 includes a gaze detection unit 11, a gaze determination unit 12, a voice recognition unit 13, a document acquisition unit 14, a recognition result determination unit 15, and an explanation state determination unit 16. , A history storage unit 16a, an explanation location estimation unit 17, and a highlight display control unit 18. Note that the explanation support apparatus 10 may include various functional units included in a known computer, for example, functional units such as various input / output devices and audio output devices, in addition to the functional units illustrated in FIG.

  The line-of-sight detection unit 11 is a processing unit that performs line-of-sight detection.

  As one embodiment, the line-of-sight detection unit 11 applies an algorithm such as a corneal reflection method to the subject image output from the camera 1, and a viewpoint position pointed by the direction of the line of sight from the center position of the pupil of the eyeball, so-called gaze point. Is detected. Thus, besides detecting the position of the line of sight using the corneal reflection method, the position of the line of sight can also be detected using another method. For example, by dividing the screen of the display unit 5, learning the shape of the eye that sees the divided area, and performing eye-gaze detection by executing template matching with the eye shape detected from the subject image input from the camera 1. Can also be done. Alternatively, the user may wear a headset that detects the position of the line of sight and acquire the position of the line of sight detected by the headset.

  As a result of the line-of-sight detection, coordinate data of the gazing point located inside or outside the screen of the display unit 5 is output from the line-of-sight detection unit 11 to the stop point detection unit 12a.

  The line-of-sight determination unit 12 is a processing unit that determines the state of the line of sight using the result of line-of-sight detection by the line-of-sight detection unit 11. As illustrated in FIG. 1, the line-of-sight determination unit 12 includes a stop point detection unit 12a and an in-screen determination unit 12b.

  The stop point detection unit 12 a is a processing unit that detects a stop point from the result of the line-of-sight detection by the line-of-sight detection unit 11.

  As one embodiment, the stop point detection unit 12a monitors whether or not the gazing point detected by the line-of-sight detection unit 11 is stopped within a predetermined range for a predetermined period, for example, 30 msec to 300 msec. Then, when the gazing point remains within a predetermined range for a predetermined period, the stationary point detection unit 12a performs predetermined statistical processing, for example, average processing, on the coordinate data of the gazing point detected over the period. Thus, the representative value of the coordinate data of the gazing point is calculated. The representative value of the coordinate data of the gazing point obtained in this way is output to the in-screen determination unit 12b as a stop point.

  The in-screen determination unit 12 b is a processing unit that determines whether or not a gazing point or a stop point exists inside the screen of the display unit 5.

  As one embodiment, when the stop point is detected by the stop point detection unit 12a, the in-screen determination unit 12b determines whether or not the stop point exists within a coordinate range corresponding to the inside of the screen of the display unit 5. Determine. Thereafter, the in-screen determination unit 12b outputs to the explanation state determination unit 16 a determination result indicating whether the stop point exists inside or outside the screen of the display unit 5 together with the coordinate data of the stop point.

  The voice recognition unit 13 is a processing unit that performs voice recognition.

  As an embodiment, the voice recognition unit 13 is activated when the presentation software receives a presentation start instruction in a state in which the document is opened, and waits until a voice signal having a predetermined time length is input from the microphone 3. For example, it waits for an audio signal having a time length of at least one frame, for example, 10 msec. Then, every time a voice signal having a predetermined time length is input from the microphone 3, the voice recognition unit 13 refers to a recognition dictionary (not shown) and performs voice recognition such as word spotting on the voice signal. After that, the voice recognition unit 13 outputs information such as a word obtained as a result of the voice recognition and the time when the word is recognized to the associating unit 15b.

  The document acquisition unit 14 is a processing unit that acquires a document used for presentation.

  As one embodiment, when receiving a presentation start instruction, the document acquisition unit 14 acquires a document displayed on the display unit 5 by an application program such as presentation software. For example, the document acquisition unit 14 can acquire a document developed on a memory by an arbitrary application program such as presentation software, word processor software, spreadsheet software, or image editing software.

  The recognition result determination unit 15 is a processing unit that determines an explanation location corresponding to the result of voice recognition among the slides displayed on the display unit 5. As shown in FIG. 1, the recognition result determination unit 15 includes an explanation unit extraction unit 15a and an association unit 15b.

  The explanation unit extraction unit 15a is a processing unit that extracts a section obtained by dividing a slide included in a document as an explanation unit.

  As an embodiment, the explanation unit extraction unit 15a divides each slide included in the document acquired by the document acquisition unit 14 into units such as paragraphs, lines, and sentences using indent information included in the slide. For example, the explanation unit extracting unit 15a scans a character string included in the slide, detects a delimiter character corresponding to a space, a punctuation point, or a line feed, and sets the delimiter character as a boundary. The explanation unit extraction unit 15a divides the character string included in the slide before and after such a boundary. As a result, the slide is divided into a plurality of sections for each delimiter. After the slide is divided into a plurality of sections in this way, the explanation unit extracting unit 15a determines, for each section, the range of coordinates that the section forms on the slide, for example, the coordinates of the upper left and lower right vertices of the section and the coordinates The word included in the section is output to the associating unit 15b as an explanation unit. Although the case where the slide is automatically divided is illustrated here, the slide may be manually divided by designating the boundary of the section via an input device or the like.

  The associating unit 15b is a processing unit that associates the result of speech recognition with the explanation location in the document.

  As an embodiment, the association unit 15b is included in the explanation unit in the slide being displayed on the display unit 5 among the word recognized by the speech recognition unit 13 and the explanation unit extracted by the explanation unit extraction unit 15a. Compare with a word. After that, the associating unit 15b outputs the explanation unit including the most words that match the words recognized by the speech recognizing unit 13 to the explanation state determining unit 16 as the explanation location corresponding to the speech recognition.

  Based on the determination result by the line-of-sight determination unit 12 and the determination result by the recognition result determination unit 15, the explanation state determination unit 16 matches the description portion corresponding to the voice recognition and the description portion corresponding to the line-of-sight detection, It is a process part which determines whether the description state corresponds to the non-coincidence state where the description part does not correspond or the non-viewing state where the line of sight is not detected in the screen.

  As one embodiment, the explanatory state determination unit 16 determines that the explanatory state is the “non-visual state” when the in-screen determination unit 12b determines that the stop point exists outside the screen of the display unit 5. On the other hand, when the in-screen determination unit 12b determines that the stop point exists inside the screen of the display unit 5, the explanation state determination unit 16 associates the coordinate position of the stop point detected by the stop point detection unit 12a with each other. It is determined by the part 15b whether it exists in the coordinate range of the description unit matched with the result of speech recognition. At this time, the explanation state determination unit 16 sets an allowable coordinate range that is larger than the explanation unit and includes the coordinate range of the explanation unit, and the stopping point detected by the stopping point detection unit 12a within the allowable coordinate range. By determining whether or not the coordinate position of the point exists, the line-of-sight error can be absorbed. When the allowable coordinate range is set in this way, both the width direction and the height direction of the coordinate range of the explanation unit may be expanded, or one of them may be expanded. Then, the explanation state determination unit 16 determines that the explanation state is “matched state” when the coordinate position of the stop point is within the above-described allowable coordinate range, while the coordinate position of the stop point is the above-described allowable position. If it exists outside the coordinate range, it is determined that the explanation state is a “mismatch state”. Thereafter, the explanation state determination unit 16 outputs the explanation state obtained as a determination result to the history storage unit 16 a and the explanation part estimation unit 17. Thus, the history storage unit 16a stores the history of the explanation state every time the explanation state is determined.

  The explanation location estimation unit 17 is a processing unit that estimates the explanation location based on the explanation state determined by the explanation state determination unit 16.

  FIG. 2 is a diagram illustrating an example of an explanation state. As shown in FIG. 2, the explanation state transitions between “matching state”, “mismatching state”, and “non-viewing state”. These “coincidence state”, “non-coincidence state”, and “non-viewing state” are states that the computer can estimate from the result of speech recognition and the result of line-of-sight detection, and correspond to the actual situation as follows.

  That is, the “matching state” corresponds to the synchronization state explained by voice while following the current explanation part, while the “mismatching state” means that the voice explains the current explanation part and the line of sight is In order to confirm the next explanation part, it can be estimated that it corresponds to the asynchronous state which is not in the present explanation part. The “non-viewing state” corresponds to a state that is explained to the audience without looking at the material. However, if the “matching state” happens to be in the asynchronous state as described above, and the same words as the words included in the current explanation part are included at the end of the line of sight, the asynchronous state is temporarily estimated even in the asynchronous state. Is included. In addition, the “mismatch state” includes a case where when the voice misrecognition occurs in the above synchronization state, it is temporarily assumed to be an asynchronous state.

  In this way, the “matching state” basically corresponds to the synchronous state, but includes some asynchronous states as an exception, while the “mismatched state” basically corresponds to the asynchronous state, but as an exception, Includes some synchronization states.

  Here, since all of the above exceptions appear as a temporary state, as shown in FIG. 3, the explanation location estimation unit 17 stores the explanation state determined by the explanation state determination unit 16 and the history storage unit 16a. By estimating the explanation location from the past history of the explanation state stored, the exception can be excluded, and the asynchronous state can be accurately determined from the coincidence state and the synchronization state.

  FIG. 3 is a diagram illustrating an example of the correspondence between the continuity of the explanation state and the explanation location. As illustrated in FIG. 3, when the description state determination unit 16 determines “matched state”, the description location estimation unit 17 determines whether or not the match state continues for a predetermined time or more. At this time, when the coincidence state continues for a predetermined time or longer, the explanation location estimation unit 17 estimates an explanation unit in which the speech recognition result and the line-of-sight detection result match as an explanation location. On the other hand, when the coincidence state does not continue for a predetermined time or longer, the explanation location estimation unit 17 takes over the current explanation unit, that is, the explanation unit estimated as the explanation location one time before the previous explanation time as the explanation location. presume.

  When the explanation state determination unit 16 determines that the state does not match, the explanation location estimation unit 17 determines whether or not the mismatch state continues for a predetermined time or more. At this time, when the disagreement state continues for a predetermined time or longer, the explanation location estimation unit 17 estimates the explanation unit corresponding to the speech recognition as the explanation location. On the other hand, when the inconsistent state has not continued for a predetermined time or more, the explanation location estimation unit 17 takes over the current explanation unit, that is, the explanation unit estimated as the explanation location one time before the previous explanation time as the explanation location. presume.

  When the explanation state determination unit 16 determines that the state is “non-viewing state”, the explanation part estimation unit 17 estimates the explanation unit corresponding to the speech recognition as the explanation part. Since it can be assumed that there are no exceptions such as “matched state” and “mismatched state”, it is not necessary to look at the materials when it is judged as “non-viewing state” regardless of continuity. This is because it can be estimated that the situation is being explained to the audience.

  Here, the case where it is determined whether or not the same explanation state continues for a predetermined time or more in order to determine the continuity is illustrated, but it is determined that it is determined whether or not it continues for a predetermined number of times. It doesn't matter.

  The highlight display control unit 18 is a processing unit that executes highlight display of the explanation portion estimated by the explanation portion estimation unit 17. The “highlight display” here refers to a highlight display in a narrow sense, that is, not just a display control that brightens or inverts a background color, but a broad highlight display. For example, it is possible to arbitrarily execute overall highlighting such as enclosing an explanation part, emphasizing the filling of the explanation part, and emphasizing a font (font size, underline or italic).

  Note that the functions of the line-of-sight detection unit 11, the line-of-sight determination unit 12, the speech recognition unit 13, the document acquisition unit 14, the recognition result determination unit 15, the description state determination unit 16, the description location estimation unit 17, and the highlight display control unit 18 are described. The part can be implemented as follows. For example, it can be realized by causing a central processing unit, a so-called CPU (Central Processing Unit), or the like to develop and execute a process that exhibits the same function as each of the above-described processing units on a memory. These processing units do not necessarily have to be executed by the central processing unit, but may be executed by an MPU (Micro Processing Unit). Each processing unit described above can also be realized by a hard wired logic such as an application specific integrated circuit (ASIC) or a field programmable gate array (FPGA).

  The history storage unit 16a may employ various types of semiconductor memory elements such as a RAM (Random Access Memory) and a flash memory as an example. The history storage unit 16a is not necessarily a main storage device and may be an auxiliary storage device. In this case, an HDD (Hard Disk Drive), an optical disk, an SSD (Solid State Drive), or the like can be employed.

  FIG. 4 is a flowchart illustrating the procedure of the explanation support process according to the first embodiment. This process is started when the presentation software receives a presentation start instruction with the document open, and is repeatedly executed until a presentation end instruction is received.

  As illustrated in FIG. 4, the line-of-sight detection unit 11 detects the point of sight by performing line-of-sight detection on the image output from the camera 1 (step S <b> 101). Subsequently, the stop point detection unit 12a detects the stop point by monitoring whether or not the gazing point detected in step S101 is stopped within a predetermined range for a predetermined period, for example, 30 msec to 300 msec. (Step S102).

  Thereafter, the in-screen determination unit 12b determines whether or not the stop point detected in step S102 is within a coordinate range corresponding to the inside of the screen of the display unit 5 (step S103).

  Further, the speech recognition unit 13 refers to a recognition dictionary (not shown) and performs speech recognition such as word spotting on the speech signal input from the microphone 3 (step S104). Then, the explanation unit extraction unit 15a divides the slide being displayed on the display unit 5 into units of paragraphs, lines, one sentence, and the like using indent information included in the slide, and the like. The coordinate range and the words included in the section are extracted as explanation units (step S105).

  Thereafter, the associating unit 15b associates the result of the speech recognition in step S104 with the explanation unit extracted in step S105 (step S106). That is, the associating unit 15b outputs the explanation unit containing the most words that match the word recognized in step S104 among the explanation units extracted in step S105 as the explanation location corresponding to the speech recognition.

  In addition, the explanation state determination unit 16 determines that the explanation state is “matched state”, “mismatched state”, or “based on the result of line-of-sight determination in steps S102 to S103 and the result of voice recognition association in step S106. It is determined which of the “non-visual state” corresponds to the state (step S107).

  Then, the explanation location estimation unit 17 estimates the explanation location based on whether or not there is continuity of the explanation state from the explanation state determined in step S107 and the explanation state history stored in the history storage unit 16a. (Step S108). After that, the highlighting control unit 18 executes highlighting of the explanation portion estimated in step S108 (step S109), and ends the process.

[One aspect of effect]
As described above, the explanation support apparatus 10 according to the present embodiment has any of the coincidence state in which the explanation portions of the voice recognition and the line-of-sight detection coincide, the disagreement state in which they do not coincide, and the non-visual state in which the line of sight is not in the screen. Depending on whether the state is the explanation state, the explanation portion based on the voice, the line of sight or the previous estimation result is highlighted. Therefore, according to the explanation support apparatus 10 according to the present embodiment, it is possible to suppress a decrease in the estimation accuracy of the explanation location.

  In the first embodiment, the case where the synchronous state is determined from the coincidence state and the asynchronous state is determined from the disagreement state based on the presence or absence of the continuity of the explanation state has been exemplified. Therefore, in the present embodiment, a case in which the case where the synchronous state is determined from the coincidence state and the asynchronous state is determined from the disagreement state is illustrated based on the moving direction of the visual line and the moving speed of the visual line.

  FIG. 5 is a diagram illustrating a functional configuration of the explanation support apparatus according to the second embodiment. Compared to the explanation support apparatus 10 shown in FIG. 1, the explanation support apparatus 20 shown in FIG. 5 has a movement direction detection section 21a, a movement speed calculation section 21b, and a reading state determination section 21c in the line-of-sight determination section 21. The difference is that the determination logic of the explanation location estimation unit 22 is different, and the history storage unit 16a is unnecessary. Here, the same reference numerals are given to the processing units that exhibit the same functions as those of the explanation support apparatus 10 shown in FIG. 1, and the explanation thereof is omitted.

  The movement direction detection unit 21a is a processing unit that detects the movement direction of the line of sight.

  As one embodiment, the movement direction detection unit 21a determines whether the stationary point detected by the stationary point detection unit 12a is moving in the horizontal direction or the vertical direction. For example, the moving direction detection unit 21a obtains an approximate straight line connecting stop points detected over a predetermined past time, for example, several seconds. Then, the movement direction detection unit 21a detects the movement direction as “horizontal direction” when the approximate straight line is outside a predetermined range from the horizontal direction, for example, 30 degrees, and the approximate straight line is predetermined from the vertical direction. If the angle is in a range other than 30 degrees, for example, the moving direction is detected as “vertical direction”.

  The movement speed calculation unit 21b is a processing unit that calculates the movement speed of the line of sight.

  As one embodiment, the movement speed calculation unit 21b specifies a movement range of a stop point at a certain time, and then searches for characters existing in the movement range, thereby extracting characters included in the movement range. Subsequently, the movement speed calculation unit 21b converts the characters included in the movement range into reading information, that is, a phonetic character string by morphological analysis, and obtains the number of mora. Then, the moving speed calculation unit 21b calculates the average moving speed (mora / sec) by dividing the number of mora by a certain time.

  The reading aloud state determination unit 21c is a processing unit that determines whether or not the reading aloud state is in effect. When not in a reading state, it is determined as a silent reading state.

  As one embodiment, the reading aloud state determination unit 21c determines whether or not it is in a reading aloud state according to the knowledge shown in FIG. FIG. 6 is a diagram illustrating an example of movement of a stop point. FIG. 6 shows an example in which the slide is horizontally written, but in the case of vertically written, the same can be said by rereading the horizontal to the vertical and the vertical to the horizontal. As shown in FIG. 6, in the state of reading aloud, it is conceivable that the degree of freedom of eye movement is lower than that during silent reading because of the activity of uttering characters continuously, and skipping and reading are less likely to occur. It is considered to be. Furthermore, in a state of reading aloud, after a gazing point appears in the sentence, the line of sight follows the sentence, and reading back is less than in silent reading. On the other hand, in the silent reading state, the gazing point tends to move randomly or vertically.

  For these reasons, the reading aloud state determination unit 21c determines that the reading aloud state occurs when the line-of-sight moving direction is the horizontal direction and the line-of-sight moving speed is within a predetermined speed, for example, 7 or 8 mora / sec. To do. When not in a reading state, it is determined as a silent reading state. In addition, although the case where both the moving direction and the moving speed are used for the determination of the reading aloud state or the silent reading state is illustrated here, both the moving direction and the moving speed are not necessarily used, and only one of them is used. You can also.

  The explanation location estimation unit 22 estimates the explanation location based on the explanation state determined by the explanation state determination unit 16 as with the explanation location estimation unit 17 shown in FIG. The difference is in distinguishing between the principle of coincidence and inconsistency and exceptions.

  FIG. 7 is a diagram illustrating an example of a correspondence relationship between the explanation state and the reading aloud state and the explanation portion. As illustrated in FIG. 7, the explanation location estimation unit 22 is synchronized when the explanation state determination unit 16 determines “matching state” and when the reading state determination unit 21 c determines “reading state”. Presumed to be in a state. In this case, the explanation location estimation unit 22 estimates an explanation unit in which the speech recognition result and the line-of-sight detection result match as an explanation location. On the other hand, the explanation location estimation unit 22 is an exception in the asynchronous state when the explanation state determination unit 16 determines “matching state” and when the reading state determination unit 21 c determines “silent reading state”. Estimated. In this case, the explanation location estimation unit 22 estimates the explanation unit corresponding to the speech recognition as the explanation location.

  The explanation location estimation unit 22 is an exception to the synchronization state when the explanation state determination unit 16 determines that it is in a “mismatched state” and when it is determined as “reading state” by the reading aloud state determination unit 21c. Estimated. In this case, the explanation location estimation unit 22 estimates the explanation unit corresponding to the gaze detection as the explanation location. On the other hand, the explanation location estimation unit 22 estimates that the state is an asynchronous state when the explanation state determination unit 16 determines the “mismatch state” and when the reading state determination unit 21c determines the “silent reading state”. To do. In this case, the explanation location estimation unit 22 estimates the explanation unit corresponding to the speech recognition as the explanation location.

  In addition, the description location estimation part 22 estimates the description unit corresponding to speech recognition as an explanation location, when the description state determination part 16 determines with a "non-viewing state". Since it can be assumed that there are no exceptions such as “matched state” and “mismatched state”, it is explained to the audience without looking at the materials when judged as “non-viewing state”. This is because it can be estimated that the current state is in progress.

  FIG. 8 is a flowchart illustrating the procedure of the explanation support process according to the second embodiment. Similar to the explanation support process shown in FIG. 4, this process is started when the presentation software receives a presentation start instruction with the document open, and is repeatedly executed until a presentation end instruction is received.

  As illustrated in FIG. 8, the line-of-sight detection unit 11 detects the point of sight by performing line-of-sight detection on the image output from the camera 1 (step S101). Subsequently, the stop point detection unit 12a detects the stop point by monitoring whether or not the gazing point detected in step S101 is stopped within a predetermined range for a predetermined period, for example, 30 msec to 300 msec. (Step S102).

  Thereafter, the in-screen determination unit 12b determines whether or not the stop point detected in step S102 is within a coordinate range corresponding to the inside of the screen of the display unit 5 (step S103).

  Here, when the stop point exists within the coordinate range corresponding to the inside of the screen of the display unit 5 (Yes in step S201), the reading state determination unit 21c determines that the movement direction of the line of sight is the horizontal direction and the movement of the line of sight. Whether the speed falls within a predetermined speed or not is determined as to whether the reading state is silent or silent (step S202). When the stop point does not exist within the coordinate range corresponding to the inside of the screen of the display unit 5 (No in step S201), the process of step S202 is skipped and the process proceeds to step S104.

  Further, the speech recognition unit 13 refers to a recognition dictionary (not shown) and performs speech recognition such as word spotting on the speech signal input from the microphone 3 (step S104). Then, the explanation unit extraction unit 15a divides the slide being displayed on the display unit 5 into units of paragraphs, lines, one sentence, and the like using indent information included in the slide, and the like. The coordinate range and the words included in the section are extracted as explanation units (step S105).

  Thereafter, the associating unit 15b associates the result of the speech recognition in step S104 with the explanation unit extracted in step S105 (step S106). That is, the associating unit 15b outputs the explanation unit containing the most words that match the word recognized in step S104 among the explanation units extracted in step S105 as the explanation location corresponding to the speech recognition.

  In addition, the explanation state determination unit 16 determines that the explanation state is “matched state”, “mismatched state”, or “based on the result of line-of-sight determination in steps S102 to S103 and the result of voice recognition association in step S106. It is determined which of the “non-visual state” corresponds to the state (step S107).

  And the description location estimation part 22 estimates an explanation location from the description state determined by step S107, and the determination result of the reading aloud state determined by step S202 (step S203). After that, the highlighting control unit 18 executes highlighting of the explanation portion estimated in step S203 (step S109), and ends the process.

[One aspect of effect]
As described above, the explanation support apparatus 20 according to the present embodiment determines whether the reading state or the silent reading state is determined from the moving direction of the line of sight and the moving speed of the line of sight, and depends on whether the reading state or the silent reading state is set. Distinguish between principles and exceptions of consistent and inconsistent states. Therefore, according to the explanation support apparatus 20 according to the present embodiment, as in the first embodiment, it is possible to suppress a decrease in the estimation accuracy of the explanation location.

  Although the embodiments related to the disclosed apparatus have been described above, the present invention may be implemented in various different forms other than the above-described embodiments. Therefore, another embodiment included in the present invention will be described below.

[Application examples of document files]
In the first embodiment, the case where a document created by presentation software is used has been exemplified. However, a document created by another application program can also be used. In other words, if the document file includes a page that is displayed on a screen-by-screen basis, the page of the word processing software document file is replaced with a slide, or the sheet of the spreadsheet software document file is replaced with a slide. The process shown in FIG. 8 can be applied in the same manner.

[Other implementation examples]
In the first embodiment, the explanation support apparatus 10 exemplifies the case where the explanation support processing is executed in a stand-alone manner in which the presentation software is independently executed without depending on an external resource. Can also be adopted. For example, a client server system can be constructed by providing a server that provides an explanation support service corresponding to the explanation support process described above for a client that executes presentation software. In this case, the server device can be implemented by installing an explanation support program for realizing the explanation support service as package software or online software. For example, the server device may be implemented as a Web server that provides the above explanation support service, or may be implemented as a cloud that provides the above explanation support service by outsourcing. In this case, the client starts the presentation after uploading the document used for the presentation and the identification information of the venue where the presentation is performed to the server device. When the presentation is started, the client uploads the audio signal collected from the microphone 3 in real time, and uploads the slide page information every time the slide page being displayed on the display unit 5 is switched. As a result, the server apparatus can execute the processes shown in FIG. 4 and FIG. Furthermore, the client can also be constructed as a thin client system by transmitting operation information related to an input device (not shown) to the server and displaying only the processing result transmitted from the server on the display unit 5. In this case, various resources such as document data are also held by the server, and the presentation software is also implemented as a virtual machine on the server. In the first embodiment, it is assumed that the explanation support program is added to the presentation software. However, when a request for referring to the explanation support program as a library is received from a client having a license authority, the explanation support program is executed. It can also be plugged in.

[Distribution and integration]
In addition, each component of each illustrated apparatus does not necessarily have to be physically configured as illustrated. In other words, the specific form of distribution / integration of each device is not limited to that shown in the figure, and all or a part thereof may be functionally or physically distributed or arbitrarily distributed in arbitrary units according to various loads or usage conditions. Can be integrated and configured. For example, the explanation support apparatus 10 includes the gaze detection unit 11, the gaze determination unit 12, the voice recognition unit 13, the document acquisition unit 14, the recognition result determination unit 15, the explanation state determination unit 16, the explanation location estimation unit 17, or the highlight display control unit 18. You may make it connect via a network as an external apparatus. Further, another device includes the line-of-sight detection unit 11, the line-of-sight determination unit 12, the voice recognition unit 13, the document acquisition unit 14, the recognition result determination unit 15, the description state determination unit 16, the description location estimation unit 17, or the highlight display control unit 18. The functions of the explanation support apparatus 10 described above may be realized by having each connected and connected via a network.

[Explanation support program]
The various processes described in the above embodiments can be realized by executing a prepared program on a computer such as a personal computer or a workstation. Therefore, in the following, an example of a computer that executes an explanation support program having the same function as in the above embodiment will be described with reference to FIG.

  FIG. 9 is a diagram illustrating a hardware configuration example of a computer that executes the explanation support program according to the first embodiment and the second embodiment. As illustrated in FIG. 9, the computer 100 includes an operation unit 110 a, a speaker 110 b, a camera 110 c, a display 120, and a communication unit 130. Further, the computer 100 includes a CPU 150, a ROM 160, an HDD 170, and a RAM 180. These units 110 to 180 are connected via a bus 140.

  As shown in FIG. 9, the HDD 170 stores an explanation support program 170 a that exhibits the same function as each processing unit shown in the first and second embodiments. This explanation support program 170a may be integrated or separated, like each component of each processing unit shown in FIG. 1 or FIG. That is, the HDD 170 does not necessarily have to store all the data shown in the first embodiment, and data used for processing may be stored in the HDD 170.

  Under such an environment, the CPU 150 reads the explanation support program 170 a from the HDD 170 and expands it on the RAM 180. As a result, the explanation support program 170a functions as an explanation support process 180a as shown in FIG. The explanation support process 180a develops various data read from the HDD 170 in an area allocated to the explanation support process 180a in the storage area of the RAM 180, and executes various processes using the developed various data. For example, as an example of processing executed by the explanation support process 180a, processing shown in FIGS. 4 and 8 and the like are included. Note that the CPU 150 does not necessarily operate all the processing units described in the first embodiment, and the processing unit corresponding to the process to be executed may be virtually realized.

  Note that the explanation support program 170a is not necessarily stored in the HDD 170 or the ROM 160 from the beginning. For example, each program is stored in a “portable physical medium” such as a flexible disk inserted into the computer 100, so-called FD, CD-ROM, DVD disk, magneto-optical disk, or IC card. Then, the computer 100 may acquire and execute each program from these portable physical media. Each program is stored in another computer or server device connected to the computer 100 via a public line, the Internet, a LAN, a WAN, etc., and the computer 100 acquires and executes each program from these. It may be.

DESCRIPTION OF SYMBOLS 1 Camera 3 Microphone 5 Display part 10 Description assistance apparatus 11 Eye-gaze detection part 12 Eye-gaze determination part 12a Stop point detection part 12b In-screen determination part 13 Voice recognition part 14 Document acquisition part 15 Recognition result determination part 15a Explanation unit extraction part 15b Correspondence 16 Description state determination unit 16a History storage unit 17 Description location estimation unit 18 Highlight display control unit

Claims (5)

A line-of-sight detection unit that performs line-of-sight detection on a predetermined display unit;
A voice recognition unit for performing voice recognition;
A recognition result determination unit for determining an explanation location corresponding to the result of the voice recognition among the pages of the document displayed on the display unit;
The first state where the explanation location corresponding to the speech recognition and the explanation location corresponding to the line-of-sight detection match, the second state where the explanation location corresponding to the speech recognition and the description location corresponding to the line-of-sight detection do not match An explanation state determination unit that determines which one of the third states in which the state or the position of the line of sight by the detection of the line of sight is not detected within the screen of the display unit;
An explanation support device comprising: an estimation unit that estimates an explanation location for executing highlighting based on the determined explanation state. A history storage unit for storing the history of the explanation state;
The estimation unit determines whether the speech recognition and the line of sight depend on whether or not the explanation state has continuity from the explanation state determined by the explanation state determination unit and the history of the explanation state stored in the history storage unit. Whether the explanation location where the detection results match is estimated as the explanation location where the highlighting is performed, the explanation location corresponding to the speech recognition is estimated as the explanation location where the highlighting is performed, or estimated last time The explanation support apparatus according to claim 1, wherein it is determined whether to continue the explanation part as the explanation part for executing the highlighting. A reading state determination unit that determines whether the reading state or the silent reading state is based on at least one of the moving direction of the line of sight obtained from the result of the line of sight detection and the moving direction of the line of sight obtained from the result of the line of sight detection. Further comprising
The estimation unit includes an explanation location where the speech recognition result and the line-of-sight detection result match according to the explanation state determined by the explanation state determination unit and the reading state or silent reading state determined by the reading state determination unit, the sound The explanation support apparatus according to claim 1, wherein an explanation location corresponding to recognition or an explanation location corresponding to the visual line detection is to be estimated as an explanation location for executing the highlighting. Computer
Execute line of sight detection for a given display,
Perform speech recognition,
Determining an explanation location corresponding to the result of the voice recognition among the pages of the document displayed on the display unit;
The first state where the explanation location corresponding to the speech recognition and the explanation location corresponding to the line-of-sight detection match, the second state where the explanation location corresponding to the speech recognition and the description location corresponding to the line-of-sight detection do not match A state or a position of the line of sight by the line-of-sight detection is determined in which of the third states that are not detected in the screen of the display unit,
An explanation support method, comprising: executing a process of estimating an explanation location for executing highlighting based on a determined explanation state. On the computer,
Execute line of sight detection for a given display,
Perform speech recognition,
Determining an explanation location corresponding to the result of the voice recognition among the pages of the document displayed on the display unit;
The first state where the explanation location corresponding to the speech recognition and the explanation location corresponding to the line-of-sight detection match, the second state where the explanation location corresponding to the speech recognition and the description location corresponding to the line-of-sight detection do not match A state or a position of the line of sight by the line-of-sight detection is determined in which of the third states that are not detected in the screen of the display unit,
An explanation support program characterized by causing a process of estimating an explanation location for executing highlighting based on a determined explanation state to be executed.

Images